OMNI data import does not work

[cpescou]

OMINI data have changed link and format and the OVT import of data and reading would need to be updated:

1. The complete files are now at: https://spdf.sci.gsfc.nasa.gov/pub/data/omni/low_res_omni/extended/ and not https://spdf.sci.gsfc.nasa.gov/pub/data/omni/low_res_omni/

2. The format changed with the last parameter removed in new files

3. The fix on the User guide does not work anymore

4. I could make it work by copying an OMNI file from an old version of OVT 3.0 with old cached files but the only system does not work.

[Erik Johansson]

This is a bit odd. As far as I can see from the OMNI2 pages, there are still OMNI2 files on the same old format available, in parallel with files on the new format. Or have they stated that they will discontinue the old format somewhere (omni2_%4d.dat)?

I find that I can use both of the settings below in ovt.conf (Linux): (1) downloading from FTP, and (2) from a local directory with files (on the old format) which I had just (today) downloaded from the OMNI2 website. OMNI2.hourlyAveraged.urlPattern=ftp\://spdf.gsfc.nasa.gov/pub/data/omni/low_res_omni/omni2_%4d.dat OMNI2.hourlyAveraged.urlPattern=file\:///home/erjo/temp/OMNI2_just_downloaded/omni2_%4d.dat (I run Linux typically, and I noted that the file URL does indeed have to start with three slashes, not two, before the first directory name. I am guessing it is the same on Mac OS.)

Have you made sure to delete old/all files in the OVT OMNI2 cache directory (cache/OMNI2/) before trying a new setting? (A failed attempt can generate a fake data file that is in reality an webpage with an error message or something similar, as you have noticed.)

I see that you tried omni_m2015.dat (with the "m") and (ovt.conf) file\://Users/pescoube/Library/ovt/3.0/cache/OMNI2/omni_m%4d.data That for sure should not work because OVT can not read the file format and I am guessing also because the URL is missing a slash (two instead of three).

(ovt.conf) https\://spdf.sci.gsfc.nasa.gov/pub/data/omni/low_res_omni/extended/omni2_%4d.dat (with http/https/ftp/file) should also not work since the file format is different.

[cpescou]

Thanks. From home where I can get ftp working and I used the conf file with: OMNI2.hourlyAveraged.urlPattern=ftp\://spdf.sci.gsfc.nasa.gov/pub/data/omni/low_res_omni/extended/omni2_%4d.dat

I deleted all files in cache OMNI2

and then after running OVT I got an error message: I/O error when trying to obtain value of IMF .. The stream/file as a number of characters that is different from the expected number.

I see that a new omni2_2015.dat was downloaded in cache but I can see that there is an extra parameter compared to old files that worked. Here is the first line of year 2015: 2015 1 0 2475 51 51 56 36 6.1 6.0 17.5 333.7 5.1 -2.5 1.8 -2.9 1.1 0.6 1.2 0.8 0.8 0.4 142461. 3.2 566. 0.0 -2.1 0.050 2.05 52100. 0.8 13. 2.2 1.1 0.012 -0.62 0.97 8.3 23 104 -14 42 999999.99 99999.99 99999.99 0.15 0.06 0.04 -1 9 132.9 999.9 -9 33 6.1 4.57

"4.57" was not in old files.

I think the problem is the same as with https://spdf.sci.gsfc.nasa.gov/pub/data/omni/low_res_omni/extended/

[Erik Johansson]

Your tried the wrong URL. I can reproduce the error you mentioned if I use

OMNI2.hourlyAveraged.urlPattern=ftp\://spdf.sci.gsfc.nasa.gov/pub/data/omni/low_res_omni/extended/omni2_%4d.dat

as you stated, but that is not the URL pattern I used. OVT+OMNI2 works for me if I use the old URL (at least I think it is the old one), i.e.

OMNI2.hourlyAveraged.urlPattern=ftp\://spdf.gsfc.nasa.gov/pub/data/omni/low_res_omni/omni2_%4d.dat

or your URL but after removing "extended", i.e.

OMNI2.hourlyAveraged.urlPattern=ftp\://spdf.sci.gsfc.nasa.gov/pub/data/omni/low_res_omni/omni2_%4d.dat

I do NOT try to read the "extended" OMNI2 files as you did. It appears to me that both types of files are available on FTP/WWW: both the old files (which OVT can read), and the extended files with an extra column (which OVT can not read).

Compare

https://spdf.sci.gsfc.nasa.gov/pub/data/omni/low_res_omni/omni2_2015.dat

and

https://spdf.sci.gsfc.nasa.gov/pub/data/omni/low_res_omni/extended/omni2_2015.dat

It is annoying that these two types of files (extended and non-extended) have the same filenames despite being different types of files. I also note that your and my URLs have slightly different internet domains but I am guessing they are some kind of mirrors.

Just a thought which should not be the problem, but still: You could double-check that the OMNI2 URL is the one you intend it to be by double-checking the URL stated in the OMNI2 window. That will verify that OVT reads the ovt.conf that you think it is reading. (Also, I assume you have not edited OMNI2.hourlyAveraged.localCacheFileNamePattern=omni2_%4d.dat , or?)

Or am I missing something?

[cpescou]

Thanks for your extensive check. I did not realise that the address I was using was slightly different (+sci)

With: OMNI2.hourlyAveraged.urlPattern=ftp\://spdf.gsfc.nasa.gov/pub/data/omni/low_res_omni/omni2_%4d.dat

It now works from home.

I would have to check when I am back in ESTEC in two weeks time.

SPDF should really check their various server and as you said not having slightly different files with the same name.

[cpescou]

I could check with a VPN to ESTEC and ftp address above in ovt.conf does not work.

The only way is to use an ftp client and download the omni file to cache.