Problem interpreting gzipped instance catalogs

[Thomas Glanzman]

The attached sample instance catalog, if gzipped, causes phoSim 3.4.2 to fail quietly (return code = 0). The log typically shows this:

Atmosphere Creator

Creating layer 0. Creating layer 1. Creating layer 2. Creating layer 3. Creating layer 4. Creating layer 5. Creating layer 6.

Instrument Configuration

---

then quietly dies.

The invocation of phoSim looks like this (simplified by removing lengthy directory paths) running on a Linux RHEL6-64 machine:

$ time .../3.4.2/phosim.py phosim_input_484933.txt.gz -s R22_S11 --sed .../SEDs/2016.01.26 -w work -o output

And the example instance catalog (uncompressed!) looks like this:

Opsim_obshistid 484933 SIM_SEED 61092532 Unrefracted_RA 53.0091385 Unrefracted_Dec -27.4389488 Opsim_moonra 207.173231 Opsim_moondec -11.0627264 Opsim_rotskypos 111.907939 Opsim_filter 4 Opsim_rawseeing 1.031902 Opsim_sunalt -26.4240114 Opsim_moonalt -47.1439987 Opsim_dist2moon 134.069431 Opsim_moonphase 15.782393 Opsim_expmjd 60287.0895 Opsim_altitude 77.4473354 Opsim_azimuth 80.5436566 exptime 30 airmass 1.024489 SIM_NSNAP 1 SIM_VISTIME 30 object 992887068677 53.0511139 -27.3601005 26.7201543 starSED/phoSimMLT/lte027-3.0-0.0a+0.0.BT-Settl.spec.gz 0 0 0 0 0 0 point CCM 0.0236970962 3.1 none object 61442015851546 53.2119138 -27.2931146 28.2196407 galaxySED/Burst.25E09.02Z.spec.gz 2.1556139 0 0 0 0 0 sersic2d 0.0960609987 0.0795992017 171.844421 4 CCM 0.0202780404 3.1 CCM 0.300000012 3.0999999 object 61457839745052 52.9157824 -27.1673504 29.6205975 agnSED/agn.spec.gz 2.17864656 0 0 0 0 0 point CCM 0.0239470335 3.1 none object 6144677134336042 53.0669558 -27.3559054 23.0794962 spectra_files/specFile_6000661264000_60287.0895_z.dat 0.4842 0 0 0 0 0 point CCM 0.0224811521 3.1 none 0.00725198456

[Thomas Glanzman]

Sorry, but bitbucket removed the linefeeds from my cut/paste of the instance catalog. I'll try again...

Opsim_obshistid 484933

SIM_SEED 61092532

Unrefracted_RA 53.0091385

Unrefracted_Dec -27.4389488

Opsim_moonra 207.173231

Opsim_moondec -11.0627264

Opsim_rotskypos 111.907939

Opsim_filter 4

Opsim_rawseeing 1.031902

Opsim_sunalt -26.4240114

Opsim_moonalt -47.1439987

Opsim_dist2moon 134.069431

Opsim_moonphase 15.782393

Opsim_expmjd 60287.0895

Opsim_altitude 77.4473354

Opsim_azimuth 80.5436566

exptime 30

airmass 1.024489

SIM_NSNAP 1

SIM_VISTIME 30

object 992887068677 53.0511139 -27.3601005 26.7201543 starSED/phoSimMLT/lte027-3.0-0.0a+0.0.BT-Settl.spec.gz 0 0 0 0 0 0 point CCM 0.0236970962 3.1 none

object 61442015851546 53.2119138 -27.2931146 28.2196407 galaxySED/Burst.25E09.02Z.spec.gz 2.1556139 0 0 0 0 0 sersic2d 0.0960609987 0.0795992017 171.844421 4 CCM 0.0202780404 3.1 CCM 0.300000012 3.0999999

object 61457839745052 52.9157824 -27.1673504 29.6205975 agnSED/agn.spec.gz 2.17864656 0 0 0 0 0 point CCM 0.0239470335 3.1 none

object 6144677134336042 53.0669558 -27.3559054 23.0794962 spectra_files/specFile_6000661264000_60287.0895_z.dat 0.4842 0 0 0 0 0 point CCM 0.0224811521 3.1 none 0.00725198456

[John Peterson]

i'm not sure why it quietly dies, but it should immediately die. so we'll look into this.

you should use the includeobj command though and reference gzipped object catalogs (since that is 99.9999999% of the bytes). you know what i mean?

[John Peterson]

i guess it is just meaningless bytes you are feeding it, since the python does not consider a gzipped instance catalog valid. i'm not sure we need to do anything different for this, as if someone provides a catalog with no sources its not necessarily an error. i'm inclined to close this.

[Thomas Glanzman]

@JohnPeterson Somewhere along the line I had been led to believe that phoSim could interpret gzipped instanceCatalogs. From what you say, that is not true (or no longer true). That being the case your explanation is not unreasonable and I would agree about closing this issue.

Given that instance catalogs (or their included object files) can be huge, would it not make sense to teach phoSim to detect and read gzipped files? (One can use the gzip module and 'gzip.open', for example.)

[John Peterson]

Tom, it can read gzipped catalogs and you certainly should! You just have to include them with the includeobj command:

includeobj subcatalog1.gz includeobj subcatalog2.gz

So you just can't gzip the main catalog with the ~20 commands. I see no point to that since it is a trivial amount of data.

[John Peterson]

those includeobj's should be on two lines. this keeps reformatting things.