Translate raster layer creates wrong .hdr file

[Agustin Lobo]

The .hdr band names created by “Translate raster layer” has a band names field such as:

band names = {
band 1 (402.4488220214844 Nanometers),
band 2 (411.3247985839844 Nanometers),
band 3 (419.3800354003906 Nanometers), …

and there are no wavelength field, which should be:

wavelength = {402.49115, 411.36606, 419.41736, …

and no wavelength units field

wavelength units = nm

Also, probably because it is not possible to select the original .he5 file, the fwhm filed is also missing

fwhm = {11.411813, 10.549804, 9.8321686,

[Andreas Janz]

Hi Agustin,

what you describe is expected QGIS/GDAL behaviour. Writing the ENVI *.hdr header not well supported via the API. Instead of writing an ENVI file, you should keep the PRISMA GeoTiff you already have and use this algorithm to append an ENVI header file to it:

‌

Now you can open the GeoTiff in ENVI.

Having said that, while testing it with a PRISMA product, I noticed a bug.

‌

I will fix this.

As a workaround for now, you can edit this file in your local installation:

C:\Users\Andreas\AppData\Roaming\QGIS\QGIS3\profiles\default\python\plugins\enmapboxplugin\enmapboxprocessing\rasterreader.py

Add those two lines:

‌

[Andreas Janz]

resolves #884

→ <<cset 940d811acabf>>

[Andreas Janz]

Should work now.

‌

[Agustin Lobo]

Thanks Andreas.

In the case of PRISMA images, which originally are .he5 files,

1. I presume that making the hdr from the tif (instead of the original .he5 file) implies that some information is no longer available for writing the .hdr
2. I might be wrong, but think that geotif are BSQ always. Having BIL files for hyperspectral images is not more efficient for at least some processes in EnMap-Box?

Therefore, I think that an independent tool for writing ENVI images with a correct hdr file ()avoiding the current API) would be useful.

‌

‌

[Andreas Janz]

1. We never use the HE5 file directly in the EnMAP-Box. We always import it and create a new GeoTiff with proper wavelength metadata (if you are missing any HE5 metadata in the GeoTiff, I’m happy to add those). So the GeoTiff has all information attached.

2. No, GeoTiffs can be BIP as well. In fact, the GDAL default is BIP, but we changed that to BSQ for the EnMAP-Box because compressed BIP GeoTiffs are painfully slow for visualization. If you really want BIP, you can define it here:

   

Having said that, in my experience, you really don’t want to use BIP in QGIS/GDAL, because both APIs are fully band-focussed and do not benefit from BIP at all.

No EnMAP-Box algorithm will be faster on BIP files!

Therefore, I think that an independent tool for writing ENVI images with a correct hdr file ()avoiding the current API) would be useful.

No, I would recommend to always use GeoTiff instead of ENVI format. Even ENVI software will handle the GeoTiffs very well.

‌

[Andreas Janz]

Agus, if you have any usecase where you think you really need ENVI format, because GeoTiff want work, I’m happy to have a look at it.

[Agustin Lobo]

Thanks again for your prompt reaction.

Note that I refer to BIL, not to BIP.

I can understand BIP is slower for display, perhaps BIL as well (although envi or HypPy do not have an slow display), but do not understand why BIP and BIL files are not much faster for processing, i.e. PCA (which, btw, seems to me much slower in enmap-box than in other toolboxes susch as HypPy; could not the BSQ vs BIP/BIL be the reason?).

Anyway, the original issue involved in this ticket has been reasonably resolved.

[Andreas Janz]

Note that I talked about compressed BIP.
I have not excessively tested BIL, but it should have the same disadvantages as BIP.

And also in ENVI, try to visualize a compressed ENVI BIL with hundreds of bands. First bands are visualized quite fast, but bands with higher band number get slower and slower. That is because it takes longer to decompress data that in “deeper” in the byte stream.

In GeoTiff BSQ this is not the case, because data is compressed band-wise. And for really large spatial extents, you can used tiled-GeoTiffs. At the end of the day, GeoTiff is the better format I would say, especially when data is compressed.

PCA speed may be an issue because of how we do the block-wise reading. I want to overhaul all the transformers in near future. But it may also be that the speed is mainly influenced by scikit-learn. Not sure about that.

[Agustin Lobo]

I referred to uncompressed BIL, never even heard about the compressed versions. All I know is the classic raw image (size= nb_cols x nb_rows * nb_bands * nb_bits/pixel) + .hdr file.

There are 2 separate issues:

1. Using envi files in uncompressed BIL/BIP within EnMapBox: I cannot argue on that, I trust your opinion. I’m just surprised that you prefer BSQ as it makes sense that BIL be faster for processing, and that agrees with the fact that PCA is a lot faster in HypPy than in EnMapBox. But there might be other reasons for this difference.
2. In my opinion, EnMapBox “Translate raster layer” tool should write a correct hdr even if QGIS does not. Envi format is still the most common format for hyperspectral imagery. This should not be a big issue, as you write a correct hdr file with “Append ENVI header”. I neede to write a BIL image for HypPy (because of #882) and had to edit the hdr to make it right.

In any case, this is really not a big issue.

‌

[Andreas Janz]

Hi Agustin, I’m not familiar with the HypPy package and can’t find it, please provide a link.

[Andreas Janz]

I referred to uncompressed BIL, never even heard about the compressed versions.

You’re right, in ENVI you can only create compressed BSQ.

I’m just surprised that you prefer BSQ as it makes sense that BIL be faster for processing, and that agrees with the fact that PCA is a lot faster in HypPy than in EnMapBox.

The main “problem” is the GDAL/QGIS API, which read the data band-wise. BIL or BIP will always be slower. I guess HypPy uses a different API for reading ENVI files that benefit from BIL/BIP. I’ll check this.

‌

In my opinion, EnMapBox “Translate raster layer” tool should write a correct hdr even if QGIS does not.

Maybe you are right: #888

[Agustin Lobo]

Hi Agustin, I’m not familiar with the HypPy package and can’t find it, please provide a link.

HypPy: https://blog.utwente.nl/bakker/hyppy/

‌

[Andreas Janz]

@Agustin Lobo writing an ENVI file will now also write a correct ENVI header.

[Andreas Janz]

resolves #884 and #888

→ <<cset 075cc460b6bf>>