1. HUB Geomatics Lab Avatar HUB Geomatics Lab
  2. EnMAP-Box
  3. enmap-box
  4. Issues

Translate raster: Bad bands not eliminated despite having set 'excludeBadBands' : True

Issue #1348 resolved
Agustin Lobo created an issue

Bad bands not eliminated despite having set 'excludeBadBands' : True

from the resulting hdr:

bbl = {1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0}

‌

Version 3.10.0.20220401T124144.develop c7d679b

QGIS version: 3.24.2-Tisler
QGIS code revision: 13c1a02865
Qt version: 5.15.2
Python version: 3.9.5
GDAL version: 3.2.2
GEOS version: 3.9.0-CAPI-1.16.2
PROJ version: Rel. 7.2.1, January 1st, 2021
PDAL version: 2.2.0 (git-version: Release)
Algorithm started at: 2022-05-02T20:18:31
Algorithm 'Translate raster layer' starting…
Input parameters:
{ 'bandList' : [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234], 'copyMetadata' : True, 'copyStyle' : True, 'creationProfile' : 'GTiff INTERLEAVE=BAND', 'dataType' : 5, 'excludeBadBands' : True, 'extent' : '-6.632194990,-6.522305681,37.665109210,37.740793857 [EPSG:4326]', 'grid' : None, 'noData' : None, 'offset' : None, 'outputTranslatedRaster' : '/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/v2/RT1_PRS_20210625_SR.tif', 'raster' : '/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/v2/PRS_L2D_STD_20210625111917_20210625111921_0001_SR.tif', 'resampleAlg' : 0, 'scale' : None, 'sourceColumns' : [nan,nan], 'sourceNoData' : None, 'sourceRows' : [nan,nan], 'spectralBandList' : None, 'spectralSubset' : None, 'unsetNoData' : False, 'unsetSourceNoData' : False, 'workingType' : None, 'writeEnviHeader' : True }

Translate raster 331x288x234 -of GTiff -co INTERLEAVE=BAND /media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/v2/RT1_PRS_20210625_SR.tif
Execution completed in 2.06 seconds
Execution completed in 2.17 seconds
Results:
{'outputTranslatedRaster': '/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/v2/RT1_PRS_20210625_SR.tif'}

Loading resulting layers
Algorithm 'Translate raster layer' finished

‌

Comments (29)

  5. Andreas Rabe

    Hmm, something is wrong with your raster.

    1. It is a GeoTiff, so the additional .hdr file is only used by ENVI software and is totally ignored by QGIS/EnMAP-Box.

    2. The BBL information is only stored inside the .hdr file:

    ‌

    3. The BBL information is not stored inside the .tif or the .tif.aux.xml files. You can see that inside the Layer Properties. The last bands should all be BBL=0:

    ‌

    Having said that, it could still be a bug.

    How was the .hdr file created?

    Have you edited the *hdr file? If so, note that EnMAP-Box isn’t aware of those edits.

  6. Andreas Rabe

    In general, I think you should use an image with bbl for testing, it would be closer to reality.

    Yes, I agree.

    As soon as real EnMAP data is available, we will update the testdata and include BBL information.

  7. Agustin Lobo reporter
    1. Here you have it in plain envi: https://www.dropbox.com/s/m2fkj70z74v4j1n/PRISMA_DESTRIPPED_AOIvAL.7z?dl=0
    2. Regarding

    It is a GeoTiff, so the additional .hdr file is only used by ENVI software and is totally ignored by QGIS/EnMAP-Box.

    I think those EnMapBox tools aimed to discard bad bands should not ignore the hdr even in the case of GTiff and other formats. Note your Translate

    has the option of writing a hdr file even at exporting to GTiff.

    I resignedly accept that EnMapBox is not dealing with the bbl in the processing, but at least those tools aimed to discard the bad bands should fully support the information in the hdr.

  8. Andreas Rabe

    I think those EnMapBox tools aimed to discard bad bands should not ignore the hdr even in the case of GTiff and other formats. Note your Translate

    has the option of writing a hdr file even at exporting to GTiff.

    I resignedly accept that EnMapBox is not dealing with the bbl in the processing, but at least those tools aimed to discard the bad bands should fully support the information in the hdr.

    I understand, why you would prefer this behaviour, but I’m still hesitant implementing it, because it is not in line with the GDAL/QGIS data model.

    Instead of editing the .hdr file, you need to edit the .tif.aux.xml file:

    ‌

    @Benjamin Jakimow , any thoughts on that from your side?

  9. Agustin Lobo reporter

    Accept both. Editing the hdr is the de facto standard for spectral information in hyperspectral imagery

    (as you actually recognize by having the option to write hdr files for non-envi formats in Translate )

    ‌

  10. Andreas Rabe

    Accept both. Editing the hdr is the de facto standard for spectral information in hyperspectral imagery

    Sorry, we don’t want to alter the behaviour of the GDAL data model.

    ‌

    (as you actually recognize by having the option to write hdr files for non-envi formats in Translate )

    We only do that to support opening the raster in ENVI Software with proper metadata.

  12. Agustin Lobo reporter

    You do not alter the GDAL data model. You read through gdal and then check the hdr to eliminate the bad bands

    as requested by the user (option excludeBadBands and hdr present. You can even specifically request the user

    to enter the name of the hdr, i.e. have the option excludeBadBands from hdr)

    ‌

  13. Benjamin Jakimow

    @Agustin Lobo the PRISMA_DESTRIPPED_AOIvAL.hdr contains two differing bbl definitions. Only the 2nd contains masked bands:

    ‌

  14. Agustin Lobo reporter

    Note that in this setup you are allowed to edit the *.hdr file:

    I do not understand what you mean, you can always edit the hdr. But the issue of GTiff and hdr

    files should be separated.

    Anyway, I think this current issue can be closed. The problem was that I was interpreting that I needed to

    set Bad bands as TRUE to get them out of the spectrum, while this option must be set to FALSE

    (I would interpret “Bad bands: false” as “not to consider Bad bands”. Perhaps this option should be clarified, eg

    “Bad bands: included” and Bad Bands: excluded”). I would have bad bands excluded by default.

    sorry, this was for #903

  15. Benjamin Jakimow

    But even removing the first bbl definition, so that that one with '0' values remain, I can confirm that using the enmpapbox:TranslateRasterLayer algorithms with exclude bad bands checked fails to remove these bands

    Other parts of the EnMAP-Box like the band properties and the spectral profiles recognize the BBL values well:

    ‌

  17. Andreas Rabe

    Ok, this is a different dataset. Now it is an ENVI file and not a GeoTiff anymore:

    ‌

    In this case you are allowed to edit the BBL information inside the *.hdr file. GDAL/QGIS will read the metadata correctly.

    BUT, after you opened the file inside QGIS, a *.aux.xml file will be created. From now on, all edits inside the *.hdr file will be ignored by GDAL/QGIS!

    ‌

    I know that this is complicated, but that is the GDAL datamodel.

  19. Agustin Lobo reporter

    Just to clarify the current situation with Version 3.10.3.20220824T155109.master:

    Input PRISMA_DESTRIPPED_AOIvAL.bsq + PRISMA_DESTRIPPED_AOIvAL.hdr

    Case 1. Output as GTiff, “Exclude Bad bands” not selected, write envi hdr

    ‌

    Output; hdr is wrong (bbl are all 1)

    Gtiff with all bands, but no bands are marked as bad according to Layer properties/Gdal metadata

    ‌

    Case 2: Output as GTiff, “Exclude Bad bands” selected, write envi hdr

    ‌

    Output identical to case 1 (all 234 bands despite having requested exclude bad bands)

  21. Agustin Lobo reporter

    I have deleted all xml files: same results, resulting GTiff keeps all bands.

    Anyway, even if it ha worked in this case, the philosophy behind:

    BUT, after you opened the file inside QGIS, a *.aux.xml file will be created. From now on, all edits inside the *.hdr file will be ignored by GDAL/QGIS!

    I know that this is complicated, but that is the GDAL datamodel.

    is absurd:

    1. XML files are always silently written, even if displaying the image in enmaptoolbox. Note you always display an image. Actually, you have to.
    2. If the presence of an XML implies ignoring the hdr, enmaptoolbox does not support the envi format, de facto.
    3. I understand you must conform to gdal for reading/writing, but once you have read the file, there is no reason to do not read the hdr if you need it, as in the case of identifying bad bands.

    ‌

  23. Andreas Rabe

    I understand you must conform to gdal for reading/writing, but once you have read the file, there is no reason to do not read the hdr if you need it, as in the case of identifying bad bands.

    Sorry, this would make it to complicated.

    What if BBL information is also stored inside the aux.xml file? Should that information be ignored?

    What if the user deletes the BBL from the *.hdr, but BBL is still stored inside the aux.xml or inside the *.tif file? Would you expect to have no BBL, because you deleted it from the *.hdr manually?

    What about all the other metadata? Another user might want to also edit the wavelength inside the *.hdr or the data offset and gain.

    I really don’t want to deal with those details. I also don’t want to implement special case solutions for some of the algorithms.

    Maybe @Benjamin Jakimow has a different thought on that?

  24. Agustin Lobo reporter

    What if BBL information is also stored inside the aux.xml file? Should that information be ignored?

    In the case of an envi file, certainly yes.

    What if the user deletes the BBL from the *.hdr, but BBL is still stored inside the aux.xml or inside the *.tif file? Would you expect to have no BBL, because you deleted it from the *.hdr manually?

    In the case of an envi file, it is the user responsibility taking care of the hdr. That’s the way it is with that format. In the case of other formats, just have an option to give preference to the hdr

    (in the few apps, such as this one, in which enmapbox deals with envi files). We can add this option as a wish, and ignore hdr files for non envi formats by now (but make it clear to the user).

    What about all the other metadata? Another user might want to also edit the wavelength inside the *.hdr or the data offset and gain.

    The same. hdr must have the preference for envi format files. Otherwise, you really do not support envi format files and it becomes confusing to the user.

    I really don’t want to deal with those details. I also don’t want to implement special case solutions for some of the algorithms.

    The point is that GDAl does not deal with envi format well, so for those few apps in which you support envi format you must deal with the hdr in your own way.

  25. Andreas Rabe

    Just to clarify the current situation with Version 3.10.3.20220824T155109.master:

    ‌

    Input PRISMA_DESTRIPPED_AOIvAL.bsq + PRISMA_DESTRIPPED_AOIvAL.hdr

    ‌

    Case 1. Output as GTiff, “Exclude Bad bands” not selected, write envi hdr

    Output; hdr is wrong (bbl are all 1)

    Gtiff with all bands, but no bands are marked as bad according to Layer properties/Gdal metadata

    ‌

    Case 2: Output as GTiff, “Exclude Bad bands” selected, write envi hdr

    Output identical to case 1 (all 234 bands despite having requested exclude bad bands)

    Both cases are fixed now.

  30. Log in to comment
Assignee
Type
bug
Priority
critical
Status
resolved
Component
Processing
Milestone
3.10 (Hotfix)
Version
–
Votes
0
Watchers
1
Jira: the preferred issue tracker for Bitbucket. Join the team!