Discussion: Bad Band List in Spectral Profiles and EnMAP-Box

[Benjamin Jakimow]

Background

The ENVI format specifies a band band list (bbl), which is a list of “bad band multiplier values of each band in an image, typically o for bad band and 1 for good bands” (https://www.l3harrisgeospatial.com/docs/enviheaderfiles.html)

I don’t know how often this metadata information is really used in the community. If the idea of a bbl is to “mask”, i.e. not use bad bands, than I’d remove these bands first and continue data processing with the remaining band. For example using a virtual VRT image that contains the “good bands” only.

A single spectral profile is basically a dictionary which can contain the following values:

{
'y':<required, list of spectral response values
'x':<optional, list of wavelength values, defaults to band number>,
'xUnit':<optional, wavelength unit string, e.g. 'nm'
'yUnit':<optional, unit/description of values in 'y', e.g. 'reflectance'>
'bbl':<optional, bad band list, True = good band, False = bad band>
}

As this example show, 'y' is the only information really required, but the dictionary can be expanded with any kind of other information.

In the QGIS Field Calculator these dictionary items can be accesses using the spectralData(…) function:

‌

To store the profile, it gets encoded by def encodeProfileValueDict(d: dict) -> QByteArray: and written into a spectral profile (QByteArray) field of a QgsVectorLayer.
To restore the profile, the QByteArray gets deserialized with def decodeProfileValueDict(dump: QByteArray, numpy_arrays: bool = False) -> dict:

I’d like to keep these dictionary as small as each for thee reasons:

1. to reduce the time that is required to encode/decode these dictionaries into/from QByteArrays, for example to plot them. This is as more important for larger spectral libraries with > 2000 profiles

2. to provide as much other profile metadata in normal attribute table fields, where they can be viewed and manipulated using the known standard QGIS tools.

Questions
1. My major question is if such a bbl should still be part of the information that is stored in spectral libraries per single spectral profile?
2. If bands are considered as “bad”, why do they still need to be stored? Wouldn’t it be much more consequent to exclude them in the beginning of a processing chain (QGIS Processing Model) or exclude them when collecting profiles?

[Andreas Janz]

My major question is if such a bbl should still be part of the information that is stored in spectral libraries per single spectral profile?

From a user perspective it makes sense to keep the BBL. E.g. when visualizing pixel profiles from a raster with bad bands, the user can see the bad bands in the plot. Via a “Show Bad Bands“ checkbox, the user should be able to show/hide bad bands.

‌

If bands are considered as “bad”, why do they still need to be stored?

For example, if the user collecs a training dataset from a raster with bad bands, trains a model and wants to apply it to the original raster again. If we would have silently deleted the bad bands in the library, the number of features in the classifier wouldn’t match the number of bands in the raster. That may be confusing.

Wouldn’t it be much more consequent to exclude them in the beginning of a processing chain (QGIS Processing Model) or exclude them when collecting profiles?

That would be a general decision for the EnMAP-Box. Do we concider bad bands in tools, algorithms and visualization, or do we ignore bad bands for the most part.

‌

@Agustin Lobo maybe you have a suggestion on that?

[Agustin Lobo]

I would prefer keeping the bbl. Note that if deleted, band i.e 54 will not be band 54 any more and it could be confusing. On the other hand, any user not liking the bbl could just delete all bad bands and have all bands in bbl set to correct.

[Benjamin Jakimow]

Are there other needs for storing the BBL than to maintain the number of bands? Or asked differently, are there use-cases where a spectral value is needed again, once he got labeled as “bad”?

I am asking because such “bad bands” could be indicated internally by None or NaN values as well, which saves some space and time.

Now:
y = [234, 43, 345, ... ]
bbl = [True, False, True, ...]

without bbl but same number of bands
y = [234, None, 345, ...]

‌

[Andreas Janz]

| Are there other needs for storing the BBL than to maintain the number of bands? Or asked differently, are there use-cases where a spectral value is needed again, once he got labeled as “bad”?

The usecase from above:
”From a user perspective it makes sense to keep the BBL. E.g. when visualizing pixel profiles from a raster with bad bands, the user can see the bad bands in the plot. Via a “Show Bad Bands“ checkbox, the user should be able to show/hide bad bands.”

[Andreas Janz]

Note that Sebastian also wants to discuss the BBL in the next team meeting with us.

[Andreas Janz]

Regarding the argument that BBL information needs extra space:

Most of the time the BBL won’t be defined. Instead of storing a list with many True values. Just store BBL = None. This is super light-weight and fast to check. Only if BBL is not None, you do the extra work.

Does that solve the problem with space and time you mentioned earlier?

[Agustin Lobo]

I am asking because such “bad bands” could be indicated internally by None or NaN values as well, which saves some space and time.

When working with R, I routinely set values of bad bands to NA for any plot or processing, and wish to be able of so doing

with enmapbox

‌

[Andreas Janz]

Today we decided in the team meeting to support bad bands on a per profile level.

In profile plotting the user should have a checkbox to show/hide values from bad bands.

See original issue #903.