[Import PRISMA L2D product algo] Exception: unknown error on MacOS

[Andreas Rabe]

Reported by Babrak Azizi via email.

Run on EnMAP-Box v3.10.0.20220401T124144.

Testdata: PRS_L2D_STD_20220416061922_20220416061927_0001.he5

‌

[Andreas Rabe]

@Benjamin Jakimow tried to reproduce it on his Mac, but couldn’t reproduce the specific error.

Instead he reported:

The import starts, but using the default settings it does not finish. Neither progress is provided, nor the output file "tmpPrisma.tif" is generated.

Algorithm started at: 2022-04-27T23:16:28

Algorithm 'Import PRISMA L2D product' starting…

Input parameters:

{ 'badBandThreshold' : None, 'badPixelType' : [0], 'file' : '/Users/jakimowb/Downloads/PRS_L2C_STD_20220422062237_20220422062242_0001.he5', 'outputPrismaL2D_spectralCube' : '/Users/jakimowb/Downloads/tmpPrisma.tif', 'spectralRegion' : 0 }

‌

[Andreas Rabe]

@Benjamin Jakimow you used the wrong L2C product, please try again with the L2D product.

If still not working, please try to execute the following lines in your QGIS Python Console:

‌

import h5py
h5py.__version__
file = h5py.File(r'C:\Users\Andreas\Downloads\PRS_L2D_STD_20220416061922_20220416061927_0001.he5', 'r')
ds = file['HDFEOS/SWATHS/PRS_L2D_HCO/Data Fields/VNIR_Cube']
ds

Should give you this:

‌

‌

‌

‌

[Benjamin Jakimow]

Same for the L2D product: No progress, process never finishes

QGIS version: 3.24.2-Tisler
QGIS code revision: 573ca40c96
Qt version: 5.15.2
Python version: 3.9.5
GDAL version: 3.3.2
GEOS version: 3.9.1-CAPI-1.14.2
PROJ version: Rel. 8.1.1, September 1st, 2021
PDAL version: 2.3.0 (git-version: Release)
Algorithm started at: 2022-04-28T09:55:40
Algorithm 'Import PRISMA L2D product' starting…
Input parameters:
{ 'badBandThreshold' : None, 'badPixelType' : [0], 'file' : '/Users/jakimowb/Downloads/PRS_L2D_STD_20220416061922_20220416061927_0001.he5', 'outputPrismaL2D_spectralCube' : '/Users/jakimowb/Downloads/exampleD.tif', 'spectralRegion' : 0 }

‌

[Andreas Rabe]

And what about the code?

‌

import h5py
h5py.__version__
file = h5py.File(r'C:\Users\Andreas\Downloads\PRS_L2D_STD_20220416061922_20220416061927_0001.he5', 'r')
ds = file['HDFEOS/SWATHS/PRS_L2D_HCO/Data Fields/VNIR_Cube']
ds

‌

[Benjamin Jakimow]

@Andreas Janz not even the logfile is created

[Benjamin Jakimow]

‌

h5py.__file__
'/Applications/QGIS.app/Contents/MacOS/lib/python3.9/site-packages/h5py/__init__.py'
file = h5py.File(path)
ds = file['HDFEOS/SWATHS/PRS_L2D_HCO/Data Fields/VNIR_Cube']
ds
<HDF5 dataset "VNIR_Cube": shape (1202, 66, 1207), type "<u2">
h5py.__version__
'2.10.0'

‌

[Andreas Rabe]

Ok, that looks very good. So h5py isn’t the problem here.

Something else is.

Any idea how to narrow it down? Can you debug on your MacOS? If so, I can give you a unittest.

[Benjamin Jakimow]

However, it worked by pasting the corresponding python command to the QGIS python shell:

processing.run("enmapbox:ImportPrismaL2DProduct", {'file':'/Users/jakimowb/Downloads/PRS_L2D_STD_20220416061922_20220416061927_0001.he5','spectralRegion':0,'badBandThreshold':None,'badPixelType':[0],'outputPrismaL2D_spectralCube':'/Users/jakimowb/Downloads/exampleD.tif'})
{'outputPrismaL2D_panCube': None, 'outputPrismaL2D_panErrorMatrix': None, 'outputPrismaL2D_panGeolocationFields': None, 'outputPrismaL2D_spectralCube': '/Users/jakimowb/Downloads/exampleD.tif', 'outputPrismaL2D_spectralErrorMatrix': None, 'outputPrismaL2D_spectralGeolocationFields': None, 'outputPrismaL2D_spectralGeometricFields': None}

‌

[Andreas Rabe]

Ok, that is interesting.

[Andreas Rabe]

@Benjamin Jakimow I think I have an idea what causes the problem.

In my unit tests, everything works fine.

In the GUI I also get an error on my Windows 10:
OSError: Unable to open file (file close degree doesn't match)

Interesting fact: HE5 files can’t be opened twice at the same time (exclusive read access).

My theory is: the algorithm dialog creates a read-handle on the HE5 file, when the dialog is initialized (e.g. for checking if the file exists).

When I hard code my HE5 filepath that I want to read and select a different HE5 file in die dialog, everything works fine again.

This would also explain, why your workaround with the processing.run is working.

‌

‌

Not quite sure how to deal with it. I could replace the input file parameter with an input text parameter for specifying the filepath.

@Benjamin Jakimow any ideas from your side?

‌

[Andreas Rabe]

What also works, is to make a copy of the HE5 before reading it. The copy has no read handle and can be used without a problem. After reading I can just delete the copy again.

Only draw back: I need 2 GB of temporary disk space.

[Andreas Rabe]

@Benjamin Jakimow I implemented the fix for the OSError: Unable to open file (file close degree doesn't match), please try again on MacOS, hopefully that was the problem.

[Benjamin Jakimow]

Alright, I can reproduce the GUI error as well (QGIS / EnMAP-Box):
An error has occurred while executing Python code:

Exception: unknown

Traceback (most recent call last):
  File "/Applications/QGIS.app/Contents/MacOS/../Resources/python/plugins/processing/gui/AlgorithmDialog.py", line 273, in runAlgorithm
    'parameters': self.algorithm().asMap(parameters, self.context)
Exception: unknown

Reason: the algorithm misses the “asMap” implementation
https://qgis.org/pyqgis/3.24/core/QgsProcessingAlgorithm.html#qgis.core.QgsProcessingAlgorithm.asMap

[Andreas Rabe]

Thanks @Benjamin Jakimow .

AFAIK, I never implemented the “asMap” methode for any of my algorithms. Here is the implementation of the base class:

So I guess this is a QGIS bug on MacOS.

[Andreas Rabe]

The problem might affect all MacOS users (which is critical).

Unfortunately, I can’t investigate it on my Windows 10 system.

[Benjamin Jakimow]

This is the error that is raised when calling the “asMap” from a GUI context:

self.algorithm().asMap(parameters, self.context)
OGR(1): Cannot open /Users/jakimowb/Downloads/PRS_L2D_STD_20220416061922_20220416061927_0001.he5 ().()
ERROR: Status 3: Unable to load mesh (null)
Traceback (most recent call last):
  File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/_pydevd_bundle/pydevd_exec2.py", line 3, in Exec
    exec(exp, global_vars, local_vars)
  File "<input>", line 1, in <module>
Exception: unknown

‌

[Babrak Azizi]

I tried to open my L2D through Python Console and I got this back, but I can’t find the file om my Mac on finder.

‌

[Babrak Azizi]

I don't think that it is a macOS-only problem. I also have a pc and I have the same problem.
My theory is that the file does not open because I always get an orange warning at the top of the enmap-box. I installed all the packages too.

‌

[Benjamin Jakimow]

Hi @Babrak Azizi do you get the “yellow” warning when running the enmapbox:ImportPrismaL2DProduct Algorithm?
If it is already there when starting the EnMAP-Box you can dismiss it, as long as you don’t want to use ensomap (EnMAP Soil Mapper).

You need to replace the 'TEMPORARY_OUTPUT' with a local path, because temporary locations are deleted after the algorithms has been finished.

processing.run("enmapbox:ImportPrismaL2DProduct", 
{'file':'<source dir>/PRS_L2D_STD_20220416061922_20220416061927_0001.he5',
'spectralRegion':0,
'badBandThreshold':None,
'badPixelType':[0],
'outputPrismaL2D_spectralCube':'<destination dir>/prismaL2Dcode.tif'}
)

‌

[Benjamin Jakimow]

I could track the macOS error down to this:

‌

# to start from PyCharm/python shell, start from here
from qgis.core import QgsProcessingContext, QgsProcessingParameterFile, QgsProcessingUtils
from qgis.testing import start_app
app = start_app()

# to start in QGIS python shell, start from here
import pathlib
context = QgsProcessingContext()
parameter = QgsProcessingParameterFile('file', fileFilter='Any file (*.*)')
path = QgsProcessingUtils.generateTempFilename('tempfile.txt')
with open(path, 'w') as f:
    f.write('Dummy')
assert pathlib.Path(path).is_file()
assert parameter.checkValueIsAcceptable(path, context)
print(f'asString={parameter.valueAsString(path, context)}')
print(f'asPythonString={parameter.valueAsPythonString(path, context)}')
print(f'asJsonObject={parameter.valueAsJsonObject(path, context)}')

The valueAs… functions to call and OGR reader which fails in case of the he5 image or any, as here, an arbitrary txt file.

/Applications/QGIS.app/Contents/MacOS/bin/python3 "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/pydevd.py" --multiproc --qt-support=auto --client 127.0.0.1 --port 49522 --file /Users/jakimowb/Repositories/enmap-box/tests/src/issues/test_issue_1330_QGIS.py
Connected to pydev debugger (build 211.7442.45)
Application state:
QGIS_PREFIX_PATH env var:       /Applications/QGIS.app/contents/MacOS
Prefix:     /Applications/QGIS.app/contents/MacOS
Plugin Path:        /Applications/QGIS.app/contents/MacOS/../PlugIns/qgis
Package Data Path:  /Applications/QGIS.app/contents/MacOS/../Resources
Active Theme Name:  
Active Theme Path:  /Applications/QGIS.app/contents/MacOS/../Resources/resources/themes//icons/
Default Theme Path: :/images/themes/default/
SVG Search Paths:   /Applications/QGIS.app/contents/MacOS/../Resources/svg/
        /tmp/QGIS-PythonTestConfigPathfe6_5gvl/profiles/default/svg/
User DB Path:   /Applications/QGIS.app/contents/MacOS/../Resources/resources/qgis.db
Auth DB Path:   /tmp/QGIS-PythonTestConfigPathfe6_5gvl/profiles/default/qgis-auth.db

OGR(1): Cannot open /private/var/folders/1d/23g39kb961387shpwfm6lh280000gn/T/processing_wqOGjl/f4ef3715d3e148c6a67fcf8e8ed255b4/tempfile.txt ().()
ERROR 4: `/private/var/folders/1d/23g39kb961387shpwfm6lh280000gn/T/processing_wqOGjl/f4ef3715d3e148c6a67fcf8e8ed255b4/tempfile.txt' not recognized as a supported file format.
ERROR 4: `/private/var/folders/1d/23g39kb961387shpwfm6lh280000gn/T/processing_wqOGjl/f4ef3715d3e148c6a67fcf8e8ed255b4/tempfile.txt' not recognized as a supported file format.
ERROR 4: `/private/var/folders/1d/23g39kb961387shpwfm6lh280000gn/T/processing_wqOGjl/f4ef3715d3e148c6a67fcf8e8ed255b4/tempfile.txt' not recognized as a supported file format.
ERROR 4: `/private/var/folders/1d/23g39kb961387shpwfm6lh280000gn/T/processing_wqOGjl/f4ef3715d3e148c6a67fcf8e8ed255b4/tempfile.txt' not recognized as a supported file format.
ERROR: Status 3: Unable to load mesh (null)
Traceback (most recent call last):
  File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/pydevd.py", line 1483, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/jakimowb/Repositories/enmap-box/tests/src/issues/test_issue_1330_QGIS.py", line 12, in <module>
    print(f'asString={parameter.valueAsString(path, context)}')
Exception: unknown

‌

[Benjamin Jakimow]

Opened QGIS issue https://github.com/qgis/QGIS/issues/48598

[Andreas Rabe]

Well done