Error in Transform Raster Layer

[Agustin Lobo]

‌

QGIS version: 3.24.0-Tisler
QGIS code revision: 6b44a42058
Qt version: 5.15.2
Python version: 3.9.5
GDAL version: 3.2.2
GEOS version: 3.9.0-CAPI-1.16.2
PROJ version: Rel. 7.2.1, January 1st, 2021
PDAL version: 2.2.0 (git-version: Release)
Algorithm started at: 2022-03-09T15:19:12
Algorithm 'Transform raster layer' starting…
Input parameters:
{ 'outputRaster' : '/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobbl_PCAwithall.tif', 'raster' : '/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobb.bil', 'transformer' : '/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobbl_forPCAall.pkl' }

Python command:
>>>processing.run('enmapbox:TransformRasterLayer', dict(raster='/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobb.bil', transformer='/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobbl_forPCAall.pkl', outputRaster='/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobbl_PCAwithall.tif'))

Console command:
>>>qgis_process run enmapbox:TransformRasterLayer --raster=/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobb.bil transformer=/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobbl_forPCAall.pkl outputRaster=/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobbl_PCAwithall.tif 

Traceback (most recent call last):
File "/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python/plugins/enmapboxplugin/site-packages/typeguard/__init__.py", line 903, in wrapper
retval = func(*args, **kwargs)
File "/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python/plugins/enmapboxplugin/enmapboxprocessing/algorithm/transformrasteralgorithm.py", line 81, in processAlgorithm
Xt0 = dump.transformer.transform(X0)
AttributeError: 'NoneType' object has no attribute 'transform'

Execution failed after 0.66 seconds

Loading resulting layers
Algorithm 'Transform raster layer' finished

‌

[Andreas Janz]

AttributeError: 'NoneType' object has no attribute 'transform'

Looks like the PKL file (PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobbl_forPCAall.pkl) you selected had no transformer in it.

You can check the content here:

‌

[Agustin Lobo]

Well, that pkl is the result of Fit PCA selecting Create unsupervised dataset (from feature raster), see the log file:

Python command:

processing.run('enmapbox:CreateUnsupervisedDatasetFromFeatureRaster', dict(featureRaster='/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobb.bil', outputUnsupervisedDataset='/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobbl_forPCAall.pkl'))

Console command:

qgis_process run enmapbox:CreateUnsupervisedDatasetFromFeatureRaster --featureRaster=/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobb.bil outputUnsupervisedDataset=/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobbl_forPCAall.pkl

Sampled data: X=array[994354, 210]
Execution completed in 140.17 seconds
Results:
{'outputUnsupervisedDataset': '/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobbl_forPCAall.pkl'}

‌

[Andreas Janz]

Alright, PKL looks good. Is probably a bug, I’ll check this.

[Agustin Lobo]

Just in case you need it

https://www.dropbox.com/s/6lvt5ihfuqs309o/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobbl_forPCAall.pkl?dl=0

‌

[Andreas Janz]

[Andreas Janz]

Oh, just noticed, that the PKL file is >800 mb.
The SampleSize option would really help here

[Andreas Janz]

Ok, I testet it, and I think you probably did something wrong.

Your PKL file has no transformer. I would guess, that you just extracted the data, but you didn’t actually fit the PCA:

‌

When I use your PKL for fitting a PCA, the result looks good:

‌

[Agustin Lobo]

I just used the Fit PCA panel as it is, there are no options beyond selecting

• Create unsupervised dataset (from feature raster)
• input file name
• output file name

how can I do anything wrong?

‌

[Andreas Janz]

Can you show an animated GIF (or a set of screenshots) showing what you do step by step?

[Andreas Janz]

Also, the log you showed indicates, that you just extracted data:

‌

[Agustin Lobo]

https://www.dropbox.com/s/i1bud35z1uw9unn/simplescreenrecorder-2022-03-10_12.05.50.mp4?dl=0

new pkl: https://www.dropbox.com/s/n2e8aq9klr8lfnq/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobbl_forPCAall.pkl?dl=0

new log: https://www.dropbox.com/s/g9j6yv2051bqid4/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobbl_forPCAall.pkl.log?dl=0

‌

‌

[Andreas Janz]

You are still just creating a dataset.

‌

Here is how it looks like with the enmap_berlin testdata:

‌

[Agustin Lobo]

This is a very weird way of doing it: why should we hit Run twice? There is nothing

to modify between the first and the second run.

Also note that the log file is not modified (does no include what you see in the Log tab). The contents of my new Log tab is:

‌

QGIS version: 3.24.0-Tisler
QGIS code revision: 6b44a42058
Qt version: 5.15.2
Python version: 3.9.5
GDAL version: 3.2.2
GEOS version: 3.9.0-CAPI-1.16.2
PROJ version: Rel. 7.2.1, January 1st, 2021
PDAL version: 2.2.0 (git-version: Release)
Algorithm started at: 2022-03-10T12:40:06
Algorithm 'Fit PCA' starting…
Input parameters:
{ 'dataset' : '/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobbl_forPCAall.pkl', 'outputTransformer' : 'TEMPORARY_OUTPUT', 'transformer' : 'from sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.decomposition import PCA\n\npca = PCA(n_components=0.95)\ntransformer = make_pipeline(StandardScaler(), pca)' }

Python command:
processing.run('enmapbox:FitPca', dict(transformer='from sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.decomposition import PCA\n\npca = PCA(n_components=0.95)\ntransformer = make_pipeline(StandardScaler(), pca)', dataset='/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobbl_forPCAall.pkl', outputTransformer='TEMPORARY_OUTPUT'))

Console command:
qgis_process run enmapbox:FitPca --transformer="from sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.decomposition import PCA\n\npca = PCA(n_components=0.95)\ntransformer = make_pipeline(StandardScaler(), pca)" dataset=/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobbl_forPCAall.pkl outputTransformer=TEMPORARY_OUTPUT 

Load training dataset: X=array[994354, 210]
Fit transformer
Execution completed in 100.3 seconds
Results:
{'outputTransformer': '/tmp/processing_zqrxJt/3d01cbe9f54e47fd9e7155b3d1698922/outputTransformer.pkl'}
Execution completed in 100.43 seconds (1 minute 40 seconds)
Results:
{'outputTransformer': '/tmp/processing_zqrxJt/3d01cbe9f54e47fd9e7155b3d1698922/outputTransformer.pkl'}

Loading resulting layers
Algorithm 'Fit PCA' finished

‌

Anyway, I still get an error when I actually try Transform Raster Layer

QGIS version: 3.24.0-Tisler
QGIS code revision: 6b44a42058
Qt version: 5.15.2
Python version: 3.9.5
GDAL version: 3.2.2
GEOS version: 3.9.0-CAPI-1.16.2
PROJ version: Rel. 7.2.1, January 1st, 2021
PDAL version: 2.2.0 (git-version: Release)
Algorithm started at: 2022-03-10T12:48:15
Algorithm 'Transform raster layer' starting…
Input parameters:
{ 'outputRaster' : '/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobb_PCAall.tif', 'raster' : '/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobb.bil', 'transformer' : '/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobbl_forPCAall.pkl' }

Python command:
processing.run('enmapbox:TransformRasterLayer', dict(raster='/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobb.bil', transformer='/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobbl_forPCAall.pkl', outputRaster='/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobb_PCAall.tif'))

Console command:
qgis_process run enmapbox:TransformRasterLayer --raster=/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobb.bil transformer=/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobbl_forPCAall.pkl outputRaster=/media/alobo/LaCieNTFS2T/Ignacio2019/RioTinto/PRISMA/PRS_L2D_STD_20210625111917_20210625111921_0001_SR_nobb_PCAall.tif 

Traceback (most recent call last):
File "/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python/plugins/enmapboxplugin/site-packages/typeguard/__init__.py", line 903, in wrapper
retval = func(*args, **kwargs)
File "/home/alobo/.local/share/QGIS/QGIS3/profiles/default/python/plugins/enmapboxplugin/enmapboxprocessing/algorithm/transformrasteralgorithm.py", line 81, in processAlgorithm
Xt0 = dump.transformer.transform(X0)
AttributeError: 'NoneType' object has no attribute 'transform'

Execution failed after 0.64 seconds

Loading resulting layers
Algorithm 'Transform raster layer' finished

‌

[Andreas Janz]

This is a very weird way of doing it: why should we hit Run twice? There is nothing to modify between the first and the second run.

That is a technical limitation of the processing framework. We can’t trigger the second run automatically.

‌

Anyway, I still get an error when I actually try Transform Raster Layer

You stored your PCA model file here:

But in the TramsformRasterLayer algo, you again used the wrong PKL file that only contains the data:

‌

[Andreas Janz]

Hmm, it’s a bit alarming to me, that you seam to have such trouble pulling it of.

[Agustin Lobo]

We can’t trigger the second run automatically.

There has to be a more intuitive way. Note the final pkl was saved to the tmp directory because making the user save the pkl again is also counterintuitive. Perhaps generating the 1st pkl (which I do not really understand what is meant for) should be either on the fly or in another tool.

[Agustin Lobo]

ok, I think I understand it now (not that I like it).

The 1st pkl is just the same information you have in the image, but instead of being as nrow*nrow*nbands image as a numpixe*nbands table. That is, IMHO, an unnecessary and inconvenient step for a hyperspectral image.

Well, that’s the way you have it (at least by now). I suggest the following to clarify the process to the user:

Have space for 3 files:

Create Unsupervised Dataset (with the 6 options in the wheel)

Unsupervised Dataset (by default should get the output of the previous one, but you can use a previously generated pkl as well)

Output transformer

‌

‌

[Andreas Janz]

ok, I think I understand it now (not that I like it).

The 1st pkl is just the same information you have in the image, but instead of being as nrow*nrow*nbands image as a numpixe*nbands table. That is, IMHO, an unnecessary and inconvenient step for a hyperspectral image.

I agree, in case of a whole image it is kind of stupid to store all data in a dataset pkl file. For all the other options it actually makes mutch more sense.

‌

Currently, I don’t see how I would handle the image case differently.

Also, I really wouldn’t recommend to use all data from an image, but rather use the sampleSize option. In both cases, the estimated statistics will be very similar.

‌

Well, that’s the way you have it (at least by now). I suggest the following to clarify the process to the user:

Have space for 3 files:

Create Unsupervised Dataset (with the 6 options in the wheel)

Oh, we have all the options as seperate algorithms available:

‌

In fact you trigger those algorithms via the menu options here:

‌

Have space for 3 files:

Create Unsupervised Dataset (with the 6 options in the wheel)

Unsupervised Dataset (by default should get the output of the previous one, but you can use a previously generated pkl as well)

We are technically limitted here, because of the processing framework. We can’t set the output of one widget as input to another widget. Sorry, that I can’t explain the details here.

[Andreas Janz]

To improve the usage, we could only show PKL files with Transformers in it here:

‌

This way you can’t accidently select a PKL file that isn’t suitable.

[Agustin Lobo]

We are technically limitted here, because of the processing framework. We can’t set the output of one widget as input to another widget. Sorry, that I can’t explain the details here.

But at least, if there are 2 steps you should have 3 boxes (input, intermediate (kind of dummy) and output). Otherwise the confusion is guaranteed.

[Agustin Lobo]

This way you can’t accidently select a PKL file that isn’t suitable.

That would certainly alleviate the problem in the “Transform raster layer” panel, but not in “Fit PCA”.

‌

[Agustin Lobo]

Anyway, at least I can run a PCA with EnMapBox, it’s been a long time! Thanks a lot for your efforts.

[Andreas Janz]

Thanks Agus, I’ll discuss the workflow with the team again, maybe we can come up with a better, more intuitive solution.