hubflow.core.Classification.reclassify fails to write proper tif classifcation images

Issue #410 resolved
Benjamin Jakimow created an issue

HUB flow fails to write class names to *.tif images when using hubflow.core.Classification.reclassify

The following unit test tests (see enmapboxtesting/coreapps/test_reclassifyapp.pyfor details) tries to reclassify a gdal source image into different file formats with the target ClassificationScheme ['Unclassified', 'Class 1', 'Class 2', 'Class 3', 'Class 4']. ENVI file formats work well, but TIFF fails.

    def test_hubflow_reclassify(self):
        import hubflow.core
        import uuid
        from enmapbox.testing import TestObjects
        dsSrc = TestObjects.createRasterDataset(10,20,nc=5)
        self.assertIsInstance(dsSrc, gdal.Dataset)
        classNamesOld = ['Unclassified', 'Class 1', 'Class 2', 'Class 3', 'Class 4']
        self.assertEqual(dsSrc.GetRasterBand(1).GetCategoryNames(), classNamesOld)
        pathSrc = dsSrc.GetFileList()[0]
        self.assertTrue(pathSrc.startswith('/vsimem/'))

        pathResultFiles = []
        uid = uuid.uuid4()
        for i, ext in enumerate(['bsq', 'BSQ', 'bil', 'BIL', 'bip', 'BIP', 'tif', 'TIF', 'tiff', 'TIFF']):

            pathDst = r'/vsimem/testclasstiff{}.{}.{}'.format(i, uid, ext)

            classification = hubflow.core.Classification(pathSrc)
            oldDef = classification.classDefinition()
            self.assertEqual(oldDef.names(), classNamesOld[1:])

            newNames = ['No Class', 'Class B', 'Class D']
            newColors = [QColor('black'), QColor('yellow'), QColor('brown')]

            # this works
            c = hubflow.core.Color(QColor('black'))

            # but this does'nt
            #newDef = hubflow.core.ClassDefinition(names=newNames[1:], colors=newColors[1:])


            newDef = hubflow.core.ClassDefinition(names=newNames[1:], colors=[c.name() for c in newColors[1:]])
            newDef.setNoDataNameAndColor(newNames[0], QColor('yellow'))

            # driver = guessRasterDriver(pathDst)
            r = classification.reclassify(filename=pathDst,
                                      classDefinition=newDef,
                                      mapping={0:0, 1:1, 2:1})#,
                                        #outclassificationDriver=driver)


            ds = gdal.Open(pathDst)

            self.assertIsInstance(ds, gdal.Dataset)
            if re.search(r'\.(bsq|bil|bip)$', pathDst, re.I):
                self.assertTrue(ds.GetDriver().ShortName == 'ENVI',
                        msg='Not opened with ENVI driver, but {}: {}'.format(ds.GetDriver().ShortName, pathDst))
            elif re.search(r'\.tiff?$', pathDst, re.I):
                self.assertTrue(ds.GetDriver().ShortName == 'GTiff',
                        msg='Not opened with GTiff driver, but {}: {}'.format(ds.GetDriver().ShortName, pathDst))
            elif re.search(r'\.vrt$', pathDst, re.I):
                self.assertTrue(ds.GetDriver().ShortName == 'VRT',
                        msg='Not opened with VRT driver, but {}: {}'.format(ds.GetDriver().ShortName, pathDst))
            else:
                self.fail('Unknown extension {}'.format(pathDst))
            pathResultFiles.append(pathDst)

        for pathDst in pathResultFiles:
            ds = gdal.Open(pathDst)
            band = ds.GetRasterBand(1)
            self.assertIsInstance(band.GetCategoryNames(), list, msg='Failed to set any category names to "{}"'.format(pathDst))
            self.assertEqual(newNames, band.GetCategoryNames(), msg='Failed to set all category names to "{}"'.format(pathDst))
            print('Success: created {}'.format(pathDst))

Output:

SET UP test_reclassifyapp.TestReclassify.test_hubflow_reclassify
Success: created /vsimem/testclasstiff0.d7ac9072-494e-40c3-a82d-7cb91761eef6.bsq
Success: created /vsimem/testclasstiff1.d7ac9072-494e-40c3-a82d-7cb91761eef6.BSQ
Success: created /vsimem/testclasstiff2.d7ac9072-494e-40c3-a82d-7cb91761eef6.bil
Success: created /vsimem/testclasstiff3.d7ac9072-494e-40c3-a82d-7cb91761eef6.BIL
Success: created /vsimem/testclasstiff4.d7ac9072-494e-40c3-a82d-7cb91761eef6.bip
Success: created /vsimem/testclasstiff5.d7ac9072-494e-40c3-a82d-7cb91761eef6.BIP
TEAR DOWN test_reclassifyapp.TestReclassify.test_hubflow_reclassify

Failure
Traceback (most recent call last):
  File "D:\miniconda3\envs\qgis_stable\lib\unittest\case.py", line 60, in testPartExecutor
    yield
  File "D:\miniconda3\envs\qgis_stable\lib\unittest\case.py", line 676, in run
    self._callTestMethod(testMethod)
  File "D:\miniconda3\envs\qgis_stable\lib\unittest\case.py", line 633, in _callTestMethod
    method()
  File "D:\Repositories\enmap-box\enmapboxtesting\coreapps\test_reclassifyapp.py", line 97, in test_hubflow_reclassify
    self.assertIsInstance(band.GetCategoryNames(), list, msg='Failed to set any category names to "{}"'.format(pathDst))
  File "D:\miniconda3\envs\qgis_stable\lib\unittest\case.py", line 1335, in assertIsInstance
    self.fail(self._formatMessage(msg, standardMsg))
  File "D:\miniconda3\envs\qgis_stable\lib\unittest\case.py", line 753, in fail
    raise self.failureException(msg)
AssertionError: None is not an instance of <class 'list'> : Failed to set any category names to "/vsimem/testclasstiff6.d7ac9072-494e-40c3-a82d-7cb91761eef6.tif"

Comments (5)

  1. Andreas Janz

    Yes, its known that the GTiff driver is a bit fragile when it comes to category colors. Even if I force the raster to flush to disk, the colors are not written until the unittest is finished. So if you reopen it in a different dataset, GDAL won’t recognise the colors. If you write the GTiff to disk, you can open it afterwards in QGIS and everything looks fine:

    Not really sure if we can make the test pass. I tried the following without success:

                r = classification.reclassify(filename=pathDst,
                                          classDefinition=newDef,
                                          mapping={0:0, 1:1, 2:1})#,
                                            #outclassificationDriver=driver)
                r.dataset().flushCache()
                r.dataset().close()
                del r
    

  2. Benjamin Jakimow reporter

    I found out that it is just required to call files = dataSet.GetFileList() once after the dataset has been opened to make all the metadata written to the aux.xml known. The hub flow object that is returned by reclassify() might implement that as well.

  3. Benjamin Jakimow reporter

    .gitignore: added test-outputs for quick access of test data results EnMAPBoxTestCass: added .tempDir(), which creates and return the pathlib.Path to test-outputs directory test_reclassifyapp.py: resolves #410

    → <<cset 7dca3a11e0e9>>

  4. Andreas Janz

    Ah, funny how some things work in the background 😸

    I included the workaround in my routine und removed it from your unittest.

  5. Log in to comment