Show info in white and errors/warnings in red on the jupyter lab interface when running the code

[Claire Michailovsky]

Found a solution, only tested on jupyter lab so far, prints the log.info in white and others in red by using sys.stdout for info and sys.stderr for errors and warnings. Will be very helpful spotting actual errors and dropped downloads, especially when teaching.

Modifications needed:

in logger.py:

add:

import sys

then modif line 14:

__handler__ = logging.StreamHandler(stream=sys.stdout)

in log_indenter.py

add:

import sys

then from line 494:

in each of the log types call self.logger.handlers[0].setStream(sys.stdout) or self.logger.handlers[0].setStream(sys.stderr) depending on whether you want white or red output respectively. Example with white info and red everything else :

    def info(self, msg, *args, **kwargs):
        self.logger.handlers[0].setStream(sys.stdout)
        super(IndentedLoggerAdapter, self).info(msg, *args, **kwargs)
        return self

    def warning(self, msg, *args, **kwargs):
        self.logger.handlers[0].setStream(sys.stderr)
        super(IndentedLoggerAdapter, self).warning(msg, *args, **kwargs)
        return self

    def error(self, msg, *args, **kwargs):
        self.logger.handlers[0].setStream(sys.stderr)
        super(IndentedLoggerAdapter, self).error(msg, *args, **kwargs)
        return self

    def exception(self, msg, *args, **kwargs):
        self.logger.handlers[0].setStream(sys.stderr)
        super(IndentedLoggerAdapter, self).exception(msg, *args, **kwargs)
        return self

    def critical(self, msg, *args, **kwargs):
        self.logger.handlers[0].setStream(sys.stderr)
        super(IndentedLoggerAdapter, self).critical(msg, *args, **kwargs)
        return self

    def log(self, level, msg, *args, **kwargs):
        super(IndentedLoggerAdapter, self).log(level, msg, *args, **kwargs)
        return self

‌

[Claire Michailovsky]

Note: is an empty log.warning calls l.260 in main - its purpose is unclear to me but with my fix adds empty red lines to the output. Can perhaps be changed to log.info if it is just to create a legible output?

Same with a log.exception(““) l.153 in downloader.

[bert.coerver]

Solved it a bit different in the end: https://bitbucket.org/cioapps/pywapor/commits/3c4056e427e6e7c3e1df4a10de9fc235fe93ebeb

Example:

[Claire Michailovsky]

Hi Bert,

On jupyter the general info is still in red, which I think may still be confusing. See example below.

‌

[Claire Michailovsky]

Suggested fix:

in logger.py:

add:

import sys

then modif line 14:

__handler__ = logging.StreamHandler(stream=sys.stdout)

The errors generated by the code will still be red, and the color you defined for warnings also still works - just regular logging messages will be on white background.

(I do still suggest to change what I had mentionned so that we don’t get weird red lines as above:

• empty log.warning calls l.260 in main - its purpose is unclear to me but with my fix adds empty red lines to the output. Can perhaps be changed to log.info if it is just to create a legible output?
• Same with a log.exception(““) l.153 in downloader.)

‌

[bert.coerver]

hi claire,

• you’re (new) suggested fix seems to not change anything for me, so I’ll implement that so the logs look nicer in jupyter 👍
• About l.260, there is actually supposed to be a warning here (but i didnt get around to writing it), can you give me the summary you’re using that triggers it?
• The log.exception call should make sure the entire traceback of an error gets written to the log file. Need to test if thats working correctly..

‌

[Claire Michailovsky]

for l.260, the empty warning gets printed after the configuration is loaded - after a quick test, it appears to be due to putting in MODIS and its interpolation method instead of VIIRS (also the reason why it gets triggered twice as MOD and MYD are defined for the thermal data).

Code:

import pywapor
project_folder = r"D:\notebooks\Jordan_MODIS2" #Path to folder
bb = [33.131,14.2510382256321311, 33.132,14.2625000000024841]
period = ["2022-03-26", "2022-03-28"] 

project = pywapor.Project(project_folder, bb, period)

summary = {
            'ENHANCE': {"lst": ["pywapor.enhancers.dms.thermal_sharpener.sharpen"],},
            'EXAMPLE': 'SENTINEL2.S2MSI2A_R20m',
            'WHITTAKER': {'SENTINEL2.S2MSI2A_R20m':{'method':'linear'},                             
                            'MODIS.MOD11A1.061':{'method':'linear'},    
                            'MODIS.MYD11A1.061':{'method':'linear'}
                },
            'elevation': {'SRTM.30M'},
            'meteorological': {'GEOS5.inst3_2d_asm_Nx'},
            'optical': {'SENTINEL2.S2MSI2A_R20m'},
            'precipitation': {'CHIRPS.P05'},
            'soil moisture': {'FILE:{folder}{sep}se_root_out*.nc'},
            'solar radiation': {'MERRA2.M2T1NXRAD.5.12.4'},
            'statics': {'STATICS.WaPOR3'},
            'thermal': {'MODIS.MOD11A1.061','MODIS.MYD11A1.061'},
            'chunks':{'time':1,'x':100,'y':100}
            }
project.load_configuration(summary = summary)

Printout (added the word test for clarity - apparently it gets triggered twice):

‌

[bert.coerver]

Okay thanks!

• The streamhandler is now set to the correct stream: https://bitbucket.org/cioapps/pywapor/commits/cd1e58fabc57f435380bbca2a834273148b40c76
• The l.260 warning is now useful (https://bitbucket.org/cioapps/pywapor/commits/ec056590e8d77ca608ffbb3b9e30d4bd3c4d0fe8). It wont warn anymore on the summary you passed, but if for example you would do:

‌

            '_WHITTAKER_': {'SENTINEL2.S2MSI2A_R20m':{'method':'linear'},                             
                            'MODIS.MOD11A1.061':{'method':'linear'},    
                            'MODIS.MYD11A1.061':{'method':'bilinear'}
                },

It will warn you that you’re specifying two different temporal interpolations for “lst” and it will pick one (kind of sort of randomly).

• The summary you gave above will now also trigger another warning, telling you that there are some invalid/unused keys in your summary, i.e.: ‘WHITTAKER’, ‘ENHANCE’ ,'EXAMPLE' (should be ‘_WHITTAKER_’, ‘_ENHANCE_’ ,'_EXAMPLE_') and ‘chunks’ doesn't do anything either.
• the log.exception is fixed now, and will only trigger on the final attempt when retrying after an error. Also, for Landsat it won’t give a warning/error anymore when waiting for orders to finish. (https://bitbucket.org/cioapps/pywapor/commits/819080d9f1548ba1fbca98b1b106ebb777d2f93b)

‌