Write stdout + stderr to Logfile

[Benjamin Jakimow]

FORCE4QGIS processes might run for a long time and during this time many events of interest might happen, e.g. important or unimportant errors, warnings, informations, connection losses and more.

It would be helpful to log all such messages to a separate log file, as already known from the force software it self.

[Andreas Janz]

@Benjamin Jakimow you can now enable logging by providing a File-like object via the logFile keyword:

    with AutoFilenames(dir='data/tmp', cleanup=not True):
        with open('data/log.txt', 'w') as logFile:
            main(parameters=parameters, logFile=logFile)

log.txt content looks like this:

Parse Level 2 folder
_level2\X0049_Y0027\20140108_LEVEL2_LND08_BOA.tif
_level2\X0049_Y0027\20140209_LEVEL2_LND08_BOA.tif
_level2\X0049_Y0027\20140225_LEVEL2_LND08_BOA.tif
_level2\X0049_Y0027\20140321_LEVEL2_LND07_BOA.tif
Prepare QAI masking
Prepare features
Exclude unwanted wavebands
Prepare composites
Prepare timeseries
Start at 2020-04-15T18:07:35..0%..Done in 0.1 sec | 0.0 min | 0.0 hrs
Build VRT stacks
Build VRT mosaics

‌

Regarding errors, warnings and every other output produced during run-time: this is a general concern with lot’s of discussions on StackOverflow. I think this should be done by the user of the FORCE4QGIS API itself, e.g. by redirecting stdout and stderr to the log file.

Here is an example with a print, a warning, the FORCE4QGIS logs and finally an exception. Use redirect_stdout and redirect_stderr in the context manager and also a try-except block:

    with AutoFilenames(dir='data/tmp', cleanup=not True):
        with open('data/log.txt', 'w') as file, redirect_stdout(file), redirect_stderr(file):
            try:
                print('PRINT')
                warn('WARNING')
                main(parameters=parameters, logFile=file)
                raise Exception('ERROR')
            except:
                print(format_exc())
                exit(1)

log.txt looks like this:

‌

PRINT
C:/source/arcoop/benjamin_jakimow/issue12.py:53: UserWarning: WARNING
  warn('WARNING')
Parse Level 2 folder
_level2\X0049_Y0027\20140108_LEVEL2_LND08_BOA.tif
_level2\X0049_Y0027\20140209_LEVEL2_LND08_BOA.tif
_level2\X0049_Y0027\20140225_LEVEL2_LND08_BOA.tif
_level2\X0049_Y0027\20140321_LEVEL2_LND07_BOA.tif
Prepare QAI masking
Prepare features
Exclude unwanted wavebands
Prepare composites
Prepare timeseries
Start at 2020-04-15T18:07:35..0%..Done in 0.1 sec | 0.0 min | 0.0 hrs
Build VRT stacks
Build VRT mosaics
Traceback (most recent call last):
  File "C:/source/arcoop/benjamin_jakimow/issue12.py", line 55, in <module>
    raise Exception('ERROR')
Exception: ERROR

‌

[Benjamin Jakimow]

It is not possible to specify this logfile in the FORCE4QGIS GUI

[Andreas Janz]

@Benjamin Jakimow it is really not recommended to redirect the stdout and stderr stream for an application that runs in an environment with other applications (i.e. QGIS), because it alters the global state of your Python. While my app runs and the redirection is activated, the output from another app that runs in a parallel thread is also redirected into the log file.

I would rather rely on the QGIS logging system.

Errors, warnings, debug print outs etc. can be found like usually in QGIS, somewhere in the Console or any of the Log panels. I now this system is not perfect, but this is how it works in QGIS.

Not sure whats the problem with the remote connection loss. Does this kill QGIS or does it still finish the computation? Let’s have a telco for this one.

[Andreas Janz]

@Benjamin Jakimow I found a better solution for your logging problem, that works for every application and not only FORCE4QGIS.

(see https://gis.stackexchange.com/questions/84790/where-does-qgis-write-its-error-log-messages)

filename = '/tmp/qgis.log'

def write_log_message(message, tag, level):
    with open(filename, 'a') as logfile:
        logfile.write('{tag}({level}): {message}'.format(tag=tag, level=level, message=message))

QgsMessageLog.instance().messageReceived.connect(write_log_message)

‌

What do you think, shall we build a small QGIS Plugin that simply connects a message logger (see code above) at startup, that redirects all QGIS log messages to a global log file?

[Andreas Janz]

let's create a QGIS Plugin that is logging all the messages