Add automatic zipping of log files

[David Platten]

I think it would be good to have OpenREM automatically compress log files either once they reach a certain size, or perhaps every 24 hours. My openrem_qr.log file gets pretty large fairly quickly.

[David Platten]

Perhaps some thing like what is being discussed here:

http://stackoverflow.com/questions/8467978/python-want-logging-with-log-rotation-and-compression

[Tim de Wit]

I'm using this for a while now. You can easily configure this yourself in settings.py. In this case it doesn't compress the log files, but it only keeps a maximum number of logs with a constrained filesize (max 5 backups of 10MB each).

    'handlers': {
        ...,
        'qr_file': {
            'level': 'DEBUG',
            'class': 'logging.handlers.RotatingFileHandler',
            'maxBytes': 10*1024*1024,
            'backupCount': 5,
            'filename': 'openrem_qrscu.log',
            'formatter': 'verbose'
        },
        ...,
    },

[David Platten]

Many thanks for your comment, @tcdewit - that's exactly the sort of thing I am looking for.

[Ed McDonagh]

We should be able to set a sensible default in settings and document how to override them in local_settings.

[David Platten]

Changed default logging to use RotatingFileHandler to ensure that log files cannot get too big. Need to document how to override this. References issue #483

→ <<cset a37aa291c0b6>>

[David Platten]

Unfortunately the RotatingFileHandler doesn't work for me on my Windows server. Once the log file reaches maxBytes it fails to roll over. The following is written to the default.log file (many, many times...):

[2017-07-19 08:40:05,463: WARNING/MainProcess] Logged from file qrscu.py, line 53
[2017-07-19 08:40:05,463: WARNING/MainProcess] Traceback (most recent call last):
[2017-07-19 08:40:05,463: WARNING/MainProcess] File "d:\server_apps\python27\lib\logging\handlers.py", line 77, in emit
[2017-07-19 08:40:05,463: WARNING/MainProcess] self.doRollover()
[2017-07-19 08:40:05,463: WARNING/MainProcess] File "d:\server_apps\python27\lib\logging\handlers.py", line 142, in doRollover
[2017-07-19 08:40:05,463: WARNING/MainProcess] os.rename(self.baseFilename, dfn)

This is a known problem when trying to use this handler on Windows (http://bugs.python.org/issue4749) and I don't think there's an easy workaround.

I think that the default logging will need to be changed back, and if (Linux) users want to use the rotating logger then they can put it in their local_settings.py.

[Tim de Wit]

Perhaps try the ConcurrentLogHandler?
https://pypi.python.org/pypi/ConcurrentLogHandler/0.9.1

[Tim de Wit]

This one looks nice as well... uses queueing to prevent multiple simultaneous writes:
https://stackoverflow.com/questions/641420/how-should-i-log-while-using-multiprocessing-in-python/894284#894284

[David Platten]

Note to self:

pip install ConcurrentLogHandler
pip install pypiwin32

Then in the settings.py logging section:

        'qr_file': {
            'level': 'DEBUG',
            'class': 'cloghandler.ConcurrentRotatingFileHandler',
            'maxBytes': 10 * 1024 * 1024,
            'backupCount': 5,
            'filename': 'openrem_qrscu.log',
            'formatter': 'verbose'
        },

Not tested yet.

[Ed McDonagh]

Hello again @dplatten, sorry for all the emails. Did you try this? Is it an easy fix for 0.8.0, or does it need to be pushed too the following release?

[David Platten]

I haven't tried the ConcurrentLagHandler yet, and won't be able to do so in the near future. I'd suggest we document how to change the settings file to use the RotatingFileHandler for Linux users, and push this issue to the following release.

[Ed McDonagh]

Created ref #587 to cover documenting linux for 0.8.0

[Ed McDonagh]

@dplatten, I think this is already configured in settings.py here. Is that overridden in your local_settings.py or removed from your settings.py?

[David Platten]

I've reverted to standard logging in my live settings.py file.

[Ed McDonagh]

Removed rotating log file configuration from settings.py, added it into local_settings.py.example, but commented out. Refs #483 and partially refs #587 as documented in local_settings.py.example

→ <<cset 8fa2cbc4af84>>