Add automatic zipping of log files

Issue #483 new
David Platten created an issue

I think it would be good to have OpenREM automatically compress log files either once they reach a certain size, or perhaps every 24 hours. My openrem_qr.log file gets pretty large fairly quickly.

Comments (17)

  1. Tim de Wit

    I'm using this for a while now. You can easily configure this yourself in settings.py. In this case it doesn't compress the log files, but it only keeps a maximum number of logs with a constrained filesize (max 5 backups of 10MB each).

        'handlers': {
            ...,
            'qr_file': {
                'level': 'DEBUG',
                'class': 'logging.handlers.RotatingFileHandler',
                'maxBytes': 10*1024*1024,
                'backupCount': 5,
                'filename': 'openrem_qrscu.log',
                'formatter': 'verbose'
            },
            ...,
        },
    
  2. David Platten reporter

    Many thanks for your comment, @tcdewit - that's exactly the sort of thing I am looking for.

  3. Ed McDonagh

    We should be able to set a sensible default in settings and document how to override them in local_settings.

  4. David Platten reporter

    Changed default logging to use RotatingFileHandler to ensure that log files cannot get too big. Need to document how to override this. References issue #483

    → <<cset a37aa291c0b6>>

  5. David Platten reporter

    Unfortunately the RotatingFileHandler doesn't work for me on my Windows server. Once the log file reaches maxBytes it fails to roll over. The following is written to the default.log file (many, many times...):

    [2017-07-19 08:40:05,463: WARNING/MainProcess] Logged from file qrscu.py, line 53
    [2017-07-19 08:40:05,463: WARNING/MainProcess] Traceback (most recent call last):
    [2017-07-19 08:40:05,463: WARNING/MainProcess] File "d:\server_apps\python27\lib\logging\handlers.py", line 77, in emit
    [2017-07-19 08:40:05,463: WARNING/MainProcess] self.doRollover()
    [2017-07-19 08:40:05,463: WARNING/MainProcess] File "d:\server_apps\python27\lib\logging\handlers.py", line 142, in doRollover
    [2017-07-19 08:40:05,463: WARNING/MainProcess] os.rename(self.baseFilename, dfn)
    

    This is a known problem when trying to use this handler on Windows (http://bugs.python.org/issue4749) and I don't think there's an easy workaround.

    I think that the default logging will need to be changed back, and if (Linux) users want to use the rotating logger then they can put it in their local_settings.py.

  6. David Platten reporter

    Note to self:

    pip install ConcurrentLogHandler
    pip install pypiwin32
    

    Then in the settings.py logging section:

            'qr_file': {
                'level': 'DEBUG',
                'class': 'cloghandler.ConcurrentRotatingFileHandler',
                'maxBytes': 10 * 1024 * 1024,
                'backupCount': 5,
                'filename': 'openrem_qrscu.log',
                'formatter': 'verbose'
            },
    

    Not tested yet.

  7. Ed McDonagh

    Hello again @dplatten, sorry for all the emails. Did you try this? Is it an easy fix for 0.8.0, or does it need to be pushed too the following release?

  8. David Platten reporter

    I haven't tried the ConcurrentLagHandler yet, and won't be able to do so in the near future. I'd suggest we document how to change the settings file to use the RotatingFileHandler for Linux users, and push this issue to the following release.

  9. Ed McDonagh

    @dplatten, I think this is already configured in settings.py here. Is that overridden in your local_settings.py or removed from your settings.py?

  10. Ed McDonagh

    Removed rotating log file configuration from settings.py, added it into local_settings.py.example, but commented out. Refs #483 and partially refs #587 as documented in local_settings.py.example

    → <<cset 8fa2cbc4af84>>

  11. Log in to comment