include command to split/rotate logs

Create issue
Issue #13 new
nomeata repo owner created an issue

Felipe Sateler wirtes at http://bugs.debian.org/751279:

Hi,

Arbtt log files can become relatively large, and arbtt can take some time to parse it. I have a file containing a bit over 3 years of samples, and it takes a while to process:

% time arbtt-stats -i --logfile capture.log.all
<snip>
arbtt-stats -i --logfile capture.log.all  20.55s user 0.06s system 100% cpu 20.610 total

Using the right invocations of arbtt-dump, arbtt-import and grep, it is possible to split the log into multiple files, but it is cumbersome.

I'm not sure what exactly is the best way to improve this.

The simplest solution would be to provide a rotate command that (for example) would break the log by year and store each year in a different file, leaving the current year as the default capture file.

A (maybe) better solution would be to convert the logfile into a logdir, where logs are stored by year-month, and then add a flag to arbtt-stats to specify start date, so that older logs are not parsed. This would of course mean more work.

Comments (1)

  1. nomeata reporter

    I don’t think it is much an issue of files, but rather of inadequat algorithms. If you specify some time constraint (i.e. last year), the tool should not spend too much time processing the old data, but rather fast-forward to the right spot somehow. That would require a better on-disk data structure though.

    I need to improve the on disk format anyways, to make it more robust. I’ll then also try to make it faster reading only recent data.

  2. Log in to comment