Archive size mismatch

Issue #961 closed
sebastiano degan created an issue

From the admin dashboard I read "1.1 GB (631 MB)", but "du -hs /var/piler/store/" gives me 9.4G.

According to "", compression and deduplication are involved, but isn't the size supposed to be reduced? Am I missing something?

Piler version is 1.3.3

Comments (11)

  1. Janos SUTO repo owner

    Please show me the contents of the counter table, and count how many .m files you have in /var/piler/store. Do you regularly purge emails?

  2. sebastiano degan reporter
    | rcvd    | virus | duplicate | ignore | size       | stored_size |
    | 1478848 |     0 |         8 |      0 | 1090733181 |   641382776 |
    find . -type f -name "*.m" | wc -l

    We never purged until now. Piler has been running for almost a year, and we use it to store a large number of very small emails.

  3. Janos SUTO repo owner

    Indeed, it's odd. Can you run recursively on /var/piler/store, and count the total size of the files?

  4. sebastiano degan reporter
    find /var/piler/store/ -type f -name "*.m" -print0 | du -shc --files0-from=- | tail -n1                                                                         │
    5.7G    total 
    find /var/piler/store/ -type f -name "*.m" -print0 | du --apparent-size -shc --files0-from=- | tail -n1                                                         │
    613M    total
    du -hs /var/piler/store/
    du --apparent-size -hs /var/piler/store/
  5. sebastiano degan reporter

    During these days I'll play a bit with blocksize/inode_ratio etc... to store all these files more efficiently.

  6. sebastiano degan reporter

    I ended up using xfs with a blocksize of 1024, it could go down to 512 but you must disable CRC on medatada as far as I understood.

    Now i get:

    sudo du -hs /var/piler/store/
    1.5G    /var/piler/store/
    sudo du -hs --apparent-size /var/piler/store/
    714M    /var/piler/store/

    which is far more reasonable.

    I don't know if it is worth to try to improve piler estimates on real archive size, since it is so dependent on fs config. In any case, if you want me to do any test, feel free to ask.

  7. Janos SUTO repo owner

    Thanks for the research. On xfs it looks much more appropriate. Btw. what filesystem and settings did you have before?

  8. sebastiano degan reporter

    Originally I used ext4 with default settings.

    Then I ran out of inodes so I re-created the filesystem using the "-T news" option, which lowers the inode_ratio to 4096. This however left the blocksize unchanged (4K), so it was pretty inefficient for many small files and directories.

    Then I tried:

    • ext4 with the "-T small" option, that corresponds to "blocksize = 1024,inode_size = 128,inode_ratio = 4096"
    • xfs with the "-b size=1024" option

    Both solved the space problem, but xfs had twice as many inodes.

    The reasons why chose xfs are:

    1. Space and inode are filled up almost at the same rate, so one is not a bottleneck for the other
    2. The deviation from the default settings was smaller, so i tend to prefer that

    Performance-wise I did not make any test, but that was never a concern to me.

  9. Log in to comment