Archive size mismatch

[sebastiano degan]

From the admin dashboard I read "1.1 GB (631 MB)", but "du -hs /var/piler/store/" gives me 9.4G.

According to "https://bitbucket.org/jsuto/piler/issues/299/archive-size", compression and deduplication are involved, but isn't the size supposed to be reduced? Am I missing something?

Piler version is 1.3.3

[Janos SUTO]

Please show me the contents of the counter table, and count how many .m files you have in /var/piler/store. Do you regularly purge emails?

[sebastiano degan]

+---------+-------+-----------+--------+------------+-------------+
| rcvd    | virus | duplicate | ignore | size       | stored_size |
+---------+-------+-----------+--------+------------+-------------+
| 1478848 |     0 |         8 |      0 | 1090733181 |   641382776 |
+---------+-------+-----------+--------+------------+-------------+

find . -type f -name "*.m" | wc -l
1478846

We never purged until now. Piler has been running for almost a year, and we use it to store a large number of very small emails.

[Janos SUTO]

Indeed, it's odd. Can you run recursively on /var/piler/store, and count the total size of the files?

[sebastiano degan]

find /var/piler/store/ -type f -name "*.m" -print0 | du -shc --files0-from=- | tail -n1                                                                         │
5.7G    total 

find /var/piler/store/ -type f -name "*.m" -print0 | du --apparent-size -shc --files0-from=- | tail -n1                                                         │
613M    total

du -hs /var/piler/store/
9.6G

du --apparent-size -hs /var/piler/store/
4.5G

[sebastiano degan]

During these days I'll play a bit with blocksize/inode_ratio etc... to store all these files more efficiently.

[Janos SUTO]

OK, let me know your findings.

[sebastiano degan]

I ended up using xfs with a blocksize of 1024, it could go down to 512 but you must disable CRC on medatada as far as I understood.

Now i get:

sudo du -hs /var/piler/store/
1.5G    /var/piler/store/
sudo du -hs --apparent-size /var/piler/store/
714M    /var/piler/store/

which is far more reasonable.

I don't know if it is worth to try to improve piler estimates on real archive size, since it is so dependent on fs config. In any case, if you want me to do any test, feel free to ask.

[Janos SUTO]

Thanks for the research. On xfs it looks much more appropriate. Btw. what filesystem and settings did you have before?

[sebastiano degan]

Originally I used ext4 with default settings.

Then I ran out of inodes so I re-created the filesystem using the "-T news" option, which lowers the inode_ratio to 4096. This however left the blocksize unchanged (4K), so it was pretty inefficient for many small files and directories.

Then I tried:

• ext4 with the "-T small" option, that corresponds to "blocksize = 1024,inode_size = 128,inode_ratio = 4096"
• xfs with the "-b size=1024" option

Both solved the space problem, but xfs had twice as many inodes.

The reasons why chose xfs are:

1. Space and inode are filled up almost at the same rate, so one is not a bottleneck for the other
2. The deviation from the default settings was smaller, so i tend to prefer that

Performance-wise I did not make any test, but that was never a concern to me.

[Janos SUTO]

Thanks. I'll update the docs with these, then close this issue.

[Janos SUTO]

The FAQ is updated.