Support persistent cache

Issue #17 resolved
Nikolaus Rath
repo owner created an issue


I would love the option to make the cache persistent between boots. Like bitcasa. I know that has a chance of having a corrupt cache(does not match the version on the server). If there was a way of tracking versions stored on the server meta data. That could help. I my case mounting the drive on other computers would be rare. Thus the risk is minimal. I am planing on having it mounted on a headless server shared via samba etc.

In similar fashion a good meta data stored on server could minimize the risk to having multiple mounts of the file system. Of course there are limits and additional risks as reported in the documentation.

Comments (10)

  1. Todd Vierling

    Perhaps this should be revisited. I was drawn to s3ql because it almost perfectly addresses my use case, except for the fact that on umount, the cache vaporizes. If the remote storage has several hundred GB in it, and there are many GB already cached locally which is actively being used, a reboot loses all that cache and it has to be fetched all over again (incurring non-trivial cost).

    I need to be able to keep around 10% of a 1TB remote repository cached locally, and the rsync mirroring is a bit pointless since the whole idea is to keep most of the data stored remotely (S3/GS). The caching is great, as long as you don't have to reboot or unmount the filesystem.

    Unfortunately, this undocumented issue (it should be mentioned somewhere in the main doc for mount, in its caching notes, and umount) is a dealbreaker for me, and I'm going to have to go elsewhere. 8-/

  2. Nikolaus Rath reporter

    Thanks for following up! Actually, these days I think having a persistent cache in S3QL would indeed be nice.

    That said, S3QL is a fully volunteer driven project[1]. This means that new features and bugfixes are implemented only when someone has a personal interest in them and therefore also does the necessary work. Feature requests are therefore in general unlikely to actually result in the feature being implemented.

    Would you be interested in working on this? If not, there isn't much point in opening this issue again (since the purpose of the issue tracker is to coordinate ongoing and planned work, rather than enumerate all the work that could possibly be done).

    [1] Actually, that's not strictly true. I do offer paid consulting with discounts for code that makes it into the official S3QL release. Contact me privately if that's something you'd be interested in.

  3. Isaac Aymerich

    Sounds fine, for the moment I am using NFS + cachefilesd and works perfect! I Actually Have a docker which do all the ugly things (keep in mind the version of s3ql compiled in this image it's not the official, it's from a fork with gdrive support)

    docker run  -d  -it --privileged -v /persistent/cache/after/restart:/var/cache/fscache -v /nonpersistent/cache/after/restart:/s3ql/cache -v /where/you/want/mountpoint:/nfs/mountpoint:shared -e STORAGE_URL=gdrive://bucket -e LOGIN_USER=............ -e LOGIN_PASSWORD=.... -e PASSPHRASE=....... -e S3QL_METADATA_UPLOAD_INTERVAL=1800 segator/s3ql

    Enjoy meanwhile we are waiting for nov-2018

  4. Log in to comment