Docker Backup Method

Issue #253 resolved
Simon Biggs created an issue

Make a method which by default backs up once per day as a database dump within a dated and time stamped directory. Have it also convert the database dump to usable sensible CSV files.

Implement a restore from database dump method within the Docker image.

Comments (22)

  1. Simon Biggs reporter

    Hi Randle,

    Were you still keen on the docker idea? Did you want to see if I can get it ready for 0.30.0? If so, what are all the things that must be operational within a docker method for it to be included within a 0.30.0 release?

    What things are not "must have", but just nice? For now, I think I would just want the "must have" within 0.30.0. the nice things can be developed over time.

  2. Randle Taylor

    Hi Simon, my apologies for the very slow responses here. I will be reviewing the Docker deploy over the weekend and will get back to you then or on Monday.

  3. Simon Biggs reporter

    All good :). I'm not overly attached to it. And we've been so very busy at work that I doubt I'll be helpful in the near future. If you're not comfortable with the method I don't mind if it is scrapped.

  4. Randle Taylor

    Hi Simon,

    Went through your README and an install again on a clean computer last night. Here's a couple notes I made:

    • First, this is a very nice way to get a QATrack+ installation up and running :). ( Haven't tried on Windows yet, but assuming it works the same there, that will be very nice there too.)

    • In the README with your link to the installation page for Docker Compose it might be a good idea to mention that people can/should use Docker CE (I think Docker EE always costs money?)

    • I ran into (minor) trouble because Docker tried to bind to my host ports 5432 & 80 which are already occupied on my system. It's simple to change the configuration to bind to a different post on the host, but it should probably be mentioned here. Even better if the port can be specified in a config file (e.g. use 80/5432 as defaults but allow users to override

    • DEBUG should be set to False by default

    • Docs should mention that you can run docker-compose up without the -d if you need to see debug info (After running docker-compose up -d for the first time, going to http://localhost was hanging and I couldn't figure out why. I eventually I realized I could just do docker-compose up without the -dto see debug info...I had a circular import on my current branch which wasn't allowing gunicorn to start up.)

    • mention in the docs how to set e.g. the ADMIN_EMAIL etc. Perhaps customise-server.env could become custom-server.env.example and we could have users copy that to custom-server.env initially (avoids Git conflicts down the line).

    Must haves:

    • Backup: - Cronjob that stores in a directory somewhere on the host (e.g. $QATROOT/backups or a user specified path?) - a db dump created from pg_dump - All files/directories from qatrack/media/uploads - site.css from qatrack/static/qatrack_core/css/site.css

      • On demand backup that does the same thing as the cronjob but triggered via command line? Once it's dumped to a host directory it will be up to the clinic to ensure that data is backed up.
    • Restore: - Reverse of above! Would be great if you could do something like docker-compose restore /path/to/backup/dir/with/above/items?

    • I'm sure it's relatively straightforward, but a bit of documentation about how I can run manage.py commands in the container. e.g. How do I do python manage.py shell?

    Nice to haves:

    • Possibly some discussion about SSL? Not sure how difficult it is to get this running with https. I guess there are probably two approaches a) SSL cert in the container itself or b) running something like nginx on the host to handle the https conection and having it reverse proxy (http) to the container. The latter is what I've typically done but I don't know what the typical docker deployment strategy is.

    Let me know if you need any clarifications on any of these points. If you're too busy to tackle these things now that's fine, either I will attempt to implement them in time for this release or we can push to the next release (maybe release it as, as an experimental installation method for 0.3.0).

  5. Simon Biggs reporter

    Hi Randle,

    Given that you're keen I'll try and make the time. I won't get the chance this week, I might be able to give it a try over the weekend or one night next week.

    If you could give the back up stuff a try that would be great.

    I'm keen to have people have minimal friction to have a full installation of QA Track installed. It'll open up the user base even further and that's a great thing.

    Thanks Randle :)

  6. Simon Biggs reporter

    Hi Randle,

    I have had a bit of interest from something else I've been building:

    https://github.com/SimonBiggs/scriptedforms

    So when I do get some home programming time I think I'll focus on cleaning up that code base. I need to write some tests and clean it up.

    I guess this is a long way to say I think it would be best just to leave this docker install method as an experimental install method. If someone really wants it I imagine they'll pick it up and make it work.

  7. Randle Taylor

    I saw your announcement about ScriptedForms...looks quite nice!

    OK, we will see if I have time to finish up the Docker install before this release, otherwise I can list it as an experimental deploy method until next release.

  8. Simon Biggs reporter

    I've never written a Django extension before but I imagine once I'm happy with where ScriptedForms is at it wouldn't be too hard to wrap it in a Django extension for use within QATrack at some point. That can be a far future goal though.

  9. Randle Taylor

    Would have to think a bit on how to best do that. API might be simplest to implement. ScriptedForms could then be used to create "alternative UI's" for performing some tests. Could be very cool :)

  10. Simon Biggs reporter

    It could be very cool :). ScriptedForms was where my brain wandered to when I was thinking about how best to rework test writing for QATrack+. Aiming to hit the "make tables", "use jupyter", "make it look how I want it to". So being able to hook it in via a REST API would be a nice full circle. :).

  11. Simon Biggs reporter

    Hi @randlet,

    So backups and restores are now occurring with postgres. Also, the docker install method has been made much more transparent. The python virtual env, the postgres data, pip-cache and others all gets dropped within deploy/docker/user-data. The qatrackplus directory is mapped into the docker image, so any changes made within the docker image to the qatrackplus directory can be seen on the user's local drive.

    There has also been just a general tidy up.

    So next is to address the points in your comment above:

    First, this is a very nice way to get a QATrack+ installation up and running :). ( Haven't tried on Windows yet, but assuming it works the same there, that will be very nice there too.)

    Thanks :). Windows 10 will work the same, but prior to that it will struggle which unfortunate. I made a note on that within the readme. I can make it so that prior Windows versions work without a hitch, but it will remove some of the niceties of how it is currently set up. Given Windows 7 is going out of support in 18 months I don't think this should be much of an issue.

    In the README with your link to the installation page for Docker Compose it might be a good idea to mention that people can/should use Docker CE (I think Docker EE always costs money?)

    Done

    I ran into (minor) trouble because Docker tried to bind to my host ports 5432 & 80 which are already occupied on my system. It's simple to change the configuration to bind to a different post on the host, but it should probably be mentioned here. Even better if the port can be specified in a config file (e.g. use 80/5432 as defaults but allow users to override

    Done

    DEBUG should be set to False by default

    Done

    Docs should mention that you can run docker-compose up without the -d if you need to see debug info (After running docker-compose up -d for the first time, going to http://localhost was hanging and I couldn't figure out why. I eventually I realized I could just do docker-compose up without the -dto see debug info...I had a circular import on my current branch which wasn't allowing gunicorn to start up.)

    Done

    mention in the docs how to set e.g. the ADMIN_EMAIL etc. Perhaps customise-server.env could become custom-server.env.example and we could have users copy that to custom-server.env initially (avoids Git conflicts down the line).

    Given these are just initial values, might it be okay to leave them as is and have the user adjust them through the QATrack interface?

    Must haves:

    Backup: - Cronjob that stores in a directory somewhere on the host (e.g. $QATROOT/backups or a user specified path?) - a db dump created from pg_dump - All files/directories from qatrack/media/uploads - site.css from qatrack/static/qatrack_core/css/site.css

    Backup is now using postgres, it is backing up media uploads, as well as site.css. It is also set to backup every 6 hours with crontab. These backup and restore methods need some serious testing with production databases.

    On demand backup that does the same thing as the cronjob but triggered via command line? Once it's dumped to a host directory it will be up to the clinic to ensure that data is backed up.

    At the moment it's just triggered each time docker-compose up is called.

    Restore: - Reverse of above! Would be great if you could do something like docker-compose restore /path/to/backup/dir/with/above/items?

    At the moment restore is triggered if a restore file is placed in the restore directory and then docker-compose up is called.

    I'm sure it's relatively straightforward, but a bit of documentation about how I can run manage.py commands in the container. e.g. How do I do python manage.py shell?

    Done

    Nice to haves:

    Possibly some discussion about SSL? Not sure how difficult it is to get this running with https. I guess there are probably two approaches a) SSL cert in the container itself or b) running something like nginx on the host to handle the https conection and having it reverse proxy (http) to the container. The latter is what I've typically done but I don't know what the typical docker deployment strategy is.

    I had a play around with setting up let's encrypt with some nice helper scripts. I did hit a few roadblocks however. In the end I think it would be multi-fold easier for end users to use CloudFlare's free 'one-click ssl', so I wrote in some basic instructions assuming that's what was being used.

  12. Simon Biggs reporter

    Thanks Randle!

    I did come across a potential hiccup on Windows 10:

    https://github.com/docker/for-win/issues/445#issuecomment-276117266

    I might have to make the postgres data storage be a docker volume instead of having it be mounted from the user-data directory like it is currently. That's unfortunate really, I was liking having all my data right there visible in my directory.

    I haven't got access to a Windows 10 machine to test if this is still an issue.

  13. Log in to comment