Backup

Automated process

Backups are automated via a cron service which runs daily. See backuper and the HuNI swarm definition.

The backed up data is stored in S3 Glacier at s3://huni-backups/. Path database for Postgres data and neo4j for neo4j data.

It runs in the swarm as a swarm_cronjob.

Backups are scheduled to run daily via the neo4j-backup_cron and postgres-backup_cron services.

Restore procedures

Open two terminals. The first is for a connection to the docker engine. The second for accessing the nodes via the bastion host.

Configure docker engine.

• Download the TLS certificates from AWS secrets manager. swarm-docker-client_cert swarm-docker-client_key and swarm-docker-ca

• Set up your environment

      export DOCKER_TLS_VERIFY="1";
      export DOCKER_HOST="tcp://api.huni.net.au:2376";
      export DOCKER_CERT_PATH="<local directory containing ca.pem, key.pem and cert.pem>";
  

• Take a local copy of the repository

Configure bastion host access

• Download the ssh key swarm-bastion-private-key from AWS secrets manager
• Copy the key to ~/.ssh

Postgres

• Download the required backup files from s3://huni-backups/database/ and gunzip it to a local directory.

• Start the database service in the docker engine terminal window. docker stack deploy -c docker-compose-database.yml huni

• Restore the database huni-harvestdb-production. e.g.

  docker run --rm -v huni-harvestdb-production-postgres.dump-06_26_2020.dump:/tmp/huni-harvestdb-production-postgres.dump-06_26_2020.dump postgres:11-alpine psql -U postgres -d huni-harvestdb-production /tmp/huni-harvestdb-production-postgres.dump-06_26_2020.dump
  

• Restore the database huni-production. e.g.

  docker run --rm -v huni-production-postgres.dump-10_29_2020.dump:/tmp/huni-production-postgres.dump-10_29_2020.dump postgres:11-alpine psql -U postgres -d huni-production /tmp/huni-production-postgres.dump-10_29_2020.dump
  

• Start the database service in the docker engine terminal window docker stack deploy -c docker-compose-database.yml huni

neo4j

• Stop the neo4j service in the docker engine terminal window docker service rm huni_neo4j

• Download the backup folder required from s3://huni-backups/neo4j/

• Find the neo4j node docker node ls -f "label=node_type=neo4j" --format '{{.Hostname}}'

• Copy the folder to the bastion host then onto the neo4j node. e.g. scp -r -i <path to swarm-bastion-private-key> huni-production-neo4j-06_26_2020 ubuntu@ssh.huni.net.au:
scp -r huni-production-neo4j-06_26_2020/ <hostname from above command>

• Access the node via the bastion host, then onto the neo4j node ssh -i <path to swarm-bastion-private-key> ubuntu@ssh.huni.net.au ssh <hostname from above command>

• Restore the backup. e.g.

  docker run -v /data/ap-southeast-2b/neo4j-data/:/data -v huni-production-neo4j-06_26_2020:/tmp/huni-production-neo4j-06_26_2020 /var/lib/neo4j/bin/neo4j-admin restore --from=/tmp/huni-production-neo4j-06_26_2020
  

• Start the neo4j service in the docker engine terminal window docker stack deploy -c docker-compose-webapp.yml huni