Backup script for Confluence Cloud (find it attached to this issue and in the Description)

Create issue
Issue #17 resolved
Dario Bonotto created an issue

The below script works fine for me. Just tested in the new infrastructure:

#!/bin/bash

##---Fill-up the user account details, instance url and timezone-----
USERNAME=USER_NAME_HERE
PASSWORD=PASSWORD_HERE
INSTANCE=INSTANCE.atlassian.net
LOCATION="/path/to/download/folder"

# Set this to your Atlassian instance's timezone.
# See this for a list of possible values:
# https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
TIMEZONE=Europe/Amsterdam

##----START-----###
echo "Starting the script..."

# Grabs cookies and generates the backup on the UI. ########
### PLEASE NOTICE THAT THE SESSION IS CREATED BY CALLING THE JIRA SESSION ENDPOINT!!! ######
#### THE SCRIPT DOES NOT WORK IF JIRA IS NOT INSTALLED !!! #######
TODAY=$(TZ=$TIMEZONE date +%Y%m%d)
COOKIE_FILE_LOCATION=jiracookie
curl --silent --cookie-jar $COOKIE_FILE_LOCATION -X POST "https://${INSTANCE}/rest/auth/1/session" -d "{\"username\": \"$USERNAME\", \"password\": \"$PASSWORD\"}" -H 'Content-Type: application/json' --output /dev/null

## The $BKPMSG variable will print the error message, you can use it if you're planning on sending an email
BKPMSG=$(curl -s --cookie $COOKIE_FILE_LOCATION --header "X-Atlassian-Token: no-check" -H "X-Requested-With: XMLHttpRequest" -H "Content-Type: application/json"  -X POST https://${INSTANCE}/wiki/rest/obm/1.0/runbackup -d '{"cbAttachments":"true" }' )

 ## Checks if the backup procedure has failed
if [ "$(echo "$BKPMSG" | grep -ic backup)" -ne 0 ]; then
rm $COOKIE_FILE_LOCATION
echo "FAILED, IT RETURNED $BKPMSG"
exit
fi

## Checks if the backup exists every 10 seconds, 2000 times. If you have a bigger instance with a larger backup file you'll probably want to increase that.
for (( c=1; c<=2000; c++ ))
do
PROGRESS_JSON=$(curl -s --cookie $COOKIE_FILE_LOCATION https://${INSTANCE}/wiki/rest/obm/1.0/getprogress.json)
FILE_NAME=$(echo "$PROGRESS_JSON" | sed -n 's/.*"fileName"[ ]*:[ ]*"\([^"]*\).*/\1/p')

##ADDED: PRINT BACKUP STATUS INFO ##
echo "$PROGRESS_JSON"

if [[ $PROGRESS_JSON == *"error"* ]]; then
break
fi

if [ ! -z "$FILE_NAME" ]; then
break
fi
sleep 10
done

#If after 2000 attempts it still fails it ends the script.
if [ -z "$FILE_NAME" ];
then
rm $COOKIE_FILE_LOCATION
exit
else

## PRINT THE FILE TO DOWNLOAD ##
echo "File to download: $FILE_NAME"

curl -s -L --cookie $COOKIE_FILE_LOCATION "https://${INSTANCE}/wiki/download/$FILE_NAME" -o "$LOCATION/CONF-backup-${TODAY}.zip"

fi
rm $COOKIE_FILE_LOCATION

Official response

Comments (29)

  1. James Craig

    Props to you Kenta!! Dropping by to 7 month old post to PROACTIVELY give us a heads up about that change.

    THANK YOU!!

  2. Devin St. Clair

    OK, so we don't need to use username and password? What would replace the lines using username and password then? I understand what API tokens are and generated one (even though the page you do so states they are only used when two factor is turned on), but I have no idea how to call it in regards to the script.

  3. Dario Bonotto reporter

    I don't understand your question. Can you try to be a bit more specific and point to the lines you are referring to?

    Also, did you see the comment from Kenta Yamamoto?

  4. Devin St. Clair

    I did see Kenta's response, I did finally get it fixed based on his example after doing research and testing not knowing what I needed to remove from your curl commands (still new to working with APIs from command line). Everything is good now for Confluence, but my Jira script seems to no longer work for some reasons. I just keep getting back:

    {"fileName":"temp/filestore/e0613346-ed9d-45de-84f1-e4439878a99d","size":406365173,"currentStatus":"Archiving attachments.","alternativePercentage":"100%","concurrentBackupInProgress":false}
    

    I contact Atlassian Support and basically got back "Oh I only use UI, I don't know anything about the API, here is a list of feature requests to add automated backups in the UI, none of them are being worked on though." Yes, seriously.

    Basically when I run the Jira script (which is pretty much the same except for slightly different endpoints) I get back that error message over and over again, I've been poking at this all week. Also the script worked perfectly fine 2 weeks ago when they still allowed cookie auth, not quite sure why the Confluence works just fine after I switched to token but the Jira one won't...

  5. Dario Bonotto reporter

    Hi Devin,

    Automatic backups are indeed not officially supported as you can read in: * https://confluence.atlassian.com/jirakb/how-to-automate-backups-for-jira-cloud-applications-779160659.html

    Now, from what I can see your script is telling you that the backup is completed (100%) and it has been saved in 'temp/filestore/e0613346-ed9d-45de-84f1-e4439878a99d'.

    Can you kindly confirm you are using the below version of the script? * https://bitbucket.org/atlassianlabs/automatic-cloud-backup/issues/16/endpoint-changed-for-jira-again-new-script

    In case you are, most likely the below line will have to be changed to reflect the new response:

    FILE_NAME=$(echo "$PROGRESS_JSON" | sed -n 's/.*"result"[ ]*:[ ]*"\([^"]*\).*/\1/p')
    

    If I have some time I will try to test and fix the script this week.

    Cheers, Dario

  6. gitadmin@goodstream.eu

    Hello Dario

    First of all, thanks a lot for your script.

    And one more request or report. We want to organise notifying (via email for example) that script sucsecfuly done. But when we check result and code of status, we find file with backup and that your script had "Last run result" is 0x1 (incorrect function call). backup_conf.png

    Could you fix it please? Thanks a lot, Dmytro

  7. Dario Bonotto reporter

    Can you kindly confirm whether you are using the script at the top of this page and or the powershell one?

  8. Dario Bonotto reporter

    Sorry for late reply. Tomorrow I am going to test the script and try to update this issue :)

  9. Dario Bonotto reporter

    I have just tested the script and it works fine for me.

    Can you kindly paste the last lines of what is logged in the console when you run the script?

    E.g. In my case I have:

    ...
    ....
    {"size":0,"currentStatus":"Archiving attachments.","alternativePercentage":"Progress: 26% longer than last time","concurrentBackupInProgress":false}
    {"size":0,"currentStatus":"Archiving attachments.","alternativePercentage":"Progress: 26% longer than last time","concurrentBackupInProgress":false}
    {"size":0,"currentStatus":"Archiving attachments.","alternativePercentage":"Progress: 27% longer than last time","concurrentBackupInProgress":false}
    {"size":0,"currentStatus":"Archiving attachments.","alternativePercentage":"Progress: 28% longer than last time","concurrentBackupInProgress":false}
    {"fileName":"temp/filestore/0dc57fde-b470-44f2-b959-c0511caf75c3","size":4561838724,"currentStatus":"Archiving attachments.","alternativePercentage":"100%","concurrentBackupInProgress":false}
    File to download: temp/filestore/0dc57fde-b470-44f2-b959-c0511caf75c3
    

    And as you can see the backup was correctly downloaded and it is about 4.5GB (the size matches the one returned by the script):

    $ ls -la
    total 4454944
    drwxr-xr-x 2 dbonotto dbonotto       4096 Feb 14 20:01 .
    drwxr-xr-x 4 dbonotto dbonotto       4096 Dec 18 18:52 ..
    -rw-rw-r-- 1 dbonotto dbonotto 4561838724 Feb 14 20:01 CONF-backup-20190214.zip
    -rwxr-x--x 1 dbonotto dbonotto       2429 Feb 14 19:20 Confluence-backup.sh
    
  10. Dainius Narsutis

    Hi, I've been trying to integrate your script, I've entered all details required, but for some reason the script cannot authenticate with the Confluence Cloud instance. It keep responding fir 401 error. The instance I am trying to backup is on Confluence Cloud.

    I tried various username/password combinations: - email + password - email + token - username + password - username + token

    I am the admin of the instance.

    Capture.PNG

    Any advise would be appreciated.

    edit: I can confirm that the jiracookie file is being created. I have remove the -s from the curl command line and this is the output

    Capture.PNG

  11. Dario Bonotto reporter

    The attached script only works with Cookie Authentication (that is going to be deprecated soon).

    I will try to update the script to use API Tokens as soon as I have time. In the meanwhile please notice that you are getting 401 Unauthorized.

    Now, the combination email/password to be used in the script are the same credentials that you can use to log-into id.atlassian.com

    Can you confirm you are able to log-into id.atlassian.com using the same credentials?

    Also, it happened to me in the past that I had to reset the password in id.atlassian.com to make this work. Can you try if the same procedure works for you?

    Please let me know

  12. Dainius Narsutis

    I had to reset my password for the script to work. It Works! Having API token authentication would be fantastic.

    Thank you, Dario.

  13. Dario Bonotto reporter

    YOOO!!! :)

    Super happy to hear so.

    Next week I should have some time to test a new script using API Tokens that I have already started writing. I don't want to set expectations but finger crossed and let's hope that everything will be fine ;)

    Enjoy the weekend!

    Dario

  14. gitadmin@goodstream.eu

    Hi Dario.

    Thank you very much for your script, it is very useful for us. With impatience, we are waiting for the next version with support for the Tokens API.

    Best wishes.

  15. Dev Pandey

    Hi Dario,

    Thank you very much for the script. but for some reason, the script cannot authenticate with the Confluence Cloud instance. It keeps responding fir 401 error. The instance I am trying to backup is on Confluence Cloud.

    Please let me know in case i am missing something in Authentication part. Screenshot 2019-03-14 at 10.52.27 PM.png

  16. Dario Bonotto reporter

    @5ae8a8ced5c9632c3013bb70 ,

    The above script has some requirements: 1. You must have Jira as well in order to be able to call the session endpoint 2. You need to provide email address and the password you use to access id.atlassian.com and not the API Token 3. In some cases, you need to reset the password in https://id.atlassian.com/manage-profile/security in order to be able to correctly authenticate

    Can you confirm all the above requirements are matched?

    Thanks, Dario

  17. Dario Bonotto reporter

    @"gitadmin@goodstream.eu" Thanks a lot, it's this kind of feedback that keep me motivated to continue maintaining these scripts! :)

    I need some time to test the script and I will let you know how it goes asap!

  18. Dev Pandey

    Hi Dario,

    Thanks a lot for your quick response yes i am using user name password in a script. after reset password, i am getting below changed error.

    Devs-MacBook-Air:Downloads devpandey$ ./Confluence-backup.sh Starting the script... FAILED, IT RETURNED User may not create a backup.

    Thanks, Dev

  19. Dario Bonotto reporter

    Hi Dev,

    From the error message it looks like the user does not have rights to create a backup. Can you kindly make sure you have the proper rights?

    Otherwise, just try to create a backup from the UI using the same credentials to log-in (it may be a good idea to try in an incognito/private browser window). It can be the case that either you don't have right or you have to wait for the backup timer to be cleared.

  20. Elmin Custo

    Hi Dario,

    I am using scripts backup-jira-api-token.sh and backup-confluence-api-token.sh to automate backups of my Atlassian cloud environment. Backup of JIRA is successful, and restore was successful as well. Backup of Confluence seems to be OK, but when I want to restore to my other cloud site, I get the error:

     Import error
    
     There was an error importing file CONF-backup-15-04-2019.zip: Validation failed. Following issues were reported:
     Error importing data: org.xml.sax.SAXException: The XML document does not contain the <entity-engine-xml> root element or it was closed too early.
    

    Do you have any suggestions or recommendation what to could be potential problem? Could it be that scripts for Confluence backup is not working properly?

    Thanks, Elmin.

  21. Dario Bonotto reporter

    Please notice that this issue is closed and it is related to a different script than the one you are using.

    As a quick answer I can tell you to check if the zip file has been created correctly since from the error message it looks like the importer is failing to find the entities.xml file (it is not supposed to be in any folder)

    Also, Confluence restore is known to be broken if you have both, Jira and Confluence:

    If this is not the case then please open a new issue and provide more details.

  22. Elmin Custo

    Thank you very much Dario for this answer, now I have a clear picture. I have both on the same subscription so that is the reason why my Confluence restore fails. Is there a way/script to automate backup of spaces one by one?

    Best regards, Elmin.

  23. Log in to comment