Powershell script was working but now fails
The powershell script is now failing with a message that says "Failed to downgrade JIRA Cloud to JIRA Server common denominator version", when it attempts to download the backup file after it is generated.
Comments (32)
-
-
The Powershell script is no longer returning the progress status.
-
It looks like they fixed the "Failed to downgrade JIRA Cloud to JIRA Server common denominator version" problem as I was getting it also and it is working now for me.
-
Yeah that error about "failed to downgrade..." is no longer occurring but I am finding the script seems to be failing at the below step and throwing the "Attempted to download from WEBDAV directory, which is no longer supported" error.
The actual backup zip file is getting created as I can download it manually from within JSD. Anyone experience the same?
$pathName = $status.fileName if ($pathName -match "ondemandbackupmanager/download/.+/(.*)") { $fileName = $Matches[1] Write-Host "Downloading: $fileName to JIRA-backup-$today.zip" $progressPreference = 'Continue' Invoke-WebRequest -Method Get -Headers @{"Accept"="*/*"} -WebSession $session -Uri "https://$hostname/$pathName" -OutFile (Join-Path -Path $destination -ChildPath "JIRA-backup-$today.zip") } else { throw "Attempted to download from WEBDAV directory, which is no longer supported" }
-
Mine stopped working too, about a week ago. Was getting authentication errors. I had to update my login command, change from invoke-webrequest to invoke-restmethod and update the content type to json, just like the backup triggering statement. Here's the new line I used. Hope it helps others...
# New Session Invoke-RestMethod -UseBasicParsing -Method Post ` -Uri "$account_url/rest/auth/1/session" -SessionVariable session ` -Body (@{username = $username; password = $password} | convertTo-Json -Compress) ` -ContentType 'application/json'
-
I followed the modification made by TT.Smith above and I still cannot authenticate with the script. I've double and triple-checked my credentials and they are correct. I can use them to log in directly with id.atlassian.net but not through the script. The script returns "[ERROR] Invoke-RestMethod : The remote server returned an error: (401) Unauthorized." in the output when attempting to hit the /rest/auth/1/session endpoint on our cloud instance. Any ideas?
-
Has any movement occurred on this? I am having the same issue ("[ERROR] Invoke-RestMethod : The remote server returned an error: (401) Unauthorized").
I've been troubleshooting it and nothing is working.
-
I think that the problem is related with the new Login Method , BTW I still not understand how they don't have a more efficient way to create backups.
-
Don't forget that Atlassian moved all accounts over to Atlassian IDs about 7 days ago, so that may be the cause of your problems. i.e. Bitbucket, Jira & Confluence all use a common authentication system now.
-
I was able to get it working again using the method from TT.Smith and my new Atlassian ID.
-
@antonybrooke-wood How? Do you mind sharing your PS script, minus your account / username / pass information? I've been trying to do @TT_Smith 's modification and still can't seem to get it to work.
-
@vespene56 I have attached both my Jira script & my Confluence one - they are slightly different.
JIRA
<# DESCRIPTION: This script backs up Jira from Atlassian Cloud. AUTHOR: Antony Brooke-Wood SOURCE: https://bitbucket.org/atlassianlabs/automatic-cloud-backup https://github.com/ghuntley/atlassian-cloud-backup/blob/master/src/AtlassianCloudBackupClient/BackupClient.cs #> # Set variables $Account = 'XXXX' $Username = 'XXXX' $Password = 'XXXX' $AwsRegion = 'XXXX' $Destination = 'D:\Backups\Jira' $Attachments = $true $Cronitor = 'XXXX' $LogName = 'BackupJira.log' $FilesToKeep = 3 $Today = Get-Date -format yyyyMMdd $ScriptName = $MyInvocation.MyCommand.Name $LogPath = (Join-Path -Path $Destination -ChildPath $LogName) $ErrorActionPreference = 'stop' # Define logging function Function Write-Log([string]$topic, [string]$logString) { $timeStamp = [string]::Format("{0}", (Get-Date).ToString("g")) $logLine = "$($timestamp) [$($topic)] $($logString)" Add-content $LogPath -value $logLine Write-Host -Object $logLine } # Prepare the shell, set the AWS configuration, create the backup directory and begin logging. Clear-Host Set-DefaultAWSRegion $AwsRegion New-Item -ItemType Directory -Force -Path $Destination | Out-Null Write-Log "$($ScriptName)" "Beginning the backup process for Jira." # Login to Atlassian Write-Log "$($ScriptName)" "Logging in to Atlassian." Invoke-RestMethod -UseBasicParsing -Method Post ` -Uri "https://$Account.atlassian.net/rest/auth/1/session" -SessionVariable session ` -Body (@{username = $Username; password = $Password} | convertTo-Json -Compress) ` -ContentType 'application/json' # Request backup creation Write-Log "$($ScriptName)" "Triggering backup." Invoke-RestMethod -Method Post -Uri "https://$Account.atlassian.net/rest/obm/1.0/runbackup" -WebSession $session -ContentType 'application/json' -Body (@{cbAttachments = $Attachments} | ConvertTo-Json -Compress) | Out-Null # Wait for backup to finish Write-Log "$($ScriptName)" "Waiting for backup generation to finish." do { $status = Invoke-RestMethod -Method Get -Uri "https://$Account.atlassian.net/rest/obm/1.0/getprogress" -WebSession $session $status.alternativePercentage -match "(\d+)" Write-Progress -Activity 'Creating backup' -Status $status.alternativePercentage -PercentComplete $Matches[1] Start-Sleep -Seconds 5 } while(! $status.fileName) # Download the backup Write-Log "$($ScriptName)" "Downloading backup https://$Account.atlassian.net$($status.fileName)" Invoke-WebRequest -UseBasicParsing -Method Get "https://$Account.atlassian.net$($status.fileName)" -WebSession $session -OutFile (Join-Path -Path $Destination -ChildPath "Jira-backup-$Today.zip") -ErrorAction Stop # Clean up old backup files Write-Log "$($ScriptName)" "Removing old backup files." $OldBackups = @( Get-ChildItem $Destination -Recurse | where{-not $_.PsIsContainer} | where{$_.Name -notlike $LogName} | sort LastWriteTime -desc | select -Skip $FilesToKeep ) foreach ($file in $OldBackups) { Write-Log "$($ScriptName)" "Removing $file" Remove-Item -Force $file.FullName } # Finishing up Write-Log "$($ScriptName)" "Pinging Cronitor." Invoke-WebRequest -UseBasicParsing "https://cronitor.link/$cronitor/complete" | Out-Null Write-Log "$($ScriptName)" "Backup complete."
CONFLUENCE
<# DESCRIPTION: This script backs up Confluence from Atlassian Cloud. AUTHOR: Antony Brooke-Wood SOURCE: https://bitbucket.org/atlassianlabs/automatic-cloud-backup https://github.com/ghuntley/atlassian-cloud-backup/blob/master/src/AtlassianCloudBackupClient/BackupClient.cs #> # Set variables $Account = 'XXXX' $Username = 'XXXX' $Password = 'XXXX' $AwsRegion = 'XXXX' $Destination = 'D:\Backups\Confluence' $Attachments = $true $Cronitor = 'XXXX' $LogName = 'BackupConfluence.log' $FilesToKeep = 3 $Today = Get-Date -format yyyyMMdd $ScriptName = $MyInvocation.MyCommand.Name $LogPath = (Join-Path -Path $Destination -ChildPath $LogName) $ErrorActionPreference = 'stop' # Define logging function Function Write-Log([string]$topic, [string]$logString) { $timeStamp = [string]::Format("{0}", (Get-Date).ToString("g")) $logLine = "$($timestamp) [$($topic)] $($logString)" Add-content $LogPath -value $logLine Write-Host -Object $logLine } # Prepare the shell, set the AWS configuration, create the backup directory and begin logging. Clear-Host Set-DefaultAWSRegion $AwsRegion New-Item -ItemType Directory -Force -Path $Destination | Out-Null Write-Log "$($ScriptName)" "Beginning the backup process for Confluence." # Login to Atlassian Write-Log "$($ScriptName)" "Logging in to Atlassian." Invoke-RestMethod -UseBasicParsing -Method Post ` -Uri "https://$Account.atlassian.net/rest/auth/1/session" -SessionVariable session ` -Body (@{username = $Username; password = $Password} | convertTo-Json -Compress) ` -ContentType 'application/json' # Request backup creation Write-Log "$($ScriptName)" "Triggering backup." Invoke-RestMethod -Method Post -Uri "https://$Account.atlassian.net/wiki/rest/obm/1.0/runbackup" -WebSession $session -ContentType 'application/json' -Body (@{cbAttachments = $Attachments} | ConvertTo-Json -Compress) -UseBasicParsing | Out-Null # Wait for backup to finish Write-Log "$($ScriptName)" "Waiting for backup generation to finish." do { $status = Invoke-RestMethod -Method Get -Uri "https://$Account.atlassian.net/wiki/rest/obm/1.0/getprogress" -WebSession $session $status.alternativePercentage -match "(\d+)" Write-Progress -Activity 'Creating backup' -Status $status.alternativePercentage -PercentComplete $Matches[1] Start-Sleep -Seconds 5 } while(! $status.fileName) # Download the backup Write-Log "$($ScriptName)" "Downloading backup https://$Account.atlassian.net/wiki/download/$($status.fileName)" Invoke-WebRequest -UseBasicParsing "https://$Account.atlassian.net/wiki/download/$($status.fileName)" -WebSession $session -OutFile (Join-Path -Path $Destination -ChildPath "Confluence-backup-$Today.zip") # Clean up old backup files Write-Log "$($ScriptName)" "Removing old backup files." $OldBackups = @( Get-ChildItem $Destination -Recurse | where{-not $_.PsIsContainer} | where{$_.Name -notlike $LogName} | sort LastWriteTime -desc | select -Skip $FilesToKeep ) foreach ($file in $OldBackups) { Write-Log "$($ScriptName)" "Removing $file" Remove-Item -Force $file.FullName } # Finishing up Write-Log "$($ScriptName)" "Pinging Cronitor." Invoke-WebRequest -UseBasicParsing "https://cronitor.link/$cronitor/complete" | Out-Null Write-Log "$($ScriptName)" "Backup complete."
-
@antonybrooke-wood Thanks, your script helped fill the holes in mine. Now I just need to get powershell to be ok with downloading 3gb+ files. Thanks again.
-
@antonybrooke-wood Thanks! This helped heaps.
I was not able to get the authentication right prior to this so this is greatly appreciated.
-
@antonybrooke-wood Awesome, thank you. Your scripts helped me as well. I had almost everything correct but when I modified the PS script in the repo to change the login call to a REST call, I was apparently using this parameter and value
-WebSession $session
whereas what worked (from your script) was
-SessionVariable session
Thanks again, man. :)
-
@gavinroberts No worries. The WebSession v SessionVariable is a little odd. Details are in the link but basically you should use SessionVariable if you want to assign it to a variable and re-use it elsewhere. https://msdn.microsoft.com/en-us/powershell/reference/5.1/microsoft.powershell.utility/invoke-restmethod
-
Now that I have a working script, I've been able to build a .NET Core console application that does that same thing. If there's any interest, I'll put it out there on GitHub when I get it cleaned up a little.
-
@gavinroberts I'd be keen to see that if you could post it.
-
@antonybrooke-wood, I've placed the .NET core v2 project out on my GitHub account. It can be found at https://github.com/gsroberts/atlassian-cloud-backup-tool.
I welcome any feedback.
(Edited to point to the public GitHub repo rather than the private BitBucket repo)
-
Big thanks to Anthony Brooke-Wood for the solution, works great.
I was able to get secure credentials working as well with the automation. In my case we use a job scheduler to execute jobs so i didnt want my password out there for the masses to see.
#Execute fist to create your secure credential txt file but do not include in your powershell script $username = "username" $password = "password" $secureStringPwd = $password | ConvertTo-SecureString -AsPlainText -Force $creds = New-Object System.Management.Automation.PSCredential -ArgumentList $username, $secureStringPwd $secureStringText = $secureStringPwd | ConvertFrom-SecureString Set-Content "C:\creds\jira.txt" $secureStringText # Powershell script with Secure Credentials included <# DESCRIPTION: This script backs up Jira from Atlassian Cloud. AUTHOR: Antony Brooke-Wood SOURCE: https://bitbucket.org/atlassianlabs/automatic-cloud-backup https://github.com/ghuntley/atlassian-cloud-backup/blob/master/src/AtlassianCloudBackupClient/BackupClient.cs UPDATES: 7-18-2017 - Generate Backups with "my company" Account 7-19-2017 - Generate Secure Credentials #> # Specify Secury Credentials $username = "username" $pwdTxt = Get-Content "C:\creds\jira.txt" $securePwd = $pwdTxt | ConvertTo-SecureString $creds = New-Object System.Management.Automation.PSCredential -ArgumentList $username, $securePwd # Set variables $Account = 'xxxxx' #$Username = 'xxxxxx' #$Password = $cred.Password $Destination = '\\my-networkshare\' $Attachments = $true $LogName = 'BackupJira.log' $FilesToKeep = 10 $Today = Get-Date -format yyyyMMdd $ScriptName = $MyInvocation.MyCommand.Name $LogPath = (Join-Path -Path $Destination -ChildPath $LogName) $ErrorActionPreference = 'stop' # Define logging function Function Write-Log([string]$topic, [string]$logString) { $timeStamp = [string]::Format("{0}", (Get-Date).ToString("g")) $logLine = "$($timestamp) [$($topic)] $($logString)" Add-content $LogPath -value $logLine Write-Host -Object $logLine } # Prepare the shell, set the AWS configuration, create the backup directory and begin logging. Clear-Host New-Item -ItemType Directory -Force -Path $Destination | Out-Null Write-Log "$($ScriptName)" "Beginning the backup process for Jira." # Login to Atlassian Write-Log "$($ScriptName)" "Logging in to Atlassian." Invoke-RestMethod -Method Post ` -Uri "https://$Account.atlassian.net/rest/auth/1/session" -SessionVariable session ` -Body (@{username = $username ; password = $password } | convertTo-Json -Compress) ` -ContentType 'application/json' # Request backup creation Write-Log "$($ScriptName)" "Triggering backup." Invoke-RestMethod -Method Post -Uri "https://$Account.atlassian.net/rest/obm/1.0/runbackup" -WebSession $session -ContentType 'application/json' -Body (@{cbAttachments = $Attachments} | ConvertTo-Json -Compress) | Out-Null # Wait for backup to finish Write-Log "$($ScriptName)" "Waiting for backup generation to finish." do { $status = Invoke-RestMethod -Method Get -Uri "https://$Account.atlassian.net/rest/obm/1.0/getprogress" -WebSession $session $status.alternativePercentage -match "(\d+)" Write-Progress -Activity 'Creating backup' -Status $status.alternativePercentage -PercentComplete $Matches[1] Start-Sleep -Seconds 5 } while(! $status.fileName) # Download the backup Write-Log "$($ScriptName)" "Downloading backup https://$Account.atlassian.net$($status.fileName)" Invoke-WebRequest -UseBasicParsing -Method Get "https://$Account.atlassian.net$($status.fileName)" -WebSession $session -OutFile (Join-Path -Path $Destination -ChildPath "Jira-backup-$Today.zip") -ErrorAction Stop # Clean up old backup files Write-Log "$($ScriptName)" "Removing old backup files." $OldBackups = @( Get-ChildItem $Destination -Recurse | where{-not $_.PsIsContainer} | where{$_.Name -notlike $LogName} | sort LastWriteTime -desc | select -Skip $FilesToKeep ) foreach ($file in $OldBackups) { Write-Log "$($ScriptName)" "Removing $file" Remove-Item -Force $file.FullName } # Finishing up Write-Log "$($ScriptName)" "Pinging Cronitor." #Invoke-WebRequest -UseBasicParsing "https://cronitor.link/$cronitor/complete" | Out-Null Write-Log "$($ScriptName)" "Backup complete."
-
Can anyone help on how to get rid from 401 error.
VERBOSE: POST https://$account.atlassian.net/rest/backup/1/export/runbackup with -1-byte payload {"message":"Client must be authenticated to access this resource.","status-code":401} Invoke-WebRequest : The remote server returned an error: (401) Unauthorized. At C:\Users\jira char:16 + $GetBackupID = Invoke-WebRequest -Method Get -WebSession $session https://$hostn ... +
~~~~~~~~~~~~~~~~+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebExeption + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommandInvoke-RestMethod : The remote server returned an error: (401) Unauthorized. At C:\Users\jira\scriptname.ps1:58 char:15 + $status = Invoke-RestMethod -Method Get -Headers @{"Accept"="application/jso ... +
~~~~~~~~~~~~~~~~ -
Hmm... I'm not seeing that particular issue, but as Dario noted in this issue:
"If you get error 401, you are most likely affected by one the the below issues:
For the both of them the solution is to actually reset the password for your account in id.atlassian.com (you can just re-set the same password as before).
Also, please be aware that some endpoints have changed in the new infrastructure:
Hope that helps. :)
-
Thanks @gavinroberts that helped me, right now I passed 401 error but stuck with another error "waiting for the back up to finish" (500) internal server error.
-
What's the method you are calling for the progress status?
I'm using
rest/backup/1/export/getProgress?taskId={0}
Where the task ID is obtained from either the backup trigger response or the
rest/backup/1/export/lastTaskId
Endpoint.
If you are familiar with C#, this is the class I'm using in my backup utility and so far it's been working.
The work is being done in the Execute method.
The full dotnetcore 2 project I'm using can be found here: https://github.com/gsroberts/atlassian-cloud-backup-tool
-
PowerShell using secure Credentials
I posted a bit ago about setting up this script with secure credentials as in my situation we can not store passwords in scripts. I was unable to get the original script to work passing -Credentials $creds as it was looking a for specific username and password and would not accept a credential file.
Since then i have found that you will have to use a system runtime method to convert the secure string to Binary String (BSTR) which could then be used to convert to a human readable string, i.e. passing the password in the password field.
below is my create script for creating the secure credentials and the two additional variables used to convert to a Binary string and then to convert it.
Creating a secure password file - (critical step) make sure you are logged into the machine with the same user account that will run or execute this script
$password = "xxxxxxxx" $securePwd = $password | ConvertTo-SecureString -AsPlainText -Force $secureStringText = $securePwd | ConvertFrom-SecureString Set-Content "C:\creds\folder\complexPassword.txt" $secureStringText
powershell variables, added $BSTR & $UnsecurePassword
$Account = 'myCompany' $username = "myUsername" $password = Get-Content "C:\creds\folder\complexPassword.txt" $securePwd = $password | ConvertTo-SecureString $credential = New-Object System.Management.Automation.PSCredential -ArgumentList $username, $securePwd $BSTR = [System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($securePwd) $UnsecurePassword = [System.Runtime.InteropServices.Marshal]::PtrToStringAuto($BSTR)
Log into Atlassian, pass the $unsecurePassword in the password field.
# Login to Atlassian Write-Log "$($ScriptName)" "Logging in to Atlassian." Invoke-RestMethod -Method Post -Uri "https://$Account.atlassian.net/rest/auth/1/session" -SessionVariable session -Body (@{username = $username ; password = $UnsecurePassword } | convertTo-Json -Compress) -ContentType 'application/json'
-
I ran into some backup issues with percentage greater than 100 on my confluence backup. I have update the script from the below
# Wait for backup to finish Write-Log "$($ScriptName)" "Waiting for backup generation to finish." do { $status = Invoke-RestMethod -Method Get -Uri "https://$Account.atlassian.net/wiki/rest/obm/1.0/getprogress" -WebSession $session $status.alternativePercentage -match "(\d+)" Write-Progress -Activity 'Creating backup' -Status $status.alternativePercentage -PercentComplete $Matches[1] Start-Sleep -Seconds 5 } while(! $status.fileName)
To
# Wait for backup to finish Write-Log "$($ScriptName)" "Waiting for backup generation to finish." do { $status = Invoke-RestMethod -Method Get -Uri "https://$Account.atlassian.net/wiki/rest/obm/1.0/getprogress" -WebSession $session if ( $status.alternativePercentage -match "(\d+)"){ $percent = $Matches[1] if ([int] $percent -gt 100) { $percent = "100"} } Write-Progress -Activity 'Creating backup' -Status $status.alternativePercentage -PercentComplete $Matches[1] Start-Sleep -Seconds 5 } while(! $status.fileName)
This is to stop the percent value at 100 and only indicate 100% even if it's greater.
-
Looks like there were further changes to the JSON that is returned by the getProgress API call. As a result, I had to modify the way that I check for percentage completeness as well as the name of the file to download. I've also added some of the changes from @jgovernale and @gavinroberts . The resultant changes are as follows:
# Wait for backup to finish Write-Log "$($ScriptName)" "Waiting for backup generation to finish." do { $taskID = Invoke-RestMethod -Method Get -Uri "https://$Account.atlassian.net/rest/backup/1/export/lastTaskId" -WebSession $session $status = Invoke-RestMethod -Method Get -Uri "https://$Account.atlassian.net/rest/backup/1/export/getProgress?taskId=$taskID" -WebSession $session if ($percent = $status.progress){ if ([int] $percent -gt 100) { $percent = "100"} } Write-Progress -Activity 'Creating backup' -Status $status.status -PercentComplete $percent Start-Sleep -Seconds 5 } while(! $status.result) # Download the backup Write-Log "$($ScriptName)" "Downloading backup https://$Account.atlassian.net/plugins/servlet/$($status.result)" Invoke-WebRequest -UseBasicParsing -Method Get "https://$Account.atlassian.net/plugins/servlet/$($status.result)" -WebSession $session -OutFile (Join-Path -Path $Destination -ChildPath "Jira-backup-$Today.zip") -ErrorAction Stop
-
Our confluence site has grown significantly and now when i go to download the backup from my powershell script i receive a system out of memory issue. Has any one experienced this, gotten past it? We tried increasing up the memory on the powershell shell it self to 3 GB but still with no luck.
it occurs once we try to invoke the download
Invoke-WebRequest -UseBasicParsing "https://$Account.atlassian.net/wiki/download/$($status.fileName)" -WebSession $session -OutFile (Join-Path -Path $Destination -ChildPath "Confluence-backup-$Today.zip")
-
@jgovernale The Invoke-WebRequest method buffers the response stream into memory and then once the entire file has been downloaded and in memory it's flushed to disk. You may want to modify the script to use
System.Net.WebClient
orSystem.Net.HttpClient
instead so that the stream gets flushed to disk regularly throughout the download process, which would hopefully help your out of memory issue. My .NET core backup utility I wrote based on this powershell script has no trouble downloading a 4.8GB Confluence backup and uses the sameSystem.Net
classes to accomplish the task. (https://github.com/gsroberts/atlassian-cloud-backup-tool) -
Thanks @gavinroberts for the information! Huge help but i am getting Stuck on how to pass the websession information as i just get 401 unauthorized. Sorry i have next to zero .net experience
-
Sorry, @jgovernale , I've been a little wrapped up the last few days but I'll see if I can't post a code snippet that demonstrates using the
System.Net.WebClient
with preauthenticated session information in PowerShell some time today or tomorrow. -
- changed status to resolved
Closing this since the PS script is using cookie authentication that has already been deprecated some time ago:
- Log in to comment
I have the same issue, this started like 3 days ago