Thanks for reporting this issue here.
We've also received your feedback from Marketplace - I think it's the same issue.
The only change in workzone 2.4.5 was to have the Stash 'Pull request approved' event spawn a new thread in workzone to check and perform a merge operation. This is the recommended strategy by Atlassian. Before Workzone version 2.4.5 came out the auto-merge condition check was performed within the 'pull request approved' event thread.
I would expect the opposite behaviour : Before Workzone 2.4.5 pull request approved events would queue up because the auto-merge check would delay event thread execution - and now event processing would be much faster.
Alex, you can safely downgrade to Workzone 2.4.4 . In fact - before you do - could you please run your stash with Workzone 2.4.5 and Workzone debugging enabled? That would be of great help! To enable Workzone debug logging please run from the terminal
We upgraded June 5th at 2pm and virtually immediately started having performance problems. I ended up downgrading to Workzone 2.4.3 (our previous version) on June 6 because the system kept coming dangerously close to crashing, even on a weekend with virtually no user activity.
It seems like there was only just enough activity to trigger some automerges in our largest repo and each time the entire system would become unresponsive.
Immediately after reverting to Workzone 2.4.3 the performance problems stopped.
I'm unwilling to upgrade in production right now in order to capture the debug logs because the symptoms caused by 2.4.5 are quite severe for us.
I will send you the logs I have though.
Creating new threads for each auto-merge all at once flooded our system with so many simultaneous requests that the system almost crashes. Have you considered if this could be related to the known bug in Stash 3.8 - https://jira.atlassian.com/browse/STASH-7337.
thanks for the update. We totally understand that you won't be able to gather more logs with Workzone 2.4.5 due to the severity of the effects. I'm glad you were able to roll back to Workzone version 2.4.3 without complications. If you do happen to have the logs from June 5 and 6 please do send them to my personal email address for us to analyse them.
I imagine there weren't many users around on that particular weekend to approve PRs ... leaves the build results trigger. Do you have 'watch builds' enabled in Workzone? We've seen issues where CI servers send more than on build result for the same successful build to Stash. Bamboo can be (mis)configured like that and send spurious build results in quick succession for some plans. Workzone does not keep track of auto-merge attempts and thinks - 'there is a new successful build result, let's auto-merge!' even though it just triggered an auto-merge attempt milliseconds before based on an earlier build result for the same PR. This may clog up core Stash's merge service and may lead to the problem you are observing. The big question though is that nothing has changed in terms of build result handling between Workzone 2.4.3 and 2.4.5. The only change is that user approval events now start their own thread and attempt an auto-merge.
So far your problem scenario with Workzone 2.4.5 does not make much sense to me yet. However I hope the logs will shed a bit more light onto the situation.