Needs Work is no propagating to new commits

[Steven Whitman]

Recently I’ve noticed that the “Needs Works” status is not propagating when new commits are pushed to an open PR. I have one from yesterday. The overview page for the PR shows that I marked the PR with “Needs Work” yesterday. Today the developer pushed a new commit for the open PR but my needs work status is no longer present. This appears to be happening for some but not all PRs. I haven’t been able to determine if there is a pattern.

I reviewed the audit log and I see where I marked the PR as Needs Work. The previous status was Unapproved. There is no other activity that I can see changing the status back to Unapproved. I then changed the status to Needs Work again and now the audit log is showing the state changed from Unapproved to Needs Work.

During this time, two other developers approved the PR.

[Julius Davies [bit-booster.com]]

Hi !

Are you able to share what version of PR-Booster you are using as well as what version of Bitbucket Data Center ?

‌

I tried to reproduce this issue using PR-Booster v2023.09.30 together with Bitbucket Data Center v8.15.1 (e.g., latest versions of both) and the feature appears to be working correctly.

Can you confirm that PR-Booster’s “Automatically Preserve Disapprovals” is set to “Yes - For All Pushes” for the repository where you are seeing this problem?

‌

[Steven Whitman]

I verified that the setting is correctly set as you can see in this screenshot.

‌

We are running Atlassian Bitbucket v7.18.2 and the plugin is at version 2023.09.01

‌

[Julius Davies [bit-booster.com]]

We're working on this. Resolving this is a high priority. Note: duplicate issue reported by another client: #528

So far from my analysis I can report that PR-Booster has not seen any changes related to this functionality in exactly 1 year, so I suspect some interplay with newer versions of Bitbucket might be the problem.

The change from 1 year ago was also very small, and was about dropping approvals - it was not about this specific feature (preserving disapprovals).

For reference here's the (somewhat related) enhancement from 1 year ago:

---

Automatically Withdraw Approvals For Service Accounts [x] Yes No (Defaults to [x] Yes)

If the "Automatically Withdraw Approvals" feature is enabled, this controls whether the logic also applies to service account pushes (e.g., accounts backed by ssh keys only, and not actual users).

---

[Julius Davies [bit-booster.com]]

Issue #528 was marked as a duplicate of this issue.

[Steven Whitman]

We had an issue in the past with another pluggin that interfered with ScriptRunner. While they were both updating different items in the PR, depending on the order in that they ran in it would sometimes not work. I believe the other plugin was Workzones. I looked at the audit log and we updated Workzones, ScriptRunner and PR-Booster on Sept 18th. I don’t know exactly when this problem started just that it was sometime prior to Nov 14th and it took at least a few weeks before it was reported. We are running Workzone for Bitbucket 7.9.4, ScriptRunner for Bitbucket 8.11.0 and PR-Booster 2023.09.01.

[Julius Davies [bit-booster.com]]

Are you running a Bitbucket cluster? If yes, how many nodes? It shouldn’t be a factor - Bitbucket’s event architecture is designed so that events will only ever be raised on the node that triggered the event in the first place. But this information does help me in my own mental modelling of the possible scenarios.

Also, do you see any mentions of “EVENT-2001 - A slow event listener was detected” in any of your Bitbucket logs? More info here: https://confluence.atlassian.com/bitbucketserver/event-system-947174309.html

The auto-drop-approvals feature of PR-Booster is based solely on the PullRequestRescopedEvent

@EventListener
public void onRescope(PullRequestRescopedEvent event) {

‌

[Steven Whitman]

We aren’t using a BitBucker cluster. In the logs I’m seeing this:

1700608683959 ; WARNING ; EVENT ; EVENT-2001 ; Slow event listener detected ; com.atlassian.analytics.analytics-client ; 6.2.1 ; com.atlassian.analytics.client.listener.BitbucketEventListener ; {"timeMillis":15034,"eventType":"com.atlassian.config.lifecycle.events.ApplicationStoppingEvent"

In the atlassian-bitbucket-alerts-2023-11-20.log I see this:
1700496769978 ; WARNING ; EVENT ; EVENT-2001 ; Slow event listener detected ; System ; 7.18.2 ; com.atlassian.stash.internal.audit.RepositoryEventListener ; {"timeMillis":21480,"eventType":"com.atlassian.bitbucket.event.repository.RepositoryOtherReadEvent"}1700496769978 ; WARNING ; EVENT ; EVENT-2001 ; Slow event listener detected ; com.atlassian.analytics.analytics-client ; 6.2.1 ; com.atlassian.analytics.client.listener.BitbucketEventListener ; {"timeMillis":21480,"eventType":"com.atlassian.bitbucket.event.repository.RepositoryOtherReadEvent"}1700496769974 ; WARNING ; EVENT ; EVENT-2001 ; Slow event listener detected ; System ; 7.18.2 ; com.atlassian.stash.internal.audit.RepositoryEventListener ; {"timeMillis":21488,"eventType":"com.atlassian.bitbucket.event.repository.RepositoryOtherReadEvent"}1700496769974 ; WARNING ; EVENT ; EVENT-2001 ; Slow event listener detected ; com.atlassian.analytics.analytics-client ; 6.2.1 ; com.atlassian.analytics.client.listener.BitbucketEventListener ; {"timeMillis":21489,"eventType":"com.atlassian.bitbucket.event.repository.RepositoryOtherReadEvent"}

‌

[Julius Davies [bit-booster.com]]

Thanks! That’s very interesting! How about any occurrences of EVENT-1001 (sorry I should have mentioned that as well in previous)?

EVENT-1001 - An event was dropped because the event queue was full

‌

[Steven Whitman]

What I showed above is all the events that I saw between 2023-11-20 and yesterday, 2023-12-7. I can look back further if you want.

[Steven Whitman]

I looked back a bit further. For the log generated on 2023-11-12 there were 27 missed events. It looks like our system might have been overtaxed that day. The log from 2023-11-14 showed 3 missed events. The 2023-11-12 log is the oldest log so that’s all the events we’ve seen.

[Julius Davies [bit-booster.com]]

Can you upgrade to v2023.12.13 and enable the debugging/tracing for the repositories where this is happening? I have added special debugging/tracing logic just for this problem, and you can enable it using <repo> → settings → PR-Booster

‌

[Steven Whitman]

I see that version 2023.12.15 is now out. I assume I can upgrade to that version to get the same debugging/tracing.

[Julius Davies [bit-booster.com]]

I plan to leave the debugging/tracing in place until this is solved, so yes, any version from v2023.12.13 or after includes the tracing. It’s “opt-in” tracing (per-repository). By default no data is collected, and after it’s enabled only 40 events max (per repository) are retained.

[Steven Whitman]

Ok, we’ve updated to the latest. When the issue occurs again, I’ll get the logs and send them to you. Please note that next week we have a company shutdown so it is possible it may take several weeks to observe an issue.

[Steven Whitman]

I just wanted to provide an update.
We haven’t seen any lost “needs work” issues.
I looked in our logs and only see an occasional EVENT-2001 but I’m not seeing any EVENT-1001 alerts.
I also spoke with our IT group that maintains the server and they did report that the system was being over-taxed due to resource issues on the server back in November, but they have addressed that issue.
In the meantime, I’ll keep an eye on this.

[Steven Whitman]

We experienced one occurance of the lost needs work (now changes requested) earlier this week. What information do you want?

[Steven Whitman]

We had another occurrence of this issue. For this and the previous occurrence I reported there were no EVENT-1001 or EVENT-2001 alerts in our logs.

[Julius Davies [bit-booster.com]]

What version of Control Freak are you on (that is, if you happen to use our Control Freak app).

(I wonder if the latest version of Control Freak could be possibly interfering with PR-Booster.)

Oh, and as usual: Bitbucket version and PR-Booster version please!

[Steven Whitman]

We are using Control Freak, version 2024.04.01.

The version of Bitbucket is 8.19.0.

The version of PR-Booster is 2023.12.27.

BTW, I’ll be out of the office for the next week so I may be very slow to respond or not be able to respond until I return.