1. Julius Davies [bit-booster.com] Avatar Julius Davies [bit-booster.com]
  2. Bit-Booster
  3. PR-Booster For Bitbucket
  4. Issues

Clean rebase requires a new build (optimistic build feature not working)

Issue #244 resolved
Marko Hyvonen created an issue

Hi,

We hope you’re doing good.

We have a show stopper issue and we’d need urgent debugging help.

Our system info:

SEN-32196408
PR-booster version: 2022.07.05
Bitbucket data center version: v7.18.1
Bamboo version: 8.0.4 build 80012 - 27 Oct 21

Technical contact (purchase / license): cvanderwist@tandemdiabetes.com)

We use Bitbucket for our GIT repo.
We have a mono repo of 60 users, we rebase rather than merge.

On "Merge strategies", we have:
Rebase and fast-forward rebase + merge --ff-only
Squash, fast-forward only --squash --ff-only

We purchased and installed PR-booster.
We enabled “Optimistic Build Status Propagation”.
We also enabled:
Automatically Withdraw Approvals (yes - For significant pushed)
Automatically Preserve Disapprovals (yes - For significant pushes)
(We can see these settings in our Bitbucket repo settings.)

We also enabled Bitbucket built-in settings:
Merge checks > Minimum successful builds = 1.
We set "Required builds" to our verification Bamboo test plan.

Expected behavior:
underlying Green build on feature branch > clean rebase, no new commits (identical patch-id) > should not require new Green build

Observed behavior:
underlying Green build on feature branch > clean rebase, no new commits > new Green build is always required

Verified by:

We have verified at terminal, patch-id doesn't change before and after rebase to mainline.

Additional info:
We had trial version 2022.03.1 (that had the internal refresh clone option) installed and updated to 2022.07.05.
Trial version 2022.03.1 worked OK on clean rebase but we also had older Bitbucket / Bamboo version at the time

Workaround(very bad, not acceptable):
If we remove Bitbucket "Required builds", a clean rebase won't require new build

Comments (41)

  1. Julius Davies [bit-booster.com] repo owner

    We’ll try and fix this within the next 48 hours.

    Thank you very much for sending in the bug report.

    Since you are on Bitbucket v7.18.1 you should be able to downgrade back down to PR-Booster v2022.03.01 while we fix this on our side. (Bitbucket 8.x, on the other hand, only works with the latest PR-Booster versions).

    ‌

    ‌

    ‌

  2. Marko Hyvonen reporter

    Thank you Julius for prompt answer!

    Unfortunately v2022.03.01 is not going to work, we tried it half a day today and finally upgraded to 2022.07.05.

    The last time “Optimistic build” worked was ~8 months ago, even then it’s hard to recall whether it really worked.

    (We might not have had "Required builds" Enabled at the time. We must have it enabled.)

    Is there any logging that can be enabled? We only have few days to get this sorted out, after that we need to Disable the plugin.

    In the mean time, I’ll try to attach support zips that our Atlassian technical expert Chuck Vanderwist (cvanderwist@tandemdiabetes.com) collected.

    So far they have been timed out on upload but I’ll try to break them down to smaller chunks.

    Thanks, Marko

  7. Julius Davies [bit-booster.com] repo owner

    ‌

    I’m sure we’ll figure this one out quickly. I just confirmed that the behaviour DOES work correctly in v2021.08.11, and that it is NOT working in v2022.07.05.

    ‌

  8. Marko Hyvonen reporter

    Excellent.

    I’m having major issues uploading “application-logs” support .zip file, which is 1.1GB in size.

    I’ll try to zip and upload logs just for today.

    ‌

    ‌

  13. Marko Hyvonen reporter

    Let me know whether those logs are any good and whether there’s any other logging we can enable and send to you.

  14. Marko Hyvonen reporter

    Let me know if downgrading to v2021.08.11 is an option for us.

    We only really need “Optimistic build feature”, to prevent “rebase fights”.

  15. Julius Davies [bit-booster.com] repo owner

    Can you try version v2022.07.06-bb7 (which we just published). Changelog:

    ​Build-status propagation broke in our Bitbucket 8.x logic.  Fixing it for Bitbucket 7.x.

    (To fix it for Bitbucket 8.x will require help from Atlassian, because it seems they have made a serious mistake with RepositoryBuildStatusSetEvent).

    ‌

  16. Julius Davies [bit-booster.com] repo owner

    We’ve also created a support ticket with Atlassian to alert them that Bitbucket 8.x will break our feature.

    Title: Bitbucket Server 8.x no longer raises RepositoryBuildStatusSetEvent events

    URL: https://support.atlassian.com/requests/CA-1880018/ (only we can see this ticket, though)

    Customer Description

    Bitbucket 7.x correctly raises BuildStatusSetEvent when a build-status is set against a commit.

    But Bitbucket 8.x removed this event, and replaced it with RepositoryBuildStatusSetEvent.

    But it doesn't ever raise this event! Ugh!

    To see for yourself:

    Create a plugin with this code:

    @EventListener
    public void onBuildEvent(RepositoryBuildStatusSetEvent event) {
    final String commitId = event.getBuildStatus().getCommitId();
    Repository r = event.getBuildStatus().getRepository();
    System.out.println("hello!");
    }

    And then run something like this command:

    curl --user admin:admin -H "Content-Type: application/json" -X POST -d '{"state":"SUCCESSFUL", "key":"K", "name":"K-9", "url":"http://bit-booster.com/"}' http://localhost:7990/bitbucket/rest/build-status/1.0/commits/d6edcbf924697ab811a867421dab60d954ccad99

    And notice the code above is never executed.

  17. Julius Davies [bit-booster.com] repo owner

    We believe this is fixed in PR-Booster v2022.07.06-bb7 (for Bitbucket 7.x).

    For Bitbucket 8.x the bug appears to be currently unfixable (due to a regression in Bitbucket itself that we cannot work around).

  18. Marko Hyvonen reporter

    Hi Julius,

    Appreciated the quick response and debugging!
    We'll try "v2022.07.06-bb7" immediately, I'll let you know the result as soon as we have it.

    Thank you for raising ticket against Atlassian.
    Please add comments from our company that this absolutely has to be fixed by Atlassian.
    Our internal security team won't allow us to stay on 7.x Bitbucket for long.

    Also we may need to upgrade to 8.x due to bugs …

    We must start gating Green builds and we can't drop rebasing workflow.
    So Bitbucket "8.x" absolutely must be able to raise BuildStatusSetEvent.

    cheers,
    Marko

  19. Marko Hyvonen reporter

    AFTER updating PR-booster to "v2022.07.06-bb7”, do we need to have another Successful build on every PR?

    Or can the system re-use the Successful build from BEFORE updating PR-booster to "v2022.07.06-bb7?

  20. Julius Musseau

    Yes, I think that should be fine.

    Background reading (for context): https://mergebase.com/doing-git-wrong/2018/06/04/preventing-git-rebase-fights/

    Here’s how the logic works:

    1.) when a rebase happens, PR-Booster takes note of all successful builds on the previous branch tip commit.

    2.) PR-Booster stores these successful build-ids in the rebased commit inside a special “SUCCESS” build that PR-Booster itself generates: “PR-Booster Build Status Propagation”. (This part of the logic never broke).

    3.) Should any subsequent “INPROGRESS” build notifications be received, PR-Booster will intercept them, check if they match previous SUCCESS builds from before the rebase, and temporarily replace them with “SUCCESS”. (This is the part of the logic that was broken in our 2022.06.* and 2022.07.* versions until now - because we had switched to the new RepositoryBuildStatusSetEvent event that Bitbucket itself never bothers to fire).

    4.) Eventually Bamboo will send the official “SUCCESS” or “FAILURE” notification. (Only the “INPROGRESS” notification is intercepted).

    ‌

    As long as you see a ““PR-Booster Build Status Propagation” SUCCESS build on the commit, then everything should work now with v2022.07.06-bb7.

    ‌

    ‌

    ‌

  21. Marko Hyvonen reporter

    PR-booster 'v2022.07.06-bb7' has been now installed.

    These are the current settings:

    • PR booster:
      Optimistic Build Status Propagation = YES
      Automatically Withdraw Approvals "Yes - For significant pushes"
      Automatically Preserve Disapprovals "Yes - For significant pushes"

    ‌

    • Merge checks:
      "Minimum successful builds" set to 1.

    • Required-builds
      Our Verification tests plan only

    We have these issues:
    -Failing build on a Unrelated Plan will prevent merging, even if Verification tests have passed
    -Clean rebase always requires a new build (verified by terminal patch-id checks)

    This is what we want to achieve:
    -We want to gate by 1 successful build on our Verification tests plan ONLY
    -We want to ignore all other plan builds

    The merge-checks have this crazy sounding line:
    "If there are more builds than specified above, they are all required to be successful in order to merge the pull request."

    How is this ever going to work?

    ‌

    We wonder if Bamboo required-builds merge check just forces a Verification tests build no matter what on the latest commit regardless of the PR booster plugin?

    Or if the PR booster plugin doesn't account for this feature?

  22. Marko Hyvonen reporter

    Update:

    I just verified that Clean rebase (no conflict files) via Bitbucket GUI also requires a new Green build (not just via Terminal), Even if PR had a previous Green build from few minutes ago.

    Bitbucket PR Info note is:

    “Successful build of EM-VT for the latest commit is required before this pull request can be merged.”

  24. Julius Musseau

    I could do that tomorrow.

    How about disabling “Merge checks > Minimum successful builds” since the “Required-builds: Verification” already mandates at least 1 successful build anyway!

    ‌

  25. Marko Hyvonen reporter

    Exactly, good point. I just Disabled “Merge checks > Minimum successful builds”.

    That helps that much that other Unrelated plan results won’t be counted towards Merge check.

    ‌

    PR-booster still doesn’t relax build requirement on Clean rebases.

    Only after removing “Required-builds:”, we don’t need to rebuild Verification tests again before Merge to mainline is possible.

    What can we do?

    Can we enable some logging?

    ‌

  26. Julius Musseau

    I think I figured it out! New release of PR-Booster will be published within the next 30 minutes!

  28. Julius Musseau

    v2022.07.07 published - should fix it

    Also looks like this will work fine with Bitbucket 8.x. Turns out the way I was setting build-status for testing purposes does not quite accurately mimic the way Bamboo sets build-status.

    For your specific scenario it’s important you leave “Minimum successful builds” disabled.

    ‌

    (Something about the way I set build-status in my own testing definitely doesn’t work with Bitbucket 8.x, though, so there still might be a bug there for people NOT using Bamboo.)

    ‌

  30. Marko Hyvonen reporter

    Unfortunately still doesn’t work.

    We installed new PR-booster version:

    These are the current settings for our repo:
    merge-checks > Minimum successful builds = Disabled
    required-builds > our Verification tests Bamboo plan (EM-VT)

    PR-booster >
    Optimistic Build Status Propagation = Yes
    Automatically Withdraw Approvals = Yes - For significant pushes
    Automatically Preserve Disapprovals = Yes - For significant pushes

    I tested with 3 Open PRs that were behind mainline and did NOT have conflict files.

    Below is one of them:

    BEFORE REBASE at Terminal:
    \$ git diff origin/release/XXXXXXXX...origin/feature/chuang-ut-tandem-nrf-watchdog-increase-coverage | git patch-id --stable
    3d3ff9378cd6759819d1bb41824cd566c9b09a70 0000000000000000000000000000000000000000

    I rebased PR via Bitbucket GUI built-in (not PR-booster) "Rebase" button.

    AFTER REBASE at Terminal:
    \$ git fetch --prune
    Unpacking objects: 100% (7/7), 871 bytes | 34.00 KiB/s, done.
    From XXXXXXXXX

    • 35c60f3955...58e7c56e2d feature/chuang-ut-tandem-nrf-watchdog-increase-coverage -> origin/feature/chuang-ut-tandem-nrf-watchdog-increase-coverage (forced update)

    \$ git diff origin/release/XXXXXXXXXX...origin/feature/chuang-ut-tandem-nrf-watchdog-increase-coverage | git patch-id --stable
    3d3ff9378cd6759819d1bb41824cd566c9b09a70 0000000000000000000000000000000000000000

    Patch-id is identical.

    After Rebase Bitbucket GUI shows:
    Here are 2 issues preventing you from merging this pull request.
    -Successful build of EM-VT for the latest commit is required before this pull request can be merged.
    -You still need 1 approval before this pull request can be merged.

  31. Marko Hyvonen reporter

    I’m open any time tomorrow whole day (6AM-6PM) for the Screen sharing.

    We can use MSFT Teams, Skype, Zoom or anything else.

  33. Julius Musseau

    Okay, well now you’ve gone and caused me to actually setup Bitbucket 7.18.1 and Bamboo 8.0.4 and connect them to each other just to be absolutely certain !!! 😉 (Until now I happened to have a copy of Bamboo 8.2.2 kicking around, so I was using that, but now it’s time to bring out the microscopes!)

    Did the 3 open PR’s have previously successful “EM-VT” builds associated with their tip commits?

    To further debug this I would like the JSON response from the following URL (adjust it to suit your Bitbucket):

    http://localhost:7990/rest/build-status/latest/commits/0a943a29376f2336b78312d99e65da17048951db

    (Might start with “/bitbucket/rest/” instead of just “/rest/”).

    Change the commit-id (e.g., “0a943a…”) to the tip commit on the PR you are rebasing. I need the JSON for 3 situations:

    1. Before the rebase.
    2. Immediately after the rebase (a different commit-id, while the Bamboo build is in-progress).
    3. After the bamboo build completes (same commit-id as #2)

    ‌

    Should look something like this:

    ‌

    {
  "size": 2,
  "limit": 25,
  "isLastPage": true,
  "values": [
    {
      "state": "SUCCESSFUL",
      "key": "PROJ-PLN0",
      "name": "PROJECT - PLAN - basic_branching",
      "url": "http://127.0.1.1:8085/browse/PROJ-PLN0-2",
      "description": "#2 successful\nin 1 minute",
      "dateAdded": 1657233518658
    },
    {
      "state": "SUCCESSFUL",
      "key": "bit-booster.com-[PROJ-PLN0]",
      "name": "PR-Booster Build Status Propagation",
      "url": "http://127.0.1.1:8085/browse/PROJ-PLN0-1",
      "description": "PR-Booster Build Status Propagation (via git patch-id) from d6edcbf924697ab811a8 of jobs: [PROJ-PLN0]",
      "dateAdded": 1657233458119
    }
  ],
  "start": 0
}

    ‌

  34. Julius Musseau

    And here’s what it should like during the “INPROGRESS” phase when PR-Booster has intercepted the call and optimistically replaced it with “SUCCESS”: 

    {
  "size": 2,
  "limit": 25,
  "isLastPage": true,
  "values": [
    {
      "state": "SUCCESSFUL",
      "key": "PROJ-PLN0",
      "name": "PROJECT - PLAN - basic_branching",
      "url": "http://127.0.1.1:8085/browse/PROJ-PLN0-3",
      "description": "#3 successful via build-status-propagation",
      "dateAdded": 1657233677934
    },
    {
      "state": "SUCCESSFUL",
      "key": "bit-booster.com-[PROJ-PLN0]",
      "name": "PR-Booster Build Status Propagation",
      "url": "http://127.0.1.1:8085/browse/PROJ-PLN0-2",
      "description": "PR-Booster Build Status Propagation (via git patch-id) from 00dc613eb044536e5e8b of jobs: [PROJ-PLN0]",
      "dateAdded": 1657233677677
    }
  ],
  "start": 0
}

    ‌

  36. Marko Hyvonen reporter

    Did the 3 open PR’s have previously successful “EM-VT” builds associated with their tip commits?
    Yes, all 3 PRs had Green builds on tip commits before Rebase.

    ‌

  37. Julius Musseau

    p.s. notice the very subtle different in the JSON:

    INPROGRESS: "description": "#3 successful via build-status-propagation",

    OFFICIAL BAMBOO SUCCESS: "description": "#2 successful\nin 1 minute",

    So this is a good way to see what’s really going on behind the scenes.

    ‌

  38. Marko Hyvonen reporter

    ok, looks promising, I’ll get those 3 JSONS tonight.

    curl -u ${USERNAME}:${PASSWD} -X GET -H "Content-Type: application/json" "https://bitbucket.XXXXXXXXXX/rest/build-status/latest/commits/$TIP_BEFORE_REBASE" | jq '.'
    {
    "size": 1,
    "limit": 25,
    "isLastPage": true,
    "values": [
    {
    "state": "SUCCESSFUL",
    "key": "EM-VT2288",
    "name": "Pump Software - Verification Tests - feature-chuang-ut-arm-param-mgmt-increase-coverage",
    "url": "https://bamboo.XXXXXXXXXXXXX.com/browse/EM-VT2288-4",
    "description": "#4 successful\nin 48 minutes",
    "dateAdded": 1657143396787
    }
    ],
    "start": 0
    }

  39. Marko Hyvonen reporter

    Alright, I did this whole process twice with identical curl responses.

    Create test branch 3 commits down from tip of origin/master

    \$ git fetch
    \$ git rev-list -3 origin/master
    b1d43c8915ac98a56b431f856f545bd4fecbc365
    f0dadba5eed571aaa1d551792ca668287d3b5348
    5fd451052f8244901c1bf620ba7a89a4dc0950ae

    \$ git checkout -b test/PR-booster-build-status-test-2 5fd451052f8244901c1bf620ba7a89a4dc0950ae
    Switched to a new branch 'test/PR-booster-build-status-test-2'

    \$ git push origin test/PR-booster-build-status-test

    \$ git rev-list ^origin/test/PR-booster-build-status-test-2 origin/master
    b1d43c8915ac98a56b431f856f545bd4fecbc365
    f0dadba5eed571aaa1d551792ca668287d3b5348

    create a new file, commit, push

    \$ echo "test" > test.txt ; git add test.txt ; git commit -m "pr-booster test file" ; git push origin test/PR-booster-build-status-test-2

    \$ git log -1 --name-status origin/test/PR-booster-build-status-test-2
    commit 0a5b43e3394c5e40d92cf83cb517dba9904ca08a (HEAD -> test/PR-booster-build-status-test-2, origin/test/PR-booster-build-status-test-2)
    Author: Mr. nobody
    Date: Thu Jul 7 17:25:47 2022 -0700

    pr-booster test file

A       test.txt

    Open PR via Bitbucket

    Note at Bitbucket: "Successful build of EM-VT for the latest commit is required before this pull request can be merged."

    started initial build on the branch

    build-status while initial build in progress:

    curl -u ${USERNAME}:${PASSWD} -X GET -H "Content-Type: application/json" "https://bitbucket.COMPANY.com/rest/build-status/latest/commits/0a5b43e3394c5e40d92cf83cb517dba9904ca08a" | jq '.'

    {
    "size": 1,
    "limit": 25,
    "isLastPage": true,
    "values": [
    {
    "state": "INPROGRESS",
    "key": "EM-VT2335",
    "name": "Pump Software - Verification Tests - test-PR-booster-build-status-test-2",
    "url": "https://bamboo.COMPANY.com/browse/EM-VT2335-2",
    "description": "#2 in progress",
    "dateAdded": 1657240104033
    }
    ],
    "start": 0
    }

    build-status after initial build, BEFORE rebase:

    curl -u ${USERNAME}:${PASSWD} -X GET -H "Content-Type: application/json" "https://bitbucket.COMPANY.com/rest/build-status/latest/commits/0a5b43e3394c5e40d92cf83cb517dba9904ca08a" | jq '.'

    {
    "size": 1,
    "limit": 25,
    "isLastPage": true,
    "values": [
    {
    "state": "SUCCESSFUL",
    "key": "EM-VT2335",
    "name": "Pump Software - Verification Tests - test-PR-booster-build-status-test-2",
    "url": "https://bamboo.COMPANY.com/browse/EM-VT2335-2",
    "description": "#2 successful\nin 30 minutes",
    "dateAdded": 1657241933800
    }
    ],
    "start": 0
    }

    At Bitbucket, PR has Green build icon: YES

    rebase to tip of origin/master:

    \$ git fetch
    \$ git rev-list ^HEAD origin/master
    b1d43c8915ac98a56b431f856f545bd4fecbc365
    f0dadba5eed571aaa1d551792ca668287d3b5348
    \$ git rebase origin/master
    Successfully rebased and updated refs/heads/test/PR-booster-build-status-test-2.
    \$ git push --force-with-lease origin test/PR-booster-build-status-test-2

    DOES PR ASK FOR ANOTHER BUILD: YES, note at PR: "Successful build of EM-VT for the latest commit is required before this pull request can be merged."

    rebased tip commit SHA

    \$ git rev-list -1 origin/test/PR-booster-build-status-test-2
    52e955e67dc2ef0c7b00d474416489d008cbddcb

    build-status just AFTER rebase, using rebased tip SHA

    curl -u ${USERNAME}:${PASSWD} -X GET -H "Content-Type: application/json" "https://bitbucket.COMPANY.com/rest/build-status/latest/commits/52e955e67dc2ef0c7b00d474416489d008cbddcb" | jq '.'
    {
    "size": 1,
    "limit": 25,
    "isLastPage": true,
    "values": [
    {
    "state": "SUCCESSFUL",
    "key": "bit-booster.com-[EM-VT2335]",
    "name": "PR-Booster Build Status Propagation",
    "url": "https://bamboo.COMPANY.com/browse/EM-VT2335-2",
    "description": "PR-Booster Build Status Propagation (via git patch-id) from 0a5b43e3394c5e40d92c of jobs: [EM-VT2335]",
    "dateAdded": 1657242127630
    }
    ],
    "start": 0
    }

    start Bamboo build on Clean rebase content

    build-status while 2nd Bamboo is building, using rebased tip SHA

    curl -u ${USERNAME}:${PASSWD} -X GET -H "Content-Type: application/json" "https://bitbucket.COMPANY.com/rest/build-status/latest/commits/52e955e67dc2ef0c7b00d474416489d008cbddcb" | jq '.'

    {
    "size": 2,
    "limit": 25,
    "isLastPage": true,
    "values": [
    {
    "state": "SUCCESSFUL",
    "key": "EM-VT2335",
    "name": "Pump Software - Verification Tests - test-PR-booster-build-status-test-2",
    "url": "https://bamboo.COMPANY.com/browse/EM-VT2335-3",
    "description": "#3 successful via build-status-propagation",
    "dateAdded": 1657242287330
    },
    {
    "state": "SUCCESSFUL",
    "key": "bit-booster.com-[EM-VT2335]",
    "name": "PR-Booster Build Status Propagation",
    "url": "https://bamboo.COMPANY.com/browse/EM-VT2335-2",
    "description": "PR-Booster Build Status Propagation (via git patch-id) from 0a5b43e3394c5e40d92c of jobs: [EM-VT2335]",
    "dateAdded": 1657242127630
    }
    ],
    "start": 0
    }

    build-status after Bamboo 2nd build finished, using rebased tip SHA

    $ curl -u ${USERNAME}:${PASSWD} -X GET -H "Content-Type: application/json" "https://bitbucket.COMPANY.com/rest/build-status/latest/commits/52e955e67dc2ef0c7b00d474416489d008cbddcb" | jq '.'

    {
    "size": 2,
    "limit": 25,
    "isLastPage": true,
    "values": [
    {
    "state": "SUCCESSFUL",
    "key": "EM-VT2335",
    "name": "Pump Software - Verification Tests - test-PR-booster-build-status-test-2",
    "url": "https://bamboo.COMPANY.com/browse/EM-VT2335-3",
    "description": "#3 successful\nin 18 minutes",
    "dateAdded": 1657243397837
    },
    {
    "state": "SUCCESSFUL",
    "key": "bit-booster.com-[EM-VT2335]",
    "name": "PR-Booster Build Status Propagation",
    "url": "https://bamboo.COMPANY.com/browse/EM-VT2335-2",
    "description": "PR-Booster Build Status Propagation (via git patch-id) from 0a5b43e3394c5e40d92c of jobs: [EM-VT2335]",
    "dateAdded": 1657242127630
    }
    ],
    "start": 0
    }

    Bitbucket GUI: PR doesn't have a note about another build required in order to be Merged.

    ‌

  42. Log in to comment
Assignee
–
Type
bug
Priority
blocker
Status
resolved
Votes
1
Watchers
1
Jira: the preferred issue tracker for Bitbucket. Join the team!