Clone wiki

gazebo / buildcop

Build Cop

A build cop monitors the continuous integration system, reports build and test failures, and farms out work to resolve issues. The role of build cop rotates on a weekly basis between OSRF developers.

Use the buildcop_stats.bash script from release-tools to count the number of jobs of a given color:

curl https://build.osrfoundation.org/view/main/view/BuildCopFail/api/json 2>/dev/null | python -c '\
import json, sys;
jobs = json.loads(sys.stdin.read())["jobs"];
print("| Type | Count | Percent | Change |")
print("|--|--|--|--|")
print("| total | %d | |  |" % len(jobs));
for c in ["blue", "yellow", "red", "aborted"]:
    jc = [j for j in jobs if j["color"].startswith(c)]
    print("| %s | %d/%d | %.1f%% |  |" % (c, len(jc), len(jobs), 100*float(len(jc)) / len(jobs)))'

It will generate output like the following:

| Type | Count | Percent | Change |
|--|--|--|--|
| total | 115 | |  |
| blue | 72/115 | 62.6% |  |
| yellow | 34/115 | 29.6% |  |
| red | 9/115 | 7.8% |  |
| aborted | 0/115 | 0.0% |  |

that renders in markdown like the following:

Type Count Percent Change
total 115
blue 72/115 62.6%
yellow 34/115 29.6%
red 9/115 7.8%
aborted 0/115 0.0%

You can manually add entries to the change column by comparing to the entries in the previous build cop entry.

Tasks

A build cop:

  1. Actively monitors the BuildCopFail page for failures. At a minimum this should be done once per day.

  2. Restart any builds that appear to have failed for spurious reasons.

  3. Send out a daily report about the build status, and create an entry on this wiki (see the report list below).

  4. Sent out a weekly report on Friday that summarizes the build status over the past week.

Reports

2016

2017

2018

Updated