release

CodeInstances Groups Failing Runs by Fingerprint So Repeats of One Bug Show as One

CodeInstances v0.4.0 adds the sixth verb, group. Point codeinstances group at a folder of instances and it tells you how many different ways your program failed, how often each one happened, and when it was first and last seen.

The problem is a familiar one. A nightly job fails 40 times in a week. A flaky suite leaves a pile of red runs. Each failure gets its own instance now, which is good, but 40 files isn’t an answer. Usually it’s two or three bugs, repeating. Somebody has to open the files, squint at the errors and sort them into piles, and the piles drift, because the messages carry request ids, timestamps and ports that change every time.

Group does the sorting. Every failing run gets a fingerprint, and runs with the same fingerprint land in the same group, biggest first. Runs that passed are counted and left out.

Many failing runs, a few bugs. Twelve failing runs on the left. In the middle, the fingerprint: how the run ended (exit 1), its error, where “request 4411 timed out after 3002ms” becomes “request timed out after ”, and the calls that failed, such as GET /v1/items?page=* with no answer. Times, ids and numbers are blanked out. On the right, three groups: 8 runs of a timeout, 2 runs of a refused order, 2 runs of a nil map panic. Below, the command codeinstances group failures/ reports 12 failing runs in 3 groups with 40 runs passed.

What Goes Into a Fingerprint

Three things. How the run ended: exit 1, a signal, a timeout. The error. And the calls that failed, meaning the ones that got no answer or a 5xx status, by method and URL.

The error is where the work is. For a Go panic or fatal error, it’s the message and the first three functions of the failing goroutine, skipping the runtime and the testing package, so a panic inside a test points at your code and not at Go’s. For a Python exception it’s the exception and the last three frames of the traceback. For Node.js, Java and C#, it’s the error and the first three at frames, without line numbers. For a failing go test run it’s the names of the tests that failed. Anything else falls back to the last line that looks like an error, skipping the trailers that every run prints, such as exit status 1 or a bare FAIL.

Then the details that change from one repeat to the next are blanked out: times, durations, UUIDs, hex ids, long mixed ids, numbers, ports and query values. “request 4411 timed out after 3002ms” and “request 9120 timed out after 2987ms” both become request <n> timed out after <duration>. Status codes after “status”, “code” or “error:” are kept, since a 404 and a 503 are different bugs. Colour codes are stripped too, so a coloured run and a plain one still match.

The result is hashed into an id like fp1:6d1f0a93be27c4e1. The fp1 is the version of these rules, so if the rules ever change, old and new ids can’t be confused.

Where It Can Be Wrong

A fingerprint is a heuristic, and the README says so. Two bugs that end the same way with the same error line share a group. One bug whose message carries a word that changes, such as a file name, splits into several. Calls are told apart by method and URL, not body, so every failing call to one GraphQL endpoint looks the same.

Tested

82 Go tests pass, and the race detector finds nothing. The fingerprint tests use real output from Go panics, fatal errors and test runs, Python tracebacks, Node.js and Java crashes. The self-checking demo has 20 checks now. It records two more failing runs, the same broken item again and then a second bug where the service refuses orders, and checks that group puts the three failing runs in 2 groups and leaves the passing run out. That took 10 ms. The smoke check on the release binaries crashes a Python script twice with different messages and checks both land in one group. An independent review before release found that every failing go test run and every Node.js crash shared one fingerprint, and that test servers on random ports split groups. All of that is fixed, and covered by tests now. The result files are in test/results.

Next on the plan is sharing and storing instances. Download v0.4.0, and the reference has the details.