CodeInstances Now Compares Two Runs and Shows Where the Failing One Went Wrong

CodeInstances v0.2.0 is out, and it adds the third verb. The first release could keep a run and replay it. This one compares two of them: codeinstances diff passing.instance failing.instance.

The usual case goes like this. A job works on Monday and fails on Tuesday, or passes on your laptop and fails in CI. You have both runs as instance files, because you put codeinstances record in front of the command. Now you want to know what was different, and you’d rather not read two logs side by side and guess.

diff pairs up the calls the two runs made and tells you which ones came back different, which ones only one run made, how each run ended, what each printed and how long things took. It exits 0 when the runs are the same and 1 when they differ, so a script can act on it too.

How It Lines Up the Calls

Calls are paired by method, URL and request body, in order. Whatever’s left over is paired again by method and URL alone. So a request that went out with a different body shows up as one changed call, with the body difference spelled out, and not as one call missing plus one extra.

For each pair it compares the status and the response body. JSON is compared member by member, down to the first value that differs, so you get a line like response body at $.items: 12 items in a, 0 in b instead of two walls of JSON. Other text is compared by line, and binary bodies by the first byte that differs. Headers are left out, since they’re full of dates and request ids that always change.

Two separate runs read different clocks and draw different random numbers. So by default diff only checks that both runs asked for the same number of hooked values of each kind. -values compares the values themselves, for when you know they should match. A different command, a different kept environment or a different CodeInstances version gets a note, but doesn’t count as a difference. That’s on purpose: comparing a fixed build against the old one is half the point.

Speed

Every call recorded with v0.2.0 now keeps how long it took, as a new duration_ms member in the file. diff shows the total time of both runs and the calls whose time changed most, so “the failing run hit a timeout on one endpoint” is visible at a glance. Speed never counts as a difference, since timings wobble from run to run.

The new member is an addition to the open format, so files recorded by v0.1.0 still read fine, and v0.1.0 can read the new files. Old recordings just don’t have per-call times, and diff leaves them out of the speed list.

Tested Like the Rest

70 Go tests pass now, and the race detector finds nothing. The self-checking demo grew from 11 checks to 14: besides replaying its recording 10 times with the servers stopped, it records a second run of a program that makes one extra request, and checks that diff finds that request and times all 23 calls the runs share. Comparing the two runs took 7 ms. The smoke check on the release binaries records a Python program twice and checks that diff points at the changed JSON member. How it was tested explains the setup, and the result files are in test/results.

Cost isn’t in this release. Money, tokens and bytes sent are left out for now, and diff reports differences and time. Next on the plan is shrinking a failing run to the smallest one that still fails, and turning recorded runs into regression tests.

Download v0.2.0, or look at every flag on the reference page.