testing

How CodeInstances Was Tested Before Its First Release

A tool that claims to replay runs exactly has to be tested on exactly that. So most of the testing for v0.1.0 does the same thing over and over in different ways: record a program against real servers, shut the servers down, replay it, and check the replay came out the same. Then change something and check the replay notices.

Every number in this post comes from the result files in the code repository, in test/results. They were recorded on October 18, 2026, at commit c2a5975, on a Linux machine with an Intel Xeon processor at 2.10GHz and 2 CPUs, with Go 1.24.7. One script, scripts/record-tests.sh, runs everything and writes the files, so anyone can run it again.

The Self-Checking Demo

Every download includes codeinstances-demo. It starts two local servers, one HTTPS and one plain HTTP, whose answers change on every request, so a replay that quietly went to the network would give itself away. It records a Go program that makes 23 calls to them, 20 of those from parallel goroutines, and takes 5 clock and random values through the hook package. Then it stops both servers and replays the run 10 times.

The self-checking demo as recorded for v0.1.0, on Linux with an Intel Xeon at 2.10GHz, 2 CPUs and Go 1.24.7. The recording with live servers took 46 ms. The 10 replays with the servers stopped took 30, 26, 29, 36, 45, 35, 27, 45, 27 and 28 ms. All 10 replays matched, 23 HTTP and HTTPS calls were kept, 5 clock and random values, 20,061 bytes in the instance file, and 11 of 11 checks passed.

All 10 replays matched. Recording took 46 ms with the live servers, and each replay took between 26 and 45 ms with the servers gone. The whole run fit in an instance file of 20,061 bytes.

The demo also tries to fool itself. It replays a changed program that makes one extra request, and the replay reports that the request isn’t in the recording. It replays a program that prints the real clock instead of the hooked one, and the replay reports that the first line of output differs. It edits the saved file and checks that verify catches it. It checks that the Set-Cookie header was redacted. In all it makes 11 checks, and all 11 passed.

The Go Tests

An independent review before the release made the file format stricter. The id is now computed by hand-written canonical rules with a published test vector, and files with repeated or misspelled members are refused.

59 Go tests pass across the packages, and a second full pass with Go’s race detector found nothing. They cover:

  • The format. Files round-trip through save and load. The id stays the same when only layout changes, and catches any edit to a value. The id matches the published test vector. Files with repeated or misspelled members are refused. Saves are atomic. The documented example file is checked against the format.
  • The proxy. HTTPS and HTTP recorded and replayed with the servers shut down. 64 concurrent requests recorded and replayed three times. Strict order when asked for. Bodies of 4 MiB up and 8 MiB down. A body over the limit fails that one exchange cleanly. Secret headers redacted unless kept. An upstream error replayed as the same 502.
  • Changed programs. An extra request, a missing request, an extra random value and an unhooked clock are each caught on replay.
  • Hard endings. Crashes, a program killed with SIGKILL and a program that hit its timeout are all recorded and replayed. A recorder killed with SIGKILL in mid-run leaves no partial file behind. Ctrl-C reaches the program while the recorder stays up to finish the file.

The Smoke Check on Release Binaries

Tests run against source. People run binaries. So the release binaries are built the same way the release workflow builds them, and a smoke check runs on them: the version is right, the demo passes all its checks, and a plain Python program that has never heard of CodeInstances is recorded against a live local server, the server is stopped, and the run is replayed offline. Then show and verify are run on the file, and an edited copy has to fail verify.

That check passed on the recorded machine. The release workflow runs it on each system’s own binaries, on Linux, macOS and Windows machines, before anything is published, and a failure stops the release. The Intel macOS build is the one exception, checked for its architecture only. On every push, the test suite runs on Linux and macOS, with the race detector on Linux, and the Windows binaries get the same smoke check.

What the Tests Don’t Show

They show that what v0.1.0 claims to do, it does, on the cases above. They don’t make it cover more than it covers. Protocols other than HTTP aren’t recorded, streamed responses are buffered, and clock capture is Go only. The reference lists every limit. A run of the demo on your own machine takes a few seconds and will give you your own numbers.