Quick Start
This takes a couple of minutes. You need the codeinstances binary from the download page and any program that talks to the network. The examples use a Python script, but the language doesn’t matter.
1. Record a Run
Put codeinstances record and -- in front of the command you’d normally type:
codeinstances record -o weather.instance -- python3 fetch_weather.py
The program runs as usual and its output passes through. When it exits, the run is kept in weather.instance: the command, what it printed, how it ended, and every HTTP and HTTPS request it made with the answer it got. record exits with the program’s own exit code, so it drops into scripts and CI without changing anything around it.
Leave out -o and the file is called run.instance.
2. Look at It
codeinstances show weather.instance
show prints what ran, how it ended and one line per call, with the time it happened, the method, the URL, the status and the size of the answer. show -json prints the whole file. It’s plain JSON, so any tool can read it.
3. Replay It
codeinstances replay weather.instance
The program runs again, and this time every request is answered from the file. Replay never opens a connection, so it works with the network off, the API down or the data on the server changed. At the end you get either
codeinstances: replay matched (3 of 3 calls served, 0 of 0 values used)
or a list of the ways this run was different:
codeinstances: replay diverged in 2 ways:
- request GET https://api.example.com/v2/forecast isn't in the recording
- recorded call 2 (GET https://api.example.com/v1/forecast) was never made
replay exits 0 when the run matched and 1 when it diverged, so a script or a CI job can act on it.
Replay runs the command stored in the file. Treat an instance from someone else like a script from someone else: run codeinstances show on it first to see what it will run, or give your own command after --.
4. Check a Fix Against a Recorded Failure
Say the recorded run failed. You fix the bug and build again. Replay the new build against the same file:
codeinstances replay failing.instance -- ./myprogram-fixed --same --flags
The fixed program sees exactly the answers the failing one saw. If it now behaves, you know the fix works on the case that actually broke.
5. Check a File Hasn’t Been Changed
codeinstances verify weather.instance
Every instance carries an id, the SHA-256 of its contents in a fixed canonical form. verify recomputes it and exits 1 if anything in the file was edited after it was recorded.
Go Programs: Clock and Random Numbers
The proxy sees a program’s HTTP traffic from outside. It can’t see the clock or random numbers. A Go program that wants those replayed exactly asks for them through the hook package:
import "github.com/codeinstances/codeinstances/hook"
start := hook.Now() // instead of time.Now()
id := hook.Uint64() // instead of math/rand or crypto/rand
pick := hook.Intn(10)
client := hook.Client() // an http.Client that trusts the run's certificate authority
Run normally, these are the real clock and crypto/rand. Under record each value is kept in the instance, and under replay the same values come back in the same order.
Compare Two Runs
Record the run that works and the run that doesn’t, and compare them:
codeinstances diff passing.instance failing.instance
diff pairs up the calls the two runs made and lists what’s different: answers that changed (JSON down to the member), calls only one run made, how each run ended and what each printed. It also shows how long both runs took and which calls got slower. It exits 0 when the runs are the same and 1 when they differ.
Shrink a Failure and Keep It as a Test
When the runs differ in many answers, let shrink find the ones that matter:
codeinstances shrink -pass passing.instance failing.instance
It replays the program offline again and again, with the passing run’s answers swapped in for some of the failing ones, until no single remaining answer can go without the failure going away. It lists those answers and writes shrunk.instance, the failing run with only them left in.
Put that file in a folder with runs that should keep working, and replay them all as regression tests:
codeinstances test testdata/
A run recorded with exit 0 has to replay exactly. A recorded failure has to exit 0 now, so the bug stays fixed. test exits 0 when every test passed and 1 when any failed.
Group Many Failures
When failures pile up, find out how many bugs they are:
codeinstances group failures/
Each failing run gets a fingerprint from how it ended, its error (Go panics, Python tracebacks, Node.js and Java traces and failed Go tests are read for their frames) and the calls that failed, with times, ids and numbers blanked out. Runs with the same fingerprint are one group, biggest first.
Share a Run
Keep runs in a store of your own and move them by id:
codeinstances serve -dir instances # prints its address and a token
export CODEINSTANCES_STORE=http://127.0.0.1:7979 CODEINSTANCES_TOKEN=the-printed-token
codeinstances push failing.instance # prints the id
codeinstances pull 3f2a9c1e77b0 # anyone with the token gets the same run
codeinstances report failing.instance # a Markdown bug report to paste into an issue
The GitHub Actions example pushes failing test runs from CI and puts the report in the job summary.
Try the Demo
Every download includes codeinstances-demo. It starts two local servers whose answers change on every request, records a Go program talking to both, stops the servers, replays the run ten times and checks that changed programs and an edited file are caught. It also compares the recording with a second run that makes one extra request, checks diff finds it, shrinks a failing run down to the one answer that breaks it, checks test fails the program with the bug and passes a fixed build, and checks group puts repeats of one bug together:
codeinstances-demo > report.json
Progress goes to the terminal and a JSON report to report.json. Any failed check makes it exit 1.
Use Cases
The case studies walk through five situations with the commands for each: a flaky CI test, a bug report that arrives as an instance, an AI agent run that went wrong, a dependency upgrade, and work against a metered API.
Before You Rely on It
v0.5.0 is an early release, and some things aren’t covered yet. HTTP is the only protocol recorded, streamed responses arrive all at once on replay, and clock and random capture is Go only. The reference has every command, flag and limit.