Reference

Every command, flag and limit in CodeInstances v0.5.0. For a first run, the quick start is shorter.

Commands

Command	What it does
`codeinstances record [flags] -- command [args...]`	Run the command and keep the run as an instance file
`codeinstances replay [flags] file [-- command [args...]]`	Run the recorded command again, or the one given, with the network replaced by the recording, and compare
`codeinstances diff [flags] a b`	Compare two instances: how they ended, their output, their calls and values, and their speed
`codeinstances shrink [flags] -pass passing failing [-- command]`	Find the fewest of the failing run’s answers that keep the failure, and keep that instance
`codeinstances test [flags] file-or-dir... [-- command]`	Replay instances as regression tests
`codeinstances group [flags] file-or-dir...`	Fingerprint failing runs and group the ones that fail the same way
`codeinstances serve [flags]`	Run a store that keeps instances under their ids
`codeinstances push [-to url] file-or-dir...`	Upload instances to a store
`codeinstances pull [-from url] [-o file] id`	Download an instance by id or unique prefix
`codeinstances report [-url store] file`	Write a Markdown bug report for an instance
`codeinstances show [-json] file`	Describe an instance, or print it whole as JSON
`codeinstances verify file`	Check the file hasn’t been changed since it was recorded
`codeinstances version [-json]`	The version, the format version and the commit it was built from

Exit codes: record exits as the program did. replay exits 0 when the run matched and 1 when it diverged. diff exits 0 when the runs are the same and 1 when they differ. shrink exits 0 when it kept an instance and 1 when the failure couldn’t be narrowed down. test exits 0 when every test passed and 1 when any failed. group, report, push and pull exit 0 when they worked. verify exits 1 for an edited file. Anything else is 2 on a usage or file error.

Flags for record

Flag	What it does
`-o file`	The instance file to write. Default `run.instance`
`-env NAME`	Keep this environment variable in the instance and set it again on replay. Repeatable
`-resolve host=addr`	Send a host name (and optional port) to another address, like curl’s `--resolve`. Repeatable
`-upstream-ca file.pem`	Also trust this file’s certificates when reaching real HTTPS servers. Repeatable
`-keep-secrets`	Keep `Authorization`, `Cookie` and similar headers instead of redacting them
`-timeout d`	Kill the program after this long, such as `30s`. Default: no limit
`-q`	Don’t pass the program’s output through

Flags for replay

Flag	What it does
`-strict`	Insist on HTTP requests in exactly the recorded order
`-ignore-stderr`	Leave stderr out of the comparison
`-json`	Print the report as JSON
`-timeout d`	Kill the program after this long
`-q`	Don’t pass the program’s output through

By default, requests may arrive in any order, since threads race. Each request gets the next recorded answer for the same method, URL and body. Headers aren’t compared.

Flags for diff

Flags go before the two files.

Flag	What it does
`-values`	Compare the hooked clock, random and bytes values themselves. By default only how many of each kind is compared
`-ignore-stderr`	Leave stderr out of the comparison
`-json`	Print the comparison as JSON

How Diff Works

Calls are paired by method, URL and request body, in order, and the calls left over are paired again by method and URL alone, so a changed request body shows up as a change and not as one call missing and one extra. For each pair, diff compares the status and the response body: JSON by member, down to the first value that differs (numbers as written, so 1.0 and 1 differ), other text by line, binary bodies by the first byte that differs. Headers aren’t compared. How the runs ended and what they printed are compared too.

A different command, kept environment or CodeInstances version is shown as a note and doesn’t count as a difference. Speed is shown and never counted: the total time of both runs and, for runs recorded with v0.2.0 or later, the calls whose time changed most. The text output lists the first 50 differences; -json lists them all.

Flags for shrink

Flag	What it does
`-pass file`	The passing run to take answers from. Required
`-o file`	The instance file to write. Default `shrunk.instance`
`-match regexp`	The program’s output must also match this to count as the same failure
`-max-runs n`	Run the program at most this many times, two of them held back for the result. Default 200, at least 4
`-timeout d`	Kill the program after this long, in each run. Default one minute
`-q`	Don’t print a line per run
`-json`	Print the result as JSON

How Shrink Works

Calls are paired as diff pairs them. The answers that differ between paired calls are the candidates. shrink replays the program with the failing run’s answers for some candidates and the passing run’s for the rest, and uses delta debugging to cut the set down until no single answer can be swapped without the failure going away. A run counts as the same failure when it ends with the same exit code or signal, and with -match, when its output matches too. The result is the failing run with only the needed answers left in and only the calls the program made in its last run. Give a command after -- to shrink against a different build; the shrunk file keeps that command.

Answers to calls only the failing run made, response headers and hooked clock and random values are never swapped. shrink refuses with exit 1 when the failing run doesn’t fail again on replay, or when the program fails with the passing run’s answers too. Ctrl-C stops the search.

Flags for test

Flag	What it does
`-expect rule`	`auto` (the default): runs recorded with exit 0 must match, the rest must exit 0. `match` or `pass` sets one rule for every file
`-ignore-stderr`	Leave stderr out when matching
`-timeout d`	Kill the program after this long, in each test. Default one minute
`-v`	List the notes of passing tests too
`-json`	Print the results as JSON

test takes files and folders. Every .instance file under a folder is run, in name order. Give the build under test after --. A failure found with -match that exits 0 is kept with exit 0, so run it with -expect pass.

Flags for group

Flag	What it does
`-v`	List every run in each group, not only the first 5
`-json`	Print the groups as JSON

How Group Works

A run failed when it didn’t exit 0. Its fingerprint has three parts. How it ended. Its error, read from stderr, or from stdout when stderr has nothing: a Go panic or fatal error with the first three functions of the failing goroutine outside the runtime and the testing package, a Python exception with the last three frames of its traceback, an error followed by at frames (Node.js, Java, C#) with its first three frames, the names of failed Go tests, or else the last line that looks like an error, skipping trailers such as exit status 1. And the calls that got no answer or a 5xx status, by method and URL.

Before hashing, times, durations, UUIDs, hex ids, long mixed ids, numbers that don’t follow a letter, ports and query values are blanked out, and escape codes removed. Status codes after status, code, errno and error: are kept. The id starts with fp1:, the version of these rules. Groups are listed biggest first, runs in the order they were recorded. A file given twice counts once. show prints a failing run’s fingerprint.

The Store

codeinstances serve keeps instances in a folder, each as <id>.instance, and serves them over HTTP. It takes an upload only when the file is intact and its id checks out, and keeps the same file once. Every request needs Authorization: Bearer <token>; when no token is given, serve makes one and prints it. -open-read lets anyone read, and uploads still need the token. Uploads must be sent as Content-Type: application/json, and a store on a loopback address answers only to loopback host names (and any -host names), so a web page can’t reach it. It reads its folder at start, keeps an index in memory, and checks each download against its id again.

Request	What it does
`POST /v1/instances`	Store the instance in the body: 201 new, 200 already there, 422 edited, 413 over `-max-size`, 503 when two uploads are already being read
`GET /v1/instances`	List the instances, newest first (`?failed=true` for failing ones)
`GET /v1/instances/{id}`	Download one, by full id or a unique prefix of 12 or more hex digits
`GET /v1/groups`	The failing instances grouped by fingerprint

Flag	What it does
`serve -dir d`	The folder. Default `instances`
`serve -addr a`	Where to listen. Default `127.0.0.1:7979`
`serve -token t`	The token. Default `$CODEINSTANCES_TOKEN`, or a random one
`serve -open-read`	Let anyone read
`serve -max-size n`	The largest upload in bytes. Default 64 MiB
`serve -host name`	Also answer to this host name, such as an HTTPS proxy’s public name. Repeatable
`push -to url`, `pull -from url`	The store. Default `$CODEINSTANCES_STORE`
`push -token t`, `pull -token t`	Default `$CODEINSTANCES_TOKEN`
`pull -o file`	The file to write. Default the first 12 digits of the id
`pull -max-size n`	The largest file to take. Default 64 MiB
`report -url store`	Add the `pull` command. Default `$CODEINSTANCES_STORE`

push checks the store answers with the file’s own id, and pull that what it gets is intact and the one asked for. Neither follows a redirect. report writes Markdown: how the run ended, the command, the fingerprint, the last lines of output, the failed calls and the commands that bring the run back, with run text in code spans and commands quoted for a POSIX shell. The CI example shows it in a GitHub Actions job.

What a Run Keeps

The command, its arguments and the working directory. Environment variables only when named with -env, since the environment often holds secrets.
How it ended: the exit code, or the signal that killed it, or a timeout, and how long it took.
Its stdout and stderr, up to 16 MiB each.
Every HTTP and HTTPS exchange in arrival order: method, URL, headers, body, and the status, headers and body of the answer, or the error when none came back. From v0.2.0 on, how long each exchange took.
For Go programs using the hook package, every clock and random value it asked for.

How Recording Works

record starts a proxy on a loopback port and runs the program with HTTP_PROXY and HTTPS_PROXY pointing at it. Plain HTTP is passed on and kept. For HTTPS, the proxy stands in for the server with a certificate from a certificate authority made for that run alone. The program is told to trust it through SSL_CERT_FILE, REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE and NODE_EXTRA_CA_CERTS, which most HTTP clients read. The authority’s key stays in memory and is gone when the run ends.

Only header values are redacted. The values of Authorization, Proxy-Authorization, Cookie, Set-Cookie, X-Api-Key, Api-Key, X-Auth-Token and X-Amz-Security-Token are replaced with [redacted] unless you pass -keep-secrets.

The file is written to a temporary file and renamed into place, so a recorder that’s killed leaves no half-written file. Ctrl-C goes to the program, and the recorder waits for it to finish and keeps what happened.

How Replay Works

replay starts the same kind of proxy with a fresh authority, and answers from the file. It never dials out. A request with no recorded answer gets a 502 and counts as a divergence, and so does every recorded request the program didn’t make. A call that failed when it was recorded gets a 502 with the same body the program saw then. Output and exit code are compared at the end.

The Go Hook Package

github.com/codeinstances/codeinstances/hook has Now(), Since(), Uint64(), Int63(), Intn(), Float64(), Read(), Transport() and Client(). Outside codeinstances they’re the real clock and crypto/rand, and nothing is written. Under record each value is kept. Under replay the recorded values come back in order. A program that asks for more values than were recorded gets fresh ones, and the replay is reported as diverged. So is one that asks for fewer.

The Instance Format

An instance is one JSON file with the format string codeinstances/0. FORMAT.md describes every member, how the id is computed and how a replay should use the file, so other tools can read and write instances too. The post on the open instance format explains why it’s built the way it is.

Limits in v0.5.0

This is an early release, and these are the things it doesn’t do yet:

HTTP only. WebSocket, raw TCP, gRPC streaming and other protocols aren’t recorded.
Bodies are buffered whole, up to 64 MiB each. A streamed response reaches the program all at once on replay.
Empty stdin. The program gets an empty stdin, both when recorded and on replay.
Clock and random capture is Go only, through the hook package. For other languages, HTTP is captured and the clock and random numbers aren’t.
Programs that ignore HTTP_PROXY and HTTPS_PROXY aren’t captured.
Go and localhost. Go’s HTTP client never uses a proxy for localhost or 127.0.0.1. A Go program talking to a local server should use a host name, and -resolve sends that name to the local address.
Pinned certificates. Programs that pin certificates won’t accept the run’s authority.
Go on macOS and Windows ignores SSL_CERT_FILE, so a Go program there needs hook.Client() or hook.Transport() for HTTPS.
Only header values are redacted. Secrets in bodies, in URL query strings (an api_key= parameter, say) and in the command line are kept as they are, and so are bytes from hook.Read, which may be keys.
Replay hands the program the redacted values, so a program that prints a cookie it was sent will diverge.
One hooked process at a time. Each process that uses the hook package replays the recorded values from the start, so exact value replay works for one hooked process at a time.
Output is kept up to 16 MiB per stream.
No time limit on replay unless you give -timeout, so give one when replaying a run that timed out.
diff doesn’t compare headers, only a call’s status and body.
No cost figures. diff reports differences and time, not money, tokens or bytes sent.
Shrink swaps whole answers to calls both runs made. It doesn’t shorten a response body, drop requests, or change headers or hooked values, and it needs a passing run to take answers from.
Fingerprints are a heuristic. Two bugs with the same ending, error line and failed calls share a group, and a bug whose message carries a changing word splits into several. Calls are told apart by method and URL, not body.
The store is small. One server, one folder, one token, no accounts, an index in memory that suits thousands of runs. It speaks plain HTTP; put an HTTPS proxy in front of it. Nothing is scrubbed before a push beyond the redacted headers.
Shrink trusts each run. A program that fails only some of the time with the same answers can mislead it.
The recording proxy has no password. It listens on a loopback port while the program runs, so other programs on the same machine could send traffic through it.