dpsapi/e2e/README.md

# dpsapi E2E suite (Playwright)

API/process-only Playwright suite (no browsers) exercising the compiled `dpsapi` binary and the final image.

## Running

```sh
make test-e2e                      # make build, then --project=host
make docker-e2e                    # make docker, then --project=image with DPSAPI_IMAGE set
cd e2e && npm ci
npx playwright test --project=host
DPSAPI_IMAGE=git.bjphoster.com/source/dpsapi:dev npx playwright test --project=image
npx playwright test --project=host tests/host/auth.spec.ts -g 'duplicated'
```

| Variable | Meaning |
|---|---|
| `DPSAPI_BIN` | Binary under test (relative paths resolve against `e2e/`). Default `../dpsapi`; only the default is (re)built by `host-setup` via `make build`. |
| `DPSAPI_E2E_SKIP_BUILD=1` | Skip that build (set by `make test-e2e`, which already built). |
| `DPSAPI_IMAGE` | Image under test for `--project=image` (required there). |
| `DPSAPI_E2E_SEED` | Run seed for all PRNG streams (default `dpsapi-e2e-default-seed-1`, printed at start). |
| `DPSAPI_E2E_WORKERS` | Worker count (default 4). |
| `DPSAPI_E2E_ALPINE` | Image for test/helper containers (default `alpine:3.22`, must exist locally). |

Prerequisites (checked by the setup projects, failing fast with `[harness:prerequisite]`): `curl`, unprivileged `unshare -Urm` with bind/tmpfs mounts, a reachable Docker daemon, `alpine:3.22`, and the binary (host) or the image (image).

Projects: `host-setup` → `host` (`tests/host/`), `image-setup` → `image` (`tests/image/`). Global setup assigns a run id and scratch dir `/tmp/dpsapi-e2e-<runId>/`; global teardown fails the run (non-zero exit) if containers/networks/volumes labelled `dpsapi.e2e=<runId>`, processes carrying `DPSAPI_E2E_PROC=<runId>`, or workspaces remain, after cleaning them up.

Reports: `reports/html/` (HTML), `reports/results.json`, per-test files in `test-results/`.

## Writing tests

Import everything from the harness, never directly from `@playwright/test`:

```ts
import { test, expect, expectErrorResponse, assertDbInvariants } from '../../harness';

test('inactive token → 401', async ({ cli, startServer, http }) => {
  const token = await cli.tokenNew('demo');
  await cli.tokenDelete(token);
  const server = await startServer({ logLevel: 'DEBUG' });       // FakeDocker (empty) by default
  expectErrorResponse(await http.get(server.url(), { token }), 401);
  expect((await server.stop()).exitCode).toBe(0);                // every server must be stopped
  assertDbInvariants(server.dbPath, { operationCount: 0 });
});
```

### Fixtures (`harness/fixtures.ts`)

| Fixture | Type | Notes |
|---|---|---|
| `ws` | `Workspace` | Fresh temp dir; `ws.dbPath` (`<dir>/dpsapi.db`, not created) is the default `DB_PATH`. `path()`, `mkdir()`, `writeFile()`. Removed after the test (root-owned files via an alpine container). |
| `cli` | `CliRunner` | Host binary, **no shell**, argv byte-exact. `run(args, {env, cwd, timeoutMs, input, allowSignal})` → `CliResult`; `shell(script, opts)` runs `/bin/sh -c` with `$DPSAPI_BIN`; `tokenNew(desc?)` → token, `tokenList(scope = 'active' | 'all', opts?)` → `TokenListRow[]` (`token list`: active tokens only, asserted; `'all'` runs `token list all`), `tokenListAll(opts?)`, `tokenDelete(tok)`, `audit(arg? \| args[], opts?)` → `AuditRow[]` (`audit()` = latest 10, `audit('all')` = every row, `audit('25')`, `audit('1h')`), `help(opts?)` (stdout exactly `USAGE_TEXT`) (all assert exit 0 and, unless `LOG_LEVEL` is overridden, empty stderr); `env(over)`; `withBinary(path)`. Env overrides: `undefined` removes a variable. Default env is only `PATH`, `DB_PATH`, leak marker. |
| `startServer` | `(opts?) => Promise<ServerHandle>` | See below. |
| `newFakeDocker` | `() => Promise<FakeDocker>` | Extra fake Engine on a workspace socket; closed automatically. |
| `http` | `HttpClient` | `fetch(url, {method, headers, token, data, timeoutMs})` / `get()` (Playwright request, no redirects) → `HttpResult`; `curl(CurlOptions)` → `CurlResult`; `raw(RawHttpOptions)` → `RawResult`. All recorded. |
| `docker` | `RealDocker` | Real Engine containers named `dpsapi-e2e-<runId>-<testKey>-<suffix>`, labelled `dpsapi.e2e=<runId>`, `dpsapi.e2e.test=<testKey>`: `run({suffix, image, command, labels, env, init, network, extraArgs})`, `runMany(n, optsFor, concurrency)`, `stop(c, sec)`, `start`, `remove`, `inspect`, `listOwn({all})`, `runningIds()`, `cleanup()`. Everything labelled for the test is force-removed at teardown. |
| `image` | `ImageRunner` | `$DPSAPI_IMAGE`: `start(ImageStartOptions)` → `ImageContainer`; `run(args, {env, dataDir, docker, user, timeoutMs, extraArgs})` (one-shot `docker run --rm`, `CliResult` with `via: 'docker-run'`); `extractBinary()` → host path of `/app`; `snapshotDataDir(dir)`; `removeDataDir(dir)`; `containerName(suffix)`. |
| `secrets` | `SecretRegistry` | `register`, `registerAllIn`, `redact`, `findIn`, `assertNoSecretsIn(text, where)`. Tokens printed by `token new`, `token list [all]` and `audit [all \| n \| n<unit>]` via `cli`/`image` (`printsTokens(args)`) are registered automatically; `help` prints none. |
| `rng` | `Rng` | Deterministic per test (seed = run seed + title path, annotated as `seed`). `next/int/bool/pick/shuffle/bytes/string/fork`. |
| `fuzz` | `FuzzRecorder` | `fuzz.record(case)` stores a JSON-safe snapshot taken at that moment (BigInts, e.g. audit selections, become decimal strings) and returns the case unchanged; `fuzz.recordLive(obj)` for a case object filled in afterwards; `fuzz.seed`, `fuzz.rng`; cases attached as `fuzz-cases.json` on failure (or `fuzz.attach()`). Use `safeStringify(v)` (not `JSON.stringify`) for labels built from cases. |
| `recorder`, `testKey`, `e2e` | internals | `e2e.allocatePort()` (parallel-safe free port), `e2e.trackedDbs.add(path)` (snapshot on failure). |
| `scanLogsForSecrets` | option, default `true` | `test.use({ scanLogsForSecrets: false })` for tests that put registered tokens into descriptions on purpose. |

### `startServer(opts)` / `ServerHandle` (`harness/server.ts`)

Options: `logLevel` (default `'INFO'`, `null` = unset), `addr` (default `'127.0.0.1'`, `null` = unset), `port` (default allocated, `null` = unset → 80, only with `privateNetwork`), `dbPath`, `docker` (`FakeDocker` | `'real'` | `'inaccessible'` (EACCES) | `'missing'` (ENOENT) | `{bindSource}`; default a new empty FakeDocker), `env`, `privateNetwork`, `waitReady` (default true), `readyTimeoutMs`, `label`, `binary`.

Non-`real` backends run the binary as `unshare -Urm sh -c 'mount --bind <src> /var/run/docker.sock && exec <bin>'` (same pid, host network). Readiness polls `OPTIONS *` over a raw socket: answered by Go itself, so it creates no app log record and no audit row.

Handle: `baseURL`, `url(path = '/api/v1/containers')` (path verbatim), `port`, `addr`, `connectHost`, `dbPath`, `env`, `fakeDocker`, `pid`, `generation`, `stdout`, `stderr`, `logs()` → `LogSet`, `allOutput()`, `isRunning()`, `exit`, `waitReady()`, `waitForLog(query, ms)`, `signal(sig)`, `stop(sig = 'SIGTERM', {timeoutMs, allowSignalExit})` → `ExitResult {exitCode, signal, durationMs, stdout, stderr}` (hang → `HarnessError('hang')` after SIGKILL; death by signal → `HarnessError('signal')`), `kill()`, `waitForExit(ms)`, `restart()` (SIGTERM, requires exit 0, same port/DB/backend).

### `FakeDocker` (`harness/fake-docker.ts`)

Fake Engine API on a unix socket: `HEAD/GET /_ping`, `GET [/vX]/containers/json`. Live switches: `empty()`, `many(n, rng?)`, `setContainers(list)`, `error(status, message)`/`error500()`, `flaky(k)` (next k list calls → 500), `destroy()`, `hang()`, `badJson(body)`, `passthrough(upstream)`, `delay(ms)`. Counters: `stats {ping, list, other, total}`, `requests[] {method, url, path, query, at}`, `listCalls()`, `resetStats()`. Summaries: `makeContainerSummary(i, rng?, overrides)`, `makeContainerSummaries(n, rng?)`. Never unlink the socket while a server uses it (bind mounts pin the inode) — switch modes instead.

### `ImageContainer` (`harness/image.ts`)

`image.start({logLevel, addr, containerPort, dbPath, env, docker: 'real' | 'none' | FakeDocker | {socketPath} (default new FakeDocker), dataDir, user, publish, extraArgs, args, waitReady, readyTimeoutMs, suffix})`. Mounts the socket at `/var/run/docker.sock`, `dataDir` at `/data` (`DB_PATH=/data/dpsapi.db`), publishes `127.0.0.1:<random>`. Handle: `name`, `id`, `hostPort`, `baseURL`, `url()`, `dataDir`, `containerDbPath`, `fakeDocker`, `waitReady()`, `exec(args, {env, user, timeoutMs})` → `CliResult` (`via: 'docker-exec'`; exit ≥125 → `HarnessError`), `execRaw(argv)` → `ProcResult`, `logs()` → `{stdout, stderr}`, `parsedLogs()`, `inspect()`, `isRunning()`, `stop({signal?, timeoutSec})` → `{exitCode, oomKilled, stdout, stderr}` (137 = killed after grace period), `waitForExit()`, `remove()`, `snapshotDb()` → host path of a host-owned copy of the (root-owned) DB + WAL. Reuse `dataDir` to replace a container on the same data.

### HTTP helpers (`harness/http.ts`, `harness/assertions.ts`)

- `HttpResult` / `CurlResult`: `status`, `headers` (lower-case, repeats joined by `, `), `headersArray`, `body`, `bodyBuffer`, `json()`, `durationMs`; curl adds `exitCode` (7 refused, 52 empty reply…), `stderr`, `writeOut`, `argv`. Curl options: `url, method, headers: string[]` (raw `-H` lines, repeatable; `'Name;'` = empty value), `token, data, pathAsIs, head, httpVersion, timeoutMs, allowTimeout, extraArgs`.
- `RawResult.outcome`: `'response' | 'closed' | 'reset' | 'refused' | 'timeout'` (timeout only with `allowTimeout`, otherwise `HarnessError`); `responses[]`/`response` (`status`, `headers` pairs, `header(n)`, `headerAll(n)`, `body`, `bodyText`), `raw`, `closedByServer`. Options: `host, port, data (string | Buffer | chunks), chunkDelayMs, timeoutMs, expectResponses, waitForClose, endAfterWrite, head`. `buildRawRequest({method, target, headers, body, host, version, close})`.
- `expectErrorResponse(res, 401|404|405|500, {head})` (exact body, Content-Type/Length, `WWW-Authenticate: Bearer` only on 401, `Allow: GET` only on 405), `expectContainersResponse(res)` → array, `expectNoInternalDetails(body)`, `view(res)`, `ERROR_BODIES`.

### CLI / DB / logs

- `CliResult` = `ProcResult` (`stdout, stderr, stdoutBuffer, stderrBuffer, exitCode, signal, timedOut, durationMs, command, argv`) + `args`, `via`.
- `expectCliSuccess(r, {stdout?, stderr?})`; `expectCliFailure(r, {exitCode, message?, usage?, exactStderr = true, stdout = ''})` — first throws `HarnessError` on signal/timeout; `message` is the text after `dpsapi: `; usage text expected for `usage error:` messages. `expectExitedNormally(r)`, `errorMessage(r)`, `USAGE_TEXT` (printed on stderr after every usage error and on stdout by `help`), `USAGE_ERROR_MESSAGES` (the complete list).
- Tables (`token list [all]`, `audit`): `padTable(lines: string[][])` reproduces the app's `text/tabwriter` layout exactly (every column but the last padded to 2 + its widest cell in runes — `runeLength(s)` = `Array.from(s).length` — over header and rows of that invocation; last cell unpadded; no trailing blanks; `\n` after every line; header only when there are no rows). `TOKEN_LIST_COLUMNS`, `AUDIT_COLUMNS`, `EMPTY_TOKEN_LIST`, `EMPTY_AUDIT` (header-only outputs), `formatTokenList(entries)` / `formatAuditOutput(rows)` (exact expected stdout for known rows; prefer `expect(r.stdout).toBe(...)` with them), and the strict parsers `parseTokenList(stdout)` (TOKEN = first 43 chars, then a Go-quoted DESCRIPTION parsed as a quoted literal — it may contain spaces and `\"` — then ACTIVE, CREATED_AT, DELETED_AT) / `parseAuditOutput(stdout)` (TIME, TOKEN, ORIGIN, then OPERATION = rest of the line), which also require the output to re-render byte-exactly through `padTable`. Rows carry the raw padded `line`; widths depend on the printed rows, so compare rows by cells, not lines, across invocations.
- Oracles (`harness/personas.ts`): `cliOutcome(args, model)` → `help` (exit 0, usage text on stdout) | `list` with `scope: 'active' | 'all'` (`token list` / `token list all`; any other single argument → `unknown token list argument`, two or more → `token list takes at most one argument`) | `new` | `delete` | `audit` (`selection`: `{all: true}` for `audit all` — exact, lowercase, one argument —, `{limit}` for no argument / `n`, `{windowNs}` for `n<unit>`) | `error`; `auditArgsOutcome(args)`, `selectAuditRows(ops, selection, nowNs?)` / `expectedAuditStdout(ops, selection, nowNs?)` (`all`: every row, oldest first — parsed time, then rowid; `nowNs` only for windows), `expectAuditWindow(...)`, `expectedTokenListStdout(dbPath, scope)` (exact stdout from the stored rows), `StateModel.verifyList(rows, label, scope = 'active')`, `HELP_NEAR_MISSES` / `LIST_ALL_NEAR_MISSES` / `AUDIT_ALL_ARGV_NEAR_MISSES` (full argv lists) and `AUDIT_ALL_NEAR_MISSES` / `AUDIT_ALL_EXTRA_ARGS` (`audit` arguments: `ALL`, `All`, `all1`, `1all`, ` all`, `alls`, `--all`… → `invalid audit argument`; `all x`… → `audit takes at most one argument`) for argv generators; `fuzzAuditArgs(rng)` also yields `all` and its near misses.
- `goQuote`, `goUnquote`, `goUnquoteBytes` (Go `strconv` compatible), `shellQuote`.
- DB (read-only `node:sqlite`): `readTokens(db)` (`descriptionHex` = exact bytes), `readToken`, `readOperations`, `countRows`, `readSchema`, `withDb`, `fileDigest`, `dbSnapshot`, `RFC3339NANO_UTC_RE`, `rfc3339ToNanos`, `msToNanos`, and `assertDbInvariants(db, {tokenCount?, operationCount?, auditBeforeDeletion?})` → `{tokens, activeTokens, inactiveTokens, operations}`: integrity_check, foreign_key_check, exact schema, storage classes, token format, active∈{0,1}, deleted_at NULL iff active, strict RFC3339Nano UTC timestamps, deleted_at ≥ created_at, operation = `GET containers`, origin is an IP, audit time ≥ token created_at.
- Logs: `parseLogs(text)` → `LogSet` (`records`, `unparsed`, `filter/find/count/has(query)`, `levels()`, `messages(level?)`, `requests()`); `LogQuery {level, msg (string|RegExp), attrs {k: string|number|boolean|RegExp|fn}}`; `LOG_LEVELS`, `visibleLevels(min)`, `levelRank`, `LOG_TIME_RE`; `tokenId(token)` = the `token_id` the app logs.
- Misc: `runProcess(cmd, argv, {env, cwd, timeoutMs, input, allowTimeout, allowSignal})`, `toolEnv()`, `dockerCli(args, {allowFailure})`, `listByLabel`, `mapLimit`, `pollUntil`, `sleep`, `occupyPort`, `isPortFree`, `HarnessError`/`isHarnessError`.

## Conventions

- **Files**: `tests/host/<area>.spec.ts`, `tests/image/<area>.spec.ts`; persona suites `tests/host/persona-<name>.spec.ts`. One concern per `test`; use `test.step` for case tables.
- **Exit codes are always explicit**: `expect(r.exitCode).toBe(0)` / `expectCliFailure(r, {exitCode: 1|2, ...})`; every server is stopped by the test and its exit code asserted.
- **Infrastructure vs application failures**: timeouts, unexpected signals, hangs, spawn/docker failures, unparseable output throw `HarnessError` (`[harness:<kind>]`) — never catch them to satisfy a negative assertion.
- **Isolation**: everything mutable lives in the test's workspace, FakeDocker, ports and Docker resources keyed by `testKey`. Never assert on global container counts — only on the suite's names/labels. Never touch resources without the `dpsapi-e2e-` prefix.
- **Cleanup/leaks**: teardown kills+fails on running servers or any live process spawned by the worker, removes labelled containers and the workspace; global teardown fails the run on any leftover.
- **Secrets**: never `console.log` tokens (worker stdout is redacted anyway). Artifacts and the report redact registered tokens and any token-shaped 43-char string as `[REDACTED:<token_id>]`. Server logs, CLI stderr and container logs are scanned for registered tokens at teardown.
- **Artifacts on failure**: per server generation `stderr`/`stdout`/exit, container logs, `cli-invocations.json`, `http-exchanges.json`, `fake-docker-N.json`, `db-snapshot-N.txt` (integrity + rows), `fuzz-cases.json` (with seeds), `harness-problems.txt`, `harness-notes.txt`.
- **DB**: read-only inspection only; write to the app DB from tests only when a test explicitly simulates lock contention (open your own `DatabaseSync` for that).
- **Timing**: wait on observable conditions (`waitReady`, `waitForLog`, `pollUntil`, `fake.listCalls()`), never fixed sleeps.

## CLI command coverage

| Command | Main specs |
|---|---|
| `help` (stdout = usage text, exit 0, no DB; config errors win; DEBUG/TRACE log records on stderr only) | `tests/host/config-usage.spec.ts`, `logging.spec.ts`, `tests/image/image-contract.spec.ts` (`docker exec … /app help`, `docker run … help`) |
| `token list` (active only) / `token list all` (every token), padded tables, `unknown token list argument` / `token list takes at most one argument` | `tests/host/cli-tokens.spec.ts`, `config-usage.spec.ts`, personas, `tests/image/image-contract.spec.ts`, `multi-transport.spec.ts`, `image-lifecycle.spec.ts` |
| `audit [all \| n \| n<unit>]` (selection forms, units, limits, ordering, window boundary, read-only, usage errors, log levels, unparseable time) | `tests/host/cli-audit.spec.ts`, `population.spec.ts`, personas, `tests/image/image-contract.spec.ts` (`docker exec … /app audit`, `/app audit all`), `multi-transport.spec.ts`, `image-lifecycle.spec.ts` |
| `audit all` (every row oldest first — parsed time, ties in insertion order —; header only when empty; exact padded table with more than 10 rows; read-only; `ALL`/`All`/`all1`/`1all`/` all`/`alls`/`--all` → `invalid audit argument`, `all x` → `audit takes at most one argument`, never echoed, no database; DEBUG `cli dispatch command=audit all=true`) | `tests/host/cli-audit.spec.ts` (`audit all` describe; also in the read-only, log-level, unparseable-time, population and usage tables), `config-usage.spec.ts`, `harness-selfcheck.spec.ts`, personas (fuzzer, operator session, cross abuse, stupid user), `tests/image/image-contract.spec.ts`, `multi-transport.spec.ts` |
| Usage-error table (every message, exact stderr, no DB) | `tests/host/config-usage.spec.ts` (checked against `USAGE_ERROR_MESSAGES` and `cliOutcome`) |

The argv fuzzers (`persona-fuzzer-user`, `persona-cross-abuse`, `persona-operator-session`) generate `help`, `token list all`, `audit all` and their near misses (`token list ALL`, `help help`, `token list all all`, `audit ALL`, `audit all x`, `audit --all`, …), check every outcome with `cliOutcome`, and require minimum counts of each (e.g. the fuzzer forces `audit all` / an `audit all` near miss every 30th case; the operator session has an `audit-all` action).