tryscript-reference.md

tryscript Reference

Complete reference for writing tryscript golden tests. This document covers all syntax, configuration, and patterns needed to write accurate CLI tests on the first try.

Overview

Tryscript is a markdown-based CLI golden testing format. Test files are markdown documents with embedded console code blocks specifying commands and expected output.

Design Philosophy:

Shell delegation: Commands run in a real shell with full shell features
Markdown-first: Test files are valid markdown, readable as documentation
Output matching: Patterns like [..] match variable output; they're not for commands

Quick Start Example

---
sandbox: true
env:
  NO_COLOR: "1"
---

# Test: Basic echo

```console
$ echo "hello world"
hello world
? 0
```

# Test: Command with variable output

```console
$ date +%Y
[..]
? 0
```

Test File Structure

┌──────────────────────────────────────┐
│ ---                                  │  YAML Frontmatter (optional)
│ env:                                 │  - Configuration
│   MY_VAR: value                      │  - Environment variables
│ sandbox: true                        │  - Patterns
│ ---                                  │
├──────────────────────────────────────┤
│ # Test: Description                  │  Test heading (# or ##)
│                                      │
│ ```console                           │  Test block
│ $ command --flag                     │  - Command starts with $
│ expected output                      │  - Expected stdout follows
│ ? 0                                  │  - Exit code (optional, default 0)
│ ```                                  │
└──────────────────────────────────────┘

Command Block Syntax

$ command [arguments...]     # Command to execute (required)
> continuation line          # Multi-line command continuation
expected output              # Expected stdout (line by line)
! stderr line                # Expected stderr (when separating streams)
? exit_code                  # Expected exit code (default: 0)

Examples

Simple command:

$ echo "hello"
hello
? 0

Non-zero exit code:

$ exit 42
? 42

Multi-line command:

$ ls -la | \
> grep ".md" | \
> wc -l
5

Stderr handling:

$ cat nonexistent 2>&1
cat: nonexistent: No such file or directory
? 1

Separate stderr assertion:

$ ./script.sh
stdout line
! stderr line
? 0

Elision Patterns

Patterns in expected output match variable content. There are three categories of wildcards, listed in order of preference:

Named Patterns

Named patterns match typed dynamic values with specific meaning:

Pattern	Matches	Example
`[CWD]`	Current working directory	`[CWD]/output.txt`
`[ROOT]`	Test file directory	`[ROOT]/fixtures/`
`[EXE]`	`.exe` on Windows, empty otherwise	`my-cli[EXE]`
`[PATTERN]`	Custom pattern from config	User-defined regex

Unknown Wildcards

Unknown wildcards are temporary placeholders for output you haven't filled in yet. They are intended to be expanded with --expand before finalizing tests. A warning is always shown when unknown wildcards are present.

Pattern	Matches	Example
`[??]`	Any text on a single line	`Result: [??]`
`???`	Zero or more complete lines	`???\nDone`

Generic Wildcards

Generic wildcards intentionally omit unpredictable or irrelevant output. Use these when the exact value doesn't matter for the test.

Pattern	Matches	Example
`[..]`	Any text on a single line	`Built in [..]ms`
`...`	Zero or more complete lines	`...\nDone`

Pattern Examples

Single-line wildcard:

$ date
[..]
? 0

Unknown wildcard (to be expanded later):

$ my-cli process data.json
[??]
? 0

Multi-line wildcard:

$ ls -la
total [..]
...
-rw-r--r-- 1 user user [..] README.md

Custom pattern:

patterns:
  VERSION: '\d+\.\d+\.\d+'

$ my-cli --version
my-cli version [VERSION]

Wildcard Best Practices

Prefer named patterns when the output has a known structure (e.g., [VERSION], [HASH]). This makes tests self-documenting.
Use unknown wildcards ([??]/???) as temporary scaffolding when writing new tests. Run with --expand to fill them in with actual output.
Use generic wildcards ([..]/...) for output that is intentionally variable (timestamps, durations, dynamic content) and should remain elided.

Configuration (Frontmatter)

All options are optional. Place at the top of the file:

---
cwd: ./subdir              # Working directory (relative to test file)
sandbox: true              # Run in isolated temp directory
env:                       # Environment variables
  NO_COLOR: "1"
  MY_VAR: value
timeout: 5000              # Command timeout in milliseconds
patterns:                  # Custom elision patterns
  UUID: '[0-9a-f]{8}-...'
fixtures:                  # Files to copy to sandbox
  - data/input.txt
before: npm run build      # Run before first test
after: rm -rf ./cache      # Run after all tests
path:                      # Directories to prepend to PATH
  - ../dist
  - $TRYSCRIPT_PACKAGE_BIN # Access node_modules/.bin via env var
---

Config Options Reference

Option	Type	Default	Description
`cwd`	path	`"."`	Working directory (relative to test file)
`sandbox`	`boolean \| path`	`false`	Run in isolated temp directory
`env`	`object`	`{}`	Environment variables passed to shell
`timeout`	`number`	`30000`	Command timeout in milliseconds
`patterns`	`object`	`{}`	Custom regex patterns for `[NAME]`
`fixtures`	`array`	`[]`	Files to copy to sandbox
`before`	`string`	-	Shell command before first test
`after`	`string`	-	Shell command after all tests
`path`	`string[]`	`[]`	Directories to prepend to PATH (supports `$VAR` expansion)

Sandbox Mode

Sandbox provides test isolation by running commands in a temporary directory:

Configuration	Behavior
`sandbox: false` (default)	Commands run in `cwd` (test file dir)
`sandbox: true`	Creates empty temp dir, commands run there
`sandbox: ./fixtures`	Copies `./fixtures/` to temp dir, runs there

When sandbox is enabled:

Fresh temp directory created for each test file
Fixtures are copied before tests run
[CWD] matches the sandbox directory
Files created by tests don't pollute source

Sandbox with Fixtures

---
sandbox: true
fixtures:
  - data/input.txt                 # Copies to sandbox/input.txt
  - source: config/settings.json   # Copies to sandbox/custom.json
    dest: custom.json
---

Environment Variables

Use env to set variables. The shell handles $VAR expansion:

env:
  CLI: ./dist/cli.mjs
  DEBUG: "true"

$ $CLI --version
1.0.0

Important: Variables are for the shell, not for output matching.

Built-in Environment Variables

Tryscript sets these environment variables for test commands:

Variable	Description
`NO_COLOR`	Set to `"1"` by default (disables colors)
`FORCE_COLOR`	Set to `"0"` (disables forced colors)
`TRYSCRIPT_TEST_DIR`	Absolute path to directory containing the test file
`TRYSCRIPT_PACKAGE_ROOT`	Absolute path to directory containing nearest `package.json` (if found)
`TRYSCRIPT_GIT_ROOT`	Absolute path to directory containing nearest `.git` (if found)
`TRYSCRIPT_PROJECT_ROOT`	Most specific of `PACKAGE_ROOT` or `GIT_ROOT` (deepest path)
`TRYSCRIPT_PACKAGE_BIN`	Absolute path to `node_modules/.bin` directory (if exists)

Project root variables help write portable tests that work across different project types:

TRYSCRIPT_PACKAGE_ROOT - For npm/Node.js projects with package.json
TRYSCRIPT_GIT_ROOT - For any git repository (Rust, Go, Python, etc.)
TRYSCRIPT_PROJECT_ROOT - Use this when you don't care about project type
TRYSCRIPT_PACKAGE_BIN - For npm packages with node_modules/.bin (use in path:)

Example using TRYSCRIPT_PROJECT_ROOT:

$ test -n "$TRYSCRIPT_PROJECT_ROOT" && echo "in a project"
in a project
? 0

Testing CLI Applications

Tryscript provides several ways to make CLI binaries available in tests.

path: Custom Binary Directories

Use path to prepend directories to PATH, making executables available by name:

---
sandbox: true
path:
  - ../dist                  # Relative to test file directory
  - $TRYSCRIPT_PACKAGE_BIN   # Use node_modules/.bin via env var
---

$ my-cli --version
1.0.0
? 0

Key behaviors:

Paths are resolved relative to the test file directory (not the sandbox CWD)
Multiple paths are prepended in order (first has highest priority)
Works with or without sandbox mode
Frontmatter and config file paths are merged (frontmatter first)
Environment variable expansion: Path entries support standard shell variable syntax:
- $VAR - expands any environment variable (lowercase or uppercase)
- ${VAR} - braced syntax also supported
- Tryscript env vars (TRYSCRIPT_*) are checked first, then process env vars
- Undefined variables expand to empty string

Using node_modules/.bin (npm/pnpm/bun)

For Node.js projects using npm, pnpm, or bun, use $TRYSCRIPT_PACKAGE_BIN to access installed CLI tools:

---
sandbox: true
path:
  - $TRYSCRIPT_PACKAGE_BIN   # Expands to node_modules/.bin
---

# Test: Run your CLI by name
```console
$ my-cli --version
1.0.0
? 0

Test: Use any installed dev dependency

$ prettier --check src/
[..]
? 0


This works for any executable installed via `npm install`, `pnpm add`, or `bun add`. The variable only expands if `node_modules/.bin` exists (i.e., after running your package manager's install command).

**Typical project setup:**

my-project/ ├── package.json # TRYSCRIPT_PACKAGE_ROOT points here ├── node_modules/ │ └── .bin/ # TRYSCRIPT_PACKAGE_BIN points here │ ├── prettier │ ├── eslint │ └── my-cli # Your package's bin entry └── tests/ └── cli.tryscript.md # Your test file


### Language-Specific Examples

**Rust CLIs:**
```yaml
---
path:
  - ../target/release
---

Python with venv:

---
path:
  - ../.venv/bin
---

Go CLIs:

---
path:
  - ../bin
---

Test Annotations

Control test execution with HTML comments:

## This test is skipped <!-- skip -->

## Only run this test <!-- only -->

Annotation	Effect
`<!-- skip -->`	Test is skipped, marked as passed
`<!-- only -->`	Only tests with this annotation run

Complete Example

Here's a complete test file demonstrating all features:

---
sandbox: true
env:
  NO_COLOR: "1"
  CLI: ./dist/my-cli.mjs
timeout: 5000
patterns:
  VERSION: '\d+\.\d+\.\d+'
  TIMESTAMP: '\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}'
fixtures:
  - test-data/config.json
before: echo "Setup complete"
---

# CLI Golden Tests

These tests validate the my-cli command-line tool.

## Basic Commands

# Test: Show version

```console
$ $CLI --version
my-cli version [VERSION]
? 0
```

# Test: Show help

```console
$ $CLI --help
Usage: my-cli [options] [command]

Options:
  --version  Show version
  --help     Show help
...
? 0
```

## Error Handling

# Test: Missing required argument

```console
$ $CLI process
Error: missing required argument 'file'
? 1
```

# Test: File not found

```console
$ $CLI process nonexistent.txt 2>&1
Error: file not found: nonexistent.txt
? 1
```

## Feature Tests

# Test: Process config file

```console
$ $CLI process config.json
Processing: config.json
Done at [TIMESTAMP][..]
? 0
```

# Test: Verbose output <!-- skip -->

```console
$ $CLI --verbose process config.json
[DEBUG] Loading config.json
...
Done
? 0
```

CLI Usage

tryscript                              # Show help (same as --help)
tryscript run [files...]               # Run golden tests
tryscript coverage <commands...>       # Run commands with merged coverage
tryscript docs                         # Show this reference
tryscript readme                       # Show README

Run Options

Option	Description
`--update`	Update test files with actual output
`--expand`	Expand unknown wildcards (`???`/`[??]`) with actual output
`--expand-generic`	Expand unknown + generic wildcards
`--expand-all`	Expand all wildcards (including named patterns)
`--capture-log <path>`	Write wildcard capture log to YAML file
`--diff` / `--no-diff`	Show/hide diff on failure
`--fail-fast`	Stop on first failure
`--filter <pattern>`	Filter tests by name
`--verbose`	Show detailed output
`--quiet`	Suppress non-essential output
`--coverage`	Enable code coverage collection (requires c8)

Coverage Options

All coverage options mirror c8 CLI flags for familiarity:

Option	Description	Default
`--coverage-dir <dir>`	Output directory for reports	`coverage-tryscript`
`--coverage-reporter <r...>`	Coverage reporters	`text`, `html`
`--coverage-exclude <p...>`	Patterns to exclude	none
`--coverage-exclude-node-modules`	Exclude node_modules	`true`
`--no-coverage-exclude-node-modules`	Include node_modules	-
`--coverage-exclude-after-remap`	Exclude after sourcemap remap	`false`
`--coverage-skip-full`	Hide 100% covered files	`false`
`--coverage-allow-external`	Allow files outside cwd	`false`
`--coverage-monocart`	Use monocart for accurate line counts	`false`
`--merge-lcov <path>`	Merge with external LCOV file (e.g., vitest coverage)	-

Code Coverage

Experimental: Coverage features are experimental. Line counts may not perfectly match other tools like vitest, especially without the --monocart flag. Use --monocart for best accuracy when merging coverage reports from multiple sources.

Collect code coverage from subprocess execution using the --coverage flag:

# Basic coverage (node_modules excluded by default)
tryscript run --coverage tests/

# Custom output directory
tryscript run --coverage --coverage-dir my-coverage tests/

# Custom reporters
tryscript run --coverage --coverage-reporter text --coverage-reporter lcov tests/

# Exclude additional patterns
tryscript run --coverage --coverage-exclude '**/vendor/**' tests/

# Include node_modules in coverage (not recommended)
tryscript run --coverage --no-coverage-exclude-node-modules tests/

Coverage uses c8 and NODE_V8_COVERAGE to track code executed by spawned CLI processes.

Required dependencies:

# Basic coverage
npm install -D c8

# For --monocart flag (recommended for merging with vitest)
npm install -D c8 monocart-coverage-reports

Default Behavior

By default, tryscript coverage:

Excludes node_modules - Your reports show only your code, not dependencies
Includes all source files - Files with 0% coverage are shown (use --coverage-skip-full to hide 100% covered files)
Uses dist/ include pattern - Tracks your built CLI output

Merging Coverage from Multiple Sources

The coverage command merges V8 coverage from multiple CLI commands into a single report:

# Merge coverage from multiple CLI test commands
tryscript coverage "tryscript run tests/cli/" "node dist/bin.mjs --help"

# With monocart for accurate line counts
tryscript coverage --monocart "tryscript run tests/"

Important: Vitest Incompatibility

The tryscript coverage command uses NODE_V8_COVERAGE to collect coverage data from subprocesses. However, vitest does not use NODE_V8_COVERAGE - it controls the V8 profiler directly via node:inspector (see vitest PR #2786).

This means tryscript coverage "vitest run" ... will NOT collect coverage from vitest tests. The coverage command will warn you if a command produces no new coverage files.

Merging Vitest + Tryscript Coverage

Use the built-in --merge-lcov flag to combine vitest and tryscript coverage in one step:

# Step 1: Run vitest with its own coverage (generates coverage/lcov.info)
vitest run --coverage

# Step 2: Run tryscript with coverage, merging vitest's LCOV file
tryscript run --coverage --merge-lcov coverage/lcov.info tests/

The --merge-lcov flag:

Automatically adds the lcov reporter if not already specified
Merges the external LCOV file with tryscript's generated coverage
Outputs the combined lcov.info and coverage-summary.json for badge generation

Alternative: Manual LCOV Merging

If you need more control, you can merge LCOV files manually:

# Step 1: Run vitest with its own coverage
vitest run --coverage

# Step 2: Run tryscript with coverage
tryscript run --coverage --coverage-reporter lcov tests/

# Step 3: Merge the LCOV files using lcov or a merge tool
lcov -a coverage/lcov.info -a coverage-tryscript/lcov.info -o coverage-merged/lcov.info

Or use tools like nyc merge, istanbul-merge, or custom scripts to combine LCOV/JSON coverage.

Coverage Command Options

Option	Description	Default
`--reports-dir <dir>`	Output directory	`coverage`
`--reporters <list>`	Comma-separated reporters	`text,json,json-summary,lcov,html`
`--include <patterns>`	Patterns to include	`dist/**`
`--exclude <patterns>`	Patterns to exclude	none
`--exclude-node-modules`	Exclude node_modules	`true`
`--no-exclude-node-modules`	Include node_modules	-
`--exclude-after-remap`	Post-sourcemap exclude	`false`
`--skip-full`	Hide 100% files	`false`
`--allow-external`	Allow external files	`false`
`--monocart`	AST-aware line counts	`false`
`--src <dir>`	Source dir for mapping	`src`
`--verbose`	Show coverage after each command	`false`

How It Works

The coverage command:

Creates a shared temporary directory for V8 coverage data
Sets NODE_V8_COVERAGE environment variable
Runs each command in sequence (all inherit the coverage env)
Shows coverage file statistics after each command (warns if none produced)
Generates a merged coverage report using c8

Debugging Coverage Issues

Use --verbose to see intermediate coverage tables after each command:

tryscript coverage --verbose "cmd1" "cmd2"

This helps identify which commands are contributing coverage and which are not.

Why Monocart?

The --monocart flag uses monocart-coverage-reports for AST-aware line counting, producing line counts ~90% aligned with vitest. Without this flag, standard c8 may inflate line counts by 3-4x, making merged coverage percentages inaccurate.

Metric	Standard c8	With --monocart	Vitest
Total lines	~1700 (inflated)	~460	~510
Accuracy	❌	✅ ~90% match	✅ baseline

Sourcemap Requirement

Important: Coverage reports map back to source files only if your build generates sourcemaps. Without sourcemaps, reports show bundled filenames instead of source paths:

Build Configuration	Coverage Report Shows
Sourcemaps disabled	`cli-BXvEEW6O.mjs` (34% coverage)
Sourcemaps enabled	`src/cli/commands/status.ts` (83% coverage)

Enable sourcemaps in your build tool:

tsdown / tsup:

// tsdown.config.ts or tsup.config.ts
export default defineConfig({
  sourcemap: true,
  // ... other options
});

esbuild:

await esbuild.build({
  sourcemap: true,
  // ... other options
});

rollup:

// rollup.config.js
export default {
  output: {
    sourcemap: true,
  },
};

Vite:

// vite.config.ts
export default defineConfig({
  build: {
    sourcemap: true,
  },
});

After enabling sourcemaps, rebuild your project before running coverage.

Configuration

Configure coverage in tryscript.config.ts:

import { defineConfig } from 'tryscript';

export default defineConfig({
  coverage: {
    reportsDir: 'coverage-tryscript',
    reporters: ['text', 'html'],
    include: ['dist/**'],
    exclude: [],                  // Additional exclude patterns
    excludeNodeModules: true,     // Exclude node_modules (recommended)
    excludeAfterRemap: false,     // Apply exclude after sourcemap remap
    skipFull: false,              // Hide 100% covered files
    allowExternal: false,         // Allow files outside cwd
    src: 'src',
    monocart: false,              // Use monocart for vitest-compatible line counts
  },
});

Config Option	CLI Flag	Description
`reportsDir`	`--coverage-dir`	Output directory
`reporters`	`--coverage-reporter`	Reporter list
`include`	-	Include patterns (config only)
`exclude`	`--coverage-exclude`	Exclude patterns
`excludeNodeModules`	`--coverage-exclude-node-modules`	Exclude node_modules
`excludeAfterRemap`	`--coverage-exclude-after-remap`	Post-sourcemap exclude
`skipFull`	`--coverage-skip-full`	Hide 100% files
`allowExternal`	`--coverage-allow-external`	Allow external files
`src`	-	Source dir for mapping (config only)
`monocart`	`--coverage-monocart`	AST-aware line counts
`mergeLcov`	`--merge-lcov`	Merge with external LCOV file

Wildcard Expansion

The --expand flags replace wildcard placeholders in your test files with actual output from a test run. This is a surgical operation -- only targeted wildcards are replaced; the rest of the file is left intact.

Expansion Workflow

Write a test with unknown wildcards as temporary placeholders:

$ my-cli status
[??]
? 0

Run with --expand to fill in actual output:

tryscript run --expand tests/my-test.tryscript.md

Review the expanded output and commit.

Expansion Flags

The three flags form a hierarchy (each includes the previous):

Flag	Expands
`--expand`	Unknown wildcards only (`???`, `[??]`)
`--expand-generic`	Unknown + generic (`...`, `[..]`)
`--expand-all`	All wildcards including named patterns

These flags are mutually exclusive with each other and with --update.

Capture Log

Use --capture-log <path> to write a YAML sidecar file recording what each wildcard matched during a test run. This is useful for debugging pattern matches and reviewing captured values.

tryscript run --capture-log captures.yaml tests/

Best Practices

DO: Use shell features directly

$ echo "hello" | tr 'a-z' 'A-Z'
HELLO

$ cat file.txt 2>/dev/null || echo "not found"
not found

DO: Use env for CLI paths

env:
  BIN: ./dist/cli.mjs

$ $BIN --version
1.0.0

DO: Use sandbox for file operations

sandbox: true

$ echo "test" > output.txt
$ cat output.txt
test

DON'T: Use patterns in commands

# ❌ WRONG: Patterns are for output matching only
$ cat [CWD]/file.txt

DON'T: Rely on exact timestamps or paths

# ❌ WRONG: Exact match will fail
$ date
Mon Jan 3 12:34:56 UTC 2026

# ✓ RIGHT: Use elision
$ date
[..]

Config File

For project-wide settings, create tryscript.config.ts:

import { defineConfig } from 'tryscript';

export default defineConfig({
  env: { NO_COLOR: '1' },
  timeout: 30000,
  patterns: {
    VERSION: '\\d+\\.\\d+\\.\\d+',
    UUID: '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}',
  },
  // CLI testing configuration
  path: ['./dist'],       // Directories to add to PATH
});

Execution Model

Test File → Parse YAML + Blocks → Create Execution Context
                                         │
                    ┌────────────────────┴────────────────────┐
                    │                                         │
              sandbox: false                            sandbox: true
              cwd = testDir/config.cwd                  cwd = /tmp/tryscript-xxx/
                    │                                         │
                    └────────────────────┬────────────────────┘
                                         │
                    spawn(command, { shell: true, cwd, env })
                                         │
                    Capture stdout + stderr → Match against expected

Key points:

Commands run in a real shell (shell: true)
Shell handles all variable expansion ($VAR)
Patterns ([..], [CWD]) only apply to output matching
Sandbox creates isolated temp directory per test file

FilesExpand file tree

tryscript-reference.md

Latest commit

History

tryscript-reference.md

File metadata and controls

tryscript Reference

Overview

Quick Start Example

Test File Structure

Command Block Syntax

Examples

Elision Patterns

Named Patterns

Unknown Wildcards

Generic Wildcards

Pattern Examples

Wildcard Best Practices

Configuration (Frontmatter)

Config Options Reference

Sandbox Mode

Sandbox with Fixtures

Environment Variables

Built-in Environment Variables

Testing CLI Applications

path: Custom Binary Directories

Using node_modules/.bin (npm/pnpm/bun)

Test: Use any installed dev dependency

Test Annotations

Complete Example

CLI Usage

Run Options

Coverage Options

Code Coverage

Default Behavior

Merging Coverage from Multiple Sources

Merging Vitest + Tryscript Coverage

Coverage Command Options

How It Works

Debugging Coverage Issues

Why Monocart?

Sourcemap Requirement

Configuration

Wildcard Expansion

Expansion Workflow

Expansion Flags

Capture Log

Best Practices

DO: Use shell features directly

DO: Use env for CLI paths

DO: Use sandbox for file operations

DON'T: Use patterns in commands

DON'T: Rely on exact timestamps or paths

Config File

Execution Model