Commands

Run harness scripts from the directory containing run.py. An editable package installation adds the view and integrate entry points. Installing the package does not add a run command.

Running an experimentCopy Anchor Link

python run.py task=examples/computer_export agent=openai/gpt-5.6-luna

run.py uses Hydra rather than argparse. Configuration overrides use key=value. -m or --multirun creates multiple jobs. --cfg job --resolve prints the composed job configuration without executing an episode. --help lists available configuration groups and overrides.

See Configuration for the supported overrides and their meanings.

With Hydra 1.3.2, you can also use these options:

All argparse commands listed below also accept --help or -h to print their usage.

An existing nonempty output directory is renamed to the next NAME.previous-N before the new run writes artifacts.

Viewing saved tracesCopy Anchor Link

view [run_dir] [--host HOST] [--port PORT] [--no-browser]

The module form is python -m harness.trace_viewer with the same arguments. Stop the server with Ctrl+C.

Inspecting saved episodesCopy Anchor Link

python inspect_run.py [run_dir] [--json]

run_dir defaults to runs/run. --json prints a JSON array of episode summaries. Without it, the script prints a text summary. The directory must contain an episode manifest or episode manifests one level below it.

Exporting a replay viewCopy Anchor Link

python replay_run.py [run_dir] [--step STEP] [--before] [--output PATH] [--no-progress]

Use this command to export a view of the saved run. To resume execution, you need a separate restoration step.

Generating an integrationCopy Anchor Link

integrate new NAME [--kind KIND] [--root PATH] [--url URL] [--endpoint URL]

Generate a task YAML and, for a custom integration, a Python driver module. Use a new name; existing files cannot be overwritten.

Checking an integrationCopy Anchor Link

integrate check TASK [--config-root PATH] [--seed N] [--report PATH]

With integrate check, you execute the actions in task.actions against the selected environment, which may connect to an external service. We validate the observations and state without making model requests. You can also use these subcommands through python -m harness.integration.

Generating case configurationsCopy Anchor Link

python scripts/generate_experiments.py [--products PATH_OR_URL] [--limit N] [--task TASK] [--exp-dir PATH]

--cases is an alias for --products. The default input is the upstream ABxLab tasks/product_pairs-matched-ratings.csv. --limit defaults to no limit and must be positive when supplied. --task defaults to abxlab/abx_social_proof. --exp-dir defaults to conf/experiment/generated/abxlab.

Generate one YAML override per case. Input IDs must be unique and valid for filenames. Existing destinations cause an assertion.

Checking documentation sourcesCopy Anchor Link

python scripts/check_documentation.py [--docs-root PATH]

--docs-root defaults to website/content/docs in the checkout. Use this command to check source paths, page navigation, internal links, Python snippet syntax, and reference coverage without importing the harness.

python scripts/check_documentation_examples.py [--work-dir PATH]

Use this command to check local environment operations, a generated driver, factorial output, a case sweep, replay export, and output archival in a temporary copy of the harness. Install the dev,browser extras and Chromium first. You can specify a new directory with --work-dir; otherwise, we create a temporary one. You can find its location in the terminal and the results in documentation-validation.json. We do not execute the LLM tutorials, make model requests, or connect to external environments during these checks.

Native desktop utilitiesCopy Anchor Link

The native setup and state commands run inside the configured Ubuntu desktop. The integration invokes them; they are not host setup commands.

During setup, we recreate the task's marked working directory on the selected desktop. Use the native task configuration to specify that desktop. For a report, provide existing native runs with the expected saved fields.