Running simulations¶

A run is one simulation of one scenario. Each agent's tools run inside the run process, the scenario briefs each agent as a round opens, and every event lands in a JSONL log that later commands read back. A round runs until the scenario declares it settled, every agent goes idle, or the time limit passes. The game clock watches for whichever comes first.

Between rounds, most scenarios hold a postmortem: a debrief phase where the agents see the round's outcome and talk freely about what worked, on a channel whose messages cost no budget. It is where teams refine their conventions between attempts. The postmortem_enabled knob turns the phase on or off per run. The postmortem ends when the agents go idle or its own time limit passes, and the clock opens the next round.

glossogen run veyru \
  --model claude-sonnet-4-6 --provider anthropic --runs-dir ./runs \
  --config knobs_default \
  > ./runs/veyru_stdout.log 2>&1 &

From a checkout, spell each command VIRTUAL_ENV= uv run --no-sync python -m glossogen ....

Flag	Does
`--model`	Model identifier, required. The default for every agent, see per-agent models
`--provider`	One of the providers; required. Also the default for every agent. The `model_overrides` and `scheduled_events` knobs take the same values. The `judge_provider` knob and `glossogen evaluate --provider` take a judge provider: `anthropic`, `huggingface` or `openai`
`--runs-dir`	Root directory for run output. Required unless `--resume` is given, which names the directory itself
`--config`	A preset the scenario ships (`knobs_default`), or a path to a JSON file of your own; required
`--max-agent-turns`	Ceiling on agentic turns per agent (default 200)
`--resume`	Path to an existing run directory to continue
`--group-slug`	Tenant group that owns the run (default `local`)

Output goes to a timestamped directory the CLI creates: <runs-dir>/<scenario>/<unix_timestamp>/. Its layout is described under Run Storage. Run simulations in the background and pipe output to a log. The web UI is the comfortable way to watch one: the run list shows its status and the round it reached, and the run page streams messages live until it completes.

Configuration¶

The run subcommand takes a base config and trailing key=value overrides, Hydra-style. Values parse as JSON, so round_count=20 is an int, enabled=true a bool, and name=alice stays a string.

glossogen run veyru \
  --model claude-sonnet-4-6 --provider anthropic --runs-dir ./runs \
  --config knobs_intern \
  max_round_duration_seconds=120 round_count=20

Each scenario ships one or more presets next to its code (knobs_default.json and friends) and publishes a JSON Schema for its knobs. The web UI and the MCP tools read both, so a preset is discoverable without reading the source.

--config takes a preset name without the .json, which is what makes one command work from a checkout and from an installed package alike. A path wins when the argument is one, so an experiment can keep its knobs JSON outside any package. The flag is required, so every run is launched under a configuration someone chose. Each launch logs what it resolved to.

Scenario knobs from preset 'knobs_default'

Knobs can depend on each other. A scenario's knobs model may carry cross-field validators that reject an override which looks fine on its own, so when overriding one knob, override every knob the scenario validates against it. Validation happens before the run claims a directory: a rejected config leaves no run behind, and the error lands in that launch's stdout log.

Per-agent models¶

--model and --provider set the default every agent runs under. model_overrides in the config replaces the pair for the agents it names, keyed by agent_id.

{
  "round_count": 12,
  "model_overrides": {
    "stabilization_engineer": {"model": "gpt-5.4", "provider": "openai"},
    "field_observer": {"model": "claude-opus-4-6", "provider": "anthropic"}
  }
}

The CLI also accepts them as dot-notation overrides, normalized into the same field, which is how a sweep varies one seat without writing a config per cell:

glossogen run veyru \
  --model claude-sonnet-4-6 --provider anthropic --runs-dir ./runs \
  --config knobs_default \
  agents.field_observer.model=claude-opus-4-7 \
  agents.stabilization_engineer.model=gpt-5.4 \
  agents.stabilization_engineer.provider=openai

Rule	Detail
An override must name a `model`	`provider` is optional and falls back to `--provider`, so `field_observer` above stays on Anthropic. Moving an agent to another provider takes both lines
The flags stay required even when every agent is overridden	No agent runs under them, but `--provider` is still the fallback for an override that names only a model, so set both to real values rather than placeholders
The launch check reads the resolved pairs	A run whose every agent is on `openai` needs no credential for the `--provider` on the command line. It needs a key for every provider some agent resolved to, plus the scenario's judge, which is its own knob (`judge_provider`) and must name a judge provider
An override can name its own `model_settings`	That agent sends them instead of the run's `agent_model_settings`. They replace it whole: a `max_tokens` the entry does not name is 16384, not the run's. A setting one provider accepts another can refuse, such as `thinking` on a self-hosted model, so a team on several providers can give each agent the settings its provider takes. On the command line, name the model too: `agents.field_observer.model=claude-opus-4-7 agents.field_observer.model_settings.thinking=high`. In a config file, put a `model_settings` object in the entry
A fork keeps an agent's own settings	It finds them in the recorded or merged `model_overrides` entry. A replaced agent sends the `model_settings` the `--knobs` entry for its seat names, else the run's `agent_model_settings`; the source's settings for that seat stay behind, since they were for the model it replaces. That entry names the same `model` and `provider` as the command's `--model` and `--provider`, or the fork is refused
`--knobs` replaces the source's `model_overrides` whole	Like every other knob, a `model_overrides` in `--knobs` replaces the recorded map rather than merging into it. The seats it leaves out keep their source model and provider, read from their registrations, but send the run's `agent_model_settings`, so list every seat whose own settings the fork should keep
A command-line entry replaces the preset's whole	`agents.<id>.model=...` on top of a preset entry for that agent drops the preset's `provider` and `model_settings`, so name them again on the command line

Providers¶

An agent can run under any of these. Each reads the variables in the third column, from the environment or the nearest .env at or above the directory the command runs in, and the launch check refuses a run that uses a provider whose variables are unset. Hosted providers are priced from the genai-prices catalog, each agent cycle at the rates in effect when it ended; a model the catalog does not list runs, but is not costed.

`--provider`	Reaches	Variables	Weights
`anthropic`	Claude	`ANTHROPIC_API_KEY`	closed
`openai`	GPT	`OPENAI_API_KEY`	closed
`google-gla`	Gemini	`GOOGLE_API_KEY` or `GEMINI_API_KEY`	closed
`xai`	Grok	`XAI_API_KEY`	closed
`groq`	open-weight models on Groq (Llama, Qwen, gpt-oss)	`GROQ_API_KEY`	open
`mistral`	Mistral	`MISTRAL_API_KEY`	not classified
`deepseek`	DeepSeek	`DEEPSEEK_API_KEY`	open
`openrouter`	models from many vendors, named `vendor/model`	`OPENROUTER_API_KEY`	not classified
`ollama`	a local Ollama server	`OLLAMA_BASE_URL`	open
`self-hosted`	any OpenAI-compatible endpoint, see below	`SELF_HOSTED_BASE_URLS`, `SELF_HOSTED_API_KEY`	open

The weights column feeds the export's model_class. A provider serving both open- and closed-weight models is not classified, so a run with an agent under it gets an empty model_class cell rather than a guess.

A scenario's judge and the judge glossogen evaluate --provider selects run only under anthropic, huggingface or openai. The launch check refuses a run whose judge_provider names any other provider.

A model you trained yourself. Serve it behind an OpenAI-compatible endpoint (vLLM, SGLang, TGI, LM Studio) and run it as self-hosted, below. A fine-tune hosted by a provider above runs under that provider with the fine-tune's model name (--provider openai --model ft:gpt-4.1-mini:...). An openai agent is sent reasoning.effort high only when pydantic-ai marks its model as a reasoning model; a fine-tune is looked up by its base model, the second field of its ft: name.

The Provider enum, PROVIDER_SPECS, JUDGE_PROVIDERS and the hosted models the UI's pickers offer all live in model_catalog.py. SIMULATION_PROVIDERS is derived from PROVIDER_SPECS, and every provider field and --provider flag is checked against these. Adding a provider that pydantic-ai and genai-prices already support takes a value in the enum and a row in PROVIDER_SPECS, plus the pydantic-ai extra for its SDK in pyproject.toml when it needs one. The row holds its variables, its pydantic-ai prefix, whether it is billed, and its weights.

Self-hosted and local models¶

--provider self-hosted points pydantic-ai at any OpenAI-compatible chat-completions endpoint. SELF_HOSTED_BASE_URLS is a JSON map from model name to /v1 URL, so several self-hosted models coexist. SELF_HOSTED_API_KEY is the bearer token shared across them. In the .env:

SELF_HOSTED_BASE_URLS='{"meta-llama/Llama-3.3-70B-Instruct": "https://my-llama.example.com/v1", "Qwen/Qwen3-32B": "https://my-qwen.example.com/v1"}'
SELF_HOSTED_API_KEY=<the token your servers were started with>

--model names a key of that map:

glossogen run veyru \
  --model meta-llama/Llama-3.3-70B-Instruct --provider self-hosted \
  --runs-dir ./runs \
  --config knobs_default

Deploying on Modal, and running locally with Ollama or MLX, are covered in Self-hosted models. Tokens under self-hosted and ollama are priced at zero, since their GPU time is billed elsewhere.

Watch the context budget. Self-hosted models are served at a small fixed context, and input plus the agent's output cap must stay under it or the server rejects the call and the run stalls. The output cap is agent_model_settings.max_tokens (default 16384). A scenario's round judge has its own, judge_model_settings.max_tokens, which never touches an agent. For swap and resume runs, where a reconstructed history keeps growing, set "agent_model_settings": {"max_tokens": 2048} in the config.

Agent model settings¶

agent_model_settings is what every agent in a run sends with each request, unless its model_overrides entry names its own model_settings. It works for every scenario, since it lives on the knobs all of them share, and no scenario reads it. Its settings are a subset of pydantic-ai's ModelSettings, under the same names and types, so pydantic-ai's documentation says what each one does and which providers honour it. More can be added as fields of AgentModelSettings. Every setting but max_tokens (default 16384, reasoning tokens included) is unset by default, and an unset one sends nothing, leaving the provider's default in place. A scenario's own choices live in its presets, so a setting the run names wins over them like any other knob. A provider that refuses a value fails the call, and the run carries on with the error in its log.

glossogen run veyru \
  --model gpt-5.4 --provider openai --runs-dir ./runs \
  --config knobs_default \
  agent_model_settings.thinking=medium agent_model_settings.max_tokens=8192

In a config file it is one object: "agent_model_settings": {"thinking": "medium"}. A fork's --knobs replaces the source's object whole rather than merging into it, so name every setting the fork should keep. That includes the ones a scenario's presets set: signal_game's parallel_tool_calls: false is lost from a fork whose --knobs names agent_model_settings without it, and from a fork of a signal_game run recorded before the setting moved into the presets, since that run's config does not carry it.

thinking takes pydantic-ai's ThinkingLevel: minimal, low, medium, high, xhigh or max, or true (on, at the provider's default) and false (off). pydantic-ai turns the level into each provider's own parameter: OpenAI's reasoning.effort, and on Anthropic adaptive thinking with an effort, or a token budget on models without adaptive thinking. A level a provider never offers becomes the nearest one it has. A level the provider offers for some models only is sent as is: OpenAI's gpt-5.4 models refuse minimal, and every call of such a run fails.

pydantic-ai decides per model whether to send the setting at all, from the model's profile. A model it does not mark as able to think gets nothing, whatever the value: gpt-4.1, and the open-weight models served self-hosted or through Ollama, even those that reason on their own. A model that always thinks, such as claude-opus-5-5, gets no false. A run is refused before it starts when any of its agents would drop the value it names, since its log would record a setting the agents never ran with.

Every agent's registration in the event log records the settings it was sent, so glossogen export and the analysis charts can group runs by knob.agent_model_settings.thinking. A run whose model_overrides entries name their own model_settings records those under knob.model_overrides.<agent_id>.model_settings.*; the agent_model_settings column holds only the run's. A config written before the object existed spells the cap agent_max_tokens; it is still read, as max_tokens. A run logged before then recorded only its agents' cap, so its thinking level reads as unset.

The round judge's settings¶

A scenario that judges its own rounds with an LLM (veyru, orbital_anomaly and the others with judge_model / judge_provider knobs) takes a third knob beside them, judge_model_settings. It has the same fields as agent_model_settings and goes to the judge alone; the agents never see it. Unlike the agents', it sends no output cap by default, so the provider's own applies. veyru, orbital_anomaly, drive_module_repair and spot_the_difference default it to {"temperature": 0.0}, so their verdicts vary as little as the provider allows; the other judged scenarios send nothing.

glossogen run veyru ... judge_model_settings.thinking=low

Setting any judge setting replaces that default whole, on the command line as in a config file, so the run above sends thinking and no temperature. Restate judge_model_settings.temperature=0 to keep it. Anthropic refuses a temperature other than 1 alongside thinking, so don't restate it with thinking on an Anthropic judge.

The judge runs through pydantic-ai like the agents, so its calls appear in the run's Langfuse session and its spend is in the run's total_cost_usd. The judge glossogen evaluate uses is configured on that command instead, with --judge-settings (see Evaluation).

Continuing a run¶

glossogen run --resume <run-dir> continues an interrupted run inside that same directory: no new run directory is created, and new events append to the run's existing JSONL. Channel messages and scenario state are rebuilt from the log, and each agent gets its history back. Pass the --config the run was launched with: nothing compares it against what the run recorded, and a knob changed midway leaves one log produced under two configurations.

glossogen run <scenario> \
  --model <model> --provider <provider> \
  --resume ./runs/<scenario>/<timestamp> \
  --config <config> \
  > ./runs/<scenario>/<timestamp>/resume_stdout.log 2>&1 &

The clock reopens the last round the log recorded, the one that never finished, and runs to the config's round_count.

This is for a run that stopped before it was done, whether it crashed, was killed, or lost its machine. For a run that finished, resuming in place would replay its final round, which already has a verdict. Use fork-at-round instead: it keeps rounds 1..N complete, plays round N+1 onward in a fresh run directory, leaves the original untouched, and can run further than the source did. To fork a finished run with a different agent in one seat, see Agent swaps and forks.

Running many simulations¶

A sweep launches runs from a queue and keeps a fixed number live per model. Give each provider its own queue: providers rate-limit independently, and a single queue mixing models stalls every slot of one provider while the spec at its head waits for a slot on the other.

#!/bin/bash
# One queue per model, run in parallel. Each spec is "<source> <knobs>".
cd "$(git rev-parse --show-toplevel)"
LOG=/tmp/sweep.log
CAP=6

declare -a GPT_SPECS=("<source_a> <knobs.json>" "<source_b> <knobs.json>")
declare -a SONNET_SPECS=("<source_c> <knobs.json>" "<source_d> <knobs.json>")

count_running_for_model() {
  pgrep -f "Python -m glossogen run veyru --model $1" 2>/dev/null | wc -l | tr -d ' '
}

launch_one() {
  local model=$1 provider=$2 source=$3 knobs=$4
  echo "$(date) [$model] launching source=$source knobs=$knobs" >> "$LOG"
  VIRTUAL_ENV= uv run --no-sync python -m glossogen replace-agent veyru \
    --source-run-dir "runs/veyru/$source" \
    --after-round 14 --rounds-after 11 \
    --replaced-agent-id field_observer \
    --model "$model" --provider "$provider" \
    --runs-dir runs --knobs "$knobs" >> "$LOG" 2>&1
  sleep 2  # run directories are named by unix second
}

process_queue() {
  local model=$1 provider=$2; shift 2
  for spec in "$@"; do
    read -r source knobs <<< "$spec"
    while [ "$(count_running_for_model "$model")" -ge "$CAP" ]; do sleep 30; done
    launch_one "$model" "$provider" "$source" "$knobs"
  done
  echo "$(date) [$model] queue complete" >> "$LOG"
}

process_queue gpt-5.4 openai "${GPT_SPECS[@]}" &
process_queue claude-sonnet-4-6 anthropic "${SONNET_SPECS[@]}" &
wait

Start it detached so it outlives your terminal, then watch the log and the live simulations:

nohup bash /tmp/sweep.sh > /tmp/sweep.stdout 2>&1 &
tail -20 /tmp/sweep.log
pgrep -af "Python -m glossogen run veyru .* --resume"

Each replace-agent, cross-run-replace-agent and fork-at-round call prepares its run directory, spawns the simulation detached, prints new_run_id= and returns. The simulation writes its own <scenario>_stdout.log in the new directory. A fresh glossogen run does not detach, so a queue of them launches each with &.

Anchor the pgrep pattern on the interpreter's process name. Homebrew's Python on macOS runs as Python, capitalised, so Python -m glossogen run matches simulations only. A loose pattern such as glossogen run veyru also matches any shell whose command line quotes it, the counting shell included. Keep the count in a function rather than inlined into a wrapping command for the same reason.

Override a knob together with the knobs its validators check against it. A config that fails validation exits before claiming a run directory, so a launcher that polls for new directories loses the spec without an error. The reason is in that launch's own output, not in the launcher's log. In veyru, postmortem_after_swap=true requires postmortem_enabled=true, and the default preset sets the former, so a sweep with postmortem_enabled=false also passes postmortem_after_swap=false. When unsure, launch one spec in the foreground before queueing the sweep.

Recompute the queue after a launcher dies. Simulations already launched keep running, but nothing launches the rest. Subtract the specs that already have a run directory from the plan and start the launcher on what remains.

Understanding cost¶

A run spends real money against your own provider keys. An agent takes many turns per round, and every turn is an API call carrying the whole conversation so far, so cost grows faster than the round count alone suggests.

The knobs that drive spend, roughly in order of impact:

Knob	Effect
`round_count`	Rounds per run, the main multiplier
Model choice	The largest single factor. The models the pickers offer span 25× on input tokens and 20× on output, from `gpt-5.4-nano` to the Opus tier
`max_round_duration_seconds`	Ceiling on how long agents keep talking before a round is cut off
`agent_model_settings`	Per-response output cap and thinking level; see Agent model settings
Number of agents	Each one is an independent conversation
`--probe-replicas`	Evaluation only: multiplies probe calls per agent per question

What a run itself spent is on its last event: simulation_ended carries total_cost_usd, its agents' spend plus its round judge's, priced from the genai-prices catalog at the rates in effect when the run started (see token_pricing.py). A model that catalog does not know is not costed. The web UI reads it, and so does grep:

grep '"simulation_ended"' ./runs/<scenario>/<timestamp>/<scenario>.jsonl

The evaluation_cost block in the evaluation report is a different number: what the judge and probe calls of glossogen evaluate spent. A run scored with deterministic metrics only has evaluation_cost.estimated_cost_usd of 0.0 however much the simulation cost.

Nothing in the platform caps spend. Billing alerts and per-key limits with your provider are the only ceiling that holds.

Next¶

Evaluation — scoring a finished run
Agent swaps and forks — forking a run at a chosen round boundary
Scenarios — what ships in the box
Creating a scenario — writing your own