Comparing two runs¶

A single run is an anecdote. The platform exists to compare runs that differ in one knob, which is what this notebook does: two runs, one changed field, the same numbers read off both.

What this notebook cannot show is a behavioural finding. The agents are scripted, so the knobs that change how agents behave move nothing here: the harness replaces the wall-clock check with one that never fires, which makes max_round_duration_seconds structurally inert, and channel noise and the postmortem switch have nothing to act on when the script is fixed.

So the knob is round_count, which does move real numbers, and the point of the notebook is the shape of the comparison rather than the result: two runs, one changed field, the same metrics read off both through the same code path.

In [1]:

import logging
import os
import tempfile
from pathlib import Path

from pytest import MonkeyPatch

# These notebooks generate their own run, so nothing here reaches a provider.
# Clearing the keys makes that a fact rather than a claim: if any cell below
# tried to call a model, it would fail here rather than spend.
for _key in ("ANTHROPIC_API_KEY", "OPENAI_API_KEY", "HF_TOKEN"):
    os.environ.pop(_key, None)

# The platform and the MCP server each log a line per tool call at INFO, which
# buries a notebook's own output in several hundred lines of it.
for _logger_name in ("glossogen", "mcp"):
    logging.getLogger(_logger_name).setLevel(logging.WARNING)

SCENARIO = "warehouse_robot_recovery"
PRESET = "knobs_default"

import logging import os import tempfile from pathlib import Path from pytest import MonkeyPatch # These notebooks generate their own run, so nothing here reaches a provider. # Clearing the keys makes that a fact rather than a claim: if any cell below # tried to call a model, it would fail here rather than spend. for _key in ("ANTHROPIC_API_KEY", "OPENAI_API_KEY", "HF_TOKEN"): os.environ.pop(_key, None) # The platform and the MCP server each log a line per tool call at INFO, which # buries a notebook's own output in several hundred lines of it. for _logger_name in ("glossogen", "mcp"): logging.getLogger(_logger_name).setLevel(logging.WARNING) SCENARIO = "warehouse_robot_recovery" PRESET = "knobs_default"

In [2]:

from glossogen.testing import run_rounds


async def generate_run(round_count, overrides):
    """Run the real round loop with the model replaced by a script.

    Every part of the platform is real here: the MCP server, the tool dispatch,
    the game clock, the world and the event logger. Only the LLM is scripted, so
    this costs nothing and needs no key. Each send sits behind a round gate, so
    every message lands in the same round with the same text on every execution;
    only the incidental cycle count varies with scheduling.
    """
    with MonkeyPatch.context() as patch:
        return await run_rounds(
            scenario_name=SCENARIO,
            preset_name=PRESET,
            round_count=round_count,
            overrides=overrides,
            tmp_path=Path(tempfile.mkdtemp()),
            monkeypatch=patch,
        )

from glossogen.testing import run_rounds async def generate_run(round_count, overrides): """Run the real round loop with the model replaced by a script. Every part of the platform is real here: the MCP server, the tool dispatch, the game clock, the world and the event logger. Only the LLM is scripted, so this costs nothing and needs no key. Each send sits behind a round gate, so every message lands in the same round with the same text on every execution; only the incidental cycle count varies with scheduling. """ with MonkeyPatch.context() as patch: return await run_rounds( scenario_name=SCENARIO, preset_name=PRESET, round_count=round_count, overrides=overrides, tmp_path=Path(tempfile.mkdtemp()), monkeypatch=patch, )

Two runs, one difference¶

Everything else is held: the same preset, the same scenario, the same scripted agents.

In [3]:

CONDITIONS = {"short": 2, "long": 5}

runs = {}
for label, rounds in CONDITIONS.items():
    runs[label] = await generate_run(round_count=rounds, overrides={})
    print(f"{label:6} round_count={rounds}  {len(runs[label].events):5} events")

CONDITIONS = {"short": 2, "long": 5} runs = {} for label, rounds in CONDITIONS.items(): runs[label] = await generate_run(round_count=rounds, overrides={}) print(f"{label:6} round_count={rounds} {len(runs[label].events):5} events")

genai-prices has no price for model 'scripted::floor_associate' under provider <Provider.ANTHROPIC: 'anthropic'>; its tokens are not costed
Traceback (most recent call last):
  File "/home/runner/work/GlossoGen/GlossoGen/src/glossogen/token_pricing.py", line 62, in _find_catalog_model
    calculation = calc_price(
        usage=Usage(input_tokens=0, output_tokens=0),
        model_ref=model,
        provider_id=provider,
    )
  File "/home/runner/work/GlossoGen/GlossoGen/.venv/lib/python3.14/site-packages/genai_prices/__init__.py", line 62, in calc_price
    return data_snapshot.get_snapshot().calc(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        usage, model_ref, provider_id, provider_api_url, genai_request_timestamp, price_context
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/runner/work/GlossoGen/GlossoGen/.venv/lib/python3.14/site-packages/genai_prices/data_snapshot.py", line 68, in calc
    provider, model = self.find_provider_model(model_ref, None, provider_id, provider_api_url)
                      ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/GlossoGen/GlossoGen/.venv/lib/python3.14/site-packages/genai_prices/data_snapshot.py", line 128, in find_provider_model
    raise LookupError(f'Unable to find model with {model_ref=!r} in {provider.id}')
LookupError: Unable to find model with model_ref='scripted::floor_associate' in anthropic

genai-prices has no price for model 'scripted::robotics_engineer' under provider <Provider.ANTHROPIC: 'anthropic'>; its tokens are not costed
Traceback (most recent call last):
  File "/home/runner/work/GlossoGen/GlossoGen/src/glossogen/token_pricing.py", line 62, in _find_catalog_model
    calculation = calc_price(
        usage=Usage(input_tokens=0, output_tokens=0),
        model_ref=model,
        provider_id=provider,
    )
  File "/home/runner/work/GlossoGen/GlossoGen/.venv/lib/python3.14/site-packages/genai_prices/__init__.py", line 62, in calc_price
    return data_snapshot.get_snapshot().calc(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        usage, model_ref, provider_id, provider_api_url, genai_request_timestamp, price_context
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/runner/work/GlossoGen/GlossoGen/.venv/lib/python3.14/site-packages/genai_prices/data_snapshot.py", line 68, in calc
    provider, model = self.find_provider_model(model_ref, None, provider_id, provider_api_url)
                      ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/GlossoGen/GlossoGen/.venv/lib/python3.14/site-packages/genai_prices/data_snapshot.py", line 128, in find_provider_model
    raise LookupError(f'Unable to find model with {model_ref=!r} in {provider.id}')
LookupError: Unable to find model with model_ref='scripted::robotics_engineer' in anthropic

genai-prices has no price for model 'scripted::fleet_safety_coordinator' under provider <Provider.ANTHROPIC: 'anthropic'>; its tokens are not costed
Traceback (most recent call last):
  File "/home/runner/work/GlossoGen/GlossoGen/src/glossogen/token_pricing.py", line 62, in _find_catalog_model
    calculation = calc_price(
        usage=Usage(input_tokens=0, output_tokens=0),
        model_ref=model,
        provider_id=provider,
    )
  File "/home/runner/work/GlossoGen/GlossoGen/.venv/lib/python3.14/site-packages/genai_prices/__init__.py", line 62, in calc_price
    return data_snapshot.get_snapshot().calc(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        usage, model_ref, provider_id, provider_api_url, genai_request_timestamp, price_context
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/runner/work/GlossoGen/GlossoGen/.venv/lib/python3.14/site-packages/genai_prices/data_snapshot.py", line 68, in calc
    provider, model = self.find_provider_model(model_ref, None, provider_id, provider_api_url)
                      ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/GlossoGen/GlossoGen/.venv/lib/python3.14/site-packages/genai_prices/data_snapshot.py", line 128, in find_provider_model
    raise LookupError(f'Unable to find model with {model_ref=!r} in {provider.id}')
LookupError: Unable to find model with model_ref='scripted::fleet_safety_coordinator' in anthropic

short  round_count=2    263 events

long   round_count=5    614 events

Score both the same way¶

The same metric list over both runs. Reading them through the real evaluation runner rather than by hand is what keeps this comparable to what glossogen evaluate reports.

In [4]:

from glossogen.evaluation.metric_core.metric_run_options import MetricRunOptions
from glossogen.testing import MetricRun, score_metrics

METRICS = [
    "round_success",
    "mean_chars_per_round",
    "mean_chars_per_message",
    "round_ended_idle",
    "round_ended_timeout",
]


async def score(simulation, label):
    """Score one run and return its measurements keyed by metric name."""
    run_dir = simulation.log_path.parent
    run = MetricRun(
        scenario=simulation.scenario,
        run_dir=run_dir,
        log_path=simulation.log_path,
        simulation=simulation,
    )
    with MonkeyPatch.context() as patch:
        scored = await score_metrics(
            run=run,
            metric_names=METRICS,
            judge_responses=[],
            options=MetricRunOptions(probe_round=None, probe_replicas=1, ontology_path=None),
            report_path=run_dir / f"report_{label}.json",
            monkeypatch=patch,
        )
    return {m.metric_name: m.score for m in scored.report.measurements}


scores = {}
for label, simulation in runs.items():
    scores[label] = await score(simulation, label)

from glossogen.evaluation.metric_core.metric_run_options import MetricRunOptions from glossogen.testing import MetricRun, score_metrics METRICS = [ "round_success", "mean_chars_per_round", "mean_chars_per_message", "round_ended_idle", "round_ended_timeout", ] async def score(simulation, label): """Score one run and return its measurements keyed by metric name.""" run_dir = simulation.log_path.parent run = MetricRun( scenario=simulation.scenario, run_dir=run_dir, log_path=simulation.log_path, simulation=simulation, ) with MonkeyPatch.context() as patch: scored = await score_metrics( run=run, metric_names=METRICS, judge_responses=[], options=MetricRunOptions(probe_round=None, probe_replicas=1, ontology_path=None), report_path=run_dir / f"report_{label}.json", monkeypatch=patch, ) return {m.metric_name: m.score for m in scored.report.measurements} scores = {} for label, simulation in runs.items(): scores[label] = await score(simulation, label)

In [5]:

import pandas as pd

table = pd.DataFrame(scores).T
table.index.name = "condition"
table

import pandas as pd table = pd.DataFrame(scores).T table.index.name = "condition" table

Out[5]:

	round_success	mean_chars_per_round	mean_chars_per_message	round_ended_idle	round_ended_timeout
condition
short	0.0	86.0	28.666667	2.0	0.0
long	0.0	86.0	28.666667	5.0	0.0

Read the difference, and check it is the one you asked for¶

Before reading a difference, check the knob did what you meant. Here that is visible in the run itself: the long condition should record more rounds than the short one. On a real experiment the same discipline applies to whichever knob you moved, and round_ended_idle against round_ended_timeout is the pair that says whether a throughput number is measuring the agents or the time limit.

In [6]:

for metric in ("round_ended_idle", "mean_chars_per_round", "mean_chars_per_message"):
    short, long = table.loc["short", metric], table.loc["long", metric]
    print(f"{metric:24} short={short:8.2f}  long={long:8.2f}  delta={long - short:+8.2f}")

for metric in ("round_ended_idle", "mean_chars_per_round", "mean_chars_per_message"): short, long = table.loc["short", metric], table.loc["long", metric] print(f"{metric:24} short={short:8.2f} long={long:8.2f} delta={long - short:+8.2f}")

round_ended_idle         short=    2.00  long=    5.00  delta=   +3.00
mean_chars_per_round     short=   86.00  long=   86.00  delta=   +0.00
mean_chars_per_message   short=   28.67  long=   28.67  delta=   +0.00

In [7]:

import matplotlib.pyplot as plt

shown = ["mean_chars_per_round", "mean_chars_per_message"]
figure, axis = plt.subplots(figsize=(6, 3))
positions = range(len(shown))
width = 0.38
axis.bar([p - width / 2 for p in positions], [table.loc["short", m] for m in shown],
         width, label="2 rounds", color="#0e6b5c")
axis.bar([p + width / 2 for p in positions], [table.loc["long", m] for m in shown],
         width, label="5 rounds", color="#8d550e")
axis.set_xticks(list(positions))
axis.set_xticklabels(shown, fontsize=8)
axis.set_ylabel("characters")
axis.set_title("Throughput at two round counts")
axis.legend()
figure.tight_layout()

import matplotlib.pyplot as plt shown = ["mean_chars_per_round", "mean_chars_per_message"] figure, axis = plt.subplots(figsize=(6, 3)) positions = range(len(shown)) width = 0.38 axis.bar([p - width / 2 for p in positions], [table.loc["short", m] for m in shown], width, label="2 rounds", color="#0e6b5c") axis.bar([p + width / 2 for p in positions], [table.loc["long", m] for m in shown], width, label="5 rounds", color="#8d550e") axis.set_xticks(list(positions)) axis.set_xticklabels(shown, fontsize=8) axis.set_ylabel("characters") axis.set_title("Throughput at two round counts") axis.legend() figure.tight_layout()

What this is not¶

A mechanism check, not a result. Two runs with scripted agents differ only in what the platform did with a fixed script, and the quickstart's three identical runs scoring 0/3, 1/3 and 0/3 is what the same comparison looks like against a real model: a real comparison needs replications at a fixed seed, because the spread is often wider than the effect.

seed=42 is the convention here for exactly that reason: it fixes the case set, so repeated runs measure model variance on an identical workload rather than a different workload each time.

For the real thing, see Running simulations on sweeps, and Evaluation on which metrics answer which question.