Scoring a run¶

Metrics are generic. A scenario opts into each one by implementing the hook that metric reads, so a measurement written once works on every scenario, including one installed from another package.

This notebook scores a run through run_scenario_evaluation, the same function glossogen evaluate calls. Nothing is reimplemented for the notebook, so what you see here is what the CLI would write.

In [1]:

import logging
import os
import tempfile
from pathlib import Path

from pytest import MonkeyPatch

# These notebooks generate their own run, so nothing here reaches a provider.
# Clearing the keys makes that a fact rather than a claim: if any cell below
# tried to call a model, it would fail here rather than spend.
for _key in ("ANTHROPIC_API_KEY", "OPENAI_API_KEY", "HF_TOKEN"):
    os.environ.pop(_key, None)

# The platform and the MCP server each log a line per tool call at INFO, which
# buries a notebook's own output in several hundred lines of it.
for _logger_name in ("glossogen", "mcp"):
    logging.getLogger(_logger_name).setLevel(logging.WARNING)

SCENARIO = "warehouse_robot_recovery"
PRESET = "knobs_default"

import logging import os import tempfile from pathlib import Path from pytest import MonkeyPatch # These notebooks generate their own run, so nothing here reaches a provider. # Clearing the keys makes that a fact rather than a claim: if any cell below # tried to call a model, it would fail here rather than spend. for _key in ("ANTHROPIC_API_KEY", "OPENAI_API_KEY", "HF_TOKEN"): os.environ.pop(_key, None) # The platform and the MCP server each log a line per tool call at INFO, which # buries a notebook's own output in several hundred lines of it. for _logger_name in ("glossogen", "mcp"): logging.getLogger(_logger_name).setLevel(logging.WARNING) SCENARIO = "warehouse_robot_recovery" PRESET = "knobs_default"

In [2]:

from glossogen.testing import run_rounds


async def generate_run(round_count, overrides):
    """Run the real round loop with the model replaced by a script.

    Every part of the platform is real here: the MCP server, the tool dispatch,
    the game clock, the world and the event logger. Only the LLM is scripted, so
    this costs nothing and needs no key. Each send sits behind a round gate, so
    every message lands in the same round with the same text on every execution;
    only the incidental cycle count varies with scheduling.
    """
    with MonkeyPatch.context() as patch:
        return await run_rounds(
            scenario_name=SCENARIO,
            preset_name=PRESET,
            round_count=round_count,
            overrides=overrides,
            tmp_path=Path(tempfile.mkdtemp()),
            monkeypatch=patch,
        )


result = await generate_run(round_count=4, overrides={})
print(f"{len(result.events)} events at {result.log_path}")

from glossogen.testing import run_rounds async def generate_run(round_count, overrides): """Run the real round loop with the model replaced by a script. Every part of the platform is real here: the MCP server, the tool dispatch, the game clock, the world and the event logger. Only the LLM is scripted, so this costs nothing and needs no key. Each send sits behind a round gate, so every message lands in the same round with the same text on every execution; only the incidental cycle count varies with scheduling. """ with MonkeyPatch.context() as patch: return await run_rounds( scenario_name=SCENARIO, preset_name=PRESET, round_count=round_count, overrides=overrides, tmp_path=Path(tempfile.mkdtemp()), monkeypatch=patch, ) result = await generate_run(round_count=4, overrides={}) print(f"{len(result.events)} events at {result.log_path}")

genai-prices has no price for model 'scripted::floor_associate' under provider <Provider.ANTHROPIC: 'anthropic'>; its tokens are not costed
Traceback (most recent call last):
  File "/home/runner/work/GlossoGen/GlossoGen/src/glossogen/token_pricing.py", line 62, in _find_catalog_model
    calculation = calc_price(
        usage=Usage(input_tokens=0, output_tokens=0),
        model_ref=model,
        provider_id=provider,
    )
  File "/home/runner/work/GlossoGen/GlossoGen/.venv/lib/python3.14/site-packages/genai_prices/__init__.py", line 62, in calc_price
    return data_snapshot.get_snapshot().calc(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        usage, model_ref, provider_id, provider_api_url, genai_request_timestamp, price_context
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/runner/work/GlossoGen/GlossoGen/.venv/lib/python3.14/site-packages/genai_prices/data_snapshot.py", line 68, in calc
    provider, model = self.find_provider_model(model_ref, None, provider_id, provider_api_url)
                      ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/GlossoGen/GlossoGen/.venv/lib/python3.14/site-packages/genai_prices/data_snapshot.py", line 128, in find_provider_model
    raise LookupError(f'Unable to find model with {model_ref=!r} in {provider.id}')
LookupError: Unable to find model with model_ref='scripted::floor_associate' in anthropic

genai-prices has no price for model 'scripted::robotics_engineer' under provider <Provider.ANTHROPIC: 'anthropic'>; its tokens are not costed
Traceback (most recent call last):
  File "/home/runner/work/GlossoGen/GlossoGen/src/glossogen/token_pricing.py", line 62, in _find_catalog_model
    calculation = calc_price(
        usage=Usage(input_tokens=0, output_tokens=0),
        model_ref=model,
        provider_id=provider,
    )
  File "/home/runner/work/GlossoGen/GlossoGen/.venv/lib/python3.14/site-packages/genai_prices/__init__.py", line 62, in calc_price
    return data_snapshot.get_snapshot().calc(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        usage, model_ref, provider_id, provider_api_url, genai_request_timestamp, price_context
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/runner/work/GlossoGen/GlossoGen/.venv/lib/python3.14/site-packages/genai_prices/data_snapshot.py", line 68, in calc
    provider, model = self.find_provider_model(model_ref, None, provider_id, provider_api_url)
                      ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/GlossoGen/GlossoGen/.venv/lib/python3.14/site-packages/genai_prices/data_snapshot.py", line 128, in find_provider_model
    raise LookupError(f'Unable to find model with {model_ref=!r} in {provider.id}')
LookupError: Unable to find model with model_ref='scripted::robotics_engineer' in anthropic

genai-prices has no price for model 'scripted::fleet_safety_coordinator' under provider <Provider.ANTHROPIC: 'anthropic'>; its tokens are not costed
Traceback (most recent call last):
  File "/home/runner/work/GlossoGen/GlossoGen/src/glossogen/token_pricing.py", line 62, in _find_catalog_model
    calculation = calc_price(
        usage=Usage(input_tokens=0, output_tokens=0),
        model_ref=model,
        provider_id=provider,
    )
  File "/home/runner/work/GlossoGen/GlossoGen/.venv/lib/python3.14/site-packages/genai_prices/__init__.py", line 62, in calc_price
    return data_snapshot.get_snapshot().calc(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        usage, model_ref, provider_id, provider_api_url, genai_request_timestamp, price_context
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/runner/work/GlossoGen/GlossoGen/.venv/lib/python3.14/site-packages/genai_prices/data_snapshot.py", line 68, in calc
    provider, model = self.find_provider_model(model_ref, None, provider_id, provider_api_url)
                      ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/GlossoGen/GlossoGen/.venv/lib/python3.14/site-packages/genai_prices/data_snapshot.py", line 128, in find_provider_model
    raise LookupError(f'Unable to find model with {model_ref=!r} in {provider.id}')
LookupError: Unable to find model with model_ref='scripted::fleet_safety_coordinator' in anthropic

497 events at /tmp/tmp2nj7al85/smoke.jsonl

The deterministic metrics cost nothing¶

These read the event log and compute. No judge, no provider, no spend, and the same answer every time.

In [3]:

from glossogen.evaluation.metric_core.metric_run_options import MetricRunOptions
from glossogen.testing import MetricRun, score_metrics

DETERMINISTIC = [
    "round_success",
    "mean_chars_per_round",
    "mean_chars_per_message",
    "round_ended_idle",
    "round_ended_timeout",
]

run_dir = result.log_path.parent
run = MetricRun(
    scenario=result.scenario,
    run_dir=run_dir,
    log_path=result.log_path,
    simulation=result,
)

with MonkeyPatch.context() as patch:
    scored = await score_metrics(
        run=run,
        metric_names=DETERMINISTIC,
        judge_responses=[],
        options=MetricRunOptions(probe_round=None, probe_replicas=1, ontology_path=None),
        report_path=run_dir / "report_deterministic.json",
        monkeypatch=patch,
    )

for measurement in scored.report.measurements:
    print(f"{measurement.metric_name:24} {measurement.score:>8.2f}  {measurement.score_unit}")

from glossogen.evaluation.metric_core.metric_run_options import MetricRunOptions from glossogen.testing import MetricRun, score_metrics DETERMINISTIC = [ "round_success", "mean_chars_per_round", "mean_chars_per_message", "round_ended_idle", "round_ended_timeout", ] run_dir = result.log_path.parent run = MetricRun( scenario=result.scenario, run_dir=run_dir, log_path=result.log_path, simulation=result, ) with MonkeyPatch.context() as patch: scored = await score_metrics( run=run, metric_names=DETERMINISTIC, judge_responses=[], options=MetricRunOptions(probe_round=None, probe_replicas=1, ontology_path=None), report_path=run_dir / "report_deterministic.json", monkeypatch=patch, ) for measurement in scored.report.measurements: print(f"{measurement.metric_name:24} {measurement.score:>8.2f} {measurement.score_unit}")

round_success                0.00  fraction of rounds succeeded (0/4)
mean_chars_per_round        86.00  chars/round
mean_chars_per_message      28.67  chars/message
round_ended_idle             4.00  rounds ended via all_agents_idle (out of 4)
round_ended_timeout          0.00  rounds ended via round_timeout (out of 4)

judge_responses=[] is the assertion that none of these calls a model. Had one tried, the stub judge would have failed with the schema it asked for rather than inventing an answer, so a metric cannot quietly score against a fabricated judgement.

Two of those numbers are worth reading together.

In [4]:

idle = scored.measurement(metric_name="round_ended_idle")
timeout = scored.measurement(metric_name="round_ended_timeout")
print(f"ended because the agents stopped talking: {idle.score:.2f}")
print(f"ended because the clock ran out:          {timeout.score:.2f}")

idle = scored.measurement(metric_name="round_ended_idle") timeout = scored.measurement(metric_name="round_ended_timeout") print(f"ended because the agents stopped talking: {idle.score:.2f}") print(f"ended because the clock ran out: {timeout.score:.2f}")

ended because the agents stopped talking: 4.00
ended because the clock ran out:          0.00

If rounds end on the timeout, the throughput numbers are measuring the time limit rather than the agents, and mean_chars_per_round stops meaning what you think. Check this pair before believing a throughput result.

Per-round detail¶

A Measurement carries more than its headline score. per_round is what a plot reads, and per_agent is what a per-agent breakdown reads.

In [5]:

import pandas as pd

throughput = scored.measurement(metric_name="mean_chars_per_round")
frame = pd.DataFrame(
    [
        {"round": observation.round_number, "value": observation.value, "note": observation.note}
        for observation in throughput.per_round
    ]
)
frame

import pandas as pd throughput = scored.measurement(metric_name="mean_chars_per_round") frame = pd.DataFrame( [ {"round": observation.round_number, "value": observation.value, "note": observation.note} for observation in throughput.per_round ] ) frame

Out[5]:

	round	value	note
0	1	86.0	3 messages
1	2	86.0	3 messages
2	3	86.0	3 messages
3	4	86.0	3 messages

The judge-backed metrics, without spending¶

The language findings come from LLM judges. In a real evaluation those cost money; here the judge is a stub whose answers this notebook chooses, which means you can see the whole path, including what the judge is shown, without a key.

That is usually the interesting part. Not what the judge said, but what it was asked.

In [6]:

from glossogen.evaluation.metrics.shorthand_codes_metric import ShorthandCodesOutput
from glossogen.models.event import RoundResultRecorded

rounds = len(result.of_type(event_type=RoundResultRecorded))
answers = [
    ShorthandCodesOutput(
        per_round_notes=[],
        systematic=False,
        messages_shortened=False,
        shared_understanding=False,
        explanation="Scripted agents send fixed text, so there is no code to find.",
    )
    for _ in range(rounds)
]

with MonkeyPatch.context() as patch:
    judged = await score_metrics(
        run=run,
        metric_names=["shorthand_codes"],
        judge_responses=answers,
        options=MetricRunOptions(probe_round=None, probe_replicas=1, ontology_path=None),
        report_path=run_dir / "report_judged.json",
        monkeypatch=patch,
    )

print(judged.measurement(metric_name="shorthand_codes").summary)
print(f"\njudge calls made: {len(judged.judge.calls)}")

from glossogen.evaluation.metrics.shorthand_codes_metric import ShorthandCodesOutput from glossogen.models.event import RoundResultRecorded rounds = len(result.of_type(event_type=RoundResultRecorded)) answers = [ ShorthandCodesOutput( per_round_notes=[], systematic=False, messages_shortened=False, shared_understanding=False, explanation="Scripted agents send fixed text, so there is no code to find.", ) for _ in range(rounds) ] with MonkeyPatch.context() as patch: judged = await score_metrics( run=run, metric_names=["shorthand_codes"], judge_responses=answers, options=MetricRunOptions(probe_round=None, probe_replicas=1, ontology_path=None), report_path=run_dir / "report_judged.json", monkeypatch=patch, ) print(judged.measurement(metric_name="shorthand_codes").summary) print(f"\njudge calls made: {len(judged.judge.calls)}")

0/4 rounds contained shorthand codes. Scripted agents send fixed text, so there is no code to find.

judge calls made: 1

One call per round, and the metric asked each one about that round's messages alone. Inspect what a judge was actually shown:

In [7]:

call = judged.judge.calls[0]
print("schema asked for:", call.output_schema.__name__)
print()
print(call.messages[0].content[:700])

call = judged.judge.calls[0] print("schema asked for:", call.output_schema.__name__) print() print(call.messages[0].content[:700])

schema asked for: ShorthandCodesOutput

You are evaluating whether agents in a multi-agent simulation developed shorthand codes, abbreviation systems, or symbolic notation during their communication across 4 rounds.

Focus on the PRIMARY CHANNEL messages — these are the budget-constrained communications where codes and compression are most likely to emerge. Messages from OTHER CHANNELS provide context but are secondary.

IMPORTANT: Other specialized evaluators cover different phenomena. Only report what belongs to THIS evaluator:

- **Neologisms** (covered separately): invented words with new meanings, compound words, portmanteaus. If agents invent a NEW WORD (e.g., "chromaflux"), that is a neologism, NOT a code. Do NOT report it 

Next: 03_compare_runs.ipynb puts two runs side by side, which is the shape of an actual experiment.