Analyzing an export¶

The runs page's Export button and glossogen export write the same CSV tables: one row per run, per round, per agent, per message, plus each round's briefings. This notebook reads a committed example of each table and turns them into plots, so it needs no server, no runs directory and no key.

The example data under notebooks/data/ is a real cohort: veyru's baseline_oss sweep, two open-weights models crossed with five round_time_budget_seconds values and the postmortem switch, two replicas per cell, 15 rounds each. Two export invocations produced the two folders, exactly as the CLI wrote them:

# data/veyru_baseline_oss/: run, round and agent tables for the 40-run cohort
glossogen export --runs-dir <runs> --out <out> \
  --run-id veyru/<timestamp> ... \
  --frames run_level,round_level,agent_level --no-repeat-run-columns

# data/veyru_baseline_oss_messages/: message and briefing tables for two of those runs
glossogen export --runs-dir <runs> --out <out> \
  --run-id veyru/1781086390 --run-id veyru/1780571883 \
  --frames message_level,round_context --no-repeat-run-columns

--no-repeat-run-columns keeps the long tables narrow; this notebook joins them back on run_id, which is the workflow that flag exists for. See Exporting runs for every column family.

In [1]:

from pathlib import Path

import matplotlib.pyplot as plt
import pandas as pd

COHORT = Path("data/veyru_baseline_oss")
MESSAGES = Path("data/veyru_baseline_oss_messages")

# The dark ink and recessive grid keep the data as the strongest thing on the
# figure; the two series hues are a colorblind-validated pair.
INK = "#0b0b0b"
INK_SOFT = "#52514e"
GRID = "#e5e4e0"
MODEL_HUES = {
    "meta-llama/Llama-3.3-70B-Instruct": "#2a78d6",
    "Qwen/Qwen3-32B": "#eb6834",
}
SHORT = {
    "meta-llama/Llama-3.3-70B-Instruct": "Llama 3.3 70B",
    "Qwen/Qwen3-32B": "Qwen3 32B",
}

plt.rcParams.update(
    {
        "axes.spines.top": False,
        "axes.spines.right": False,
        "axes.edgecolor": INK_SOFT,
        "axes.labelcolor": INK_SOFT,
        "axes.titlecolor": INK,
        "xtick.color": INK_SOFT,
        "ytick.color": INK_SOFT,
        "axes.grid": True,
        "axes.grid.axis": "y",
        "grid.color": GRID,
        "grid.linewidth": 0.8,
        "axes.axisbelow": True,
        "figure.dpi": 110,
    }
)

from pathlib import Path import matplotlib.pyplot as plt import pandas as pd COHORT = Path("data/veyru_baseline_oss") MESSAGES = Path("data/veyru_baseline_oss_messages") # The dark ink and recessive grid keep the data as the strongest thing on the # figure; the two series hues are a colorblind-validated pair. INK = "#0b0b0b" INK_SOFT = "#52514e" GRID = "#e5e4e0" MODEL_HUES = { "meta-llama/Llama-3.3-70B-Instruct": "#2a78d6", "Qwen/Qwen3-32B": "#eb6834", } SHORT = { "meta-llama/Llama-3.3-70B-Instruct": "Llama 3.3 70B", "Qwen/Qwen3-32B": "Qwen3 32B", } plt.rcParams.update( { "axes.spines.top": False, "axes.spines.right": False, "axes.edgecolor": INK_SOFT, "axes.labelcolor": INK_SOFT, "axes.titlecolor": INK, "xtick.color": INK_SOFT, "ytick.color": INK_SOFT, "axes.grid": True, "axes.grid.axis": "y", "grid.color": GRID, "grid.linewidth": 0.8, "axes.axisbelow": True, "figure.dpi": 110, } )

Read the legend before the data¶

A multi-table export ships columns.csv: each column's family, its unit, and how many of the selected runs filled it. The coverage count is what disambiguates a blank cell, because a knob a scenario never declared and a metric that never ran both render empty.

In [2]:

legend = pd.read_csv(COHORT / "columns.csv")
print(legend.groupby("group").size().to_string())
legend[legend["runs_with_value"] < legend["run_count"]]

legend = pd.read_csv(COHORT / "columns.csv") print(legend.groupby("group").size().to_string()) legend[legend["runs_with_value"] < legend["run_count"]]

group
agent_identity     6
agent_level        4
identity           2
knob              28
label              1
lineage            1
metric            22
round_level        1
run_metadata      14

Out[2]:

	column	group	unit	runs_with_value	run_count
7	derivation_type	lineage	NaN	0.0	40.0
14	knob.easy_round_numbers	knob	NaN	39.0	40.0
16	knob.intern_join_round	knob	NaN	0.0	40.0
17	knob.intern_takeover_round	knob	NaN	0.0	40.0
21	knob.model_overrides	knob	NaN	1.0	40.0
22	knob.model_overrides.field_observer.model	knob	NaN	39.0	40.0
23	knob.model_overrides.field_observer.provider	knob	NaN	39.0	40.0
24	knob.model_overrides.stabilization_engineer.model	knob	NaN	39.0	40.0
25	knob.model_overrides.stabilization_engineer.pr...	knob	NaN	39.0	40.0
35	knob.scheduled_events	knob	NaN	39.0	40.0
37	knob.swap_round	knob	NaN	0.0	40.0
58	metric.language_emergence	metric	rounds with novel language patterns (out of 15)	1.0	40.0
61	metric.mean_chars_per_round	metric	chars/round	23.0	40.0
62	metric.mean_message_length	metric	words/message	1.0	40.0
63	metric.mean_word_length	metric	chars/word	1.0	40.0
67	metric.postmortem_ended_timeout	metric	postmortem phases ended via timeout (out of 15)	20.0	40.0
68	metric.protocol_probe	metric	probe responses	5.0	40.0

run_level.csv: one row per run, wide in metrics¶

Every table is wide in metrics: a metric is a column (metric.round_success), never a value in a metric_name column, so a row is already a design-matrix row. The knob columns come from what each run recorded and the metric columns from what its report carries, not from any list this repository maintains: this cohort was evaluated by a research checkout whose registry differs from this one, and columns like metric.language_emergence came along anyway.

In [3]:

runs = pd.read_csv(COHORT / "run_level.csv")
runs["model"] = runs["agent_model.field_observer"]
runs["budget"] = runs["knob.round_time_budget_seconds"].astype(int)
runs["postmortem"] = runs["knob.postmortem_enabled"]

# The committed cohort is two replicas per (model, budget, postmortem) cell.
cell_sizes = runs.groupby(["model", "budget", "postmortem"]).size()
assert len(runs) == 40, f"expected 40 runs, got {len(runs)}"
assert (cell_sizes == 2).all(), f"uneven cells:\n{cell_sizes[cell_sizes != 2]}"

runs[["run_id", "model", "budget", "postmortem", "metric.round_success",
      "metric.mean_chars_per_message", "total_cost_usd"]].head(8)

runs = pd.read_csv(COHORT / "run_level.csv") runs["model"] = runs["agent_model.field_observer"] runs["budget"] = runs["knob.round_time_budget_seconds"].astype(int) runs["postmortem"] = runs["knob.postmortem_enabled"] # The committed cohort is two replicas per (model, budget, postmortem) cell. cell_sizes = runs.groupby(["model", "budget", "postmortem"]).size() assert len(runs) == 40, f"expected 40 runs, got {len(runs)}" assert (cell_sizes == 2).all(), f"uneven cells:\n{cell_sizes[cell_sizes != 2]}" runs[["run_id", "model", "budget", "postmortem", "metric.round_success", "metric.mean_chars_per_message", "total_cost_usd"]].head(8)

Out[3]:

	run_id	model	budget	postmortem	metric.round_success	metric.mean_chars_per_message	total_cost_usd
0	veyru/1777920823	Qwen/Qwen3-32B	800	True	0.133333	54.142857	0.0
1	veyru/1780571879	meta-llama/Llama-3.3-70B-Instruct	800	True	0.133333	143.725275	0.0
2	veyru/1780571883	meta-llama/Llama-3.3-70B-Instruct	2000	True	0.666667	147.000000	0.0
3	veyru/1780571887	Qwen/Qwen3-32B	150	True	0.000000	48.400000	0.0
4	veyru/1780572066	Qwen/Qwen3-32B	150	True	0.000000	84.617647	0.0
5	veyru/1780576077	Qwen/Qwen3-32B	250	True	0.000000	51.235294	0.0
6	veyru/1780576803	Qwen/Qwen3-32B	250	True	0.000000	58.294118	0.0
7	veyru/1780578775	Qwen/Qwen3-32B	450	True	0.000000	50.903226	0.0

Budget against success¶

metric.round_success is the fraction of the run's 15 rounds the judge scored as stabilized. One line per model over the budget sweep, with every replica shown, because at this replica count the spread is part of the answer.

In [4]:

budgets = sorted(runs["budget"].unique())
positions = range(len(budgets))

figure, axis = plt.subplots(figsize=(7.0, 3.4))
for model, hue in MODEL_HUES.items():
    mine = runs[runs["model"] == model]
    means = mine.groupby("budget")["metric.round_success"].mean().reindex(budgets)
    for position, budget in zip(positions, budgets):
        replicas = mine.loc[mine["budget"] == budget, "metric.round_success"]
        axis.scatter([position] * len(replicas), replicas, s=34, color=hue,
                     alpha=0.35, edgecolor="white", linewidth=0.8, zorder=2)
    axis.plot(positions, means.to_numpy(), color=hue, linewidth=2, marker="o",
              markersize=7, markeredgecolor="white", zorder=3, label=SHORT[model])
    axis.annotate(SHORT[model], (len(budgets) - 1, means.iloc[-1]),
                  xytext=(10, 0), textcoords="offset points",
                  va="center", color=INK_SOFT, fontsize=9)

axis.set_xticks(list(positions), [str(b) for b in budgets])
axis.set_xlabel("round_time_budget_seconds")
axis.set_ylabel("round_success")
axis.set_ylim(-0.04, 1.0)
axis.set_xlim(-0.3, len(budgets) - 0.25)
axis.set_title("Rounds won per run across the budget sweep")
axis.legend(frameon=False, loc="upper left")
figure.tight_layout()

budgets = sorted(runs["budget"].unique()) positions = range(len(budgets)) figure, axis = plt.subplots(figsize=(7.0, 3.4)) for model, hue in MODEL_HUES.items(): mine = runs[runs["model"] == model] means = mine.groupby("budget")["metric.round_success"].mean().reindex(budgets) for position, budget in zip(positions, budgets): replicas = mine.loc[mine["budget"] == budget, "metric.round_success"] axis.scatter([position] * len(replicas), replicas, s=34, color=hue, alpha=0.35, edgecolor="white", linewidth=0.8, zorder=2) axis.plot(positions, means.to_numpy(), color=hue, linewidth=2, marker="o", markersize=7, markeredgecolor="white", zorder=3, label=SHORT[model]) axis.annotate(SHORT[model], (len(budgets) - 1, means.iloc[-1]), xytext=(10, 0), textcoords="offset points", va="center", color=INK_SOFT, fontsize=9) axis.set_xticks(list(positions), [str(b) for b in budgets]) axis.set_xlabel("round_time_budget_seconds") axis.set_ylabel("round_success") axis.set_ylim(-0.04, 1.0) axis.set_xlim(-0.3, len(budgets) - 0.25) axis.set_title("Rounds won per run across the budget sweep") axis.legend(frameon=False, loc="upper left") figure.tight_layout()

Both models sit near zero until the budget crosses several hundred characters, then Llama pulls away while Qwen barely moves. A mean alone would hide how far apart the 2000-budget Llama replicas are, which is why the dots stay on the plot.

round_level.csv joins back on run_id¶

One row per run and round. Exported with --no-repeat-run-columns, so the run's knobs live only on run_level.csv and the join brings them over.

The blank cells here are the empty-versus-zero rule in action: metric.round_success is 0.0 on a lost round, a real observation, while metric.language_strangeness is blank on every round where the judge flagged nothing. Filling those blanks with zeros would bias any mean over the column.

In [5]:

rounds = pd.read_csv(COHORT / "round_level.csv")
assert len(rounds) == 40 * 15, f"expected one row per run and round, got {len(rounds)}"

rounds = rounds.merge(runs[["run_id", "model", "budget", "postmortem"]], on="run_id")

flagged = rounds["metric.language_strangeness"].notna().sum()
lost = (rounds["metric.round_success"] == 0.0).sum()
print(f"rounds with a strangeness flag: {flagged} of {len(rounds)} (the rest are blank, not zero)")
print(f"rounds scored 0.0 on round_success: {lost} (a counted loss, not a blank)")

rounds = pd.read_csv(COHORT / "round_level.csv") assert len(rounds) == 40 * 15, f"expected one row per run and round, got {len(rounds)}" rounds = rounds.merge(runs[["run_id", "model", "budget", "postmortem"]], on="run_id") flagged = rounds["metric.language_strangeness"].notna().sum() lost = (rounds["metric.round_success"] == 0.0).sum() print(f"rounds with a strangeness flag: {flagged} of {len(rounds)} (the rest are blank, not zero)") print(f"rounds scored 0.0 on round_success: {lost} (a counted loss, not a blank)")

rounds with a strangeness flag: 233 of 600 (the rest are blank, not zero)
rounds scored 0.0 on round_success: 543 (a counted loss, not a blank)

In [6]:

# One hue, light to dark: the cell value is a magnitude, the share of Llama
# replicas that won that round at that budget.
from matplotlib.colors import LinearSegmentedColormap

RAMP = LinearSegmentedColormap.from_list(
    "sequential_blue",
    ["#cde2fb", "#9ec5f4", "#6da7ec", "#3987e5", "#256abf", "#1c5cab", "#104281", "#0d366b"],
)

llama = rounds[rounds["model"] == "meta-llama/Llama-3.3-70B-Instruct"]
share = llama.pivot_table(index="budget", columns="round_number",
                          values="metric.round_success", aggfunc="mean")

figure, axis = plt.subplots(figsize=(7.6, 2.9))
mesh = axis.pcolormesh(share.columns, range(len(share.index)), share.to_numpy(),
                       cmap=RAMP, vmin=0.0, vmax=1.0,
                       edgecolors="white", linewidth=1.5)
axis.set_yticks(range(len(share.index)), [str(b) for b in share.index])
axis.set_xticks(list(share.columns))
axis.set_xlabel("round")
axis.set_ylabel("budget")
axis.set_title("Llama 3.3 70B: share of replicas winning each round")
axis.grid(False)
colorbar = figure.colorbar(mesh, ax=axis, label="share of replicas")
colorbar.outline.set_visible(False)
figure.tight_layout()

# One hue, light to dark: the cell value is a magnitude, the share of Llama # replicas that won that round at that budget. from matplotlib.colors import LinearSegmentedColormap RAMP = LinearSegmentedColormap.from_list( "sequential_blue", ["#cde2fb", "#9ec5f4", "#6da7ec", "#3987e5", "#256abf", "#1c5cab", "#104281", "#0d366b"], ) llama = rounds[rounds["model"] == "meta-llama/Llama-3.3-70B-Instruct"] share = llama.pivot_table(index="budget", columns="round_number", values="metric.round_success", aggfunc="mean") figure, axis = plt.subplots(figsize=(7.6, 2.9)) mesh = axis.pcolormesh(share.columns, range(len(share.index)), share.to_numpy(), cmap=RAMP, vmin=0.0, vmax=1.0, edgecolors="white", linewidth=1.5) axis.set_yticks(range(len(share.index)), [str(b) for b in share.index]) axis.set_xticks(list(share.columns)) axis.set_xlabel("round") axis.set_ylabel("budget") axis.set_title("Llama 3.3 70B: share of replicas winning each round") axis.grid(False) colorbar = figure.colorbar(mesh, ax=axis, label="share of replicas") colorbar.outline.set_visible(False) figure.tight_layout()

The wins live in the top row and cluster in specific rounds rather than spreading evenly: at this budget some cases are within reach and most are not. Per-round rows are what make that readable at all; the run-level score collapses it to one number.

agent_level.csv: the roster¶

Keyed on the run's registered agents, not on what the metrics reported, so it answers who ran under which model even for a metric-free run.

In [7]:

agents = pd.read_csv(COHORT / "agent_level.csv")
assert len(agents) == 40 * 2, "veyru registers two agents per run"

agents[["agent_id", "agent_role", "agent_model"]].drop_duplicates().reset_index(drop=True)

agents = pd.read_csv(COHORT / "agent_level.csv") assert len(agents) == 40 * 2, "veyru registers two agents per run" agents[["agent_id", "agent_role", "agent_model"]].drop_duplicates().reset_index(drop=True)

Out[7]:

	agent_id	agent_role	agent_model
0	field_observer	Field Observer	Qwen/Qwen3-32B
1	stabilization_engineer	Stabilization Engineer	Qwen/Qwen3-32B
2	field_observer	Field Observer	meta-llama/Llama-3.3-70B-Instruct
3	stabilization_engineer	Stabilization Engineer	meta-llama/Llama-3.3-70B-Instruct

message_level.csv: a message beside its own numbers¶

The second export folder holds every channel message of two Llama runs from the same cohort, budgets 150 and 2000. This table is the distribution the run-level means summarize. It reads event logs, which is why it is opt-in rather than in the default set.

Every channel is exported; is_primary_channel marks the ones the throughput metrics read. And repetition_factor is blank on every row because the language_repetition metric never ran on these runs: an empty cell means no number exists, not zero.

In [8]:

messages = pd.read_csv(MESSAGES / "message_level.csv")
assert messages["repetition_factor"].isna().all(), "these runs never ran language_repetition"

link = messages[messages["is_primary_channel"]].copy()
link["budget"] = link["run_id"].map({"veyru/1781086390": 150, "veyru/1780571883": 2000})
print(link.groupby("budget")["chars"].describe().round(1).to_string())
link[["round_number", "sender_agent_id", "chars", "character_entropy_bits", "text"]].head(4)

messages = pd.read_csv(MESSAGES / "message_level.csv") assert messages["repetition_factor"].isna().all(), "these runs never ran language_repetition" link = messages[messages["is_primary_channel"]].copy() link["budget"] = link["run_id"].map({"veyru/1781086390": 150, "veyru/1780571883": 2000}) print(link.groupby("budget")["chars"].describe().round(1).to_string()) link[["round_number", "sender_agent_id", "chars", "character_entropy_bits", "text"]].head(4)

        count   mean   std   min   25%    50%    75%    max
budget                                                     
150      30.0  133.5  63.9  31.0  95.0  142.0  184.5  254.0
2000    122.0  147.0  60.2  41.0  81.8  155.5  185.0  290.0

Out[8]:

	round_number	sender_agent_id	chars	character_entropy_bits	text
0	1	field_observer	179	4.222060	The Veyru is dim overall, all faces are faint....
1	1	stabilization_engineer	126	4.194817	To stabilize Veyru 1, place a warm stone besid...
2	1	field_observer	67	4.238306	The stabilization was ineffective. Please prov...
3	1	stabilization_engineer	173	4.092369	To stabilize Veyru 1, at each corner of the ba...

In [9]:

# Two steps of one blue ramp: the two conditions are ordered budgets, so an
# ordinal ramp encodes them rather than two unrelated hues.
BUDGET_HUES = {150: "#86b6ef", 2000: "#184f95"}

figure, axis = plt.subplots(figsize=(7.0, 3.2))
bins = range(0, int(link["chars"].max()) + 40, 40)
for budget, hue in BUDGET_HUES.items():
    sample = link.loc[link["budget"] == budget, "chars"]
    axis.hist(sample, bins=bins, histtype="step", linewidth=2, color=hue,
              label=f"budget {budget} ({len(sample)} messages)")
axis.set_xlabel("characters per link message")
axis.set_ylabel("messages")
axis.set_title("The tight budget does not shorten Llama's messages")
axis.legend(frameon=False)
figure.tight_layout()

# Two steps of one blue ramp: the two conditions are ordered budgets, so an # ordinal ramp encodes them rather than two unrelated hues. BUDGET_HUES = {150: "#86b6ef", 2000: "#184f95"} figure, axis = plt.subplots(figsize=(7.0, 3.2)) bins = range(0, int(link["chars"].max()) + 40, 40) for budget, hue in BUDGET_HUES.items(): sample = link.loc[link["budget"] == budget, "chars"] axis.hist(sample, bins=bins, histtype="step", linewidth=2, color=hue, label=f"budget {budget} ({len(sample)} messages)") axis.set_xlabel("characters per link message") axis.set_ylabel("messages") axis.set_title("The tight budget does not shorten Llama's messages") axis.legend(frameon=False) figure.tight_layout()

Under a 150-character round budget the messages are not shorter, there are simply far fewer of them before the budget is gone, which is one way to lose every round. The run-level means barely register it: 134 against 147 characters per message under a 13x budget change. The distribution is where the behaviour is visible.

round_context.csv: what the agent knew going in¶

One row per run and round, one column per agent's briefing. Joined to the message table on run_id and round_number, it puts the question next to the answer: the case the observer was handed, and what it then said on the link.

In [10]:

import textwrap

context = pd.read_csv(MESSAGES / "round_context.csv")
assert len(context) == 2 * 15, f"expected one row per run and round, got {len(context)}"

run_id, round_number = "veyru/1780571883", 3
briefing = context.set_index(["run_id", "round_number"]).loc[
    (run_id, round_number), "injection.field_observer"
]
print(f"--- briefing to field_observer, {run_id} round {round_number} ---")
print(textwrap.fill(briefing[:600], width=88))
print()
print("--- what the link then carried ---")
said = link[(link["run_id"] == run_id) & (link["round_number"] == round_number)]
for message in said.itertuples():
    print(f"[{message.sender_agent_id}] {message.text[:200]}")

import textwrap context = pd.read_csv(MESSAGES / "round_context.csv") assert len(context) == 2 * 15, f"expected one row per run and round, got {len(context)}" run_id, round_number = "veyru/1780571883", 3 briefing = context.set_index(["run_id", "round_number"]).loc[ (run_id, round_number), "injection.field_observer" ] print(f"--- briefing to field_observer, {run_id} round {round_number} ---") print(textwrap.fill(briefing[:600], width=88)) print() print("--- what the link then carried ---") said = link[(link["run_id"] == run_id) & (link["round_number"] == round_number)] for message in said.itertuples(): print(f"[{message.sender_agent_id}] {message.text[:200]}")

--- briefing to field_observer, veyru/1780571883 round 3 ---
--- PREVIOUS VEYRU RESULT --- Veyru 2: STABILIZED   --- NEW VEYRU --- Veyru 3 Time
budget: 2000s (one character = one second; this is the total for both of you)  What you
observe: A Veyru in a corner of the room. Edges appear to wobble or flex when touched.
Faces look slightly curved or bulging, the box shape is subtly distorted. The hum sounds
muffled as if underwater.

--- what the link then carried ---
[field_observer] The Veyru has edges that appear to wobble or flex when touched. Faces look slightly curved or bulging, the box shape is subtly distorted. The hum sounds muffled as if underwater.
[stabilization_engineer] To stabilize Veyru 3, rotate the Veyru slowly over 20 seconds while sounding a steady tone at firm volume near opposite faces, starting from the back face. After rotation, chime a bell once near each 
[stabilization_engineer] Please describe the new symptoms of Veyru 3.
[field_observer] The stabilization was successful, but new symptoms have appeared. Faces are flickering randomly between light and dark patches that form no pattern. The hum is broken and irregular, starting and stopp
[stabilization_engineer] To stabilize Veyru 3, place a warm stone beside the back face at firm warmth for 20 seconds. Rotate and repeat for each face.

Where to go from here¶

• The same tables come from the runs page's Export button, as a zip with the same columns.csv legend.
• glossogen analyze aggregates without the detour through CSV when the question is one group-by away; see Analysis.
• Exporting runs documents every column family, the empty-versus-zero rule, and the knob filter grammar used to select cohorts like this one.