Comparison Methodology

Kairn will publish named comparative results only after every arm passes the same trace-integrity, provider, model, tool, and verification controls.

No winner is assumed in advance.

Controlled four-arm comparison

The controlled experiment uses one instrumented Anthropic Messages agent loop:

1. Vanilla
2. ReasonBlocks
3. Kairn
4. Kairn + ReasonBlocks

All arms use the same pinned model, temperature, prompts, repository tools, context policy, step limit, timeout, and final verifier. ReasonBlocks model routing is disabled so it cannot change the base model.

The ReasonBlocks arms use its official Anthropic-native steering lifecycle, trajectory monitors, E-traces, and within-session codebase memory. Each attempt starts with an empty unique memory ID. Prepopulated memory, language-specific import-graph advantages, and LangChain-only compression or early exit are excluded from the claim.

The combined arm always delivers Kairn repository evidence before ReasonBlocks trajectory guidance.

Privacy boundary

ReasonBlocks-enabled runs require a ReasonBlocks API key plus an explicit --allow-external-trajectory flag and use public benchmark repositories only. They send trajectory information to ReasonBlocks' hosted service under its published trust disclosure.

This is not part of normal Kairn operation. Vanilla and Kairn-only runs do not send repository evidence or agent trajectories to Kairn.

Instrumented repository tools

Every controlled arm receives the same tools:

• list_files
• search_repo
• read_file
• apply_patch
• run_verification

Uninstrumented repository reads invalidate an attempt.

Measures

The primary measures are:

• time and model tokens before required production source is first hinted and first read
• required-source content and explicit-read recall
• explicit noise and other extraneous files opened
• unnecessary evidence tokens on preregistered abstention turns
• repeated unchanged source reads across turns
• cached input, uncached input, output, tool-result, repository-content, and total signed token deltas
• final verification, required touches, source invariants, excluded edits, and diff scope

Token and timing improvements count only on quality-passing pairs. Never-reached source is reported as a miss rather than removed from timing summaries. Regressions remain signed and visible.

Task matrix

The sealed publication holdout crosses Python, TypeScript, Go, Java, and C++ with source localization, failure-first debugging, and persistent multi-turn work.

The planned publication campaign contains two repository-disjoint tasks per language/scenario cell, five repeats, and four arms: 600 attempts after smoke and pilot gates pass.

Existing development fixtures may validate the harness but cannot serve as the unseen publication set.

Ecological track

A separate track compares off-the-shelf Codex against Codex + Kairn through hooks and MCP interfaces from the frozen source checkout. It tests whether Kairn helps an agent loop Kairn does not control. Its results are not merged with the controlled four-arm experiment.

Publication rules

The site will publish the frozen manifest, arm definitions, versions, task digest, denominators, misses, exclusions, infrastructure failures, sanitized attempt rows, uncertainty, and negative outcomes. ReasonBlocks will not be described as closed source, expensive, or universally difficult to configure, and no named outcome will appear before all arms pass protocol checks.

The site build recomputes every complete-campaign aggregate, stratum, paired count, median, outcome, metric effect, exclusion, and checksum from the embedded attempt rows. A mismatch stops the build instead of rendering the claim.

A complete campaign remains current for 180 days after its latest embedded attempt ended. Its generated timestamp must be at or after that attempt. When the window expires, the campaign must be rerun or explicitly moved to legacy evidence before the site can publish again. Methodology-only and partial artifacts never expose comparative results and are exempt from this age window.