Results

Read the Results Carefully

The static snapshot above covers 217 programs and reports a legacy composite direction-normalized ratio of 1.32x. This is not a measured language advantage. Comprehension is 1.84x, error detection is 1.49x, token economics is 1.42x, and information density is 0.97x—the category whose normalized ratio favors C#. Seven category ratios favor Calor and one favors C#. The Calor parser accepted 217 of 217 inputs; the Roslyn syntax parser accepted 217 of 217 C# inputs. These are parse checks, not evidence that the pairs compute the same result.

The snapshot was recorded on 2026-09-15 from source revision c2a8816d. Its Directory.Build.props declares compiler 0.21.0, before the release version bump; the JSON's version: "1.0" identifies the data format, not the compiler release. The corpus is tests/TestData/Benchmarks. The exact compiler binary used for the measurement was not independently recorded. These are fixed-source results, rerun for the 0.22.0 release. They are not a live measure of coding-agent performance. The separate agent-study snapshots below were not rerun.

These measurements need important caveats:

• Source pairs are not all behaviorally equivalent. For example, the C# CsvParser implements parsing, while its Calor counterpart contains only helper functions. The 0.19 rerun fixes an invalid newline literal in the C# fixture, but does not establish equal work across the corpus. The ratios therefore cannot establish a language advantage. See the benchmark integrity follow-up.
• The metrics are deterministic static analyses. The exporter reports both zero-width intervals from singleton category aggregates and pooled intervals from repeated program scores. Thirty repetitions are not independent samples. Neither interval establishes sampling uncertainty over the corpus or a measured language, agent-productivity, correctness, or safety advantage.
• These C#-versus-Calor micro-benchmarks are not the v0.12 release gates. PP-A1 covered adoption readiness, while PP-W5 measured toolchain tax and concluded only that no large tax was detected—not that parity was proven.

Token economics is a composite of token, character, and line ratios. Calor still pays a § token premium on small programs even though the current composite favors Calor.

Separate Agent Studies

The static dashboard does not measure effect rows or live coding-agent behavior. Those questions use separate, pre-registered agent studies. The research evidence status explains why the proposed follow-on adopter study stopped unadjudicated.

Redesigned PP-W-rows: paused, not complete

Current status, 2026-09-11: paused by the user. The central pause handoff records two consumed invalid/censored attempts and 442 untouched scheduled identities out of 444. Neither the pilot nor confirmation has completed. These interrupted attempts do not supply an effect-row benefit, negative, or null result. They must not be erased, replaced, or described as an entirely unrun redesigned study.

The single total experiment ceiling is USD1,000. Two reservations of USD25.52 each remain held: USD51.04 total. Actual provider charges are unknown; the held amount is not measured spend or reconciled charges. The conditionally authorized permanent-retention exception has not been applied. Its implementation and final reviews remain unfinished in the accounting issue and paused draft checkpoint.

There is no permission to resume. A new explicit instruction and the remaining implementation, review, and operator-readiness gates are required. This product release neither resumes collection nor changes the protocol, consumed attempts, unknown costs, or existing ceiling. Confirmation would still need its own prospective design and affordability decision; it is not automatic.

Earlier decision, 2026-09-10: USD250 total, pilot only (superseded ceiling). The recorded user decision and historical authorization receipt accepted the registered stopping rules and publication of negative or null results under a conservative pilot-only interpretation. Its dated BUDGET_NOT_RUN assessment described an earlier pre-execution hold, not the current pause after consumed attempts. Historical planning estimates were not cost lower bounds. Neither that hold nor today's pause is a pilot null or the conditional stage-2 UNDERPOWERED-CARRIED outcome.

Historical M0 disposition, 2026-09-09: unchanged. The M0 decision records formal M0 as UNADJUDICATED, with a separate maintainer administrative stop. It deferred this narrower proposal pending a separate explicit decision. Later PP-W financial decisions do not re-arm M0, supply its missing human or participant approvals, or establish that effect rows help, fail, or cannot be studied.

Local buildability passed on 2026-09-10. The reviewed gate closure records Exit A under the frozen v0.18 release criteria. This deterministic engineering check is not an agent observation. Subsequent engineering work does not turn the two interrupted attempts into a completed study or benefit result.

The proposed methodology explains the old design defect: the unwanted behavior was already visibly forbidden, so avoiding it did not require effect enforcement. The redesigned question concerns violations hidden behind an abstraction while visible tests pass. Historical zero escapes do not answer that question.

The historical benefit ledger records the old registration as UNDERPOWERED, epochRun: false, and legA/legB: null. Null means uncollected for that historical registration, not zero violations. It is not the record of the two consumed redesigned attempts. This publication changes neither ledger nor scientific gate.

Later agent-result publication must follow actual adjudication by the scientific owners. The current pause is not a completed pilot, unaffordable confirmation verdict, or confirmatory result.

Historical PP-E1: effect-row toolchain cost

The 2026-08-27 PP-E1 report records two distinct legs, not one pooled sample:

• A seeded-fixture check caught 10 of 10 registered mutations at compiler revision 121c6681, with frozen unmutated diagnostic baselines reproduced. These are hand-seeded cases, not agent trials or a population detection rate.
• Four ordinary programming tasks ran five times on each of two compiler configurations, for 40 valid green runs, with no dropped task pairs or censored runs in the archived analysis. The median paired output-token ratio was 1.1835. Its one-sided 95% two-level cluster-bootstrap lower bound was 0.9012, below the registered 1.0 bound gate, and the point estimate stayed below the 1.35 failure threshold.

The epoch pins record e1-rows-parity-001, started 2026-08-27, claude-opus-4-8, Claude Code 2.1.243, and the v0.14.3 control at 63316987 versus the 0.15.0 release-build treatment at b775acb4. These are different compiler versions, not the redesigned one-compiler policy contrast.

The combined registered verdict was HIT. The tasks measured compiler feedback cost on ordinary code; because they contained no callbacks, they did not measure the safety benefit of effect rows.

Historical PP-W-rows: practice rounds, not confirmation

The 2026-09-01 practice report describes historical dry runs used for sizing, not a confirmatory verdict. Both used claude-opus-4-8 with permissive v0.14.3-pre-rows control 283ec9f9 versus strict v0.15.0 treatment 3bb2601e.

The second dry-run analysis reports a descriptive output-tokens-to-green ratio of 1.1175, with a one-sided 95% cluster-bootstrap lower bound of 0.8634, over its two cost-eligible task pairs (12 runs). It is not a result over all 18 runs or a measurement of general development effort.

Its old-design sizing required nine runs per task/arm, while the retired ceiling afforded three, with calculated power 0.48 under that registration. These are historical sizing assumptions, not the redesigned sample size or funding. The benefit ledger therefore remains UNDERPOWERED: the claim is neither supported nor refuted. The redesigned defect analysis also means buying more of the old tasks would not isolate the intended benefit.

The ledger must be read with care. Its top-level achievablePower field is 0.868, but the recorded reason and power curve use 0.868 for the smallest unaffordable design, nine runs per cell. The largest affordable design was three runs per cell at 0.48 power. The public interpretation therefore does not present 0.868 as affordable power.

The practice result exposed a fixture-design defect: the undesirable hidden effect was already forbidden by the visible task specification. The 2026-09-08 redesign separates visible requirements from a held-out effect-observing test and adds eight task-qualification rules. At the 2026-09-09 M0 closeout, no candidate had established both R1 (a convenient effect-hiding API) and R4 (the laundering solution passes visible tests). The local buildability gate subsequently passed on 2026-09-10, as recorded above. The two later consumed attempts and the user-directed pause are separate from these practice results.

On 2026-09-09, M0 recorded these missing prerequisites and the lack of a suitable isolated evaluator. Formal M0 status is UNADJUDICATED; the maintainer action was an administrative stop. At that closeout, no redesigned pilot or confirmatory collection was authorized. The old w-rows-001 epoch remains unrun, not a null result; this does not erase the later redesigned attempts.

Keep these statements separate:

• The 1.32x dashboard is a deterministic source/structure comparison.
• The 0.15 proof point did not detect a large toolchain tax.
• The historical PP-W-rows registration did not run a confirmatory epoch.
• The redesigned study is paused after two invalid/censored attempts, with 442 untouched identities; neither stage has completed and no benefit or null result exists.
• The redesign and M0 records contain no protected three-arm adopter comparison, independent methods adjudication, or safety/economic advantage result.

See the artifact inventory, the proposed redesign, and the M0 evidence status for source links and execution states.

When to Use Calor

These scores are not sufficient evidence for choosing a language. Consider Calor when you want to experiment with:

• Explicit contracts and the documented limits of optional static verification
• Explicit IDs and structural markers for agent-editing experiments
• Structured source representations, while measuring costs on your own tasks

Use C# when:

• Ecosystem libraries are needed — Leverage existing tooling
• Human readability is priority — Familiar syntax for human developers

Methodology

The current static runner is automated and can regenerate a new artifact from the checked-out source. That is not the same as exactly reproducing every historical artifact: several omit the measurement binary, working-tree state, source revision, model, or runtime configuration. Proposed protocols have not run.

The published dashboard framework:

1. Compiles paired Calor and C# programs.
2. Applies deterministic metric calculators to source, syntax trees, and compilation results.
3. Aggregates per-program direction-normalized ratios. Values above 1 favor Calor, including lower-is-better metrics whose raw score ratios are inverted.
4. Repeats the deterministic evaluation 30 times for the reported statistical artifact. Greater than 1.0 favors Calor and less than 1.0 favors C#; neither is a measured language advantage.

LLM-based task runners exist separately and are not folded into this dashboard result.

See Methodology for full details on the evaluation framework.

Running Benchmarks

To regenerate benchmark results:

# Run the evaluation framework
dotnet run --project tests/Calor.Evaluation -c Release -- run -f website -o website/public/data/benchmark-results.json --statistical --runs 30

# Start the website to view results
cd website && npm run dev

Results are automatically loaded from benchmark-results.json and displayed in the dashboard above.