Benchmarking
The published v0.12 dashboard compares 217 paired programs across eight deterministic metrics. It reports a 1.32x overall Calor/C# composite, with Calor ahead in seven metrics and C# ahead in information density.
The Metrics
Published v0.12 Metrics
| Category | What It Measures | Why It Matters |
|---|---|---|
| Comprehension | Structural clarity, semantic extractability | Can agents understand code without deep analysis? |
| Error Detection | Bug identification, contract violation detection | Can agents find issues using explicit semantics? |
| Edit Precision | Heuristic targetability and change isolation | Do explicit IDs and markers provide stronger source targets? |
| Generation Accuracy | Compilation success, structural correctness | Can agents produce valid code? |
| Token Economics | Tokens required to represent logic | How much context window does code consume? |
| Information Density | Semantic elements per token | How much meaning per token? |
| Refactoring Stability | ID-based reference preservation | Are explicit IDs preserved by modeled transformations? |
| Correctness | Edge case handling, bug prevention | Does code produce correct results for edge cases? |
The repository also contains separate LLM and estimation-mode suites for task completion, safety, and effect discipline. Their results are not included in the v0.12 eight-metric headline or the 1.32x composite.
Summary Results
| Category | Calor vs C# | Winner |
|---|---|---|
| Comprehension | 1.84x | Calor |
| Error Detection | 1.49x | Calor |
| Edit Precision | 1.36x | Calor |
| Refactoring Stability | 1.38x | Calor |
| Correctness | 1.29x | Calor |
| Generation Accuracy | 1.02x | Calor |
| Token Economics | 1.42x | Calor |
| Information Density | 0.98x | C# |
Current pattern: Calor leads seven categories. C# leads information density at 0.98x Calor/C#.
Key Insight
The strongest current advantages are comprehension (1.84x), error detection (1.49x), token economics (1.42x composite), and refactoring stability (1.38x). Generation accuracy is close at 1.02x. Information density is 0.98x, the only current C# win.
Token economics is a composite of token, character, and line ratios. Calor still uses more raw tokens on small programs, so the 1.42x composite must not be read as a raw-token advantage.
Agent Task Benchmark
The Agent Task Benchmark tests Claude's ability to generate correct Calor code from natural language prompts. It validates skill documentation quality by measuring how well Claude can learn Calor syntax.
The latest committed agent-task artifact is dated 2026-02-16 and reports an 86.5% pass rate (77 of 89 tasks across 17 categories). It is a separate, single-run benchmark and is not part of the v0.12 dashboard composite.
Learn More
- Methodology - How benchmarks work
- Results - Detailed results table
- Agent Tasks - Claude code generation benchmark
- Individual Metrics - Deep dive into each metric