Results

Live Benchmark Results

Evaluated across 217 programs with 8 metrics

Updated: Aug 12, 2026, 11:02 PM17d253d

Overall Advantage

1.32x

Calor/C# composite

Programs Tested

217

217 Calor successes

Calor Wins

of 8 metrics

C# Wins

of 8 metrics

How to read this run: these are deterministic static-analysis micro-benchmarks. Repeating them 30 times produced identical values, so the zero-width confidence intervals and reported p-values do not measure uncertainty in the 217-program sample. These results are also separate from the v0.12 release gates.

Agent Refactoring Benchmark

Measures Claude Code agent success rates on real refactoring tasks (rename, extract, inline, move, add contracts, change signature).

Calor19/20 tasks

95%

C#19/20 tasks

95%

View category breakdown

Rename Symbol

2/3

3/3

Extract Method

4/4

Inline Function

3/3

Move Method

3/3

Add Contract

4/4

3/4

Change Signature

3/3

Majority voting (2/3 runs) with compilation + Z3 verification580e189

Metric Breakdown

ComprehensionCalor wins

1.84x

Explicit structure aids understanding

Error DetectionCalor wins

1.49x

Contracts surface invariant violations

Token EconomicsCalor wins

1.42x

More compact representation

Refactoring StabilityCalor wins

1.38x

More stable during refactoring

Edit PrecisionCalor wins

1.36x

Explicit IDs and markers aid targeting

CorrectnessCalor wins

1.29x

Contracts help prevent edge case bugs

Generation AccuracyCalor wins

1.02x

Better code generation from prompts

Information DensityC# wins

0.98x

Calor trades density for explicitness

Calor better

Tie

C# better

|Center line = 1.0x (equal)

Current Status

Calor leads in 7 of 8 metrics, demonstrating advantages in areas where explicitness matters.

Per-Program Results

Filter by level:

Program	Lvl	Adv	Tokens	Gen Acc	Comp	Edit	Err Det	Info Den	Refactor	Correct
Abs	1	1.31	1.04	1.00	1.68	1.31	1.67	0.91	1.30	1.60
AbsoluteContracts	2	1.56	0.55	1.14	3.50	1.39	2.14	0.82	1.37	1.58
AbstractClass	2	1.12	1.73	1.00	1.15	1.33	0.86	0.57	1.35	1.00
Adapter	2	1.55	2.13	1.00	1.57	1.66	1.83	1.05	1.52	1.60
AggregateStats	2	1.11	1.52	1.00	1.09	1.31	1.00	0.63	1.32	1.00
AreaOfCircle	1	1.50	1.59	1.00	2.27	1.31	2.14	1.04	1.30	1.33
ArrayContracts	3	1.47	1.60	1.03	1.44	1.35	2.14	1.02	1.40	1.80
ArraySlice	2	1.51	1.14	1.00	2.78	1.43	2.14	0.79	1.30	1.50
ArraySum	2	1.19	1.40	1.00	1.48	1.23	1.00	1.23	1.32	0.83
AsyncChain	3	1.16	1.47	1.00	1.17	1.43	1.00	0.85	1.35	1.00
AsyncEffect	3	1.27	1.69	1.00	1.36	1.07	1.33	1.27	1.47	1.00
AsyncErrorHandling	3	1.39	1.49	1.00	1.83	1.48	1.33	1.59	1.47	0.91
AsyncLoop	3	1.39	1.91	1.00	1.96	1.43	1.00	1.48	1.32	1.00
AsyncPure	3	1.04	1.45	1.00	0.73	1.03	1.00	0.79	1.32	1.00
AsyncReturn	3	1.14	1.33	1.00	1.29	1.31	1.00	0.85	1.30	1.00
AutoProps	2	1.18	1.01	1.00	1.70	1.33	1.00	1.05	1.35	1.00
Average	2	1.38	1.44	1.00	1.88	1.31	1.57	1.53	1.30	1.00
BankAccount	3	1.65	1.45	1.00	2.81	1.65	2.14	1.24	1.37	1.50
BasicTryCatch	2	1.14	1.34	1.00	1.26	1.31	1.00	0.80	1.35	1.08
BclCoverage	2	1.39	2.65	1.14	1.41	1.43	1.14	0.88	1.47	1.00
BinarySearch	3	1.19	1.95	1.00	1.32	1.35	1.00	0.66	1.40	0.86
BinaryTree	4	1.47	3.92	1.00	1.05	1.36	0.86	0.82	1.84	0.92
BitContracts	3	1.40	0.74	1.14	2.47	1.39	1.83	0.61	1.37	1.60
BitSet	3	1.50	0.60	1.00	2.71	1.31	2.50	0.69	1.44	1.80
BMICalculator	2	1.45	0.91	1.00	2.09	1.31	2.50	0.63	1.30	1.90
BreadthFirstSearch	3	1.34	2.55	1.00	1.45	1.35	1.00	1.15	1.40	0.86
BubbleSort	2	1.44	1.40	1.00	2.36	1.27	1.83	1.12	1.37	1.17
BuggyContracts	2	1.34	1.32	1.14	1.43	1.39	1.67	1.08	1.46	1.25
Builder	3	1.25	2.07	1.00	1.06	1.36	0.86	0.98	1.49	1.20
Calculator	2	1.09	1.21	1.00	1.29	1.35	1.00	0.52	1.37	1.00
Calendar	3	1.45	0.61	1.00	2.30	1.31	2.50	0.66	1.35	1.90
CancellableTask	3	1.24	1.61	1.00	1.58	1.43	1.00	0.96	1.32	1.00
Capitalize	2	1.30	0.90	1.00	1.82	1.31	1.83	0.64	1.30	1.60
CelsiusToKelvin	1	1.51	1.55	1.00	2.27	1.31	2.14	1.01	1.30	1.50
ChainOfResponsibility	3	1.32	1.91	1.00	1.33	1.36	1.57	0.69	1.43	1.23
CircularBuffer	3	1.52	1.62	1.00	2.24	1.33	2.14	1.18	1.35	1.29
Clamp	2	1.44	2.06	1.00	1.33	1.35	1.67	0.92	1.25	1.90
CollectionLib	2	1.38	0.98	1.10	2.29	1.52	1.67	0.81	1.40	1.25
Command	3	1.29	1.90	1.00	1.56	1.68	1.00	0.95	1.40	0.83
CompactClass	2	1.40	0.71	1.00	2.38	1.31	2.00	0.81	1.39	1.60
ComposedEffects	3	1.37	1.72	1.14	1.84	1.39	1.33	1.27	1.56	0.71
Composite	3	1.66	2.03	1.00	1.76	1.68	2.50	1.04	1.49	1.80
Composition	3	1.44	1.64	1.00	1.62	1.36	1.83	1.14	1.56	1.40
CompoundInterest	2	1.46	1.30	1.00	2.21	1.31	2.14	0.93	1.32	1.50
ConstructorInit	2	1.55	1.26	1.00	2.75	1.33	2.14	1.22	1.35	1.33
Contains	2	1.14	1.16	1.00	1.44	1.31	1.00	0.66	1.32	1.20
ContractedDivide	3	1.31	1.96	1.00	0.87	1.31	1.67	0.69	1.19	1.80
CorrectEffects	2	1.31	1.45	1.14	1.72	1.39	1.33	0.88	1.59	1.00
CountOccurrences	2	1.35	0.94	1.00	1.90	1.31	1.67	1.05	1.32	1.60
CountVowels	2	1.11	1.18	1.00	1.50	1.23	1.00	0.65	1.35	1.00
CsvParser	2	1.48	1.25	1.00	1.97	1.31	1.67	2.29	1.32	1.00
CurrencyConverter	2	1.51	0.68	1.00	2.70	1.31	2.50	0.75	1.29	1.80
CustomException	2	1.35	1.53	1.00	1.45	1.35	1.57	0.87	1.43	1.60
DatabaseEffect	3	1.35	1.14	1.14	2.21	1.39	1.33	1.07	1.50	1.00
DateDiff	2	1.57	0.77	1.00	3.13	1.31	2.50	0.64	1.29	1.90
Decorator	3	1.63	1.60	1.00	2.07	1.71	2.50	0.93	1.43	1.80
DelayedResult	3	1.41	1.24	1.00	1.83	1.31	2.17	0.93	1.39	1.40
DepthFirstSearch	3	1.33	2.19	1.00	1.55	1.31	1.00	1.02	1.35	1.20
Deque	3	1.58	1.85	1.00	2.33	1.33	2.14	1.15	1.32	1.50
DictOps	2	1.44	1.26	1.00	2.35	1.43	1.67	1.21	1.27	1.33
DigitCount	2	1.53	0.52	1.00	2.64	1.31	2.50	0.90	1.44	1.90
Dijkstra	4	1.71	1.78	1.00	2.54	1.34	2.50	1.42	1.32	1.80
DisjointSet	3	1.29	2.13	1.00	1.24	1.52	1.00	0.86	1.37	1.20
DistanceCalculator	3	1.19	0.74	1.00	1.43	1.31	1.67	0.52	1.34	1.50
DivisionContracts	3	1.54	1.28	1.14	2.35	1.39	2.14	0.73	1.46	1.80
EditDistance	3	1.21	1.65	1.00	1.47	1.23	1.00	1.02	1.34	1.00
EmailValidator	2	1.08	0.97	1.00	1.55	1.31	1.00	0.53	1.30	1.00
Encapsulation	3	1.18	1.98	1.00	1.07	1.36	1.00	0.68	1.37	1.00
EnumMatch	2	1.32	1.03	1.00	2.07	1.31	1.00	1.66	1.30	1.20
EnumType	3	1.26	1.00	1.00	2.09	1.31	1.00	0.97	1.32	1.40
EnumWithMethods	3	1.22	0.87	1.00	2.09	1.31	1.00	0.79	1.30	1.40
ErrorPropagation	2	1.26	1.04	1.00	1.88	1.31	1.57	0.83	1.30	1.17
ExceptionChain	2	1.40	1.45	1.00	1.79	1.31	1.43	1.47	1.35	1.36
ExhaustiveMatch	2	1.30	1.14	1.00	2.13	1.31	0.86	1.48	1.32	1.17
ExpressionBodied	1	1.14	1.02	1.00	1.69	1.31	1.00	0.85	1.27	1.00
Factorial	2	1.14	1.15	1.00	1.31	1.31	1.00	0.79	1.37	1.20
Factory	3	1.63	2.21	1.00	1.99	1.35	2.14	0.97	1.89	1.50
Fibonacci	2	1.14	0.86	1.00	1.41	1.31	1.00	0.75	1.37	1.40
FileEffects	3	1.39	1.61	1.10	2.15	1.39	1.33	1.05	1.50	1.00
Filter	2	1.24	1.41	1.00	2.06	1.43	1.00	0.55	1.27	1.20
FizzBuzz	2	1.23	1.85	1.00	1.18	1.27	1.33	0.26	1.56	1.40
Flyweight	3	1.57	1.54	1.00	1.88	1.36	2.50	1.32	1.46	1.50
FormatHeader	2	1.14	1.43	1.00	1.29	1.31	1.00	0.66	1.44	1.00
GCD	2	1.10	1.13	1.00	1.24	1.31	1.00	0.72	1.37	1.00
GenericClass	2	1.42	1.89	1.00	1.68	1.33	1.43	1.18	1.32	1.50
GenericConstraints	3	1.30	1.13	1.00	2.35	1.43	1.00	0.86	1.27	1.40
GenericFunction	2	1.25	1.12	1.00	2.32	1.43	1.00	0.88	1.27	1.00
GradeCalculator	2	1.28	0.86	1.00	1.68	1.31	1.83	0.60	1.30	1.70
Graph	3	1.64	1.48	1.00	2.65	1.61	2.50	1.03	1.32	1.50
GroupBy	2	1.49	1.73	1.00	2.39	1.43	1.83	0.85	1.27	1.40
GuardClause	2	1.33	1.40	1.00	2.00	1.31	1.57	0.65	1.35	1.33
GuardMatch	2	1.31	1.33	1.00	1.86	1.31	1.00	1.32	1.30	1.40
HashMap	3	1.62	1.95	1.00	1.94	1.28	2.50	1.35	1.43	1.50
HelloWorld	1	1.40	1.57	1.00	2.61	1.31	1.33	0.94	1.39	1.00
HiddenNetworkEffect	3	1.50	2.18	1.14	1.89	1.43	1.14	1.65	1.60	1.00
Hypotenuse	2	1.47	0.95	1.00	2.69	1.31	2.14	0.89	1.30	1.50
Inheritance	3	1.29	1.33	1.00	1.44	1.48	1.00	1.28	1.37	1.40
InlineSigs	2	1.12	0.86	1.00	1.74	1.31	1.00	0.59	1.27	1.20
InsertionSort	2	1.24	1.43	1.00	2.04	1.23	1.00	1.04	1.35	0.83
InterfaceImpl	3	1.55	1.92	1.00	1.81	1.68	1.83	1.16	1.59	1.40
InterpolationSearch	3	1.31	1.39	1.00	1.55	1.31	1.83	0.86	1.37	1.14
Inventory	3	1.54	1.43	1.00	2.66	1.33	2.14	0.94	1.32	1.50
IsAlpha	1	1.11	0.82	1.00	1.91	1.31	1.00	0.37	1.27	1.20
IsEven	1	1.14	1.61	1.00	1.21	1.31	1.00	0.52	1.30	1.20
IsOdd	1	1.12	1.62	1.00	1.21	1.31	1.00	0.52	1.30	1.00
IsPrime	3	1.22	1.45	1.00	1.18	1.35	1.22	0.63	1.30	1.60
Iterator	2	1.41	1.42	1.00	1.75	1.34	1.83	1.20	1.32	1.40
Knapsack	4	1.58	1.20	1.00	2.66	1.23	2.50	1.10	1.35	1.58
LCS	3	1.22	1.45	1.00	1.59	1.23	1.00	1.03	1.30	1.17
LeapYear	2	1.18	1.49	1.00	1.36	1.35	1.00	0.41	1.40	1.40
LinearSearch	2	1.10	1.04	1.00	1.57	1.23	1.00	0.78	1.32	0.86
LinkedList	4	1.60	4.48	1.00	1.00	1.36	0.86	1.07	1.67	1.40
LinqPipeline	2	1.19	1.26	1.00	1.62	1.31	1.00	0.87	1.27	1.17
ListInvariant	3	1.60	0.52	1.14	3.58	1.39	2.14	0.85	1.37	1.80
ListOps	2	1.34	1.00	1.00	2.06	1.31	1.67	1.08	1.27	1.33
LogPipeline	2	1.15	1.33	1.00	1.58	1.31	1.00	0.53	1.42	1.00
Map	2	1.40	1.67	1.00	2.95	1.43	1.00	0.87	1.27	1.00
MathLib	2	1.39	0.77	1.10	2.47	1.39	1.67	0.74	1.37	1.60
MathOperations	3	1.32	2.40	1.00	0.82	1.35	1.11	0.91	1.37	1.60
MatrixMultiply	3	1.42	1.57	1.00	1.95	1.34	1.57	1.30	1.30	1.33
MaxHeap	3	1.48	2.44	1.00	1.63	1.36	1.57	1.28	1.43	1.17
MaxTwo	1	1.26	0.95	1.00	1.70	1.31	1.67	0.76	1.30	1.40
MaxValue	2	1.10	1.12	1.00	1.33	1.23	0.67	1.20	1.23	1.00
Mediator	3	1.30	2.46	1.00	1.40	1.36	0.86	1.12	1.37	0.83
MergeSort	3	1.52	2.88	1.00	1.44	1.35	1.83	0.97	1.37	1.33
MethodOverloading	3	1.08	0.90	1.00	1.62	1.31	1.00	0.57	1.27	1.00
MethodOverriding	3	1.25	1.13	1.00	1.56	1.43	1.00	1.55	1.32	1.00
MinStack	3	1.31	2.12	1.00	1.39	1.61	0.86	1.01	1.32	1.17
MinTwo	1	1.26	0.95	1.00	1.70	1.31	1.67	0.76	1.30	1.40
MissingEffects	2	1.38	1.72	1.14	1.95	1.39	1.33	0.92	1.56	1.00
MixedContracts	3	1.46	1.47	1.14	1.55	1.39	2.14	1.01	1.49	1.50
MixedSyntax	2	1.40	1.23	1.00	2.11	1.31	2.14	0.79	1.32	1.33
ModuloContracts	3	1.64	0.43	1.14	3.64	1.39	2.50	0.68	1.40	1.90
MultipleCatch	2	1.14	1.28	1.00	1.27	1.35	1.00	0.73	1.37	1.09
NamedConfig	2	1.12	1.32	1.00	1.25	1.31	1.00	0.62	1.42	1.00
NestedMatch	2	1.31	1.35	1.00	1.86	1.31	1.00	1.43	1.30	1.20
NetworkEffect	3	1.36	1.15	1.14	2.17	1.39	1.33	1.20	1.53	1.00
NullCheck	2	1.10	1.16	1.00	1.66	1.31	0.86	0.70	1.27	0.80
Observer	3	1.29	2.10	1.00	1.40	1.36	1.00	0.86	1.40	1.20
OptionalIds	2	1.09	0.94	1.00	1.62	1.31	1.00	0.59	1.27	1.00
OptionType	2	1.48	1.16	1.10	2.43	1.41	1.57	1.24	1.43	1.50
OverflowSafe	3	1.51	1.63	1.03	1.46	1.35	2.14	1.29	1.40	1.80
OverflowUnsafe	3	1.61	1.36	1.03	1.62	1.31	2.50	2.00	1.25	1.80
PairLogger	2	1.11	1.23	1.00	1.42	1.31	1.00	0.47	1.42	1.00
Palindrome	2	1.18	1.26	1.00	1.49	1.31	1.00	1.01	1.35	1.00
ParallelTasks	3	1.29	1.38	1.00	2.16	1.43	1.00	1.06	1.30	1.00
ParseAndDouble	2	1.14	1.42	1.00	1.29	1.31	1.00	0.66	1.44	1.00
PasswordValidator	2	1.11	1.10	1.00	1.49	1.31	1.00	0.69	1.32	1.00
PhoneBook	2	1.52	1.27	1.00	2.38	1.33	2.50	0.91	1.27	1.50
Polymorphism	3	1.25	1.46	1.00	1.19	1.68	1.00	0.82	1.43	1.40
Power	2	1.26	1.70	1.00	1.06	1.35	1.22	0.89	1.27	1.60
PriorityQueue	3	1.54	2.77	1.00	1.71	1.36	1.57	1.05	1.49	1.33
Properties	2	1.39	1.39	1.10	1.83	1.48	1.57	1.04	1.40	1.33
PropertyAccess	1	1.83	1.34	1.10	1.63	1.48	1.83	4.25	1.43	1.60
PropertyMatch	2	1.37	1.46	1.00	1.79	1.65	1.00	1.48	1.37	1.20
ProvableContracts	2	1.48	1.01	1.14	2.10	1.39	2.14	0.98	1.49	1.58
Proxy	3	1.30	1.59	1.00	1.41	1.68	1.14	0.88	1.53	1.20
PureComputation	2	1.34	0.83	1.14	2.10	1.39	1.67	0.75	1.37	1.50
PureFunctions	1	1.36	0.99	1.14	1.50	1.43	1.83	0.74	1.58	1.70
Queue	3	1.30	2.29	1.00	1.22	1.36	0.86	1.09	1.40	1.17
QuickSort	4	1.57	3.69	1.00	1.44	1.27	1.57	0.90	1.56	1.14
RangeContracts	3	1.50	0.77	1.14	2.45	1.39	2.50	0.73	1.46	1.58
RangeMatch	2	1.54	1.11	1.00	2.32	1.31	1.83	1.78	1.30	1.70
RecordType	3	1.44	0.56	1.10	2.80	1.71	1.67	0.74	1.43	1.50
Reduce	2	1.23	1.24	1.00	2.00	1.31	1.00	0.79	1.27	1.20
ResultType	2	1.24	1.71	1.00	1.53	1.36	1.00	1.02	1.37	0.92
ReturnMapped	2	1.13	1.45	1.00	1.25	1.31	1.00	0.64	1.42	1.00
ReverseArray	2	1.53	1.02	1.00	2.87	1.31	1.83	1.59	1.32	1.33
ReverseString	2	1.22	1.80	1.00	1.40	1.31	1.00	1.10	1.32	0.83
ScoreBoard	2	1.57	1.34	1.00	2.37	1.33	2.50	0.95	1.27	1.80
SealedClass	2	1.24	1.96	1.00	1.46	1.33	1.00	0.83	1.35	1.00
SearchContracts	3	1.57	0.62	1.14	2.92	1.39	2.50	0.77	1.40	1.80
SelectionSort	2	1.44	1.40	1.00	2.23	1.23	1.83	1.27	1.35	1.17
SetOps	2	1.48	1.48	1.00	2.22	1.43	1.67	1.20	1.35	1.50
ShoppingCart	4	1.71	3.23	1.00	1.90	1.28	2.14	1.06	1.93	1.19
Sign	1	1.10	1.23	1.00	1.38	1.31	1.00	0.61	1.30	1.00
SimpleAsync	3	1.33	1.34	1.00	1.96	1.31	1.33	1.34	1.39	1.00
SimpleClass	2	1.63	1.40	1.00	2.36	1.65	2.50	0.88	1.43	1.80
SimpleMatch	2	1.37	0.90	1.00	2.15	1.31	1.00	1.87	1.30	1.40
Singleton	2	1.17	1.71	1.00	1.31	1.36	1.00	0.83	1.37	0.77
Sort	2	1.16	1.55	1.00	1.58	1.23	1.00	0.78	1.30	0.83
SortedList	2	1.37	1.50	1.00	1.65	1.59	1.67	0.97	1.30	1.25
SortingContracts	3	1.56	0.68	1.14	3.07	1.39	2.50	0.55	1.37	1.80
Stack	3	1.33	2.46	1.00	1.15	1.36	0.86	0.99	1.40	1.40
State	3	1.44	1.24	1.00	2.05	1.36	1.00	2.07	1.37	1.40
StateEffect	3	1.39	1.07	1.14	2.00	1.41	1.33	1.66	1.50	1.00
StaticMembers	3	1.26	1.48	1.10	1.37	1.43	1.00	1.34	1.32	1.00
Strategy	3	1.26	2.12	1.00	1.33	1.50	1.00	0.75	1.40	1.00
StringContracts	3	1.37	1.42	1.03	1.45	1.31	2.14	1.01	1.32	1.29
StringLib	2	1.32	0.80	1.10	2.33	1.39	1.67	0.65	1.37	1.25
StringUtils	2	1.08	1.30	1.00	1.29	1.35	1.00	0.51	1.37	0.83
SumDigits	2	1.55	0.42	1.00	2.87	1.31	2.50	0.95	1.44	1.90
SumRange	2	1.31	1.71	1.00	1.48	1.23	1.57	0.95	1.34	1.17
SwitchExpression	2	1.56	1.11	1.00	1.93	1.31	1.83	2.22	1.35	1.70
TaxCalculator	3	1.42	0.73	1.00	2.28	1.31	2.50	0.65	1.30	1.58
TemperatureConverter	1	1.11	1.29	1.00	1.28	1.31	1.00	0.63	1.35	1.00
TemperatureRange	2	1.12	1.47	1.00	1.14	1.31	1.00	0.69	1.32	1.00
TemplateMethod	3	1.00	1.44	0.76	0.37	0.94	0.86	0.73	1.83	1.10
TernaryChain	2	1.19	0.89	1.00	1.61	1.31	1.00	1.02	1.27	1.40
ThreeWayMerge	2	1.09	1.33	1.00	1.15	1.31	1.00	0.52	1.42	1.00
Timer	2	1.53	1.11	1.00	2.57	1.36	2.14	1.19	1.40	1.50
TimerEffect	3	1.59	0.66	1.10	3.13	1.39	2.50	0.69	1.40	1.90
TodoList	2	1.54	1.44	1.00	2.06	1.59	2.14	1.23	1.32	1.50
TowerOfHanoi	3	1.50	1.18	1.00	2.33	1.35	2.14	0.89	1.49	1.58
Trie	4	1.62	2.31	1.00	1.80	1.36	2.50	1.11	1.46	1.39
TruncateString	2	1.26	1.04	1.00	1.85	1.31	1.57	0.93	1.30	1.07
TryFinally	2	1.18	1.28	1.00	1.48	1.31	1.00	1.01	1.35	1.00
TupleMatch	2	1.24	0.91	1.00	1.91	1.31	1.00	1.07	1.30	1.40
TypeAlias	3	1.45	1.34	1.10	2.56	1.41	1.38	1.38	1.30	1.17
TypeMatch	2	1.41	1.27	1.00	1.75	1.35	1.00	2.14	1.37	1.40
UnitConverter	2	1.35	0.69	1.00	2.37	1.31	1.83	0.69	1.27	1.60
UrlParser	2	1.16	0.96	1.00	1.55	1.31	1.00	0.75	1.27	1.40
Visitor	3	1.04	1.50	0.76	0.42	0.98	1.00	0.55	2.13	1.00
VoidSequence	2	1.32	1.14	1.00	3.03	1.35	1.00	0.60	1.47	1.00
VotingSystem	2	1.40	1.20	1.00	1.95	1.33	1.83	0.95	1.30	1.60
WildcardMatch	2	1.20	0.80	1.00	1.84	1.31	1.00	0.93	1.30	1.40
Zip	2	1.28	1.36	1.00	2.43	1.31	1.00	0.84	1.27	1.00

Showing 217 of 217 programs. Values above 1.0 favor Calor (highlighted in pink), values below 1.0 favor C# (highlighted in teal).

Read the Results Carefully

The v0.12 run covers 217 programs, reports an overall Calor/C# advantage of 1.32x, and has Calor ahead in seven of eight metrics. Comprehension is 1.84x, error detection is 1.49x, token economics is 1.42x, and information density is 0.98x—the one category where C# leads.

These measurements need two important caveats:

The metrics are deterministic static analyses. Thirty runs are thirty identical observations, so the zero-width confidence intervals and p-values measure no run-to-run variance; they do not measure sampling uncertainty over the corpus.
These C#-versus-Calor micro-benchmarks are not the v0.12 release gates. PP-A1 covered adoption readiness, while PP-W5 measured toolchain tax and concluded only that no large tax was detected—not that parity was proven.

Token economics is a composite of token, character, and line ratios. Calor still pays a § token premium on small programs even though the current composite favors Calor.

When to Use Calor

Based on results, Calor is most valuable when:

Contract verification matters — The error detection advantage validates explicit contracts
Edit precision is important — Optional explicit IDs and structural markers provide targeting signals
Agent comprehension is critical — Explicit structure provides clear signals
Compact source matters across several dimensions — The composite token, character, and line measure currently favors Calor

Use C# when:

Ecosystem libraries are needed — Leverage existing tooling
Human readability is priority — Familiar syntax for human developers

Methodology

All benchmarks are automated and reproducible. Each program is evaluated across 8 metrics comparing Calor and C# implementations.

The published dashboard framework:

Compiles paired Calor and C# programs.
Applies deterministic metric calculators to source, syntax trees, and compilation results.
Aggregates the per-program Calor/C# ratios.
Repeats the deterministic evaluation 30 times for the reported statistical artifact. Greater than 1.0 favors Calor; less than 1.0 favors C#.

LLM-based task runners exist separately and are not folded into this dashboard result.

See Methodology for full details on the evaluation framework.

Running Benchmarks

To regenerate benchmark results:

Bash

# Run the evaluation framework
dotnet run --project tests/Calor.Evaluation -- run -f website -o website/public/data/benchmark-results.json

# Start the website to view results
cd website && npm run dev

Results are automatically loaded from benchmark-results.json and displayed in the dashboard above.