Generation Accuracy

Category: Generation Accuracy v0.12 result: Calor narrowly wins (1.02x across 217 programs) What it measures: Compilation success and structural correctness

Overview

The Generation Accuracy metric measures how correctly AI agents can generate valid code in each language.

Why It Matters

When an agent generates code, it must:

• Produce syntactically valid code
• Create structurally complete programs
• Avoid compilation errors

Languages with familiar patterns and extensive training data have an advantage.

How It's Measured

Composite Score

Score = (CompilationSuccess × 0.5) + (StructureScore × 0.3) + (NoErrors × 0.2)

Structure Score Calculation

Calor Structure Score

C# Structure Score

Why the Result Is Close

The current static compilation-and-structure calculator gives Calor a small 1.02x lead. That is close enough that the following C# familiarity advantages still matter when interpreting actual LLM generation, which this metric does not directly measure.

1. Training Data Advantage

LLMs have seen vastly more C# code than Calor code. This creates:

• Better pattern recognition
• Fewer syntax errors
• More accurate boilerplate generation

2. Familiar Patterns

// C#: Very common pattern, well-learned
public static int Add(int a, int b)
{
    return a + b;
}

// Calor: Novel syntax, less exposure
§F{f001:Add:pub}
  §I{i32:a}
  §I{i32:b}
  §O{i32}
  §R (+ a b)

3. Error Recovery

C# errors often have clear fixes:

• Missing semicolon → add semicolon
• Missing brace → add brace
• Wrong type → cast or convert

Calor errors may be less familiar:

• Misindented block body → requires understanding indent-only scope
• Wrong block nesting → requires indentation-level understanding

Example Comparison

Agent Task: "Write a function that adds two integers"

C# generation (typical):

public static int Add(int a, int b)
{
    return a + b;
}

• Compiles: Yes
• Structure: Complete
• Errors: 0
• Score: 1.0

Calor generation (typical):

§F{f001:Add:pub}
  §I{i32:a}
  §I{i32:b}
  §O{i32}
  §R (+ a b)

• Compiles: Yes
• Structure: Complete
• Errors: 0
• Score: 1.0

Calor generation (common error):

§F{f001:Add:pub}
§I{i32:a}
§I{i32:b}
§O{i32}
§R (+ a b)

• Compiles: No
• Error: Function body is not indented
• Score: 0.0

Error Patterns

Common Calor Errors

Common C# Errors

Mitigation Strategies

For Calor

1. Template-based generation: Start with valid templates
2. Indentation tracking: Maintain consistent block levels
3. Validation pass: Check structure before outputting

For C#

1. Standard patterns: Use well-established code patterns
2. Roslyn validation: Parse generated code before output
3. Error correction: Attempt to fix common errors

Detailed Metrics

The framework provides detailed breakdowns:

results.Add(MetricResult.CreateHigherIsBetter(
    Category,
    "CompilationSuccess",
    calorCompilation.Success ? 1.0 : 0.0,
    csharpCompilation.Success ? 1.0 : 0.0));

results.Add(MetricResult.CreateLowerIsBetter(
    Category,
    "ErrorCount",
    calorCompilation.Errors.Count,
    csharpCompilation.Errors.Count));

results.Add(MetricResult.CreateHigherIsBetter(
    Category,
    "StructureCompleteness",
    CalculateCalorStructureScore(calorCompilation),
    CalculateCSharpStructureScore(csharpCompilation)));

Interpretation

The v0.12 ratio is 1.02x, a small Calor lead in the deterministic calculator. Actual model-generated code may behave differently because:

• C# has more training data exposure
• C# patterns are more familiar to LLMs
• C# tooling provides better error feedback

Treat the result as near parity on the measured compilation and structural signals, not as proof that an LLM generates either language more reliably.

Next

• Task Completion - End-to-end success rates