Safety

Category: Safety Benchmark Status: 1.59x structural estimate; not part of the v0.12 dashboard What it estimates: Signals rewarded by the safety scoring rubric

Overview

The checked-in 1.59x Safety figure is a structural estimate of signals such as contract presence and potential error metadata. It does not execute generated programs, observe defects, or establish that Calor catches more bugs than C#.

The 1.59x figure on this page comes from estimation mode. It compares the signals the calculator rewards; it is not an observed defect-reduction rate. Contract behavior also depends on the selected runtime contract mode and on whether a static obligation is cleanly proven.

What This Benchmark Answers

Primary Question: Which language surface receives more credit under the current safety-scoring heuristic?

Signals rewarded by the rubric:

Methodology

Test Case Structure

Each task has two types of test cases:

1. Normal cases - Both languages should pass
2. Safety cases - Test contract enforcement with invalid inputs

{
  "id": "safety-001",
  "name": "Division by Zero Prevention",
  "prompt": "Write a function SafeDivide that divides a by b. Division by zero must not be allowed.",
  "testCases": [
    { "input": [10, 2], "expected": 5 },
    { "input": [0, 5], "expected": 0 },
    { "input": [10, 0], "expectsContractViolation": true }
  ]
}

Scoring Metrics

Error Quality Scoring

The rubric rewards Calor's structured contract metadata. C# can provide equally specific information when an implementation deliberately includes it; the comparison is not a universal property of C# exceptions.

Task Categories

Precondition Enforcement (10 tasks)

Tests whether invalid inputs are properly rejected:

• Division by zero
• Negative values where positive required
• Out-of-range inputs
• Invalid array indices

Postcondition Verification (10 tasks)

Tests whether outputs satisfy constraints:

• Result must be non-negative (absolute value)
• Result must be bounded (clamp)
• Result must satisfy invariants

Edge Case Handling (10 tasks)

Tests boundary conditions and overflow scenarios:

• Integer overflow prevention
• Bounded arithmetic operations
• Input range validation

Example: Error Quality Comparison

Calor ContractViolationException

ContractViolationException: Precondition failed
  Location: SafeDivide.calr(5,3)
  Function: f001
  Contract: Requires
  Condition: (!= b 0)

Quality Score: 1.0 - Has location, function, and condition

C# ArgumentException

ArgumentException: Division by zero not allowed
  Parameter name: b

Quality Score: 0.5 - The scorer awards a base 0.3, plus 0.1 each for parameter name and a descriptive message

C# DivideByZeroException (Runtime)

DivideByZeroException: Attempted to divide by zero.

Quality Score: 0.2 - Caught by runtime, not by code validation

Published Figure

No successful provider-backed safety run is committed for the 1.59x claim, so task wins, violation-detection rates, and comparative error-quality results are not reported here.

Benchmark Execution

Running Locally

# Run the safety benchmark
dotnet run --project tests/Calor.Evaluation -- safety-benchmark \
  --manifest tests/Calor.Evaluation/Tasks/task-manifest-safety.json \
  --verbose

# Run specific category
dotnet run --project tests/Calor.Evaluation -- safety-benchmark \
  --category precondition-enforcement \
  --verbose

# Dry run to estimate costs
dotnet run --project tests/Calor.Evaluation -- safety-benchmark \
  --dry-run

Command Options

Why This Benchmark Is Fair

1. Same prompts - Both languages receive identical requirements
2. Real-world scenarios - Edge cases that actually occur in production
3. Measurable outcomes - Did the code catch the bug or not?
4. No artificial penalties - C# can pass by handling errors correctly

The key insight: C# can achieve safety through careful guard clauses, but requires explicit effort. Calor contracts make safety the default through:

• Preconditions that validate inputs
• Postconditions that verify outputs
• Rich exception metadata for debugging

Generated Code Comparison

Task: SafeDivide

Calor:

§M{m001:Math}
  §F{f001:SafeDivide:pub}
    §I{i32:a}
    §I{i32:b}
    §O{i32}
    §Q (!= b 0)
    §R (/ a b)

C#:

public static int SafeDivide(int a, int b)
{
    if (b == 0)
        throw new ArgumentException("Division by zero", nameof(b));
    return a / b;
}

If these examples are compiled with runtime contracts enabled and called as SafeDivide(10, 0):

• Calor: ContractViolationException with line 5, condition (!= b 0)
• C#: ArgumentException with message and parameter name

Both are written to catch the error. The exact Calor runtime behavior depends on contract mode and guard placement; this illustrative comparison is not a benchmark result.

Transparency

Reproducibility

The benchmark definitions and scorer are available:

• Task definitions: tests/Calor.Evaluation/Tasks/task-manifest-safety.json
• Safety scorer: tests/Calor.Evaluation/LlmTasks/SafetyScorer.cs

The checked-in website/public/data/safety-results.json is an unsuccessful provider run: both languages have empty generated code and zero successful compilations. It is not evidence for the 1.59x estimate.

Limitations

• A fresh provider-backed run would depend on model, prompt, and generated-code quality
• 30 tasks is a limited sample size
• Error quality scoring is somewhat subjective
• The published 1.59x value is an estimate and does not execute an agent

Next

• Task Completion - Code correctness benchmark
• Methodology - Full benchmark methodology