v0.13.2—Control flow and dataflow are rebuilt on explicit semantics, Z3 translation matches executable C# numeric rules, and the build chain is hermetic and supply-chain verified with hash- and size-pinned Z3 binaries.See what's new

benchmark

Compare Calor vs C# across evaluation metrics.

Bash
calor benchmark [project] [options]

Overview

The benchmark command compares paired Calor and C# source across seven categories:

  1. Token Economics - Token count and density
  2. Generation Accuracy - Code correctness
  3. Comprehension - Understandability
  4. Edit Precision - Targeted modification accuracy
  5. Error Detection - Bug identification
  6. Information Density - Meaning per token
  7. Task Completion - End-to-end success

This command is not the same surface as the published v0.12 website artifact, which is generated by tests/Calor.Evaluation over 217 programs and currently contains eight metrics. Do not expect one local calor benchmark invocation to reproduce the 1.32x headline.

Quick Start

Bash
# Compare two files
calor benchmark --calor Calculator.calr --csharp Calculator.cs

# Benchmark entire project
calor benchmark ./src

# Quick token-only comparison
calor benchmark --calor file.calr --csharp file.cs --quick

# Generate markdown report
calor benchmark ./src --format markdown --output report.md

Options

OptionShortDefaultDescription
--calorNoneCalor file to benchmark
--csharp, --csNoneC# file to benchmark
--category-cAllFilter by category
--format-fconsoleOutput format: console, markdown, json
--output-ostdoutSave results to file
--verbose-vfalseShow detailed per-metric breakdown
--quick-qfalseQuick token-only benchmark

File-Level Benchmark

Compare a specific Calor file against its C# equivalent:

Bash
calor benchmark --calor PaymentService.calr --csharp PaymentService.cs

Use --category with one of the seven exact category names above to narrow the report. --format json emits the shared schema 2.0 envelope with benchmark data under data; human status text moves to stderr.

Quick Benchmark

For a fast token, character, and line comparison without the full seven-metric evaluation:

Bash
calor benchmark --calor file.calr --csharp file.cs --quick

Quick mode is descriptive only; inspect the separate token, character, and line values instead of treating its aggregate as proof of an agent advantage.

Exit Codes

CodeMeaning
0Benchmark completed successfully
1Invalid category, missing input, missing file, or another command error

Finding no paired files in project mode emits a warning and currently exits 0, so automation should inspect diagnostics as well as the process exit code.

See Also