LUNA EVALUATION LAB / 20 SEPTEMBER 2026
Nine models. Four everyday agent tasks. Measured latency and estimated inference spend, with correctness kept in view.
Loading the benchmark snapshot…
Observations from this small workload, not a general model ranking. Findings below compare only models with all planned trials recorded and fully metered.
| Model | Recorded | Correctness | Full pass | Mean task | Median turn | p95 turn | First text, mean | USD / task | USD / correct task |
|---|
Pass counts use recorded trials as the denominator; Recorded shows progress against planned trials. Full pass includes timing and cache performance checks. Cost per correct task includes spend on failed attempts. “Unknown” means incomplete metering or missing trials, never zero.
Three repetitions of each named test per model, run sequentially with model order rotated. Each trial uses a fresh disposable Luna instance, frozen core and plugins, a pinned model, and no fallback or judge. Provider generation defaults were retained; reasoning budgets were not normalized.
Task time sums all user-turn durations, including tools and continuation calls. Setup and cleanup are excluded. Multi-turn tasks naturally take longer. First text differs from first activity: tool calls may start before visible text. Provider latency and shared host load were not controlled.
Only three repetitions per test: p95 is descriptive, not a dependable tail-latency estimate. Correctness uses limited deterministic checks; this is not a broad intelligence or coding evaluation.
Foreground inference only, using reported token usage and the campaign’s dated standard paid text rates. Includes observed cache reads and Anthropic five-minute cache writes. Excludes auxiliary/background calls, external tool fees, infrastructure and cache storage. Context above 200k is not priced.
Caches were not cleared, so a fresh local database does not imply a cold provider cache. Gemini promotional rates in the snapshot expire December 31, 2026. No free-tier savings are assumed.
Grok and Kimi remain outside this comparison because the benchmark environment has no configured inference credentials for their providers. No speed or cost results are inferred for them.
Rates below are USD per million tokens from the September 20, 2026 campaign snapshot. Follow the provider links for current terms.
| Model / source | Input | Output | Cache read | Cache write / input |
|---|
Downloads contain aggregate metrics, test hashes and pricing provenance. Raw conversations, credentials and local filesystem paths are excluded. This is a published snapshot; it does not update automatically as new trials finish.