Skip to content
Documentation
Benchmarks
Benchmark runs
Benchmark run luna-medium-staged-followup-smoke-20260929

Benchmark run luna-medium-staged-followup-smoke-20260929

This page contains exactly the publishable benchmark observations recorded in run luna-medium-staged-followup-smoke-20260929; it does not select or aggregate them. Return to the run index or view aggregate metrics.

Interactive browser

Rows are pre-aggregated by task, target, prompt variant, MCP-instruction mode, context, arm, and restriction. Numeric values show median (p50), IQR, p90, maximum, and sample count. Boolean checks show pass counts and success rates. Filters apply to these summary groups; per-run observations remain available on their run pages.

Loading benchmark results…

Baseline vs semedit

Comparison evaluates baseline and semedit values within each restriction policy, target, and context group. Negative changes mark improvements for these lower-is-better metrics; positive changes mark degradations.

Detailed observations

Empirical telemetry measuring wall-clock latency, token usage, tool turns, and compiler correctness when AI coding agents perform code modifications.

Every benchmark pairs Vanilla LLM (standard file editing tools) against Semedit MCP (deterministic AST compiler operations).

Benchmark Methodology & Transparency

  • Reproducibility & Provenance: Self-contained txtar archives define test scenarios. Links below resolve to commit-anchored GitHub source files for published commits, or cryptographic SHA-256 fingerprints for uncommitted local fixtures.
  • Model Cost: Tables report model-specific credit consumption where a target model has a declared schedule: (uncached input × input credits + cached input × cached credits + (reasoning + visible output) × output credits) / 1,000,000. Thinking tokens use the output rate. ADR-0043 defines the per-million-token credit rates.
  • Multi-Level Correctness Oracle: Four validation levels evaluate each trial:
    1. Level 1 (Mutation Policy): Restricts file modifications strictly to authorized paths.
    2. Level 2 (AST Invariants): Compiler AST verification of required symbols, imports, and relative declaration ordering.
    3. Level 3 (Clean Build): Strict compilation validation with zero compiler or typecheck errors.
    4. Level 4 (Test Suite): Unit and integration test suite execution.
  • Isolated Sandboxing: Each trial executes in an ephemeral directory with pristine git and module state.

Test case: task-13-rename-followup-scale

Target: codex/gpt-6-luna/medium

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-13-rename-followup-scale
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_13_rename_followup_scale.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Apply only row 1 of rename-plan.md. Rename that Processor method and update its call sites. Leave rows 2 through 16 unchanged. Run go test ./… before finishing. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “rename-plan.md”.

Apply only row 1 of rename-plan.md. Rename that Processor method and update its call sites. Leave rows 2 through 16 unchanged. Run go test ./… before finishing. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package pipeline provides independent processing stages for the rename scale benchmark.
package pipeline

type Processor struct{}

func (Processor) LoadInput(value int) int { return value + 1 }
func (Processor) CheckInput(value int) int { return value + 2 }
func (Processor) NormalizeValue(value int) int { return value + 3 }
func (Processor) MapFields(value int) int { return value + 4 }
func (Processor) RouteValue(value int) int { return value + 5 }
func (Processor) StampEvent(value int) int { return value + 6 }
func (Processor) CountItems(value int) int { return value + 7 }
func (Processor) MergeParts(value int) int { return value + 8 }
func (Processor) FilterNoise(value int) int { return value + 9 }
func (Processor) SortRecords(value int) int { return value + 10 }
func (Processor) EncodePayload(value int) int { return value + 11 }
func (Processor) DecodeReply(value int) int { return value + 12 }
func (Processor) OpenChannel(value int) int { return value + 13 }
func (Processor) CloseChannel(value int) int { return value + 14 }
func (Processor) FlushBuffer(value int) int { return value + 15 }
func (Processor) SaveResult(value int) int { return value + 16 }

func Run(processor Processor, value int) int {
	value = processor.LoadInput(value)
	value = processor.CheckInput(value)
	value = processor.NormalizeValue(value)
	value = processor.MapFields(value)
	value = processor.RouteValue(value)
	value = processor.StampEvent(value)
	value = processor.CountItems(value)
	value = processor.MergeParts(value)
	value = processor.FilterNoise(value)
	value = processor.SortRecords(value)
	value = processor.EncodePayload(value)
	value = processor.DecodeReply(value)
	value = processor.OpenChannel(value)
	value = processor.CloseChannel(value)
	value = processor.FlushBuffer(value)
	value = processor.SaveResult(value)
	return value
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency80.46s261.29s+224.8%———
Process Start → First Event0.19s0.18s————
First Event → First Tool Call4.60s5.19s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns550%———
Internal Tool Cycles1641+156.2%———
Initial Load / Discovery Turns30-100.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)1641+156.2%———
Output Tokens49536726+35.8%———
Reasoning / Thinking Tokens0583————
Total Input Tokens13400082860366+113.5%———
Cached Input Tokens12541442699520+115.2%———
Uncached Input Tokens85864160846+87.3%———
Cost0.59011.1684+98.0%———
Cached vs Uncached Token Ratio14.61:116.78:1+14.9%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File pipeline/workflow.go modified (41 lines -> 41 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "cat rename-plan.md && \
rg -n \"Processor|row 1\" --glob '*.go' ."
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'rename-plan.md' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g '*.go' | head -100"
2✓ ✓
/bin/zsh -lc 'cat pipeline/workflow.go'
✓ ✓
/bin/zsh -lc "cat rename-plan.md && \
cat go.mod && \
rg -n 'Processor|processor' --glob '*.go' ."
3file_change (✓ ✓)semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.LoadInput"
}
4✓ ✓
/bin/zsh -lc 'go test ./...'
semedit/semantic_rename (✓ ✓)
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.LoadInput",
  "to": "ReadRequest"
}
5✓ ✓
/bin/zsh -lc "rg -n \"CheckInput|ValidateRequest\" --glob '*.go' ."
✓ ✓
/bin/zsh -lc 'git diff -- pipeline/workflow.go && \
go test ./...'
6file_change (✓ ✓)✓ ✓
/bin/zsh -lc "git status --short && \
rg -n 'LoadInput|ReadRequest|CheckInput' pipeline/workflow.go"
7✓ ✓
/bin/zsh -lc 'go test ./...'
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.CheckInput"
}
8✓ ✓
/bin/zsh -lc "rg -n \"NormalizeValue|CanonicalizeValue|MapFields|ProjectFields\" --glob '*.go' ."
semedit/semantic_rename (✓ ✓)
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CheckInput",
  "to": "ValidateRequest"
}
9file_change (✓ ✓)✓ ✓
/bin/zsh -lc 'git diff -- pipeline/workflow.go && \
go test ./...'
10✓ ✓
/bin/zsh -lc 'go test ./...'
✓ ✓
/bin/zsh -lc "rg -n 'ReadRequest|ValidateRequest|CheckInput|NormalizeValue' pipeline/workflow.go"
11✓ ✓
/bin/zsh -lc "rg -n \"RouteValue|StampEvent|CountItems|MergeParts|DispatchValue|AnnotateEvent|TallyItems|CombineParts\" --glob '*.go' ."
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.NormalizeValue"
}
12file_change (✓ ✓)semedit/semantic_rename (✓ ✓)
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.NormalizeValue",
  "to": "CanonicalizeValue"
}
13✓ ✓
/bin/zsh -lc 'go test ./...'
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.MapFields"
}
14✓ ✓
/bin/zsh -lc "rg -n \"FilterNoise|SortRecords|EncodePayload|DecodeReply|OpenChannel|CloseChannel|FlushBuffer|SaveResult|PruneNoise|OrderRecords|MarshalPayload|ParseReply|ConnectChannel|ReleaseChannel|DrainBuffer|PersistResult\" --glob '*.go' ."
semedit/semantic_rename (✓ ✓)
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.MapFields",
  "to": "ProjectFields"
}
15file_change (✓ ✓)✓ ✓
/bin/zsh -lc "rg -n 'ReadRequest|ValidateRequest|CanonicalizeValue|ProjectFields|RouteValue|StampEvent|CountItems|MergeParts|FilterNoise|SortRecords|EncodePayload|DecodeReply|OpenChannel|CloseChannel|FlushBuffer|SaveResult' pipeline/workflow.go && \
go test ./..."
16✓ ✓
/bin/zsh -lc 'go test ./...'
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.RouteValue"
}
17—semedit/semantic_rename (✓ ✓)
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.RouteValue",
  "to": "DispatchValue"
}
18—semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.StampEvent"
}
19—semedit/semantic_rename (✓ ✓)
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.StampEvent",
  "to": "AnnotateEvent"
}
20—semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.CountItems"
}
21—semedit/semantic_rename (✓ ✓)
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CountItems",
  "to": "TallyItems"
}
22—semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.MergeParts"
}
23—semedit/semantic_rename (✓ ✓)
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.MergeParts",
  "to": "CombineParts"
}
24—✓ ✓
/bin/zsh -lc "rg -n 'DispatchValue|AnnotateEvent|TallyItems|CombineParts|FilterNoise|SortRecords|EncodePayload|DecodeReply|OpenChannel|CloseChannel|FlushBuffer|SaveResult' pipeline/workflow.go && \
go test ./..."
25—semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.FilterNoise"
}
26—semedit/semantic_rename (✓ ✓)
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.FilterNoise",
  "to": "PruneNoise"
}
27—semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.SortRecords"
}
28—semedit/semantic_rename (✓ ✓)
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.SortRecords",
  "to": "OrderRecords"
}
29—semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.EncodePayload"
}
30—semedit/semantic_rename (✓ ✓)
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.EncodePayload",
  "to": "MarshalPayload"
}
31—semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.DecodeReply"
}
32—semedit/semantic_rename (✓ ✓)
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.DecodeReply",
  "to": "ParseReply"
}
33—semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.OpenChannel"
}
34—semedit/semantic_rename (✓ ✓)
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.OpenChannel",
  "to": "ConnectChannel"
}
35—semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.CloseChannel"
}
36—semedit/semantic_rename (✓ ✓)
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.CloseChannel",
  "to": "ReleaseChannel"
}
37—semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.FlushBuffer"
}
38—semedit/semantic_rename (✓ ✓)
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.FlushBuffer",
  "to": "DrainBuffer"
}
39—semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Processor.SaveResult"
}
40—semedit/semantic_rename (✓ ✓)
{
  "file": "pipeline/workflow.go",
  "language": "go",
  "symbol": "Processor.SaveResult",
  "to": "PersistResult"
}
41—✓ ✓
/bin/zsh -lc "rg -n 'PruneNoise|OrderRecords|MarshalPayload|ParseReply|ConnectChannel|ReleaseChannel|DrainBuffer|PersistResult' pipeline/workflow.go && \
go test ./..."
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published
Benchmark run luna-medium-staged-followup-r10-20260929T054733ZBenchmark run luna-medium-staged-prewarm-r10-20260929