Benchmark run luna6-multiswe-go-hints-smoke-20261007-r1
This page contains exactly the publishable benchmark observations recorded in run luna6-multiswe-go-hints-smoke-20261007-r1; it does not select or aggregate them. Return to the run index or view aggregate metrics.
Interactive browser
Rows are pre-aggregated by task, target, prompt variant, MCP-instruction mode, context, arm, and restriction. Numeric values show median (p50), IQR, p90, maximum, and sample count. Boolean checks show pass counts and success rates. Filters apply to these summary groups; per-run observations remain available on their run pages.
Loading benchmark results…
Baseline vs semedit
Comparison evaluates baseline and semedit values within each restriction policy, target, and context group. Negative changes mark improvements for these lower-is-better metrics; positive changes mark degradations.
Detailed observations
Empirical telemetry measuring wall-clock latency, token usage, tool turns, and compiler correctness when AI coding agents perform code modifications.
Every benchmark pairs Vanilla LLM (standard file editing tools) against Semedit MCP (deterministic AST compiler operations).
Benchmark Methodology & Transparency
- Reproducibility & Provenance: Self-contained txtar archives define test scenarios. Links below resolve to commit-anchored GitHub source files for published commits, or cryptographic SHA-256 fingerprints for uncommitted local fixtures.
- Model Cost: Tables report model-specific credit consumption where a target model has a declared schedule: (uncached input × input credits + cached input × cached credits + (reasoning + visible output) × output credits) / 1,000,000. Thinking tokens use the output rate. ADR-0043 defines the per-million-token credit rates.
- Multi-Level Correctness Oracle: Four validation levels evaluate each trial:
- Level 1 (Mutation Policy): Restricts file modifications strictly to authorized paths.
- Level 2 (AST Invariants): Compiler AST verification of required symbols, imports, and relative declaration ordering.
- Level 3 (Clean Build): Strict compilation validation with zero compiler or typecheck errors.
- Level 4 (Test Suite): Unit and integration test suite execution.
- Isolated Sandboxing: Each trial executes in an ephemeral directory with pristine git and module state.
Test case: zeromicro__go-zero-2463
Target: codex/gpt-6-luna/high
explicit prompt · none MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | zeromicro__go-zero-2463 |
| Target | codex/gpt-6-luna/high |
| Prompt variant | explicit |
| MCP server instructions | none |
| Semedit arm restriction | write |
| Fixture | testdata/bench/zeromicro__go-zero-2463.txtar |
Vanilla LLM Prompt:
logx能否支持一下全局添加field功能,或默认加一个serviceName,收集日志想根据服务分索引很不方便
| Metric | Vanilla (Small) | Semedit Explicit (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 326.68s | 238.09s | -27.1% | — | — | — |
| Process Start → First Event | 1.03s | 0.14s | — | — | — | — |
| First Event → First Tool Call | 4.64s | 37.50s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 4 | 1 | -75.0% | — | — | — |
| Internal Tool Cycles | 49 | 4 | -91.8% | — | — | — |
| Initial Load / Discovery Turns | 7 | 0 | -100.0% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 49 | 4 | -91.8% | — | — | — |
| Output Tokens | 52851 | 3107 | -94.1% | — | — | — |
| Reasoning / Thinking Tokens | 28743 | 708 | -97.5% | — | — | — |
| Total Input Tokens | 6423405 | 215455 | -96.6% | — | — | — |
| Cached Input Tokens | 6131968 | 183552 | -97.0% | — | — | — |
| Uncached Input Tokens | 291437 | 31903 | -89.1% | — | — | — |
| Cost | 3.2815 | 0.1733 | -94.7% | — | — | — |
| Cached vs Uncached Token Ratio | 21.04:1 | 5.75:1 | -72.7% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | FAIL | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Explicit in Small Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | ✓ ✓ |
| 2 | ? ✗ | semedit/semantic_batch (✓ ✓) |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | ✓ ✓ |
| 5 | ✓ ✓ | — |
| 6 | ✓ ✓ | — |
| 7 | ✓ ✓ | — |
| 8 | ✓ ✓ | — |
| 9 | file_change (✓ ✓) | — |
| 10 | file_change (✓ ✓) | — |
| 11 | file_change (✓ ✓) | — |
| 12 | file_change (✓ ✓) | — |
| 13 | file_change (✓ ✓) | — |
| 14 | ✓ ✓ | — |
| 15 | file_change (✓ ✓) | — |
| 16 | ✓ ✓ | — |
| 17 | ✓ ✓ | — |
| 18 | ✓ ✓ | — |
| 19 | ✓ ✓ | — |
| 20 | file_change (✓ ✓) | — |
| 21 | file_change (✓ ✓) | — |
| 22 | ✓ ✓ | — |
| 23 | file_change (✓ ✓) | — |
| 24 | file_change (✓ ✓) | — |
| 25 | file_change (✓ ✓) | — |
| 26 | file_change (✓ ✓) | — |
| 27 | ? ✗ | — |
| 28 | ✓ ✓ | — |
| 29 | file_change (✓ ✓) | — |
| 30 | file_change (✓ ✓) | — |
| 31 | ✓ ✓ | — |
| 32 | ✓ ✓ | — |
| 33 | file_change (✓ ✓) | — |
| 34 | file_change (✓ ✓) | — |
| 35 | file_change (✓ ✓) | — |
| 36 | file_change (✓ ✓) | — |
| 37 | file_change (✓ ✓) | — |
| 38 | file_change (✓ ✓) | — |
| 39 | file_change (✓ ✓) | — |
| 40 | ✓ ✓ | — |
| 41 | ✓ ✓ | — |
| 42 | file_change (✓ ✓) | — |
| 43 | ✓ ✓ | — |
| 44 | ? ✗ | — |
| 45 | ✓ ✓ | — |
| 46 | ✓ ✓ | — |
| 47 | file_change (✓ ✓) | — |
| 48 | file_change (✓ ✓) | — |
| 49 | ✓ ✓ | — |
Semedit integration feedback (agent opinion, excluded from measurements):
I’d ask my owner to add semedit: its single ordered batch let me scaffold and wire a multi-file Go change cleanly, which felt reassuring, even though the verification output was unwieldy.
Baseline vs Semedit Explicit in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
implicit prompt · none MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | zeromicro__go-zero-2463 |
| Target | codex/gpt-6-luna/high |
| Prompt variant | implicit |
| MCP server instructions | none |
| Semedit arm restriction | write |
| Fixture | testdata/bench/zeromicro__go-zero-2463.txtar |
Vanilla LLM Prompt:
logx能否支持一下全局添加field功能,或默认加一个serviceName,收集日志想根据服务分索引很不方便
Semedit MCP Prompt:
logx能否支持一下全局添加field功能,或默认加一个serviceName,收集日志想根据服务分索引很不方便 Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.
Target: codex/gpt-6-luna/medium
explicit prompt · none MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | zeromicro__go-zero-2463 |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | explicit |
| MCP server instructions | none |
| Semedit arm restriction | write |
| Fixture | testdata/bench/zeromicro__go-zero-2463.txtar |
Vanilla LLM Prompt:
logx能否支持一下全局添加field功能,或默认加一个serviceName,收集日志想根据服务分索引很不方便
| Metric | Vanilla (Small) | Semedit Explicit (Small) | Δ (Small) | Vanilla (Large) | MCP (Large) | Δ (Large) |
|---|---|---|---|---|---|---|
| Wall-Clock Latency | 190.07s | 184.35s | -3.0% | — | — | — |
| Process Start → First Event | 0.57s | 0.62s | — | — | — | — |
| First Event → First Tool Call | 4.01s | 46.08s | — | — | — | — |
| MCP Initialize → First Semantic Call | — | — | — | — | — | — |
| MCP Server Start → Initialize | — | — | — | — | — | — |
| Top-Level User Turns | 4 | 1 | -75.0% | — | — | — |
| Internal Tool Cycles | 28 | 3 | -89.3% | — | — | — |
| Initial Load / Discovery Turns | 6 | 0 | -100.0% | — | — | — |
| MCP Discovery / Schema Turns | 0 | 0 | — | — | — | — |
| Tool Invocations (per run) | 28 | 3 | -89.3% | — | — | — |
| Output Tokens | 24304 | 2868 | -88.2% | — | — | — |
| Reasoning / Thinking Tokens | 7259 | 368 | -94.9% | — | — | — |
| Total Input Tokens | 3038871 | 205946 | -93.2% | — | — | — |
| Cached Input Tokens | 2830080 | 179200 | -93.7% | — | — | — |
| Uncached Input Tokens | 208791 | 26746 | -87.2% | — | — | — |
| Cost | 1.6240 | 0.1521 | -90.6% | — | — | — |
| Cached vs Uncached Token Ratio | 13.55:1 | 6.70:1 | -50.6% | — | — | — |
| Oracle L1: Mutation Policy | PASS | PASS | — | — | — | — |
| Oracle L2: AST Invariants | PASS | PASS | — | — | — | — |
| Oracle L3: Clean Build | PASS | PASS | — | — | — | — |
| Oracle L4: Verification Test | FAIL | PASS | — | — | — | — |
| Semantic Tool Invocation Verified (per run) | N/A | PASS | — | — | — | — |
Baseline vs Semedit Explicit in Small Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | ✓ ✓ | semedit/semantic_batch (✓ ✓) |
| 2 | ✓ ✓ | ✓ ✓ |
| 3 | ✓ ✓ | ✓ ✓ |
| 4 | ✓ ✓ | — |
| 5 | ✓ ✓ | — |
| 6 | ✓ ✓ | — |
| 7 | ✓ ✓ | — |
| 8 | ✓ ✓ | — |
| 9 | file_change (✓ ✓) | — |
| 10 | file_change (✓ ✓) | — |
| 11 | file_change (✓ ✓) | — |
| 12 | file_change (✓ ✓) | — |
| 13 | ✓ ✓ | — |
| 14 | file_change (✓ ✓) | — |
| 15 | ✓ ✓ | — |
| 16 | ✓ ✓ | — |
| 17 | ✓ ✓ | — |
| 18 | ✓ ✓ | — |
| 19 | file_change (✓ ✓) | — |
| 20 | file_change (✓ ✓) | — |
| 21 | file_change (✓ ✓) | — |
| 22 | file_change (✓ ✓) | — |
| 23 | ✓ ✓ | — |
| 24 | ✓ ✓ | — |
| 25 | ✓ ✓ | — |
| 26 | ✓ ✓ | — |
| 27 | file_change (✓ ✓) | — |
| 28 | ✓ ✓ | — |
Semedit integration feedback (agent opinion, excluded from measurements):
Semedit’s ordered semantic batch applied the full logging change in one call, which felt efficient, so I’d ask my owner to add its tools.
Baseline vs Semedit Explicit in Large Context
| # | Vanilla | Semedit MCP |
|---|---|---|
| 1 | not published | not published |
implicit prompt · none MCP instructions · write semedit restriction
| Setting | Value |
|---|---|
| Test case | zeromicro__go-zero-2463 |
| Target | codex/gpt-6-luna/medium |
| Prompt variant | implicit |
| MCP server instructions | none |
| Semedit arm restriction | write |
| Fixture | testdata/bench/zeromicro__go-zero-2463.txtar |
Vanilla LLM Prompt:
logx能否支持一下全局添加field功能,或默认加一个serviceName,收集日志想根据服务分索引很不方便
Semedit MCP Prompt:
logx能否支持一下全局添加field功能,或默认加一个serviceName,收集日志想根据服务分索引很不方便 Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.