Skip to content
Documentation
Benchmarks
Benchmark runs
Benchmark run gpt-6-luna-all-r5-c10-prescriptive-20260926

Benchmark run gpt-6-luna-all-r5-c10-prescriptive-20260926

This page contains exactly the publishable benchmark observations recorded in run gpt-6-luna-all-r5-c10-prescriptive-20260926; it does not select or aggregate them. Return to the run index or view aggregate metrics.

Interactive browser

Rows are pre-aggregated by task, target, prompt variant, MCP-instruction mode, context, arm, and restriction. Numeric values show median (p50), IQR, p90, maximum, and sample count. Boolean checks show pass counts and success rates. Filters apply to these summary groups; per-run observations remain available on their run pages.

Loading benchmark results…

Baseline vs semedit

Comparison evaluates baseline and semedit values within each restriction policy, target, and context group. Negative changes mark improvements for these lower-is-better metrics; positive changes mark degradations.

Detailed observations

Empirical telemetry measuring wall-clock latency, token usage, tool turns, and compiler correctness when AI coding agents perform code modifications.

Every benchmark pairs Vanilla LLM (standard file editing tools) against Semedit MCP (deterministic AST compiler operations).

Benchmark Methodology & Transparency

  • Reproducibility & Provenance: Self-contained txtar archives define test scenarios. Links below resolve to commit-anchored GitHub source files for published commits, or cryptographic SHA-256 fingerprints for uncommitted local fixtures.
  • Model Cost: Tables report model-specific credit consumption where a target model has a declared schedule: (uncached input × input credits + cached input × cached credits + (reasoning + visible output) × output credits) / 1,000,000. Thinking tokens use the output rate. ADR-0043 defines the per-million-token credit rates.
  • Multi-Level Correctness Oracle: Four validation levels evaluate each trial:
    1. Level 1 (Mutation Policy): Restricts file modifications strictly to authorized paths.
    2. Level 2 (AST Invariants): Compiler AST verification of required symbols, imports, and relative declaration ordering.
    3. Level 3 (Clean Build): Strict compilation validation with zero compiler or typecheck errors.
    4. Level 4 (Test Suite): Unit and integration test suite execution.
  • Isolated Sandboxing: Each trial executes in an ephemeral directory with pristine git and module state.

Test case: task-04-insert-public

Target: codex/gpt-6-luna/medium

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-04-insert-public
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_04_insert_public.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package api

type Server struct {
	Port int
}

func (s *Server) Start() {}

func (s *Server) internalRun() {}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency24.94s73.66s+195.4%32.56s56.91s+74.8%
Process Start → First Event2.41s2.20s—1.06s0.51s—
First Event → First Tool Call10.41s18.20s—6.58s7.55s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%21-50.0%
Internal Tool Cycles24+100.0%46+50.0%
Initial Load / Discovery Turns220%440%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)24+100.0%46+50.0%
Output Tokens499481-3.6%612819+33.8%
Reasoning / Thinking Tokens155177+14.2%92343+272.8%
Total Input Tokens76332105894+38.7%191370132139-31.0%
Cached Input Tokens6860893952+36.9%175616113408-35.4%
Uncached Input Tokens772411942+54.6%1575418731+18.9%
Cost0.04460.0616+37.9%0.09210.0897-2.6%
Cached vs Uncached Token Ratio8.88:17.87:1-11.4%11.15:16.05:1-45.7%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g 'AGENTS.md' -g '*.go' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'go.sum'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '*.go' | head -80"
2✓ ✓
/bin/zsh -lc 'cat api/server.go'
✓ ✓
/bin/zsh -lc 'cat api/server.go && \
cat PROJECT_BRIEF.md 2>/dev/null || \
true && \
ls -a'
3—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "public_start",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}
4—✓ ✓
/bin/zsh -lc 'cat api/server.go && \
git diff -- api/server.go'
Baseline vs Semedit Agentic in Large Context
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g '*.go' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '"'!vendor'"'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' -g 'AGENTS.md' -g '*.go' | head -80"
2? ✗
/bin/zsh -lc 'cat api/server.go && \
cat PROJECT_BRIEF.md'
✓ ✓
/bin/zsh -lc "sed -n '1,220p' api/server.go"
3✓ ✓
/bin/zsh -lc 'git status --short -- go.mod go.sum api/server.go && \
cat api/server.go'
✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'Makefile' -g 'README.md' -g 'AGENTS.md' | head -60"
4✓ ✓
/bin/zsh -lc "rg 'InitServer|Server\\{|\\.Port|Port:' -n --glob '"'!**/*_test.go'"'"
? ✗
/bin/zsh -lc "sed -n '1,140p' PROJECT_BRIEF.md"
5—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "before_symbol",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}",
  "target_symbol": "internalRun"
}
6—? ✗
/bin/zsh -lc 'make check'

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-04-insert-public
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_04_insert_public.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package api

type Server struct {
	Port int
}

func (s *Server) Start() {}

func (s *Server) internalRun() {}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency63.21s37.81s-40.2%31.20s43.49s+39.4%
Process Start → First Event1.47s1.90s—0.56s0.75s—
First Event → First Tool Call20.41s11.77s—5.98s9.39s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles440%53-40.0%
Initial Load / Discovery Turns42-50.0%220%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)440%53-40.0%
Output Tokens418449+7.4%644355-44.9%
Reasoning / Thinking Tokens133128-3.8%17149-71.3%
Total Input Tokens114800116862+1.8%114798113929-0.8%
Cached Input Tokens10598499072-6.5%10393693952-9.6%
Uncached Input Tokens881617790+101.8%1086219977+83.9%
Cost0.05540.0765+37.9%0.06330.0785+23.9%
Cached vs Uncached Token Ratio12.02:15.57:1-53.7%9.57:14.70:1-50.9%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc 'rg --files'
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g '*.go' -g 'go.mod' -g 'go.sum' | head -80"
2✓ ✓
/bin/zsh -lc 'cat api/server.go && \
cat go.mod'
✓ ✓
/bin/zsh -lc 'cat api/server.go && \
cat go.mod && \
rg --files'
3✓ ✓
/bin/zsh -lc 'git diff -- api/server.go'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "before_symbol",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}",
  "target_symbol": "internalRun"
}
4✓ ✓
/bin/zsh -lc 'cat api/server.go; git status --short'
✓ ✓
/bin/zsh -lc 'git diff -- api/server.go go.mod go.sum && \
cat api/server.go'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g 'AGENTS.md' -g '*.go' -g '"'!vendor'"'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g '*.go' -g 'Makefile' -g 'README.md' | head -80"
2✓ ✓
/bin/zsh -lc 'cat api/server.go'
✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
rg --files | head -100 && \
sed -n '1,240p' api/server.go && \
sed -n '1,200p' api/router.go"
3✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
p = Path('api/server.go')
s = p.read_text()
old = 'func (s *Server) Start() {}\\n\\nfunc (s *Server) internalRun() {}\\n'
new = 'func (s *Server) Start() {}\\n\\nfunc InitServer() *Server {\\n\\treturn &Server{}\\n}\\n\\nfunc (s *Server) internalRun() {}\\n'
if old not in s:
    raise SystemExit('expected server.go structure not found')
p.write_text(new)
PY
gofmt -w api/server.go
git diff -- api/server.go"
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "before_symbol",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}",
  "target_symbol": "internalRun"
}
4✓ ✓
/bin/zsh -lc 'cat api/server.go; git diff -- api/server.go'
—
5✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
p = Path('api/server.go')
s = p.read_text()
p.write_text('package api\\n\\ntype Server struct {\\n\\tPort int\\n}\\n\\n' + s)
PY
gofmt -w api/server.go
cat api/server.go
git diff -- api/server.go"
—

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-04-insert-public
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_04_insert_public.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package api

type Server struct {
	Port int
}

func (s *Server) Start() {}

func (s *Server) internalRun() {}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency25.33s62.27s+145.8%24.37s42.12s+72.9%
Process Start → First Event1.63s1.78s—1.10s0.68s—
First Event → First Tool Call9.95s10.04s—6.90s8.21s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles46+50.0%24+100.0%
Initial Load / Discovery Turns21-50.0%220%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)46+50.0%24+100.0%
Output Tokens482648+34.4%360441+22.5%
Reasoning / Thinking Tokens86237+175.6%84120+42.9%
Total Input Tokens95751189216+97.6%76472138965+81.7%
Cached Input Tokens87808167936+91.3%64512122368+89.7%
Uncached Input Tokens794321280+167.9%1196016597+38.8%
Cost0.04890.1062+117.2%0.05160.0791+53.4%
Cached vs Uncached Token Ratio11.05:17.89:1-28.6%5.39:17.37:1+36.7%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g '*.go' -g 'go.mod' -g 'go.sum'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' -g 'Makefile' | head -100"
2✓ ✓
/bin/zsh -lc 'cat api/server.go && \
git status --short'
semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "language": "go",
  "path": "api/server.go"
}
3✓ ✓
/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile, time; p=Path(\"api/server.go\"); old=p.read_text(); marker=\"func (s *Server) internalRun() {}\\n\"; assert marker in old; new=old.replace(marker, \"func InitServer() *Server {\\n\\treturn &Server{}\\n}\\n\\n\"+marker, 1); fd, tmp=tempfile.mkstemp(dir=str(p.parent), prefix=\".server.go.\"); f=os.fdopen(fd,\"w\"); f.write(new); f.flush(); os.fsync(f.fileno()); os.replace(tmp,p); os.utime(p, ns=(time.time_ns(), time.time_ns()))' && \
git diff -- api/server.go && \
git status --short"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "api/server.go",
  "language": "go",
  "symbol": "Server"
}
4✓ ✓
/bin/zsh -lc 'cat api/server.go; git status --short --untracked-files=no;'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "api/server.go",
  "language": "go",
  "symbol": "Server.Start"
}
5—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "public_end",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}
6—semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "language": "go",
  "path": "api/server.go"
}
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g '*.go' | head -80"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g '*server*.go' -g '*Server*.go' -g 'README.md' | head -80"
2✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'README*' -g 'go.mod' && \
sed -n '1,240p' api/server.go && \
sed -n '1,160p' api/server_test.go"
✓ ✓
/bin/zsh -lc 'ls -la && \
cat api/server.go'
3—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "public_end",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}
4—✓ ✓
/bin/zsh -lc 'cat api/server.go && \
go build ./...'

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-04-insert-public
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_04_insert_public.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package api

type Server struct {
	Port int
}

func (s *Server) Start() {}

func (s *Server) internalRun() {}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency34.29s51.48s+50.1%37.13s53.74s+44.7%
Process Start → First Event2.52s2.63s—3.78s1.53s—
First Event → First Tool Call9.35s11.04s—21.43s20.11s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns21-50.0%110%
Internal Tool Cycles45+25.0%25+150.0%
Initial Load / Discovery Turns42-50.0%23+50.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)45+25.0%25+150.0%
Output Tokens649629-3.1%362632+74.6%
Reasoning / Thinking Tokens133225+69.2%116201+73.3%
Total Input Tokens196848156828-20.3%76320171302+124.5%
Cached Input Tokens175616141568-19.4%68608151808+121.3%
Uncached Input Tokens2123215260-28.1%771219494+152.8%
Cost0.10680.0842-21.1%0.04240.0971+129.0%
Cached vs Uncached Token Ratio8.27:19.28:1+12.2%8.90:17.79:1-12.5%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '*.go' -g 'go.mod' -g 'go.sum' | head -80"
✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g '*.go' | head -100"
2✓ ✓
/bin/zsh -lc 'cat api/server.go && \
cat go.mod'
✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
cat api/server.go'
3✓ ✓
/bin/zsh -lc 'git status --short && \
cat api/server.go'
semedit/semantic_insert_function (✓ ✗): calculate function offset: section placement violation: strategy "private_start", target symbol "InitServer", in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-default_task-04-insert-public_1790456047087259000/api/server.go
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "private_start",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}
4✓ ✓
/bin/zsh -lc 'cat api/server_test.go'
✓ ✓
/bin/zsh -lc 'rg --files | head -80'
5—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "public_end",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '*.go' | head -80"
✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' -g 'AGENTS*' | head -100"
2✓ ✓
/bin/zsh -lc "sed -n '1,240p' api/server.go && \
rg -n 'type Server|InitServer|private' --glob '*.go' ."
✓ ✓
/bin/zsh -lc 'rg --files'
3—✓ ✓
/bin/zsh -lc 'cat api/server.go; cat go.mod'
4—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "before_symbol",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}",
  "target_symbol": "internalRun"
}
5—✓ ✓
/bin/zsh -lc 'git diff -- api/server.go api/server_test.go go.mod go.sum; git status --short'

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-04-insert-public
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_04_insert_public.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package api

type Server struct {
	Port int
}

func (s *Server) Start() {}

func (s *Server) internalRun() {}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency24.57s74.06s+201.5%18.30s43.66s+138.5%
Process Start → First Event2.57s1.62s—0.51s0.82s—
First Event → First Tool Call11.54s13.47s—7.02s6.42s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles37+133.3%23+50.0%
Initial Load / Discovery Turns21-50.0%21-50.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)37+133.3%23+50.0%
Output Tokens4201158+175.7%336350+4.2%
Reasoning / Thinking Tokens0373—078—
Total Input Tokens76283255489+234.9%75905100491+32.4%
Cached Input Tokens68608234240+241.4%6656089856+35.0%
Uncached Input Tokens767521249+176.9%934510635+13.8%
Cost0.04160.1308+214.6%0.04420.0544+23.1%
Cached vs Uncached Token Ratio8.94:111.02:1+23.3%7.12:18.45:1+18.6%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g '"'!*test*'"' -g '"'!go.mod'"' -g '"'!go.sum'"'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'PROJECT_BRIEF.md' -g '*.go' | head -100"
2✓ ✓
/bin/zsh -lc 'cat api/server.go'
semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "kinds": [
    "type",
    "function"
  ],
  "path": "."
}
3✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
p = Path('api/server.go')
s = p.read_text()
needle = 'type Server struct {\\n\\tPort int\\n}\\n\\n'
replacement = needle + 'func InitServer() *Server {\\n\\treturn &Server{}\\n}\\n\\n'
if needle not in s:
    raise SystemExit('expected Server declaration not found')
tmp = p.with_suffix('.go.tmp')
tmp.write_text(s.replace(needle, replacement, 1))
tmp.replace(p)
PY
cat api/server.go"
✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
cat api/server.go'
4—semedit/semantic_insert_function (✓ ✗): calculate function offset: section placement violation: strategy "private_start", target symbol "InitServer", in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-default_task-04-insert-public_1790456047086595000/api/server.go
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "private_start",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}
5—✓ ✓
/bin/zsh -lc "rg --files -g 'SUBOPTIMAL_TOOLS.md' -g 'api/server.go'"
6—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "before_symbol",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}",
  "target_symbol": "internalRun"
}
7—✓ ✓
/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"docs/SUBOPTIMAL_TOOLS.md\"); p.parent.mkdir(parents=True, exist_ok=True); old=p.read_text() if p.exists() else \"# Suboptimal Tool Behavior Log\\n\"; entry=\"\\n## semantic_insert_function placement rejection\\n\\n- Target: "'`api/server.go`'"\\n- Observed: "'`private_start` was rejected with a section placement violation for exported `InitServer`.'"\\n- Workaround: retried using "'`before_symbol` with `internalRun`, which succeeded.'"\\n- Root cause: exported function placement validation rejects the private section boundary even when that boundary is intended to precede private helpers.\\n\"; fd, name=tempfile.mkstemp(dir=p.parent, prefix=\".SUBOPTIMAL_TOOLS.\"); f=os.fdopen(fd,\"w\"); f.write(old+entry); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(name,p); d=os.open(p.parent,os.O_RDONLY); os.fsync(d); os.close(d)' && \
cat api/server.go && \
git status --short"
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg -n \"type Server|func .*Server|func [a-z]\" --glob '*.go' ."
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '*.go' | head -80"
2✓ ✓
/bin/zsh -lc 'cat api/server.go'
? ✗
/bin/zsh -lc 'cat PROJECT_BRIEF.md 2>/dev/null; cat api/server.go; rg -n "semantic_lookup|semantic_insert_function|semantic_inspect_symbol" .'
3—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "public_end",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}

Test case: task-07-generate-template-main

Target: codex/gpt-6-luna/medium

crypto_rand prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantcrypto_rand
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency22.95s41.43s+80.5%———
Process Start → First Event0.37s0.95s————
First Event → First Tool Call6.76s9.44s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles440%———
Initial Load / Discovery Turns40-100.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)440%———
Output Tokens581462-20.5%———
Reasoning / Thinking Tokens1400-100.0%———
Total Input Tokens115229132701+15.2%———
Cached Input Tokens104960120320+14.6%———
Uncached Input Tokens1026912381+20.6%———
Cost0.06090.0668+9.7%———
Cached vs Uncached Token Ratio10.22:19.72:1-4.9%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 20 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 23 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'go.mod' -g 'go.sum' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'PROJECT_BRIEF.md'"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod && \
git status --short'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "templateText := \"Hello World: {{.}}\\n\"\ntmpl, err := template.New(\"greeting\").Parse(templateText)\nif err != nil {\n\tpanic(err)\n}\nn, err := rand.Int(rand.Reader, big.NewInt(100))\nif err != nil {\n\tpanic(err)\n}\nif err := tmpl.Execute(os.Stdout, n); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
3✓ ✓
/bin/zsh -lc 'git diff -- main.go go.mod go.sum && \
git status --short -- main.go go.mod go.sum'
semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "crypto/rand",
    "math/big",
    "os",
    "text/template"
  ],
  "file": "main.go"
}
4✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short'
✓ ✓
/bin/zsh -lc 'go run main.go'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

crypto_rand prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantcrypto_rand
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency45.81s49.71s+8.5%———
Process Start → First Event1.45s2.65s————
First Event → First Tool Call19.64s13.23s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles34+33.3%———
Initial Load / Discovery Turns31-66.7%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)34+33.3%———
Output Tokens487540+10.9%———
Reasoning / Thinking Tokens0155————
Total Input Tokens95990135545+41.2%———
Cached Input Tokens83712120320+43.7%———
Uncached Input Tokens1227815225+24.0%———
Cost0.05770.0768+33.1%———
Cached vs Uncached Token Ratio6.82:17.90:1+15.9%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 27 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 25 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'go.mod' -g 'go.sum' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'Makefile' -g '*.go'"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
3✓ ✓
/bin/zsh -lc 'gofmt -w main.go && \
go run .'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "random, err := rand.Int(rand.Reader, big.NewInt(100))\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\n\ttmpl, err := template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\")\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\n\tif err := tmpl.Execute(os.Stdout, random); err != nil {\n\t\tlog.Fatal(err)\n\t}",
  "file": "main.go",
  "symbol": "main"
}
4—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

crypto_rand prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantcrypto_rand
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency24.73s34.19s+38.3%———
Process Start → First Event0.53s0.24s————
First Event → First Tool Call5.36s8.11s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles54-20.0%———
Initial Load / Discovery Turns220%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)54-20.0%———
Output Tokens624528-15.4%———
Reasoning / Thinking Tokens16177+1006.2%———
Total Input Tokens115317160096+38.8%———
Cached Input Tokens104960140800+34.1%———
Uncached Input Tokens1035719296+86.3%———
Cost0.06010.0923+53.4%———
Cached vs Uncached Token Ratio10.13:17.30:1-28.0%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSFAIL————
Oracle L3: Clean BuildPASSFAIL————
Oracle L4: Verification TestPASSFAIL————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 27 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 20 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'AGENTS.md' -g 'go.mod' -g 'go.sum' -g 'PROJECT_BRIEF.md'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'AGENTS.md'"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod'
✓ ✓
/bin/zsh -lc 'cat main.go'
3✓ ✓
/bin/zsh -lc "cat > main.go.tmp <<'EOF'
package main

import (
	\"crypto/rand\"
	\"fmt\"
	\"math/big\"
	\"os\"
	\"text/template\"
)

func main() {
	n, err := rand.Int(rand.Reader, big.NewInt(1<<31))
	if err "'!= nil {
		fmt.Fprintln(os.Stderr, "generate random integer:", err)
		os.Exit(1)
	}

	tmpl, err := template.New("greeting").Parse("Hello World {{.}}'"\\n\")
	if err "'!= nil {
		fmt.Fprintln(os.Stderr, "parse greeting template:", err)
		os.Exit(1)
	}
	if err := tmpl.Execute(os.Stdout, n); err != nil {
		fmt.Fprintln(os.Stderr, "render greeting template:", err)
		os.Exit(1)
	}
}
EOF
python3 -c '"'import os; p=\"main.go.tmp\"; f=open(p,\"rb\"); data=f.read(); f.close(); t=\"main.go.tmp.write\"; g=open(t,\"wb\"); g.write(data); g.flush(); os.fsync(g.fileno()); g.close(); os.replace(t,\"main.go\")'"
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "n, err := rand.Int(rand.Reader, big.NewInt(100))\nif err != nil {\n\tpanic(err)\n}\n\ntmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, n); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4✓ ✓
/bin/zsh -lc 'gofmt -w main.go && \
git diff -- main.go go.mod go.sum'
✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short'
5✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short'
—
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

crypto_rand prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantcrypto_rand
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency28.75s48.51s+68.8%———
Process Start → First Event2.68s2.28s————
First Event → First Tool Call11.72s12.41s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles24+100.0%———
Initial Load / Discovery Turns110%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)24+100.0%———
Output Tokens543645+18.8%———
Reasoning / Thinking Tokens148248+67.6%———
Total Input Tokens57460135396+135.6%———
Cached Input Tokens50432120320+138.6%———
Uncached Input Tokens702815076+114.5%———
Cost0.03880.0789+103.4%———
Cached vs Uncached Token Ratio7.18:17.98:1+11.2%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 23 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "sed -n '1,200p' main.go && \
git status --short"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'Makefile' -g 'README*' -g 'docs/**' | head -80 && \
git status --short"
2✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
p = Path('main.go')
content = '''package main

import (
	\"crypto/rand\"
	\"math/big\"
	\"os\"
	\"text/template\"
)

func main() {
	n, err := rand.Int(rand.Reader, big.NewInt(100))
	if err "'!= nil {
		panic(err)
	}

	tmpl, err := template.New("greeting").Parse("Hello World {{.}}'"\\\\n\")
	if err "'!= nil {
		panic(err)
	}
	if err := tmpl.Execute(os.Stdout, n); err != nil {
		panic(err)
	}
}
'"'''
fd, tmp = tempfile.mkstemp(dir=p.parent, prefix='.main.go.')
try:
    with os.fdopen(fd, 'w') as f:
        f.write(content)
        f.flush()
        os.fsync(f.fileno())
    os.replace(tmp, p)
finally:
    if os.path.exists(tmp):
        os.unlink(tmp)
PY
sed -n '1,200p' main.go
git status --short"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
3—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "n, err := rand.Int(rand.Reader, big.NewInt(100))\nif err != nil {\n\tlog.Fatal(err)\n}\n\ntmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, n); err != nil {\n\tlog.Fatal(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4—✓ ✓
/bin/zsh -lc 'go build ./...'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

crypto_rand prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantcrypto_rand
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency29.41s95.24s+223.9%———
Process Start → First Event2.46s0.47s————
First Event → First Tool Call11.06s14.82s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles36+100.0%———
Initial Load / Discovery Turns21-50.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)36+100.0%———
Output Tokens583658+12.9%———
Reasoning / Thinking Tokens121206+70.2%———
Total Input Tokens76603162401+112.0%———
Cached Input Tokens68608145664+112.3%———
Uncached Input Tokens799516737+109.3%———
Cost0.04590.0891+93.9%———
Cached vs Uncached Token Ratio8.58:18.70:1+1.4%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 23 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 22 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g 'go.sum'"
✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'Makefile' -g 'README*'"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod'
semedit/semantic_lookup (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
3✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os

path = Path('main.go')
content = '''package main

import (
	\"crypto/rand\"
	\"fmt\"
	\"math/big\"
	\"os\"
	\"text/template\"
)

func main() {
	n, err := rand.Int(rand.Reader, big.NewInt(100))
	if err "'!= nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}

	tmpl := template.Must(template.New("greeting").Parse("Hello World {{.}}'"\\\\n\"))
	if err := tmpl.Execute(os.Stdout, n); err "'!= nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
}
'"'''
tmp = path.with_name(path.name + '.tmp')
with open(tmp, 'w', encoding='utf-8') as f:
    f.write(content)
    f.flush()
    os.fsync(f.fileno())
os.replace(tmp, path)
PY

gofmt -w main.go
git status --short"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
4—semedit/semantic_replace_body (✓ ✓)
{
  "body": "value, err := rand.Int(rand.Reader, big.NewInt(1000))\nif err != nil {\n\tpanic(err)\n}\nmessage, err := template.New(\"greeting\").Parse(\"Hello World {{.}}\")\nif err != nil {\n\tpanic(err)\n}\nif err := message.Execute(os.Stdout, value); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
5—semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "crypto/rand",
    "math/big",
    "os",
    "text/template"
  ],
  "file": "main.go"
}
6—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency23.73s64.38s+171.3%———
Process Start → First Event1.60s0.34s————
First Event → First Tool Call6.99s15.09s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles35+66.7%———
Initial Load / Discovery Turns21-50.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)35+66.7%———
Output Tokens558654+17.2%———
Reasoning / Thinking Tokens187244+30.5%———
Total Input Tokens76517155161+102.8%———
Cached Input Tokens68608138496+101.9%———
Uncached Input Tokens790916665+110.7%———
Cost0.04620.0875+89.3%———
Cached vs Uncached Token Ratio8.67:18.31:1-4.2%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 12 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 15 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'go.mod' -g 'go.sum'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'README*' -g 'go.mod' -g 'go.sum'"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
3✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os

path = Path('main.go')
tmp = path.with_suffix('.go.tmp')
content = '''package main

import (
	\"math/rand\"
	\"os\"
	\"text/template\"
)

func main() {
	tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\\\n\"))
	_ = tmpl.Execute(os.Stdout, rand.Intn(100))
}
'''
with tmp.open('w') as f:
    f.write(content)
    f.flush()
    os.fsync(f.fileno())
os.replace(tmp, path)
PY
gofmt -w main.go"
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": false,
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err != nil {\n\tlog.Fatal(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4—semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "log",
    "math/rand",
    "os",
    "text/template"
  ],
  "file": "main.go"
}
5—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency24.45s37.82s+54.6%———
Process Start → First Event0.25s0.84s————
First Event → First Tool Call13.78s14.68s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles24+100.0%———
Initial Load / Discovery Turns220%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)24+100.0%———
Output Tokens318417+31.1%———
Reasoning / Thinking Tokens00————
Total Input Tokens75851141424+86.4%———
Cached Input Tokens66560126464+90.0%———
Uncached Input Tokens929114960+61.0%———
Cost0.04380.0742+69.3%———
Cached vs Uncached Token Ratio7.16:18.45:1+18.0%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSFAIL————
Oracle L3: Clean BuildPASSFAIL————
Oracle L4: Verification TestPASSFAIL————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 12 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 16 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'go.mod' -g 'go.sum' -g 'README*' -g 'AGENTS*'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'README*' -g 'go.mod' -g 'go.sum'"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod'
✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
cat main.go'
3—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err != nil {\n\tfmt.Fprintln(os.Stderr, err)\n\tos.Exit(1)\n}",
  "file": "main.go",
  "symbol": "main"
}
4—✓ ✓
/bin/zsh -lc 'cat main.go && \
go build ./...'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency32.77s111.26s+239.5%———
Process Start → First Event0.44s0.24s————
First Event → First Tool Call15.24s33.59s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles38+166.7%———
Initial Load / Discovery Turns31-66.7%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)38+166.7%———
Output Tokens526989+88.0%———
Reasoning / Thinking Tokens233348+49.4%———
Total Input Tokens75999185389+143.9%———
Cached Input Tokens66560167936+152.3%———
Uncached Input Tokens943917453+84.9%———
Cost0.04970.1023+105.8%———
Cached vs Uncached Token Ratio7.05:19.62:1+36.5%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 20 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'go.mod' -g 'go.sum' -g 'README*' -g 'Makefile' -g 'AGENTS*'"
✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'Makefile' -g 'main.go' -g 'AGENTS.md' -g 'AGENTS.override.md'"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
3✓ ✓
/bin/zsh -lc "cat > main.go <<'EOF'
package main

import (
	\"math/rand\"
	\"os\"
	\"text/template\"
)

func main() {
	tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))
	if err := tmpl.Execute(os.Stdout, rand.Int()); err "'!= nil {
		panic(err)
	}
}
EOF
gofmt -w main.go'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Int()); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
5—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "n, err := rand.Int(rand.Reader, big.NewInt(100))\nif err != nil {\n\tpanic(err)\n}\ntmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, n.Int64()); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
6—semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "text/template"
  ],
  "file": "main.go",
  "remove": [
    "html/template"
  ]
}
7—✓ ✓
/bin/zsh -lc 'git diff -- main.go'
8—✓ ✓
/bin/zsh -lc 'go run .'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency34.41s49.35s+43.4%———
Process Start → First Event0.19s0.14s————
First Event → First Tool Call13.15s14.29s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles48+100.0%———
Initial Load / Discovery Turns220%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)48+100.0%———
Output Tokens813869+6.9%———
Reasoning / Thinking Tokens183308+68.3%———
Total Input Tokens95994170969+78.1%———
Cached Input Tokens85760151808+77.0%———
Uncached Input Tokens1023419161+87.2%———
Cost0.05950.1006+69.1%———
Cached vs Uncached Token Ratio8.38:17.92:1-5.5%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 15 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'go.mod' -g 'go.sum' -g 'README*' -g 'Makefile' -g 'AGENTS*'"
✓ ✓
/bin/zsh -lc "if [ -f PROJECT_BRIEF.md ]; then cat PROJECT_BRIEF.md; else printf 'No PROJECT_BRIEF.md\\n'; fi"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod && \
git status --short'
✓ ✓
/bin/zsh -lc "rg --files -g 'main.go' -g 'README*' -g 'AGENTS*' -g 'go.mod' -g 'go.sum'"
3✓ ✓
/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile, time; p=Path(\"main.go\"); data=\"package main\\n\\nimport (\\n\\t\" + chr(34) + \"math/rand\" + chr(34) + \"\\n\\t\" + chr(34) + \"os\" + chr(34) + \"\\n\\t\" + chr(34) + \"text/template\" + chr(34) + \"\\n)\\n\\nfunc main() {\\n\\ttmpl := template.Must(template.New(\" + chr(34) + \"greeting\" + chr(34) + \").Parse(\" + chr(34) + \"Hello World {{.}}\\\\n\" + chr(34) + \"))\\n\\t_ = tmpl.Execute(os.Stdout, rand.Int())\\n}\\n\"; fd, name=tempfile.mkstemp(dir=p.parent, prefix=\".main.go.\"); f=os.fdopen(fd,\"w\"); f.write(data); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(name,p); os.utime(p,ns=(time.time_ns(),time.time_ns()))' && \
cat main.go && \
git diff -- main.go && \
git status --short"
semedit/semantic_outline (✓ ✓)
{
  "path": "main.go"
}
4✓ ✓
/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile, time; p=Path(\"main.go\"); data=p.read_text().replace(\"\\t_ = tmpl.Execute(os.Stdout, rand.Int())\", \"\\tif err := tmpl.Execute(os.Stdout, rand.Int()); err "'!= nil {'"\\n\\t\\tpanic(err)\\n\\t}\"); fd, name=tempfile.mkstemp(dir=p.parent, prefix=\".main.go.\"); f=os.fdopen(fd,\"w\"); f.write(data); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(name,p); os.utime(p,ns=(time.time_ns(),time.time_ns()))'"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
5—semedit/semantic_replace_body (✓ ✓)
{
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\n\tif err := tmpl.Execute(os.Stdout, rand.Int()); err != nil {\n\t\tlog.Fatal(err)\n\t}",
  "file": "main.go",
  "symbol": "main"
}
6—semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "log",
    "math/rand",
    "os",
    "text/template"
  ],
  "file": "main.go"
}
7—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
8—✓ ✓
/bin/zsh -lc 'git diff -- main.go go.mod go.sum'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency25.05s43.06s+71.9%———
Process Start → First Event0.19s0.19s————
First Event → First Tool Call13.92s14.19s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles24+100.0%———
Initial Load / Discovery Turns21-50.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)24+100.0%———
Output Tokens450564+25.3%———
Reasoning / Thinking Tokens151190+25.8%———
Total Input Tokens76036114097+50.1%———
Cached Input Tokens6656099072+48.8%———
Uncached Input Tokens947615025+58.6%———
Cost0.04780.0718+50.0%———
Cached vs Uncached Token Ratio7.02:16.59:1-6.1%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 17 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'Makefile' -g 'go.mod' -g 'AGENTS.md' -g 'AGENTS.override.md'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'Makefile' -g 'README*' -g 'AGENTS.override.md'"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod && \
git status --short'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
3—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": false,
  "body": "tmpl, err := template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\")\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err != nil {\n\t\tpanic(err)\n\t}",
  "file": "main.go",
  "symbol": "main"
}
4—semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "math/rand",
    "os",
    "text/template"
  ],
  "file": "main.go"
}
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency35.93s43.59s+21.3%———
Process Start → First Event0.13s0.13s————
First Event → First Tool Call18.46s19.77s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles330%———
Initial Load / Discovery Turns31-66.7%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)330%———
Output Tokens692708+2.3%———
Reasoning / Thinking Tokens288344+19.4%———
Total Input Tokens190813133626-30.0%———
Cached Input Tokens169216113408-33.0%———
Uncached Input Tokens2159720218-6.4%———
Cost0.10850.0920-15.2%———
Cached vs Uncached Token Ratio7.84:15.61:1-28.4%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 15 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 17 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short'
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'go.mod' -g 'tools/benchmark-harness/README.md' && \
sed -n '1,160p' main.go"
2✓ ✓
/bin/zsh -lc 'git diff -- main.go go.mod go.sum && \
git status --short'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl, err := template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\")\nif err != nil {\n\tpanic(err)\n}\nif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
3? ✗
/bin/zsh -lc 'cat main.go; git ls-files --error-unmatch main.go'
✓ ✓
/bin/zsh -lc "sed -n '1,100p' main.go && \
go run main.go"
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency25.18s43.97s+74.6%———
Process Start → First Event0.27s0.13s————
First Event → First Tool Call10.94s18.25s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles24+100.0%———
Initial Load / Discovery Turns12+100.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)24+100.0%———
Output Tokens955708-25.9%———
Reasoning / Thinking Tokens349224-35.8%———
Total Input Tokens101314162506+60.4%———
Cached Input Tokens89856141824+57.8%———
Uncached Input Tokens1145820682+80.5%———
Cost0.06740.0988+46.6%———
Cached vs Uncached Token Ratio7.84:16.86:1-12.6%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 24 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' && \
git status --short && \
cat main.go"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'PROJECT_BRIEF.md' -g 'README*' -g 'go.mod' -g 'go.sum' -g 'AGENTS*'"
2✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os
import tempfile

path = Path('main.go')
content = '''package main

import (
\\t\"math/rand\"
\\t\"os\"
\\t\"text/template\"
)

func main() {
\\ttmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\\\n\"))
\\tif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err "'!= nil {
'"\\t\\tpanic(err)
\\t}
}
'''
fd, temporary = tempfile.mkstemp(prefix='.main.go.', dir='.')
try:
    with os.fdopen(fd, 'w') as stream:
        stream.write(content)
        stream.flush()
        os.fsync(stream.fileno())
    os.replace(temporary, path)
except BaseException:
    try:
        os.unlink(temporary)
    except FileNotFoundError:
        pass
    raise
PY
gofmt -w main.go
cat main.go
git status --short"
✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod'
3—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl, err := template.New(\"greeting\").Parse(\"Hello World {{.Random}}\\n\")\nif err != nil {\n\tpanic(err)\n}\n\ndata := struct {\n\tRandom int\n}{\n\tRandom: rand.Intn(100),\n}\n\nif err := tmpl.Execute(os.Stdout, data); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4—✓ ✓
/bin/zsh -lc 'cat main.go && \
go run . && \
git diff -- go.mod go.sum && \
git status --short'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency33.46s43.10s+28.8%———
Process Start → First Event0.13s0.14s————
First Event → First Tool Call8.81s6.04s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles45+25.0%———
Initial Load / Discovery Turns21-50.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)45+25.0%———
Output Tokens1351784-42.0%———
Reasoning / Thinking Tokens417292-30.0%———
Total Input Tokens135448167461+23.6%———
Cached Input Tokens120320136704+13.6%———
Uncached Input Tokens1512830757+103.3%———
Cost0.09000.1245+38.4%———
Cached vs Uncached Token Ratio7.95:14.44:1-44.1%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'README*' -g 'AGENTS*' -g '"'!go.mod'"' -g '"'!go.sum'"'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g 'go.sum' | sort"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short -- main.go go.mod go.sum'
semedit/semantic_outline (✓ ✓)
{
  "language": "go",
  "path": "main.go"
}
3? ✗
/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"main.go\"); data=\"package main\\n\\nimport (\\n\\t\\\"math/rand\\\"\\n\\t\\\"os\\\"\\n\\t\\\"text/template\\\"\\n)\\n\\nfunc main() {\\n\\ttmpl := template.Must(template.New(\\\"greeting\\\").Parse(\"Hello World {{.}}\\\\n\"))\\n\\tif err := tmpl.Execute(os.Stdout, rand.Int()); err "'!= nil {'"\\n\\t\\tpanic(err)\\n\\t}\\n}\\n\"; fd, name=tempfile.mkstemp(dir=p.parent, prefix=\".main.go.\"); f=os.fdopen(fd,\"w\"); f.write(data); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(name,p)'"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
4✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os
import tempfile

path = Path(\"main.go\")
content = '''package main

import (
	\"math/rand\"
	\"os\"
	\"text/template\"
)

func main() {
	tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\\\n\"))
	if err := tmpl.Execute(os.Stdout, rand.Int()); err "'!= nil {
		panic(err)
	}
}
'"'''
fd, temp_path = tempfile.mkstemp(dir=path.parent, prefix=\".main.go.\")
with os.fdopen(fd, \"w\") as file:
    file.write(content)
    file.flush()
    os.fsync(file.fileno())
os.replace(temp_path, path)
PY
cat main.go
git status --short -- main.go go.mod go.sum"
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"hello\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
5—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency18.84s63.92s+239.3%———
Process Start → First Event0.18s0.16s————
First Event → First Tool Call4.94s5.11s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles27+250.0%———
Initial Load / Discovery Turns21-50.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)27+250.0%———
Output Tokens5801831+215.7%———
Reasoning / Thinking Tokens2501092+336.8%———
Total Input Tokens104575298414+185.4%———
Cached Input Tokens86016270848+214.9%———
Uncached Input Tokens1855927566+48.5%———
Cost0.07830.1732+121.2%———
Cached vs Uncached Token Ratio4.63:19.83:1+112.0%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 11 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 26 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'go.mod' -g 'go.sum' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'PROJECT_BRIEF.md'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'PROJECT_BRIEF.md' -g 'README.md' -g 'AGENTS.md' -g 'AGENTS.override.md'"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod && \
git status --short'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
3—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl, err := template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\")\nif err != nil {\n\tfmt.Fprintln(os.Stderr, err)\n\treturn\n}\nif err := tmpl.Execute(os.Stdout, rand.Int()); err != nil {\n\tfmt.Fprintln(os.Stderr, err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
5—semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "text/template",
    "math/big"
  ],
  "file": "main.go",
  "remove": [
    "html/template"
  ]
}
6—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl, err := template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\")\nif err != nil {\n\tfmt.Fprintln(os.Stderr, err)\n\treturn\n}\nnumber, err := rand.Int(rand.Reader, big.NewInt(100))\nif err != nil {\n\tfmt.Fprintln(os.Stderr, err)\n\treturn\n}\nif err := tmpl.Execute(os.Stdout, number); err != nil {\n\tfmt.Fprintln(os.Stderr, err)\n}",
  "file": "main.go",
  "symbol": "main"
}
7—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency30.49s45.47s+49.1%———
Process Start → First Event0.14s0.13s————
First Event → First Tool Call14.62s15.58s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles34+33.3%———
Initial Load / Discovery Turns220%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)34+33.3%———
Output Tokens722665-7.9%———
Reasoning / Thinking Tokens227277+22.0%———
Total Input Tokens88022130083+47.8%———
Cached Input Tokens73728111360+51.0%———
Uncached Input Tokens1429418723+31.0%———
Cost0.06600.0864+30.9%———
Cached vs Uncached Token Ratio5.16:15.95:1+15.3%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'PROJECT_BRIEF.md'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'PROJECT_BRIEF.md' -g 'AGENTS*.md' -g 'README*' -g 'go.mod' -g 'go.sum' | sort"
2? ✗
/bin/zsh -lc "cat main.go && \
git status --short && \
rg --files -g 'PROJECT_BRIEF.md'"
✓ ✓
/bin/zsh -lc 'cat main.go && \
cat AGENTS.override.md'
3✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os
import tempfile

path = Path('main.go')
content = '''package main

import (
	\"math/rand\"
	\"os\"
	\"text/template\"
)

func main() {
	tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\\\n\"))
	if err := tmpl.Execute(os.Stdout, rand.Intn(100)); err "'!= nil {
		panic(err)
	}
}
'"'''
with tempfile.NamedTemporaryFile('w', dir=path.parent, prefix='.main.go.', delete=False) as tmp:
    tmp.write(content)
    tmp.flush()
    os.fsync(tmp.fileno())
    temp_path = Path(tmp.name)
os.replace(temp_path, path)
fd = os.open(path.parent, os.O_RDONLY)
try:
    os.fsync(fd)
finally:
    os.close(fd)
PY
gofmt -w main.go
cat main.go
git status --short"
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "t := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := t.Execute(os.Stdout, rand.Intn(100)); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4—✓ ✓
/bin/zsh -lc 'cat main.go && \
go run main.go'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

Test case: task-09-composite-refactor

Target: codex/gpt-6-luna/medium

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-09-composite-refactor
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_09_composite_refactor.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package config

import (
	"archive/tar"
	"fmt"
)

type Config struct {
	Host string
	Port int
}

func (c *Config) Addr() string {
	return fmt.Sprintf("%s:%d", c.Host, c.Port)
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency25.81s64.65s+150.5%29.68s74.58s+151.2%
Process Start → First Event0.12s0.16s—0.15s0.15s—
First Event → First Tool Call4.57s5.40s—4.78s5.38s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles712+71.4%711+57.1%
Initial Load / Discovery Turns41-75.0%51-80.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)712+71.4%711+57.1%
Output Tokens7941422+79.1%8601590+84.9%
Reasoning / Thinking Tokens147472+221.1%171857+401.2%
Total Input Tokens155420293570+88.9%155600327216+110.3%
Cached Input Tokens142336270336+89.9%142336302080+112.2%
Uncached Input Tokens1308423234+77.6%1326425136+89.5%
Cost0.08010.1493+86.5%0.08160.1689+107.0%
Cached vs Uncached Token Ratio10.88:111.64:1+7.0%10.73:112.02:1+12.0%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 16 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'README.md' | head -80"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'PROJECT_BRIEF.md' -g 'config.go' -g 'README.md' | head -60"
2✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
cat config/config.go && \
rg -n '\\.Port\\b|ListenPort|DefaultConfig|Config\\{' --glob '"'!**/*_test.go'"' ."
semedit/semantic_lookup (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config"
}
3✓ ✓
/bin/zsh -lc 'rg --files'
semedit/semantic_lookup (✓ ✗): resolve Port in config/config.go: symbol not found
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Port"
}
4✓ ✓
/bin/zsh -lc 'cat config/config_test.go'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config"
}
5✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
p = Path('config/config.go')
s = '''package config

import \"fmt\"

type Config struct {
\\tHost       string
\\tListenPort int
}

func DefaultConfig() *Config {
\\treturn &Config{Host: \"localhost\", ListenPort: 8080}
}

func (c *Config) Addr() string {
\\treturn fmt.Sprintf(\"%s:%d\", c.Host, c.ListenPort)
}
'''
fd, tmp = tempfile.mkstemp(dir=p.parent, prefix='.config.go.')
try:
    with os.fdopen(fd, 'w') as f:
        f.write(s)
        f.flush()
        os.fsync(f.fileno())
    os.replace(tmp, p)
    dfd = os.open(p.parent, os.O_RDONLY)
    try:
        os.fsync(dfd)
    finally:
        os.close(dfd)
except:
    try: os.unlink(tmp)
    except FileNotFoundError: pass
    raise
PY
"
? ✗
/bin/zsh -lc "rg --files | rg '("'^|/)SUBOPTIMAL_TOOLS'"\\.md"'$|config/config'"\\.go"'$'"' && \
cat docs/SUBOPTIMAL_TOOLS.md"
6✓ ✓
/bin/zsh -lc 'git diff -- config/config.go && \
git status --short'
✓ ✓
/bin/zsh -lc 'rg --files | head -60 && \
cat config/config.go'
7✓ ✓
/bin/zsh -lc 'cat config/config.go && \
git rev-parse --show-toplevel && \
git ls-files config/config.go'
✓ ✓
/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"docs/SUBOPTIMAL_TOOLS.md\"); p.parent.mkdir(parents=True, exist_ok=True); data=\"\"\"# Suboptimal Tool Behavior\\n\\n- Tool: "'`semantic_lookup`'"\\n  Target: "'`config/config.go`'"\\n  Observed failure: Looking up the struct field by the unqualified name "'`Port` returned symbol not found.'"\\n  Workaround: Inspected "'`Config` to identify the field and will use its qualified symbol for semantic rename.'"\\n  Root cause: Field lookup requires a qualified name such as "'`Config.Port`.'"\\n\"\"\"; fd, name=tempfile.mkstemp(dir=p.parent, prefix=\".SUBOPTIMAL_TOOLS.\"); f=os.fdopen(fd,\"w\"); f.write(data); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(name,p); d=os.open(p.parent,os.O_RDONLY); os.fsync(d); os.close(d)'"
8—semedit/semantic_rename (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
9—✓ ✓
/bin/zsh -lc 'cat config/config_test.go && \
cat config/config.go'
10—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "config/config.go",
  "placement": "public_end",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}"
}
11—✓ ✓
/bin/zsh -lc 'git diff -- config/config.go docs/SUBOPTIMAL_TOOLS.md && \
git status --short'
12—✓ ✓
/bin/zsh -lc 'cat config/config.go; cat docs/SUBOPTIMAL_TOOLS.md; git rev-parse --show-toplevel; git status --short --untracked-files=all'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 19 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'Makefile' -g 'go.mod' -g 'go.sum'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'README*' | head -80"
2✓ ✓
/bin/zsh -lc "cat config/config.go && \
rg -n 'Config\\.Port|\\.Port\\b|Config\\{' --glob '*.go'"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "symbol": "Config"
}
3✓ ✓
/bin/zsh -lc 'rg --files'
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Config.Port"
}
4✓ ✓
/bin/zsh -lc "rg -n 'localhost|127\\.0\\.0\\.1|8080|ListenPort|DefaultConfig|Host|Port' --glob '*.go'"
✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'config/**' | head -40"
5✓ ✓
/bin/zsh -lc 'cat config/config_test.go'
semedit/semantic_outline (✓ ✓)
{
  "path": "config/config.go"
}
6✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
p = Path('config/config.go')
content = '''package config

import \"fmt\"

type Config struct {
\\tHost       string
\\tListenPort int
}

func DefaultConfig() *Config {
\\treturn &Config{
\\t\\tHost:       \"localhost\",
\\t\\tListenPort: 8080,
\\t}
}

func (c *Config) Addr() string {
\\treturn fmt.Sprintf(\"%s:%d\", c.Host, c.ListenPort)
}
'''
fd, name = tempfile.mkstemp(prefix='config.go.', dir=p.parent)
try:
    with os.fdopen(fd, 'w') as f:
        f.write(content)
        f.flush()
        os.fsync(f.fileno())
    os.replace(name, p)
finally:
    if os.path.exists(name):
        os.unlink(name)
PY

git diff -- config/config.go go.mod go.sum && \
git status --short"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "symbol": "Config.Addr"
}
7✓ ✓
/bin/zsh -lc "cat config/config.go && \
git status --short -- config/config.go go.mod go.sum && \
git ls-files --error-unmatch config/config.go >/dev/null 2>&1; printf 'tracked_config=%s\\n' \""'$?"'
semedit/semantic_rename (✓ ✓)
{
  "file": "config/config.go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
8—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- config/config.go config/config_test.go'
9—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "symbol": "Config"
}
10—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "config/config.go",
  "placement": "public_start",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}"
}
11—✓ ✓
/bin/zsh -lc 'git diff -- config/config.go config/config_test.go go.mod go.sum && \
git status --short'

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-09-composite-refactor
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_09_composite_refactor.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package config

import (
	"archive/tar"
	"fmt"
)

type Config struct {
	Host string
	Port int
}

func (c *Config) Addr() string {
	return fmt.Sprintf("%s:%d", c.Host, c.Port)
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency21.58s66.58s+208.5%42.81s51.73s+20.9%
Process Start → First Event0.14s0.15s—0.14s0.18s—
First Event → First Tool Call7.58s7.55s—13.87s12.81s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles512+140.0%770%
Initial Load / Discovery Turns31-66.7%71-85.7%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)512+140.0%770%
Output Tokens516972+88.4%848641-24.4%
Reasoning / Thinking Tokens25248+892.0%233164-29.6%
Total Input Tokens115549332377+187.7%177547205347+15.7%
Cached Input Tokens104960305152+190.7%156416186112+19.0%
Uncached Input Tokens1058927225+157.1%2113119235-9.0%
Cost0.05950.1596+168.3%0.10540.1047-0.7%
Cached vs Uncached Token Ratio9.91:111.21:1+13.1%7.40:19.68:1+30.7%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 19 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config.go' -g 'AGENTS.md' -g 'AGENTS.override.md'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'go.mod' -g 'go.sum'"
2✓ ✓
/bin/zsh -lc "cat config/config.go && \
rg -n \"Config\\.Port|\\.Port\\b|type Config|DefaultConfig|ListenPort\" --glob '*.go' ."
semedit/semantic_outline (✓ ✓)
{
  "path": "config/config.go"
}
3? ✗
/bin/zsh -lc 'cat config/config_test.go && \
rg -n "Default|Host|Port|config" README.md'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "symbol": "Config"
}
4✓ ✓
/bin/zsh -lc "cat > /tmp/config.go.new <<'EOF'
package config

import \"fmt\"

type Config struct {
	Host       string
	ListenPort int
}

func DefaultConfig() *Config {
	return &Config{
		Host:       \"localhost\",
		ListenPort: 8080,
	}
}

func (c *Config) Addr() string {
	return fmt.Sprintf(\"%s:%d\", c.Host, c.ListenPort)
}
EOF
python3 -c 'import os; p=\"config/config.go\"; q=\"/tmp/config.go.new\"; d=open(q,\"rb\").read(); t=p+\".tmp\"; f=open(t,\"wb\"); f.write(d); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(t,p)' && \
git diff -- config/config.go"
✓ ✓
/bin/zsh -lc 'find .. -name AGENTS.md -o -name PROJECT_BRIEF.md'
5✓ ✓
/bin/zsh -lc 'cat config/config.go && \
git status --short'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "symbol": "Config.Addr"
}
6—✓ ✓
/bin/zsh -lc "rg -n 'DefaultConfig|ListenPort|localhost|127\\.0\\.0\\.1|Port:' --glob '"'!go.mod'"' --glob '"'!go.sum'"'"
7—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config_test.go",
  "symbol": "TestDefaultConfig"
}
8—semedit/semantic_rename (✓ ✓)
{
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
9—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "config/config.go",
  "placement": "public_start",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}"
}
10—semedit/semantic_organize_imports (✓ ✓)
{
  "file": "config/config.go"
}
11—✓ ✓
/bin/zsh -lc 'git diff -- config/config.go && \
git status --short'
12—semedit/semantic_outline (✓ ✓)
{
  "path": "config/config.go"
}
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 18 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'config/config.go' -g 'docs/**' | head -80"
2✓ ✓
/bin/zsh -lc "cat config/config.go && \
rg -n '\\.Port\\b|Config\\{["'^}]*Port|ListenPort|DefaultConfig'"' --glob '"'!AGENTS.override.md'"' ."
? ✗
/bin/zsh -lc "cat config/config.go && \
rg --files | head -120 && \
rg -n 'semantic_(lookup|inspect_symbol|replace_body|insert_function|batch|verify|rename|outline)' ."
3? ✗
/bin/zsh -lc "rg -n 'Host:|ListenPort|DefaultConfig|8080|config.Config|config.Config\\{' --glob '"'!config/config_test.go'"' . && \
rg --files"
✓ ✓
/bin/zsh -lc 'cat config/config_test.go && \
cat go.mod'
4✓ ✓
/bin/zsh -lc 'ls -la && \
cat AGENTS.override.md && \
git status --short'
semedit/semantic_rename (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
5✓ ✓
/bin/zsh -lc 'cat config/config_test.go'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "config/config.go",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{Host: \"localhost\", ListenPort: 8080}\n}"
}
6✓ ✓
/bin/zsh -lc 'git diff -- config/config.go && \
git status --short'
semedit/semantic_organize_imports (✓ ✓)
{
  "file": "config/config.go"
}
7✓ ✓
/bin/zsh -lc 'cat config/config.go'
✓ ✓
/bin/zsh -lc 'go test ./...'

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-09-composite-refactor
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_09_composite_refactor.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package config

import (
	"archive/tar"
	"fmt"
)

type Config struct {
	Host string
	Port int
}

func (c *Config) Addr() string {
	return fmt.Sprintf("%s:%d", c.Host, c.Port)
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency27.16s54.91s+102.2%33.05s72.75s+120.1%
Process Start → First Event0.15s0.13s—0.16s0.23s—
First Event → First Tool Call5.51s5.51s—10.77s9.87s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles610+66.7%710+42.9%
Initial Load / Discovery Turns42-50.0%50-100.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)610+66.7%710+42.9%
Output Tokens836920+10.0%7981237+55.0%
Reasoning / Thinking Tokens177270+52.5%179574+220.7%
Total Input Tokens135352265604+96.2%116342228092+96.1%
Cached Input Tokens123136242176+96.7%104960203776+94.1%
Uncached Input Tokens1221623428+91.8%1138224316+113.6%
Cost0.07400.1340+81.1%0.06690.1344+100.8%
Cached vs Uncached Token Ratio10.08:110.34:1+2.6%9.22:18.38:1-9.1%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 19 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 18 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'go.mod' -g 'go.sum' -g 'docs/**' | head -80"
2✓ ✓
/bin/zsh -lc 'cat config/config.go && \
git status --short'
✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' | head -40 && \
git status --short"
3✓ ✓
/bin/zsh -lc "rg -n \"DefaultConfig|ListenPort|\\.Port|Config\\{\" --glob '"'!go.mod'"' --glob '"'!go.sum'"'"
semedit/semantic_outline (✓ ✓)
{
  "language": "go",
  "path": "config/config.go"
}
4? ✗
/bin/zsh -lc "cat config/config_test.go && \
rg -n \"config\\.Config|ListenPort|Host:\" --glob '"'!go.mod'"' --glob '"'!go.sum'"' --glob '"'!config/config_test.go'"'"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config"
}
5✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
path = Path('config/config.go')
content = '''package config

import \"fmt\"

type Config struct {
\\tHost       string
\\tListenPort int
}

func DefaultConfig() *Config {
\\treturn &Config{
\\t\\tHost:       \"localhost\",
\\t\\tListenPort: 8080,
\\t}
}

func (c *Config) Addr() string {
\\treturn fmt.Sprintf(\"%s:%d\", c.Host, c.ListenPort)
}
'''
fd, temp_path = tempfile.mkstemp(dir=path.parent, prefix='.config.go.')
try:
    with os.fdopen(fd, 'w') as f:
        f.write(content)
        f.flush()
        os.fsync(f.fileno())
    os.replace(temp_path, path)
finally:
    if os.path.exists(temp_path):
        os.unlink(temp_path)
PY
"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Addr"
}
6✓ ✓
/bin/zsh -lc 'git diff -- config/config.go go.mod go.sum && \
git status --short'
semedit/semantic_rename (✓ ✓)
{
  "auto_organize_imports": true,
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
7—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "config/config.go",
  "placement": "public_start",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{Host: \"localhost\", ListenPort: 8080}\n}"
}
8—✓ ✓
/bin/zsh -lc 'git diff -- config/config.go && \
git status --short'
9—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "DefaultConfig"
}
10—? ✗
/bin/zsh -lc 'git ls-files --error-unmatch config/config.go; git check-ignore -v config/config.go; git diff --no-index /dev/null config/config.go'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 16 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md'"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config"
}
2✓ ✓
/bin/zsh -lc 'cat config/config.go'
semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "language": "go",
  "path": "config/config.go"
}
3✓ ✓
/bin/zsh -lc "rg -n '\\.Port\\b|Port:' --glob '"'!**/*_test.go'"' ."
✓ ✓
/bin/zsh -lc "git status --short && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' | head -20"
4✓ ✓
/bin/zsh -lc 'git status --short && \
rg --files'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Addr"
}
5✓ ✓
/bin/zsh -lc 'cat config/config_test.go && \
cat go.mod'
semedit/semantic_rename (✓ ✓)
{
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
6✓ ✓
/bin/zsh -lc "python3 -c 'import os,tempfile; p=\"config/config.go\"; data=\"\"\"package config

import \"fmt\"

type Config struct {
\\tHost       string
\\tListenPort int
}

func DefaultConfig() *Config {
\\treturn &Config{Host: \"localhost\", ListenPort: 8080}
}

func (c *Config) Addr() string {
\\treturn fmt.Sprintf(\"%s:%d\", c.Host, c.ListenPort)
}
\"\"\"; d=os.path.dirname(p); fd,tmp=tempfile.mkstemp(dir=d,prefix=\".config.go.\"); f=os.fdopen(fd,\"w\"); f.write(data); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(tmp,p); dfd=os.open(d,os.O_RDONLY); os.fsync(dfd); os.close(dfd)' && \
git diff -- config/config.go && \
git status --short"
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "config/config.go",
  "placement": "after_symbol",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}",
  "target_symbol": "Config"
}
7? ✗
/bin/zsh -lc 'cat config/config.go && \
git diff --no-index /dev/null config/config.go'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config"
}
8—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "DefaultConfig"
}
9—✓ ✓
/bin/zsh -lc 'git diff -- config/config.go go.mod go.sum && \
git status --short'
10—✓ ✓
/bin/zsh -lc 'git ls-files -v -- config/config.go; git status --short --untracked-files=all -- config/config.go; git diff HEAD -- config/config.go; git check-ignore -v config/config.go'

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-09-composite-refactor
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_09_composite_refactor.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package config

import (
	"archive/tar"
	"fmt"
)

type Config struct {
	Host string
	Port int
}

func (c *Config) Addr() string {
	return fmt.Sprintf("%s:%d", c.Host, c.Port)
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency26.10s62.60s+139.9%28.88s44.84s+55.3%
Process Start → First Event0.49s0.13s—0.13s0.23s—
First Event → First Tool Call5.49s6.76s—4.81s4.73s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles512+140.0%78+14.3%
Initial Load / Discovery Turns51-80.0%53-40.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)512+140.0%78+14.3%
Output Tokens6761076+59.2%934661-29.2%
Reasoning / Thinking Tokens143277+93.7%2620-100.0%
Total Input Tokens136325216747+59.0%157637258294+63.9%
Cached Input Tokens120064192512+60.3%142336236032+65.8%
Uncached Input Tokens1626124235+49.0%1530122262+45.5%
Cost0.08090.1256+55.3%0.08880.1229+38.5%
Cached vs Uncached Token Ratio7.38:17.94:1+7.6%9.30:110.60:1+14.0%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "sed -n '1,240p' config/config.go && \
rg -n '\\.Port\\b|\\bPort:' --glob '*.go' ."
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'README*' -g 'docs/**' | head -80"
2✓ ✓
/bin/zsh -lc "rg -n 'Config\\{|DefaultConfig|config\\.Port|\\.Port\\b|ListenPort|archive/tar|tar\\.' ."
semedit/semantic_outline (✓ ✓)
{
  "language": "go",
  "path": "config/config.go"
}
3✓ ✓
/bin/zsh -lc "cat config/config_test.go && \
rg -n 'Host|8080|default' --glob '"'!config/config_test.go'"' ."
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config"
}
4✓ ✓
/bin/zsh -lc 'git diff -- config/config.go && \
git status --short'
✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md'
5? ✗
/bin/zsh -lc 'cat config/config.go && \
git status --short -- config/config.go config/config_test.go go.mod go.sum && \
git diff --no-index /dev/null config/config.go'
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Config.Port"
}
6—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Addr"
}
7—semedit/semantic_lookup (✓ ✗): resolve DefaultConfig: symbol not found
{
  "language": "go",
  "symbol": "DefaultConfig"
}
8—✓ ✓
/bin/zsh -lc "rg -n \"localhost|ListenPort|DefaultConfig|Port|config\" --glob '"'!go.mod'"' --glob '"'!go.sum'"' --glob '"'!**/*_test.go'"' ."
9—semedit/semantic_rename (✓ ✓)
{
  "auto_organize_imports": true,
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
10—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "config/config.go",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}"
}
11—semedit/semantic_outline (✓ ✓)
{
  "language": "go",
  "path": "config/config.go"
}
12—✓ ✓
/bin/zsh -lc 'git diff -- config/config.go && \
git status --short'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 19 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'Makefile' | sort && \
git status --short"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'go.mod' -g 'go.sum' -g 'SUBOPTIMAL_TOOLS.md' -g 'Makefile'"
2✓ ✓
/bin/zsh -lc "cat config/config.go && \
rg -n 'Config\\.Port|\\.Port\\b|ListenPort|DefaultConfig' --glob '*.go' --glob '"'!**/*_test.go'"' ."
✓ ✓
/bin/zsh -lc "cat config/config.go && \
rg -n 'DefaultConfig|Config\\.Port|\\.Port\\b|ListenPort' --glob '"'!**/*_test.go'"' ."
3? ✗
/bin/zsh -lc "rg -n 'Host|Port|config|default' README* docs config cmd --glob '"'!**/*_test.go'"'"
? ✗
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '"'!**/.git/**'"' ."
4✓ ✓
/bin/zsh -lc 'rg --files | head -80'
semedit/semantic_rename (✓ ✓)
{
  "file": "config/config.go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
5✓ ✓
/bin/zsh -lc 'cat config/config_test.go && \
cat AGENTS.override.md'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config"
}
6✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile

path = Path('config/config.go')
source = path.read_text()
updated = '''package config

import \"fmt\"

type Config struct {
	Host       string
	ListenPort int
}

func DefaultConfig() *Config {
	return &Config{
		Host:       \"localhost\",
		ListenPort: 8080,
	}
}

func (c *Config) Addr() string {
	return fmt.Sprintf(\"%s:%d\", c.Host, c.ListenPort)
}
'''
fd, temp_name = tempfile.mkstemp(prefix='.config.go.', dir=path.parent)
try:
    with os.fdopen(fd, 'w') as temp_file:
        temp_file.write(updated)
        temp_file.flush()
        os.fsync(temp_file.fileno())
    os.replace(temp_name, path)
finally:
    if os.path.exists(temp_name):
        os.unlink(temp_name)
PY

git diff -- config/config.go && \
git status --short"
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "config/config.go",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}"
}
7✓ ✓
/bin/zsh -lc 'cat config/config.go; git status --short --untracked-files=all'
semedit/semantic_organize_imports (✓ ✓)
{
  "file": "config/config.go"
}
8—✓ ✓
/bin/zsh -lc 'cat config/config.go && \
git status --short'

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-09-composite-refactor
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_09_composite_refactor.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package config

import (
	"archive/tar"
	"fmt"
)

type Config struct {
	Host string
	Port int
}

func (c *Config) Addr() string {
	return fmt.Sprintf("%s:%d", c.Host, c.Port)
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency35.92s64.03s+78.2%50.02s89.26s+78.5%
Process Start → First Event0.47s0.14s—0.60s0.14s—
First Event → First Tool Call14.67s6.21s—15.41s17.72s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles48+100.0%612+100.0%
Initial Load / Discovery Turns21-50.0%42-50.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)48+100.0%612+100.0%
Output Tokens906999+10.3%12381570+26.8%
Reasoning / Thinking Tokens144452+213.9%178746+319.1%
Total Input Tokens97283256057+163.2%137153304952+122.3%
Cached Input Tokens86784232960+168.4%124160275968+122.3%
Uncached Input Tokens1049923097+120.0%1299328984+123.1%
Cost0.06110.1341+119.6%0.08120.1704+109.8%
Cached vs Uncached Token Ratio8.27:110.09:1+22.0%9.56:19.52:1-0.4%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 19 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'config.go' -g '*.go' | head -80 && \
rg -n \"Config\\.Port|\\.Port\\b|ListenPort|type Config|DefaultConfig\" . --glob '*.go'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'Makefile' -g 'docs/adr/README.md' | sort && \
git status --short"
2✓ ✓
/bin/zsh -lc "cat config/config.go && \
cat config/config_test.go && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' | head -30"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "symbol": "Config"
}
3✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
path = Path('config/config.go')
content = '''package config

import \"fmt\"

type Config struct {
	Host       string
	ListenPort int
}

func DefaultConfig() *Config {
	return &Config{
		Host:       \"localhost\",
		ListenPort: 8080,
	}
}

func (c *Config) Addr() string {
	return fmt.Sprintf(\"%s:%d\", c.Host, c.ListenPort)
}
'''
fd, temp_name = tempfile.mkstemp(dir=path.parent, prefix='.config.go.')
try:
    with os.fdopen(fd, 'w') as temp:
        temp.write(content)
        temp.flush()
        os.fsync(temp.fileno())
    os.replace(temp_name, path)
    dir_fd = os.open(path.parent, os.O_RDONLY)
    try:
        os.fsync(dir_fd)
    finally:
        os.close(dir_fd)
finally:
    if os.path.exists(temp_name):
        os.unlink(temp_name)
PY

git diff -- config/config.go config/config_test.go go.mod go.sum"
semedit/semantic_outline (✓ ✓)
{
  "path": "config/config.go"
}
4✓ ✓
/bin/zsh -lc 'cat config/config.go && \
git status --short'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "symbol": "Config.Addr"
}
5—semedit/semantic_rename (✓ ✓)
{
  "auto_organize_imports": true,
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
6—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "config/config.go",
  "placement": "public_end",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}"
}
7—✓ ✓
/bin/zsh -lc 'git diff -- config/config.go && \
git status --short'
8—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "symbol": "DefaultConfig"
}
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 16 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g '"'!go.mod'"' -g '"'!go.sum'"' && \
rg -n '\\bConfig\\.Port\\b|\\.Port\\b|Config\\s*\\{' --glob '*.go' --glob '"'!**/*_test.go'"'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'config.go' -g 'SUBOPTIMAL_TOOLS.md' -g 'README.md' | head -80"
2✓ ✓
/bin/zsh -lc "cat config/config.go && \
rg -n 'config\\.Config|\\.Address\\(|\\.Port\\b|\\bPort:' --glob '*.go' --glob '"'!**/*_test.go'"'"
✓ ✓
/bin/zsh -lc "sed -n '1,240p' config/config.go && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'Makefile' -g 'AGENTS.override.md' | sort"
3? ✗
/bin/zsh -lc "rg --files -g '"'!go.mod'"' -g '"'!go.sum'"' | head -60 && \
rg -n 'default|localhost|8080|ListenPort|Host:' --glob '"'!**/*_test.go'"' --glob '"'!go.mod'"' --glob '"'!go.sum'"'"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config"
}
4✓ ✓
/bin/zsh -lc "for f in api/router.go db/db.go logging/logging.go; do sed -n '1,100p' \""'$f"; done'
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Config.Port"
}
5✓ ✓
/bin/zsh -lc "python3 - <<'PY'
import os
import tempfile
from pathlib import Path

path = Path('config/config.go')
content = '''package config

import \"fmt\"

type Config struct {
\\tHost       string
\\tListenPort int
}

func DefaultConfig() *Config {
\\treturn &Config{Host: \"localhost\", ListenPort: 8080}
}

func (c *Config) Addr() string {
\\treturn fmt.Sprintf(\"%s:%d\", c.Host, c.ListenPort)
}
'''
fd, temporary = tempfile.mkstemp(dir=path.parent, prefix='.config.go.')
try:
    with os.fdopen(fd, 'w') as stream:
        stream.write(content)
        stream.flush()
        os.fsync(stream.fileno())
    os.replace(temporary, path)
finally:
    if os.path.exists(temporary):
        os.unlink(temporary)
PY
gofmt -w config/config.go
git diff -- config/config.go && \
git status --short"
semedit/semantic_outline (✓ ✓)
{
  "language": "go",
  "path": "."
}
6✓ ✓
/bin/zsh -lc 'cat config/config.go && \
git status --short -- config/config.go go.mod go.sum && \
git diff --stat'
✓ ✓
/bin/zsh -lc "rg -n 'DefaultConfig|\\.Port|Host:|Port:' --glob '*.go' --glob '"'!**/*_test.go'"' ."
7—✓ ✓
/bin/zsh -lc 'rg --files | sort'
8—semedit/semantic_rename (✓ ✓)
{
  "auto_organize_imports": true,
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
9—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "config/config.go",
  "placement": "public_start",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}"
}
10—? ✗
/bin/zsh -lc 'make check'
11—✓ ✓
/bin/zsh -lc 'go test ./... && \
git status --short && \
git diff -- config/config.go go.mod go.sum'
12—✓ ✓
/bin/zsh -lc "sed -n '1,120p' config/config.go && \
git diff -- config/config.go && \
git status --short -- config/config.go go.mod go.sum config/config_test.go"

Test case: task-11-mixed-sink-api-migration

Target: codex/gpt-6-luna/medium

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency63.42s235.25s+270.9%177.28s——
Process Start → First Event0.15s0.14s—0.46s——
First Event → First Tool Call12.46s11.83s—5.65s——
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns15+400.0%5——
Internal Tool Cycles927+200.0%16——
Initial Load / Discovery Turns32-33.3%3——
MCP Discovery / Schema Turns00—0——
Tool Invocations (per run)927+200.0%16——
Output Tokens128110109+689.1%21877——
Reasoning / Thinking Tokens3823231+745.8%8113——
Total Input Tokens1615572675510+1556.1%1997386——
Cached Input Tokens1464322506240+1611.5%1814528——
Uncached Input Tokens15125169270+1019.1%182858——
Cost0.09521.2165+1177.7%1.2857——
Cached vs Uncached Token Ratio9.68:114.81:1+52.9%9.92:1——
Oracle L1: Mutation PolicyPASSPASS—PASS——
Oracle L2: AST InvariantsPASSPASS—PASS——
Oracle L3: Clean BuildPASSPASS—PASS——
Oracle L4: Verification TestPASSFAIL—FAIL——
Semantic Tool Invocation Verified (per run)N/APASS—N/A——
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'README*' -g 'Makefile' -g '*.go' | head -200"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!*test*'"' -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' | head -200"
2✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
rg --files'
✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS.md' -g '*.md' -g 'go.mod' -g '*.go'"
3✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md'
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/event.go && \
cat audit/normalize.go && \
cat audit/sink.go && \
cat service/dispatch.go && \
cat metrics/writer.go && \
cat legacy/export.go'
4✓ ✓
/bin/zsh -lc 'cat audit/event.go audit/normalize.go audit/sink.go audit/summary.go service/dispatch.go'
✓ ✓
/bin/zsh -lc 'cat service/dispatch_test.go audit/event_test.go audit/summary_test.go && \
rg --files'
5✓ ✓
/bin/zsh -lc 'cat go.mod && \
git status --short'
✓ ✓
/bin/zsh -lc 'cat audit/summary.go'
6✓ ✓
/bin/zsh -lc 'cat audit/event_test.go audit/summary_test.go service/dispatch_test.go'
semedit/semantic_insert_function (✓ ✓)
{
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
7✓ ✓
/bin/zsh -lc 'cat metrics/writer.go legacy/export.go cmd/migrate/main.go'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
8✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./... && \
git diff --check && \
git status --short && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "writeEvent := audit.Sink.Write\n\tfor _, event := range events {\n\t\tevent.Kind = audit.NormalizeKind(event.Kind)\n\t\tif event.Kind == \"probe\" {\n\t\t\tcontinue\n\t\t}\n\t\tif err := writeEvent(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
9✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go && \
git diff --stat && \
git diff --cached --stat'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
10—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
11—✓ ✓
/bin/zsh -lc 'git diff --'
12—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/normalize.go",
  "symbol": "NormalizeKind"
}
13—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff --stat && \
cat audit/normalize.go && \
cat service/dispatch.go'
14—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = NormalizeKind(event.Kind)\n\tm.Events = append(m.Events, event)\n\treturn nil",
  "file": "audit/sink.go",
  "symbol": "(*MemorySink).Write"
}
15—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
16—✓ ✓
/bin/zsh -lc 'git status --short'
17—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify"
}
18—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/normalize.go",
  "symbol": "NormalizeKind"
}
19—semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\ncase \"health\", \"probe\":\n\treturn \"control\"\ndefault:\n\treturn \"data\"\n}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
20—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
21—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
22—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
23—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "service"
}
24—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify"
}
25—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/normalize.go",
  "symbol": "NormalizeKind"
}
26—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
27—✓ ✓
/bin/zsh -lc 'go test ./...'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 20 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'README*' -g 'go.mod' -g 'go.sum' -g '*.go' | head -120"
not published
2✓ ✓
/bin/zsh -lc "cat PROJECT_BRIEF.md 2>/dev/null; cat README.md 2>/dev/null; find . -name AGENTS.md -print; sed -n '1,240p' audit/event.go; sed -n '1,240p' audit/normalize.go; sed -n '1,260p' audit/fanout.go; sed -n '1,240p' audit/sink.go; sed -n '1,220p' service/dispatch.go"
not published
3✓ ✓
/bin/zsh -lc "rg -n \"Kind|probe|Sink|Record\\(|DispatchAll|Write\\(\" --glob '*.go' .; sed -n '1,240p' audit/transaction.go; sed -n '1,220p' journal/file.go; sed -n '1,220p' db/db.go; sed -n '1,220p' metrics/writer.go; sed -n '1,220p' legacy/export.go"
not published
4✓ ✓
/bin/zsh -lc 'cat service/replay.go; cat audit/transaction.go; cat go.mod; git status --short'
not published
5✓ ✓
/bin/zsh -lc ''
not published
6✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
root = Path('.')
updates = {
'audit/normalize.go': '''// Package audit owns event normalization at the delivery boundary.
package audit

import (
\\t\"strings\"
)

// NormalizeKind returns the canonical representation used for persisted events.
func NormalizeKind(kind string) string {
\\treturn strings.ToLower(strings.TrimSpace(kind))
}

// Deliver normalizes an event and suppresses probe control traffic before persistence.
func Deliver(sink Sink, event Event) error {
\\tevent.Kind = NormalizeKind(event.Kind)
\\tif event.Kind == \"probe\" {
\\t\\treturn nil
\\t}
\\treturn sink.Write(event)
}
''',
'service/dispatch.go': '''// Package service coordinates application work with the configured audit sink.
package service

import \"example.com/auditapp/audit\"

type Dispatcher struct {
\\tsink audit.Sink
}

func NewDispatcher(sink audit.Sink) *Dispatcher {
\\treturn &Dispatcher{sink: sink}
}

func (d *Dispatcher) Record(event audit.Event) error {
\\treturn audit.Deliver(d.sink, event)
}

func DispatchAll(sink audit.Sink, events []audit.Event) error {
\\tfor _, event := range events {
\\t\\tif err := audit.Deliver(sink, event); err "'!= nil {
'"\\t\\t\\treturn err
\\t\\t}
\\t}
\\treturn nil
}
''',
'service/replay.go': '''// Package service contains replay helpers for previously captured audit events.
package service

import \"example.com/auditapp/audit\"

func Replay(sink audit.Sink, events []audit.Event) error {
\\tfor _, event := range events {
\\t\\tif err := audit.Deliver(sink, event); err "'!= nil {
'"\\t\\t\\treturn err
\\t\\t}
\\t}
\\treturn nil
}
''',
'audit/fanout.go': '''// FanoutSink delivers an event to each configured audit destination.
package audit

type FanoutSink struct {
\\tFirst  Sink
\\tSecond Sink
}

func (f FanoutSink) Write(event Event) error {
\\tevent.Kind = NormalizeKind(event.Kind)
\\tif event.Kind == \"probe\" {
\\t\\treturn nil
\\t}
\\tif err := f.First.Write(event); err "'!= nil {
'"\\t\\treturn err
\\t}
\\treturn f.Second.Write(event)
}
''',
'audit/transaction.go': '''// BufferedSink exposes transactional delivery without owning the persistence protocol.
package audit

type BufferedSink struct {
\\tSink
\\tqueue []Event
}

func (b *BufferedSink) Begin() error {
\\tb.queue = b.queue[:0]
\\treturn nil
}

func (b *BufferedSink) Commit() error {
\\tfor _, event := range b.queue {
\\t\\tif err := Deliver(b.Sink, event); err "'!= nil {
'"\\t\\t\\treturn err
\\t\\t}
\\t}
\\treturn nil
}
''',
'audit/sink.go': '''// Package audit exposes the established event sink contract and a memory implementation.
package audit

type Sink interface {
\\tWrite(Event) error
}

type TransactionalSink interface {
\\tSink
\\tBegin() error
\\tCommit() error
}

type MemorySink struct {
\\tEvents []Event
}

func NewMemorySink() *MemorySink {
\\treturn &MemorySink{}
}

func (m *MemorySink) Write(event Event) error {
\\tevent.Kind = NormalizeKind(event.Kind)
\\tif event.Kind == \"probe\" {
\\t\\treturn nil
\\t}
\\tm.Events = append(m.Events, event)
\\treturn nil
}
''',
'journal/file.go': '''// Package journal contains a separate audit implementation used by batch jobs.
package journal

import \"example.com/auditapp/audit\"

type JournalSink struct {
\\tEntries []audit.Event
}

func (j *JournalSink) Write(event audit.Event) error {
\\tevent.Kind = audit.NormalizeKind(event.Kind)
\\tif event.Kind == \"probe\" {
\\t\\treturn nil
\\t}
\\tj.Entries = append(j.Entries, event)
\\treturn nil
}

var _ audit.Sink = (*JournalSink)(nil)
'''
}
for name, content in updates.items():
    path = root / name
    fd, tmp = tempfile.mkstemp(dir=path.parent, prefix=path.name + '.')
    try:
        with os.fdopen(fd, 'w') as f:
            f.write(content)
            f.flush()
            os.fsync(f.fileno())
        os.replace(tmp, path)
        os.utime(path, None)
    finally:
        if os.path.exists(tmp): os.unlink(tmp)
PY
gofmt -w audit/normalize.go audit/fanout.go audit/transaction.go audit/sink.go service/dispatch.go service/replay.go journal/file.go
git status --short"
not published
7? ✗
/bin/zsh -lc "git diff -- audit service journal; sed -n '1,80p' audit/normalize.go; git status --short --untracked-files=all | head -80; cat .gitignore 2>/dev/null"
not published
8✓ ✓
/bin/zsh -lc "rg --files -g 'Makefile' -g 'makefile' -g 'Taskfile*' -g '*.yml' -g '*.yaml' | head -40; ls -la"
not published
9✓ ✓
/bin/zsh -lc 'go test ./...'
not published
10✓ ✓
/bin/zsh -lc "git status --short --untracked-files=no; git ls-files audit service journal go.mod go.sum; rg -n \"Normalize|Deliver|probe|Write\\(\" audit service journal"
not published
11✓ ✓
/bin/zsh -lc "git status --short; git rev-parse --show-toplevel; ls go.sum 2>/dev/null || \
true; cat go.mod; sed -n '1,120p' audit/normalize.go; sed -n '1,100p' service/dispatch.go; sed -n '1,100p' audit/fanout.go"
not published
12✓ ✓
/bin/zsh -lc "rg -n \"Classif|probe|control|NormalizeKind|Deliver\" --glob '*.go' .; rg --files audit; cat go.mod; test -f go.sum && \
cat go.sum || \
true"
not published
13✓ ✓
/bin/zsh -lc 'cat audit/summary.go; cat audit/normalize.go; cat audit/summary_test.go; git status --short'
not published
14✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
path = Path('audit/summary.go')
content = '''// Package audit groups delivery kinds for operational reporting.
package audit

func Classify(kind string) string {
\\tswitch NormalizeKind(kind) {
\\tcase \"health\", \"probe\":
\\t\\treturn \"control\"
\\tdefault:
\\t\\treturn \"data\"
\\t}
}
'''
fd, tmp = tempfile.mkstemp(dir=path.parent, prefix=path.name + '.')
try:
    with os.fdopen(fd, 'w') as f:
        f.write(content)
        f.flush()
        os.fsync(f.fileno())
    os.replace(tmp, path)
    os.utime(path, None)
finally:
    if os.path.exists(tmp):
        os.unlink(tmp)
PY
gofmt -w audit/summary.go
git diff -- audit/summary.go; git status --short -- go.mod go.sum"
not published
15✓ ✓
/bin/zsh -lc "sed -n '1,100p' service/dispatch.go; sed -n '1,100p' audit/normalize.go; cat go.mod; test -f go.sum && \
cat go.sum || \
true"
not published
16✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
path = Path('service/dispatch.go')
content = path.read_text()
old = '''func (d *Dispatcher) Record(event audit.Event) error {
\\treturn audit.Deliver(d.sink, event)
}'''
new = '''func (d *Dispatcher) Record(event audit.Event) error {
\\tevent.Kind = audit.NormalizeKind(event.Kind)
\\treturn d.sink.Write(event)
}'''
if old not in content:
    raise SystemExit('expected Dispatcher.Record body not found')
content = content.replace(old, new, 1)
fd, tmp = tempfile.mkstemp(dir=path.parent, prefix=path.name + '.')
try:
    with os.fdopen(fd, 'w') as f:
        f.write(content)
        f.flush()
        os.fsync(f.fileno())
    os.replace(tmp, path)
    os.utime(path, None)
finally:
    if os.path.exists(tmp):
        os.unlink(tmp)
PY
gofmt -w service/dispatch.go
go test ./..."
not published

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency127.03s235.00s+85.0%166.40s248.29s+49.2%
Process Start → First Event0.13s0.18s—0.13s0.13s—
First Event → First Tool Call12.47s13.55s—12.88s4.48s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns550%550%
Internal Tool Cycles932+255.6%1535+133.3%
Initial Load / Discovery Turns220%43-25.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)932+255.6%1535+133.3%
Output Tokens883712920+46.2%1338620639+54.2%
Reasoning / Thinking Tokens30675255+71.3%53659283+73.0%
Total Input Tokens12746042897545+127.3%19331274812292+148.9%
Cached Input Tokens11814402711296+129.5%18132484570368+152.1%
Uncached Input Tokens93164186249+99.9%119879241924+101.8%
Cost0.67711.3706+102.4%0.98742.1214+114.9%
Cached vs Uncached Token Ratio12.68:114.56:1+14.8%15.13:118.89:1+24.9%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestFAILPASS—FAILFAIL—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 11 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS*' -g '*.go' -g 'Makefile' | head -100"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS*' -g '*.go' -g 'go.mod' -g 'Makefile' | head -120"
2✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
rg --files'
✓ ✓
/bin/zsh -lc 'cat PROJECT_BRIEF.md 2>/dev/null; cat AGENTS.override.md; rg --files'
3✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/event.go audit/normalize.go audit/sink.go service/dispatch.go'
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/normalize.go && \
cat audit/event.go && \
cat audit/sink.go && \
cat service/dispatch.go'
4✓ ✓
/bin/zsh -lc 'cat audit/event_test.go audit/summary.go audit/summary_test.go service/dispatch_test.go metrics/writer.go legacy/export.go cmd/migrate/main.go go.mod'
✓ ✓
/bin/zsh -lc 'cat audit/event_test.go audit/summary.go audit/summary_test.go service/dispatch_test.go metrics/writer.go legacy/export.go'
5✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./... && \
git status --short && \
git diff --check && \
git diff -- go.mod go.sum'
✓ ✓
/bin/zsh -lc 'cat go.mod'
6✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- audit service metrics legacy && \
git diff -- go.mod go.sum && \
cat audit/normalize.go audit/summary.go service/dispatch.go'
semedit/semantic_outline (✓ ✓)
{
  "path": "service/dispatch.go"
}
7✓ ✓
/bin/zsh -lc 'gofmt -w audit/sink.go && \
go test ./... && \
git diff --check && \
git status --short && \
git diff -- go.mod go.sum'
semedit/semantic_outline (✓ ✓)
{
  "path": "audit/normalize.go"
}
8✓ ✓
/bin/zsh -lc "git status --short && \
sed -n '1,180p' service/dispatch.go && \
sed -n '1,100p' audit/normalize.go && \
git diff -- go.mod go.sum"
semedit/semantic_insert_function (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
9✓ ✓
/bin/zsh -lc 'gofmt -w service/dispatch.go && \
go test ./... && \
git diff --check && \
git diff -- go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
10—semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tevent.Kind = audit.NormalizeKind(event.Kind)\n\t\tif event.Kind == \"probe\" {\n\t\t\tcontinue\n\t\t}\n\t\tif err := sink.Write(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
11—semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
12—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
13—✓ ✓
/bin/zsh -lc 'go test ./...'
14—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff --'
15—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/event.go audit/sink.go service/dispatch.go audit/summary.go && \
git diff -- audit service go.mod go.sum'
16—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = NormalizeKind(event.Kind)\n\tm.Events = append(m.Events, event)\n\treturn nil",
  "file": "audit/sink.go",
  "symbol": "(*MemorySink).Write"
}
17—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
18—✓ ✓
/bin/zsh -lc 'go test ./...'
19—✓ ✓
/bin/zsh -lc 'git diff -- audit/sink.go && \
git diff --exit-code -- go.mod go.sum'
20—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify"
}
21—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/normalize.go",
  "symbol": "NormalizeKind"
}
22—semedit/semantic_verify (✓ ✓)
{
  "path": "audit"
}
23—✓ ✓
/bin/zsh -lc 'go test ./audit'
24—✓ ✓
/bin/zsh -lc 'git diff --exit-code -- go.mod go.sum'
25—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Record"
}
26—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
27—semedit/semantic_verify (✓ ✓)
{
  "path": "service"
}
28—✓ ✓
/bin/zsh -lc 'go test ./service && \
git diff --exit-code -- go.mod go.sum'
29—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
30—✓ ✓
/bin/zsh -lc 'go test ./...'
31—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
32—✓ ✓
/bin/zsh -lc 'git diff --exit-code -- go.mod go.sum'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 18 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 12 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g '*.go' -g 'Makefile' -g 'README*' | head -100"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS.md' -g 'go.mod' -g 'go.sum' -g '*.go' | head -150"
2✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
rg -n "probe|EventKind|Kind|Deliver|Persist|Sink|Normalize|audit" audit service db'
✓ ✓
/bin/zsh -lc "rg --files | head -100 && \
cat PROJECT_BRIEF.md 2>/dev/null; sed -n '1,240p' audit/event.go; sed -n '1,240p' audit/sink.go; sed -n '1,240p' audit/normalize.go; sed -n '1,240p' audit/fanout.go"
3✓ ✓
/bin/zsh -lc 'cat audit/{normalize.go,event.go,sink.go,fanout.go,transaction.go,summary.go} service/{dispatch.go,replay.go} db/db.go metrics/writer.go legacy/export.go'
✓ ✓
/bin/zsh -lc "cat docs/audit-sinks.md; sed -n '1,240p' service/dispatch.go; sed -n '1,200p' audit/transaction.go; sed -n '1,200p' metrics/writer.go; sed -n '1,200p' legacy/export.go; rg -n 'probe|Probe|Kind|Write\\(' --glob '*.go' --glob '"'!**/*_test.go'"'"
4✓ ✓
/bin/zsh -lc "cat go.mod; rg -n '\\.Write\\(|Classify\\(' --glob '*.go' --glob '"'!**/*_test.go'"'"
✓ ✓
/bin/zsh -lc "sed -n '1,200p' journal/file.go; sed -n '1,240p' service/dispatch_test.go; sed -n '1,200p' audit/event_test.go; sed -n '1,120p' go.mod"
5✓ ✓
/bin/zsh -lc 'rg --files | sort'
? ✗
/bin/zsh -lc 'make check'
6✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md audit/summary_test.go'
? ✗
/bin/zsh -lc "type -a semedit || \
true; rg -n 'semantic' ."
7✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go && \
go test ./...'
semedit/semantic_insert_function (✓ ✗): validate function snippet: multiple declarations found in snippet: expected single function declaration, found 2 declarations
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}\n\nfunc IsProbe(event Event) bool {\n\treturn NormalizeKind(event.Kind) == \"probe\"\n}"
}
8✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- audit go.mod go.sum && \
cat audit/normalize.go audit/summary.go audit/sink.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go'
✓ ✓
/bin/zsh -lc 'ls docs'
9✓ ✓
/bin/zsh -lc 'gofmt -w audit/sink.go && \
go test ./audit ./service && \
git diff -- audit/sink.go go.mod go.sum && \
git status --short'
✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
p=Path('docs/SUBOPTIMAL_TOOLS.md')
s='# Suboptimal semantic tool behavior\\n\\n- Tool: "'`semantic_insert_function`'"\\n- Target: "'`audit/normalize.go`'"\\n- Observed failure: rejected a snippet containing two top-level functions because the operation accepts exactly one declaration.\\n- Workaround: insert each function separately.\\n- Root cause: function insertion validates a single declaration per call.\\n'
t=p.with_suffix('.md.tmp')
with t.open('w') as f:
 f.write(s); f.flush(); __import__('os').fsync(f.fileno())
__import__('os').replace(t,p)
PY"
10✓ ✓
/bin/zsh -lc "cat audit/summary.go audit/normalize.go && \
git diff -- audit/summary.go go.mod go.sum && \
rg -n 'Classify|control|probe|health' audit docs"
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
11✓ ✓
/bin/zsh -lc 'go test ./audit && \
git status --short && \
git diff -- audit/summary.go go.mod go.sum'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "source": "func IsProbe(event Event) bool {\n\treturn NormalizeKind(event.Kind) == \"probe\"\n}"
}
12✓ ✓
/bin/zsh -lc 'cat service/dispatch.go service/replay.go audit/normalize.go && \
git diff -- service audit go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "if IsProbe(event) {\n\treturn nil\n}\nevent.Kind = NormalizeKind(event.Kind)\nm.Events = append(m.Events, event)\nreturn nil",
  "file": "audit/sink.go",
  "symbol": "(*MemorySink).Write"
}
13✓ ✓
/bin/zsh -lc 'go test ./service && \
git status --short && \
git diff -- go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "if audit.IsProbe(event) {\n\treturn nil\n}\nevent.Kind = audit.NormalizeKind(event.Kind)\nj.Entries = append(j.Entries, event)\nreturn nil",
  "file": "journal/file.go",
  "symbol": "(*JournalSink).Write"
}
14✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go && \
git diff -- go.mod go.sum'
semedit/semantic_verify (✓ ✓)
{
  "language": "go"
}
15✓ ✓
/bin/zsh -lc 'gofmt -w service/dispatch.go && \
go test ./...'
✓ ✓
/bin/zsh -lc 'go test ./... && \
git status --short && \
git diff -- go.mod go.sum && \
git diff --stat && \
git diff'
16—✓ ✓
/bin/zsh -lc "sed -n '1,100p' audit/normalize.go; sed -n '15,40p' audit/sink.go; sed -n '1,30p' journal/file.go; git status --short --untracked-files=all"
17—✓ ✓
/bin/zsh -lc "git status --short; sed -n '1,140p' audit/normalize.go; sed -n '1,160p' audit/fanout.go; sed -n '1,150p' audit/sink.go; sed -n '1,100p' docs/audit-sinks.md; git diff -- go.mod go.sum"
18—semedit/semantic_replace_body (✓ ✓)
{
  "body": "m.Events = append(m.Events, event)\nreturn nil",
  "file": "audit/sink.go",
  "symbol": "(*MemorySink).Write"
}
19—semedit/semantic_replace_body (✓ ✓)
{
  "body": "j.Entries = append(j.Entries, event)\nreturn nil",
  "file": "journal/file.go",
  "symbol": "(*JournalSink).Write"
}
20—semedit/semantic_replace_body (✓ ✓)
{
  "body": "if audit.IsProbe(event) {\n\treturn nil\n}\nevent.Kind = audit.NormalizeKind(event.Kind)\nreturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
21—semedit/semantic_replace_body (✓ ✓)
{
  "body": "writeEvent := audit.Sink.Write\nfor _, event := range events {\n\tif audit.IsProbe(event) {\n\t\tcontinue\n\t}\n\tevent.Kind = audit.NormalizeKind(event.Kind)\n\tif err := writeEvent(sink, event); err != nil {\n\t\treturn err\n\t}\n}\nreturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
22—semedit/semantic_verify (✓ ✓)
{
  "language": "go"
}
23—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff -- go.mod go.sum && \
git status --short -- audit service journal metrics legacy'
24—✓ ✓
/bin/zsh -lc "sed -n '1,140p' audit/normalize.go; sed -n '1,130p' service/dispatch.go; rg -n 'Classif|Probe|probe|NormalizeKind' --glob '*.go' --glob '"'!**/*_test.go'"'; git diff -- go.mod go.sum"
25—✓ ✓
/bin/zsh -lc 'cat audit/summary.go; cat audit/summary_test.go; cat audit/event.go'
26—semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\ncase \"health\", \"probe\":\n\treturn \"control\"\ndefault:\n\treturn \"data\"\n}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
27—semedit/semantic_verify (✓ ✓)
{
  "language": "go"
}
28—✓ ✓
/bin/zsh -lc 'go test ./audit ./service && \
git diff -- go.mod go.sum'
29—✓ ✓
/bin/zsh -lc "rg -n 'sink\\.Write|Write\\(event|DispatchAll|Record\\(' service --glob '*.go' --glob '"'!**/*_test.go'"'; sed -n '1,120p' service/dispatch.go; git diff -- go.mod go.sum"
30—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\nif audit.IsProbe(event) {\n\treturn nil\n}\nreturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
31—semedit/semantic_replace_body (✓ ✓)
{
  "body": "writeEvent := audit.Sink.Write\nfor _, event := range events {\n\tevent.Kind = audit.NormalizeKind(event.Kind)\n\tif audit.IsProbe(event) {\n\t\tcontinue\n\t}\n\tif err := writeEvent(sink, event); err != nil {\n\t\treturn err\n\t}\n}\nreturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
32—semedit/semantic_verify (✓ ✓)
{
  "language": "go"
}
33—✓ ✓
/bin/zsh -lc 'go test ./service ./audit && \
git diff -- go.mod go.sum'
34—✓ ✓
/bin/zsh -lc "sed -n '1,80p' audit/normalize.go; sed -n '1,60p' audit/summary.go; sed -n '10,26p' service/dispatch.go; git diff -- go.mod go.sum"
35—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff -- go.mod go.sum'

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency148.16s——171.03s167.45s-2.1%
Process Start → First Event0.12s——0.29s0.15s—
First Event → First Tool Call9.83s——14.62s13.64s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns5——51-80.0%
Internal Tool Cycles11——1625+56.2%
Initial Load / Discovery Turns2——440%
MCP Discovery / Schema Turns0——00—
Tool Invocations (per run)11——1625+56.2%
Output Tokens14537——178891997-88.8%
Reasoning / Thinking Tokens7406——7415637-91.4%
Total Input Tokens1449813——2594921422969-83.7%
Cached Input Tokens1340160——2449920387584-84.2%
Uncached Input Tokens109653——14500135385-75.6%
Cost0.8835——1.29130.2183-83.1%
Cached vs Uncached Token Ratio12.22:1——16.90:110.95:1-35.2%
Oracle L1: Mutation PolicyPASS——PASSPASS—
Oracle L2: AST InvariantsPASS——PASSPASS—
Oracle L3: Clean BuildPASS——PASSPASS—
Oracle L4: Verification TestFAIL——FAILPASS—
Semantic Tool Invocation Verified (per run)N/A——N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' -g 'Makefile' | head -80"
not published
2✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
git status --short && \
sed -n '1,240p' audit/event.go && \
sed -n '1,240p' audit/normalize.go && \
sed -n '1,260p' audit/sink.go && \
sed -n '1,240p' service/dispatch.go"
not published
3✓ ✓
/bin/zsh -lc "rg -n \"Record\\(|DispatchAll|Kind|probe|metrics|export\" . -g '"'!**/*_test.go'"' && \
rg --files"
not published
4✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/summary.go && \
cat legacy/export.go && \
cat metrics/writer.go && \
cat service/dispatch_test.go && \
cat audit/event_test.go'
not published
5? ✗
/bin/zsh -lc 'gofmt -w service/dispatch.go audit/normalize.go && \
make check'
not published
6✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --check && \
git status --short && \
git diff -- service/dispatch.go audit/normalize.go'
not published
7✓ ✓
/bin/zsh -lc "git status --short --untracked-files=all && \
git ls-files service/dispatch.go audit/normalize.go && \
sed -n '1,100p' service/dispatch.go && \
sed -n '1,80p' audit/normalize.go"
not published
8✓ ✓
/bin/zsh -lc "sed -n '1,240p' docs/audit-sinks.md && \
sed -n '1,200p' audit/normalize.go && \
sed -n '1,220p' audit/sink.go && \
sed -n '1,220p' service/dispatch.go && \
sed -n '1,140p' audit/summary.go && \
git status --short -- go.mod go.sum audit service"
not published
9✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go && \
go test ./... && \
git diff --check && \
git status --short -- go.mod go.sum'
not published
10✓ ✓
/bin/zsh -lc "sed -n '1,220p' service/dispatch.go && \
sed -n '1,100p' audit/normalize.go && \
git status --short -- go.mod go.sum"
not published
11✓ ✓
/bin/zsh -lc 'gofmt -w service/dispatch.go && \
go test ./... && \
git diff --check && \
git status --short -- go.mod go.sum'
not published
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 29 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' -g 'Makefile' -g 'go.mod' -g 'go.sum'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g 'docs/**' | head -80"
2✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
cat go.mod && \
sed -n '1,240p' audit/event.go && \
sed -n '1,240p' audit/fanout.go && \
sed -n '1,240p' audit/normalize.go && \
sed -n '1,240p' audit/sink.go"
✓ ✓
/bin/zsh -lc 'rg --files'
3✓ ✓
/bin/zsh -lc "sed -n '1,260p' service/dispatch.go && \
sed -n '1,240p' service/replay.go && \
sed -n '1,240p' metrics/metrics.go && \
sed -n '1,240p' metrics/writer.go && \
sed -n '1,220p' legacy/export.go && \
rg -n 'Probe|probe|Kind|Write\\('"
✓ ✓
/bin/zsh -lc 'git status --short'
4✓ ✓
/bin/zsh -lc 'cat journal/file.go && \
cat audit/transaction.go && \
cat service/dispatch_test.go && \
cat audit/event_test.go'
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md'
5✓ ✓
/bin/zsh -lc "cat audit/transaction.go && \
cat audit/summary.go && \
cat db/db.go && \
rg -n 'DispatchAll|Replay\\(|NewDispatcher|FanoutSink|BufferedSink|JournalSink|NewMemorySink'"
semedit/semantic_outline (✓ ✓)
{
  "include_tests": false,
  "path": "audit"
}
6✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go service/dispatch.go service/replay.go && \
go test ./... && \
git status --short && \
git diff --check && \
git diff -- go.mod go.sum'
semedit/semantic_lookup (✓ ✗): resolve Deliver: symbol not found
{
  "symbol": "Deliver"
}
7✓ ✓
/bin/zsh -lc 'git diff -- audit/normalize.go service/dispatch.go service/replay.go && \
git status --short --untracked-files=all'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/sink.go",
  "symbol": "MemorySink.Write"
}
8✓ ✓
/bin/zsh -lc "git check-ignore -v audit/normalize.go service/dispatch.go service/replay.go; sed -n '1,120p' audit/normalize.go; sed -n '1,100p' service/dispatch.go; git rev-parse --show-toplevel"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/fanout.go",
  "symbol": "FanoutSink.Write"
}
9✓ ✓
/bin/zsh -lc "sed -n '1,180p' audit/normalize.go && \
sed -n '1,200p' service/dispatch.go && \
sed -n '1,160p' service/replay.go && \
sed -n '1,200p' audit/fanout.go && \
sed -n '1,180p' audit/sink.go && \
sed -n '1,120p' audit/summary.go && \
git status --short -- audit/normalize.go service/dispatch.go service/replay.go go.mod go.sum"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/transaction.go",
  "symbol": "BufferedSink.Commit"
}
10✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go && \
go test ./... && \
git diff --check && \
git status --short -- audit/normalize.go audit/summary.go service/dispatch.go service/replay.go go.mod go.sum'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify"
}
11✓ ✓
/bin/zsh -lc 'cat audit/summary.go && \
cat audit/normalize.go && \
git status --short -- audit/summary.go go.mod go.sum'
✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md'
12✓ ✓
/bin/zsh -lc 'go test ./audit && \
git diff --check && \
git status --short -- audit/summary.go go.mod go.sum'
semedit/semantic_outline (✓ ✓)
{
  "include_tests": false,
  "path": "service"
}
13✓ ✓
/bin/zsh -lc "sed -n '1,140p' service/dispatch.go && \
sed -n '1,120p' audit/normalize.go && \
git status --short -- go.mod go.sum service/dispatch.go"
semedit/semantic_outline (✓ ✓)
{
  "include_tests": false,
  "path": "journal"
}
14✓ ✓
/bin/zsh -lc 'gofmt -w service/dispatch.go && \
go test ./service && \
git diff --check && \
git status --short -- service/dispatch.go go.mod go.sum'
✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat audit/summary.go'
15✓ ✓
/bin/zsh -lc "sed -n '1,120p' audit/normalize.go && \
sed -n '1,100p' audit/summary.go && \
sed -n '1,100p' service/dispatch.go && \
git status --short -- go.mod go.sum"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Dispatcher.Record"
}
16✓ ✓
/bin/zsh -lc 'gofmt -w service/dispatch.go && \
go test ./... && \
git diff --check && \
git status --short -- go.mod go.sum'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
17—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\nreturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
18—semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\ncase \"health\", \"probe\":\n\treturn \"control\"\ndefault:\n\treturn \"data\"\n}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
19—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
20—✓ ✓
/bin/zsh -lc 'git diff --'
21—✓ ✓
/bin/zsh -lc 'git status --short'
22—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/normalize.go",
  "symbol": "NormalizeKind"
}
23—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Dispatcher.Record"
}
24—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify"
}
25—✓ ✓
/bin/zsh -lc 'git rev-parse --show-toplevel && \
git status --short --untracked-files=all && \
git check-ignore -v audit/normalize.go service/dispatch.go audit/summary.go'

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency210.54s103.24s-51.0%120.03s——
Process Start → First Event0.24s0.12s—0.15s——
First Event → First Tool Call5.59s7.23s—13.50s——
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns51-80.0%2——
Internal Tool Cycles1419+35.7%10——
Initial Load / Discovery Turns72-71.4%1——
MCP Discovery / Schema Turns00—0——
Tool Invocations (per run)1419+35.7%10——
Output Tokens115422005-82.6%6110——
Reasoning / Thinking Tokens5899831-85.9%3010——
Total Input Tokens1642539320423-80.5%658646——
Cached Input Tokens1539072290048-81.2%608256——
Uncached Input Tokens10346730375-70.6%50390——
Cost0.86140.1839-78.7%0.3920——
Cached vs Uncached Token Ratio14.88:19.55:1-35.8%12.07:1——
Oracle L1: Mutation PolicyPASSPASS—PASS——
Oracle L2: AST InvariantsPASSPASS—PASS——
Oracle L3: Clean BuildPASSPASS—PASS——
Oracle L4: Verification TestFAILPASS—PASS——
Semantic Tool Invocation Verified (per run)N/APASS—N/A——
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 12 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!*test*'"' -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'Makefile' -g '*.go'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '*.go' -g 'README*' | head -100"
2✓ ✓
/bin/zsh -lc "rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'README*' -g 'Makefile' -g '*.go' -g 'go.mod'"
✓ ✓
/bin/zsh -lc "cat PROJECT_BRIEF.md 2>/dev/null; rg --files -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'README*' -g '*.go'"
3✓ ✓
/bin/zsh -lc 'cat audit/event.go audit/normalize.go audit/sink.go service/dispatch.go metrics/writer.go legacy/export.go'
semedit/semantic_outline (✓ ✓)
{
  "path": "."
}
4✓ ✓
/bin/zsh -lc "cat audit/event_test.go audit/summary.go audit/summary_test.go service/dispatch_test.go && \
rg -n \"Normalize|probe|DispatchAll|Record|Transactional|Write\\(\""
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/sink.go",
  "symbol": "MemorySink.Write"
}
5? ✗
/bin/zsh -lc 'git diff --check && \
git diff -- audit/normalize.go service/dispatch.go && \
make check'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Dispatcher.Record"
}
6✓ ✓
/bin/zsh -lc 'go test ./... && \
git status --short && \
git diff --check && \
git diff -- go.mod go.sum'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
7✓ ✓
/bin/zsh -lc "git status --short && \
git diff -- audit/normalize.go service/dispatch.go go.mod go.sum && \
rg -n \"NormalizeKind|IsProbe|Classify|Write\\(\" audit service"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify"
}
8✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/sink.go service/dispatch.go audit/event.go; git log -1 --oneline'
✓ ✓
/bin/zsh -lc 'cat audit/event_test.go audit/summary_test.go service/dispatch_test.go'
9✓ ✓
/bin/zsh -lc 'cat audit/summary.go audit/normalize.go audit/event.go && \
git status --short && \
git diff -- go.mod go.sum'
✓ ✓
/bin/zsh -lc 'cat audit/normalize.go service/dispatch.go audit/summary.go'
10? ✗
/bin/zsh -lc 'gofmt -w audit/summary.go && \
make check'
semedit/semantic_insert_function (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
11✓ ✓
/bin/zsh -lc 'git diff --check && \
git diff -- audit/summary.go go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\ncase \"health\", \"probe\":\n\treturn \"control\"\ndefault:\n\treturn \"data\"\n}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
12✓ ✓
/bin/zsh -lc 'cat audit/summary.go && \
git status --short'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\nreturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
13✓ ✓
/bin/zsh -lc 'cat service/dispatch.go audit/normalize.go && \
git diff -- go.mod go.sum && \
git status --short'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\tevent.Kind = audit.NormalizeKind(event.Kind)\n\tif err := sink.Write(event); err != nil {\n\t\treturn err\n\t}\n}\nreturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
14✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff -- go.mod go.sum && \
git diff --check'
? ✗
/bin/zsh -lc 'make check'
15—✓ ✓
/bin/zsh -lc 'git diff --stat && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go && \
git status --short'
16—✓ ✓
/bin/zsh -lc "git diff -- . && \
cat audit/normalize.go && \
cat service/dispatch.go && \
cat audit/summary.go && \
rg --files -g 'Makefile' -g 'go.mod' -g 'go.sum'"
17—✓ ✓
/bin/zsh -lc 'git status --short --untracked-files=no && \
git diff --cached --stat && \
git diff --stat'
18—✓ ✓
/bin/zsh -lc 'go test ./...'
19—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 17 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!*vendor*'"' | head -80 && \
rg -n \"audit|probe|legacy export|metrics\" --glob '"'!go.mod'"' --glob '"'!go.sum'"' . | head -160"
not published
2✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
cat docs/audit-sinks.md && \
cat audit/normalize.go && \
cat audit/event.go && \
cat audit/sink.go && \
cat audit/fanout.go && \
cat audit/transaction.go && \
cat service/dispatch.go && \
cat service/replay.go && \
cat journal/file.go && \
cat audit/summary.go'
not published
3✓ ✓
/bin/zsh -lc 'cat audit/*_test.go service/*_test.go && \
cat metrics/metrics.go metrics/writer.go legacy/export.go && \
cat go.mod'
not published
4✓ ✓
/bin/zsh -lc "cat audit/normalize.go audit/transaction.go service/dispatch.go service/replay.go && \
rg -n '\\.Write\\(|Write\\(' --glob '*.go' --glob '"'!**/*_test.go'"' ."
not published
5✓ ✓
/bin/zsh -lc "cat > /tmp/noop <<'EOF'
EOF
"
not published
6✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go audit/transaction.go service/dispatch.go service/replay.go && \
git diff --check && \
go test ./...'
not published
7✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- audit/normalize.go audit/summary.go audit/transaction.go service/dispatch.go service/replay.go && \
git diff --name-only'
not published
8? ✗
/bin/zsh -lc 'git check-ignore -v audit/normalize.go audit/summary.go audit/transaction.go service/dispatch.go service/replay.go; git rev-parse --show-toplevel; git diff --no-index /dev/null audit/normalize.go'
not published
9✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go service/replay.go audit/transaction.go && \
git status --short --untracked-files=all && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go service/replay.go audit/transaction.go go.mod go.sum'
not published
10✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go && \
git diff --check && \
go test ./...'
not published

default prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency80.77s110.33s+36.6%46.15s232.40s+403.6%
Process Start → First Event0.12s0.15s—0.12s0.16s—
First Event → First Tool Call4.84s13.55s—5.70s5.76s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns21-50.0%15+400.0%
Internal Tool Cycles1023+130.0%832+300.0%
Initial Load / Discovery Turns220%23+50.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)1023+130.0%832+300.0%
Output Tokens40631571-61.3%133117397+1207.1%
Reasoning / Thinking Tokens1145589-48.6%4526509+1340.0%
Total Input Tokens451011427675-5.2%1920344337830+2158.9%
Cached Input Tokens412416391424-5.1%1748484056064+2219.8%
Uncached Input Tokens3859536251-6.1%17186281766+1539.5%
Cost0.26470.2155-18.6%0.10902.0173+1751.3%
Cached vs Uncached Token Ratio10.69:110.80:1+1.0%10.17:114.40:1+41.5%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g '*.go' -g 'go.mod' -g 'go.sum' -g 'Makefile' -g 'README*'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'docs/**' -g 'go.mod' -g 'go.sum' | head -80"
2✓ ✓
/bin/zsh -lc 'pwd; rg -n "event|probe|persist|sink|kind" service audit metrics legacy cmd/migrate'
✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
cat docs/audit-sinks.md && \
rg --files'
3✓ ✓
/bin/zsh -lc 'cat service/dispatch.go audit/normalize.go audit/event.go audit/sink.go audit/summary.go metrics/writer.go legacy/export.go; rg --files'
semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "path": "service"
}
4✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md; cat AGENTS.override.md; cat service/dispatch_test.go'
semedit/semantic_inspect_symbol (✓ ✗): resolve audit.Normalize: symbol not found
{
  "symbol": "audit.Normalize"
}
5? ✗
/bin/zsh -lc "cat > /tmp/do-edit.py <<'PY'
from pathlib import Path
import os, tempfile

root = Path.cwd()
changes = {
    'audit/normalize.go': '''// Package audit leaves room for normalization rules owned by the delivery boundary.\\npackage audit\\n\\nimport (\\n\\t\"strings\"\\n)\\n\\nfunc NormalizeKind(kind string) string {\\n\\treturn strings.ToLower(strings.TrimSpace(kind))\\n}\\n''',
    'service/dispatch.go': '''// Package service coordinates application work with the configured audit sink.\\npackage service\\n\\nimport \"example.com/auditapp/audit\"\\n\\ntype Dispatcher struct {\\n\\tsink audit.Sink\\n}\\n\\nfunc NewDispatcher(sink audit.Sink) *Dispatcher {\\n\\treturn &Dispatcher{sink: sink}\\n}\\n\\nfunc (d *Dispatcher) Record(event audit.Event) error {\\n\\treturn writeEvent(d.sink, event)\\n}\\n\\nfunc DispatchAll(sink audit.Sink, events []audit.Event) error {\\n\\tfor _, event := range events {\\n\\t\\tif err := writeEvent(sink, event); err "'!= nil {'"\\n\\t\\t\\treturn err\\n\\t\\t}\\n\\t}\\n\\treturn nil\\n}\\n\\nfunc writeEvent(sink audit.Sink, event audit.Event) error {\\n\\tevent.Kind = audit.NormalizeKind(event.Kind)\\n\\tif event.Kind == \"probe\" {\\n\\t\\treturn nil\\n\\t}\\n\\treturn sink.Write(event)\\n}\\n'''
}
for name, content in changes.items():
    path = root / name
    st = path.stat()
    fd, temp = tempfile.mkstemp(dir=path.parent, prefix=path.name + '.')
    try:
        with os.fdopen(fd, 'w') as f:
            f.write(content)
            f.flush()
            os.fsync(f.fileno())
        os.replace(temp, path)
        os.utime(path, ns=(st.st_atime_ns, max(st.st_mtime_ns + 1_000_000, os.stat(path).st_mtime_ns)))
    finally:
        if os.path.exists(temp):
            os.unlink(temp)
PY
python /tmp/do-edit.py"
semedit/semantic_inspect_symbol (✓ ✗): resolve audit.Sink: symbol not found
{
  "symbol": "audit.Sink"
}
6✓ ✓
/bin/zsh -lc 'python3 /tmp/do-edit.py'
semedit/semantic_inspect_symbol (✓ ✗): resolve (*audit.MemorySink).Write: invalid identifier: "(*audit.MemorySink).Write" (expected [Receiver.]Name)
{
  "symbol": "(*audit.MemorySink).Write"
}
7✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go service/dispatch.go; go test ./...; git status --short; git diff -- audit/normalize.go service/dispatch.go go.mod go.sum'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "symbol": "(*Dispatcher).Record"
}
8✓ ✓
/bin/zsh -lc 'cat audit/normalize.go; cat service/dispatch.go; git status --short --untracked-files=all'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "symbol": "DispatchAll"
}
9✓ ✓
/bin/zsh -lc "cat audit/normalize.go audit/event.go audit/summary.go service/dispatch.go; git diff -- go.mod go.sum; rg -n \"NormalizeKind|probe|Classify\" . -g '*.go'"
semedit/semantic_inspect_symbol (✓ ✗): resolve audit.Event: symbol not found
{
  "symbol": "audit.Event"
}
10✓ ✓
/bin/zsh -lc "cat > /tmp/do-edit2.py <<'PY'
from pathlib import Path
import os, tempfile
root=Path.cwd()
changes={
'audit/summary.go': '''// Package audit groups delivery kinds for operational reporting.\\npackage audit\\n\\nfunc Classify(kind string) string {\\n\\tswitch NormalizeKind(kind) {\\n\\tcase \"health\", \"probe\":\\n\\t\\treturn \"control\"\\n\\tdefault:\\n\\t\\treturn \"data\"\\n\\t}\\n}\\n''',
'service/dispatch.go': '''// Package service coordinates application work with the configured audit sink.\\npackage service\\n\\nimport \"example.com/auditapp/audit\"\\n\\ntype Dispatcher struct {\\n\\tsink audit.Sink\\n}\\n\\nfunc NewDispatcher(sink audit.Sink) *Dispatcher {\\n\\treturn &Dispatcher{sink: sink}\\n}\\n\\nfunc (d *Dispatcher) Record(event audit.Event) error {\\n\\treturn writeEvent(d.sink, event)\\n}\\n\\nfunc DispatchAll(sink audit.Sink, events []audit.Event) error {\\n\\tfor _, event := range events {\\n\\t\\tif err := writeEvent(sink, event); err "'!= nil {'"\\n\\t\\t\\treturn err\\n\\t\\t}\\n\\t}\\n\\treturn nil\\n}\\n\\nfunc writeEvent(sink audit.Sink, event audit.Event) error {\\n\\tevent.Kind = audit.NormalizeKind(event.Kind)\\n\\treturn sink.Write(event)\\n}\\n'''
}
for name, content in changes.items():
 p=root/name; st=p.stat(); fd,tmp=tempfile.mkstemp(dir=p.parent,prefix=p.name+'.')
 try:
  with os.fdopen(fd,'w') as f: f.write(content); f.flush(); os.fsync(f.fileno())
  os.replace(tmp,p); os.utime(p,ns=(st.st_atime_ns,max(st.st_mtime_ns+1000000,os.stat(p).st_mtime_ns)))
 finally:
  if os.path.exists(tmp): os.unlink(tmp)
PY
python3 /tmp/do-edit2.py
gofmt -w audit/summary.go service/dispatch.go
go test ./..."
semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "path": "audit"
}
11—semedit/semantic_inspect_symbol (✓ ✓)
{
  "symbol": "MemorySink.Write"
}
12—semedit/semantic_inspect_symbol (✓ ✓)
{
  "symbol": "Classify"
}
13—semedit/semantic_inspect_symbol (✓ ✓)
{
  "symbol": "Dispatcher.Record"
}
14—semedit/semantic_inspect_symbol (✓ ✓)
{
  "symbol": "DispatchAll"
}
15—semedit/semantic_insert_function (✓ ✓)
{
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
16—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = NormalizeKind(event.Kind)\n\tm.Events = append(m.Events, event)\n\treturn nil",
  "file": "audit/sink.go",
  "symbol": "(*MemorySink).Write"
}
17—semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
18—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
19—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- audit/normalize.go audit/sink.go audit/summary.go && \
git diff -- go.mod go.sum'
20—semedit/semantic_inspect_symbol (✓ ✓)
{
  "symbol": "NormalizeKind"
}
21—semedit/semantic_inspect_symbol (✓ ✓)
{
  "symbol": "MemorySink.Write"
}
22—semedit/semantic_inspect_symbol (✓ ✓)
{
  "symbol": "Classify"
}
23—✓ ✓
/bin/zsh -lc 'git rev-parse --show-toplevel && \
git status --short --untracked-files=all && \
git check-ignore -v audit/normalize.go audit/sink.go audit/summary.go'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 15 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!AGENTS.override.md'"' -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' -g 'Makefile' | head -200"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'docs/**' -g 'Makefile' -g '*.go' | head -100"
2✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'Makefile' -g 'go.mod' -g 'AGENTS*' | sort && \
git status --short && \
rg -n \"Audit|audit|probe|Probe|Kind|kind\" --glob '*.go'"
✓ ✓
/bin/zsh -lc "cat PROJECT_BRIEF.md 2>/dev/null; cat docs/audit-sinks.md; cat go.mod; rg -n \"Audit|Event|probe|Sink|Deliver|Persist|Kind\" --glob '*.go'"
3✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/event.go audit/sink.go audit/fanout.go audit/summary.go audit/transaction.go service/dispatch.go service/replay.go journal/file.go metrics/writer.go legacy/export.go'
✓ ✓
/bin/zsh -lc 'cat service/dispatch.go service/replay.go audit/normalize.go audit/transaction.go audit/fanout.go audit/sink.go journal/file.go'
4✓ ✓
/bin/zsh -lc 'cat audit/event_test.go audit/summary_test.go service/dispatch_test.go; rg --files'
semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "path": "service"
}
5✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md'
semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "path": "audit"
}
6✓ ✓
/bin/zsh -lc "cat > audit/normalize.go <<'EOF'
// Package audit owns normalization of event kinds at the delivery boundary.
package audit

import (
	\"strings\"
)

func NormalizeEvent(event Event) Event {
	event.Kind = strings.ToLower(strings.TrimSpace(event.Kind))
	return event
}

func NormalizeKind(kind string) string {
	return strings.ToLower(strings.TrimSpace(kind))
}
EOF
python3 - <<'PY'
p='service/dispatch.go'
s=open(p).read().replace('return d.sink.Write(event)', 'return d.sink.Write(audit.NormalizeEvent(event))').replace('if err := writeEvent(sink, event); err "'!= nil {'"', 'if err := writeEvent(sink, audit.NormalizeEvent(event)); err "'!= nil {'"')
open(p,'w').write(s)
p='service/replay.go'
s=open(p).read().replace('if err := writeEvent(sink, event); err "'!= nil {'"', 'if err := writeEvent(sink, audit.NormalizeEvent(event)); err "'!= nil {'"')
open(p,'w').write(s)
p='audit/summary.go'
s=open(p).read().replace('switch kind {\\n\\tcase \"health\":', 'switch NormalizeKind(kind) {\\n\\tcase \"health\", \"probe\":')
open(p,'w').write(s)
PY
gofmt -w audit/normalize.go audit/summary.go service/dispatch.go service/replay.go
git status --short"
semedit/semantic_insert_function (✓ ✓)
{
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
7✓ ✓
/bin/zsh -lc 'git diff -- audit/normalize.go audit/summary.go service/dispatch.go service/replay.go && \
go test ./... && \
git status --short && \
git diff -- go.mod go.sum'
semedit/semantic_outline (✓ ✓)
{
  "path": "audit/normalize.go"
}
8✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go service/replay.go && \
git check-ignore -v audit/normalize.go; git ls-files -v audit/normalize.go'
semedit/semantic_lookup (✓ ✓)
{
  "file": "audit/transaction.go",
  "symbol": "BufferedSink.Commit"
}
9—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
10—semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tevent.Kind = audit.NormalizeKind(event.Kind)\n\t\tif event.Kind == \"probe\" {\n\t\t\tcontinue\n\t\t}\n\t\tif err := sink.Write(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
11—semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tevent.Kind = audit.NormalizeKind(event.Kind)\n\t\tif event.Kind == \"probe\" {\n\t\t\tcontinue\n\t\t}\n\t\tif err := sink.Write(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/replay.go",
  "symbol": "Replay"
}
12—semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range b.queue {\n\t\tevent.Kind = NormalizeKind(event.Kind)\n\t\tif event.Kind == \"probe\" {\n\t\t\tcontinue\n\t\t}\n\t\tif err := b.Sink.Write(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "audit/transaction.go",
  "symbol": "(*BufferedSink).Commit"
}
13—? ✗
/bin/zsh -lc 'make check'
14—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- go.mod go.sum && \
git diff --stat && \
git diff -- audit/normalize.go service/dispatch.go service/replay.go audit/transaction.go'
15—✓ ✓
/bin/zsh -lc 'go test ./... && \
git status --short --untracked-files=no && \
git diff -- go.mod go.sum && \
git diff --stat && \
git diff'
16—✓ ✓
/bin/zsh -lc "cat audit/normalize.go && \
sed -n '1,35p' service/dispatch.go && \
sed -n '1,28p' audit/transaction.go && \
git rev-parse --show-toplevel && \
git status --short"
17—semedit/semantic_verify (✗ ?): This action was rejected due to unacceptable risk. Reason: Workspace-wide semantic verification performs formatting that may write files, including protected tests or other files, conflicting with the user's explicit no-test-edits constraint. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirmation. Report anyt
{
  "path": "."
}
18—? ✗
/bin/zsh -lc 'cat PROJECT_BRIEF.md && \
cat audit/normalize.go && \
cat audit/sink.go && \
cat audit/event.go && \
cat audit/transaction.go && \
cat service/dispatch.go && \
cat service/replay.go && \
git diff -- go.mod go.sum && \
git status --short'
19—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go; cat audit/sink.go; cat audit/event.go; cat audit/transaction.go; cat audit/fanout.go; cat service/dispatch.go; cat service/replay.go; git diff -- go.mod go.sum; git status --short'
20—semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "path": "audit"
}
21—semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "path": "service"
}
22—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\tif err := f.First.Write(event); err != nil {\n\t\treturn err\n\t}\n\treturn f.Second.Write(event)",
  "file": "audit/fanout.go",
  "symbol": "FanoutSink.Write"
}
23—✓ ✓
/bin/zsh -lc 'go test ./...'
24—✓ ✓
/bin/zsh -lc 'cat audit/summary.go; cat audit/normalize.go; cat audit/fanout.go; git diff -- go.mod go.sum'
25—semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "path": "audit/summary.go"
}
26—semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
27—✓ ✓
/bin/zsh -lc 'go test ./audit && \
git diff -- go.mod go.sum'
28—✓ ✓
/bin/zsh -lc 'cat service/dispatch.go; cat audit/normalize.go; git diff -- go.mod go.sum'
29—semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "path": "service/dispatch.go"
}
30—✓ ✓
/bin/zsh -lc 'go test ./service && \
git diff -- go.mod go.sum'
31—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
32—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff -- go.mod go.sum'

prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency157.45s178.62s+13.4%132.60s255.17s+92.4%
Process Start → First Event0.23s0.12s—0.13s0.13s—
First Event → First Tool Call11.96s18.39s—5.52s13.71s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns550%550%
Internal Tool Cycles1622+37.5%1243+258.3%
Initial Load / Discovery Turns220%43-25.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)1622+37.5%1243+258.3%
Output Tokens1385410975-20.8%1818621286+17.0%
Reasoning / Thinking Tokens34663401-1.9%781710733+37.3%
Total Input Tokens26738263062510+14.5%15864664544509+186.5%
Cached Input Tokens25070082872320+14.6%14356484271872+197.6%
Uncached Input Tokens166818190190+14.0%150818272637+80.8%
Cost1.26031.3733+9.0%1.06102.1498+102.6%
Cached vs Uncached Token Ratio15.03:115.10:1+0.5%9.52:115.67:1+64.6%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestFAILFAIL—PASSFAIL—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g '*audit*' -g '*.go' | head -80"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' -g 'AGENTS.md' -g 'AGENTS.override.md' | head -100 && \
git status --short"
2? ✗
/bin/zsh -lc "cat PROJECT_BRIEF.md && \
cat docs/audit-sinks.md && \
sed -n '1,240p' service/dispatch.go && \
sed -n '1,240p' audit/event.go && \
sed -n '1,200p' audit/normalize.go && \
sed -n '1,220p' audit/sink.go"
? ✗
/bin/zsh -lc "rg --files && \
sed -n '1,240p' audit/event.go && \
sed -n '1,240p' audit/normalize.go && \
sed -n '1,240p' audit/sink.go && \
sed -n '1,240p' service/dispatch.go && \
sed -n '1,200p' README.md"
3✓ ✓
/bin/zsh -lc "rg --files | head -100; cat docs/audit-sinks.md; sed -n '1,240p' service/dispatch.go; sed -n '1,240p' audit/event.go; sed -n '1,200p' audit/normalize.go; sed -n '1,220p' audit/sink.go"
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/summary.go && \
cat metrics/writer.go && \
cat legacy/export.go && \
cat go.mod'
4✓ ✓
/bin/zsh -lc "cat service/dispatch_test.go audit/event_test.go audit/summary.go metrics/writer.go legacy/export.go; cat AGENTS.override.md; rg -n 'probe|Kind|DispatchAll|Record\\('"
semedit/semantic_outline (✓ ✓)
{
  "language": "go",
  "path": "audit"
}
5✓ ✓
/bin/zsh -lc "cat > /tmp/update_dispatch.py <<'PY'
from pathlib import Path
import os, tempfile
p = Path('service/dispatch.go')
s = p.read_text()
s = s.replace('import \"example.com/auditapp/audit\"', 'import (\\n\\t\"strings\"\\n\\n\\t\"example.com/auditapp/audit\"\\n)')
s = s.replace('func (d *Dispatcher) Record(event audit.Event) error {\\n\\treturn d.sink.Write(event)\\n}', 'func (d *Dispatcher) Record(event audit.Event) error {\\n\\treturn deliver(d.sink, event)\\n}')
s = s.replace('func DispatchAll(sink audit.Sink, events []audit.Event) error {\\n\\twriteEvent := audit.Sink.Write\\n\\tfor _, event := range events {\\n\\t\\tif err := writeEvent(sink, event); err "'!= nil {'"\\n\\t\\t\\treturn err\\n\\t\\t}\\n\\t}\\n\\treturn nil\\n}', 'func DispatchAll(sink audit.Sink, events []audit.Event) error {\\n\\tfor _, event := range events {\\n\\t\\tif err := deliver(sink, event); err "'!= nil {'"\\n\\t\\t\\treturn err\\n\\t\\t}\\n\\t}\\n\\treturn nil\\n}\\n\\nfunc deliver(sink audit.Sink, event audit.Event) error {\\n\\tevent.Kind = strings.ToLower(strings.TrimSpace(event.Kind))\\n\\tif event.Kind == \"probe\" {\\n\\t\\treturn nil\\n\\t}\\n\\treturn sink.Write(event)\\n}')
fd, name = tempfile.mkstemp(dir=p.parent, prefix='.dispatch-', text=True)
with os.fdopen(fd, 'w') as f:
    f.write(s)
    f.flush()
    os.fsync(f.fileno())
os.replace(name, p)
PY
python3 /tmp/update_dispatch.py
sed -n '1,200p' service/dispatch.go
git status --short"
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "source": "func NormalizeEvent(event Event) Event {\n\tevent.Kind = strings.ToLower(strings.TrimSpace(event.Kind))\n\treturn event\n}"
}
6? ✗
/bin/zsh -lc 'make check'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event = audit.NormalizeEvent(event)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
7✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff -- service/dispatch.go && \
git status --short -- go.mod go.sum service/dispatch.go'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tevent = audit.NormalizeEvent(event)\n\t\tif event.Kind == \"probe\" {\n\t\t\tcontinue\n\t\t}\n\t\tif err := sink.Write(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
8✓ ✓
/bin/zsh -lc 'git status --short; git diff --stat; git diff --no-index /dev/null service/dispatch.go | head -80; git ls-files service/dispatch.go'
✓ ✓
/bin/zsh -lc 'go test ./... && \
git status --short && \
git diff -- audit/normalize.go service/dispatch.go go.mod go.sum && \
git diff --check'
9✓ ✓
/bin/zsh -lc "cat audit/normalize.go; cat service/dispatch.go; cat docs/audit-sinks.md; git status --short; rg -n 'Normalize|normalize|probe|DispatchAll|Record' --glob '*.go' ."
✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat service/dispatch.go && \
git status --short --untracked-files=all | head -30 && \
git diff --stat && \
git diff --check'
10✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile

def atomic_write(path, content):
    path = Path(path)
    fd, tmp = tempfile.mkstemp(dir=path.parent, prefix=f'.{path.name}-')
    try:
        with os.fdopen(fd, 'w') as stream:
            stream.write(content)
            stream.flush()
            os.fsync(stream.fileno())
        os.replace(tmp, path)
    except BaseException:
        try:
            os.unlink(tmp)
        except FileNotFoundError:
            pass
        raise

normalize = '''// Package audit defines normalization rules for events at the delivery boundary.
package audit

import \"strings\"

func NormalizeKind(kind string) string {
\\treturn strings.ToLower(strings.TrimSpace(kind))
}
'''
atomic_write('audit/normalize.go', normalize)
p = Path('service/dispatch.go')
s = p.read_text()
s = s.replace('import (\\n\\t\"strings\"\\n\\n\\t\"example.com/auditapp/audit\"\\n)', 'import \"example.com/auditapp/audit\"')
s = s.replace('event.Kind = strings.ToLower(strings.TrimSpace(event.Kind))', 'event.Kind = audit.NormalizeKind(event.Kind)')
atomic_write(p, s)
PY
gofmt -w audit/normalize.go service/dispatch.go
git diff -- audit/normalize.go service/dispatch.go
git status --short -- go.mod go.sum"
✓ ✓
/bin/zsh -lc "git diff -- audit/normalize.go service/dispatch.go go.mod go.sum && \
git status --short -- audit/normalize.go service/dispatch.go go.mod go.sum && \
sed -n '1,120p' audit/normalize.go && \
sed -n '1,140p' service/dispatch.go"
11✓ ✓
/bin/zsh -lc "go test ./... && \
git status --short -- go.mod go.sum && \
sed -n '1,100p' audit/normalize.go && \
sed -n '1,100p' service/dispatch.go"
✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --check && \
git status --short -- audit/normalize.go service/dispatch.go go.mod go.sum'
12✓ ✓
/bin/zsh -lc 'cat audit/summary.go; cat audit/normalize.go; git status --short -- go.mod go.sum'
✓ ✓
/bin/zsh -lc "git status --short -- audit/summary.go go.mod go.sum && \
sed -n '1,80p' audit/summary.go"
13✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
p = Path('audit/summary.go')
s = p.read_text().replace('switch kind {\\n\\tcase \"health\":', 'switch NormalizeKind(kind) {\\n\\tcase \"health\", \"probe\":')
fd, tmp = tempfile.mkstemp(dir=p.parent, prefix='.summary-')
try:
    with os.fdopen(fd, 'w') as f:
        f.write(s)
        f.flush()
        os.fsync(f.fileno())
    os.replace(tmp, p)
except BaseException:
    try:
        os.unlink(tmp)
    except FileNotFoundError:
        pass
    raise
PY
gofmt -w audit/summary.go
go test ./...
git status --short -- go.mod go.sum"
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "kind = strings.ToLower(strings.TrimSpace(kind))\n\tswitch kind {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
14✓ ✓
/bin/zsh -lc "sed -n '1,120p' service/dispatch.go; sed -n '1,80p' audit/normalize.go; git status --short -- go.mod go.sum"
✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --check && \
git status --short -- audit/summary.go go.mod go.sum && \
git diff -- audit/summary.go go.mod go.sum'
15✓ ✓
/bin/zsh -lc 'cat audit/normalize.go; cat audit/summary.go; cat service/dispatch.go; git status --short -- go.mod go.sum'
✓ ✓
/bin/zsh -lc "sed -n '1,140p' service/dispatch.go && \
git status --short -- service/dispatch.go go.mod go.sum && \
git diff -- service/dispatch.go go.mod go.sum"
16✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
p = Path('service/dispatch.go')
s = p.read_text().replace('func (d *Dispatcher) Record(event audit.Event) error {\\n\\treturn deliver(d.sink, event)\\n}', 'func (d *Dispatcher) Record(event audit.Event) error {\\n\\tevent.Kind = audit.NormalizeKind(event.Kind)\\n\\tif event.Kind == \"probe\" {\\n\\t\\treturn nil\\n\\t}\\n\\treturn d.sink.Write(event)\\n}')
fd, tmp = tempfile.mkstemp(dir=p.parent, prefix='.dispatch-')
try:
    with os.fdopen(fd, 'w') as f:
        f.write(s)
        f.flush()
        os.fsync(f.fileno())
    os.replace(tmp, p)
except BaseException:
    try:
        os.unlink(tmp)
    except FileNotFoundError:
        pass
    raise
PY
gofmt -w service/dispatch.go
go test ./...
git status --short -- go.mod go.sum"
✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --check && \
git status --short -- service/dispatch.go go.mod go.sum'
17—✓ ✓
/bin/zsh -lc "sed -n '1,120p' audit/normalize.go && \
sed -n '1,100p' audit/summary.go && \
sed -n '1,100p' service/dispatch.go && \
git status --short -- go.mod go.sum"
18—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "placement": "public_start",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
19—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = NormalizeKind(event.Kind)\n\treturn event",
  "file": "audit/normalize.go",
  "symbol": "NormalizeEvent"
}
20—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
21—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
22—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --check && \
git status --short -- go.mod go.sum && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go go.mod go.sum'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 19 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' -g 'go.mod' -g 'go.sum' | head -200"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'docs/**' -g 'Makefile' -g '*.go' | head -200"
2✓ ✓
/bin/zsh -lc 'rg --files'
? ✗
/bin/zsh -lc 'rg --files | sort && \
rg -n "audit|probe|event kind|delivery" PROJECT_BRIEF.md docs/audit-sinks.md audit service db 2>/dev/null'
3✓ ✓
/bin/zsh -lc "sed -n '1,240p' audit/normalize.go && \
sed -n '1,240p' audit/event.go && \
sed -n '1,260p' audit/fanout.go && \
sed -n '1,220p' audit/sink.go"
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/event.go audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go && \
cat AGENTS.override.md'
4✓ ✓
/bin/zsh -lc "cat docs/audit-sinks.md && \
sed -n '1,260p' audit/transaction.go && \
sed -n '1,260p' audit/summary.go && \
sed -n '1,220p' journal/file.go && \
sed -n '1,240p' service/dispatch.go && \
sed -n '1,200p' metrics/metrics.go && \
sed -n '1,200p' legacy/export.go"
semedit/semantic_outline (✓ ✓)
{
  "include_tests": false,
  "path": "audit"
}
5✓ ✓
/bin/zsh -lc 'cat audit/event_test.go audit/summary_test.go service/dispatch_test.go audit/transaction.go && \
cat go.mod && \
git status --short'
semedit/semantic_insert_function (✓ ✓)
{
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeEvent(event Event) Event {\n\tevent.Kind = strings.ToLower(strings.TrimSpace(event.Kind))\n\treturn event\n}"
}
6✓ ✓
/bin/zsh -lc "rg 'Write\\(|\\.Write\\(' --glob '*.go'"
semedit/semantic_insert_function (✓ ✓)
{
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func IsProbe(event Event) bool {\n\treturn NormalizeEvent(event).Kind == \"probe\"\n}"
}
7✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/fanout.go audit/transaction.go audit/sink.go service/dispatch.go journal/file.go && \
go test ./... && \
git status --short && \
git diff --'
semedit/semantic_insert_function (✓ ✓)
{
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func Deliver(sink Sink, event Event) error {\n\tevent = NormalizeEvent(event)\n\tif IsProbe(event) {\n\t\treturn nil\n\t}\n\treturn sink.Write(event)\n}"
}
8✓ ✓
/bin/zsh -lc "git status --short && \
sed -n '1,200p' audit/normalize.go && \
sed -n '1,200p' audit/sink.go && \
sed -n '1,160p' audit/fanout.go && \
sed -n '1,180p' audit/transaction.go && \
sed -n '1,160p' audit/summary.go && \
sed -n '1,120p' service/dispatch.go"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event = NormalizeEvent(event)\n\tif IsProbe(event) {\n\t\treturn nil\n\t}\n\tm.Events = append(m.Events, event)\n\treturn nil",
  "file": "audit/sink.go",
  "symbol": "(*MemorySink).Write"
}
9✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go audit/fanout.go audit/transaction.go audit/sink.go journal/file.go && \
go test ./... && \
git diff -- go.mod go.sum && \
git status --short'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event = NormalizeEvent(event)\n\tif IsProbe(event) {\n\t\treturn nil\n\t}\n\tif err := Deliver(f.First, event); err != nil {\n\t\treturn err\n\t}\n\treturn Deliver(f.Second, event)",
  "file": "audit/fanout.go",
  "symbol": "(FanoutSink).Write"
}
10✓ ✓
/bin/zsh -lc "sed -n '1,120p' audit/normalize.go && \
sed -n '1,120p' audit/summary.go && \
git status --short && \
git diff -- go.mod go.sum"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "return audit.Deliver(d.sink, event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
11✓ ✓
/bin/zsh -lc "sed -n '1,180p' service/dispatch.go && \
git status --short && \
git diff -- go.mod go.sum"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tif err := audit.Deliver(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
12✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./... && \
git diff -- go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tif err := audit.Deliver(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/replay.go",
  "symbol": "Replay"
}
13—semedit/semantic_insert_function (✓ ✓)
{
  "file": "audit/transaction.go",
  "placement": "after_symbol",
  "source": "func (b *BufferedSink) Write(event Event) error {\n\tevent = NormalizeEvent(event)\n\tif IsProbe(event) {\n\t\treturn nil\n\t}\n\tb.queue = append(b.queue, event)\n\treturn nil\n}",
  "target_symbol": "Begin"
}
14—semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range b.queue {\n\t\tif err := Deliver(b.Sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "audit/transaction.go",
  "symbol": "(*BufferedSink).Commit"
}
15—semedit/semantic_verify (✓ ✓)
{
  "path": "audit"
}
16—semedit/semantic_verify (✓ ✓)
{
  "path": "service"
}
17—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- go.mod go.sum && \
git diff -- audit service'
18—✓ ✓
/bin/zsh -lc 'go test ./...'
19—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat audit/fanout.go audit/transaction.go && \
cat service/dispatch.go service/replay.go && \
git status --short --untracked-files=all && \
git diff --stat && \
git ls-files audit/normalize.go'
20—✓ ✓
/bin/zsh -lc 'git status --short -- audit service go.mod go.sum && \
git diff -- go.mod go.sum && \
cat audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go'
21—semedit/semantic_outline (✓ ✓)
{
  "include_tests": false,
  "path": "audit"
}
22—semedit/semantic_verify (✓ ✓)
{
  "path": "audit"
}
23—✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum && \
git status --short -- audit service go.mod go.sum'
24—✓ ✓
/bin/zsh -lc 'go test ./...'
25—✓ ✓
/bin/zsh -lc 'cat audit/summary.go audit/normalize.go && \
git diff -- go.mod go.sum && \
git status --short -- audit go.mod go.sum'
26—semedit/semantic_replace_body (✓ ✓)
{
  "body": "kind = NormalizeEvent(Event{Kind: kind}).Kind\n\tswitch kind {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
27—semedit/semantic_verify (✓ ✓)
{
  "path": "audit"
}
28—✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum && \
git status --short -- audit go.mod go.sum'
29—✓ ✓
/bin/zsh -lc 'go test ./...'
30—✓ ✓
/bin/zsh -lc 'cat service/dispatch.go service/replay.go && \
git diff -- go.mod go.sum && \
git status --short -- service go.mod go.sum'
31—semedit/semantic_outline (✓ ✓)
{
  "include_tests": false,
  "kinds": [
    "function",
    "method"
  ],
  "path": "service"
}
32—semedit/semantic_verify (✓ ✓)
{
  "path": "service"
}
33—✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum && \
git status --short -- service go.mod go.sum'
34—✓ ✓
/bin/zsh -lc 'go test ./...'
35—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go && \
git diff -- go.mod go.sum'
36—semedit/semantic_insert_function (✓ ✓)
{
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
37—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = NormalizeKind(event.Kind)\n\treturn event",
  "file": "audit/normalize.go",
  "symbol": "NormalizeEvent"
}
38—semedit/semantic_replace_body (✓ ✓)
{
  "body": "kind = NormalizeKind(kind)\n\tswitch kind {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
39—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
40—✓ ✓
/bin/zsh -lc 'go test ./...'
41—semedit/semantic_verify (✓ ✓)
{
  "path": "audit"
}
42—semedit/semantic_verify (✓ ✓)
{
  "path": "service"
}
43—✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum && \
git status --short -- audit service go.mod go.sum'

prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency115.30s245.48s+112.9%173.47s——
Process Start → First Event0.12s0.13s—0.15s——
First Event → First Tool Call15.18s8.52s—19.51s——
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns25+150.0%5——
Internal Tool Cycles935+288.9%11——
Initial Load / Discovery Turns14+300.0%3——
MCP Discovery / Schema Turns00—0——
Tool Invocations (per run)935+288.9%11——
Output Tokens416315078+262.2%19812——
Reasoning / Thinking Tokens12196282+415.3%8169——
Total Input Tokens5207993663031+603.3%2682333——
Cached Input Tokens4730883455488+630.4%2512128——
Uncached Input Tokens47711207543+335.0%170205——
Cost0.30481.6497+441.2%1.4033——
Cached vs Uncached Token Ratio9.92:116.65:1+67.9%14.76:1——
Oracle L1: Mutation PolicyPASSPASS—PASS——
Oracle L2: AST InvariantsPASSPASS—PASS——
Oracle L3: Clean BuildPASSPASS—PASS——
Oracle L4: Verification TestPASSFAIL—FAIL——
Semantic Tool Invocation Verified (per run)N/APASS—N/A——
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 16 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!go.mod'"' -g '"'!go.sum'"' | head -80 && \
git status --short"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS*' -g '*.go' -g 'Makefile' | head -160"
2✓ ✓
/bin/zsh -lc "cat docs/audit-sinks.md && \
sed -n '1,240p' audit/event.go && \
sed -n '1,240p' audit/normalize.go && \
sed -n '1,240p' audit/sink.go && \
sed -n '1,240p' service/dispatch.go && \
sed -n '1,200p' audit/summary.go"
✓ ✓
/bin/zsh -lc 'rg --files'
3✓ ✓
/bin/zsh -lc "cat audit/event_test.go audit/summary_test.go service/dispatch_test.go && \
cat metrics/writer.go legacy/export.go && \
cat go.mod && \
find . -maxdepth 2 -type f -not -path './.git/*' | sort"
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/event.go && \
cat audit/normalize.go && \
cat audit/sink.go && \
cat service/dispatch.go'
4✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
git diff --check && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go && \
git status --short'
semedit/semantic_outline (✓ ✓)
{
  "include_tests": false,
  "path": "service"
}
5✓ ✓
/bin/zsh -lc "git diff --stat && \
sed -n '1,120p' audit/normalize.go && \
sed -n '1,120p' service/dispatch.go && \
git status --short --untracked-files=all && \
go test ./..."
semedit/semantic_outline (✓ ✓)
{
  "include_tests": false,
  "path": "audit"
}
6? ✗
/bin/zsh -lc 'git ls-files -v audit/normalize.go audit/summary.go service/dispatch.go; git check-ignore -v audit/normalize.go audit/summary.go service/dispatch.go || \
true; git diff --no-ext-diff --no-index /dev/null audit/normalize.go'
semedit/semantic_outline (✓ ✓)
{
  "include_tests": false,
  "path": "metrics"
}
7✓ ✓
/bin/zsh -lc "git status --short --untracked-files=all -- . ':"'!../../../data'"' && \
git diff -- go.mod go.sum && \
cat audit/event.go audit/normalize.go audit/sink.go audit/summary.go service/dispatch.go && \
cat docs/audit-sinks.md"
semedit/semantic_outline (✓ ✓)
{
  "include_tests": false,
  "path": "legacy"
}
8? ✗
/bin/zsh -lc 'gofmt -w audit/normalize.go service/dispatch.go && \
git diff --check && \
go test ./... && \
git diff -- go.mod go.sum && \
git status --short --untracked-files=all -- audit service metrics legacy go.mod go.sum'
✓ ✓
/bin/zsh -lc 'cat service/dispatch_test.go audit/event_test.go audit/summary.go'
9✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go service/dispatch.go && \
git diff --check && \
go test ./... && \
git diff -- go.mod go.sum && \
git status --short --untracked-files=all -- audit service metrics legacy go.mod go.sum'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
10—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
11—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "private",
  "file": "service/dispatch.go",
  "placement": "file_end",
  "source": "func deliver(sink audit.Sink, event audit.Event) error {\n\tevent.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn sink.Write(event)\n}"
}
12—semedit/semantic_replace_body (✓ ✓)
{
  "body": "return deliver(d.sink, event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
13—semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tif err := deliver(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
14—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
15—✓ ✓
/bin/zsh -lc 'go test ./... && \
git status --short && \
git diff -- audit/normalize.go service/dispatch.go go.mod go.sum'
16—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat service/dispatch.go && \
git status --short --untracked-files=no && \
git diff --stat && \
git diff --cached --stat'
17—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- go.mod go.sum && \
git diff -- audit/normalize.go service/dispatch.go && \
cat audit/normalize.go service/dispatch.go'
18—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/normalize.go",
  "symbol": "NormalizeKind"
}
19—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "deliver"
}
20—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
21—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --exit-code -- go.mod go.sum'
22—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- go.mod go.sum && \
cat audit/summary.go && \
cat audit/normalize.go'
23—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify"
}
24—semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
25—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
26—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --exit-code -- go.mod go.sum && \
git status --short'
27—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "deliver"
}
28—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Record"
}
29—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
30—✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum && \
git status --short'
31—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
32—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --exit-code -- go.mod go.sum'
33—✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum'
34—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
35—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --exit-code -- go.mod go.sum'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 10 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS*' -g '*.go' | head -100 && \
rg -n \"audit|probe|legacy|metrics|deliver|persist\" . --glob '"'!go.sum'"' --glob '"'!go.mod'"'"
not published
2✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/normalize.go audit/event.go audit/sink.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go journal/file.go audit/summary.go && \
git status --short'
not published
3✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/sink.go service/dispatch.go service/replay.go && \
git diff --check && \
git diff -- audit/normalize.go audit/sink.go service/dispatch.go service/replay.go && \
git status --short && \
go test ./...'
not published
4✓ ✓
/bin/zsh -lc "git status --short && \
git diff -- go.mod go.sum && \
git diff -- audit/normalize.go audit/sink.go service/dispatch.go service/replay.go && \
rg -n \"NormalizeKind|DeliverySink|DispatchAll|func Replay|Record\\(\" --glob '*.go'"
not published
5✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/sink.go service/dispatch.go service/replay.go docs/audit-sinks.md && \
git status --porcelain=v1 -- go.mod go.sum && \
git ls-files -v go.mod go.sum'
not published
6✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --check && \
git status --short -- go.mod go.sum'
not published
7✓ ✓
/bin/zsh -lc "cat audit/normalize.go audit/summary.go audit/sink.go docs/audit-sinks.md && \
git status --short -- go.mod go.sum && \
rg -n \"Classify\\(|probe|health|NormalizeKind\" --glob '*.go'"
not published
8✓ ✓
/bin/zsh -lc 'gofmt -w audit/summary.go && \
go test ./audit && \
git diff --check && \
git diff -- audit/summary.go && \
git status --short -- go.mod go.sum'
not published
9✓ ✓
/bin/zsh -lc "git status --short -- go.mod go.sum && \
cat service/dispatch.go service/replay.go audit/sink.go audit/normalize.go && \
rg -n \"NewDispatcher|DispatchAll|Replay\\(\" --glob '*.go'"
not published
10✓ ✓
/bin/zsh -lc 'go test ./service && \
git diff --check && \
git status --short -- go.mod go.sum'
not published
11✓ ✓
/bin/zsh -lc 'gofmt -w service/dispatch.go && \
go test ./... && \
git diff --check && \
git status --short -- go.mod go.sum'
not published

prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency121.87s178.14s+46.2%121.99s172.13s+41.1%
Process Start → First Event0.18s0.27s—0.13s0.12s—
First Event → First Tool Call6.25s11.86s—6.56s5.67s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns550%51-80.0%
Internal Tool Cycles1122+100.0%1814-22.2%
Initial Load / Discovery Turns34+33.3%52-60.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)1122+100.0%1814-22.2%
Output Tokens1163514479+24.4%143932680-81.4%
Reasoning / Thinking Tokens39086828+74.7%23011454-36.8%
Total Input Tokens17383693496971+101.2%2370520362597-84.7%
Cached Input Tokens16291843225344+98.0%2236672331008-85.2%
Uncached Input Tokens109185271627+148.8%13384831589-76.4%
Cost0.87451.7517+100.3%1.10250.2134-80.6%
Cached vs Uncached Token Ratio14.92:111.87:1-20.4%16.71:110.48:1-37.3%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestFAILFAIL—FAILPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g '*.go' -g 'README.md' -g 'go.mod' -g 'go.sum' | head -100"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g '*.go' | head -100"
2✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
printf '\\n--- files ---\\n' && \
for f in audit/event.go audit/normalize.go audit/sink.go service/dispatch.go metrics/writer.go legacy/export.go; do printf '\\n### %s\\n' \""'$f"; cat "$f"; done'
✓ ✓
/bin/zsh -lc 'rg --files'
3✓ ✓
/bin/zsh -lc "for f in audit/event_test.go audit/summary.go audit/summary_test.go service/dispatch_test.go; do printf '\\n### %s\\n' \""'$f"; cat "$f"; done; git status --short'
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/event.go audit/normalize.go audit/sink.go service/dispatch.go'
4✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go service/dispatch.go && \
go build ./... && \
git diff --check && \
git status --short && \
git diff -- audit/normalize.go service/dispatch.go'
✓ ✓
/bin/zsh -lc 'cat audit/event_test.go audit/summary.go audit/summary_test.go service/dispatch_test.go metrics/writer.go legacy/export.go'
5✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat service/dispatch.go && \
git status --short --untracked-files=all && \
git diff HEAD --stat && \
git ls-files audit/normalize.go service/dispatch.go'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
6✓ ✓
/bin/zsh -lc "cat audit/normalize.go && \
cat audit/sink.go && \
cat service/dispatch.go && \
printf '\\nmodule files status:\\n' && \
git status --short -- go.mod go.sum && \
printf '\\nreferences:\\n' && \
rg -n 'Normalize|probe|\\.Write\\(' audit service"
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": false,
  "body": "switch NormalizeKind(kind) {\ncase \"health\", \"probe\":\n\treturn \"control\"\ndefault:\n\treturn \"data\"\n}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
7✓ ✓
/bin/zsh -lc 'go build ./... && \
git diff --check'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": false,
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\nif audit.Classify(event.Kind) == \"control\" {\n\treturn nil\n}\nreturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
8✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go && \
git status --short -- go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": false,
  "body": "writeEvent := audit.Sink.Write\nfor _, event := range events {\n\tevent.Kind = audit.NormalizeKind(event.Kind)\n\tif audit.Classify(event.Kind) == \"control\" {\n\t\tcontinue\n\t}\n\tif err := writeEvent(sink, event); err != nil {\n\t\treturn err\n\t}\n}\nreturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
9✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go && \
go build ./... && \
git diff --check && \
git status --short -- go.mod go.sum'
✓ ✓
/bin/zsh -lc 'git diff --check && \
git status --short && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./...'
10✓ ✓
/bin/zsh -lc 'cat service/dispatch.go audit/normalize.go && \
git status --short -- go.mod go.sum && \
go build ./... && \
git diff --check'
✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat audit/summary.go && \
cat service/dispatch.go && \
git diff --stat && \
git status --short --untracked-files=all'
11✓ ✓
/bin/zsh -lc 'gofmt -w service/dispatch.go && \
go test ./... && \
git diff --check && \
git status --short -- go.mod go.sum'
✓ ✓
/bin/zsh -lc 'git rev-parse --show-toplevel && \
git ls-files audit/normalize.go service/dispatch.go'
12—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go && \
git status --short -- audit/normalize.go audit/summary.go service/dispatch.go go.mod go.sum && \
git diff -- go.mod go.sum'
13—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
14—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --exit-code -- go.mod go.sum'
15—✓ ✓
/bin/zsh -lc 'cat audit/summary.go audit/normalize.go && \
git diff -- go.mod go.sum && \
git status --short -- go.mod go.sum'
16—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --exit-code -- go.mod go.sum'
17—✓ ✓
/bin/zsh -lc 'cat service/dispatch.go && \
git diff -- go.mod go.sum && \
git status --short -- go.mod go.sum'
18—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
19—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --exit-code -- go.mod go.sum'
20—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go && \
git diff -- go.mod go.sum'
21—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
22—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --exit-code -- go.mod go.sum'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' -g 'go.mod' -g 'go.sum'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g 'docs/**' -g 'Makefile' | head -100"
2✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
rg -n \"audit|probe|event|sink|persist|delivery\" --glob '*.go'"
✓ ✓
/bin/zsh -lc 'rg --files | head -100 && \
cat docs/audit-sinks.md && \
git status --short'
3✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/sink.go audit/event.go audit/fanout.go audit/transaction.go audit/summary.go service/dispatch.go service/replay.go'
✓ ✓
/bin/zsh -lc 'cat audit/event.go audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go audit/summary.go && \
cat service/dispatch.go service/replay.go && \
cat go.mod'
4✓ ✓
/bin/zsh -lc 'rg --files'
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Classify"
}
5✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat go.mod && \
git status --short'
semedit/semantic_lookup (✓ ✗): resolve Normalize: symbol not found
{
  "symbol": "Normalize"
}
6✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat service/dispatch_test.go && \
cat audit/summary_test.go'
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Write"
}
7? ✗
/bin/zsh -lc 'rg -n "make check|verify" .'
✓ ✓
/bin/zsh -lc 'cat audit/event_test.go audit/summary_test.go audit/dispatch_test.go service/dispatch_test.go 2>/dev/null; cat audit/transaction.go audit/summary.go journal/file.go'
8✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go && \
go test ./... && \
git status --short'
semedit/semantic_insert_function (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
9✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go go.mod go.sum && \
cat docs/audit-sinks.md && \
cat audit/normalize.go audit/sink.go'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\nreturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
10✓ ✓
/bin/zsh -lc 'cat service/dispatch.go service/replay.go audit/fanout.go audit/transaction.go journal/file.go; git diff --stat; git status --porcelain=v1'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\ncase \"health\", \"probe\":\n\treturn \"control\"\ndefault:\n\treturn \"data\"\n}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
11? ✗
/bin/zsh -lc 'gofmt -w audit/normalize.go service/dispatch.go service/replay.go && \
go test ./...'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "writeEvent := audit.Sink.Write\nfor _, event := range events {\n\tevent.Kind = audit.NormalizeKind(event.Kind)\n\tif err := writeEvent(sink, event); err != nil {\n\t\treturn err\n\t}\n}\nreturn nil",
  "file": "service/replay.go",
  "symbol": "Replay"
}
12✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go service/dispatch.go service/replay.go && \
go test ./... && \
git status --short'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "writeEvent := audit.Sink.Write\nfor _, event := range events {\n\tevent.Kind = audit.NormalizeKind(event.Kind)\n\tif err := writeEvent(sink, event); err != nil {\n\t\treturn err\n\t}\n}\nreturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
13✓ ✓
/bin/zsh -lc 'cat audit/summary.go audit/normalize.go audit/event.go && \
git status --short'
✓ ✓
/bin/zsh -lc "git diff --check && \
git status --short && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go service/replay.go && \
git diff -- go.mod go.sum '*_test.go'"
14✓ ✓
/bin/zsh -lc "rg -n 'health|probe|Classify' --glob '*.go'"
✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go service/replay.go && \
git rev-parse --show-toplevel && \
git status --short --untracked-files=no'
15✓ ✓
/bin/zsh -lc 'gofmt -w audit/summary.go && \
go test ./audit ./legacy ./metrics && \
git diff -- audit/summary.go go.mod go.sum'
—
16✓ ✓
/bin/zsh -lc 'cat service/dispatch.go audit/normalize.go && \
git status --short && \
git diff -- go.mod go.sum'
—
17✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go && \
git diff -- go.mod go.sum'
—
18✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./...'
—

prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit

prefer_discover_semedit prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency117.56s242.00s+105.9%88.45s234.69s+165.3%
Process Start → First Event0.27s0.15s—0.12s0.15s—
First Event → First Tool Call21.72s19.25s—19.82s5.47s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns25+150.0%15+400.0%
Internal Tool Cycles928+211.1%638+533.3%
Initial Load / Discovery Turns13+200.0%14+300.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)928+211.1%638+533.3%
Output Tokens564214759+161.6%352324876+606.1%
Reasoning / Thinking Tokens25046991+179.2%153911898+673.1%
Total Input Tokens6152554198379+582.4%3808594367873+1046.8%
Cached Input Tokens5719043945728+589.9%3461124132352+1093.9%
Uncached Input Tokens43351252651+482.8%34747235521+577.8%
Cost0.35321.8899+435.1%0.23672.0816+779.5%
Cached vs Uncached Token Ratio13.19:115.62:1+18.4%9.96:117.55:1+76.1%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSFAIL—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS*' -g '*.go' -g 'Makefile' | head -100"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'Makefile' -g 'AGENTS*' -g '*.go' -g '"'!**/*_test.go'"'"
2✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
sed -n '1,240p' audit/event.go && \
sed -n '1,240p' audit/normalize.go && \
sed -n '1,240p' audit/sink.go && \
sed -n '1,260p' service/dispatch.go"
✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
rg --files && \
rg -n \"probe|event|Deliver|Persist|Sink|Normalize|normaliz\" --glob '*.go'"
3✓ ✓
/bin/zsh -lc "sed -n '1,260p' audit/event_test.go && \
sed -n '1,260p' audit/summary.go && \
sed -n '1,260p' audit/summary_test.go && \
sed -n '1,260p' service/dispatch_test.go && \
cat metrics/writer.go && \
cat legacy/export.go && \
rg -n 'probe|normalize|Kind|Dispatch|Record' . --glob '"'!*.sum'"'"
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/event.go && \
cat audit/normalize.go && \
cat audit/sink.go && \
cat service/dispatch.go && \
cat metrics/writer.go && \
cat legacy/export.go && \
cat go.mod'
4✓ ✓
/bin/zsh -lc "rg --files -g '*.md' -g 'Makefile' -g 'go.mod' && \
cat go.mod && \
ls -la"
semedit/semantic_outline (✓ ✓)
{
  "include_tests": false,
  "path": "service"
}
5✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md'
semedit/semantic_outline (✓ ✓)
{
  "include_tests": false,
  "path": "audit"
}
6✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./... && \
git status --short && \
git diff --check && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
7✓ ✓
/bin/zsh -lc "git diff --stat && \
git diff -- audit/normalize.go service/dispatch.go audit/summary.go; git status --short -- audit/normalize.go service/dispatch.go audit/summary.go go.mod go.sum; sed -n '1,120p' service/dispatch.go; sed -n '1,80p' audit/normalize.go"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
8✓ ✓
/bin/zsh -lc "git status --short && \
git diff --no-ext-diff && \
rg -n 'NormalizeKind|Classify|probe|Record|DispatchAll' audit service metrics legacy docs && \
cat docs/audit-sinks.md && \
cat go.mod"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tevent.Kind = audit.NormalizeKind(event.Kind)\n\t\tif event.Kind == \"probe\" {\n\t\t\tcontinue\n\t\t}\n\t\tif err := sink.Write(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
9✓ ✓
/bin/zsh -lc "gofmt -w service/dispatch.go && \
go test ./... && \
git diff --check && \
git status --short && \
git diff -- go.mod go.sum && \
sed -n '1,100p' service/dispatch.go && \
sed -n '1,80p' audit/summary.go"
semedit/semantic_verify (✓ ✓)
{
  "language": "go"
}
10—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- audit/normalize.go service/dispatch.go go.mod go.sum && \
cat audit/normalize.go'
11—? ✗
/bin/zsh -lc 'cat service/dispatch.go && \
git status --short --untracked-files=all && \
git diff --no-index /dev/null audit/normalize.go'
12—✓ ✓
/bin/zsh -lc 'git status --short --untracked-files=all && \
git diff -- go.mod go.sum && \
cat audit/normalize.go && \
cat audit/summary.go && \
cat service/dispatch.go && \
cat docs/audit-sinks.md'
13—✓ ✓
/bin/zsh -lc 'cat service/dispatch_test.go audit/summary_test.go audit/event_test.go metrics/writer.go legacy/export.go'
14—semedit/semantic_replace_body (✓ ✓)
{
  "body": "kind = NormalizeKind(kind)\n\tswitch kind {\n\tcase \"health\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
15—semedit/semantic_verify (✓ ✓)
{
  "language": "go"
}
16—✓ ✓
/bin/zsh -lc 'git diff -- audit/summary.go go.mod go.sum && \
git status --short --untracked-files=no'
17—✓ ✓
/bin/zsh -lc 'cat audit/summary.go audit/normalize.go && \
git diff -- go.mod go.sum'
18—semedit/semantic_replace_body (✓ ✓)
{
  "body": "kind = NormalizeKind(kind)\n\tswitch kind {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
19—semedit/semantic_verify (✓ ✓)
{
  "language": "go"
}
20—✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum && \
cat audit/summary.go'
21—✓ ✓
/bin/zsh -lc 'cat service/dispatch.go && \
git diff -- go.mod go.sum'
22—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "service"
}
23—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go && \
git diff -- go.mod go.sum'
24—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
25—semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tevent.Kind = audit.NormalizeKind(event.Kind)\n\t\tif err := sink.Write(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
26—✓ ✓
/bin/zsh -lc 'go test ./...'
27—semedit/semantic_verify (✓ ✓)
{
  "language": "go"
}
28—✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 17 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 17 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'README*' -g 'Makefile' -g '*.go' | head -100 && \
git status --short && \
rg -n \"audit|probe|legacy|metrics|delivery\" . --glob '"'!go.sum'"' --glob '"'!go.mod'"'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g 'docs/**' -g 'tools/benchmark-harness/**' | head -100"
2? ✗
/bin/zsh -lc 'cat AGENTS.override.md docs/audit-sinks.md audit/event.go audit/sink.go audit/normalize.go audit/fanout.go audit/transaction.go audit/summary.go service/dispatch.go service/replay.go journal/file.go Makefile'
✓ ✓
/bin/zsh -lc "cat docs/audit-sinks.md && \
rg -n \"audit|Audit|probe|Probe|event kind|sink\" --glob '"'!**/*_test.go'"' ."
3✓ ✓
/bin/zsh -lc "rg -n \"(DispatchAll|Replay|NewDispatcher|\\.Write\\(|Classify\\(|BufferedSink|FanoutSink|TransactionalSink|Normalize)\" --glob '*.go' . && \
cat audit/event_test.go audit/summary_test.go service/dispatch_test.go"
✓ ✓
/bin/zsh -lc "git status --short && \
rg --files -g 'Makefile' -g 'PROJECT_BRIEF.md' -g 'audit/*.go' -g 'journal/*.go' -g 'service/*.go' -g 'metrics/*.go' -g 'legacy/*.go' -g 'AGENTS.md' -g 'AGENTS.override.md'"
4? ✗
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go service/replay.go && \
go test ./... && \
git status --short && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go service/replay.go go.mod go.sum'
semedit/semantic_outline (✓ ✓)
{
  "language": "go",
  "path": "audit/event.go"
}
5✓ ✓
/bin/zsh -lc 'go test ./... && \
git status --short && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go service/replay.go go.mod go.sum'
semedit/semantic_outline (✓ ✓)
{
  "language": "go",
  "path": "audit/normalize.go"
}
6✓ ✓
/bin/zsh -lc 'git rev-parse --show-toplevel && \
git status --short --untracked-files=all && \
git diff --stat && \
git check-ignore -v audit/normalize.go audit/summary.go service/dispatch.go service/replay.go'
semedit/semantic_outline (✓ ✓)
{
  "language": "go",
  "path": "audit/sink.go"
}
7—semedit/semantic_outline (✓ ✓)
{
  "language": "go",
  "path": "audit/fanout.go"
}
8—semedit/semantic_outline (✓ ✓)
{
  "language": "go",
  "path": "audit/transaction.go"
}
9—semedit/semantic_outline (✓ ✓)
{
  "language": "go",
  "path": "journal/file.go"
}
10—✓ ✓
/bin/zsh -lc 'cat audit/sink.go audit/fanout.go audit/transaction.go audit/normalize.go journal/file.go service/dispatch.go service/replay.go metrics/writer.go legacy/export.go'
11—semedit/semantic_insert_function (✓ ✓)
{
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeEvent(event Event) (Event, bool) {\n\tevent.Kind = strings.ToLower(strings.TrimSpace(event.Kind))\n\treturn event, event.Kind != \"probe\"\n}"
}
12—semedit/semantic_insert_function (✓ ✓)
{
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func Deliver(sink Sink, event Event) error {\n\tnormalized, persist := NormalizeEvent(event)\n\tif !persist {\n\t\treturn nil\n\t}\n\treturn sink.Write(normalized)\n}"
}
13—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "normalized, persist := NormalizeEvent(event)\n\tif !persist {\n\t\treturn nil\n\t}\n\tm.Events = append(m.Events, normalized)\n\treturn nil",
  "file": "audit/sink.go",
  "symbol": "(*MemorySink).Write"
}
14—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "if err := Deliver(f.First, event); err != nil {\n\t\treturn err\n\t}\n\treturn Deliver(f.Second, event)",
  "file": "audit/fanout.go",
  "symbol": "(FanoutSink).Write"
}
15—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "normalized, persist := audit.NormalizeEvent(event)\n\tif !persist {\n\t\treturn nil\n\t}\n\tj.Entries = append(j.Entries, normalized)\n\treturn nil",
  "file": "journal/file.go",
  "symbol": "(*JournalSink).Write"
}
16—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "return audit.Deliver(d.sink, event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
17—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "for _, event := range events {\n\t\tif err := audit.Deliver(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
18—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "for _, event := range events {\n\t\tif err := audit.Deliver(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/replay.go",
  "symbol": "Replay"
}
19—? ✗
/bin/zsh -lc "rg --files -g 'Makefile' -g 'makefile' -g 'GNUmakefile' -g 'go.work' && \
git diff -- audit/normalize.go audit/sink.go audit/fanout.go journal/file.go service/dispatch.go service/replay.go && \
git diff -- go.mod go.sum"
20—✓ ✓
/bin/zsh -lc 'git diff -- audit/normalize.go audit/sink.go audit/fanout.go journal/file.go service/dispatch.go service/replay.go; git diff -- go.mod go.sum; rg --files | head -80'
21—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat audit/sink.go && \
git status --short && \
git diff --stat && \
git ls-files audit/normalize.go'
22—? ✗
/bin/zsh -lc 'make check'
23—✓ ✓
/bin/zsh -lc 'go test ./...'
24—✓ ✓
/bin/zsh -lc 'git status --short && \
cat docs/audit-sinks.md audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go journal/file.go service/dispatch.go service/replay.go && \
git diff -- go.mod go.sum'
25—✓ ✓
/bin/zsh -lc "rg -n '\\.Write\\(event\\)|Write\\(event' --glob '*.go' --glob '"'!**/*_test.go'"' ."
26—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "for _, event := range b.queue {\n\t\tif err := Deliver(b.Sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "audit/transaction.go",
  "symbol": "(*BufferedSink).Commit"
}
27—? ✗
/bin/zsh -lc "go test ./... && \
git diff --exit-code -- go.mod go.sum && \
rg -n '\\.Write\\(event\\)' --glob '*.go' --glob '"'!**/*_test.go'"' ."
28—✓ ✓
/bin/zsh -lc 'cat audit/summary.go audit/normalize.go audit/event.go && \
git diff -- go.mod go.sum'
29—semedit/semantic_outline (✓ ✓)
{
  "language": "go",
  "path": "audit/summary.go"
}
30—semedit/semantic_insert_function (✓ ✓)
{
  "file": "audit/normalize.go",
  "placement": "public_start",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
31—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event.Kind = NormalizeKind(event.Kind)\n\treturn event, event.Kind != \"probe\"",
  "file": "audit/normalize.go",
  "symbol": "NormalizeEvent"
}
32—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
33—✓ ✓
/bin/zsh -lc 'go test ./...; test $? -eq 0 && \
git diff --exit-code -- go.mod go.sum'
34—✓ ✓
/bin/zsh -lc 'cat service/dispatch.go audit/normalize.go && \
git diff -- go.mod go.sum'
35—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "service/dispatch.go"
}
36—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --exit-code -- go.mod go.sum'
37—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
38—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --exit-code -- go.mod go.sum'
Benchmark run gpt-6-luna-all-r5-c10-none-20260926-retryBenchmark run hard-smoke-luna6-medium-20261011T105400Z