Skip to content
Documentation
Benchmarks
Benchmark runs
Benchmark run run-20260924-codex-descriptive

Benchmark run run-20260924-codex-descriptive

This page contains exactly the publishable benchmark observations recorded in run run-20260924-codex-descriptive; it does not select or aggregate them. Return to the run index or view aggregate metrics.

Interactive browser

Rows are pre-aggregated by task, target, prompt variant, MCP-instruction mode, context, arm, and restriction. Numeric values show median (p50), IQR, p90, maximum, and sample count. Boolean checks show pass counts and success rates. Filters apply to these summary groups; per-run observations remain available on their run pages.

Loading benchmark results…

Baseline vs semedit

Comparison evaluates baseline and semedit values within each restriction policy, target, and context group. Negative changes mark improvements for these lower-is-better metrics; positive changes mark degradations.

Detailed observations

Empirical telemetry measuring wall-clock latency, token usage, tool turns, and compiler correctness when AI coding agents perform code modifications.

Every benchmark pairs Vanilla LLM (standard file editing tools) against Semedit MCP (deterministic AST compiler operations).

Benchmark Methodology & Transparency

  • Reproducibility & Provenance: Self-contained txtar archives define test scenarios. Links below resolve to commit-anchored GitHub source files for published commits, or cryptographic SHA-256 fingerprints for uncommitted local fixtures.
  • Model Cost: Tables report model-specific credit consumption where a target model has a declared schedule: (uncached input × input credits + cached input × cached credits + (reasoning + visible output) × output credits) / 1,000,000. Thinking tokens use the output rate. ADR-0043 defines the per-million-token credit rates.
  • Multi-Level Correctness Oracle: Four validation levels evaluate each trial:
    1. Level 1 (Mutation Policy): Restricts file modifications strictly to authorized paths.
    2. Level 2 (AST Invariants): Compiler AST verification of required symbols, imports, and relative declaration ordering.
    3. Level 3 (Clean Build): Strict compilation validation with zero compiler or typecheck errors.
    4. Level 4 (Test Suite): Unit and integration test suite execution.
  • Isolated Sandboxing: Each trial executes in an ephemeral directory with pristine git and module state.

Test case: task-04-insert-public

Target: codex/gpt-5.6-luna/high

default prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-04-insert-public
Targetcodex/gpt-5.6-luna/high
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/bench/task_04_insert_public.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
package api

type Server struct {
	Port int
}

func (s *Server) Start() {}

func (s *Server) internalRun() {}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)⚠ MCP (Large)Δ (Large)
Wall-Clock Latency33.89s81.96s+141.9%49.17s58.85sN/A
Process Start → First Event0.12s0.46s—0.13s0.44s—
First Event → First Tool Call6.58s13.60s—7.16s8.29s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%11N/A
Internal Tool Cycles49+125.0%68N/A
Initial Load / Discovery Turns43-25.0%62N/A
MCP Discovery / Schema Turns01—00—
Tool Invocations (per run)49+125.0%68N/A
Output Tokens12042581+114.4%16591954N/A
Reasoning / Thinking Tokens5311130+112.8%648774N/A
Total Input Tokens122632338029+175.6%166822209450N/A
Cached Input Tokens103936311552+199.8%123904190976N/A
Uncached Input Tokens1869626477+41.6%4291818474N/A
Cost0.19750.3995+102.3%0.34580.2697N/A
Cached vs Uncached Token Ratio5.56:111.77:1+111.7%2.89:110.34:1N/A
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/AFAIL—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!*test*'"' -g '"'!go.mod'"' -g '"'!go.sum'"' | sort && \
rg -n \"type Server|func .*Server|private|helper\" -g '*.go' -g '"'!**/*_test.go'"'"
✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- files ---'; rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS*.md' -g '*.go' -g 'Makefile' -g 'README*' | sort; printf '%s\\n' '--- status ---'; git status --short; printf '%s\\n' '--- top-level ---'; rg --files -g '"'!vendor'"' | sed -n '1,80p'"
2✓ ✓
/bin/zsh -lc "sed -n '1,120p' api/server.go && \
git status --short"
✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- api/server.go ---'; nl -ba api/server.go; printf '%s\\n' '--- api/server_test.go ---'; nl -ba api/server_test.go; printf '%s\\n' '--- Makefile ---'; if [ -f Makefile ]; then nl -ba Makefile | sed -n '1,180p'; else printf '%s\\n' 'no Makefile in fixture'; fi"
3✓ ✓
/bin/zsh -lc 'gofmt -w api/server.go && \
git diff -- api/server.go && \
go test ./api'
cua_repl/js (✓ ✓)
{
  "code": "const state = await cua.getState(); nodeRepl.write(state);",
  "title": "Inspect available workspace surfaces"
}
4✓ ✓
/bin/zsh -lc "sed -n '1,80p' api/server.go && \
git status --short -- api/server.go && \
git diff --no-index /dev/null api/server.go || \
true"
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "api/server.go",
  "placement": "public_end",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}
5—? ✗
/bin/zsh -lc "printf '%s\\n' '--- diff ---'; git diff -- api/server.go api/server_test.go go.mod go.sum; printf '%s\\n' '--- diff check ---'; git diff --check; printf '%s\\n' '--- make check ---'; make check"
6—✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- api/server.go ---'; nl -ba api/server.go; printf '%s\\n' '--- status ---'; git status --short -- api/server.go api/server_test.go go.mod go.sum; printf '%s\\n' '--- metadata ---'; git diff --stat -- api/server.go"
7—✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- HEAD api/server.go ---'; git show HEAD:api/server.go | nl -ba; printf '%s\\n' '--- index/worktree hashes ---'; git rev-parse HEAD:api/server.go; shasum api/server.go; git ls-files -v api/server.go api/server_test.go go.mod go.sum"
8—✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- root ---'; git rev-parse --show-toplevel; printf '%s\\n' '--- tracked sample ---'; git ls-files | sed -n '1,30p'; printf '%s\\n' '--- ignored fixture files ---'; git status --short --ignored --untracked-files=all -- api/server.go api/server_test.go go.mod | sed -n '1,30p'; printf '%s\\n' '--- all fixture entries ---'; ls -la"
9—✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- gofmt check ---'; test -z \""'$(gofmt -d api/server.go)"; printf '"'%s\\n' 'gofmt: clean'; printf '%s\\n' '--- go test ---'; go test ./..."
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '"'!AGENTS.override.md'"' -g '*.go' -g 'Makefile' -g 'README*' | sort"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '*.go' -g 'Makefile' -g 'README*' | sort | head -200"
2✓ ✓
/bin/zsh -lc "sed -n '1,240p' api/server.go && \
sed -n '1,180p' api/router.go && \
git status --short"
✓ ✓
/bin/zsh -lc "sed -n '1,240p' api/server.go && \
printf '\\n--- repository files ---\\n' && \
rg --files -g '"'!*.sum'"' -g '"'!go.mod'"' | sort | head -200"
3? ✗
/bin/zsh -lc "sed -n '1,240p' api/server_test.go && \
rg -n \"InitServer|Server\\{\" . -g '*.go' -g '"'!*_test.go'"'"
✓ ✓
/bin/zsh -lc "command -v semedit || \
true
command -v go || \
true
rg -n \"semantic_insert|semedit|InitServer|internalRun|func \\(s \\*Server\\)\" ."
4✓ ✓
/bin/zsh -lc "rg --files -g '"'!AGENTS.override.md'"' | sort && \
git diff -- api/server.go"
✓ ✓
/bin/zsh -lc "sed -n '1,160p' api/server_test.go && \
sed -n '1,160p' api/router.go && \
sed -n '1,160p' AGENTS.override.md"
5? ✗
/bin/zsh -lc "sed -n '1,80p' api/server.go && \
git status --short -- api/server.go api/server_test.go && \
git ls-files --error-unmatch api/server.go"
✓ ✓
/bin/zsh -lc 'gofmt -w api/server.go
git diff --check
go test ./...'
6✓ ✓
/bin/zsh -lc 'gofmt -d api/server.go && \
go test ./... && \
git diff --no-index /dev/null api/server.go >/tmp/server-diff.txt || \
true; test ! -e Makefile; test -z "$(git diff -- go.mod go.sum)"'
✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- api/server.go'
7—✓ ✓
/bin/zsh -lc "sed -n '1,80p' api/server.go
pwd
git rev-parse --show-toplevel
git diff --no-ext-diff -- api/server.go
stat -f '%Sm %N' api/server.go"
8—✓ ✓
/bin/zsh -lc 'git ls-files --stage -- api/server.go api/server_test.go go.mod go.sum
git check-ignore -v api/server.go api/server_test.go || \
true
ls -la
find . -maxdepth 2 -type f -print | sort'
✕ Why no semantic edit tool was used
I did not call a `semantic_*` tool because no semedit MCP tools were exposed or discoverable in the available tool set. I used the minimal ordinary source edit instead.

Target: codex/gpt-5.6-luna/medium

default prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-04-insert-public
Targetcodex/gpt-5.6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/bench/task_04_insert_public.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
package api

type Server struct {
	Port int
}

func (s *Server) Start() {}

func (s *Server) internalRun() {}
MetricVanilla (Small)⚠ MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency32.74s41.66sN/A37.34s66.93s+79.2%
Process Start → First Event0.14s0.14s—0.11s0.13s—
First Event → First Tool Call6.40s10.76s—6.58s11.20s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns11N/A110%
Internal Tool Cycles44N/A58+60.0%
Initial Load / Discovery Turns42N/A52-60.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)44N/A58+60.0%
Output Tokens10051087N/A12151679+38.2%
Reasoning / Thinking Tokens340304N/A346462+33.5%
Total Input Tokens120663122733N/A143692249732+73.8%
Cached Input Tokens8857699840N/A115968199168+71.7%
Uncached Input Tokens3208722893N/A2772450564+82.4%
Cost0.24510.2061N/A0.24340.4166+71.1%
Cached vs Uncached Token Ratio2.76:14.36:1N/A4.18:13.94:1-5.8%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/AFAIL—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '*.go' -g 'Makefile' | sort"
✓ ✓
/bin/zsh -lc "pwd; rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '*.go' -g 'Makefile' -g 'README*' | sort; rg -n \"type Server|func .*Server|private|InitServer\" -g '*.go' ."
2✓ ✓
/bin/zsh -lc "sed -n '1,240p' api/server.go && \
printf '\\n--- tests (read-only) ---\\n' && \
sed -n '1,240p' api/server_test.go && \
printf '\\n--- root files ---\\n' && \
rg --files -g '"'!api/server.go'"' -g '"'!api/server_test.go'"' | sort"
✓ ✓
/bin/zsh -lc "sed -n '1,160p' api/server.go; sed -n '1,120p' api/server_test.go; if [ -f PROJECT_BRIEF.md ]; then sed -n '1,200p' PROJECT_BRIEF.md; fi; if [ -f Makefile ]; then sed -n '1,200p' Makefile; fi; git status --short"
3✓ ✓
/bin/zsh -lc 'gofmt -w api/server.go && \
go test ./... && \
git diff -- api/server.go && \
git status --short'
✓ ✓
/bin/zsh -lc "rg --files -g '"'!api/server_test.go'"' -g '"'!api/server.go'"' | sort; command -v semedit || \
true; find . -maxdepth 2 -type f -perm -111 -print"
4? ✗
/bin/zsh -lc "sed -n '1,80p' api/server.go && \
git diff --no-ext-diff -- api/server.go && \
git show HEAD:api/server.go"
✓ ✓
/bin/zsh -lc 'gofmt -w api/server.go && \
go test ./... && \
git diff -- api/server.go go.mod go.sum api/server_test.go && \
git status --short -- api/server.go go.mod go.sum api/server_test.go'
⚠ Why no semantic edit tool was used
I did not call a semantic_* tool because none was available or discoverable in the provided tool context. I used a minimal ordinary source edit instead.
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!*AGENTS.override.md'"' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '*.go' -g 'Makefile' | head -200"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '*.go' -g 'Makefile' -g 'README*' | head -80 && \
rg -n \"type Server|func InitServer|private|func [a-z]\" --glob '*.go' ."
2✓ ✓
/bin/zsh -lc "sed -n '1,240p' api/server.go && \
printf '\\n--- router ---\\n' && \
sed -n '1,220p' api/router.go && \
printf '\\n--- project files ---\\n' && \
rg --files -g 'PROJECT_BRIEF.md' -g 'Makefile' -g 'go.mod' -g 'README*'"
✓ ✓
/bin/zsh -lc "sed -n '1,220p' api/server.go && \
printf '\\n--- Makefile/files ---\\n' && \
rg --files -g 'Makefile' -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS*.md' -g 'go.mod' -g 'go.sum' | sort && \
printf '\\n--- git status ---\\n' && \
git status --short"
3✓ ✓
/bin/zsh -lc "sed -n '1,240p' api/server_test.go && \
sed -n '1,160p' go.mod && \
git status --short"
✓ ✓
/bin/zsh -lc "sed -n '1,160p' go.mod; printf '\\n--- root files ---\\n'; rg --files -g '"'!go.sum'"' -g '"'!go.mod'"' | sort; printf '\\n--- semantic operation references ---\\n'; rg -n \"semantic_insert_function|insert.*function|semedit\" . --glob '"'!*.sum'"' --glob '"'!go.mod'"' | head -120"
4✓ ✓
/bin/zsh -lc 'gofmt -w api/server.go && \
go test ./... && \
git diff -- api/server.go go.mod go.sum && \
git status --short -- api/server.go go.mod go.sum'
✓ ✓
/bin/zsh -lc "sed -n '1,220p' api/server_test.go; sed -n '1,180p' api/router.go; sed -n '1,180p' api/router_test.go"
5✓ ✓
/bin/zsh -lc "sed -n '1,80p' api/server.go && \
git diff --no-ext-diff -- api/server.go; git status --short --untracked-files=all -- api/server.go"
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": false,
  "file": "api/server.go",
  "placement": "before_symbol",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}",
  "target_symbol": "internalRun"
}
6—✓ ✓
/bin/zsh -lc "sed -n '1,120p' api/server.go && \
printf '\\n--- diff ---\\n' && \
git diff -- api/server.go && \
printf '\\n--- protected-file status ---\\n' && \
git status --short -- go.mod go.sum && \
printf '\\n--- tests ---\\n' && \
go test ./..."
7—? ✗
/bin/zsh -lc 'git status --short -- api/server.go && \
git diff --no-ext-diff -- api/server.go && \
git show HEAD:api/server.go'
8—✓ ✓
/bin/zsh -lc "git status --short --untracked-files=all | rg 'server.go|api/' || \
true; git check-ignore -v api/server.go || \
true"

Target: codex/gpt-6-luna/high

default prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-04-insert-public
Targetcodex/gpt-6-luna/high
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/bench/task_04_insert_public.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
package api

type Server struct {
	Port int
}

func (s *Server) Start() {}

func (s *Server) internalRun() {}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency26.89s31.21s+16.1%34.69s42.13s+21.4%
Process Start → First Event0.12s0.12s—0.12s0.13s—
First Event → First Tool Call6.58s5.53s—7.47s6.43s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles440%45+25.0%
Initial Load / Discovery Turns42-50.0%23+50.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)440%45+25.0%
Output Tokens828535-35.4%1433859-40.1%
Reasoning / Thinking Tokens310197-36.5%698431-38.3%
Total Input Tokens115004103953-9.6%97399150675+54.7%
Cached Input Tokens10291286784-15.7%85760130304+51.9%
Uncached Input Tokens1209217169+42.0%1163920371+75.0%
Cost0.07020.0738+5.1%0.07720.0996+29.1%
Cached vs Uncached Token Ratio8.51:15.05:1-40.6%7.37:16.40:1-13.2%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg -n \"type Server|func [a-z]\" . --glob '*.go'"
✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '*.go' -g 'README*' | head -80"
2✓ ✓
/bin/zsh -lc 'cat api/server.go'
✓ ✓
/bin/zsh -lc "cat api/server.go && \
rg --files -g 'PROJECT_BRIEF.md' -g 'docs/**' | head -40"
3✓ ✓
/bin/zsh -lc 'git diff -- api/server.go'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "public_end",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}
4✓ ✓
/bin/zsh -lc "sed -n '1,30p' api/server.go && \
git status --short"
✓ ✓
/bin/zsh -lc 'cat api/server.go && \
git diff -- api/server.go go.mod go.sum api/server_test.go'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc 'rg -n "type Server|func .*Server|''^func [a-z]" --glob '"'*.go'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g '*.go' -g 'go.mod' -g 'go.sum' | head -80"
2✓ ✓
/bin/zsh -lc 'cat api/server.go && \
cat api/server_test.go'
✓ ✓
/bin/zsh -lc "rg --files | head -100 && \
sed -n '1,240p' api/server.go && \
sed -n '1,180p' api/router.go"
3✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os
import tempfile
import time

path = Path('api/server.go')
old_stat = path.stat()
content = path.read_text()
needle = 'func (s *Server) Start() {}\\n\\n'
replacement = needle + 'func InitServer() *Server {\\n\\treturn &Server{}\\n}\\n\\n'
if content.count(needle) "'!= 1:
    raise SystemExit('"'expected exactly one Start method insertion point')
updated = content.replace(needle, replacement, 1)
fd, temp_name = tempfile.mkstemp(prefix='.server.go.', dir=path.parent)
try:
    with os.fdopen(fd, 'w') as temp_file:
        temp_file.write(updated)
        temp_file.flush()
        os.fsync(temp_file.fileno())
    os.utime(temp_name, ns=(old_stat.st_atime_ns, max(time.time_ns(), old_stat.st_mtime_ns + 1)))
    os.replace(temp_name, path)
    directory_fd = os.open(path.parent, os.O_RDONLY)
    try:
        os.fsync(directory_fd)
    finally:
        os.close(directory_fd)
except BaseException:
    try:
        os.unlink(temp_name)
    except FileNotFoundError:
        pass
    raise
PY
git diff -- api/server.go"
✓ ✓
/bin/zsh -lc "sed -n '1,200p' api/server_test.go"
4✓ ✓
/bin/zsh -lc 'cat api/server.go && \
git status --short'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "before_symbol",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}",
  "target_symbol": "internalRun"
}
5—✓ ✓
/bin/zsh -lc "sed -n '1,100p' api/server.go && \
git diff -- api/server.go && \
git status --short"

Target: codex/gpt-6-luna/medium

default prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-04-insert-public
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/bench/task_04_insert_public.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
package api

type Server struct {
	Port int
}

func (s *Server) Start() {}

func (s *Server) internalRun() {}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency20.62s27.33s+32.5%19.16s21.25s+10.9%
Process Start → First Event0.12s0.13s—0.12s0.12s—
First Event → First Tool Call4.60s7.66s—6.04s5.76s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns21-50.0%110%
Internal Tool Cycles330%330%
Initial Load / Discovery Turns32-33.3%20-100.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)330%330%
Output Tokens504348-31.0%470424-9.8%
Reasoning / Thinking Tokens5850-13.8%940-100.0%
Total Input Tokens169161102161-39.6%7720877545+0.4%
Cached Input Tokens15129678592-48.1%6348867584+6.5%
Uncached Input Tokens1786523569+31.9%137209961-27.4%
Cost0.08950.0835-6.7%0.05720.0471-17.7%
Cached vs Uncached Token Ratio8.47:13.33:1-60.6%4.63:16.78:1+46.6%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g 'AGENTS.md' -g '*.go' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'go.sum'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g '*.go' -g 'AGENTS.md' | head -100 && \
rg -n \"type Server|func .*Server|private helper|func [a-z]\" --glob '*.go' ."
2✓ ✓
/bin/zsh -lc 'cat api/server.go'
✓ ✓
/bin/zsh -lc "cat api/server.go && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'go.mod' -g 'AGENTS.md' ."
3✓ ✓
/bin/zsh -lc 'git status --short -- go.mod go.sum api/server.go && \
cat api/server.go'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "public_start",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g '*.go' -g 'PROJECT_BRIEF.md' | head -80"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g '*.go' -g 'PROJECT_BRIEF.md' -g 'README*' | head -80 && \
rg -n \"type Server|func NewServer|func InitServer|semantic_insert_function|semantic_verify\" . --glob '*.go' --glob '*.md' --glob '*.json'"
2✓ ✓
/bin/zsh -lc 'cat api/server.go && \
cat api/router.go && \
git status --short'
? ✗
/bin/zsh -lc "cat api/server.go && \
cat api/router.go && \
rg -n \"semantic_insert_function|semantic_verify|semantic\" . --glob '"'!*.sum'"' --glob '"'!go.mod'"'"
3✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
p = Path('api/server.go')
s = p.read_text()
needle = 'func (s *Server) Start() {}\\n\\nfunc (s *Server) internalRun() {}\\n'
replacement = 'func InitServer() *Server {\\n\\treturn &Server{}\\n}\\n\\nfunc (s *Server) Start() {}\\n\\nfunc (s *Server) internalRun() {}\\n'
if needle not in s:
    raise SystemExit('expected source layout not found')
tmp = p.with_suffix('.go.tmp')
with tmp.open('w') as f:
    f.write(s.replace(needle, replacement, 1))
    f.flush()
    import os
    os.fsync(f.fileno())
os.replace(tmp, p)
PY
"
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "public_start",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}

Target: codex/gpt-6-luna/xhigh

default prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-04-insert-public
Targetcodex/gpt-6-luna/xhigh
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/bench/task_04_insert_public.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
package api

type Server struct {
	Port int
}

func (s *Server) Start() {}

func (s *Server) internalRun() {}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency43.86s64.65s+47.4%45.21s37.75s-16.5%
Process Start → First Event0.12s0.13s—0.48s0.14s—
First Event → First Tool Call12.07s11.52s—9.86s8.01s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles34+33.3%64-33.3%
Initial Load / Discovery Turns220%42-50.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)34+33.3%64-33.3%
Output Tokens16701905+14.1%17471285-26.4%
Reasoning / Thinking Tokens9921514+52.6%797883+10.8%
Total Input Tokens79776110162+38.1%100523111253+10.7%
Cached Input Tokens6451294976+47.2%8780892928+5.8%
Uncached Input Tokens1526415186-0.5%1271518325+44.1%
Cost0.08760.1044+19.3%0.08550.0961+12.4%
Cached vs Uncached Token Ratio4.23:16.25:1+48.0%6.91:15.07:1-26.6%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g '*.go' -g 'PROJECT_BRIEF.md' -g '"'!vendor'"' && \
git status --short"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '*.go' -g 'docs/adr/README.md' -g 'docs/research/README.md' | head -80"
2✓ ✓
/bin/zsh -lc "sed -n '1,240p' api/server.go"
✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'api/*.go' -g 'go.mod' -g 'go.sum' -g 'README*' | sort && \
sed -n '1,240p' api/server.go"
3✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os
import tempfile

path = Path('api/server.go')
original = path.read_bytes()
needle = b'func (s *Server) Start() {}\\n\\n'
if original.count(needle) "'!= 1:
    raise SystemExit('"'expected insertion point exactly once')
updated = original.replace(
    needle,
    needle + b'func InitServer() *Server {\\n\\treturn &Server{}\\n}\\n\\n',
)
fd, temporary = tempfile.mkstemp(prefix='.server.go.', dir=path.parent)
try:
    with os.fdopen(fd, 'wb') as target:
        target.write(updated)
        target.flush()
        os.fsync(target.fileno())
    os.chmod(temporary, path.stat().st_mode)
    os.replace(temporary, path)
    directory_fd = os.open(path.parent, os.O_RDONLY)
    try:
        os.fsync(directory_fd)
    finally:
        os.close(directory_fd)
except BaseException:
    try:
        os.unlink(temporary)
    except FileNotFoundError:
        pass
    raise
PY
sed -n '1,120p' api/server.go"
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "public_end",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}
4—✓ ✓
/bin/zsh -lc "git status --short && \
git diff --check && \
git diff -- api/server.go go.mod go.sum api/server_test.go && \
sed -n '1,120p' api/server.go"
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc 'rg -n "type Server|func .*Server|''^func [a-z]" --glob '"'*.go' ."
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g '*.go' -g 'go.mod' -g 'go.sum' | head -100"
2✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS.md' -g 'AGENTS.override.md'"
✓ ✓
/bin/zsh -lc "sed -n '1,240p' api/server.go && \
rg -n \"type Server|func \\(.*\\) Init|func [a-z].*Server|func InitServer\" --glob '*.go' ."
3✓ ✓
/bin/zsh -lc "sed -n '1,120p' api/server.go"
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": false,
  "file": "api/server.go",
  "placement": "before_symbol",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}",
  "target_symbol": "internalRun"
}
4✓ ✓
/bin/zsh -lc 'git status --short'
✓ ✓
/bin/zsh -lc "sed -n '1,120p' api/server.go && \
git diff -- api/server.go go.mod go.sum api/server_test.go && \
git status --short"
5✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os
import tempfile

path = Path('api/server.go')
data = path.read_bytes()
old = b'func (s *Server) Start() {}\\n\\nfunc (s *Server) internalRun() {}\\n'
new = b'func (s *Server) Start() {}\\n\\nfunc InitServer() *Server {\\n\\treturn &Server{}\\n}\\n\\nfunc (s *Server) internalRun() {}\\n'
if data.count(old) "'!= 1:
    raise SystemExit('"'expected server method block exactly once')
mode = path.stat().st_mode
fd, temporary = tempfile.mkstemp(prefix='.server.go.', dir=path.parent)
try:
    with os.fdopen(fd, 'wb') as stream:
        stream.write(data.replace(old, new))
        stream.flush()
        os.fsync(stream.fileno())
    os.chmod(temporary, mode)
    os.replace(temporary, path)
    directory_fd = os.open(path.parent, os.O_RDONLY)
    try:
        os.fsync(directory_fd)
    finally:
        os.close(directory_fd)
except BaseException:
    try:
        os.unlink(temporary)
    except FileNotFoundError:
        pass
    raise
PY
sed -n '1,120p' api/server.go
git diff -- api/server.go"
—
6? ✗
/bin/zsh -lc 'git rev-parse --show-toplevel && \
git status --short -- api/server.go && \
git ls-files --stage -- api/server.go && \
git diff --no-index /dev/null api/server.go'
—

Test case: task-07-generate-template-main

Target: codex/gpt-5.6-luna/high

crypto_rand prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-5.6-luna/high
Prompt variantcrypto_rand
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)⚠ MCP (Large)Δ (Large)
Wall-Clock Latency73.24s134.25s+83.3%65.94s76.15sN/A
Process Start → First Event0.12s0.12s—0.77s0.13s—
First Event → First Tool Call7.80s8.98s—9.46s7.97s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%11N/A
Internal Tool Cycles713+85.7%47N/A
Initial Load / Discovery Turns71-85.7%43N/A
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)713+85.7%47N/A
Output Tokens22885240+129.0%23772812N/A
Reasoning / Thinking Tokens10283017+193.5%13961559N/A
Total Input Tokens212622413133+94.3%129812195700N/A
Cached Input Tokens158208379136+139.6%98816154368N/A
Uncached Input Tokens5441433997-37.5%3099641332N/A
Cost0.45070.6073+34.8%0.31760.4150N/A
Cached vs Uncached Token Ratio2.91:111.15:1+283.6%3.19:13.73:1N/A
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/AFAIL—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 22 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 25 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'Makefile' -g '*.md' | sort && \
sed -n '1,200p' main.go"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'Makefile' -g 'AGENTS*.md' -g 'README*' | sort && \
if [ -f PROJECT_BRIEF.md ]; then sed -n '1,240p' PROJECT_BRIEF.md; fi && \
printf '\\n--- main.go ---\\n' && \
sed -n '1,240p' main.go && \
printf '\\n--- Makefile ---\\n' && \
if [ -f Makefile ]; then sed -n '1,200p' Makefile; fi"
2✓ ✓
/bin/zsh -lc "git status --short && \
find . -maxdepth 2 -type f -not -path './.git/*' -print | sort"
✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- files ---' && \
rg --files -g '"'!AGENTS.override.md'"' | sort && \
printf '%s\\n' '--- status ---' && \
git status --short && \
printf '%s\\n' '--- available semedit commands ---' && \
command -v semedit || \
true && \
command -v semantic-editor || \
true && \
rg -n \"semantic_(replace_body|organize_imports|insert_function)|semedit\" .codex .agents 2>/dev/null || \
true"
3✓ ✓
/bin/zsh -lc "sed -n '1,200p' main.go"
✓ ✓
/bin/zsh -lc "sed -n '1,160p' go.mod && \
find . -maxdepth 3 -type f -not -path './.git/*' -print"
4✓ ✓
/bin/zsh -lc 'gofmt -w main.go'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl, err := template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\")\nif err != nil {\n\tlog.Fatal(err)\n}\n\nvalue, err := rand.Int(rand.Reader, big.NewInt(100))\nif err != nil {\n\tlog.Fatal(err)\n}\n\nif err := tmpl.Execute(os.Stdout, value); err != nil {\n\tlog.Fatal(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
5✓ ✓
/bin/zsh -lc 'go test ./...'
✓ ✓
/bin/zsh -lc "sed -n '1,200p' main.go && \
printf '%s\\n' '--- diff ---' && \
git diff -- main.go go.mod go.sum && \
printf '%s\\n' '--- gofmt check ---' && \
gofmt -d main.go"
6✓ ✓
/bin/zsh -lc 'go run .'
semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "text/template"
  ],
  "file": "main.go",
  "remove": [
    "html/template"
  ]
}
7✓ ✓
/bin/zsh -lc 'git diff -- main.go go.mod go.sum && \
git status --short -- main.go go.mod go.sum'
✓ ✓
/bin/zsh -lc "sed -n '1,200p' main.go && \
printf '%s\\n' '--- protected files ---' && \
git diff -- go.mod go.sum && \
printf '%s\\n' '--- status in fixture ---' && \
git status --short -- main.go go.mod go.sum docs/SUBOPTIMAL_TOOLS.md"
8—✓ ✓
/bin/zsh -lc "gofmt -w main.go && \
sed -n '1,160p' main.go && \
printf '%s\\n' '--- fixture git metadata ---' && \
git rev-parse --show-toplevel && \
git ls-files --stage -- main.go docs/SUBOPTIMAL_TOOLS.md go.mod go.sum && \
git status --short --untracked-files=all -- main.go docs/SUBOPTIMAL_TOOLS.md go.mod go.sum && \
printf '%s\\n' '--- gofmt diff ---' && \
gofmt -d main.go"
9—? ✗
/bin/zsh -lc 'make check'
10—✓ ✓
/bin/zsh -lc 'gofmt -d main.go'
11—✓ ✓
/bin/zsh -lc 'go test ./...'
12—✓ ✓
/bin/zsh -lc 'go run .'
13—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "main.go"
}
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 20 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!*go.mod'"' -g '"'!*go.sum'"' | sort && \
printf '\\n--- main.go ---\\n' && \
sed -n '1,200p' main.go && \
printf '\\n--- project docs ---\\n' && \
(test -f PROJECT_BRIEF.md && \
sed -n '1,240p' PROJECT_BRIEF.md || \
true)"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'Makefile' -g 'AGENTS*' -g 'docs/adr/README.md' -g 'docs/research/README.md' -g 'go.mod' -g 'go.sum'"
2✓ ✓
/bin/zsh -lc "git status --short && \
printf '\\n--- files ---\\n' && \
ls -la && \
printf '\\n--- related source ---\\n' && \
sed -n '1,160p' api/router.go && \
sed -n '1,160p' auth/auth.go"
✓ ✓
/bin/zsh -lc "sed -n '1,200p' main.go && \
sed -n '1,120p' go.mod && \
git status --short"
3✓ ✓
/bin/zsh -lc "gofmt -w main.go && \
go test ./... && \
printf '\\n--- main.go ---\\n' && \
sed -n '1,200p' main.go && \
printf '\\n--- diff ---\\n' && \
git diff -- main.go && \
printf '\\n--- protected-file status ---\\n' && \
git status --short -- go.mod go.sum"
✓ ✓
/bin/zsh -lc "rg --files -uu | sort | sed -n '1,160p'"
4✓ ✓
/bin/zsh -lc "go run . && \
printf '\\n--- repository metadata ---\\n' && \
git rev-parse --show-toplevel && \
git ls-files --error-unmatch main.go >/dev/null 2>&1; printf 'main.go tracked=%s\\n' \""'$?" && \
git status --short -- main.go go.mod go.sum'
✓ ✓
/bin/zsh -lc "command -v semedit || \
true; sed -n '1,120p' api/router.go; sed -n '1,80p' auth/auth.go"
5—✓ ✓
/bin/zsh -lc 'gofmt -w main.go && \
go test ./... && \
output=$(go run .) && \
printf '"'%s\\n' \""'$output" && \
test "$(printf '"'%s' \""'$output" | awk '"'{print NF}')\" -eq 3 && \
git diff --check && \
git diff -- main.go go.mod go.sum"
6—✓ ✓
/bin/zsh -lc 'git status --short -- main.go go.mod go.sum && \
git diff -- main.go'
7—✓ ✓
/bin/zsh -lc "sed -n '1,120p' main.go; git ls-files --stage -- main.go; git show HEAD:main.go 2>&1 | sed -n '1,120p'; git status --short --untracked-files=all | sed -n '1,80p'"
✕ Why no semantic edit tool was used
I did not call a `semantic_*` tool because no semedit MCP server tools were available or discoverable in the fixture environment. I judged the direct edit simpler for this minimal standalone `main.go` change.

default prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-5.6-luna/high
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)⚠ MCP (Small)Δ (Small)Vanilla (Large)⚠ MCP (Large)Δ (Large)
Wall-Clock Latency51.28s82.43sN/A61.51s67.82sN/A
Process Start → First Event0.12s0.12s—0.12s0.49s—
First Event → First Tool Call7.93s8.59s—9.85s8.72s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns11N/A11N/A
Internal Tool Cycles58N/A56N/A
Initial Load / Discovery Turns51N/A56N/A
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)58N/A56N/A
Output Tokens20222489N/A20622435N/A
Reasoning / Thinking Tokens11761259N/A10861329N/A
Total Input Tokens143772214450N/A146785172085N/A
Cached Input Tokens129280177664N/A122112126976N/A
Uncached Input Tokens1449236786N/A2467345109N/A
Cost0.23300.3852N/A0.27890.4020N/A
Cached vs Uncached Token Ratio8.92:14.83:1N/A4.95:12.81:1N/A
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/AFAIL—N/AFAIL—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 18 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!*go.mod'"' -g '"'!*go.sum'"' | sort && \
printf '\\n--- main.go ---\\n' && \
sed -n '1,200p' main.go && \
if [ -f PROJECT_BRIEF.md ]; then printf '\\n--- PROJECT_BRIEF.md ---\\n'; sed -n '1,200p' PROJECT_BRIEF.md; fi"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!go.mod'"' -g '"'!go.sum'"' | sort && \
if [ -f PROJECT_BRIEF.md ]; then sed -n '1,220p' PROJECT_BRIEF.md; fi && \
printf '\\n--- main.go ---\\n' && \
sed -n '1,220p' main.go"
2✓ ✓
/bin/zsh -lc 'gofmt -w main.go'
✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- go.mod ---' && \
sed -n '1,120p' go.mod && \
printf '%s\\n' '--- available project files ---' && \
rg --files -uu -g '"'!/.git/**'"' | sort && \
printf '%s\\n' '--- semantic tooling ---' && \
command -v semedit || \
true && \
command -v semantic-editor || \
true && \
command -v go || \
true && \
printf '%s\\n' '--- git status ---' && \
git status --short"
3✓ ✓
/bin/zsh -lc 'go run main.go'
? ✗
/bin/zsh -lc 'make check'
4✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- main.go go.mod go.sum'
✓ ✓
/bin/zsh -lc 'gofmt -w main.go'
5✓ ✓
/bin/zsh -lc "sed -n '1,120p' main.go && \
git status --short --untracked-files=all -- main.go go.mod go.sum && \
git diff --name-only -- main.go go.mod go.sum && \
git check-ignore -v main.go go.mod go.sum || \
true"
✓ ✓
/bin/zsh -lc 'go test ./...'
6—✓ ✓
/bin/zsh -lc 'go run .'
7—✓ ✓
/bin/zsh -lc 'git diff --check -- main.go && \
git status --short -- main.go go.mod go.sum && \
git diff -- main.go'
8—✓ ✓
/bin/zsh -lc "sed -n '1,220p' main.go && \
printf '%s\\n' '--- repository metadata ---' && \
git rev-parse --show-toplevel && \
git check-ignore -v main.go || \
true && \
printf '%s\\n' '--- protected-file status ---' && \
git status --short --untracked-files=all -- go.mod go.sum main.go"
✕ Why no semantic edit tool was used
I did not call a `semantic_*` tool because no semedit MCP tools were available in the callable tool list, and no local semantic-editor executable was discovered. I therefore used ordinary editing.
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 14 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'Makefile' -g 'go.mod' -g 'go.sum' -g 'docs/**' | sort && \
if [ -f PROJECT_BRIEF.md ]; then sed -n '1,220p' PROJECT_BRIEF.md; fi && \
printf '\\n--- main.go ---\\n' && \
sed -n '1,220p' main.go && \
printf '\\n--- Makefile ---\\n' && \
if [ -f Makefile ]; then sed -n '1,220p' Makefile; fi"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!*go.mod'"' -g '"'!*go.sum'"' | sort && \
printf '\\n--- main.go ---\\n' && \
sed -n '1,200p' main.go && \
printf '\\n--- status ---\\n' && \
git status --short"
2? ✗
/bin/zsh -lc "sed -n '1,120p' go.mod && \
git status --short && \
rg -n \"Hello World|text/template|rand\" . --glob '"'!go.mod'"' --glob '"'!go.sum'"'"
✓ ✓
/bin/zsh -lc 'gofmt -w main.go'
3? ✗
/bin/zsh -lc 'make check'
✓ ✓
/bin/zsh -lc 'go run .'
4✓ ✓
/bin/zsh -lc 'gofmt -w main.go && \
go test ./... && \
go run . && \
git diff -- main.go go.mod go.sum && \
git status --short -- main.go go.mod go.sum'
✓ ✓
/bin/zsh -lc 'git diff --check -- main.go && \
git diff -- main.go && \
git status --short -- main.go go.mod go.sum'
5✓ ✓
/bin/zsh -lc "sed -n '1,120p' main.go; git ls-files --stage -- main.go; git status --short --untracked-files=all -- main.go go.mod go.sum; git diff --no-ext-diff --no-textconv -- main.go"
✓ ✓
/bin/zsh -lc 'go test ./...'
6—✓ ✓
/bin/zsh -lc "sed -n '1,120p' main.go && \
git status --short --untracked-files=all -- main.go go.mod go.sum && \
git ls-files --stage -- main.go go.mod go.sum"
⚠ Why no semantic edit tool was used
I did not discover a callable `semantic_*` tool from the available semedit MCP tools in this session. I therefore used ordinary editing, which was sufficient for this small change.

prefer_discover_semedit prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-5.6-luna/high
Prompt variantprefer_discover_semedit
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency106.31s168.01s+58.0%65.28s116.66s+78.7%
Process Start → First Event0.13s0.12s—0.12s0.15s—
First Event → First Tool Call34.59s28.18s—18.50s19.78s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles514+180.0%48+100.0%
Initial Load / Discovery Turns550%42-50.0%
MCP Discovery / Schema Turns03—00—
Tool Invocations (per run)514+180.0%48+100.0%
Output Tokens22353627+62.3%23403485+48.9%
Reasoning / Thinking Tokens11052028+83.5%13382270+69.7%
Total Input Tokens304269591033+94.2%227035373394+64.5%
Cached Input Tokens271872552704+103.3%172032320512+86.3%
Uncached Input Tokens3239738329+18.3%5500352882-3.9%
Cost0.39810.6376+60.2%0.47140.5973+26.7%
Cached vs Uncached Token Ratio8.39:114.42:1+71.8%3.13:16.06:1+93.8%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'PROJECT_BRIEF.md' -g 'Makefile' -g 'main.go' -g 'go.mod' -g 'README*' -g 'docs/**' | sort"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!go.mod'"' -g '"'!go.sum'"' | sort | sed -n '1,120p'"
2✓ ✓
/bin/zsh -lc "sed -n '1,240p' AGENTS.override.md && \
printf '\\n--- main.go ---\\n' && \
sed -n '1,200p' main.go && \
printf '\\n--- go.mod ---\\n' && \
sed -n '1,120p' go.mod && \
printf '\\n--- status ---\\n' && \
git status --short"
✓ ✓
/bin/zsh -lc "sed -n '1,200p' main.go && \
printf '\\n--- status ---\\n' && \
git status --short && \
printf '\\n--- protected files ---\\n' && \
git status --short -- go.mod go.sum"
3✓ ✓
/bin/zsh -lc 'gofmt -w main.go && \
go test ./... && \
output=$(go run .) && \
printf '"'%s\\n' \""'$output" && \
case "$output" in '"'Hello World '*) ;; *) exit 1 ;; esac"
cua_repl/js (✓ ✓)
{
  "code": "await cua.getState();",
  "title": "Inspect available computer surfaces"
}
4✓ ✓
/bin/zsh -lc 'git diff --check -- main.go && \
git diff -- main.go go.mod go.sum; test ! -e go.sum'
cua_repl/js (✓ ✗): Browser is not available: iab
{
  "code": "const tab = await cua.createBrowserTab(\"iab\", \"about:blank\", {visible:false}); await tab.close();",
  "title": "Close unused browser surface"
}
5✓ ✓
/bin/zsh -lc "sed -n '1,120p' main.go && \
git status --short -- main.go go.mod go.sum && \
git ls-files --stage -- main.go go.mod go.sum"
cua_repl/js (✓ ✓)
{
  "code": "await cua.getState();",
  "title": "Confirm no external app is needed"
}
6—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"hello\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Int()); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
7—✓ ✓
/bin/zsh -lc "sed -n '1,160p' main.go"
8—semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "math/rand"
  ],
  "file": "main.go",
  "remove": [
    "crypto/rand"
  ]
}
9—✓ ✓
/bin/zsh -lc "sed -n '1,160p' main.go && \
printf '\\n--- module/toolchain ---\\n' && \
sed -n '1,80p' go.mod"
10—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "main.go"
}
11—✓ ✓
/bin/zsh -lc 'go run .'
12—✓ ✓
/bin/zsh -lc 'go test ./...'
13—✓ ✓
/bin/zsh -lc "git diff -- main.go && \
printf '\\n--- scoped status ---\\n' && \
git status --short -- main.go go.mod go.sum AGENTS.override.md"
14—✓ ✓
/bin/zsh -lc "pwd; ls -la; printf '\\n--- main content ---\\n'; sed -n '1,160p' main.go; printf '\\n--- tracked? ---\\n'; git ls-files --error-unmatch main.go; printf '\\n--- diff stat ---\\n'; git diff --stat -- main.go; printf '\\n--- status ---\\n'; git status --short --untracked-files=all | sed -n '1,40p'"
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 14 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd; rg --files -g '"'!go.mod'"' -g '"'!go.sum'"' | sort; if [ -f PROJECT_BRIEF.md ]; then sed -n '1,240p' PROJECT_BRIEF.md; fi; if [ -f main.go ]; then sed -n '1,240p' main.go; fi; if [ -f Makefile ]; then sed -n '1,200p' Makefile; fi"
✓ ✓
/bin/zsh -lc "pwd; rg --files -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'Makefile' -g 'README*' -g 'docs/**' -g 'go.mod' -g 'go.sum' | sort; git status --short"
2✓ ✓
/bin/zsh -lc "sed -n '1,160p' go.mod; for f in api/router.go auth/auth.go cache/cache.go config/config.go db/db.go health/health.go logging/logging.go metrics/metrics.go models/models.go utils/utils.go; do echo \"--- "'$f"; sed -n '"'1,140p' \""'$f"; done; git status --short'
✓ ✓
/bin/zsh -lc "sed -n '1,200p' main.go; if [ -f PROJECT_BRIEF.md ]; then sed -n '1,240p' PROJECT_BRIEF.md; fi; if [ -f Makefile ]; then sed -n '1,220p' Makefile; fi; sed -n '1,120p' go.mod"
3✓ ✓
/bin/zsh -lc 'gofmt -w main.go; go test ./...; output=$(go run .); printf '"'PROGRAM_OUTPUT=%s\\n' \""'$output"; git diff --check; printf '"'%s\\n' '--- main.go diff'; git diff -- main.go; printf '%s\\n' '--- protected-file changes'; git diff --name-only -- go.mod go.sum"
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"hello\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Int()); err != nil {\n    panic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4✓ ✓
/bin/zsh -lc "sed -n '1,120p' main.go; git rev-parse --show-toplevel; git ls-files --stage -- main.go; git status --short --untracked-files=all -- main.go; git check-ignore -v main.go || \
true"
✓ ✓
/bin/zsh -lc "sed -n '1,200p' main.go; if [ -f docs/SUBOPTIMAL_TOOLS.md ]; then sed -n '1,220p' docs/SUBOPTIMAL_TOOLS.md; else echo 'docs/SUBOPTIMAL_TOOLS.md: absent'; fi; git diff -- main.go"
5—semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "math/rand"
  ],
  "file": "main.go",
  "remove": [
    "crypto/rand"
  ]
}
6—✓ ✓
/bin/zsh -lc "sed -n '1,200p' main.go; git diff -- main.go; git diff -- docs/SUBOPTIMAL_TOOLS.md"
7—✓ ✓
/bin/zsh -lc 'git status --short -- main.go docs/SUBOPTIMAL_TOOLS.md; git ls-files --stage -- main.go docs/SUBOPTIMAL_TOOLS.md; gofmt -d main.go; go test ./...'
8—✓ ✓
/bin/zsh -lc 'set -e
gofmt -d main.go
go test ./...
output=$(go run .)
printf '"'%s\\n' \""'$output"
printf '"'%s\\n' \""'$output" | rg -q '"'"'^Hello World [0-9]+$'"'
if [ -e go.sum ]; then git diff --no-ext-diff -- go.sum; fi
git diff --no-ext-diff -- go.mod"

Target: codex/gpt-5.6-luna/medium

crypto_rand prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-5.6-luna/medium
Prompt variantcrypto_rand
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency44.43s97.13s+118.6%43.00s74.80s+74.0%
Process Start → First Event1.47s1.46s—0.12s0.14s—
First Event → First Tool Call8.17s8.17s—8.15s9.72s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles38+166.7%48+100.0%
Initial Load / Discovery Turns31-66.7%41-75.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)38+166.7%48+100.0%
Output Tokens14933229+116.3%15481983+28.1%
Reasoning / Thinking Tokens6061698+180.2%589643+9.2%
Total Input Tokens122958301534+145.2%143436280978+95.9%
Cached Input Tokens91648270592+195.3%107776253440+135.2%
Uncached Input Tokens3131030942-1.2%3566027538-22.8%
Cost0.26530.4378+65.0%0.29630.3432+15.8%
Cached vs Uncached Token Ratio2.93:18.75:1+198.8%3.02:19.20:1+204.5%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 20 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!go.mod'"' -g '"'!go.sum'"' | sort && \
printf '\\n--- main.go ---\\n' && \
sed -n '1,200p' main.go && \
printf '\\n--- project docs ---\\n' && \
if [ -f PROJECT_BRIEF.md ]; then sed -n '1,240p' PROJECT_BRIEF.md; fi"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!*go.mod'"' -g '"'!*go.sum'"' | head -80 && \
printf '\\n--- main.go ---\\n' && \
sed -n '1,200p' main.go && \
printf '\\n--- project docs ---\\n' && \
if [ -f PROJECT_BRIEF.md ]; then sed -n '1,220p' PROJECT_BRIEF.md; fi && \
if [ -f Makefile ]; then sed -n '1,220p' Makefile; fi"
2✓ ✓
/bin/zsh -lc 'gofmt -w main.go && \
go run main.go && \
git diff --check && \
git status --short && \
git diff -- main.go'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "n, err := rand.Int(rand.Reader, big.NewInt(100))\nif err != nil {\n\tlog.Fatal(err)\n}\n\ntmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World: {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, n); err != nil {\n\tlog.Fatal(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
3? ✗
/bin/zsh -lc "sed -n '1,200p' main.go && \
git check-ignore -v main.go go.mod go.sum 2>/dev/null || \
true && \
test "'! -e go.mod && \
test ! -e go.sum && \
git diff --check'
✓ ✓
/bin/zsh -lc "sed -n '1,200p' main.go && \
printf '\\n--- diff ---\\n' && \
git diff -- main.go && \
printf '\\n--- format check ---\\n' && \
test -z \""'$(gofmt -d main.go)" && \
echo gofmt-ok && \
printf '"'\\n--- run ---\\n' && \
go run ./main.go && \
printf '\\n--- status ---\\n' && \
git status --short"
4—semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "text/template"
  ],
  "file": "main.go",
  "remove": [
    "html/template"
  ]
}
5—✓ ✓
/bin/zsh -lc 'mkdir -p docs'
6—✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- main.go ---' && \
sed -n '1,200p' main.go && \
printf '%s\\n' '--- validation ---' && \
test -z \""'$(gofmt -d main.go)" && \
echo gofmt-ok && \
GO111MODULE=off go test ./... && \
printf '"'%s\\n' '--- execution ---' && \
GO111MODULE=off go run ./main.go && \
printf '%s\\n' '--- protected files ---' && \
if [ -e go.mod ] || \
[ -e go.sum ]; then echo 'protected file present; unchanged check requires git status'; else echo 'no go.mod or go.sum in fixture'; fi && \
git status --short -- main.go go.mod go.sum docs/SUBOPTIMAL_TOOLS.md"
7—semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "crypto/rand",
    "log",
    "math/big",
    "os",
    "text/template"
  ],
  "file": "main.go",
  "remove": [
    "crypto/rand",
    "log",
    "math/big",
    "os",
    "text/template"
  ]
}
8—✓ ✓
/bin/zsh -lc "sed -n '1,100p' main.go && \
test -z \""'$(gofmt -d main.go)" && \
GO111MODULE=off go test ./... && \
GO111MODULE=off go run ./main.go'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 25 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 20 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!*go.mod'"' -g '"'!*go.sum'"' | head -80 && \
printf '\\n--- main.go ---\\n' && \
sed -n '1,200p' main.go && \
printf '\\n--- project docs ---\\n' && \
if [ -f PROJECT_BRIEF.md ]; then sed -n '1,200p' PROJECT_BRIEF.md; fi"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!AGENTS.override.md'"' -g 'PROJECT_BRIEF.md' -g 'README*' -g 'main.go' -g 'Makefile' -g 'go.mod' -g 'go.sum' -g '.codex/**' | sort && \
printf '\\n--- main.go ---\\n' && \
sed -n '1,200p' main.go && \
printf '\\n--- project brief ---\\n' && \
if [ -f PROJECT_BRIEF.md ]; then sed -n '1,240p' PROJECT_BRIEF.md; fi && \
printf '\\n--- make targets ---\\n' && \
if [ -f Makefile ]; then sed -n '1,200p' Makefile; fi"
2? ✗
/bin/zsh -lc 'gofmt -w main.go && \
make check'
✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- files ---' && \
rg --files -uu | sort && \
printf '%s\\n' '--- git status ---' && \
git status --short && \
printf '%s\\n' '--- go.mod ---' && \
sed -n '1,120p' go.mod && \
printf '%s\\n' '--- available tools ---' && \
command -v semedit || \
true && \
command -v go || \
true"
3✓ ✓
/bin/zsh -lc "go test ./... && \
go run . && \
printf '\\n--- diff ---\\n' && \
git diff -- main.go go.mod go.sum && \
printf '\\n--- status ---\\n' && \
git status --short -- main.go go.mod go.sum"
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nn, err := rand.Int(rand.Reader, big.NewInt(100))\nif err != nil {\n\tlog.Fatal(err)\n}\nif err := tmpl.Execute(os.Stdout, n); err != nil {\n\tlog.Fatal(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4✓ ✓
/bin/zsh -lc "sed -n '1,200p' main.go && \
git status --short && \
git diff --no-ext-diff --no-textconv -- main.go; git ls-files --stage main.go; git check-ignore -v main.go || \
true"
✓ ✓
/bin/zsh -lc "sed -n '1,200p' main.go && \
printf '\\n--- diff ---\\n' && \
git diff -- main.go go.mod go.sum && \
printf '\\n--- test ---\\n' && \
go test ./... && \
printf '\\n--- run ---\\n' && \
go run ."
5—semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "text/template"
  ],
  "file": "main.go",
  "remove": [
    "html/template"
  ]
}
6—✓ ✓
/bin/zsh -lc "sed -n '1,80p' main.go && \
printf '\\n--- verification ---\\n' && \
go test ./... && \
output="'$(go run .) && \
printf '"'%s\\n' \""'$output" && \
case "$output" in '"'Hello World '* ) ;; * ) exit 1 ;; esac && \
test \""'$(git diff -- go.mod go.sum | wc -c | tr -d '"' ')\" -eq 0"
7—semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "crypto/rand",
    "log",
    "math/big",
    "os",
    "text/template"
  ],
  "file": "main.go",
  "remove": [
    "crypto/rand",
    "log",
    "math/big",
    "os",
    "text/template"
  ]
}
8—✓ ✓
/bin/zsh -lc "sed -n '1,40p' main.go && \
gofmt -d main.go && \
go test ./... >/tmp/semedit-test-output && \
tail -n 5 /tmp/semedit-test-output && \
output="'$(go run .) && \
printf '"'%s\\n' \""'$output" && \
case "$output" in '"'Hello World '* ) ;; * ) exit 1 ;; esac"

default prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-5.6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency36.93s60.47s+63.7%44.29s49.99s+12.9%
Process Start → First Event0.14s0.14s—0.12s0.13s—
First Event → First Tool Call8.20s8.26s—8.17s13.15s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles47+75.0%54-20.0%
Initial Load / Discovery Turns41-75.0%51-80.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)47+75.0%54-20.0%
Output Tokens11161756+57.3%1700890-47.6%
Reasoning / Thinking Tokens479653+36.3%756401-47.0%
Total Input Tokens120420231911+92.6%145999167339+14.6%
Cached Input Tokens105984183040+72.7%128256144896+13.0%
Uncached Input Tokens1443648871+238.5%1774322443+26.5%
Cost0.17300.4081+135.9%0.22650.2234-1.4%
Cached vs Uncached Token Ratio7.34:13.75:1-49.0%7.23:16.46:1-10.7%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 15 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!*go.mod'"' -g '"'!*go.sum'"' | sed -n '1,80p' && \
printf '\\n--- main.go ---\\n' && \
sed -n '1,200p' main.go"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!AGENTS.override.md'"' -g '"'!go.mod'"' -g '"'!go.sum'"' | sort && \
printf '\\n--- main.go ---\\n' && \
sed -n '1,200p' main.go && \
printf '\\n--- project docs ---\\n' && \
if [ -f PROJECT_BRIEF.md ]; then sed -n '1,240p' PROJECT_BRIEF.md; fi"
2✓ ✓
/bin/zsh -lc 'gofmt -w main.go'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"hello\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Int()); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
3✓ ✓
/bin/zsh -lc 'go run main.go'
✓ ✓
/bin/zsh -lc "sed -n '1,160p' main.go && \
printf '\\n--- protected-file status ---\\n' && \
git status --short -- go.mod go.sum main.go"
4✓ ✓
/bin/zsh -lc 'git diff -- main.go go.mod go.sum && \
git status --short'
semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "math/rand",
    "text/template"
  ],
  "file": "main.go",
  "remove": [
    "crypto/rand",
    "html/template"
  ]
}
5—? ✗
/bin/zsh -lc "sed -n '1,160p' main.go && \
printf '\\n--- make check ---\\n' && \
make check && \
printf '\\n--- run ---\\n' && \
go run ."
6—✓ ✓
/bin/zsh -lc 'gofmt -d main.go && \
go test ./... && \
go run .'
7—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- main.go go.mod go.sum'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 15 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 15 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!*go.sum'"' -g '"'!*go.mod'"' | head -80 && \
printf '\\n--- main.go ---\\n' && \
sed -n '1,200p' main.go && \
printf '\\n--- project docs ---\\n' && \
for f in PROJECT_BRIEF.md Makefile; do if [ -f \""'$f" ]; then echo "### $f"; sed -n '"'1,220p' \""'$f"; fi; done'
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'Makefile' -g 'README*' -g 'AGENTS*' | sort && \
sed -n '1,200p' main.go"
2✓ ✓
/bin/zsh -lc "sed -n '1,120p' go.mod && \
printf '\\n--- relevant source conventions ---\\n' && \
sed -n '1,160p' api/router.go && \
sed -n '1,120p' logging/logging.go"
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": false,
  "body": "tmpl := template.Must(template.New(\"hello\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Int()); err != nil {\n\tlog.Fatal(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
3✓ ✓
/bin/zsh -lc 'gofmt -w main.go && \
git diff --check && \
go test ./...'
semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "log",
    "math/rand",
    "os",
    "text/template"
  ],
  "file": "main.go"
}
4✓ ✓
/bin/zsh -lc "go run . && \
printf '\\n--- diff ---\\n' && \
git diff -- main.go go.mod go.sum"
✓ ✓
/bin/zsh -lc "sed -n '1,120p' main.go && \
git diff -- main.go go.mod go.sum && \
go run ."
5✓ ✓
/bin/zsh -lc "sed -n '1,120p' main.go && \
git status --short --untracked-files=all && \
git diff --no-ext-diff --no-color -- main.go"
—

prefer_discover_semedit prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-5.6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency36.35s80.26s+120.8%46.11s71.58s+55.2%
Process Start → First Event0.12s0.11s—0.12s0.12s—
First Event → First Tool Call14.21s16.53s—12.69s16.10s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles28+300.0%47+75.0%
Initial Load / Discovery Turns220%42-50.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)28+300.0%47+75.0%
Output Tokens12402076+67.4%17531978+12.8%
Reasoning / Thinking Tokens559778+39.2%791927+17.2%
Total Input Tokens168427325525+93.3%145269303447+108.9%
Cached Input Tokens144896298240+105.8%129280273920+111.9%
Uncached Input Tokens2353127285+16.0%1598929527+84.7%
Cost0.24410.3712+52.1%0.22090.3717+68.3%
Cached vs Uncached Token Ratio6.16:110.93:1+77.5%8.09:19.28:1+14.7%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 15 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 15 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!go.mod'"' -g '"'!go.sum'"' | sort && \
if [ -f PROJECT_BRIEF.md ]; then sed -n '1,240p' PROJECT_BRIEF.md; fi && \
if [ -f main.go ]; then sed -n '1,240p' main.go; fi && \
if [ -f Makefile ]; then sed -n '1,200p' Makefile; fi"
✓ ✓
/bin/zsh -lc "pwd; rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'README*' -g 'main.go' -g 'Makefile' -g 'docs/adr/README.md' -g 'docs/research/README.md' | sort; sed -n '1,240p' PROJECT_BRIEF.md 2>/dev/null; sed -n '1,220p' main.go 2>/dev/null"
2✓ ✓
/bin/zsh -lc 'gofmt -w main.go && \
git diff --check && \
go test ./... && \
go run . && \
git diff -- main.go go.mod go.sum'
✓ ✓
/bin/zsh -lc "rg --files -g '"'!AGENTS.override.md'"' | sort; git status --short; sed -n '1,120p' go.mod 2>/dev/null; sed -n '1,160p' AGENTS.override.md"
3—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"hello\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Int()); err != nil {\n\tlog.Fatal(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4—✓ ✓
/bin/zsh -lc "sed -n '1,120p' main.go"
5—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": false,
  "body": "tmpl := texttemplate.Must(texttemplate.New(\"hello\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Int()); err != nil {\n\tlog.Fatal(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
6—semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "math/rand",
    "texttemplate text/template"
  ],
  "file": "main.go",
  "remove": [
    "crypto/rand",
    "html/template"
  ]
}
7—✓ ✓
/bin/zsh -lc "sed -n '1,120p' main.go; git diff -- main.go; git diff -- go.mod go.sum"
8—✓ ✓
/bin/zsh -lc 'set -e
gofmt -w main.go
go test ./...
go vet ./...
git diff --check
test ! -e Makefile || \
make check
if git diff --quiet -- go.mod go.sum; then
  echo '"'protected module files unchanged'
else
  echo 'protected module files changed' >&2
  exit 1
fi
sed -n '1,120p' main.go
git diff -- main.go"
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 15 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 19 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- available files ---' && \
rg --files -g '"'!go.mod'"' -g '"'!go.sum'"' | sort && \
printf '%s\\n' '--- project brief ---' && \
if [ -f PROJECT_BRIEF.md ]; then sed -n '1,240p' PROJECT_BRIEF.md; else printf '%s\\n' '(none)'; fi && \
printf '%s\\n' '--- main.go ---' && \
sed -n '1,240p' main.go"
✓ ✓
/bin/zsh -lc "pwd; printf '\\n-- files --\\n'; rg --files -g '"'!*AGENTS.override.md'"' | sort; printf '\\n-- main.go --\\n'; sed -n '1,200p' main.go; printf '\\n-- project brief --\\n'; if [ -f PROJECT_BRIEF.md ]; then sed -n '1,240p' PROJECT_BRIEF.md; fi; printf '\\n-- make targets --\\n'; if [ -f Makefile ]; then sed -n '1,220p' Makefile; fi"
2✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- module metadata ---' && \
sed -n '1,120p' go.mod && \
printf '%s\\n' '--- main references ---' && \
rg -n \"Hello World|text/template|rand|func main|main\\(\" -g '"'!go.sum'"' -g '"'!go.mod'"' ."
✓ ✓
/bin/zsh -lc "sed -n '1,120p' go.mod; printf '\\n-- related source patterns --\\n'; rg -n \"template|rand|Hello|func main\" --glob '*.go' ."
3✓ ✓
/bin/zsh -lc 'gofmt -w main.go && \
go test ./... && \
git diff --check && \
git diff -- go.mod go.sum main.go && \
git status --short -- go.mod go.sum main.go'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"hello\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Int()); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4✓ ✓
/bin/zsh -lc "sed -n '1,160p' main.go && \
printf '%s\\n' '--- tracked diff ---' && \
git diff -- main.go && \
printf '%s\\n' '--- status ---' && \
git status --short"
✓ ✓
/bin/zsh -lc "sed -n '1,120p' main.go; printf '\\n-- git diff -- --\\n'; git diff -- main.go go.mod go.sum; printf '\\n-- status --\\n'; git status --short"
5—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"hello\").Parse(\"Hello World {{.}}\\n\"))\nvalue, err := rand.Int(rand.Reader, big.NewInt(1_000_000))\nif err != nil {\n\tpanic(err)\n}\nif err := tmpl.Execute(os.Stdout, value); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
6—✓ ✓
/bin/zsh -lc "sed -n '1,120p' main.go; printf '\\n-- module files unchanged --\\n'; git diff --no-ext-diff -- go.mod go.sum; printf '\\n-- checks --\\n'; gofmt -d main.go; go test ./..."
7—✓ ✓
/bin/zsh -lc 'output=$(go run .); case "$output" in '"'Hello World '* ) printf '%s\\n' \""'$output";; *) printf '"'unexpected output: %s\\n' \""'$output" >&2; exit 1;; esac'

Target: codex/gpt-6-luna/high

crypto_rand prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/high
Prompt variantcrypto_rand
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency37.30s35.62s-4.5%28.76s32.69s+13.7%
Process Start → First Event0.13s0.12s—0.16s0.13s—
First Event → First Tool Call6.37s7.08s—7.43s6.57s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles440%34+33.3%
Initial Load / Discovery Turns42-50.0%12+100.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)440%34+33.3%
Output Tokens14511144-21.2%1048882-15.8%
Reasoning / Thinking Tokens736641-12.9%579526-9.2%
Total Input Tokens117661106077-9.8%79375122646+54.5%
Cached Input Tokens9062491904+1.4%69632106240+52.6%
Uncached Input Tokens2703714173-47.6%974316406+68.4%
Cost0.11760.0807-31.4%0.06210.0852+37.2%
Cached vs Uncached Token Ratio3.35:16.48:1+93.5%7.15:16.48:1-9.4%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 24 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 20 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'main.go' -g 'PROJECT_BRIEF.md' -g 'Makefile'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'go.sum' -g 'Makefile'"
2✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
cat main.go'
✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod'
3✓ ✓
/bin/zsh -lc 'gofmt -d main.go && \
git diff -- main.go go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "n, err := rand.Int(rand.Reader, big.NewInt(100))\nif err != nil {\n\tpanic(err)\n}\n\ntmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, n.Int64()); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short'
✓ ✓
/bin/zsh -lc 'cat main.go'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 24 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 25 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short'
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'AGENTS.md' -g 'AGENTS.override.md'"
2✓ ✓
/bin/zsh -lc "python3 - <<'PY'
import os
import tempfile

path = 'main.go'
content = '''package main

import (
	\"crypto/rand\"
	\"log\"
	\"math/big\"
	\"os\"
	\"text/template\"
)

func main() {
	value, err := rand.Int(rand.Reader, big.NewInt(100))
	if err "'!= nil {
		log.Fatal(err)
	}

	greeting, err := template.New("greeting").Parse("Hello World {{.}}'"\\\\n\")
	if err "'!= nil {
		log.Fatal(err)
	}
	if err := greeting.Execute(os.Stdout, value); err != nil {
		log.Fatal(err)
	}
}
'"'''
fd, temporary_path = tempfile.mkstemp(dir='.', prefix='.main.go.')
try:
    with os.fdopen(fd, 'w') as temporary_file:
        temporary_file.write(content)
        temporary_file.flush()
        os.fsync(temporary_file.fileno())
    os.replace(temporary_path, path)
finally:
    if os.path.exists(temporary_path):
        os.unlink(temporary_path)
PY
gofmt -w main.go
git diff -- main.go go.mod go.sum"
✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short'
3? ✗
/bin/zsh -lc 'cat main.go && \
git status --short -- main.go go.mod go.sum && \
git diff --no-index /dev/null main.go'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "n, err := rand.Int(rand.Reader, big.NewInt(100))\nif err != nil {\n\tlog.Fatal(err)\n}\n\ntmpl, err := template.New(\"hello\").Parse(\"Hello World {{.}}\\n\")\nif err != nil {\n\tlog.Fatal(err)\n}\n\nif err := tmpl.Execute(os.Stdout, n.Int64()); err != nil {\n\tlog.Fatal(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4—✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short -- main.go go.mod go.sum'

default prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/high
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency29.92s32.96s+10.1%38.43s73.86s+92.2%
Process Start → First Event0.12s0.14s—0.12s0.13s—
First Event → First Tool Call10.49s7.28s—7.90s7.99s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles24+100.0%37+133.3%
Initial Load / Discovery Turns12+100.0%31-66.7%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)24+100.0%37+133.3%
Output Tokens710996+40.3%11842441+106.2%
Reasoning / Thinking Tokens380669+76.1%6721489+121.6%
Total Input Tokens56778111088+95.7%99735190264+90.8%
Cached Input Tokens4428892928+109.8%88832173824+95.7%
Uncached Input Tokens1249018160+45.4%1090316440+50.8%
Cost0.05590.0894+59.9%0.07270.1337+84.0%
Cached vs Uncached Token Ratio3.55:15.12:1+44.3%8.15:110.57:1+29.8%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "sed -n '1,200p' main.go"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'README*'"
2✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os
import tempfile

path = Path('main.go')
content = '''package main

import (
	\"math/rand\"
	\"os\"
	\"text/template\"
)

func main() {
	tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\\\n\"))
	if err := tmpl.Execute(os.Stdout, rand.Int()); err "'!= nil {
		panic(err)
	}
}
'"'''
fd, temp_name = tempfile.mkstemp(dir=path.parent, prefix='.main.go.')
try:
    with os.fdopen(fd, 'w') as f:
        f.write(content)
        f.flush()
        os.fsync(f.fileno())
    os.replace(temp_name, path)
finally:
    if os.path.exists(temp_name):
        os.unlink(temp_name)
PY
sed -n '1,120p' main.go"
✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short'
3—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4—✓ ✓
/bin/zsh -lc 'cat main.go'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 14 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "sed -n '1,200p' main.go && \
git status --short"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'PROJECT_BRIEF.md' -g 'README*' | head -30 && \
sed -n '1,160p' main.go"
2✓ ✓
/bin/zsh -lc 'git diff -- main.go && \
git status --short -- main.go go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Int()); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
3✓ ✓
/bin/zsh -lc "pwd; sed -n '1,100p' main.go; git rev-parse --show-toplevel; git status --short -- main.go go.mod go.sum"
✓ ✓
/bin/zsh -lc "cat main.go && \
sed -n '1,30p' go.mod && \
rg --files | head -80"
4—semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "math/rand/v2"
  ],
  "file": "main.go",
  "remove": [
    "crypto/rand"
  ]
}
5—✓ ✓
/bin/zsh -lc 'cat main.go'
6—✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile, time
path = Path('docs/SUBOPTIMAL_TOOLS.md')
path.parent.mkdir(parents=True, exist_ok=True)
entry = \"\"\"# Suboptimal Tool Behavior

## semantic_replace_body import resolution in "'`main.go`

- Tool: `semantic_replace_body` with `auto_organize_imports: true`
- Target: `main.go`
- Observed failure: the generated call `rand.Int()` was paired with `crypto/rand`, whose `Int` requires a reader and a `*big.Int`; semantic diagnostics reported the resulting compile error.
- Workaround: used `semantic_organize_imports` to replace `crypto/rand` with `math/rand/v2`, where `rand.Int()` has the required zero-argument API.
- Root cause: import resolution selected the wrong standard-library package for the symbol based on its name.
"""
old = path.read_text() if path.exists() else '"''
content = old.rstrip() + ('\\n\\n' if old else '') + entry
fd, name = tempfile.mkstemp(dir=path.parent, prefix='.SUBOPTIMAL_TOOLS.')
try:
    with os.fdopen(fd, 'w') as f:
        f.write(content)
        f.flush()
        os.fsync(f.fileno())
    if path.exists():
        previous = path.stat().st_mtime_ns
        now = time.time_ns()
        os.utime(name, ns=(now, max(now, previous + 1)))
    os.replace(name, path)
    dfd = os.open(path.parent, os.O_RDONLY)
    try:
        os.fsync(dfd)
    finally:
        os.close(dfd)
finally:
    if os.path.exists(name):
        os.unlink(name)
PY
cat main.go && \
cat docs/SUBOPTIMAL_TOOLS.md"
7—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- main.go docs/SUBOPTIMAL_TOOLS.md'

prefer_discover_semedit prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/high
Prompt variantprefer_discover_semedit
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency64.28s47.34s-26.3%46.38s45.78s-1.3%
Process Start → First Event0.12s0.12s—0.13s0.61s—
First Event → First Tool Call9.52s19.45s—17.32s14.61s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles440%440%
Initial Load / Discovery Turns220%42-50.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)440%440%
Output Tokens21161357-35.9%15671656+5.7%
Reasoning / Thinking Tokens1495875-41.5%8261100+33.2%
Total Input Tokens136762231348+69.2%219236192860-12.0%
Cached Input Tokens115456206848+79.2%196608163328-16.9%
Uncached Input Tokens2130624500+15.0%2262829532+30.5%
Cost0.12730.1409+10.7%0.13560.1491+9.9%
Cached vs Uncached Token Ratio5.42:18.44:1+55.8%8.69:15.53:1-36.3%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 15 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'go.sum' | sort"
✓ ✓
/bin/zsh -lc "pwd
rg --files -g 'main.go' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'AGENTS*.md'"
2✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
printf '\\n--- main.go ---\\n' && \
cat main.go"
✓ ✓
/bin/zsh -lc 'cat main.go
go version
cat go.mod'
3✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os
import tempfile

path = Path('main.go')
old_stat = path.stat()
content = '''package main

import (
\\t\"math/rand\"
\\t\"os\"
\\t\"text/template\"
)

func main() {
\\ttmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\\\n\"))
\\tif err := tmpl.Execute(os.Stdout, rand.Int()); err "'!= nil {
'"\\t\\tpanic(err)
\\t}
}
'''
fd, tmp_name = tempfile.mkstemp(prefix='.main.go.', dir=path.parent)
try:
    with os.fdopen(fd, 'w') as tmp:
        tmp.write(content)
        tmp.flush()
        os.fsync(tmp.fileno())
    os.replace(tmp_name, path)
    timestamp = max(old_stat.st_mtime_ns + 1, path.stat().st_mtime_ns)
    os.utime(path, ns=(timestamp, timestamp))
    dir_fd = os.open(path.parent, os.O_RDONLY)
    try:
        os.fsync(dir_fd)
    finally:
        os.close(dir_fd)
except BaseException:
    try:
        os.unlink(tmp_name)
    except FileNotFoundError:
        pass
    raise
PY

git diff -- main.go"
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "greeting := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := greeting.Execute(os.Stdout, rand.Intn(100)); err != nil {\n\tlog.Fatal(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short -- main.go go.mod go.sum'
✓ ✓
/bin/zsh -lc 'cat main.go
git status --short
git diff -- main.go go.mod go.sum'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 14 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g '"'!go.mod'"' -g '"'!go.sum'"'"
? ✗
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'go.mod' -g 'go.sum' | sort && \
cat AGENTS.md 2>/dev/null && \
cat PROJECT_BRIEF.md 2>/dev/null && \
cat main.go"
2✓ ✓
/bin/zsh -lc 'cat main.go'
✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
cat main.go && \
cat go.mod'
3✓ ✓
/bin/zsh -lc 'git diff -- main.go && \
gofmt -d main.go'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short -- main.go go.mod go.sum'
✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short && \
git diff -- main.go go.mod go.sum'

Target: codex/gpt-6-luna/medium

crypto_rand prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantcrypto_rand
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency18.31s36.14s+97.4%15.63s30.31s+93.9%
Process Start → First Event0.12s0.12s—0.11s0.14s—
First Event → First Tool Call4.91s6.31s—4.55s5.14s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles14+300.0%34+33.3%
Initial Load / Discovery Turns12+100.0%32-33.3%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)14+300.0%34+33.3%
Output Tokens636499-21.5%356720+102.2%
Reasoning / Thinking Tokens215142-34.0%0306—
Total Input Tokens58223121483+108.7%75373118738+57.5%
Cached Input Tokens49408109056+120.7%6656097024+45.8%
Uncached Input Tokens881512427+41.0%881321714+146.4%
Cost0.04500.0663+47.3%0.04310.0914+111.9%
Cached vs Uncached Token Ratio5.60:18.78:1+56.6%7.55:14.47:1-40.8%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSFAIL—PASSFAIL—
Oracle L3: Clean BuildPASSFAIL—PASSFAIL—
Oracle L4: Verification TestPASSFAIL—PASSFAIL—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 27 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 20 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short'
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'Makefile' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g '*semantic*' -g 'README*' | head -80 && \
git status --short"
2—✓ ✓
/bin/zsh -lc 'cat main.go'
3—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "n, err := rand.Int(rand.Reader, big.NewInt(1_000_000))\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\n\ttmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\n\tif err := tmpl.Execute(os.Stdout, n); err != nil {\n\t\tpanic(err)\n\t}",
  "file": "main.go",
  "symbol": "main"
}
4—✓ ✓
/bin/zsh -lc 'cat main.go'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 23 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 24 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g 'go.sum'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'Makefile' -g '*.md' | head -80 && \
git status --short"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod'
✓ ✓
/bin/zsh -lc 'cat main.go && \
ls -la && \
cat AGENTS.override.md'
3✓ ✓
/bin/zsh -lc "cat > main.go <<'EOF'
package main

import (
	\"crypto/rand\"
	\"fmt\"
	\"math/big\"
	\"os\"
	\"text/template\"
)

func main() {
	n, err := rand.Int(rand.Reader, big.NewInt(1<<31))
	if err "'!= nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}

	tmpl := template.Must(template.New("greeting").Parse("Hello World {{.}}'"\\n\"))
	if err := tmpl.Execute(os.Stdout, n); err "'!= nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
}
EOF
gofmt -w main.go
go run main.go'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "n, err := rand.Int(rand.Reader, big.NewInt(1000))\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\n\ttmpl, err := template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\")\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tif err := tmpl.Execute(os.Stdout, n.Int64()); err != nil {\n\t\tlog.Fatal(err)\n\t}",
  "file": "main.go",
  "symbol": "main"
}
4—✓ ✓
/bin/zsh -lc 'cat main.go && \
git diff -- main.go go.mod go.sum && \
go run main.go'

default prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency18.82s23.62s+25.5%22.17s23.98s+8.2%
Process Start → First Event0.13s0.12s—0.13s0.45s—
First Event → First Tool Call7.21s5.13s—4.47s4.69s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles24+100.0%34+33.3%
Initial Load / Discovery Turns220%32-33.3%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)24+100.0%34+33.3%
Output Tokens386492+27.5%602475-21.1%
Reasoning / Thinking Tokens0162—251145-42.2%
Total Input Tokens75314104368+38.6%97207105019+8.0%
Cached Input Tokens6246491904+47.1%8678491904+5.9%
Uncached Input Tokens1285012464-3.0%1042313115+25.8%
Cost0.05260.0623+18.5%0.05840.0635+8.7%
Cached vs Uncached Token Ratio4.86:17.37:1+51.7%8.33:17.01:1-15.8%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 12 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g 'main.go' -g 'go.mod' -g 'go.sum' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'PROJECT_BRIEF.md'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'Makefile' -g 'go.mod' -g 'docs/**'"
2✓ ✓
/bin/zsh -lc 'cat main.go'
✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod'
3—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4—✓ ✓
/bin/zsh -lc 'cat main.go'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 14 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'go.mod' -g 'go.sum' -g 'AGENTS.md' -g 'PROJECT_BRIEF.md'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'Makefile' -g '*.md' | head -80"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod && \
git status --short'
✓ ✓
/bin/zsh -lc 'cat main.go && \
cat AGENTS.override.md'
3✓ ✓
/bin/zsh -lc 'gofmt -w main.go && \
git diff -- main.go go.mod go.sum && \
go run .'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4—✓ ✓
/bin/zsh -lc 'cat main.go'

prefer_discover_semedit prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency20.91s57.76s+176.3%26.52s42.05s+58.6%
Process Start → First Event0.13s0.12s—0.12s0.12s—
First Event → First Tool Call9.88s9.91s—7.33s8.67s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles16+500.0%56+20.0%
Initial Load / Discovery Turns12+100.0%12+100.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)16+500.0%56+20.0%
Output Tokens6261708+172.8%825781-5.3%
Reasoning / Thinking Tokens2421045+331.8%166336+102.4%
Total Input Tokens134522309917+130.4%133062194484+46.2%
Cached Input Tokens110336276992+151.0%113408175104+54.4%
Uncached Input Tokens2418632925+36.1%1965419380-1.4%
Cost0.09890.1860+88.0%0.08990.1062+18.2%
Cached vs Uncached Token Ratio4.56:18.41:1+84.4%5.77:19.04:1+56.6%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 11 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 15 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS*' | head -40 && \
sed -n '1,200p' main.go"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'PROJECT_BRIEF.md' -g 'README*' -g 'Makefile' -g 'go.mod' -g 'AGENTS.md' -g 'AGENTS.override.md'"
2—✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod && \
git status --short'
3—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Int()); err != nil {\n\tlog.Fatal(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4—✓ ✓
/bin/zsh -lc 'cat main.go && \
rg --files'
5—semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "math/rand",
    "text/template"
  ],
  "file": "main.go",
  "remove": [
    "crypto/rand",
    "html/template"
  ]
}
6—✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short -- main.go go.mod go.sum'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 17 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc 'cat main.go'
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod'"
2? ✗
/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"main.go\"); data=\"package main\\n\\nimport (\\n\\t\\\"math/rand\\\"\\n\\t\\\"os\\\"\\n\\t\\\"text/template\\\"\\n)\\n\\nfunc main() {\\n\\ttmpl := template.Must(template.New(\\\"greeting\\\").Parse(\"Hello World {{.}}\\\\n\"))\\n\\tif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err "'!= nil {'"\\n\\t\\tpanic(err)\\n\\t}\\n}\\n\"; fd, tmp = tempfile.mkstemp(dir=p.parent); f=os.fdopen(fd,\"w\"); f.write(data); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(tmp,p)'"
✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod'
3✓ ✓
/bin/zsh -lc 'gofmt -w main.go && \
cat main.go'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl, err := template.New(\"greeting\").Parse(\"Hello World: {{.}}\\n\")\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err != nil {\n\t\tpanic(err)\n\t}",
  "file": "main.go",
  "symbol": "main"
}
4✓ ✓
/bin/zsh -lc 'go run main.go'
✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short'
5✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os
import tempfile

path = Path('main.go')
data = '''package main

import (
	\"math/rand\"
	\"os\"
	\"text/template\"
)

func main() {
	tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\\\n\"))
	if err := tmpl.Execute(os.Stdout, rand.Intn(100)); err "'!= nil {
		panic(err)
	}
}
'"'''
fd, temporary = tempfile.mkstemp(dir=path.parent)
with os.fdopen(fd, 'w') as output:
    output.write(data)
    output.flush()
    os.fsync(output.fileno())
os.replace(temporary, path)
PY
gofmt -w main.go
cat main.go
go run main.go"
semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "text/template"
  ],
  "file": "main.go",
  "remove": [
    "html/template"
  ]
}
6—✓ ✓
/bin/zsh -lc 'cat main.go'

Target: codex/gpt-6-luna/xhigh

crypto_rand prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/xhigh
Prompt variantcrypto_rand
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency65.13s54.89s-15.7%69.24s55.06s-20.5%
Process Start → First Event0.12s0.12s—0.12s0.13s—
First Event → First Tool Call16.50s8.03s—8.02s9.46s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles35+66.7%34+33.3%
Initial Load / Discovery Turns220%12+100.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)35+66.7%34+33.3%
Output Tokens15261737+13.8%33882183-35.6%
Reasoning / Thinking Tokens10151260+24.1%27641648-40.4%
Total Input Tokens77030134353+74.4%83401106568+27.8%
Cached Input Tokens62464118272+89.3%7065691904+30.1%
Uncached Input Tokens1456616081+10.4%1274514664+15.1%
Cost0.08380.1072+28.0%0.12640.1075-15.0%
Cached vs Uncached Token Ratio4.29:17.35:1+71.5%5.54:16.27:1+13.1%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 24 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 19 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'go.sum'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'go.sum'"
2✓ ✓
/bin/zsh -lc 'cat main.go'
✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod'
3✓ ✓
/bin/zsh -lc "python3 - <<'PY'
import os
import tempfile
from pathlib import Path

path = Path('main.go')
content = '''package main

import (
\\t\"crypto/rand\"
\\t\"log\"
\\t\"math/big\"
\\t\"os\"
\\t\"text/template\"
)

func main() {
\\tn, err := rand.Int(rand.Reader, big.NewInt(100))
\\tif err "'!= nil {
'"\\t\\tlog.Fatal(err)
\\t}

\\ttmpl, err := template.New(\"greeting\").Parse(\"Hello World {{.}}\\\\n\")
\\tif err "'!= nil {
'"\\t\\tlog.Fatal(err)
\\t}
\\tif err := tmpl.Execute(os.Stdout, n); err "'!= nil {
'"\\t\\tlog.Fatal(err)
\\t}
}
'''
fd, temp_name = tempfile.mkstemp(prefix='.main.go.', dir='.')
try:
    with os.fdopen(fd, 'w') as temp_file:
        temp_file.write(content)
        temp_file.flush()
        os.fsync(temp_file.fileno())
    os.replace(temp_name, path)
    os.utime(path, None)
except Exception:
    try:
        os.unlink(temp_name)
    except FileNotFoundError:
        pass
    raise
PY
cat main.go"
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "greeting := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nvalue, err := rand.Int(rand.Reader, big.NewInt(100))\nif err != nil {\n\tpanic(err)\n}\nif err := greeting.Execute(os.Stdout, value); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4—✓ ✓
/bin/zsh -lc 'git diff -- main.go go.mod go.sum && \
cat main.go'
5—✓ ✓
/bin/zsh -lc 'git status --short && \
git ls-files --stage -- main.go go.mod go.sum'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 25 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 25 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "sed -n '1,200p' main.go && \
git status --short"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'go.sum'"
2✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os
import tempfile
import time

path = Path('main.go')
old_mtime = path.stat().st_mtime_ns
content = '''// Package main renders a greeting with a cryptographically generated integer.
package main

import (
\\t\"crypto/rand\"
\\t\"log\"
\\t\"math/big\"
\\t\"os\"
\\t\"text/template\"
)

func main() {
\\tn, err := rand.Int(rand.Reader, big.NewInt(100))
\\tif err "'!= nil {
'"\\t\\tlog.Fatalf(\"generate random integer: %v\", err)
\\t}

\\tgreeting, err := template.New(\"greeting\").Parse(\"Hello World {{.}}\\\\n\")
\\tif err "'!= nil {
'"\\t\\tlog.Fatalf(\"parse greeting template: %v\", err)
\\t}
\\tif err := greeting.Execute(os.Stdout, n); err "'!= nil {
'"\\t\\tlog.Fatalf(\"execute greeting template: %v\", err)
\\t}
}
'''
fd, temp_name = tempfile.mkstemp(prefix='.main.go.', dir=str(path.parent))
try:
    with os.fdopen(fd, 'w') as temp_file:
        temp_file.write(content)
        temp_file.flush()
        os.fsync(temp_file.fileno())
    os.replace(temp_name, path)
    new_mtime = max(time.time_ns(), old_mtime + 1)
    os.utime(path, ns=(new_mtime, new_mtime))
finally:
    if os.path.exists(temp_name):
        os.unlink(temp_name)
PY
gofmt -w main.go
git diff -- main.go go.mod go.sum
git status --short -- main.go go.mod go.sum"
✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short'
3✓ ✓
/bin/zsh -lc "sed -n '1,120p' main.go && \
git check-ignore -v main.go go.mod go.sum || \
true"
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "number, err := rand.Int(rand.Reader, big.NewInt(100))\nif err != nil {\n\tlog.Fatalf(\"generate random integer: %v\", err)\n}\n\ngreeting, err := template.New(\"greeting\").Parse(\"Hello World: {{.}}\\n\")\nif err != nil {\n\tlog.Fatalf(\"parse greeting template: %v\", err)\n}\n\nif err := greeting.Execute(os.Stdout, number); err != nil {\n\tlog.Fatalf(\"execute greeting template: %v\", err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4—✓ ✓
/bin/zsh -lc 'cat main.go && \
git diff -- main.go'

default prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/xhigh
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency42.44s130.08s+206.5%38.34s58.61s+52.9%
Process Start → First Event0.13s0.12s—0.54s0.12s—
First Event → First Tool Call8.81s9.69s—7.64s9.11s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles37+133.3%25+150.0%
Initial Load / Discovery Turns12+100.0%13+200.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)37+133.3%25+150.0%
Output Tokens19355759+197.6%15461619+4.7%
Reasoning / Thinking Tokens13194732+258.8%10661161+8.9%
Total Input Tokens78393196618+150.8%59152135786+129.6%
Cached Input Tokens63488173056+172.6%49408116224+135.2%
Uncached Input Tokens1490523562+58.1%974419562+100.8%
Cost0.09380.2333+148.7%0.06940.1127+62.5%
Cached vs Uncached Token Ratio4.26:17.34:1+72.4%5.07:15.94:1+17.2%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "sed -n '1,200p' main.go"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'PROJECT_BRIEF.md' -g 'README*' -g 'Makefile' -g 'AGENTS.md' -g 'AGENTS.override.md'"
2✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os
import tempfile
import time

path = Path('main.go')
content = '''package main

import (
	\"math/rand\"
	\"os\"
	\"text/template\"
)

func main() {
	t := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\\\n\"))
	if err := t.Execute(os.Stdout, rand.Intn(100)); err "'!= nil {
		panic(err)
	}
}
'"'''
old_mtime = path.stat().st_mtime_ns if path.exists() else 0
fd, tmp_name = tempfile.mkstemp(dir=path.parent, prefix=f'.{path.name}.')
try:
    with os.fdopen(fd, 'w', encoding='utf-8', newline='') as f:
        f.write(content)
        f.flush()
        os.fsync(f.fileno())
    os.replace(tmp_name, path)
    new_time = max(time.time_ns(), old_mtime + 1)
    os.utime(path, ns=(new_time, new_time))
    dir_fd = os.open(path.parent, os.O_RDONLY)
    try:
        os.fsync(dir_fd)
    finally:
        os.close(dir_fd)
except BaseException:
    try:
        os.unlink(tmp_name)
    except FileNotFoundError:
        pass
    raise
PY"
✓ ✓
/bin/zsh -lc "sed -n '1,200p' main.go"
3✓ ✓
/bin/zsh -lc "sed -n '1,120p' main.go"
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "t := template.Must(template.New(\"hello\").Parse(\"Hello World {{.}}\\n\"))\nif err := t.Execute(os.Stdout, rand.Int()); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4—✓ ✓
/bin/zsh -lc "sed -n '1,100p' main.go; rg --files docs | rg 'SUBOPTIMAL_TOOLS\\.md"'$'"' || \
true"
5—✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os
import tempfile

path = Path('docs/SUBOPTIMAL_TOOLS.md')
path.parent.mkdir(parents=True, exist_ok=True)
old = path.read_text() if path.exists() else '<"'!-- This log records semantic editing tool failures and their corrections. -->'"\\n\\n# Suboptimal Tool Behavior\\n'
entry = '''\\n## 2026-09-24: semantic_replace_body import inference\\n\\n- Tool: "'`semantic_replace_body` with `auto_organize_imports: true`'"\\n- Target: "'`main.go`'"\\n- Observed failure: import inference selected "'`crypto/rand` and `html/template` for `rand.Int()` and `template.New(...)`. Diagnostics reported that `crypto/rand.Int` requires an `io.Reader` and `*big.Int`; the request requires a standard random integer and `text/template`.'"\\n- Workaround: explicitly replace the inferred imports with "'`math/rand` and `text/template`, then re-check diagnostics.'"\\n- Root cause: package inference selected packages by matching identifiers without preserving the intended standard-library package semantics.\\n'''
new = old.rstrip() + '\\n' + entry
fd, tmp_name = tempfile.mkstemp(prefix='.SUBOPTIMAL_TOOLS.', dir=path.parent)
try:
    with os.fdopen(fd, 'w') as tmp:
        tmp.write(new)
        tmp.flush()
        os.fsync(tmp.fileno())
    os.replace(tmp_name, path)
    os.utime(path, None)
    dir_fd = os.open(path.parent, os.O_RDONLY)
    try:
        os.fsync(dir_fd)
    finally:
        os.close(dir_fd)
except BaseException:
    try:
        os.unlink(tmp_name)
    except FileNotFoundError:
        pass
    raise
PY"
6—semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "math/rand",
    "text/template"
  ],
  "file": "main.go",
  "remove": [
    "crypto/rand",
    "html/template"
  ]
}
7—✓ ✓
/bin/zsh -lc "sed -n '1,80p' main.go && \
git status --short && \
git diff --check && \
git diff -- main.go docs/SUBOPTIMAL_TOOLS.md"
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 16 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 15 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "sed -n '1,200p' main.go && \
printf '\\n--- status ---\\n' && \
git status --short"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'docs/**' | head -80 && \
git status --short"
2✓ ✓
/bin/zsh -lc "python3 - <<'PY'
import os
import tempfile
import time
from pathlib import Path

path = Path('main.go')
source = '''package main

import (
	\"math/rand\"
	\"os\"
	\"text/template\"
	\"time\"
)

func main() {
	rng := rand.New(rand.NewSource(time.Now().UnixNano()))
	greeting := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\\\n\"))
	if err := greeting.Execute(os.Stdout, rng.Intn(100)); err "'!= nil {
		panic(err)
	}
}
'"'''
old_mtime_ns = path.stat().st_mtime_ns
fd, temp_name = tempfile.mkstemp(dir=path.parent, prefix=f'.{path.name}.')
try:
    with os.fdopen(fd, 'w', encoding='utf-8', newline='') as stream:
        stream.write(source)
        stream.flush()
        os.fsync(stream.fileno())
    os.replace(temp_name, path)
    directory_fd = os.open(path.parent, os.O_RDONLY)
    try:
        os.fsync(directory_fd)
    finally:
        os.close(directory_fd)
    new_mtime_ns = max(time.time_ns(), old_mtime_ns + 1)
    os.utime(path, ns=(new_mtime_ns, new_mtime_ns))
finally:
    if os.path.exists(temp_name):
        os.unlink(temp_name)
PY
sed -n '1,120p' main.go"
✓ ✓
/bin/zsh -lc "cat main.go && \
rg --files -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'go.sum' -g 'AGENTS.md' -g 'AGENTS.override.md'"
3—✓ ✓
/bin/zsh -lc 'cat go.mod'
4—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err != nil {\n\tlog.Fatal(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
5—✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short -- main.go go.mod go.sum && \
git diff -- main.go go.mod go.sum'

prefer_discover_semedit prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/xhigh
Prompt variantprefer_discover_semedit
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency81.03s58.56s-27.7%101.52s108.55s+6.9%
Process Start → First Event0.12s0.13s—0.13s0.12s—
First Event → First Tool Call23.88s18.26s—23.34s32.28s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles64-33.3%440%
Initial Load / Discovery Turns32-33.3%32-33.3%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)64-33.3%440%
Output Tokens35041722-50.9%50423651-27.6%
Reasoning / Thinking Tokens23371241-46.9%41702916-30.1%
Total Input Tokens262880228941-12.9%172110202467+17.6%
Cached Input Tokens233216196864-15.6%146944168192+14.5%
Uncached Input Tokens2966432077+8.1%2516634275+36.2%
Cost0.20550.1664-19.0%0.21480.2098-2.3%
Cached vs Uncached Token Ratio7.86:16.14:1-21.9%5.84:14.91:1-16.0%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSFAIL—
Oracle L3: Clean BuildPASSPASS—PASSFAIL—
Oracle L4: Verification TestPASSPASS—PASSFAIL—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 16 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 18 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'go.mod' -g 'go.sum' -g 'AGENTS.md' -g 'AGENTS.override.md'"
✓ ✓
/bin/zsh -lc "rg --files -g 'main.go' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'README*' -g 'go.mod' -g 'go.sum' -g '"'!vendor'"'"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short -- main.go go.mod go.sum'
✓ ✓
/bin/zsh -lc 'cat main.go'
3✓ ✓
/bin/zsh -lc 'cat go.mod'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl, err := template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\")\nif err != nil {\n\tlog.Fatal(err)\n}\nif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err != nil {\n\tlog.Fatal(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4✓ ✓
/bin/zsh -lc "python3 - main.go <<'PY'
import os
import pathlib
import tempfile
import time

path = pathlib.Path(\"main.go\")
old_stat = path.stat()
content = \"\"\"// main.go defines the executable entry point that renders the greeting template.
package main

import (
	\"log\"
	\"math/rand\"
	\"os\"
	\"text/template\"
)

func main() {
	greeting := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\\\n\"))
	if err := greeting.Execute(os.Stdout, rand.Intn(100)); err "'!= nil {
		log.Fatal(err)
	}
}
"""
fd, temporary_name = tempfile.mkstemp(prefix=".main.go.", dir=path.parent)
try:
    with os.fdopen(fd, "w", encoding="utf-8", newline="") as temporary:
        temporary.write(content)
        temporary.flush()
        os.fsync(temporary.fileno())
    os.chmod(temporary_name, old_stat.st_mode)
    mtime_ns = max(time.time_ns(), old_stat.st_mtime_ns + 1)
    os.utime(temporary_name, ns=(mtime_ns, mtime_ns))
    with open(temporary_name, "rb") as temporary:
        os.fsync(temporary.fileno())
    os.replace(temporary_name, path)
    directory_fd = os.open(path.parent, os.O_RDONLY)
    try:
        os.fsync(directory_fd)
    finally:
        os.close(directory_fd)
except BaseException:
    try:
        os.unlink(temporary_name)
    except FileNotFoundError:
        pass
    raise
PY
'
✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short -- main.go go.mod go.sum'
5✓ ✓
/bin/zsh -lc 'git diff -- main.go go.mod go.sum && \
git status --short -- main.go go.mod go.sum'
—
6✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short && \
git rev-parse --show-toplevel'
—
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 19 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File main.go modified (4 lines -> 15 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g 'main.go' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g 'go.sum'"
✓ ✓
/bin/zsh -lc "ls -la && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'main.go' -g 'AGENTS.md'"
2✓ ✓
/bin/zsh -lc 'git status --short'
✓ ✓
/bin/zsh -lc "cat main.go && \
cat go.mod && \
find . -maxdepth 2 -type f -not -path './.git/*' -not -path './.scratch/*' -print"
3✓ ✓
/bin/zsh -lc 'cat main.go'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err != nil {\n\tlog.Fatal(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4✓ ✓
/bin/zsh -lc "python3 - <<'PY'
import os
import pathlib
import tempfile
import time

path = pathlib.Path('main.go')
old_stat = path.stat()
content = '''// This file keeps the fixture's standalone template example executable from its root.\\npackage main\\n\\nimport (\\n\\t\"log\"\\n\\t\"math/rand\"\\n\\t\"os\"\\n\\t\"text/template\"\\n)\\n\\nfunc main() {\\n\\tgreeting, err := template.New(\"greeting\").Parse(\"Hello World"'! Random integer: {{.}}'"\\\\n\")\\n\\tif err "'!= nil {'"\\n\\t\\tlog.Fatal(err)\\n\\t}\\n\\tif err := greeting.Execute(os.Stdout, rand.Intn(100)); err "'!= nil {'"\\n\\t\\tlog.Fatal(err)\\n\\t}\\n}\\n'''
fd, temp_name = tempfile.mkstemp(prefix='.main.go.', dir=path.parent)
try:
    with os.fdopen(fd, 'w', encoding='utf-8', newline='') as temp:
        temp.write(content)
        temp.flush()
        os.fsync(temp.fileno())
    os.chmod(temp_name, old_stat.st_mode)
    now = time.time_ns()
    mtime = max(now, old_stat.st_mtime_ns + 1)
    os.utime(temp_name, ns=(old_stat.st_atime_ns, mtime))
    os.replace(temp_name, path)
    dir_fd = os.open(path.parent, os.O_RDONLY)
    try:
        os.fsync(dir_fd)
    finally:
        os.close(dir_fd)
except BaseException:
    try:
        os.unlink(temp_name)
    except FileNotFoundError:
        pass
    raise
PY"
✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- main.go go.mod go.sum && \
cat main.go'

Test case: task-09-composite-refactor

Target: codex/gpt-5.6-luna/high

default prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-09-composite-refactor
Targetcodex/gpt-5.6-luna/high
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/bench/task_09_composite_refactor.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
package config

import (
	"archive/tar"
	"fmt"
)

type Config struct {
	Host string
	Port int
}

func (c *Config) Addr() string {
	return fmt.Sprintf("%s:%d", c.Host, c.Port)
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency60.07s116.36s+93.7%55.00s97.24s+76.8%
Process Start → First Event0.12s0.12s—0.12s0.53s—
First Event → First Tool Call9.19s15.24s—7.50s10.65s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles711+57.1%811+37.5%
Initial Load / Discovery Turns74-42.9%84-50.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)711+57.1%811+37.5%
Output Tokens19742354+19.3%20493245+58.4%
Reasoning / Thinking Tokens7091002+41.3%7861802+129.3%
Total Input Tokens189434363428+91.8%214048331846+55.0%
Cached Input Tokens152320317184+108.2%195072304128+55.9%
Uncached Input Tokens3711446244+24.6%1897627718+46.1%
Cost0.34220.4905+43.3%0.27750.4421+59.3%
Cached vs Uncached Token Ratio4.10:16.86:1+67.1%10.28:110.97:1+6.7%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 18 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS*.md' -g 'Makefile' -g 'go.mod' -g 'go.sum' && \
sed -n '1,220p' config/config.go && \
if [ -f PROJECT_BRIEF.md ]; then sed -n '1,220p' PROJECT_BRIEF.md; fi"
✓ ✓
/bin/zsh -lc "pwd; rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS*.md' -g 'Makefile' -g 'config/config.go' -g 'docs/adr/README.md' -g 'docs/research/README.md' -g '"'!go.mod'"' -g '"'!go.sum'"' | sort"
2✓ ✓
/bin/zsh -lc "rg -n --glob '"'!AGENTS.override.md'"' --glob '"'!go.sum'"' --glob '"'!go.mod'"' 'Config|\\.Port\\b|Port:' ."
✓ ✓
/bin/zsh -lc "git status --short; sed -n '1,220p' config/config.go; rg -n --glob '"'!go.mod'"' --glob '"'!go.sum'"' --glob '"'!*_test.go'"' 'Config\\.Port|\\.Port\\b|ListenPort|DefaultConfig|config\\.Config|Config\\{' ."
3✓ ✓
/bin/zsh -lc "sed -n '1,220p' config/config_test.go && \
sed -n '1,220p' Makefile 2>/dev/null || \
true && \
git status --short"
✓ ✓
/bin/zsh -lc "rg --files | sort | sed -n '1,200p'; rg -n --glob '"'!go.mod'"' --glob '"'!go.sum'"' 'Host|Port|Addr\\(' ."
4✓ ✓
/bin/zsh -lc "rg -n --glob '"'!config/config.go'"' --glob '"'!config/config_test.go'"' 'localhost|8080|ListenPort|DefaultConfig' . || \
true"
✓ ✓
/bin/zsh -lc "sed -n '1,160p' config/config_test.go; sed -n '1,160p' AGENTS.override.md; sed -n '1,120p' go.mod"
5? ✗
/bin/zsh -lc 'gofmt -w config/config.go && \
make check'
semedit/semantic_rename (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
6✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --check && \
git diff -- config/config.go && \
git diff --name-only -- go.mod go.sum config/config.go'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "config/config.go",
  "language": "go",
  "placement": "before_symbol",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{Host: \"localhost\", ListenPort: 8080}\n}",
  "target_symbol": "Addr"
}
7✓ ✓
/bin/zsh -lc "sed -n '1,160p' config/config.go && \
git status --short -- config/config.go config/config_test.go go.mod go.sum"
semedit/semantic_organize_imports (✓ ✓)
{
  "file": "config/config.go"
}
8—✓ ✓
/bin/zsh -lc "sed -n '1,160p' config/config.go; git diff -- config/config.go config/config_test.go go.mod go.sum; git status --short -- config/config.go config/config_test.go go.mod go.sum"
9—✓ ✓
/bin/zsh -lc "git rev-parse --show-toplevel; git ls-files -v -- config/config.go config/config_test.go go.mod go.sum; git diff --no-ext-diff -- config/config.go; git show HEAD:config/config.go 2>&1 | sed -n '1,100p'"
10—✓ ✓
/bin/zsh -lc 'ls -la; gofmt -d config/config.go; go test ./...'
11—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "config/config.go"
}
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'config/config.go' -g 'Makefile' -g 'go.mod' -g 'go.sum' && \
sed -n '1,220p' config/config.go"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'docs/adr/README.md' -g 'docs/research/README.md' -g 'config/config.go' -g 'Makefile' -g 'go.mod' -g 'AGENTS*.md'"
2✓ ✓
/bin/zsh -lc "rg -n --glob '"'!AGENTS.override.md'"' --glob '"'!go.mod'"' --glob '"'!go.sum'"' 'Config|\\.Port|ListenPort|DefaultConfig|archive/tar' ."
✓ ✓
/bin/zsh -lc "sed -n '1,240p' config/config.go && \
printf '\\n--- references ---\\n' && \
rg -n --glob '*.go' '\\.Port\\b|Port\\s*:' ."
3✓ ✓
/bin/zsh -lc "sed -n '1,200p' config/config_test.go && \
rg --files | sort | sed -n '1,200p' && \
git status --short"
✓ ✓
/bin/zsh -lc "rg --files -g '"'!AGENTS.override.md'"' | sort && \
printf '\\n--- module ---\\n' && \
sed -n '1,120p' go.mod"
4✓ ✓
/bin/zsh -lc 'gofmt -w config/config.go'
✓ ✓
/bin/zsh -lc "sed -n '1,240p' config/config_test.go && \
rg -n --glob '*.go' 'Config\\{|DefaultConfig|ListenPort|Host:' ."
5✓ ✓
/bin/zsh -lc 'go test ./config'
semedit/semantic_rename (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
6✓ ✓
/bin/zsh -lc 'git diff --check && \
git diff -- config/config.go config/config_test.go go.mod go.sum && \
git status --short -- config/config.go config/config_test.go go.mod go.sum'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "config/config.go",
  "placement": "before_symbol",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}",
  "target_symbol": "Config"
}
7✓ ✓
/bin/zsh -lc "sed -n '1,200p' config/config.go && \
git status --short && \
git diff --no-ext-diff -- config/config.go"
✓ ✓
/bin/zsh -lc "sed -n '1,180p' config/config.go && \
git diff -- config/config.go && \
git status --short"
8✓ ✓
/bin/zsh -lc 'git rev-parse --show-toplevel && \
git ls-files --stage -- config/config.go config/config_test.go go.mod go.sum && \
git check-ignore -v config/config.go config/config_test.go go.mod go.sum || \
true && \
git diff HEAD --stat -- config/config.go'
✓ ✓
/bin/zsh -lc 'git status --short -- config/config.go config/config_test.go go.mod go.sum && \
git diff --no-ext-diff -- config/config.go'
9—✓ ✓
/bin/zsh -lc 'ls -la .git .gitignore 2>/dev/null; git diff --no-index /dev/null config/config.go || \
true'
10—✓ ✓
/bin/zsh -lc "rg -n --glob '*.go' '\\bPort\\b|archive/tar|ListenPort|DefaultConfig' . && \
go test ./..."
11—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "config/config.go"
}

Target: codex/gpt-5.6-luna/medium

default prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-09-composite-refactor
Targetcodex/gpt-5.6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/bench/task_09_composite_refactor.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
package config

import (
	"archive/tar"
	"fmt"
)

type Config struct {
	Host string
	Port int
}

func (c *Config) Addr() string {
	return fmt.Sprintf("%s:%d", c.Host, c.Port)
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency49.02s56.52s+15.3%36.83s62.68s+70.2%
Process Start → First Event0.12s0.12s—0.12s0.13s—
First Event → First Tool Call7.11s7.32s—6.31s8.63s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles78+14.3%58+60.0%
Initial Load / Discovery Turns70-100.0%50-100.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)78+14.3%58+60.0%
Output Tokens15441636+6.0%13331851+38.9%
Reasoning / Thinking Tokens369505+36.9%376537+42.8%
Total Input Tokens188643263901+39.9%163664236270+44.4%
Cached Input Tokens153344236032+53.9%148480205568+38.4%
Uncached Input Tokens3529927869-21.0%1518430702+102.2%
Cost0.31060.3216+3.6%0.20140.3279+62.8%
Cached vs Uncached Token Ratio4.34:18.47:1+95.0%9.78:16.70:1-31.5%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 18 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'Makefile' -g 'AGENTS*.md' | sort && \
sed -n '1,220p' config/config.go"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'Makefile' -g 'AGENTS*' -g 'docs/adr/README.md' -g 'docs/research/README.md' -g '*semantic*' -g '*semedit*'"
2✓ ✓
/bin/zsh -lc "rg -n --glob '"'!AGENTS.override.md'"' --glob '"'!*.sum'"' --glob '"'!go.mod'"' '\\bPort\\b|Config\\s*\\{' . && \
rg -n 'make check|go test|gofmt' Makefile README.md docs 2>/dev/null || \
true && \
git status --short"
✓ ✓
/bin/zsh -lc "sed -n '1,240p' AGENTS.override.md && \
printf '\\n--- config/config.go ---\\n' && \
nl -ba config/config.go && \
printf '\\n--- files ---\\n' && \
rg --files"
3✓ ✓
/bin/zsh -lc "rg --files | sort | head -200 && \
find . -maxdepth 3 -type f -not -path './.git/*' -print | sort"
✓ ✓
/bin/zsh -lc "sed -n '1,240p' config/config_test.go && \
printf '\\n--- go.mod ---\\n' && \
sed -n '1,120p' go.mod && \
printf '\\n--- available executables ---\\n' && \
command -v semedit || \
true && \
command -v go || \
true"
4✓ ✓
/bin/zsh -lc "sed -n '1,240p' config/config_test.go && \
sed -n '1,120p' go.mod"
semedit/semantic_rename (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
5✓ ✓
/bin/zsh -lc 'gofmt -w config/config.go && \
go test ./...'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "config/config.go",
  "placement": "after_symbol",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}",
  "target_symbol": "Config"
}
6✓ ✓
/bin/zsh -lc 'git diff -- config/config.go go.mod go.sum config/config_test.go && \
git status --short -- config/config.go go.mod go.sum config/config_test.go'
✓ ✓
/bin/zsh -lc 'nl -ba config/config.go && \
git diff -- config/config.go && \
git status --short'
7✓ ✓
/bin/zsh -lc "sed -n '1,120p' config/config.go && \
git status --short --untracked-files=all | head -20 && \
git check-ignore -v config/config.go config/config_test.go go.mod 2>/dev/null || \
true"
✓ ✓
/bin/zsh -lc 'gofmt -d config/config.go && \
go test ./... && \
git diff -- go.mod go.sum && \
git diff -- config/config.go'
8—✓ ✓
/bin/zsh -lc "git status --short -- config/config.go go.mod go.sum config/config_test.go && \
sed -n '1,120p' config/config.go && \
test "'! -e go.sum || \
echo go.sum-present'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'Makefile' -g 'AGENTS*' | sort && \
sed -n '1,220p' config/config.go"
? ✗
/bin/zsh -lc "pwd; rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'Makefile' -g 'docs/adr/README.md' -g 'docs/research/README.md' -g 'AGENTS*.md'; printf '\\n--- config/config.go ---\\n'; sed -n '1,240p' config/config.go; printf '\\n--- tools matching semantic ---\\n'; for (const x of ALL_TOOLS) if (/semantic|semedit|codex_review|codex_ask/i.test(x.name+' '+x.description)) text(x.name+' :: '+x.description);"
2✓ ✓
/bin/zsh -lc "rg -n \"Config\\{|\\.Port\\b|DefaultConfig|ListenPort|Addr\\(\" . -g '"'!AGENTS.override.md'"' -g '"'!go.mod'"' -g '"'!go.sum'"' && \
rg --files | sort | head -80"
✓ ✓
/bin/zsh -lc "pwd; rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'Makefile' -g 'docs/adr/README.md' -g 'docs/research/README.md' -g 'AGENTS*.md'; printf '\\n--- config/config.go ---\\n'; sed -n '1,240p' config/config.go"
3✓ ✓
/bin/zsh -lc "sed -n '1,160p' config/config_test.go && \
sed -n '1,180p' api/router.go && \
rg -n \"archive/tar|Config|ListenPort|Port\" --glob '*.go' --glob '"'!config/config_test.go'"'"
✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- references ---'; rg -n --glob '*.go' --glob '"'!**/*_test.go'"' 'Config|\\.Port|Port:' .; printf '%s\\n' '--- repository files ---'; rg --files | sort | sed -n '1,160p'; printf '%s\\n' '--- git status ---'; git status --short"
4✓ ✓
/bin/zsh -lc 'gofmt -w config/config.go && \
git diff -- config/config.go config/config_test.go go.mod go.sum && \
if [ -f Makefile ]; then make check; else go test ./config; fi'
✓ ✓
/bin/zsh -lc "sed -n '1,240p' config/config_test.go; printf '\\n--- go.mod ---\\n'; sed -n '1,120p' go.mod; printf '\\n--- make/check availability ---\\n'; if test -f Makefile; then sed -n '1,180p' Makefile; else echo 'no Makefile in fixture'; fi"
5✓ ✓
/bin/zsh -lc 'git status --short && \
git diff --check && \
git diff -- config/config.go'
semedit/semantic_rename (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
6—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "config/config.go",
  "language": "go",
  "placement": "public_start",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}"
}
7—✓ ✓
/bin/zsh -lc "sed -n '1,160p' config/config.go; printf '\\n--- diff ---\\n'; git diff -- config/config.go; printf '\\n--- formatting check ---\\n'; test -z \""'$(gofmt -l config/config.go)"; printf '"'gofmt: clean\\n'; printf '\\n--- tests ---\\n'; go test ./..."
8—✓ ✓
/bin/zsh -lc 'git status --short -- config/config.go config/config_test.go go.mod go.sum; git diff --no-ext-diff -- config/config.go; git ls-files --stage -- config/config.go; git check-ignore -v config/config.go || \
true'

Target: codex/gpt-6-luna/high

default prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-09-composite-refactor
Targetcodex/gpt-6-luna/high
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/bench/task_09_composite_refactor.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
package config

import (
	"archive/tar"
	"fmt"
)

type Config struct {
	Host string
	Port int
}

func (c *Config) Addr() string {
	return fmt.Sprintf("%s:%d", c.Host, c.Port)
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency33.46s48.98s+46.4%34.30s44.50s+29.8%
Process Start → First Event0.11s0.12s—0.13s0.13s—
First Event → First Tool Call7.00s7.27s—7.76s4.06s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles46+50.0%48+100.0%
Initial Load / Discovery Turns43-25.0%35+66.7%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)46+50.0%48+100.0%
Output Tokens12741501+17.8%13601115-18.0%
Reasoning / Thinking Tokens515858+66.6%461568+23.2%
Total Input Tokens118789181626+52.9%97330204826+110.4%
Cached Input Tokens102912163840+59.2%74496187136+151.2%
Uncached Input Tokens1587717786+12.0%2283417690-22.5%
Cost0.08780.1149+30.9%0.09850.1120+13.8%
Cached vs Uncached Token Ratio6.48:19.21:1+42.1%3.26:110.58:1+224.2%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 18 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md' && \
sed -n '1,240p' config/config.go && \
rg -n '\\bConfig\\.Port\\b|\\.Port\\b|ListenPort|DefaultConfig' --glob '*.go' ."
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS*.md' -g 'config/config.go' -g 'docs/**' | head -80 && \
sed -n '1,240p' config/config.go"
2✓ ✓
/bin/zsh -lc "sed -n '1,200p' config/config_test.go && \
rg -n 'Host|ListenPort|8080|DefaultConfig' --glob '"'!config/config_test.go'"' ."
✓ ✓
/bin/zsh -lc "rg -n 'Config\\{|\\.Port|Port:' --glob '*.go' . | head -120 && \
rg -n 'DefaultConfig|default.*(port|host)|localhost|127\\.0\\.0\\.1' config --glob '*.go'"
3✓ ✓
/bin/zsh -lc 'git diff -- config/config.go && \
git status --short'
✓ ✓
/bin/zsh -lc "sed -n '1,120p' config/config_test.go && \
rg -n 'Config|8080' README* *.go config --glob '*.go' --glob '*.md' 2>/dev/null | head -100"
4✓ ✓
/bin/zsh -lc "sed -n '1,160p' config/config.go && \
git status --short -- config/config.go"
semedit/semantic_rename (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
5—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "config/config.go",
  "placement": "public_start",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{Host: \"localhost\", ListenPort: 8080}\n}"
}
6—✓ ✓
/bin/zsh -lc "sed -n '1,180p' config/config.go && \
git diff -- config/config.go go.mod go.sum && \
git status --short"
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 19 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'config/*'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'config.go' -g '*semantic*' -g 'Makefile'"
2✓ ✓
/bin/zsh -lc "cat config/config.go && \
rg -n '\\bConfig\\s*\\{|\\.Port\\b|DefaultConfig\\(' --glob '*.go'"
✓ ✓
/bin/zsh -lc 'cat config/config.go && \
git status --short'
3✓ ✓
/bin/zsh -lc "cat config/config_test.go && \
rg -n 'ListenPort|Host:|Port:' . --glob '"'!go.mod'"' --glob '"'!go.sum'"'"
✓ ✓
/bin/zsh -lc "rg -n '\\bPort\\b|Config\\s*\\{' --glob '*.go' ."
4✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os
import tempfile

path = Path('config/config.go')
source = path.read_text()
updated = '''package config

import \"fmt\"

type Config struct {
\\tHost       string
\\tListenPort int
}

func DefaultConfig() *Config {
\\treturn &Config{
\\t\\tHost:       \"localhost\",
\\t\\tListenPort: 8080,
\\t}
}

func (c *Config) Addr() string {
\\treturn fmt.Sprintf(\"%s:%d\", c.Host, c.ListenPort)
}
'''
if source == updated:
    raise SystemExit('config/config.go already has the requested contents')
old_mtime_ns = path.stat().st_mtime_ns
fd, temp_name = tempfile.mkstemp(prefix='.config.go.', dir=path.parent)
try:
    with os.fdopen(fd, 'w') as temp:
        temp.write(updated)
        temp.flush()
        os.fsync(temp.fileno())
    os.replace(temp_name, path)
    directory_fd = os.open(path.parent, os.O_RDONLY)
    try:
        os.fsync(directory_fd)
    finally:
        os.close(directory_fd)
    now_ns = path.stat().st_mtime_ns
    if now_ns <= old_mtime_ns:
        os.utime(path, ns=(max(now_ns, old_mtime_ns + 1), old_mtime_ns + 1))
except BaseException:
    try:
        os.unlink(temp_name)
    except FileNotFoundError:
        pass
    raise
PY
cat config/config.go
git status --short"
✓ ✓
/bin/zsh -lc "rg -n 'DefaultConfig|ListenPort|archive/tar|config\\.Config|config' --glob '"'!*.sum'"' --glob '"'!*.mod'"' ."
5—✓ ✓
/bin/zsh -lc 'cat config/config_test.go'
6—semedit/semantic_rename (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
7—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "config/config.go",
  "placement": "public_start",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}"
}
8—✓ ✓
/bin/zsh -lc 'cat config/config.go && \
git status --short config/config.go config/config_test.go go.mod go.sum'

Target: codex/gpt-6-luna/medium

default prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-09-composite-refactor
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/bench/task_09_composite_refactor.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
package config

import (
	"archive/tar"
	"fmt"
)

type Config struct {
	Host string
	Port int
}

func (c *Config) Addr() string {
	return fmt.Sprintf("%s:%d", c.Host, c.Port)
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency23.57s55.16s+134.0%27.72s36.08s+30.2%
Process Start → First Event0.11s0.12s—0.12s0.13s—
First Event → First Tool Call5.35s7.33s—6.82s4.58s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles59+80.0%56+20.0%
Initial Load / Discovery Turns31-66.7%330%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)59+80.0%56+20.0%
Output Tokens7221191+65.0%757727-4.0%
Reasoning / Thinking Tokens203295+45.3%85244+187.1%
Total Input Tokens117628243475+107.0%116158146121+25.8%
Cached Input Tokens105984225536+112.8%104960128256+22.2%
Uncached Input Tokens1164417939+54.1%1119817865+59.5%
Cost0.06720.1198+78.4%0.06480.0889+37.2%
Cached vs Uncached Token Ratio9.10:112.57:1+38.1%9.37:17.18:1-23.4%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 18 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g 'go.sum'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'config.go' -g 'README*' -g 'Makefile' -g 'go.mod' -g 'go.sum'"
2✓ ✓
/bin/zsh -lc 'cat config/config.go && \
git status --short'
✓ ✓
/bin/zsh -lc "cat config/config.go && \
rg -n \"semantic_(rename|insert_function|replace_body|organize_imports)|semantic editor|DefaultConfig|Config\\.Port\" . --glob '"'!*.sum'"' --glob '"'!go.sum'"'"
3✓ ✓
/bin/zsh -lc "rg -n 'Config\\{|\\.Port|DefaultConfig|ListenPort|config.Config' --glob '"'!**/*_test.go'"' ."
✓ ✓
/bin/zsh -lc "cat config/config_test.go && \
rg -n \"\\.Port\\b|DefaultConfig|Host:\" --glob '*.go' ."
4✓ ✓
/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"config/config.go\"); s=p.read_text(); s=s.replace(\"\\t\\\"archive/tar\\\"\\n\", \"\").replace(\"\\tPort int\", \"\\tListenPort int\").replace(\"c.Host, c.Port\", \"c.Host, c.ListenPort\"); s=s.replace(\"func (c *Config) Addr() string {\", \"func DefaultConfig() *Config {\\n\\treturn &Config{Host: \\\"localhost\\\", ListenPort: 8080}\\n}\\n\\nfunc (c *Config) Addr() string {\"); fd, name=tempfile.mkstemp(dir=str(p.parent), prefix=\".config.go.\"); f=os.fdopen(fd,\"w\"); f.write(s); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(name,p)' && \
cat config/config.go"
semedit/semantic_rename (✓ ✗): resolve Port in config/config.go: symbol not found
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Port",
  "to": "ListenPort"
}
5✓ ✓
/bin/zsh -lc 'gofmt -w config/config.go && \
git diff -- config/config.go && \
git status --short -- config/config.go go.mod go.sum'
✓ ✓
/bin/zsh -lc 'rg --files docs 2>/dev/null | head -20; test -f docs/SUBOPTIMAL_TOOLS.md && \
cat docs/SUBOPTIMAL_TOOLS.md || \
true'
6—semedit/semantic_rename (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
7—✓ ✓
/bin/zsh -lc 'cat config/config.go'
8—✓ ✓
/bin/zsh -lc "python3 -c 'import os,tempfile; p=\"docs/SUBOPTIMAL_TOOLS.md\"; os.makedirs(\"docs\",exist_ok=True); data=\"# Suboptimal Tool Behavior\\n\\n- Tool: "'`semantic_rename`'"\\n  Target: "'`config/config.go`'"\\n  Observed: Resolving the bare symbol "'`Port` failed with `symbol not found`.'"\\n  Workaround: Retried using the qualified symbol "'`Config.Port`, which succeeded.'"\\n  Root cause: The rename backend requires a qualified field symbol for this declaration.\\n\"; fd,t=tempfile.mkstemp(dir=\"docs\",prefix=\".SUBOPTIMAL_TOOLS.\"); f=os.fdopen(fd,\"w\"); f.write(data); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(t,p); d=os.open(\"docs\",os.O_RDONLY); os.fsync(d); os.close(d)'"
9—semedit/semantic_insert_function (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "config/config.go",
  "placement": "public_start",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}"
}
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 19 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "sed -n '1,240p' config/config.go && \
rg -n '\\.Port|Port:' --glob '*.go' ."
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'config.go' -g 'go.mod' -g 'go.sum' -g 'Makefile' -g 'SUBOPTIMAL_TOOLS.md' | sort"
2✓ ✓
/bin/zsh -lc "rg --files && \
rg -n 'DefaultConfig|ListenPort|Config\\{' . --glob '"'!AGENTS.override.md'"'"
✓ ✓
/bin/zsh -lc "cat config/config.go && \
git status --short && \
rg -n 'Config\\.Port|\\.Port|DefaultConfig' --glob '*.go' ."
3✓ ✓
/bin/zsh -lc 'cat config/config_test.go && \
cat go.mod'
✓ ✓
/bin/zsh -lc 'cat config/config_test.go'
4✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
p = Path('config/config.go')
s = p.read_text()
s = '''package config

import \"fmt\"

type Config struct {
\\tHost       string
\\tListenPort int
}

func DefaultConfig() *Config {
\\treturn &Config{
\\t\\tHost:       \"localhost\",
\\t\\tListenPort: 8080,
\\t}
}

func (c *Config) Addr() string {
\\treturn fmt.Sprintf(\"%s:%d\", c.Host, c.ListenPort)
}
'''
tmp = p.with_suffix('.go.tmp')
tmp.write_text(s)
tmp.replace(p)
PY
gofmt -w config/config.go
git diff -- config/config.go"
semedit/semantic_rename (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
5✓ ✓
/bin/zsh -lc 'cat config/config.go && \
git status --short'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "config/config.go",
  "placement": "public_start",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}"
}
6—✓ ✓
/bin/zsh -lc 'cat config/config.go && \
git diff -- config/config.go && \
git status --short -- config/config.go go.mod go.sum'

Target: codex/gpt-6-luna/xhigh

default prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-09-composite-refactor
Targetcodex/gpt-6-luna/xhigh
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/bench/task_09_composite_refactor.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
package config

import (
	"archive/tar"
	"fmt"
)

type Config struct {
	Host string
	Port int
}

func (c *Config) Addr() string {
	return fmt.Sprintf("%s:%d", c.Host, c.Port)
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency58.69s71.38s+21.6%69.55s74.18s+6.7%
Process Start → First Event0.12s0.44s—0.55s0.14s—
First Event → First Tool Call13.77s9.38s—9.76s7.65s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles58+60.0%87-12.5%
Initial Load / Discovery Turns440%440%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)58+60.0%87-12.5%
Output Tokens25812348-9.0%31282470-21.0%
Reasoning / Thinking Tokens15721752+11.5%16671822+9.3%
Total Input Tokens119570198638+66.1%189069179423-5.1%
Cached Input Tokens107008177920+66.3%171776162816-5.2%
Uncached Input Tokens1256220718+64.9%1729316607-4.0%
Cost0.11010.1475+34.0%0.14610.1359-7.0%
Cached vs Uncached Token Ratio8.52:18.59:1+0.8%9.93:19.80:1-1.3%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 19 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 18 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "sed -n '1,240p' config/config.go && \
rg -n '\\.Port\\b|\\bPort\\s*:' --glob '*.go' ."
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'go.mod' -g 'AGENTS.md' -g 'AGENTS.override.md'"
2? ✗
/bin/zsh -lc "rg -n 'Config\\{|DefaultConfig|ListenPort|config\\.Config|\\.Addr\\(\\)' --glob '"'!**/*_test.go'"' . && \
rg -n 'config' README* docs config --glob '*.md'"
✓ ✓
/bin/zsh -lc "sed -n '1,240p' config/config.go && \
rg -n '\\bConfig\\b|\\.Port\\b|Port:' --glob '*.go' ."
3✓ ✓
/bin/zsh -lc "rg --files -g '"'!**/*_test.go'"' | head -80 && \
rg -n 'localhost|127\\.0\\.0\\.1|8080|port' --glob '"'!**/*_test.go'"' ."
✓ ✓
/bin/zsh -lc "rg --files -g '*.go' -g '*.md' | head -80 && \
rg -n 'localhost|127\\.0\\.0\\.1|8080|DefaultConfig|ListenPort|Host string' . --glob '"'!go.sum'"' --glob '"'!go.mod'"'"
4✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- config/config.go'
✓ ✓
/bin/zsh -lc "sed -n '1,120p' config/config_test.go"
5✓ ✓
/bin/zsh -lc "python3 - <<'PY'
import os
import stat
import tempfile
from pathlib import Path

path = Path('config/config.go')
old = path.read_text()
new = '''package config

import \"fmt\"

type Config struct {
	Host       string
	ListenPort int
}

func DefaultConfig() *Config {
	return &Config{
		Host:       \"localhost\",
		ListenPort: 8080,
	}
}

func (c *Config) Addr() string {
	return fmt.Sprintf(\"%s:%d\", c.Host, c.ListenPort)
}
'''
if old == new:
    raise SystemExit('config/config.go already has the requested contents')
mode = stat.S_IMODE(path.stat().st_mode)
fd, temporary = tempfile.mkstemp(prefix='.config.go.', dir=path.parent)
try:
    with os.fdopen(fd, 'w') as stream:
        stream.write(new)
        stream.flush()
        os.fsync(stream.fileno())
    os.chmod(temporary, mode)
    os.replace(temporary, path)
    directory_fd = os.open(path.parent, os.O_RDONLY)
    try:
        os.fsync(directory_fd)
    finally:
        os.close(directory_fd)
except BaseException:
    try:
        os.unlink(temporary)
    except FileNotFoundError:
        pass
    raise
PY

git diff -- config/config.go
sed -n '1,120p' config/config.go"
semedit/semantic_rename (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
6—✓ ✓
/bin/zsh -lc "sed -n '1,120p' config/config.go"
7—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "config/config.go",
  "placement": "public_end",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{Host: \"localhost\", ListenPort: 8080}\n}"
}
8—✓ ✓
/bin/zsh -lc "git status --short && \
git diff -- config/config.go config/config_test.go go.mod go.sum && \
sed -n '1,120p' config/config.go"
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 19 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 18 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'config.go' -g '*.go' | head -200"
✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'config.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'Makefile' -g 'go.mod' -g 'go.sum'"
2✓ ✓
/bin/zsh -lc "sed -n '1,240p' config/config.go && \
rg -n 'Config\\s*\\{|\\.Port\\b|Port:' --glob '*.go' --glob '"'!**/*_test.go'"'"
✓ ✓
/bin/zsh -lc "sed -n '1,240p' config/config.go && \
git status --short"
3✓ ✓
/bin/zsh -lc "sed -n '1,220p' config/config_test.go && \
rg -n 'ListenPort|default.*(host|port)|localhost|127\\.0\\.0\\.1|8080|Config' --glob '"'!**/*_test.go'"'"
✓ ✓
/bin/zsh -lc "rg --files -g '"'!go.mod'"' -g '"'!go.sum'"' | sort | head -100 && \
rg -n 'DefaultConfig|ListenPort|\\.Port\\b|Config\\{' . -g '*.go' -g '"'!**/*_test.go'"'"
4✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- config/config.go'
✓ ✓
/bin/zsh -lc "sed -n '1,240p' config/config_test.go && \
rg -n 'Host:|Port:|host|port|127\\.0\\.0\\.1|localhost|8080' . -g '"'!go.mod'"' -g '"'!go.sum'"' -g '"'!**/*_test.go'"'"
5✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os
import stat
import tempfile
import time

path = Path('config/config.go')
old = path.read_text()
expected = '''package config

import (
	\"archive/tar\"
	\"fmt\"
)

type Config struct {
	Host string
	Port int
}

func (c *Config) Addr() string {
	return fmt.Sprintf(\"%s:%d\", c.Host, c.Port)
}
'''
if old "'!= expected:
    raise SystemExit('"'config/config.go changed from the inspected version; refusing to overwrite')
new = '''package config

import \"fmt\"

type Config struct {
	Host       string
	ListenPort int
}

func DefaultConfig() *Config {
	return &Config{
		Host:       \"localhost\",
		ListenPort: 8080,
	}
}

func (c *Config) Addr() string {
	return fmt.Sprintf(\"%s:%d\", c.Host, c.ListenPort)
}
'''
old_stat = path.stat()
fd, temp_name = tempfile.mkstemp(prefix='.config.go.', dir=path.parent)
try:
    os.fchmod(fd, stat.S_IMODE(old_stat.st_mode))
    with os.fdopen(fd, 'w', encoding='utf-8', newline='') as temp:
        temp.write(new)
        temp.flush()
        os.fsync(temp.fileno())
    os.replace(temp_name, path)
    new_mtime = max(time.time(), old_stat.st_mtime + 1)
    os.utime(path, (new_mtime, new_mtime))
    dir_fd = os.open(path.parent, os.O_RDONLY)
    try:
        os.fsync(dir_fd)
    finally:
        os.close(dir_fd)
except BaseException:
    try:
        os.unlink(temp_name)
    except FileNotFoundError:
        pass
    raise
PY
sed -n '1,200p' config/config.go
git diff -- config/config.go"
semedit/semantic_rename (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
6✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
print(repr(Path('config/config.go').read_text()))
PY"
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "config/config.go",
  "placement": "public_start",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{Host: \"localhost\", ListenPort: 8080}\n}"
}
7✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os
import stat
import tempfile
import time

path = Path('config/config.go')
old = path.read_text()
if 'Port int' not in old or 'archive/tar' not in old or 'ListenPort' in old:
    raise SystemExit('config/config.go changed from the inspected version; refusing to overwrite')
new = '''package config

import \"fmt\"

type Config struct {
	Host       string
	ListenPort int
}

func DefaultConfig() *Config {
	return &Config{
		Host:       \"localhost\",
		ListenPort: 8080,
	}
}

func (c *Config) Addr() string {
	return fmt.Sprintf(\"%s:%d\", c.Host, c.ListenPort)
}
'''
old_stat = path.stat()
fd, temp_name = tempfile.mkstemp(prefix='.config.go.', dir=path.parent)
try:
    os.fchmod(fd, stat.S_IMODE(old_stat.st_mode))
    with os.fdopen(fd, 'w', encoding='utf-8', newline='') as temp:
        temp.write(new)
        temp.flush()
        os.fsync(temp.fileno())
    os.replace(temp_name, path)
    new_mtime = max(time.time(), old_stat.st_mtime + 1)
    os.utime(path, (new_mtime, new_mtime))
    dir_fd = os.open(path.parent, os.O_RDONLY)
    try:
        os.fsync(dir_fd)
    finally:
        os.close(dir_fd)
except BaseException:
    try:
        os.unlink(temp_name)
    except FileNotFoundError:
        pass
    raise
PY
sed -n '1,200p' config/config.go
git diff -- config/config.go"
✓ ✓
/bin/zsh -lc "sed -n '1,200p' config/config.go && \
git diff -- config/config.go config/config_test.go go.mod go.sum"
8✓ ✓
/bin/zsh -lc "rg -n 'Port\\b|ListenPort|DefaultConfig|archive/tar' config --glob '*.go'"
—

Test case: task-11-mixed-sink-api-migration

Target: codex/gpt-5.6-luna/high

default prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-5.6-luna/high
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency99.78s134.42s+34.7%164.08s278.39s+69.7%
Process Start → First Event0.13s0.13s—0.12s0.48s—
First Event → First Tool Call7.21s13.57s—9.58s14.12s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles814+75.0%928+211.1%
Initial Load / Discovery Turns110%21-50.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)814+75.0%928+211.1%
Output Tokens44424659+4.9%674410722+59.0%
Reasoning / Thinking Tokens21842687+23.0%36017235+100.9%
Total Input Tokens257879529987+105.5%332439923150+177.7%
Cached Input Tokens220416489472+122.1%261888864256+230.0%
Uncached Input Tokens3746340515+8.1%7055158894-16.5%
Cost0.49630.6677+34.5%0.79401.2653+59.3%
Cached vs Uncached Token Ratio5.88:112.08:1+105.3%3.71:114.67:1+295.3%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '*.go' -g 'Makefile' -g 'README*' | sed -n '1,160p'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' -g 'docs/adr/README.md' -g 'docs/research/README.md' -g 'AGENTS*.md' -g '"'!go.mod'"' -g '"'!go.sum'"' | sort && \
git status --short && \
rg -n -i \"audit|probe|deliver|persist|event kind|eventkind|legacy export|metrics\" --glob '"'!go.mod'"' --glob '"'!go.sum'"' --glob '"'!**/*_test.go'"' ."
2✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- files ---' && \
rg --files -uu | sed -n '1,200p' && \
printf '%s\\n' '--- git status ---' && \
git status --short && \
printf '%s\\n' '--- implementation ---' && \
sed -n '1,240p' audit/event.go && \
sed -n '1,240p' audit/normalize.go && \
sed -n '1,240p' audit/sink.go && \
sed -n '1,260p' service/dispatch.go"
✓ ✓
/bin/zsh -lc "sed -n '1,220p' audit/event.go && \
sed -n '1,220p' audit/sink.go && \
sed -n '1,220p' audit/normalize.go && \
sed -n '1,160p' audit/summary.go && \
sed -n '1,180p' service/dispatch.go && \
sed -n '1,160p' metrics/writer.go && \
sed -n '1,160p' legacy/export.go && \
sed -n '1,200p' docs/audit-sinks.md && \
rg --files | sort"
3✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- docs/audit-sinks.md ---' && \
sed -n '1,260p' docs/audit-sinks.md && \
printf '%s\\n' '--- audit tests ---' && \
sed -n '1,280p' audit/event_test.go && \
sed -n '1,260p' audit/summary.go && \
sed -n '1,280p' audit/summary_test.go && \
printf '%s\\n' '--- other protocols ---' && \
sed -n '1,260p' metrics/writer.go && \
sed -n '1,260p' legacy/export.go && \
printf '%s\\n' '--- module ---' && \
sed -n '1,120p' go.mod"
✓ ✓
/bin/zsh -lc "sed -n '1,240p' audit/event_test.go && \
sed -n '1,240p' audit/summary_test.go && \
sed -n '1,260p' service/dispatch_test.go && \
sed -n '1,220p' metrics/writer_test.go && \
sed -n '1,220p' legacy/export_test.go && \
sed -n '1,120p' cmd/migrate/main.go && \
git status --short -- . && \
git diff -- audit service metrics legacy cmd docs"
4✓ ✓
/bin/zsh -lc "rg -n \"Normalize|normalize|Classify|Record|DispatchAll|Write\\(\" -uu --glob '"'!*.sum'"' --glob '"'!go.mod'"' --glob '"'!AGENTS.override.md'"' . && \
printf '%s\\n' '--- service tests ---' && \
sed -n '1,320p' service/dispatch_test.go && \
printf '%s\\n' '--- command and remaining tests ---' && \
sed -n '1,240p' cmd/migrate/main.go && \
sed -n '1,260p' metrics/writer_test.go && \
sed -n '1,260p' legacy/export_test.go"
semedit/semantic_lookup (✓ ✓)
{
  "file": "service/dispatch.go",
  "language": "go",
  "symbol": "(*Dispatcher).Record"
}
5✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go && \
git diff --check && \
go test ./...'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
6✓ ✓
/bin/zsh -lc "git diff -- audit/normalize.go audit/summary.go service/dispatch.go && \
printf '%s\\n' '--- scoped status ---' && \
git status --short -- audit/normalize.go audit/summary.go service/dispatch.go go.mod go.sum '*_test.go' && \
printf '%s\\n' '--- files ---' && \
sed -n '1,120p' audit/normalize.go && \
sed -n '1,100p' audit/summary.go && \
sed -n '1,120p' service/dispatch.go"
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
7✓ ✓
/bin/zsh -lc 'git rev-parse --show-toplevel && \
git ls-files --stage -- audit/normalize.go audit/summary.go service/dispatch.go && \
git show HEAD:audit/normalize.go 2>/dev/null || \
true && \
git diff --no-ext-diff --stat -- . && \
git check-ignore -v audit/normalize.go audit/summary.go service/dispatch.go || \
true'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "writeEvent := audit.Sink.Write\n\tfor _, event := range events {\n\t\tevent.Kind = audit.NormalizeKind(event.Kind)\n\t\tif err := writeEvent(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
8✓ ✓
/bin/zsh -lc "go vet ./... && \
printf '%s\\n' '--- protected files ---' && \
if [ -e go.mod ]; then stat -f '%N %z bytes' go.mod; fi && \
if [ -e go.sum ]; then stat -f '%N %z bytes' go.sum; else echo 'go.sum absent'; fi && \
printf '%s\\n' '--- modified fixture files ---' && \
find audit service -maxdepth 1 -type f -newer go.mod -print | sort"
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
9—✓ ✓
/bin/zsh -lc 'git diff -- audit service metrics legacy cmd docs && \
git status --short -- audit service metrics legacy cmd docs go.mod go.sum'
10—? ✗
/bin/zsh -lc "sed -n '1,120p' audit/normalize.go && \
sed -n '1,100p' service/dispatch.go && \
sed -n '1,100p' audit/summary.go && \
git rev-parse --show-toplevel && \
git ls-files --error-unmatch audit/normalize.go service/dispatch.go audit/summary.go"
11—? ✗
/bin/zsh -lc 'make check'
12—✓ ✓
/bin/zsh -lc 'go test ./...'
13—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
14—✓ ✓
/bin/zsh -lc 'gofmt -d audit service && \
go vet ./...'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'Makefile' -g 'go.mod' -g 'go.sum' -g '*.go' -g 'AGENTS*' | sed -n '1,160p'"
✓ ✓
/bin/zsh -lc "pwd; rg --files -g 'PROJECT_BRIEF.md' -g 'docs/adr/README.md' -g 'docs/research/README.md' -g 'Makefile' -g 'go.mod' -g '*.go' -g 'AGENTS*.md' | sort | sed -n '1,160p'"
2✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- files ---' && \
rg --files | sort && \
printf '%s\\n' '--- git status ---' && \
git status --short && \
printf '%s\\n' '--- makefile ---' && \
if [ -f Makefile ]; then sed -n '1,220p' Makefile; fi && \
printf '%s\\n' '--- audit files ---' && \
sed -n '1,240p' audit/event.go && \
sed -n '1,260p' audit/normalize.go && \
sed -n '1,260p' audit/fanout.go && \
sed -n '1,240p' audit/sink.go && \
sed -n '1,260p' audit/transaction.go"
✓ ✓
/bin/zsh -lc "sed -n '1,240p' audit/event.go; sed -n '1,240p' audit/normalize.go; sed -n '1,280p' audit/fanout.go; sed -n '1,240p' audit/sink.go; sed -n '1,260p' service/dispatch.go; sed -n '1,220p' audit/transaction.go; sed -n '1,220p' audit/summary.go; sed -n '1,220p' legacy/export.go; sed -n '1,220p' metrics/writer.go"
3✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- audit docs ---' && \
sed -n '1,260p' docs/audit-sinks.md && \
printf '%s\\n' '--- service dispatch ---' && \
sed -n '1,300p' service/dispatch.go && \
printf '%s\\n' '--- service replay ---' && \
sed -n '1,280p' service/replay.go && \
printf '%s\\n' '--- journal ---' && \
sed -n '1,280p' journal/file.go && \
printf '%s\\n' '--- metrics ---' && \
sed -n '1,260p' metrics/metrics.go && \
sed -n '1,300p' metrics/writer.go && \
printf '%s\\n' '--- legacy ---' && \
sed -n '1,300p' legacy/export.go && \
printf '%s\\n' '--- models/db/api ---' && \
sed -n '1,260p' models/models.go && \
sed -n '1,260p' db/db.go && \
sed -n '1,260p' api/router.go"
✓ ✓
/bin/zsh -lc "rg -n \"audit\\.|Event\\{|\\.Write\\(|TransactionalSink|MemorySink|BufferedSink|probe|health|Classify|Persist|persist|journal|db\" --glob '"'!AGENTS.override.md'"' --glob '"'!go.mod'"' --glob '"'!go.sum'"' .; sed -n '1,260p' audit/event_test.go; sed -n '1,300p' audit/summary_test.go; sed -n '1,300p' service/dispatch_test.go; sed -n '1,260p' metrics/writer_test.go; sed -n '1,260p' legacy/export_test.go; sed -n '1,220p' api/router.go; sed -n '1,220p' db/db.go; sed -n '1,220p' journal/file.go; sed -n '1,220p' logging/logging.go"
4✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- audit tests ---' && \
sed -n '1,320p' audit/event_test.go && \
sed -n '1,320p' audit/summary_test.go && \
printf '%s\\n' '--- service tests ---' && \
sed -n '1,320p' service/dispatch_test.go && \
printf '%s\\n' '--- other sinks/usages ---' && \
rg -n \"audit\\.Event|Sink|\\.Write\\(|probe|kind|Normalize|normalize|DispatchAll|Replay|NewDispatcher\" --glob '*.go' --glob '*.md' ."
✓ ✓
/bin/zsh -lc "sed -n '1,240p' docs/audit-sinks.md; sed -n '1,160p' service/replay.go; sed -n '1,180p' audit/event.go; sed -n '1,180p' journal/file.go; sed -n '1,160p' go.mod; git status --short; git diff -- audit go.mod go.sum"
5✓ ✓
/bin/zsh -lc "sed -n '1,220p' audit/summary.go && \
printf '%s\\n' '--- module ---' && \
sed -n '1,160p' go.mod && \
printf '%s\\n' '--- git log ---' && \
git log -5 --oneline --all -- audit service docs/audit-sinks.md"
✓ ✓
/bin/zsh -lc "rg -n \"normalize|control traffic|probe|delivery boundary|persistence|sink\" . --glob '"'!go.mod'"' --glob '"'!go.sum'"' --glob '"'!AGENTS.override.md'"'"
6✓ ✓
/bin/zsh -lc "gofmt -w audit/normalize.go audit/sink.go audit/fanout.go audit/summary.go service/dispatch.go service/replay.go journal/file.go && \
git diff -- audit/normalize.go audit/sink.go audit/fanout.go audit/summary.go service/dispatch.go service/replay.go journal/file.go && \
printf '%s\\n' '--- protected file checksums ---' && \
git diff --stat -- go.mod go.sum && \
printf '%s\\n' '--- tests ---' && \
go test ./..."
✓ ✓
/bin/zsh -lc "git log --oneline -8 -- .; rg -n \"func .*Write|type .*Sink|Sink interface\" --glob '*.go' ."
7✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- changed implementation ---' && \
sed -n '1,180p' audit/normalize.go && \
sed -n '1,120p' audit/sink.go && \
sed -n '1,120p' audit/fanout.go && \
sed -n '1,120p' audit/summary.go && \
sed -n '1,140p' service/dispatch.go && \
sed -n '1,120p' service/replay.go && \
sed -n '1,100p' journal/file.go && \
printf '%s\\n' '--- status ---' && \
git status --short -- audit service journal go.mod go.sum && \
printf '%s\\n' '--- diff check ---' && \
git diff --check -- audit service journal && \
printf '%s\\n' '--- diff stat ---' && \
git diff --stat -- audit service journal"
semedit/semantic_insert_function (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
8✓ ✓
/bin/zsh -lc "git rev-parse --show-toplevel && \
git ls-files --stage -- audit/normalize.go audit/sink.go service/dispatch.go journal/file.go && \
git check-ignore -v audit/normalize.go audit/sink.go service/dispatch.go journal/file.go || \
true && \
git diff --no-ext-diff --no-color --no-index /dev/null audit/normalize.go | sed -n '1,120p'"
✓ ✓
/bin/zsh -lc "sed -n '1,120p' audit/normalize.go"
9✓ ✓
/bin/zsh -lc 'gofmt -w audit/transaction.go && \
go test ./... && \
go vet ./... && \
test ! -e go.sum || \
git diff --exit-code -- go.sum && \
git diff --exit-code -- go.mod && \
printf '"'%s\\n' 'verified: tests, vet, protected module files'"
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event.Kind = NormalizeKind(event.Kind)\nm.Events = append(m.Events, event)\nreturn nil",
  "file": "audit/sink.go",
  "symbol": "(*MemorySink).Write"
}
10—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event.Kind = NormalizeKind(event.Kind)\nif err := f.First.Write(event); err != nil {\n\treturn err\n}\nreturn f.Second.Write(event)",
  "file": "audit/fanout.go",
  "symbol": "FanoutSink.Write"
}
11—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "kind = NormalizeKind(kind)\nswitch kind {\ncase \"health\", \"probe\":\n\treturn \"control\"\ndefault:\n\treturn \"data\"\n}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
12—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\nj.Entries = append(j.Entries, event)\nreturn nil",
  "file": "journal/file.go",
  "symbol": "(*JournalSink).Write"
}
13—semedit/semantic_insert_function (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeEvent(event Event) Event {\n\tevent.Kind = NormalizeKind(event.Kind)\n\treturn event\n}"
}
14—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event = NormalizeEvent(event)\nm.Events = append(m.Events, event)\nreturn nil",
  "file": "audit/sink.go",
  "symbol": "(*MemorySink).Write"
}
15—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event = NormalizeEvent(event)\nif err := f.First.Write(event); err != nil {\n\treturn err\n}\nreturn f.Second.Write(event)",
  "file": "audit/fanout.go",
  "symbol": "FanoutSink.Write"
}
16—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event = audit.NormalizeEvent(event)\nj.Entries = append(j.Entries, event)\nreturn nil",
  "file": "journal/file.go",
  "symbol": "(*JournalSink).Write"
}
17—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event = audit.NormalizeEvent(event)\nreturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
18—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "writeEvent := audit.Sink.Write\nfor _, event := range events {\n\tevent = audit.NormalizeEvent(event)\n\tif err := writeEvent(sink, event); err != nil {\n\t\treturn err\n\t}\n}\nreturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
19—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "writeEvent := audit.Sink.Write\nfor _, event := range events {\n\tevent = audit.NormalizeEvent(event)\n\tif err := writeEvent(sink, event); err != nil {\n\t\treturn err\n\t}\n}\nreturn nil",
  "file": "service/replay.go",
  "symbol": "Replay"
}
20—semedit/semantic_insert_function (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "audit/transaction.go",
  "placement": "file_end",
  "source": "func (b *BufferedSink) Write(event Event) error {\n\tevent = NormalizeEvent(event)\n\tb.queue = append(b.queue, event)\n\treturn nil\n}"
}
21—✓ ✓
/bin/zsh -lc "git diff -- audit journal service; sed -n '1,180p' audit/normalize.go; sed -n '1,180p' audit/transaction.go; sed -n '1,120p' service/dispatch.go; sed -n '1,120p' service/replay.go"
22—✓ ✓
/bin/zsh -lc 'git status --short -- audit journal service; git ls-files --stage -- audit/normalize.go audit/sink.go; git rev-parse --show-toplevel; git check-ignore -v audit/normalize.go || \
true'
23—? ✗
/bin/zsh -lc 'make check'
24—✓ ✓
/bin/zsh -lc 'gofmt -w audit/fanout.go audit/normalize.go audit/sink.go audit/summary.go audit/transaction.go journal/file.go service/dispatch.go service/replay.go && \
go test ./...'
25—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
26—✓ ✓
/bin/zsh -lc 'go vet ./...'
27—✓ ✓
/bin/zsh -lc "gofmt -d audit/fanout.go audit/normalize.go audit/sink.go audit/summary.go audit/transaction.go journal/file.go service/dispatch.go service/replay.go; rg -n \"Normalize|probe|func \\(.*BufferedSink\\).*Write|Classify\" audit journal service metrics legacy"
28—✓ ✓
/bin/zsh -lc "test -f go.sum && \
sha256sum go.mod go.sum || \
sha256sum go.mod; test -f go.sum || \
echo 'go.sum absent in fixture'; find . -maxdepth 2 -name '*_test.go' -print | sort"

prefer_discover_semedit prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-5.6-luna/high
Prompt variantprefer_discover_semedit
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit

Target: codex/gpt-5.6-luna/medium

default prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-5.6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)⚠ MCP (Large)Δ (Large)
Wall-Clock Latency261.42s125.85s-51.9%105.11s120.52sN/A
Process Start → First Event0.14s0.12s—0.13s0.12s—
First Event → First Tool Call8.01s14.86s—7.01s8.23s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns51-80.0%11N/A
Internal Tool Cycles1513-13.3%109N/A
Initial Load / Discovery Turns21-50.0%22N/A
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)1513-13.3%109N/A
Output Tokens276213337-87.9%44194792N/A
Reasoning / Thinking Tokens128991218-90.6%19062175N/A
Total Input Tokens2212802419083-81.1%298316391104N/A
Cached Input Tokens1925888365824-81.0%272384347904N/A
Uncached Input Tokens28691453259-81.4%2593243200N/A
Cost3.61310.5859-83.8%0.45560.5990N/A
Cached vs Uncached Token Ratio6.71:16.87:1+2.3%10.50:18.05:1N/A
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestFAILPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/AFAIL—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 17 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!AGENTS.override.md'"' -g 'PROJECT_BRIEF.md' -g 'README*' -g 'docs/**' -g '*.go' -g 'Makefile' | head -200"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS*.md' -g '"'!AGENTS.override.md'"' -g '*.go' -g 'Makefile' -g 'README*' | sort | head -200"
2✓ ✓
/bin/zsh -lc "sed -n '1,240p' audit/event.go && \
sed -n '1,240p' audit/normalize.go && \
sed -n '1,260p' audit/sink.go && \
sed -n '1,260p' service/dispatch.go && \
sed -n '1,240p' docs/audit-sinks.md && \
sed -n '1,220p' audit/summary.go && \
sed -n '1,220p' legacy/export.go && \
sed -n '1,220p' metrics/writer.go"
✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- files ---' && \
rg --files -g '"'!AGENTS.override.md'"' | sort && \
printf '%s\\n' '--- audit/event.go ---' && \
sed -n '1,240p' audit/event.go && \
printf '%s\\n' '--- audit/normalize.go ---' && \
sed -n '1,240p' audit/normalize.go && \
printf '%s\\n' '--- audit/sink.go ---' && \
sed -n '1,240p' audit/sink.go && \
printf '%s\\n' '--- service/dispatch.go ---' && \
sed -n '1,260p' service/dispatch.go"
3✓ ✓
/bin/zsh -lc "sed -n '1,260p' audit/event_test.go && \
sed -n '1,260p' audit/summary_test.go && \
sed -n '1,280p' service/dispatch_test.go && \
sed -n '1,220p' legacy/export_test.go && \
sed -n '1,220p' metrics/writer_test.go && \
git status --short && \
git diff -- . ':(exclude)AGENTS.override.md'"
✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- docs/audit-sinks.md ---' && \
sed -n '1,260p' docs/audit-sinks.md && \
printf '%s\\n' '--- audit/summary.go ---' && \
sed -n '1,260p' audit/summary.go && \
printf '%s\\n' '--- metrics/writer.go ---' && \
sed -n '1,260p' metrics/writer.go && \
printf '%s\\n' '--- legacy/export.go ---' && \
sed -n '1,260p' legacy/export.go && \
printf '%s\\n' '--- cmd/migrate/main.go ---' && \
sed -n '1,260p' cmd/migrate/main.go && \
printf '%s\\n' '--- go.mod ---' && \
sed -n '1,120p' go.mod"
4✓ ✓
/bin/zsh -lc "rg -n \"probe|Normalize|normalize|Write\\(|Record\\(|DispatchAll|Classify\" . -g '"'!AGENTS.override.md'"' -g '"'!go.mod'"' -g '"'!go.sum'"'"
✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- audit/event_test.go ---' && \
sed -n '1,280p' audit/event_test.go && \
printf '%s\\n' '--- audit/summary_test.go ---' && \
sed -n '1,260p' audit/summary_test.go && \
printf '%s\\n' '--- service/dispatch_test.go ---' && \
sed -n '1,320p' service/dispatch_test.go && \
printf '%s\\n' '--- git status ---' && \
git status --short"
5✓ ✓
/bin/zsh -lc "sed -n '1,120p' go.mod && \
sed -n '1,120p' cmd/migrate/main.go && \
find . -maxdepth 2 -type f -not -path './.git/*' -print"
semedit/semantic_insert_function (✓ ✓)
{
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
6? ✗
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/sink.go audit/summary.go service/dispatch.go && \
make check'
✓ ✓
/bin/zsh -lc "sed -n '1,160p' audit/normalize.go && \
sed -n '1,160p' audit/sink.go && \
sed -n '1,120p' audit/summary.go"
7✓ ✓
/bin/zsh -lc 'go test ./... && \
go vet ./... && \
git diff --check -- audit service && \
git diff -- go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = NormalizeKind(event.Kind)\n\tm.Events = append(m.Events, event)\n\treturn nil",
  "file": "audit/sink.go",
  "symbol": "(*MemorySink).Write"
}
8✓ ✓
/bin/zsh -lc 'git diff -- audit/normalize.go audit/sink.go audit/summary.go service/dispatch.go && \
git status --short -- audit service go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
9✓ ✓
/bin/zsh -lc "git rev-parse --show-toplevel && \
git status --short --untracked-files=all -- audit service go.mod go.sum && \
sed -n '1,160p' audit/normalize.go && \
sed -n '1,120p' service/dispatch.go"
? ✗
/bin/zsh -lc 'git diff -- audit/normalize.go audit/sink.go audit/summary.go go.mod go.sum && \
git diff --check && \
make check'
10✓ ✓
/bin/zsh -lc "sed -n '1,180p' audit/normalize.go && \
sed -n '1,180p' audit/sink.go && \
sed -n '1,160p' audit/summary.go && \
sed -n '1,220p' service/dispatch.go && \
git status --short --untracked-files=all -- audit service go.mod go.sum && \
git diff -- go.mod go.sum"
✓ ✓
/bin/zsh -lc "git status --short -- audit go.mod go.sum && \
printf '%s\\n' '--- normalize ---' && \
sed -n '1,120p' audit/normalize.go && \
printf '%s\\n' '--- sink ---' && \
sed -n '1,180p' audit/sink.go && \
printf '%s\\n' '--- summary ---' && \
sed -n '1,120p' audit/summary.go && \
printf '%s\\n' '--- make targets ---' && \
(make -qp 2>/dev/null | rg '"'^[A-Za-z0-9_.-]+:'"' | head -40 || \
true)"
11✓ ✓
/bin/zsh -lc "git ls-files -- audit/normalize.go audit/sink.go audit/summary.go service/dispatch.go go.mod go.sum; find . -maxdepth 2 -type f -not -path './.git/*' -print | sort"
semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
12✓ ✓
/bin/zsh -lc 'gofmt -d audit/normalize.go audit/sink.go audit/summary.go service/dispatch.go && \
go test ./... && \
go vet ./... && \
git diff --check'
✓ ✓
/bin/zsh -lc 'go test ./... && \
go vet ./... && \
git diff --check && \
cmp -s go.mod go.mod && \
test ! -e go.sum'
13✓ ✓
/bin/zsh -lc "sed -n '1,120p' audit/summary.go && \
sed -n '1,140p' audit/normalize.go && \
go test ./audit ./service ./metrics ./legacy && \
git diff -- go.mod go.sum"
✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- tracked state ---' && \
git ls-files --stage -- audit/normalize.go audit/sink.go audit/summary.go audit/event_test.go service/dispatch_test.go go.mod go.sum && \
printf '%s\\n' '--- ignored state ---' && \
git check-ignore -v audit/normalize.go audit/sink.go audit/summary.go audit/event_test.go service/dispatch_test.go go.mod 2>/dev/null || \
true && \
printf '%s\\n' '--- hashes ---' && \
shasum audit/event_test.go service/dispatch_test.go metrics/writer.go legacy/export.go go.mod"
14✓ ✓
/bin/zsh -lc "sed -n '1,220p' service/dispatch.go && \
sed -n '1,120p' audit/normalize.go && \
go test ./service ./audit && \
git diff -- go.mod go.sum"
—
15✓ ✓
/bin/zsh -lc 'gofmt -w service/dispatch.go && \
go test ./... && \
git diff --check && \
git diff -- go.mod go.sum'
—
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!AGENTS.override.md'"' | sort && \
printf '\\n--- git status ---\\n' && \
git status --short"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'docs/**' -g '*.go' -g 'Makefile' -g 'go.mod' -g 'go.sum' | head -200"
2✓ ✓
/bin/zsh -lc "sed -n '1,240p' docs/audit-sinks.md
sed -n '1,240p' audit/event.go
sed -n '1,240p' audit/normalize.go
sed -n '1,260p' audit/fanout.go
sed -n '1,220p' audit/sink.go
sed -n '1,220p' audit/transaction.go
sed -n '1,220p' audit/summary.go"
✓ ✓
/bin/zsh -lc "sed -n '1,240p' docs/audit-sinks.md && \
sed -n '1,240p' audit/event.go && \
sed -n '1,240p' audit/normalize.go && \
sed -n '1,240p' audit/fanout.go && \
sed -n '1,240p' audit/sink.go && \
sed -n '1,240p' audit/transaction.go && \
sed -n '1,240p' models/models.go"
3✓ ✓
/bin/zsh -lc "rg -n \"audit|Event|Classify|Write\\(\" --glob '*.go' --glob '*.md'"
✓ ✓
/bin/zsh -lc "sed -n '1,260p' service/dispatch.go && \
sed -n '1,260p' service/replay.go && \
sed -n '1,240p' api/router.go && \
sed -n '1,240p' db/db.go && \
sed -n '1,220p' journal/file.go && \
sed -n '1,220p' metrics/writer.go && \
sed -n '1,220p' legacy/export.go && \
rg -n \"audit|Event|Kind|Write\\(\" --glob '*.go' --glob '"'!**/*_test.go'"' ."
4✓ ✓
/bin/zsh -lc "sed -n '1,180p' journal/file.go
sed -n '1,180p' service/dispatch.go
sed -n '1,160p' service/replay.go
sed -n '1,160p' metrics/writer.go
sed -n '1,160p' legacy/export.go
sed -n '1,220p' audit/*_test.go
sed -n '1,180p' service/dispatch_test.go"
✓ ✓
/bin/zsh -lc "sed -n '1,240p' audit/summary.go && \
sed -n '1,260p' audit/event_test.go && \
sed -n '1,260p' audit/summary_test.go && \
sed -n '1,260p' audit/transaction.go && \
sed -n '1,220p' audit/fanout.go && \
git status --short && \
git log -5 --oneline"
5? ✗
/bin/zsh -lc 'make check'
✓ ✓
/bin/zsh -lc "git diff -- audit service journal && \
git diff --name-only -- audit service journal go.mod go.sum '*_test.go'"
6✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go audit/summary.go journal/file.go service/dispatch.go service/replay.go
git diff --check
git status --short -- audit journal service go.mod go.sum

git diff -- audit journal service'
✓ ✓
/bin/zsh -lc "git status --short --untracked-files=all -- audit service journal go.mod go.sum && \
sed -n '1,120p' audit/normalize.go && \
git rev-parse --show-toplevel && \
git ls-files -- audit/normalize.go"
7✓ ✓
/bin/zsh -lc "sed -n '1,120p' audit/normalize.go
sed -n '1,100p' audit/transaction.go
git status --short"
? ✗
/bin/zsh -lc 'ls -la && \
gofmt -w audit/normalize.go audit/summary.go audit/sink.go audit/fanout.go audit/transaction.go journal/file.go service/dispatch.go service/replay.go && \
make check'
8✓ ✓
/bin/zsh -lc 'git ls-files --stage audit/normalize.go audit/sink.go service/dispatch.go journal/file.go
git check-ignore -v audit/normalize.go
git diff --no-ext-diff --stat -- .
git rev-parse --show-toplevel'
✓ ✓
/bin/zsh -lc 'go test ./...'
9✓ ✓
/bin/zsh -lc 'go test ./...'
✓ ✓
/bin/zsh -lc "go vet ./... && \
printf 'protected files:\\n' && \
stat -f '%N %z bytes' go.mod go.sum 2>/dev/null || \
true && \
printf 'test files changed by this work:\\n' && \
find . -type f -name '*_test.go' -newer audit/normalize.go -print && \
printf 'audit implementation:\\n' && \
sed -n '1,120p' audit/normalize.go && \
sed -n '1,100p' audit/summary.go && \
printf 'workspace files:\\n' && \
find . -type f -not -path './.scratch/*' | sort"
10✓ ✓
/bin/zsh -lc 'gofmt -d audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go audit/summary.go journal/file.go service/dispatch.go service/replay.go
go vet ./...
rg -n "Normalize|probe|Classify|metrics|legacy" audit journal service metrics legacy'
—
⚠ Why no semantic edit tool was used
I did not call a semantic_* tool because no applicable semedit MCP operation was exposed in the available tools. Ordinary editing was sufficient for this small, localized change.

prefer_discover_semedit prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-5.6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency119.54s115.93s-3.0%152.82s179.01s+17.1%
Process Start → First Event0.12s0.12s—0.12s0.12s—
First Event → First Tool Call18.52s14.46s—17.55s14.66s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles713+85.7%920+122.2%
Initial Load / Discovery Turns110%110%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)713+85.7%920+122.2%
Output Tokens41623651-12.3%55904390-21.5%
Reasoning / Thinking Tokens16511509-8.6%20071757-12.5%
Total Input Tokens390932552860+41.4%510070893233+75.1%
Cached Input Tokens351232495616+41.1%462080848640+83.7%
Uncached Input Tokens3970057244+44.2%4799044593-7.1%
Cost0.54850.6888+25.6%0.69890.8317+19.0%
Cached vs Uncached Token Ratio8.85:18.66:1-2.1%9.63:119.03:1+97.6%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd; rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS*.md' -g '"'!go.mod'"' -g '"'!go.sum'"' | sort; printf '\\n--- status ---\\n'; git status --short; printf '\\n--- top-level ---\\n'; rg --files -g '"'!go.mod'"' -g '"'!go.sum'"' | sed -n '1,120p'"
✓ ✓
/bin/zsh -lc "pwd; printf '%s\\n' '--- files ---'; rg --files -g '"'!go.sum'"' -g '"'!go.mod'"' | sort; printf '%s\\n' '--- brief ---'; if test -f PROJECT_BRIEF.md; then sed -n '1,240p' PROJECT_BRIEF.md; fi; printf '%s\\n' '--- status ---'; git status --short"
2✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- audit/event.go ---'; sed -n '1,240p' audit/event.go; printf '%s\\n' '--- audit/normalize.go ---'; sed -n '1,240p' audit/normalize.go; printf '%s\\n' '--- audit/sink.go ---'; sed -n '1,240p' audit/sink.go; printf '%s\\n' '--- audit/summary.go ---'; sed -n '1,260p' audit/summary.go; printf '%s\\n' '--- service/dispatch.go ---'; sed -n '1,280p' service/dispatch.go; printf '%s\\n' '--- docs/audit-sinks.md ---'; sed -n '1,260p' docs/audit-sinks.md"
✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- audit/event.go ---'; sed -n '1,240p' audit/event.go; printf '%s\\n' '--- audit/normalize.go ---'; sed -n '1,240p' audit/normalize.go; printf '%s\\n' '--- audit/sink.go ---'; sed -n '1,280p' audit/sink.go; printf '%s\\n' '--- audit/summary.go ---'; sed -n '1,260p' audit/summary.go; printf '%s\\n' '--- service/dispatch.go ---'; sed -n '1,300p' service/dispatch.go; printf '%s\\n' '--- docs ---'; sed -n '1,240p' docs/audit-sinks.md; printf '%s\\n' '--- cmd/migrate ---'; sed -n '1,220p' cmd/migrate/main.go"
3✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- audit tests ---'; sed -n '1,280p' audit/event_test.go; sed -n '1,260p' audit/summary_test.go; printf '%s\\n' '--- service tests ---'; sed -n '1,320p' service/dispatch_test.go; printf '%s\\n' '--- metrics ---'; sed -n '1,260p' metrics/writer.go; sed -n '1,280p' metrics/writer_test.go; printf '%s\\n' '--- legacy ---'; sed -n '1,260p' legacy/export.go; sed -n '1,260p' legacy/export_test.go; printf '%s\\n' '--- cmd ---'; sed -n '1,260p' cmd/migrate/main.go"
✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- tests and protocols ---'; sed -n '1,260p' audit/event_test.go; sed -n '1,260p' audit/summary_test.go; sed -n '1,300p' service/dispatch_test.go; sed -n '1,240p' metrics/writer.go; sed -n '1,220p' legacy/export.go; sed -n '1,260p' legacy/export_test.go; sed -n '1,220p' metrics/writer_test.go"
4✓ ✓
/bin/zsh -lc "rg -n \"probe|health|normalize|Normalize|Record|DispatchAll|Transactional|delivery|sink\" -g '"'!go.mod'"' -g '"'!go.sum'"' .; printf '\\n--- make/check candidates ---\\n'; rg --files -g 'Makefile' -g 'scripts/**' -g '*.sh' -g 'README*' -g '"'!go.mod'"' -g '"'!go.sum'"' | sort"
✓ ✓
/bin/zsh -lc "rg -n \"probe|normalize|Classify|Dispatch|Sink|audit\" . --glob '"'!go.mod'"' --glob '"'!go.sum'"'"
5✓ ✓
/bin/zsh -lc "gofmt -w audit/normalize.go audit/summary.go service/dispatch.go
git diff --check
go test ./...
printf '\\n--- diff ---\\n'
git diff -- audit/normalize.go audit/summary.go service/dispatch.go
printf '\\n--- protected/status check ---\\n'
git status --short -- go.mod go.sum audit service metrics legacy cmd
"
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
6✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- files ---'; sed -n '1,120p' audit/normalize.go; sed -n '1,120p' audit/summary.go; sed -n '1,120p' service/dispatch.go; printf '%s\\n' '--- git identity ---'; git rev-parse --show-toplevel; git ls-files --error-unmatch audit/normalize.go 2>&1 || \
true; printf '%s\\n' '--- ignored? ---'; git check-ignore -v audit/normalize.go audit/summary.go service/dispatch.go || \
true; printf '%s\\n' '--- all status ---'; git status --short --untracked-files=all | sed -n '1,80p'"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = NormalizeKind(event.Kind)\n\tm.Events = append(m.Events, event)\n\treturn nil",
  "file": "audit/sink.go",
  "symbol": "(*MemorySink).Write"
}
7? ✗
/bin/zsh -lc "gofmt -w audit/normalize.go audit/summary.go service/dispatch.go
gofmt -d audit/normalize.go audit/summary.go service/dispatch.go
git diff --check
go test ./...
go vet ./...
printf '\\n--- resulting source ---\\n'
sed -n '1,120p' audit/normalize.go
sed -n '1,100p' audit/summary.go
sed -n '1,140p' service/dispatch.go
printf '\\n--- protected files present and untouched by workflow ---\\n'
stat -f '%N %z bytes' go.mod go.sum"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
8—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
9—semedit/semantic_replace_body (✓ ✓)
{
  "body": "writeEvent := audit.Sink.Write\n\tfor _, event := range events {\n\t\tevent.Kind = audit.NormalizeKind(event.Kind)\n\t\tif err := writeEvent(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
10—✓ ✓
/bin/zsh -lc "git diff -- audit/normalize.go audit/sink.go audit/summary.go service/dispatch.go; printf '%s\\n' '--- protected diff check ---'; git diff -- go.mod go.sum; printf '%s\\n' '--- fixture status ---'; git status --short -- audit service metrics legacy cmd go.mod go.sum"
11—✓ ✓
/bin/zsh -lc "sed -n '1,120p' audit/normalize.go; sed -n '14,35p' audit/sink.go; git rev-parse --show-toplevel; git status --short --untracked-files=all | head -80; git diff --no-ext-diff --no-color -- audit/normalize.go"
12—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
13—✓ ✓
/bin/zsh -lc 'ls -la; go env GOMOD; go test ./...'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd; rg --files -g '"'!AGENTS.override.md'"' | sort; printf '\\n--- PROJECT_BRIEF ---\\n'; if [ -f PROJECT_BRIEF.md ]; then sed -n '1,240p' PROJECT_BRIEF.md; fi; printf '\\n--- git status ---\\n'; git status --short"
✓ ✓
/bin/zsh -lc "pwd && \
printf '%s\\n' '--- files ---' && \
rg --files -g '"'!AGENTS.override.md'"' | sort && \
printf '%s\\n' '--- git status ---' && \
git status --short && \
printf '%s\\n' '--- project brief ---' && \
if test -f PROJECT_BRIEF.md; then sed -n '1,240p' PROJECT_BRIEF.md; fi"
2✓ ✓
/bin/zsh -lc "for f in audit/event.go audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go audit/summary.go api/router.go service/dispatch.go service/replay.go journal/file.go logging/logging.go metrics/writer.go legacy/export.go cmd/migrate/main.go cmd/replay/main.go docs/audit-sinks.md; do printf '\\n--- %s ---\\n' \""'$f"; sed -n '"'1,260p' \""'$f"; done'
✓ ✓
/bin/zsh -lc "printf '%s\\n' '--- audit sources ---' && \
for f in audit/event.go audit/normalize.go audit/fanout.go audit/sink.go audit/summary.go audit/transaction.go docs/audit-sinks.md service/dispatch.go api/router.go; do echo \"### "'$f"; sed -n '"'1,260p' \""'$f"; done'
3? ✗
/bin/zsh -lc 'for f in audit/*_test.go service/*_test.go journal/*_test.go metrics/*_test.go legacy/*_test.go; do [ -f "$f" ] || \
continue; printf '"'\\n--- %s ---\\n' \""'$f"; sed -n '"'1,300p' \""'$f"; done; printf '"'\\n--- references ---\\n'; rg -n \"probe|health|Kind|Sink|Write\\(|Normalize|Classify|Fanout|Buffered|Journal\" --glob '*.go' --glob '*.md'"
✓ ✓
/bin/zsh -lc "rg -n \"Event|Kind|health|probe|metrics|legacy|Write\\(\" --glob '"'!go.mod'"' --glob '"'!go.sum'"' ."
4? ✗
/bin/zsh -lc 'for f in audit/*_test.go service/*_test.go journal/*_test.go metrics/*_test.go legacy/*_test.go; do if [ -f "$f" ]; then printf '"'\\n--- %s ---\\n' \""'$f"; sed -n '"'1,300p' \""'$f"; fi; done; printf '"'\\n--- references ---\\n'; rg -n \"probe|health|Kind|Sink|Write\\(|Normalize|Classify|Fanout|Buffered|Journal\" --glob '*.go' --glob '*.md'"
? ✗
/bin/zsh -lc 'for f in service/replay.go journal/file.go audit/*_test.go metrics/writer.go legacy/export.go cmd/migrate/main.go cmd/replay/main.go; do echo "### $f"; sed -n '"'1,220p' \""'$f"; done && \
printf '"'%s\\n' '--- make/check ---' && \
sed -n '1,220p' Makefile"
5✓ ✓
/bin/zsh -lc 'for f in $(rg --files -g '"'*_test.go' | sort); do printf '\\n--- %s ---\\n' \""'$f"; sed -n '"'1,300p' \""'$f"; done; printf '"'\\n--- references ---\\n'; rg -n \"probe|health|Kind|Sink|Write\\(|Normalize|Classify|Fanout|Buffered|Journal\" --glob '*.go' --glob '*.md'"
semedit/semantic_insert_function (✓ ✗): validate function snippet: multiple declarations found in snippet: expected single function declaration, found 2 declarations
{
  "access_modifier": "public",
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}\n\nfunc NormalizeEvent(event Event) Event {\n\tevent.Kind = NormalizeKind(event.Kind)\n\treturn event\n}"
}
6? ✗
/bin/zsh -lc "gofmt -w audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go audit/summary.go service/dispatch.go service/replay.go journal/file.go; git diff -- audit service journal; printf '\\n--- protected file status ---\\n'; git status --short -- go.mod go.sum AGENTS.override.md; printf '\\n--- make check ---\\n'; make check"
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
7✓ ✓
/bin/zsh -lc "git rev-parse --show-toplevel; git ls-files -- audit/normalize.go audit/sink.go service/dispatch.go journal/file.go; sed -n '1,120p' audit/normalize.go; sed -n '1,100p' audit/transaction.go; git status --short --untracked-files=all | head -80"
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeEvent(event Event) Event {\n\tevent.Kind = NormalizeKind(event.Kind)\n\treturn event\n}"
}
8✓ ✓
/bin/zsh -lc "go test ./...; go vet ./...; if [ -e Makefile ]; then make check; else echo 'make check unavailable: no Makefile in fixture'; fi; printf '\\n--- protected file hashes ---\\n'; shasum go.mod go.sum; printf '\\n--- changed source files ---\\n'; for f in audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go audit/summary.go service/dispatch.go service/replay.go journal/file.go; do echo \""'$f"; done'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "kind = NormalizeKind(kind)\n\tswitch kind {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
9✓ ✓
/bin/zsh -lc "gofmt -d audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go audit/summary.go service/dispatch.go service/replay.go journal/file.go; printf '\\n--- audit behavior ---\\n'; rg -n \"Normalize|probe|health\" audit service journal; printf '\\n--- protected protocols unchanged ---\\n'; sed -n '1,80p' metrics/writer.go; sed -n '1,80p' legacy/export.go; printf '\\n--- test files untouched by mtime-independent content check ---\\n'; rg --files -g '*_test.go' | sort"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event = NormalizeEvent(event)\n\tm.Events = append(m.Events, event)\n\treturn nil",
  "file": "audit/sink.go",
  "symbol": "(*MemorySink).Write"
}
10—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event = audit.NormalizeEvent(event)\n\tj.Entries = append(j.Entries, event)\n\treturn nil",
  "file": "journal/file.go",
  "symbol": "(*JournalSink).Write"
}
11—semedit/semantic_replace_body (✓ ✓)
{
  "body": "return d.sink.Write(audit.NormalizeEvent(event))",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
12—semedit/semantic_replace_body (✓ ✓)
{
  "body": "writeEvent := audit.Sink.Write\n\tfor _, event := range events {\n\t\tif err := writeEvent(sink, audit.NormalizeEvent(event)); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
13—semedit/semantic_replace_body (✓ ✓)
{
  "body": "writeEvent := audit.Sink.Write\n\tfor _, event := range events {\n\t\tif err := writeEvent(sink, audit.NormalizeEvent(event)); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/replay.go",
  "symbol": "Replay"
}
14—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event = NormalizeEvent(event)\n\tif err := f.First.Write(event); err != nil {\n\t\treturn err\n\t}\n\treturn f.Second.Write(event)",
  "file": "audit/fanout.go",
  "symbol": "(FanoutSink).Write"
}
15—semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range b.queue {\n\t\tif err := b.Sink.Write(NormalizeEvent(event)); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "audit/transaction.go",
  "symbol": "(*BufferedSink).Commit"
}
16—✓ ✓
/bin/zsh -lc "git diff -- audit journal service docs/SUBOPTIMAL_TOOLS.md && \
printf '%s\\n' '--- protected/test status ---' && \
git status --short -- go.mod go.sum '*_test.go' && \
printf '%s\\n' '--- diff check ---' && \
git diff --check"
17—✓ ✓
/bin/zsh -lc "sed -n '1,120p' audit/normalize.go; sed -n '1,100p' audit/summary.go; git status --short -- .; git diff --stat -- .; git rev-parse --show-toplevel"
18—✓ ✓
/bin/zsh -lc 'go test ./...'
19—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
20—? ✗
/bin/zsh -lc "printf '%s\\n' '--- final audit implementation ---' && \
for f in audit/normalize.go audit/summary.go audit/sink.go audit/fanout.go audit/transaction.go journal/file.go service/dispatch.go service/replay.go; do echo \"### "'$f"; sed -n '"'1,180p' \""'$f"; done && \
printf '"'%s\\n' '--- protected files present ---' && \
shasum go.mod go.sum && \
printf '%s\\n' '--- test files untouched by status ---' && \
git status --short -- '*_test.go' 'go.mod' 'go.sum'"

Target: codex/gpt-6-luna/high

default prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/high
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency138.78s103.09s-25.7%219.17s154.00s-29.7%
Process Start → First Event0.12s0.12s—0.13s0.12s—
First Event → First Tool Call12.96s7.72s—6.18s7.84s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%51-80.0%
Internal Tool Cycles714+100.0%1716-5.9%
Initial Load / Discovery Turns12+100.0%41-75.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)714+100.0%1716-5.9%
Output Tokens34662981-14.0%330123916-88.1%
Reasoning / Thinking Tokens19701686-14.4%180992570-85.8%
Total Input Tokens197488312129+58.0%1590395323110-79.7%
Cached Input Tokens179968283648+57.6%1466112293888-80.0%
Uncached Input Tokens1752028481+62.6%12428329222-76.5%
Cost0.15670.2005+27.9%1.31610.2276-82.7%
Cached vs Uncached Token Ratio10.27:19.96:1-3.0%11.80:110.06:1-14.7%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—FAILPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g '"'!*test*'"' -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'Makefile' -g '*.go' -g '*.md' | head -150"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g '*.go' -g 'go.mod' -g 'Makefile' -g 'docs/**' | head -160"
2✓ ✓
/bin/zsh -lc "git status --short && \
sed -n '1,240p' audit/event.go && \
sed -n '1,240p' audit/normalize.go && \
sed -n '1,240p' audit/sink.go && \
sed -n '1,260p' service/dispatch.go && \
sed -n '1,200p' docs/audit-sinks.md"
✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
cat docs/audit-sinks.md'
3✓ ✓
/bin/zsh -lc "sed -n '1,220p' audit/event_test.go && \
sed -n '1,240p' service/dispatch_test.go && \
sed -n '1,180p' audit/summary.go && \
sed -n '1,180p' metrics/writer.go && \
sed -n '1,200p' legacy/export.go && \
cat go.mod && \
rg -n \"probe|Kind|DispatchAll|Record\\(\" --glob '"'!go.mod'"' --glob '"'!go.sum'"' ."
✓ ✓
/bin/zsh -lc "sed -n '1,240p' audit/event.go && \
sed -n '1,240p' audit/normalize.go && \
sed -n '1,260p' audit/sink.go && \
sed -n '1,280p' service/dispatch.go"
4✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./...'
✓ ✓
/bin/zsh -lc "git status --short && \
rg --files -g 'Makefile' -g 'README*' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g '.codex/**'"
5✓ ✓
/bin/zsh -lc "git diff --check -- audit/normalize.go audit/summary.go service/dispatch.go && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go && \
sed -n '1,200p' service/dispatch.go && \
sed -n '1,120p' audit/normalize.go && \
sed -n '1,80p' audit/summary.go"
✓ ✓
/bin/zsh -lc "sed -n '1,240p' service/dispatch_test.go && \
sed -n '1,220p' audit/event_test.go && \
sed -n '1,220p' audit/summary.go && \
sed -n '1,220p' audit/summary_test.go && \
sed -n '1,240p' metrics/writer.go && \
sed -n '1,240p' legacy/export.go"
6✓ ✓
/bin/zsh -lc 'mkdir -p .scratch && \
verify_file=$(mktemp .scratch/verify-audit-XXXXXX.go) && \
trap '"'rm -f \""'$verify_file"'"' EXIT && \
cat > \""'$verify_file" <<'"'EOF'
package main

import (
	\"fmt\"
	\"example.com/auditapp/audit\"
	\"example.com/auditapp/service\"
)

func main() {
	sink := audit.NewMemorySink()
	dispatcher := service.NewDispatcher(sink)
	if err := dispatcher.Record(audit.Event{ID: \"one\", Kind: \"  AuDiT\\t\"}); err "'!= nil || \
sink.Events[0].Kind != "audit" {
		panic(fmt.Sprintf("record normalization failed: %#v, %v", sink.Events, err))
	}
	if err := service.DispatchAll(sink, []audit.Event{{ID: "two", Kind: " PROBE "}}); err != nil || \
sink.Events[1].Kind != "probe" {
		panic(fmt.Sprintf("batch normalization failed: %#v, %v", sink.Events, err))
	}
	if audit.Classify(" PrObE'"\\n\") "'!= "control" {
		panic("probe classification failed")
	}
}
EOF
go run "$verify_file" && \
git diff --check -- audit/normalize.go audit/summary.go service/dispatch.go'
✓ ✓
/bin/zsh -lc "cat go.mod && \
rg -n \"Probe|probe|Kind|DispatchAll|Record\\(|Write\\(\" --glob '*.go' ."
7✓ ✓
/bin/zsh -lc 'git status --short -- audit/normalize.go audit/summary.go service/dispatch.go audit/event_test.go service/dispatch_test.go go.mod go.sum && \
find .scratch -maxdepth 1 -type f -print'
semedit/semantic_insert_function (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
8—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\nreturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
9—semedit/semantic_replace_body (✓ ✓)
{
  "body": "writeEvent := audit.Sink.Write\nfor _, event := range events {\n\tevent.Kind = audit.NormalizeKind(event.Kind)\n\tif err := writeEvent(sink, event); err != nil {\n\t\treturn err\n\t}\n}\nreturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
10—semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\ncase \"health\", \"probe\":\n\treturn \"control\"\ndefault:\n\treturn \"data\"\n}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
11—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
12—✓ ✓
/bin/zsh -lc 'git diff -- audit/normalize.go service/dispatch.go audit/summary.go && \
git status --short -- audit/normalize.go service/dispatch.go audit/summary.go go.mod go.sum'
13—✓ ✓
/bin/zsh -lc 'go test ./...'
14—✓ ✓
/bin/zsh -lc "sed -n '1,160p' audit/normalize.go && \
sed -n '1,140p' service/dispatch.go && \
sed -n '1,100p' audit/summary.go && \
git status --short"
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 21 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS*' -g '*.go' | head -160"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'Makefile' -g '*.go' -g 'docs/**' | head -200"
2✓ ✓
/bin/zsh -lc 'cat go.mod'
✓ ✓
/bin/zsh -lc "sed -n '1,260p' audit/sink.go && \
sed -n '1,260p' audit/transaction.go && \
sed -n '1,260p' service/dispatch.go && \
sed -n '1,220p' metrics/writer.go && \
sed -n '1,220p' legacy/export.go"
3✓ ✓
/bin/zsh -lc "sed -n '1,240p' audit/normalize.go && \
sed -n '1,240p' audit/event.go && \
sed -n '1,260p' audit/fanout.go && \
sed -n '1,240p' audit/sink.go"
✓ ✓
/bin/zsh -lc "cat docs/audit-sinks.md && \
sed -n '1,240p' audit/event.go && \
sed -n '1,240p' audit/normalize.go && \
sed -n '1,260p' audit/fanout.go"
4✓ ✓
/bin/zsh -lc "sed -n '1,260p' service/dispatch.go && \
sed -n '1,240p' service/replay.go && \
rg -n \"Kind|probe|Probe|audit|Deliver|Persist\" --glob '*.go' ."
✓ ✓
/bin/zsh -lc "cat audit/event_test.go audit/summary.go audit/summary_test.go audit/transaction.go audit/fanout.go; cat service/dispatch_test.go; rg -n \"probe|Normalize|normaliz|BufferedSink|FanoutSink|Write\\(\""
5✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'Makefile' -g 'README*' -g '*.md'"
✓ ✓
/bin/zsh -lc 'cat journal/file.go && \
cat api/router.go && \
cat cmd/replay/main.go && \
cat config/config.go && \
cat models/models.go && \
cat db/db.go'
6✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/transaction.go && \
cat db/db.go && \
cat journal/file.go'
semedit/semantic_insert_function (✓ ✓)
{
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.Join(strings.Fields(kind), \" \"))\n}"
}
7✓ ✓
/bin/zsh -lc "cat metrics/writer.go metrics/metrics.go legacy/export.go audit/summary.go && \
rg -n 'Write\\(|TransactionalSink|FanoutSink|JournalSink|MemorySink' --glob '*.go' ."
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = NormalizeKind(event.Kind)\n\tm.Events = append(m.Events, event)\n\treturn nil",
  "file": "audit/sink.go",
  "symbol": "(*MemorySink).Write"
}
8? ✗
/bin/zsh -lc "ls -la && \
rg -n '"'^check:|make check|''^fmt:|''^lint:'"' Makefile"
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\tj.Entries = append(j.Entries, event)\n\treturn nil",
  "file": "journal/file.go",
  "symbol": "(*JournalSink).Write"
}
9✓ ✓
/bin/zsh -lc "git status --short && \
rg -n \"func .*Write|\\.Write\\(event\\)\" --glob '*.go' ."
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
10✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile

root = Path('.')
changes = {
'audit/normalize.go': '''// Package audit owns event normalization and persistence eligibility at delivery.
package audit

import \"strings\"

func NormalizeKind(kind string) string {
	return strings.ToLower(strings.TrimSpace(kind))
}

func PrepareEvent(event Event) (Event, bool) {
	event.Kind = NormalizeKind(event.Kind)
	return event, event.Kind "'!= "probe"
}

func Deliver(sink Sink, event Event) error {
	event, persist := PrepareEvent(event)
	if !persist {
		return nil
	}
	return sink.Write(event)
}
'"''',
'audit/sink.go': '''// Package audit exposes the established event sink contract and a memory implementation.
package audit

type Sink interface {
	Write(Event) error
}

type TransactionalSink interface {
	Sink
	Begin() error
	Commit() error
}

type MemorySink struct {
	Events []Event
}

func NewMemorySink() *MemorySink {
	return &MemorySink{}
}

func (m *MemorySink) Write(event Event) error {
	event, persist := PrepareEvent(event)
	if "'!persist {
		return nil
	}
	m.Events = append(m.Events, event)
	return nil
}
'"''',
'audit/fanout.go': '''// FanoutSink delivers an event to each configured audit destination.
package audit

type FanoutSink struct {
	First  Sink
	Second Sink
}

func (f FanoutSink) Write(event Event) error {
	event, persist := PrepareEvent(event)
	if "'!persist {
		return nil
	}
	if err := Deliver(f.First, event); err != nil {
		return err
	}
	return Deliver(f.Second, event)
}
'"''',
'audit/transaction.go': '''// BufferedSink exposes transactional delivery without owning the persistence protocol.
package audit

type BufferedSink struct {
	Sink
	queue []Event
}

func (b *BufferedSink) Begin() error {
	b.queue = b.queue[:0]
	return nil
}

func (b *BufferedSink) Write(event Event) error {
	event, persist := PrepareEvent(event)
	if "'!persist {
		return nil
	}
	b.queue = append(b.queue, event)
	return nil
}

func (b *BufferedSink) Commit() error {
	for _, event := range b.queue {
		if err := Deliver(b.Sink, event); err != nil {
			return err
		}
	}
	return nil
}
'"''',
'service/dispatch.go': '''// Package service coordinates application work with the configured audit sink.
package service

import \"example.com/auditapp/audit\"

type Dispatcher struct {
	sink audit.Sink
}

func NewDispatcher(sink audit.Sink) *Dispatcher {
	return &Dispatcher{sink: sink}
}

func (d *Dispatcher) Record(event audit.Event) error {
	return audit.Deliver(d.sink, event)
}

func DispatchAll(sink audit.Sink, events []audit.Event) error {
	for _, event := range events {
		if err := audit.Deliver(sink, event); err "'!= nil {
			return err
		}
	}
	return nil
}
'"''',
'service/replay.go': '''// Package service contains replay helpers for previously captured audit events.
package service

import \"example.com/auditapp/audit\"

func Replay(sink audit.Sink, events []audit.Event) error {
	for _, event := range events {
		if err := audit.Deliver(sink, event); err "'!= nil {
			return err
		}
	}
	return nil
}
'"''',
'journal/file.go': '''// Package journal contains a separate audit implementation used by batch jobs.
package journal

import \"example.com/auditapp/audit\"

type JournalSink struct {
	Entries []audit.Event
}

func (j *JournalSink) Write(event audit.Event) error {
	event, persist := audit.PrepareEvent(event)
	if "'!persist {
		return nil
	}
	j.Entries = append(j.Entries, event)
	return nil
}

var _ audit.Sink = (*JournalSink)(nil)
'"''',
}
for name, content in changes.items():
    path = root / name
    st = path.stat()
    fd, tmp = tempfile.mkstemp(prefix=f'.{path.name}.', dir=path.parent)
    try:
        with os.fdopen(fd, 'w', encoding='utf-8', newline='') as f:
            f.write(content)
            f.flush()
            os.fsync(f.fileno())
        os.replace(tmp, path)
        os.utime(path, ns=(st.st_atime_ns, max(st.st_mtime_ns + 1, path.stat().st_mtime_ns + 1)))
    finally:
        if os.path.exists(tmp):
            os.unlink(tmp)
PY
gofmt -w audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go journal/file.go
git diff -- audit service journal"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "writeEvent := audit.Sink.Write\n\tfor _, event := range events {\n\t\tevent.Kind = audit.NormalizeKind(event.Kind)\n\t\tif err := writeEvent(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
11✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
p = Path('audit/summary.go')
st = p.stat()
content = '''// Package audit groups delivery kinds for operational reporting.
package audit

func Classify(kind string) string {
	switch NormalizeKind(kind) {
	case \"health\", \"probe\":
		return \"control\"
	default:
		return \"data\"
	}
}
'''
fd, tmp = tempfile.mkstemp(prefix=f'.{p.name}.', dir=p.parent)
try:
    with os.fdopen(fd, 'w', encoding='utf-8', newline='') as f:
        f.write(content)
        f.flush()
        os.fsync(f.fileno())
    os.replace(tmp, p)
    os.utime(p, ns=(st.st_atime_ns, max(st.st_mtime_ns + 1, p.stat().st_mtime_ns + 1)))
finally:
    if os.path.exists(tmp):
        os.unlink(tmp)
PY
gofmt -w audit/summary.go
go test ./..."
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = NormalizeKind(event.Kind)\n\tif err := f.First.Write(event); err != nil {\n\t\treturn err\n\t}\n\treturn f.Second.Write(event)",
  "file": "audit/fanout.go",
  "symbol": "(FanoutSink).Write"
}
12✓ ✓
/bin/zsh -lc "git status --short -- go.mod go.sum ':(glob)**/*_test.go' && \
git diff --check -- audit service journal && \
sed -n '1,160p' audit/normalize.go && \
sed -n '1,120p' audit/summary.go && \
sed -n '1,120p' audit/transaction.go"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
13✓ ✓
/bin/zsh -lc "sed -n '1,200p' audit/normalize.go && \
sed -n '1,140p' audit/sink.go && \
sed -n '1,140p' audit/fanout.go && \
sed -n '1,160p' audit/transaction.go && \
sed -n '1,180p' service/dispatch.go && \
sed -n '1,120p' service/replay.go && \
sed -n '1,100p' journal/file.go && \
sed -n '1,100p' audit/summary.go && \
cat go.mod && \
test "'! -e go.sum || \
cat go.sum && \
git status --short -- go.mod go.sum '"':(glob)**/*_test.go'"
✓ ✓
/bin/zsh -lc "cat audit/normalize.go && \
git status --short && \
git diff -- audit/normalize.go audit/sink.go audit/summary.go audit/fanout.go service/dispatch.go journal/file.go && \
rg --files -g 'Makefile' -g 'go.work' -g 'go.mod' -g 'go.sum'"
14✓ ✓
/bin/zsh -lc "sed -n '1,120p' audit/summary.go && \
sed -n '1,120p' audit/normalize.go && \
sed -n '1,120p' audit/summary_test.go && \
cat go.mod && \
if test -e go.sum; then cat go.sum; else printf 'go.sum absent\\n'; fi && \
git status --short -- go.mod go.sum ':(glob)**/*_test.go'"
✓ ✓
/bin/zsh -lc 'go test ./...'
15✓ ✓
/bin/zsh -lc "sed -n '1,120p' service/dispatch.go && \
sed -n '1,100p' audit/normalize.go && \
cat go.mod && \
if test -e go.sum; then cat go.sum; else printf 'go.sum absent\\n'; fi && \
git status --short -- go.mod go.sum ':(glob)**/*_test.go'"
✓ ✓
/bin/zsh -lc 'git status --short -- . && \
git diff -- . && \
git diff --exit-code -- go.mod go.sum'
16✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
p = Path('service/dispatch.go')
s = p.read_text()
old = '''func (d *Dispatcher) Record(event audit.Event) error {
	return audit.Deliver(d.sink, event)
}'''
new = '''func (d *Dispatcher) Record(event audit.Event) error {
	event.Kind = audit.NormalizeKind(event.Kind)
	if event.Kind == \"probe\" {
		return nil
	}
	return d.sink.Write(event)
}'''
if old not in s:
    raise SystemExit('expected Dispatcher.Record implementation not found')
st = p.stat()
fd, tmp = tempfile.mkstemp(prefix=f'.{p.name}.', dir=p.parent)
try:
    with os.fdopen(fd, 'w', encoding='utf-8', newline='') as f:
        f.write(s.replace(old, new, 1))
        f.flush()
        os.fsync(f.fileno())
    os.replace(tmp, p)
    os.utime(p, ns=(st.st_atime_ns, max(st.st_mtime_ns + 1, p.stat().st_mtime_ns + 1)))
finally:
    if os.path.exists(tmp):
        os.unlink(tmp)
PY
gofmt -w service/dispatch.go
go test ./..."
semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
17✓ ✓
/bin/zsh -lc "sed -n '1,55p' service/dispatch.go && \
git status --short -- go.mod go.sum ':(glob)**/*_test.go' && \
git diff --check -- service/dispatch.go"
—

prefer_discover_semedit prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/high
Prompt variantprefer_discover_semedit
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency124.69s286.13s+129.5%99.32s——
Process Start → First Event0.12s0.13s—0.12s——
First Event → First Tool Call8.92s12.84s—15.42s——
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns15+400.0%1——
Internal Tool Cycles731+342.9%5——
Initial Load / Discovery Turns21-50.0%1——
MCP Discovery / Schema Turns00—0——
Tool Invocations (per run)731+342.9%5——
Output Tokens418728556+582.0%4387——
Reasoning / Thinking Tokens275819525+607.9%2673——
Total Input Tokens2125883981357+1772.8%282373——
Cached Input Tokens1845763729152+1920.4%251648——
Uncached Input Tokens28012252205+800.3%30725——
Cost0.20302.1638+966.0%0.2280——
Cached vs Uncached Token Ratio6.59:114.79:1+124.4%8.19:1——
Oracle L1: Mutation PolicyPASSPASS—PASS——
Oracle L2: AST InvariantsPASSPASS—PASS——
Oracle L3: Clean BuildPASSPASS—PASS——
Oracle L4: Verification TestPASSFAIL—PASS——
Semantic Tool Invocation Verified (per run)N/APASS—N/A——
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'go.sum' -g '*.go' | head -200"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'Makefile' -g '*.go' -g '*.md' | head -180 && \
git status --short"
2✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'Makefile' -g 'AGENTS*' -g '*.go' -g 'go.mod' -g 'go.sum' | sort"
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/event.go && \
cat audit/normalize.go && \
cat audit/sink.go && \
cat service/dispatch.go && \
cat cmd/migrate/main.go && \
cat metrics/writer.go && \
cat legacy/export.go'
3✓ ✓
/bin/zsh -lc "git status --short && \
sed -n '1,240p' AGENTS.override.md && \
sed -n '1,240p' audit/event.go && \
sed -n '1,240p' audit/normalize.go && \
sed -n '1,240p' audit/sink.go && \
sed -n '1,260p' service/dispatch.go"
? ✗
/bin/zsh -lc "cat audit/event_test.go && \
cat service/dispatch_test.go && \
cat audit/summary.go && \
cat audit/summary_test.go && \
cat go.mod && \
rg --files -g 'Makefile' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md'"
4✓ ✓
/bin/zsh -lc 'for f in go.mod audit/*.go service/*.go metrics/*.go legacy/*.go cmd/migrate/*.go; do echo "--- $f"; sed -n '"'1,280p' \""'$f"; done; rg --files -g '"'Makefile' -g 'README*' -g 'PROJECT_BRIEF.md' -g '*.md' | sort"
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
5✓ ✓
/bin/zsh -lc "cat docs/audit-sinks.md; rg -n \"Normalize|probe|TransactionalSink|DispatchAll|Sink|delivery|whitespace|kind\" . -g '"'!go.sum'"' -g '"'!go.mod'"'"
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "private",
  "auto_organize_imports": true,
  "file": "service/dispatch.go",
  "source": "func dispatchEvent(sink audit.Sink, event audit.Event) error {\n\tevent.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn sink.Write(event)\n}"
}
6✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os
import tempfile

root = Path.cwd()
changes = {
    \"audit/normalize.go\": '''// Package audit owns normalization rules for the event delivery boundary.
package audit

import \"strings\"

func NormalizeKind(kind string) string {
\\treturn strings.ToLower(strings.TrimSpace(kind))
}

func NormalizeEvent(event Event) Event {
\\tevent.Kind = NormalizeKind(event.Kind)
\\treturn event
}
''',
    \"audit/summary.go\": '''// Package audit groups delivery kinds for operational reporting.
package audit

func Classify(kind string) string {
\\tswitch NormalizeKind(kind) {
\\tcase \"health\", \"probe\":
\\t\\treturn \"control\"
\\tdefault:
\\t\\treturn \"data\"
\\t}
}
''',
    \"service/dispatch.go\": '''// Package service coordinates application work with the configured audit sink.
package service

import \"example.com/auditapp/audit\"

type Dispatcher struct {
\\tsink audit.Sink
}

func NewDispatcher(sink audit.Sink) *Dispatcher {
\\treturn &Dispatcher{sink: sink}
}

func (d *Dispatcher) Record(event audit.Event) error {
\\treturn deliver(d.sink, event)
}

func DispatchAll(sink audit.Sink, events []audit.Event) error {
\\tfor _, event := range events {
\\t\\tif err := deliver(sink, event); err "'!= nil {
'"\\t\\t\\treturn err
\\t\\t}
\\t}
\\treturn nil
}

func deliver(sink audit.Sink, event audit.Event) error {
\\treturn sink.Write(audit.NormalizeEvent(event))
}
''',
}

for name, contents in changes.items():
    target = root / name
    fd, temp_name = tempfile.mkstemp(prefix=f\".{target.name}.\", dir=target.parent)
    try:
        with os.fdopen(fd, \"w\", encoding=\"utf-8\", newline=\"\\n\") as stream:
            stream.write(contents)
            stream.flush()
            os.fsync(stream.fileno())
        os.replace(temp_name, target)
        dir_fd = os.open(target.parent, os.O_RDONLY)
        try:
            os.fsync(dir_fd)
        finally:
            os.close(dir_fd)
    finally:
        if os.path.exists(temp_name):
            os.unlink(temp_name)
PY
gofmt -w audit/normalize.go audit/summary.go service/dispatch.go
git diff -- audit/normalize.go audit/summary.go service/dispatch.go; git status --short"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "return dispatchEvent(d.sink, event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
7✓ ✓
/bin/zsh -lc 'go test ./... && \
test -z "$(gofmt -d audit/normalize.go audit/summary.go service/dispatch.go)" && \
printf '"'gofmt: clean\\n' && \
cat go.mod && \
sed -n '1,200p' audit/normalize.go && \
sed -n '1,120p' audit/summary.go && \
sed -n '1,180p' service/dispatch.go"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tif err := dispatchEvent(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
8—? ✗
/bin/zsh -lc 'make check'
9—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
10—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat service/dispatch.go'
11—✓ ✓
/bin/zsh -lc 'git diff -- audit/normalize.go service/dispatch.go go.mod go.sum && \
git status --short -- audit/normalize.go service/dispatch.go go.mod go.sum'
12—✓ ✓
/bin/zsh -lc 'go test ./...'
13—✓ ✓
/bin/zsh -lc "rg --files -g 'go.mod' -g 'go.sum' -g '*.go' -g '*.md' && \
cat audit/event.go && \
cat audit/normalize.go && \
cat audit/sink.go && \
cat service/dispatch.go && \
cat docs/audit-sinks.md && \
cat go.mod && \
if [ -f go.sum ]; then cat go.sum; else echo 'go.sum absent'; fi"
14—✓ ✓
/bin/zsh -lc 'git status --short -- go.mod go.sum && \
cat go.mod && \
test ! -f go.sum'
15—✓ ✓
/bin/zsh -lc 'go test ./...'
16—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
17—✓ ✓
/bin/zsh -lc 'cat audit/summary.go && \
cat audit/normalize.go && \
cat audit/sink.go && \
cat service/dispatch.go && \
cat go.mod && \
test ! -f go.sum'
18—semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
19—✓ ✓
/bin/zsh -lc 'cat go.mod && \
test ! -f go.sum && \
git status --short -- go.mod go.sum'
20—✓ ✓
/bin/zsh -lc 'go test ./...'
21—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
22—✓ ✓
/bin/zsh -lc 'cat service/dispatch.go && \
cat audit/normalize.go && \
cat go.mod && \
test ! -f go.sum && \
git status --short -- go.mod go.sum'
23—✓ ✓
/bin/zsh -lc 'go test ./...'
24—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
25—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat audit/summary.go && \
cat service/dispatch.go && \
cat go.mod && \
test ! -f go.sum'
26—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\treturn dispatchEvent(d.sink, event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
27—semedit/semantic_replace_body (✓ ✓)
{
  "body": "dispatcher := NewDispatcher(sink)\n\tfor _, event := range events {\n\t\tif err := dispatcher.Record(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
28—semedit/semantic_replace_body (✓ ✓)
{
  "body": "if event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "dispatchEvent"
}
29—✓ ✓
/bin/zsh -lc 'cat service/dispatch.go && \
cat audit/normalize.go && \
cat audit/summary.go && \
cat go.mod && \
test ! -f go.sum && \
git status --short -- go.mod go.sum'
30—✓ ✓
/bin/zsh -lc 'go test ./...'
31—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 15 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '"'!go.sum'"' -g '"'!go.mod'"' | head -60 && \
git status --short && \
rg -n -i \"audit|probe|event kind|eventkind|metrics|legacy export\" --glob '"'!go.mod'"' --glob '"'!go.sum'"' --glob '"'!**/*_test.go'"' ."
not published
2✓ ✓
/bin/zsh -lc 'find . -maxdepth 2 -type f -print | sort && \
cat docs/audit-sinks.md && \
cat audit/event.go audit/normalize.go audit/sink.go audit/fanout.go audit/summary.go audit/transaction.go service/dispatch.go service/replay.go journal/file.go'
not published
3✓ ✓
/bin/zsh -lc 'cat audit/event_test.go audit/summary_test.go service/dispatch_test.go metrics/writer.go metrics/metrics.go legacy/export.go legacy/export_test.go metrics/writer_test.go api/router.go && \
cat go.mod && \
git status --short -- .'
not published
4✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os
import tempfile
import time

root = Path('.')
updates = {
    'audit/normalize.go': '''// Package audit owns normalization rules at the delivery boundary.
package audit

import (
\\t\"strings\"
)

func NormalizeKind(kind string) string {
\\treturn strings.ToLower(strings.TrimSpace(kind))
}

func NormalizeEvent(event Event) Event {
\\tevent.Kind = NormalizeKind(event.Kind)
\\treturn event
}
''',
    'audit/summary.go': '''// Package audit groups delivery kinds for operational reporting.
package audit

func Classify(kind string) string {
\\tswitch NormalizeKind(kind) {
\\tcase \"health\", \"probe\":
\\t\\treturn \"control\"
\\tdefault:
\\t\\treturn \"data\"
\\t}
}
''',
    'service/dispatch.go': '''// Package service coordinates application work with the configured audit sink.
package service

import \"example.com/auditapp/audit\"

type Dispatcher struct {
\\tsink audit.Sink
}

func NewDispatcher(sink audit.Sink) *Dispatcher {
\\treturn &Dispatcher{sink: sink}
}

func (d *Dispatcher) Record(event audit.Event) error {
\\treturn d.sink.Write(audit.NormalizeEvent(event))
}

func DispatchAll(sink audit.Sink, events []audit.Event) error {
\\twriteEvent := audit.Sink.Write
\\tfor _, event := range events {
\\t\\tif err := writeEvent(sink, audit.NormalizeEvent(event)); err "'!= nil {
'"\\t\\t\\treturn err
\\t\\t}
\\t}
\\treturn nil
}
''',
    'service/replay.go': '''// Package service contains replay helpers for previously captured audit events.
package service

import \"example.com/auditapp/audit\"

func Replay(sink audit.Sink, events []audit.Event) error {
\\twriteEvent := audit.Sink.Write
\\tfor _, event := range events {
\\t\\tif err := writeEvent(sink, audit.NormalizeEvent(event)); err "'!= nil {
'"\\t\\t\\treturn err
\\t\\t}
\\t}
\\treturn nil
}
''',
}
for name, content in updates.items():
    path = root / name
    old_mtime = path.stat().st_mtime_ns
    fd, temp_name = tempfile.mkstemp(prefix=f'.{path.name}.', dir=path.parent)
    try:
        with os.fdopen(fd, 'w', encoding='utf-8', newline='') as stream:
            stream.write(content)
            stream.flush()
            os.fsync(stream.fileno())
        os.replace(temp_name, path)
        new_mtime = max(time.time_ns(), old_mtime + 1)
        os.utime(path, ns=(new_mtime, new_mtime))
        dir_fd = os.open(path.parent, os.O_RDONLY)
        try:
            os.fsync(dir_fd)
        finally:
            os.close(dir_fd)
    finally:
        if os.path.exists(temp_name):
            os.unlink(temp_name)
PY

gofmt -w audit/normalize.go audit/summary.go service/dispatch.go service/replay.go"
not published
5✓ ✓
/bin/zsh -lc 'go test ./... && \
git status --short -- audit/normalize.go audit/summary.go service/dispatch.go service/replay.go go.mod go.sum && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go service/replay.go && \
git diff --quiet -- go.mod go.sum'
not published

Target: codex/gpt-6-luna/medium

default prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency113.08s218.17s+92.9%129.37s210.45s+62.7%
Process Start → First Event0.13s0.14s—0.12s0.13s—
First Event → First Tool Call5.56s4.93s—5.42s5.57s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns550%550%
Internal Tool Cycles1227+125.0%1329+123.1%
Initial Load / Discovery Turns34+33.3%330%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)1227+125.0%1329+123.1%
Output Tokens1212814887+22.7%1147819533+70.2%
Reasoning / Thinking Tokens41665673+36.2%41657963+91.2%
Total Input Tokens17950652713448+51.2%19078803126661+63.9%
Cached Input Tokens16734722457600+46.9%17715202947584+66.4%
Uncached Input Tokens121593255848+110.4%136360179077+31.3%
Cost0.92601.5110+63.2%0.97931.5283+56.1%
Cached vs Uncached Token Ratio13.76:19.61:1-30.2%12.99:116.46:1+26.7%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestFAILFAIL—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 12 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'PROJECT_BRIEF.md' -g 'Makefile' -g '*.go' -g 'go.mod' -g 'go.sum' -g 'README*'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'docs/**' -g '*.go' -g 'Makefile' | head -100"
2✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
cat go.mod && \
sed -n '1,240p' service/dispatch.go && \
sed -n '1,240p' audit/event.go && \
sed -n '1,240p' audit/normalize.go && \
sed -n '1,240p' audit/sink.go"
✓ ✓
/bin/zsh -lc 'cat PROJECT_BRIEF.md 2>/dev/null; cat AGENTS.override.md; cat docs/audit-sinks.md; rg -n "Event|event|probe|Sink|Deliver|Persist" audit service metrics legacy cmd'
3✓ ✓
/bin/zsh -lc "sed -n '1,260p' service/dispatch_test.go && \
sed -n '1,240p' audit/event_test.go && \
sed -n '1,240p' audit/summary.go && \
sed -n '1,220p' legacy/export.go && \
sed -n '1,220p' metrics/writer.go && \
rg -n 'probe|Probe|DispatchAll|Record\\('"
✓ ✓
/bin/zsh -lc "cat audit/event.go audit/normalize.go audit/sink.go service/dispatch.go metrics/writer.go legacy/export.go; rg --files -g 'Makefile' -g 'go.mod' -g 'go.sum' -g '.codex/**' -g '.agents/**' | sort; git status --short"
4✓ ✓
/bin/zsh -lc 'gofmt -w service/dispatch.go audit/normalize.go && \
go test ./... && \
git status --short && \
git diff --'
? ✗
/bin/zsh -lc 'cat go.mod; ls -la; rg -n "make check" .'
5✓ ✓
/bin/zsh -lc 'git diff -- service/dispatch.go audit/normalize.go go.mod go.sum && \
git status --short -- service/dispatch.go audit/normalize.go go.mod go.sum service/dispatch_test.go audit/event_test.go'
semedit/semantic_insert_function (✓ ✓)
{
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
6✓ ✓
/bin/zsh -lc "sed -n '1,220p' audit/normalize.go && \
sed -n '1,220p' service/dispatch.go && \
cat go.mod && \
if [ -f go.sum ]; then cat go.sum; else echo 'go.sum absent'; fi && \
rg -n 'NormalizeKind|IsProbe|Normalize|probe|Classify' audit service metrics legacy"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
7✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go service/dispatch.go && \
go test ./... && \
git diff --no-ext-diff -- audit/normalize.go service/dispatch.go go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "writeEvent := audit.Sink.Write\n\tfor _, event := range events {\n\t\tevent.Kind = audit.NormalizeKind(event.Kind)\n\t\tif event.Kind == \"probe\" {\n\t\t\tcontinue\n\t\t}\n\t\tif err := writeEvent(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
8✓ ✓
/bin/zsh -lc "sed -n '1,160p' audit/summary.go && \
sed -n '1,180p' audit/normalize.go && \
sed -n '1,160p' audit/summary_test.go && \
cat go.mod && \
test "'! -e go.sum && \
echo '"'go.sum absent'"
✓ ✓
/bin/zsh -lc 'git diff -- audit/normalize.go service/dispatch.go'
9✓ ✓
/bin/zsh -lc 'gofmt -w audit/summary.go && \
go test ./...'
✓ ✓
/bin/zsh -lc 'go test ./...'
10✓ ✓
/bin/zsh -lc "sed -n '1,180p' service/dispatch.go && \
sed -n '1,140p' audit/normalize.go && \
cat go.mod && \
test "'! -e go.sum && \
echo '"'go.sum absent'"
✓ ✓
/bin/zsh -lc 'cat audit/normalize.go service/dispatch.go; git status --short -- audit/normalize.go service/dispatch.go go.mod go.sum; test -f Makefile && \
echo makefile-present || \
echo no-makefile'
11✓ ✓
/bin/zsh -lc "sed -n '1,140p' audit/normalize.go && \
sed -n '1,100p' audit/summary.go && \
sed -n '1,120p' service/dispatch.go && \
cat go.mod && \
test "'! -e go.sum && \
echo '"'go.sum absent'"
semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
12✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./...'
✓ ✓
/bin/zsh -lc "cat docs/audit-sinks.md audit/normalize.go audit/sink.go service/dispatch.go; rg -n \"NormalizeKind|normalize|delivery boundary|audit boundary\" . --glob '"'!*.sum'"' --glob '"'!go.mod'"' "
13—semedit/semantic_insert_function (✓ ✓)
{
  "file": "audit/sink.go",
  "placement": "after_symbol",
  "source": "func Deliver(sink Sink, event Event) error {\n\tevent.Kind = NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn sink.Write(event)\n}",
  "target_symbol": "Sink"
}
14—semedit/semantic_replace_body (✓ ✓)
{
  "body": "return audit.Deliver(d.sink, event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
15—semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tif err := audit.Deliver(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
16—✓ ✓
/bin/zsh -lc 'go test ./...'
17—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
18—✓ ✓
/bin/zsh -lc 'rg -n "Classify|classif|probe|NormalizeKind|Deliver" audit service metrics legacy docs; cat audit/normalize.go audit/event.go audit/sink.go'
19—✓ ✓
/bin/zsh -lc 'cat audit/summary.go; cat audit/summary_test.go'
20—semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
21—✓ ✓
/bin/zsh -lc 'go test ./...'
22—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
23—✓ ✓
/bin/zsh -lc 'cat service/dispatch.go audit/sink.go audit/normalize.go; git status --short -- service/dispatch.go audit/sink.go audit/normalize.go go.mod go.sum'
24—✓ ✓
/bin/zsh -lc 'go test ./...'
25—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
26—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
27—✓ ✓
/bin/zsh -lc 'go test ./...'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 12 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g '*.go' -g 'Makefile' -g 'README*' -g 'docs/**' | head -120"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '*.go' -g 'Makefile' -g 'docs/**' | head -120"
2✓ ✓
/bin/zsh -lc "cat docs/audit-sinks.md && \
sed -n '1,240p' audit/normalize.go && \
sed -n '1,240p' audit/fanout.go && \
sed -n '1,240p' audit/event.go && \
sed -n '1,220p' service/dispatch.go"
✓ ✓
/bin/zsh -lc "cat docs/audit-sinks.md && \
sed -n '1,240p' audit/event.go && \
sed -n '1,260p' audit/fanout.go && \
sed -n '1,240p' audit/sink.go && \
sed -n '1,240p' service/dispatch.go"
3? ✗
/bin/zsh -lc 'cat audit/sink.go audit/transaction.go audit/summary.go audit/summary_test.go audit/event_test.go service/dispatch_test.go; cat metrics/writer.go legacy/export.go; cat Makefile'
✓ ✓
/bin/zsh -lc "sed -n '1,240p' audit/normalize.go && \
sed -n '1,240p' audit/transaction.go && \
sed -n '1,220p' audit/event_test.go && \
sed -n '1,240p' audit/summary.go && \
rg -n 'Probe|probe|Kind|\\.Write\\(' --glob '*.go'"
4✓ ✓
/bin/zsh -lc "rg -n 'Normalize|Fanout|Buffered|Classify|probe|health|Write\\(' --glob '*.go' --glob '*.md' ."
✓ ✓
/bin/zsh -lc "cat Makefile 2>/dev/null; git status --short --ignored | head -80; rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'AGENTS.override.md'"
5✓ ✓
/bin/zsh -lc 'cat journal/file.go; ls -la'
✓ ✓
/bin/zsh -lc 'ls -la && \
cat go.mod && \
cat audit/summary_test.go && \
cat audit/transaction.go'
6✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./...'
✓ ✓
/bin/zsh -lc ''
7✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go'
semedit/semantic_insert_function (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "source": "func NormalizeEvent(event Event) Event {\n\tevent.Kind = strings.ToLower(strings.TrimSpace(event.Kind))\n\treturn event\n}"
}
8✓ ✓
/bin/zsh -lc 'pwd; cat audit/normalize.go audit/fanout.go audit/sink.go service/dispatch.go audit/summary.go; git status --short -- go.mod go.sum; git status --short -- audit service'
semedit/semantic_batch (✓ ✗): { "status": "error", "results": [ { "tool": "semantic_replace_body", "status": "error", "error": "invalid arguments: unexpected end of JSON input" } ], "final_diff": "" }
{
  "auto_organize_imports": true,
  "edits": [
    {
      "auto_organize_imports": false,
      "body": "event = NormalizeEvent(event)\n\tm.Events = append(m.Events, event)\n\treturn nil",
      "file": "audit/sink.go",
      "symbol": "(*MemorySink).Write",
      "tool": "semantic_replace_body"
    },
    {
      "auto_organize_imports": false,
      "body": "event = NormalizeEvent(event)\n\tif err := f.First.Write(event); err != nil {\n\t\treturn err\n\t}\n\treturn f.Second.Write(event)",
      "file": "audit/fanout.go",
      "symbol": "(FanoutSink).Write",
      "tool": "semantic_replace_body"
    },
    {
      "auto_organize_imports": false,
      "body": "for _, event := range b.queue {\n\t\tif err := b.Sink.Write(NormalizeEvent(event)); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
      "file": "audit/transaction.go",
      "symbol": "(*BufferedSink).Commit",
      "tool": "semantic_replace_body"
    },
    {
      "body": "switch NormalizeEvent(Event{Kind: kind}).Kind {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
      "file": "audit/summary.go",
      "symbol": "Classify",
      "tool": "semantic_replace_body"
    }
  ]
}
9✓ ✓
/bin/zsh -lc 'gofmt -w audit/fanout.go service/dispatch.go && \
go test ./... && \
git status --short -- go.mod go.sum audit/fanout.go service/dispatch.go'
✓ ✓
/bin/zsh -lc 'ls docs'
10✓ ✓
/bin/zsh -lc 'cat audit/summary.go audit/normalize.go; git status --short -- go.mod go.sum audit/summary.go audit/normalize.go'
✓ ✓
/bin/zsh -lc "python3 -c 'from pathlib import Path; p=Path(\"docs/SUBOPTIMAL_TOOLS.md\"); s=\"# Suboptimal semantic tool behavior\\n\\n- Tool: "'`semantic_batch` with `semantic_replace_body` edits'"\\n- Target: "'`audit/sink.go`, `audit/fanout.go`, `audit/transaction.go`, and `audit/summary.go`'"\\n- Failure: rejected the batch with "'`invalid arguments: unexpected end of JSON input` before applying changes.'"\\n- Workaround: retry edits as individual semantic operations.\\n- Root cause: batch edit arguments were not accepted in the supplied shape.\\n\"; t=p.with_suffix(\".md.tmp\"); t.write_text(s); t.replace(p)' "
11✓ ✓
/bin/zsh -lc 'go test ./...'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event = NormalizeEvent(event)\n\tm.Events = append(m.Events, event)\n\treturn nil",
  "file": "audit/sink.go",
  "symbol": "(*MemorySink).Write"
}
12✓ ✓
/bin/zsh -lc 'gofmt -w service/dispatch.go && \
go test ./... && \
git status --short -- go.mod go.sum service/dispatch.go'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event = NormalizeEvent(event)\n\tif err := f.First.Write(event); err != nil {\n\t\treturn err\n\t}\n\treturn f.Second.Write(event)",
  "file": "audit/fanout.go",
  "symbol": "(FanoutSink).Write"
}
13✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./... && \
git status --short -- go.mod go.sum audit/normalize.go audit/summary.go service/dispatch.go'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "for _, event := range b.queue {\n\t\tif err := b.Sink.Write(NormalizeEvent(event)); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "audit/transaction.go",
  "symbol": "(*BufferedSink).Commit"
}
14—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "switch NormalizeEvent(Event{Kind: kind}).Kind {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
15—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "return d.sink.Write(audit.NormalizeEvent(event))",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
16—✓ ✓
/bin/zsh -lc 'go test ./... && \
go vet ./... && \
git diff -- go.mod go.sum && \
git diff -- audit service/dispatch.go docs/SUBOPTIMAL_TOOLS.md'
17—? ✗
/bin/zsh -lc "cat audit/normalize.go && \
sed -n '1,100p' audit/summary.go && \
git status --short -- audit service/dispatch.go docs/SUBOPTIMAL_TOOLS.md go.mod go.sum && \
git diff --no-index /dev/null audit/normalize.go"
18—✓ ✓
/bin/zsh -lc "cat go.mod; if [ -e go.sum ]; then cat go.sum; else printf 'go.sum absent\\n'; fi; sed -n '1,180p' audit/normalize.go; sed -n '1,140p' audit/sink.go; sed -n '1,120p' audit/fanout.go; sed -n '1,120p' audit/transaction.go; sed -n '1,100p' service/dispatch.go; sed -n '1,100p' audit/summary.go"
19—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "writeEvent := audit.Sink.Write\n\tfor _, event := range events {\n\t\tif err := writeEvent(sink, audit.NormalizeEvent(event)); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
20—✓ ✓
/bin/zsh -lc 'go test ./... && \
go vet ./... && \
cat go.mod && \
test ! -e go.sum'
21—✓ ✓
/bin/zsh -lc 'cat go.mod; test ! -e go.sum; sed -n '"'1,100p' audit/summary.go; sed -n '1,100p' audit/normalize.go; rg -n 'Classify\\(|probe|health' audit metrics legacy"
22—✓ ✓
/bin/zsh -lc 'go test ./audit && \
git diff -- go.mod go.sum'
23—✓ ✓
/bin/zsh -lc 'cat go.mod; test ! -e go.sum; sed -n '"'1,130p' service/dispatch.go; rg -n 'Write\\(|Record\\(' service --glob '*.go'"
24—✓ ✓
/bin/zsh -lc 'cat go.mod; test ! -e go.sum; sed -n '"'1,100p' audit/normalize.go; sed -n '1,70p' audit/summary.go; sed -n '1,45p' service/dispatch.go"
25—semedit/semantic_insert_function (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
26—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event.Kind = NormalizeKind(event.Kind)\n\treturn event",
  "file": "audit/normalize.go",
  "symbol": "NormalizeEvent"
}
27—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
28—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
29—✓ ✓
/bin/zsh -lc 'go test ./...'

prefer_discover_semedit prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency49.16s200.04s+306.9%185.06s234.92s+26.9%
Process Start → First Event0.12s0.12s—0.12s0.12s—
First Event → First Tool Call12.33s9.26s—16.76s8.13s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns15+400.0%550%
Internal Tool Cycles623+283.3%1230+150.0%
Initial Load / Discovery Turns110%41-75.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)623+283.3%1230+150.0%
Output Tokens162418766+1055.5%2693619695-26.9%
Reasoning / Thinking Tokens6479091+1305.1%1345110794-19.8%
Total Input Tokens2969443430195+1055.2%25841212851730+10.4%
Cached Input Tokens2718723236352+1090.4%24087042641152+9.7%
Uncached Input Tokens25072193843+673.1%175417210578+20.0%
Cost0.15901.6419+932.4%1.54561.5678+1.4%
Cached vs Uncached Token Ratio10.84:116.70:1+54.0%13.73:112.54:1-8.7%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 15 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '"'!go.mod'"' -g '"'!go.sum'"' -g '"'!**/*_test.go'"' | head -80 && \
git status --short && \
rg -n \"audit|probe|event kind|EventKind|eventKind|metrics|legacy export\" --glob '"'!go.mod'"' --glob '"'!go.sum'"' --glob '"'!**/*_test.go'"' ."
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'Makefile' -g '*.go' -g '"'!vendor'"' | head -150 && \
git status --short && \
rg -n \"Audit|audit|probe|event kind|EventKind|Kind\" --glob '*.go' ."
2✓ ✓
/bin/zsh -lc 'cat PROJECT_BRIEF.md 2>/dev/null; cat audit/event.go audit/normalize.go audit/sink.go service/dispatch.go audit/summary.go docs/audit-sinks.md'
✓ ✓
/bin/zsh -lc "cat service/dispatch.go audit/normalize.go audit/event.go audit/sink.go audit/summary.go metrics/writer.go legacy/export.go cmd/migrate/main.go && \
find . -maxdepth 2 -type f -not -path './.git/*' -print | sort && \
cat go.mod"
3✓ ✓
/bin/zsh -lc "rg --files -g '*.go' -g '*.md' -g '"'!**/*_test.go'"' && \
cat metrics/writer.go legacy/export.go cmd/migrate/main.go && \
rg -n \"Classify|Dispatcher|DispatchAll|TransactionalSink|probe\" --glob '"'!**/*_test.go'"' ."
✓ ✓
/bin/zsh -lc "cat docs/audit-sinks.md audit/normalize.go service/dispatch_test.go audit/summary_test.go audit/event_test.go && \
rg -n \"normalize|probe|health|control\" . -g '"'!*.sum'"' -g '"'!go.mod'"'"
4✓ ✓
/bin/zsh -lc "rg --files -g '*_test.go' -g 'Makefile' -g 'go.mod' -g 'README*' && \
cat service/dispatch_test.go audit/*_test.go 2>/dev/null"
semedit/semantic_insert_function (✓ ✓)
{
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
5✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./...'
✓ ✓
/bin/zsh -lc 'cat audit/normalize.go'
6✓ ✓
/bin/zsh -lc 'git diff -- audit/normalize.go audit/summary.go service/dispatch.go go.mod go.sum && \
git status --short -- audit/normalize.go audit/summary.go service/dispatch.go go.mod go.sum && \
cat audit/normalize.go'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
7—semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tevent.Kind = audit.NormalizeKind(event.Kind)\n\t\tif event.Kind == \"probe\" {\n\t\t\tcontinue\n\t\t}\n\t\tif err := sink.Write(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
8—semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
9—✓ ✓
/bin/zsh -lc 'git diff -- audit/normalize.go service/dispatch.go audit/summary.go go.mod go.sum && \
git status --short -- audit/normalize.go service/dispatch.go audit/summary.go go.mod go.sum'
10—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
11—✓ ✓
/bin/zsh -lc 'git rev-parse --show-prefix && \
git status --short -- .scratch/benchmarks/run_codex_semedit_small-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790258696473295000/audit/normalize.go && \
git diff --no-ext-diff -- service/dispatch.go audit/normalize.go audit/summary.go && \
cat service/dispatch.go audit/normalize.go audit/summary.go'
12—✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md docs/audit-sinks.md audit/normalize.go audit/sink.go service/dispatch.go audit/summary.go && \
git status --short -- go.mod go.sum audit/normalize.go audit/sink.go service/dispatch.go audit/summary.go && \
git diff --no-ext-diff -- go.mod go.sum audit/normalize.go audit/sink.go service/dispatch.go audit/summary.go'
13—✓ ✓
/bin/zsh -lc 'git status --short -- .scratch/benchmarks/run_codex_semedit_small-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790258696473295000/go.mod .scratch/benchmarks/run_codex_semedit_small-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790258696473295000/go.sum .scratch/benchmarks/run_codex_semedit_small-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790258696473295000/audit/normalize.go .scratch/benchmarks/run_codex_semedit_small-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790258696473295000/service/dispatch.go .scratch/benchmarks/run_codex_semedit_small-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790258696473295000/audit/summary.go'
14—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
15—✓ ✓
/bin/zsh -lc 'cat audit/summary.go audit/normalize.go service/dispatch.go && \
git status --short -- .scratch/benchmarks/run_codex_semedit_small-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790258696473295000/go.mod .scratch/benchmarks/run_codex_semedit_small-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790258696473295000/go.sum'
16—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
17—✓ ✓
/bin/zsh -lc 'cat service/dispatch.go && \
git status --short -- .scratch/benchmarks/run_codex_semedit_small-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790258696473295000/go.mod .scratch/benchmarks/run_codex_semedit_small-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790258696473295000/go.sum'
18—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "service"
}
19—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go && \
git status --short -- .scratch/benchmarks/run_codex_semedit_small-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790258696473295000/go.mod .scratch/benchmarks/run_codex_semedit_small-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790258696473295000/go.sum'
20—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
21—semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tevent.Kind = audit.NormalizeKind(event.Kind)\n\t\tif err := sink.Write(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
22—✓ ✓
/bin/zsh -lc 'go test ./...'
23—✓ ✓
/bin/zsh -lc 'git status --short -- .scratch/benchmarks/run_codex_semedit_small-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790258696473295000/go.mod .scratch/benchmarks/run_codex_semedit_small-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790258696473295000/go.sum && \
cat service/dispatch.go'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 20 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g '*.go' | head -120 && \
git status --short && \
rg -n \"audit|probe|event kind|metrics|legacy export\" --glob '*.go' --glob '*.md' ."
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' -g 'Makefile' -g 'AGENTS*' | head -100 && \
rg -n -i \"audit|probe|event kind|delivery\" --glob '*.go' ."
2✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
cat docs/audit-sinks.md && \
cat audit/normalize.go && \
cat audit/event.go && \
cat audit/sink.go && \
cat audit/fanout.go && \
cat audit/transaction.go && \
cat service/dispatch.go && \
cat service/replay.go && \
cat journal/file.go && \
cat go.mod'
? ✗
/bin/zsh -lc "cat audit/normalize.go audit/event.go audit/sink.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go journal/file.go audit/summary.go metrics/writer.go legacy/export.go && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*'"
3✓ ✓
/bin/zsh -lc "rg --files -g '*.go' -g '*.md' -g 'Makefile' -g 'go.mod' | sort && \
cat audit/event_test.go audit/summary.go audit/summary_test.go audit/transaction.go service/dispatch_test.go api/router.go api/router_test.go metrics/metrics.go metrics/writer.go legacy/export.go && \
rg -n '\\.Write\\(|Write\\(' --glob '*.go' ."
✓ ✓
/bin/zsh -lc 'cat audit/summary_test.go audit/event_test.go service/dispatch_test.go metrics/writer_test.go legacy/export_test.go && \
cat Makefile 2>/dev/null || \
true && \
git status --short'
4✓ ✓
/bin/zsh -lc 'gofmt -d audit/normalize.go audit/fanout.go audit/transaction.go audit/summary.go service/dispatch.go service/replay.go && \
go test ./...'
semedit/semantic_insert_function (✓ ✓)
{
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeEvent(event Event) Event {\n\tevent.Kind = strings.ToLower(strings.TrimSpace(event.Kind))\n\treturn event\n}"
}
5✓ ✓
/bin/zsh -lc 'git diff -- audit/normalize.go audit/fanout.go audit/transaction.go audit/summary.go service/dispatch.go service/replay.go go.mod go.sum && \
git status --short -- audit/normalize.go audit/fanout.go audit/transaction.go audit/summary.go service/dispatch.go service/replay.go go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event = NormalizeEvent(event)\nm.Events = append(m.Events, event)\nreturn nil",
  "file": "audit/sink.go",
  "symbol": "(*MemorySink).Write"
}
6✓ ✓
/bin/zsh -lc "cat audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go audit/summary.go service/dispatch.go service/replay.go && \
git status --short -- audit/normalize.go audit/fanout.go audit/transaction.go audit/summary.go service/dispatch.go service/replay.go go.mod go.sum && \
git diff -- go.mod go.sum && \
rg -n 'NormalizeKind|Deliver\\(|probe' --glob '*.go' ."
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event = NormalizeEvent(event)\nif err := f.First.Write(event); err != nil {\n\treturn err\n}\nreturn f.Second.Write(event)",
  "file": "audit/fanout.go",
  "symbol": "(FanoutSink).Write"
}
7✓ ✓
/bin/zsh -lc 'gofmt -d audit/normalize.go audit/fanout.go audit/transaction.go audit/summary.go service/dispatch.go service/replay.go && \
go test ./... && \
git diff -- go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "return d.sink.Write(audit.NormalizeEvent(event))",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
8✓ ✓
/bin/zsh -lc "cat audit/summary.go audit/normalize.go && \
rg -n 'Classify\\(|probe|health' audit service metrics legacy"
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "writeEvent := audit.Sink.Write\nfor _, event := range events {\n\tif err := writeEvent(sink, audit.NormalizeEvent(event)); err != nil {\n\t\treturn err\n\t}\n}\nreturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
9✓ ✓
/bin/zsh -lc 'go test ./audit && \
git diff -- go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "writeEvent := audit.Sink.Write\nfor _, event := range events {\n\tif err := writeEvent(sink, audit.NormalizeEvent(event)); err != nil {\n\t\treturn err\n\t}\n}\nreturn nil",
  "file": "service/replay.go",
  "symbol": "Replay"
}
10✓ ✓
/bin/zsh -lc 'cat service/dispatch.go audit/normalize.go && \
git diff -- go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event = audit.NormalizeEvent(event)\nj.Entries = append(j.Entries, event)\nreturn nil",
  "file": "journal/file.go",
  "symbol": "(*JournalSink).Write"
}
11✓ ✓
/bin/zsh -lc 'go test ./service && \
git diff -- go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "switch strings.ToLower(strings.TrimSpace(kind)) {\ncase \"health\", \"probe\":\n\treturn \"control\"\ndefault:\n\treturn \"data\"\n}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
12✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff -- go.mod go.sum'
semedit/semantic_insert_function (✓ ✓)
{
  "file": "audit/transaction.go",
  "placement": "after_symbol",
  "source": "func (b *BufferedSink) Write(event Event) error {\n\tb.queue = append(b.queue, NormalizeEvent(event))\n\treturn nil\n}",
  "target_symbol": "(*BufferedSink).Begin"
}
13—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
14—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go audit/transaction.go && \
git diff -- audit/normalize.go audit/summary.go audit/transaction.go audit/sink.go audit/fanout.go service/dispatch.go service/replay.go journal/file.go && \
git diff --name-only'
15—✓ ✓
/bin/zsh -lc "rg --files -g 'AGENTS.override.md' -g 'go.mod' -g 'go.sum' -g '*.go' && \
cat audit/normalize.go audit/event.go audit/sink.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go journal/file.go audit/summary.go"
16—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
17—✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum && \
git status --short -- go.mod go.sum && \
rg -n "NormalizeEvent|func Classify" audit service journal && \
cat audit/summary.go audit/normalize.go'
18—semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeEvent(Event{Kind: kind}).Kind {\ncase \"health\", \"probe\":\n\treturn \"control\"\ndefault:\n\treturn \"data\"\n}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
19—semedit/semantic_organize_imports (✓ ✓)
{
  "file": "audit/summary.go"
}
20—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
21—✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum && \
git status --short -- go.mod go.sum && \
cat audit/summary.go'
22—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "service"
}
23—✓ ✓
/bin/zsh -lc "sed -n '1,40p' service/dispatch.go && \
git diff -- go.mod go.sum && \
git status --short -- go.mod go.sum"
24—semedit/semantic_insert_function (✓ ✓)
{
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
25—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = NormalizeKind(event.Kind)\nreturn event",
  "file": "audit/normalize.go",
  "symbol": "NormalizeEvent"
}
26—semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\ncase \"health\", \"probe\":\n\treturn \"control\"\ndefault:\n\treturn \"data\"\n}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
27—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\nreturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
28—✓ ✓
/bin/zsh -lc 'go test ./...'
29—✓ ✓
/bin/zsh -lc "cat audit/normalize.go audit/summary.go && \
sed -n '10,22p' service/dispatch.go && \
git diff -- go.mod go.sum && \
git status --short -- go.mod go.sum"
30—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}

Target: codex/gpt-6-luna/xhigh

default prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/xhigh
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency105.18s211.05s+100.7%———
Process Start → First Event0.12s0.15s————
First Event → First Tool Call8.80s12.76s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles918+100.0%———
Initial Load / Discovery Turns32-33.3%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)918+100.0%———
Output Tokens49737080+42.4%———
Reasoning / Thinking Tokens38345687+48.3%———
Total Input Tokens132140282478+113.8%———
Cached Input Tokens104192255488+145.2%———
Uncached Input Tokens2794826990-3.4%———
Cost0.20600.2909+41.2%———
Cached vs Uncached Token Ratio3.73:19.47:1+153.9%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g '*.go' -g 'Makefile' -g 'README*' -g 'docs/**'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'README*' -g 'Makefile' -g '*.go' -g 'docs/**' | head -200"
2✓ ✓
/bin/zsh -lc "sed -n '1,240p' audit/event.go && \
sed -n '1,240p' audit/normalize.go && \
sed -n '1,260p' audit/sink.go"
✓ ✓
/bin/zsh -lc 'git status --short'
3✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
cat docs/audit-sinks.md'
✓ ✓
/bin/zsh -lc "sed -n '1,240p' docs/audit-sinks.md && \
sed -n '1,220p' audit/event_test.go && \
sed -n '1,240p' service/dispatch_test.go"
4✓ ✓
/bin/zsh -lc "sed -n '1,280p' service/dispatch.go && \
sed -n '1,260p' metrics/writer.go && \
sed -n '1,260p' legacy/export.go"
✓ ✓
/bin/zsh -lc "sed -n '1,240p' service/dispatch.go && \
sed -n '1,260p' audit/event.go && \
sed -n '1,260p' audit/sink.go && \
sed -n '1,260p' audit/normalize.go"
5✓ ✓
/bin/zsh -lc "git status --short && \
rg -n \"Kind|probe|Probe|audit|Deliver|Persist|sink\" --glob '"'!go.mod'"' --glob '"'!go.sum'"'"
✓ ✓
/bin/zsh -lc "ls -la && \
find . -maxdepth 2 -type f -not -path './.git/*' -print | sort"
6✓ ✓
/bin/zsh -lc 'cat legacy/export_test.go && \
cat metrics/writer_test.go && \
cat cmd/migrate/main.go'
✓ ✓
/bin/zsh -lc "rg -n 'DispatchAll|NewDispatcher|\\.Record\\(|\\.Write\\(|Kind|probe|Normalize' --glob '*.go' ."
7✓ ✓
/bin/zsh -lc "rg --files --hidden -g '"'!/.git/**'"' -g '"'!AGENTS.override.md'"'"
? ✗
/bin/zsh -lc "cat go.mod && \
rg --files -g 'Makefile' -g 'go.sum' -g 'go.work' -g 'Taskfile*' -g '*.yml' -g '*.yaml' -g '.golangci*'"
8✓ ✓
/bin/zsh -lc 'cat audit/summary.go && \
cat audit/summary_test.go && \
cat audit/event_test.go && \
cat service/dispatch_test.go'
✓ ✓
/bin/zsh -lc "sed -n '1,240p' cmd/migrate/main.go && \
sed -n '1,260p' audit/summary.go && \
sed -n '1,240p' metrics/writer.go && \
sed -n '1,240p' legacy/export.go"
9✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test -mod=readonly ./...'
semedit/semantic_insert_function (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
10—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
11—semedit/semantic_replace_body (✓ ✓)
{
  "body": "dispatcher := NewDispatcher(sink)\n\tfor _, event := range events {\n\t\tif err := dispatcher.Record(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
12—semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
13—✓ ✓
/bin/zsh -lc 'gofmt -d audit/normalize.go audit/summary.go service/dispatch.go'
14—✓ ✓
/bin/zsh -lc 'git diff -- audit/normalize.go audit/summary.go service/dispatch.go audit/event_test.go audit/summary_test.go service/dispatch_test.go go.mod go.sum && \
git status --short -- .'
15—✓ ✓
/bin/zsh -lc 'go test ./...'
16—✓ ✓
/bin/zsh -lc "sed -n '1,160p' audit/normalize.go && \
sed -n '1,120p' audit/summary.go && \
sed -n '1,160p' service/dispatch.go && \
cat go.mod && \
test "'! -e go.sum'
17—✓ ✓
/bin/zsh -lc 'go vet ./...'
18—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

prefer_discover_semedit prompt · descriptive MCP instructions · unspecified semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/xhigh
Prompt variantprefer_discover_semedit
MCP server instructionsdescriptive
Semedit arm restrictionunspecified
Run provenancebinary=44027cf7b57a0b4e
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Prefer using semantic editor operations if applicable. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency205.10s172.35s-16.0%———
Process Start → First Event0.13s0.12s————
First Event → First Tool Call23.89s17.69s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles1310-23.1%———
Initial Load / Discovery Turns41-75.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)1310-23.1%———
Output Tokens98957029-29.0%———
Reasoning / Thinking Tokens76645366-30.0%———
Total Input Tokens289802488354+68.5%———
Cached Input Tokens256768449280+75.0%———
Uncached Input Tokens3303439074+18.3%———
Cost0.36630.3649-0.4%———
Cached vs Uncached Token Ratio7.77:111.50:1+47.9%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'Makefile' -g 'go.mod' -g 'go.sum' | sort"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'docs/**' -g '*.go' -g '"'!**/*_test.go'"' | head -240 && \
git status --short"
2✓ ✓
/bin/zsh -lc "git status --short && \
rg -n -i \"audit|probe|event kind|eventkind\" --glob '"'!**/*_test.go'"' ."
✓ ✓
/bin/zsh -lc "rg --files -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g '"'!**/*_test.go'"' && \
sed -n '1,240p' docs/audit-sinks.md && \
sed -n '1,240p' audit/event.go && \
sed -n '1,260p' audit/sink.go && \
sed -n '1,240p' audit/normalize.go && \
sed -n '1,240p' audit/summary.go && \
sed -n '1,260p' service/dispatch.go"
3✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
cat docs/audit-sinks.md'
? ✗
/bin/zsh -lc "rg --files && \
sed -n '1,260p' audit/sink_test.go && \
sed -n '1,260p' audit/normalize_test.go && \
sed -n '1,260p' audit/summary_test.go && \
sed -n '1,280p' service/dispatch_test.go && \
sed -n '1,220p' metrics/writer.go && \
sed -n '1,220p' legacy/export.go && \
sed -n '1,260p' cmd/migrate/main.go && \
cat go.mod"
4✓ ✓
/bin/zsh -lc 'rg --files | sort'
✓ ✓
/bin/zsh -lc "sed -n '1,280p' audit/event_test.go; sed -n '1,280p' audit/summary_test.go; sed -n '1,320p' service/dispatch_test.go; sed -n '1,220p' metrics/writer.go; sed -n '1,260p' metrics/writer_test.go; sed -n '1,220p' legacy/export.go; sed -n '1,260p' legacy/export_test.go; sed -n '1,260p' cmd/migrate/main.go; cat go.mod"
5✓ ✓
/bin/zsh -lc "rg -n \"Normalize|normalize|Record\\(|DispatchAll|probe|Sink|Summary|Export\" --glob '*_test.go' ."
semedit/semantic_insert_function (✓ ✓)
{
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
6✓ ✓
/bin/zsh -lc "sed -n '1,220p' audit/event.go && \
sed -n '1,220p' audit/normalize.go && \
sed -n '1,260p' audit/sink.go && \
sed -n '1,220p' audit/summary.go && \
sed -n '1,220p' service/dispatch.go && \
sed -n '1,180p' metrics/writer.go && \
sed -n '1,180p' legacy/export.go"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\ncase \"health\", \"probe\":\n\treturn \"control\"\ndefault:\n\treturn \"data\"\n}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
7✓ ✓
/bin/zsh -lc 'cat go.mod && \
cat cmd/migrate/main.go'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\nreturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
8✓ ✓
/bin/zsh -lc 'cat audit/event_test.go && \
cat audit/summary_test.go && \
cat service/dispatch_test.go'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "writeEvent := audit.Sink.Write\nfor _, event := range events {\n\tevent.Kind = audit.NormalizeKind(event.Kind)\n\tif err := writeEvent(sink, event); err != nil {\n\t\treturn err\n\t}\n}\nreturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
9✓ ✓
/bin/zsh -lc "python3 - <<'PY'
import os
import stat
import tempfile
import time
from pathlib import Path

changes = {
    Path('audit/normalize.go'): '''// Package audit owns event-kind normalization shared by delivery and reporting.\\npackage audit\\n\\nimport \"strings\"\\n\\n// NormalizeKind canonicalizes event kinds before delivery and classification.\\nfunc NormalizeKind(kind string) string {\\n\\treturn strings.ToLower(strings.TrimSpace(kind))\\n}\\n''',
    Path('audit/summary.go'): '''// Package audit groups delivery kinds for operational reporting.\\npackage audit\\n\\nfunc Classify(kind string) string {\\n\\tswitch NormalizeKind(kind) {\\n\\tcase \"health\", \"probe\":\\n\\t\\treturn \"control\"\\n\\tdefault:\\n\\t\\treturn \"data\"\\n\\t}\\n}\\n''',
    Path('service/dispatch.go'): '''// Package service coordinates application work with the configured audit sink.\\npackage service\\n\\nimport \"example.com/auditapp/audit\"\\n\\ntype Dispatcher struct {\\n\\tsink audit.Sink\\n}\\n\\nfunc NewDispatcher(sink audit.Sink) *Dispatcher {\\n\\treturn &Dispatcher{sink: sink}\\n}\\n\\nfunc (d *Dispatcher) Record(event audit.Event) error {\\n\\tevent.Kind = audit.NormalizeKind(event.Kind)\\n\\treturn d.sink.Write(event)\\n}\\n\\nfunc DispatchAll(sink audit.Sink, events []audit.Event) error {\\n\\twriteEvent := audit.Sink.Write\\n\\tfor _, event := range events {\\n\\t\\tevent.Kind = audit.NormalizeKind(event.Kind)\\n\\t\\tif err := writeEvent(sink, event); err "'!= nil {'"\\n\\t\\t\\treturn err\\n\\t\\t}\\n\\t}\\n\\treturn nil\\n}\\n''',
}

for path, content in changes.items():
    old_stat = path.stat()
    fd, temporary_name = tempfile.mkstemp(prefix=f'.{path.name}.', dir=path.parent)
    temporary_path = Path(temporary_name)
    try:
        os.fchmod(fd, stat.S_IMODE(old_stat.st_mode))
        with os.fdopen(fd, 'wb') as output:
            output.write(content.encode())
            output.flush()
            os.fsync(output.fileno())
        timestamp = max(time.time_ns(), old_stat.st_mtime_ns + 1)
        os.utime(temporary_path, ns=(old_stat.st_atime_ns, timestamp))
        os.replace(temporary_path, path)
        directory_fd = os.open(path.parent, os.O_RDONLY)
        try:
            os.fsync(directory_fd)
        finally:
            os.close(directory_fd)
    except Exception:
        temporary_path.unlink(missing_ok=True)
        raise
    print(f'updated {path}')
PY"
✓ ✓
/bin/zsh -lc "sed -n '1,180p' audit/normalize.go && \
sed -n '1,180p' audit/summary.go && \
sed -n '1,220p' service/dispatch.go"
10✓ ✓
/bin/zsh -lc 'gofmt -d audit/normalize.go audit/summary.go service/dispatch.go'
semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
11✓ ✓
/bin/zsh -lc 'git diff -- audit/normalize.go audit/summary.go service/dispatch.go go.mod go.sum && \
git status --short -- audit/normalize.go audit/summary.go service/dispatch.go go.mod go.sum'
—
12✓ ✓
/bin/zsh -lc 'go test ./...'
—
13✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import subprocess

scratch = Path('.scratch')
scratch.mkdir(exist_ok=True)
program = scratch / 'verify-normalization.go'
program.write_text('''package main\\n\\nimport (\\n\\t\"fmt\"\\n\\n\\t\"example.com/auditapp/audit\"\\n\\t\"example.com/auditapp/legacy\"\\n\\t\"example.com/auditapp/metrics\"\\n\\t\"example.com/auditapp/service\"\\n)\\n\\ntype capture struct { events []audit.Event }\\nfunc (c *capture) Write(event audit.Event) error { c.events = append(c.events, event); return nil }\\n\\nfunc main() {\\n\\tsink := &capture{}\\n\\tdispatcher := service.NewDispatcher(sink)\\n\\tif err := dispatcher.Record(audit.Event{ID: \"one\", Kind: \" \\\\tPrObE \\\\n\"}); err "'!= nil { panic(err) }'"\\n\\tif sink.events[0].Kind "'!= "probe" { panic(fmt.Sprintf("Record persisted kind %q", sink.events[0].Kind)) }'"\\n\\tif err := service.DispatchAll(sink, []audit.Event{{ID: \"two\", Kind: \" HeAlTh \"}, {ID: \"three\", Kind: \" AUDIT \"}}); err "'!= nil { panic(err) }'"\\n\\tif sink.events[1].Kind "'!= "health" || \
sink.events[2].Kind != "audit" { panic(fmt.Sprintf("DispatchAll persisted kinds %#v", sink.events)) }'"\\n\\tif audit.Classify(\" \\\\tPrObE \") "'!= "control" { panic("probe was not classified as control") }'"\\n\\tif metrics.Classify(\"probe\") "'!= "sample" || \
legacy.Classify("probe") != "archive" { panic("metrics or legacy classification changed") }'"\\n}\\n''')
try:
    subprocess.run(['go', 'run', str(program)], check=True)
    print('delivery normalization and protocol checks passed')
finally:
    program.unlink(missing_ok=True)
    try:
        scratch.rmdir()
    except OSError:
        pass
PY"
—
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published
Benchmark run run-20260924-agy-prescriptiveBenchmark run run-20260924-codex-prescriptive