Skip to content
Documentation
Benchmarks
Benchmark runs
Benchmark run gpt-6-luna-all-r5-c10-descriptive-20260926

Benchmark run gpt-6-luna-all-r5-c10-descriptive-20260926

This page contains exactly the publishable benchmark observations recorded in run gpt-6-luna-all-r5-c10-descriptive-20260926; it does not select or aggregate them. Return to the run index or view aggregate metrics.

Interactive browser

Rows are pre-aggregated by task, target, prompt variant, MCP-instruction mode, context, arm, and restriction. Numeric values show median (p50), IQR, p90, maximum, and sample count. Boolean checks show pass counts and success rates. Filters apply to these summary groups; per-run observations remain available on their run pages.

Loading benchmark results…

Baseline vs semedit

Comparison evaluates baseline and semedit values within each restriction policy, target, and context group. Negative changes mark improvements for these lower-is-better metrics; positive changes mark degradations.

Detailed observations

Empirical telemetry measuring wall-clock latency, token usage, tool turns, and compiler correctness when AI coding agents perform code modifications.

Every benchmark pairs Vanilla LLM (standard file editing tools) against Semedit MCP (deterministic AST compiler operations).

Benchmark Methodology & Transparency

  • Reproducibility & Provenance: Self-contained txtar archives define test scenarios. Links below resolve to commit-anchored GitHub source files for published commits, or cryptographic SHA-256 fingerprints for uncommitted local fixtures.
  • Model Cost: Tables report model-specific credit consumption where a target model has a declared schedule: (uncached input × input credits + cached input × cached credits + (reasoning + visible output) × output credits) / 1,000,000. Thinking tokens use the output rate. ADR-0043 defines the per-million-token credit rates.
  • Multi-Level Correctness Oracle: Four validation levels evaluate each trial:
    1. Level 1 (Mutation Policy): Restricts file modifications strictly to authorized paths.
    2. Level 2 (AST Invariants): Compiler AST verification of required symbols, imports, and relative declaration ordering.
    3. Level 3 (Clean Build): Strict compilation validation with zero compiler or typecheck errors.
    4. Level 4 (Test Suite): Unit and integration test suite execution.
  • Isolated Sandboxing: Each trial executes in an ephemeral directory with pristine git and module state.

Test case: task-04-insert-public

Target: codex/gpt-6-luna/medium

default prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-04-insert-public
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_04_insert_public.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package api

type Server struct {
	Port int
}

func (s *Server) Start() {}

func (s *Server) internalRun() {}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency19.90s46.69s+134.6%67.33s38.88s-42.3%
Process Start → First Event1.56s1.96s—0.78s1.01s—
First Event → First Tool Call9.76s13.36s—9.55s7.14s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles26+200.0%63-50.0%
Initial Load / Discovery Turns21-50.0%41-75.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)26+200.0%63-50.0%
Output Tokens312486+55.8%1055329-68.8%
Reasoning / Thinking Tokens880-100.0%16547-71.5%
Total Input Tokens76072128427+68.8%136432100680-26.2%
Cached Input Tokens66560111360+67.3%12723287808-31.0%
Uncached Input Tokens951217067+79.4%920012872+39.9%
Cost0.04540.0766+68.6%0.07010.0588-16.0%
Cached vs Uncached Token Ratio7.00:16.52:1-6.8%13.83:16.82:1-50.7%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g '*.go' -g '"'!vendor'"'"
✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' | head -80"
2✓ ✓
/bin/zsh -lc 'cat api/server.go && \
git status --short'
semedit/semantic_outline (✓ ✓)
{
  "include_tests": false,
  "include_unexported": true,
  "path": "."
}
3—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "api/server.go",
  "symbol": "Server"
}
4—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "api/server.go",
  "symbol": "internalRun"
}
5—✓ ✓
/bin/zsh -lc 'rg --files'
6—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "before_symbol",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}",
  "target_symbol": "internalRun"
}
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g '*.go' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '"'!**/*_test.go'"'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS*.md' -g '*.go' -g 'README.md' | head -80"
2✓ ✓
/bin/zsh -lc 'cat api/server.go'
? ✗
/bin/zsh -lc "cat AGENTS.override.md && \
sed -n '1,220p' api/server.go && \
rg -n \"semantic_insert|semantic_lookup|semantic_outline|semantic_verify\" ."
3✓ ✓
/bin/zsh -lc "cat api/router.go && \
rg 'Server\\{|InitServer|internalRun'"
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "public_end",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}
4? ✗
/bin/zsh -lc "cat api/server_test.go && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md'"
—
5? ✗
/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"api/server.go\"); s=p.read_text(); needle=\"type Server struct {\\n\\tPort int\\n}\\n\"; replacement=needle+\"\\nfunc InitServer() *Server {\\n\\treturn &Server{}\\n}\\n\"; assert needle in s and \"func InitServer()\" not in s; fd, tmp=tempfile.mkstemp(dir=str(p.parent), prefix=\".server.go.\");\\nwith os.fdopen(fd, \"w\") as f: f.write(s.replace(needle,replacement,1)); f.flush(); os.fsync(f.fileno())\\nos.replace(tmp,p)'"
—
6✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os
import tempfile

path = Path('api/server.go')
source = path.read_text()
needle = 'type Server struct {\\n\\tPort int\\n}\\n'
replacement = needle + '\\nfunc InitServer() *Server {\\n\\treturn &Server{}\\n}\\n'
assert needle in source and 'func InitServer()' not in source
fd, temporary = tempfile.mkstemp(dir=path.parent, prefix='.server.go.')
with os.fdopen(fd, 'w') as output:
    output.write(source.replace(needle, replacement, 1))
    output.flush()
    os.fsync(output.fileno())
os.replace(temporary, path)
PY
cat api/server.go"
—

default prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-04-insert-public
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_04_insert_public.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package api

type Server struct {
	Port int
}

func (s *Server) Start() {}

func (s *Server) internalRun() {}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency22.68s55.00s+142.5%27.22s45.89s+68.6%
Process Start → First Event2.17s2.37s—0.81s0.91s—
First Event → First Tool Call10.78s11.07s—6.22s8.52s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles26+200.0%440%
Initial Load / Discovery Turns220%41-75.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)26+200.0%440%
Output Tokens339795+134.5%444522+17.6%
Reasoning / Thinking Tokens97198+104.1%128167+30.5%
Total Input Tokens76144212595+179.2%115040133179+15.8%
Cached Input Tokens66560189440+184.6%105984118272+11.6%
Uncached Input Tokens958423155+141.6%905614907+64.6%
Cost0.04610.1177+155.5%0.05630.0754+34.0%
Cached vs Uncached Token Ratio6.94:18.18:1+17.8%11.70:17.93:1-32.2%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g 'AGENTS.md' -g '*.go' -g 'go.mod' -g 'go.sum' -g 'PROJECT_BRIEF.md'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g '*.go' | head -80 && \
rg -n 'type Server|func .*Server|private' --glob '*.go' ."
2✓ ✓
/bin/zsh -lc 'cat api/server.go && \
git status --short'
✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
cat api/server.go'
3—semedit/semantic_insert_function (✓ ✗): calculate function offset: section placement violation: strategy "private_start", target symbol "InitServer", in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-default_task-04-insert-public_1790456047051019000/api/server.go
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "private_start",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}
4—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "public_end",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}
5—✓ ✓
/bin/zsh -lc 'rg --files | sort && \
git diff -- api/server.go go.mod go.sum'
6—✓ ✓
/bin/zsh -lc 'cat api/server.go && \
git status --short'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g '"'!*test*'"' -g '"'!go.mod'"' -g '"'!go.sum'"'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'tools/benchmark-harness/README.md' -g '*.go' | head -200 && \
rg -n 'type Server|func .*Server|private helper|func [a-z]' --glob '*.go' ."
2✓ ✓
/bin/zsh -lc "sed -n '1,240p' api/server.go"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "api/server.go",
  "language": "go",
  "symbol": "Server"
}
3✓ ✓
/bin/zsh -lc 'git diff -- api/server.go'
semedit/semantic_outline (✓ ✓)
{
  "language": "go",
  "path": "api/server.go"
}
4✓ ✓
/bin/zsh -lc "sed -n '1,80p' api/server.go"
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "language": "go",
  "placement": "public_end",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}

default prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-04-insert-public
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_04_insert_public.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package api

type Server struct {
	Port int
}

func (s *Server) Start() {}

func (s *Server) internalRun() {}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency30.47s62.79s+106.1%24.91s42.11s+69.0%
Process Start → First Event1.55s2.73s—1.07s1.05s—
First Event → First Tool Call11.75s10.77s—8.09s10.61s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles45+25.0%26+200.0%
Initial Load / Discovery Turns21-50.0%24+100.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)45+25.0%26+200.0%
Output Tokens508565+11.2%361546+51.2%
Reasoning / Thinking Tokens91204+124.2%98138+40.8%
Total Input Tokens95958149374+55.7%76574126337+65.0%
Cached Input Tokens85760130560+52.2%68608113152+64.9%
Uncached Input Tokens1019818814+84.5%796613185+65.5%
Cost0.05440.0893+64.1%0.04280.0698+63.1%
Cached vs Uncached Token Ratio8.41:16.94:1-17.5%8.61:18.58:1-0.4%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g 'AGENTS.md' -g 'AGENTS.override.md' -g '*.go' -g 'PROJECT_BRIEF.md' -g 'README.md'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '*.go' -g 'Makefile' | head -80"
2✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
cat api/server.go'
semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "path": "api/server.go"
}
3✓ ✓
/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"api/server.go\"); data=p.read_text(); old=\"func (s *Server) Start() {}\\n\\nfunc (s *Server) internalRun() {}\\n\"; new=\"func (s *Server) Start() {}\\n\\nfunc InitServer() *Server {\\n\\treturn &Server{}\\n}\\n\\nfunc (s *Server) internalRun() {}\\n\"; assert old in data; fd, tmp=tempfile.mkstemp(dir=p.parent, prefix=\".server.go.\"); f=os.fdopen(fd,\"w\"); f.write(data.replace(old,new)); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(tmp,p)' && \
git diff -- api/server.go"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "api/server.go",
  "symbol": "Start"
}
4✓ ✓
/bin/zsh -lc 'cat api/server.go; git status --short; git diff --no-ext-diff -- api/server.go'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "api/server.go",
  "symbol": "Server"
}
5—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "before_symbol",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}",
  "target_symbol": "internalRun"
}
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g '*.go' -g 'go.mod' -g 'go.sum' -g 'PROJECT_BRIEF.md' && \
git status --short"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'README*' -g '*.go' | head -100"
2✓ ✓
/bin/zsh -lc "sed -n '1,240p' api/server.go && \
sed -n '1,180p' api/server_test.go"
✓ ✓
/bin/zsh -lc 'git status --short'
3—✓ ✓
/bin/zsh -lc 'cat api/server.go'
4—? ✗
/bin/zsh -lc "cat AGENTS.override.md && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' -g 'tools/benchmark-harness/README.md'"
5—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "public_start",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}
6—semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "language": "go",
  "path": "api/server.go"
}

default prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-04-insert-public
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_04_insert_public.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package api

type Server struct {
	Port int
}

func (s *Server) Start() {}

func (s *Server) internalRun() {}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency27.41s46.34s+69.1%72.59s52.61s-27.5%
Process Start → First Event2.87s2.79s—3.26s1.77s—
First Event → First Tool Call9.97s11.31s—23.07s19.55s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles45+25.0%74-42.9%
Initial Load / Discovery Turns32-33.3%32-33.3%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)45+25.0%74-42.9%
Output Tokens489478-2.2%803399-50.3%
Reasoning / Thinking Tokens57115+101.8%13172-45.0%
Total Input Tokens95418114496+20.0%155649125748-19.2%
Cached Input Tokens86784101120+16.5%14438498048-32.1%
Uncached Input Tokens863413376+54.9%1126527700+145.9%
Cost0.05010.0661+32.0%0.07590.0996+31.2%
Cached vs Uncached Token Ratio10.05:17.56:1-24.8%12.82:13.54:1-72.4%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g '*.go' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' | head -80"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' -g 'AGENTS.md' -g '*.go' | head -80"
2✓ ✓
/bin/zsh -lc 'cat api/server.go'
✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'tools/benchmark-harness/README.md' -g 'README.md' -g 'AGENTS.md' | head -50"
3✓ ✓
/bin/zsh -lc 'cat api/server_test.go'
semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "language": "go",
  "path": "api/server.go"
}
4✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
p = Path('api/server.go')
s = p.read_text()
needle = 'func (s *Server) Start() {}\\n\\nfunc (s *Server) internalRun() {}\\n'
replacement = 'func (s *Server) Start() {}\\n\\nfunc InitServer() *Server {\\n\\treturn &Server{}\\n}\\n\\nfunc (s *Server) internalRun() {}\\n'
if needle not in s:
    raise SystemExit('expected source layout not found')
fd, tmp = tempfile.mkstemp(dir=p.parent, prefix='.server.go.')
try:
    with os.fdopen(fd, 'w') as f:
        f.write(s.replace(needle, replacement, 1))
        f.flush()
        os.fsync(f.fileno())
    os.replace(tmp, p)
except Exception:
    try: os.unlink(tmp)
    except FileNotFoundError: pass
    raise
PY"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "api/server.go",
  "language": "go",
  "symbol": "Server"
}
5—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "public_end",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g '*.go' -g 'AGENTS.md' -g 'AGENTS.override.md' | head -80"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g '*.go' -g 'Makefile' | head -80"
2✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
rg --files'
✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
rg -n "type Server|func .*Server|private" api/server.go'
3✓ ✓
/bin/zsh -lc 'cat api/server.go'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "api/server.go",
  "language": "go",
  "symbol": "Server"
}
4✓ ✓
/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"api/server.go\"); data=p.read_text(); old=\"func (s *Server) Start() {}\\n\\nfunc (s *Server) internalRun() {}\\n\"; new=\"func (s *Server) Start() {}\\n\\nfunc InitServer() *Server {\\n\\treturn &Server{}\\n}\\n\\nfunc (s *Server) internalRun() {}\\n\"; assert old in data; fd, name=tempfile.mkstemp(dir=str(p.parent)); f=os.fdopen(fd,\"w\"); f.write(new); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(name,p)'"
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "public_start",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}
5✓ ✓
/bin/zsh -lc 'cat api/server.go && \
git diff -- api/server.go'
—
6✓ ✓
/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"api/server.go\"); new=\"package api\\n\\ntype Server struct {\\n\\tPort int\\n}\\n\\nfunc (s *Server) Start() {}\\n\\nfunc InitServer() *Server {\\n\\treturn &Server{}\\n}\\n\\nfunc (s *Server) internalRun() {}\\n\"; fd, name=tempfile.mkstemp(dir=str(p.parent)); f=os.fdopen(fd,\"w\"); f.write(new); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(name,p)'"
—
7✓ ✓
/bin/zsh -lc 'cat api/server.go && \
git diff -- api/server.go'
—

default prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-04-insert-public
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_04_insert_public.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Add public constructor func InitServer() *Server placed before private helpers. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package api

type Server struct {
	Port int
}

func (s *Server) Start() {}

func (s *Server) internalRun() {}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency—51.89s—21.86s47.03s+115.1%
Process Start → First Event—2.04s—0.15s0.16s—
First Event → First Tool Call—10.55s—11.70s6.33s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns—1—110%
Internal Tool Cycles—4—24+100.0%
Initial Load / Discovery Turns—1—220%
MCP Discovery / Schema Turns—0—00—
Tool Invocations (per run)—4—24+100.0%
Output Tokens—525—333481+44.4%
Reasoning / Thinking Tokens—208—0165—
Total Input Tokens—159174—76363107995+41.4%
Cached Input Tokens—135680—6041692928+53.8%
Uncached Input Tokens—23494—1594715067-5.5%
Cost—0.1018—0.05910.0690+16.6%
Cached vs Uncached Token Ratio—5.78:1—3.79:16.17:1+62.8%
Oracle L1: Mutation Policy—PASS—PASSPASS—
Oracle L2: AST Invariants—PASS—PASSPASS—
Oracle L3: Clean Build—PASS—PASSPASS—
Oracle L4: Verification Test—PASS—PASSPASS—
Semantic Tool Invocation Verified (per run)—PASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1not published✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g '*.go' -g 'go.mod' -g 'go.sum' | head -80"
2not publishedsemedit/semantic_outline (✓ ✓)
{
  "path": "api/server.go"
}
3not publishedsemedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "public_end",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}
4not publishedsemedit/semantic_outline (✓ ✓)
{
  "path": "api/server.go"
}
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File api/server.go modified (9 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc 'rg -n "type Server|func .*Server|''^func " --glob '"'*.go' ."
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g '*benchmark*' -g 'AGENTS.md' -g '*.go' | head -80"
2✓ ✓
/bin/zsh -lc 'cat api/server.go'
✓ ✓
/bin/zsh -lc 'rg --files'
3—semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "language": "go",
  "path": "api/server.go"
}
4—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "api/server.go",
  "placement": "public_start",
  "source": "func InitServer() *Server {\n\treturn \u0026Server{}\n}"
}

Test case: task-07-generate-template-main

Target: codex/gpt-6-luna/medium

crypto_rand prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantcrypto_rand
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency29.92s62.54s+109.0%———
Process Start → First Event0.64s0.95s————
First Event → First Tool Call14.79s15.40s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles310+233.3%———
Initial Load / Discovery Turns21-50.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)310+233.3%———
Output Tokens616960+55.8%———
Reasoning / Thinking Tokens183289+57.9%———
Total Input Tokens76680166364+117.0%———
Cached Input Tokens68608149760+118.3%———
Uncached Input Tokens807216604+105.7%———
Cost0.04730.0946+99.8%———
Cached vs Uncached Token Ratio8.50:19.02:1+6.1%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 20 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'go.mod' -g 'go.sum' -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'README*'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'Makefile' -g 'go.mod' -g 'go.sum'"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod && \
git status --short'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "symbol": "main"
}
3✓ ✓
/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"main.go\"); data=\"\"\"package main

import (
\\t\\\"crypto/rand\\\"
\\t\\\"math/big\\\"
\\t\\\"os\\\"
\\t\\\"text/template\\\"
)

func main() {
\\tn, err := rand.Int(rand.Reader, big.NewInt(1_000_000))
\\tif err "'!= nil {
'"\\t\\tpanic(err)
\\t}

\\ttmpl := template.Must(template.New(\\\"greeting\\\").Parse(\\\"Hello World {{.}}\\\\n\\\"))
\\tif err := tmpl.Execute(os.Stdout, n); err "'!= nil {
'"\\t\\tpanic(err)
\\t}
}
\"\"\"; fd, name=tempfile.mkstemp(dir=str(p.parent), prefix=\".main.go.\"); f=os.fdopen(fd,\"w\"); f.write(data); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(name,p)'"
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "n, err := rand.Int(rand.Reader, big.NewInt(100))\nif err != nil {\n\tlog.Fatal(err)\n}\ntmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World: {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, n); err != nil {\n\tlog.Fatal(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4—✓ ✓
/bin/zsh -lc 'go build -o /tmp/semedit-hello-world .'
5—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "symbol": "main"
}
6—✓ ✓
/bin/zsh -lc 'git status --short -- main.go go.mod go.sum && \
git diff -- main.go'
7—semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "text/template"
  ],
  "file": "main.go",
  "remove": [
    "html/template"
  ]
}
8—✓ ✓
/bin/zsh -lc 'git status --short -- main.go go.mod go.sum'
9—✓ ✓
/bin/zsh -lc 'go build -o /tmp/semedit-hello-world .'
10—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "symbol": "main"
}
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

crypto_rand prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantcrypto_rand
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency35.72s35.44s-0.8%———
Process Start → First Event1.69s1.45s————
First Event → First Tool Call18.44s8.66s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles42-50.0%———
Initial Load / Discovery Turns20-100.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)42-50.0%———
Output Tokens462261-43.5%———
Reasoning / Thinking Tokens150-100.0%———
Total Input Tokens9623987103-9.5%———
Cached Input Tokens8780875776-13.7%———
Uncached Input Tokens843111327+34.3%———
Cost0.04900.0505+3.1%———
Cached vs Uncached Token Ratio10.41:16.69:1-35.8%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 20 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 22 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'go.mod' -g 'go.sum' -g 'AGENTS*' -g 'PROJECT_BRIEF.md'"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod && \
git status --short'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "n, err := rand.Int(rand.Reader, big.NewInt(100))\nif err != nil {\n\tpanic(err)\n}\ndata := struct {\n\tMessage string\n\tNumber  *big.Int\n}{Message: \"Hello World\", Number: n}\nif err := template.New(\"hello\").Execute(os.Stdout, data); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
3✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
p = Path('main.go')
s = '''package main

import (
	\"crypto/rand\"
	\"math/big\"
	\"os\"
	\"text/template\"
)

func main() {
	value, err := rand.Int(rand.Reader, big.NewInt(1<<31))
	if err "'!= nil {
		panic(err)
	}

	tmpl := template.Must(template.New("greeting").Parse("Hello World {{.}}'"\\\\n\"))
	if err := tmpl.Execute(os.Stdout, value); err "'!= nil {
		panic(err)
	}
}
'"'''
tmp = p.with_suffix('.go.tmp')
tmp.write_text(s)
tmp.replace(p)
PY
gofmt -w main.go
git diff -- main.go"
—
4? ✗
/bin/zsh -lc 'cat main.go && \
git status --short -- main.go go.mod go.sum && \
git diff --no-index /dev/null main.go'
—
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

crypto_rand prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantcrypto_rand
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency26.75s40.19s+50.3%———
Process Start → First Event1.13s1.61s————
First Event → First Tool Call7.57s6.60s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles34+33.3%———
Initial Load / Discovery Turns31-66.7%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)34+33.3%———
Output Tokens471714+51.6%———
Reasoning / Thinking Tokens0325————
Total Input Tokens95966131812+37.4%———
Cached Input Tokens8780893952+7.0%———
Uncached Input Tokens815837860+364.1%———
Cost0.04820.1311+171.9%———
Cached vs Uncached Token Ratio10.76:12.48:1-76.9%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 23 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 24 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'README*' -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'go.sum'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'README*' -g 'go.mod' -g 'go.sum'"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
3✓ ✓
/bin/zsh -lc 'go run main.go'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "number, err := rand.Int(rand.Reader, big.NewInt(100))\nif err != nil {\n\tpanic(err)\n}\n\ngreeting, err := template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\")\nif err != nil {\n\tpanic(err)\n}\n\nif err := greeting.Execute(os.Stdout, number.Int64()); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

crypto_rand prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantcrypto_rand
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency27.91s43.86s+57.1%———
Process Start → First Event0.71s0.64s————
First Event → First Tool Call7.73s9.89s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles36+100.0%———
Initial Load / Discovery Turns31-66.7%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)36+100.0%———
Output Tokens505702+39.0%———
Reasoning / Thinking Tokens0240————
Total Input Tokens96039133591+39.1%———
Cached Input Tokens81664118272+44.8%———
Uncached Input Tokens1437515319+6.6%———
Cost0.06270.0796+27.1%———
Cached vs Uncached Token Ratio5.68:17.72:1+35.9%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSFAIL————
Oracle L3: Clean BuildPASSFAIL————
Oracle L4: Verification TestPASSFAIL————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 27 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 24 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'go.mod' -g 'go.sum' -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'README*'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'PROJECT_BRIEF.md' -g 'Makefile' -g 'AGENTS.md' -g 'AGENTS.override.md' | head -50"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod'
semedit/semantic_lookup (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
3✓ ✓
/bin/zsh -lc 'gofmt -w main.go && \
go run main.go && \
git status --short'
✓ ✓
/bin/zsh -lc 'rg --files'
4—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
5—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "randomInt, err := rand.Int(rand.Reader, big.NewInt(100))\nif err != nil {\n\tlog.Fatal(err)\n}\n\nmessage, err := template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\")\nif err != nil {\n\tlog.Fatal(err)\n}\nif err := message.Execute(os.Stdout, randomInt); err != nil {\n\tlog.Fatal(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
6—✓ ✓
/bin/zsh -lc 'go run . && \
git diff -- main.go go.mod go.sum'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

crypto_rand prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantcrypto_rand
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer from crypto/rand, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency19.46s33.17s+70.5%———
Process Start → First Event0.64s0.51s————
First Event → First Tool Call6.00s6.66s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles45+25.0%———
Initial Load / Discovery Turns41-75.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)45+25.0%———
Output Tokens480445-7.3%———
Reasoning / Thinking Tokens980-100.0%———
Total Input Tokens95542116351+21.8%———
Cached Input Tokens84736101120+19.3%———
Uncached Input Tokens1080615231+40.9%———
Cost0.05540.0689+24.4%———
Cached vs Uncached Token Ratio7.84:16.64:1-15.3%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 20 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 20 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'go.mod' -g 'go.sum' -g 'AGENTS.md' -g 'AGENTS.override.md'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' | head -80"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "symbol": "main"
}
3✓ ✓
/bin/zsh -lc "cat > main.go <<'EOF'
package main

import (
	\"crypto/rand\"
	\"math/big\"
	\"os\"
	\"text/template\"
)

func main() {
	n, err := rand.Int(rand.Reader, big.NewInt(100))
	if err "'!= nil {
		panic(err)
	}

	tmpl := template.Must(template.New("greeting").Parse("Hello World {{.}}'"\\n\"))
	if err := tmpl.Execute(os.Stdout, n); err "'!= nil {
		panic(err)
	}
}
EOF
gofmt -w main.go
git diff -- main.go go.mod go.sum'
semedit/semantic_outline (✓ ✓)
{
  "path": "main.go"
}
4✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "n, err := rand.Int(rand.Reader, big.NewInt(100))\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\ttmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\n\tif err := tmpl.Execute(os.Stdout, n); err != nil {\n\t\tlog.Fatal(err)\n\t}",
  "file": "main.go",
  "symbol": "main"
}
5—✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short -- go.mod go.sum main.go && \
go build ./...'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

default prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency37.03s41.42s+11.9%———
Process Start → First Event0.26s0.17s————
First Event → First Tool Call13.97s13.62s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles43-25.0%———
Initial Load / Discovery Turns21-50.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)43-25.0%———
Output Tokens789632-19.9%———
Reasoning / Thinking Tokens201301+49.8%———
Total Input Tokens96228108857+13.1%———
Cached Input Tokens8576093952+9.6%———
Uncached Input Tokens1046814905+42.4%———
Cost0.06000.0724+20.7%———
Cached vs Uncached Token Ratio8.19:16.30:1-23.1%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSFAIL————
Oracle L3: Clean BuildPASSFAIL————
Oracle L4: Verification TestPASSFAIL————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 12 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 15 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'go.mod' -g 'go.sum' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'PROJECT_BRIEF.md' -g 'Makefile'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'README*' -g 'Makefile' -g 'go.mod' -g 'go.sum' -g 'AGENTS.override.md'"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod && \
git status --short'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
3? ✗
/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile; p=Path(\"main.go\"); data=\"package main\\n\\nimport (\\n\\t\\\"math/rand\\\"\\n\\t\\\"os\\\"\\n\\t\\\"text/template\\\"\\n)\\n\\nfunc main() {\\n\\tt := template.Must(template.New(\\\"greeting\\\").Parse(\"Hello World {{.}}\\\\n\"))\\n\\t_ = t.Execute(os.Stdout, rand.Intn(100))\\n}\\n\"; fd, tmp=tempfile.mkstemp(dir=p.parent, prefix=\".main.go.\"); f=os.fdopen(fd,\"w\"); f.write(data); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(tmp,p)' && \
cat main.go"
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nvalue := rand.Intn(100)\nif err := tmpl.Execute(os.Stdout, value); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os
import tempfile

path = Path('main.go')
content = '''package main

import (
	\"math/rand\"
	\"os\"
	\"text/template\"
)

func main() {
	t := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\\\n\"))
	_ = t.Execute(os.Stdout, rand.Intn(100))
}
'''
fd, temporary = tempfile.mkstemp(dir=path.parent, prefix='.main.go.')
with os.fdopen(fd, 'w') as output:
    output.write(content)
    output.flush()
    os.fsync(output.fileno())
os.replace(temporary, path)
PY
cat main.go"
—
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

default prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency31.86s40.10s+25.9%———
Process Start → First Event0.13s0.15s————
First Event → First Tool Call12.70s16.69s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles34+33.3%———
Initial Load / Discovery Turns32-33.3%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)34+33.3%———
Output Tokens623608-2.4%———
Reasoning / Thinking Tokens281246-12.5%———
Total Input Tokens95657104428+9.2%———
Cached Input Tokens8576091904+7.2%———
Uncached Input Tokens989712524+26.5%———
Cost0.05750.0650+13.0%———
Cached vs Uncached Token Ratio8.67:17.34:1-15.3%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 17 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'Makefile' -g 'go.mod'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'Makefile' -g 'README*' -g 'go.mod' -g 'go.sum'"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod && \
rg --files'
✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod'
3✓ ✓
/bin/zsh -lc 'gofmt -w main.go && \
cat main.go && \
git status --short'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl, err := template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\")\nif err != nil {\n\tpanic(err)\n}\nif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4—✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

default prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency29.60s38.72s+30.8%———
Process Start → First Event0.18s0.19s————
First Event → First Tool Call13.08s13.71s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles440%———
Initial Load / Discovery Turns21-50.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)440%———
Output Tokens673616-8.5%———
Reasoning / Thinking Tokens160215+34.4%———
Total Input Tokens95563134268+40.5%———
Cached Input Tokens84736116224+37.2%———
Uncached Input Tokens1082718044+66.7%———
Cost0.05870.0846+44.1%———
Cached vs Uncached Token Ratio7.83:16.44:1-17.7%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSFAIL————
Oracle L3: Clean BuildPASSFAIL————
Oracle L4: Verification TestPASSFAIL————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 15 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'go.mod' -g 'go.sum'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'Makefile' -g 'README*' -g '"'!go.mod'"' -g '"'!go.sum'"'"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod'
? ✗
/bin/zsh -lc "cat main.go && \
cat AGENTS.override.md && \
rg -n \"semantic_(lookup|inspect_symbol|replace_body|organize_imports|verify)|semedit\" . --glob '"'!go.mod'"' --glob '"'!go.sum'"' --glob '"'!AGENTS.override.md'"'"
3✓ ✓
/bin/zsh -lc "cat > main.go.tmp <<'EOF'
package main

import (
	\"math/rand\"
	\"os\"
	\"text/template\"
)

func main() {
	tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))
	_ = tmpl.Execute(os.Stdout, rand.Int())
}
EOF
python3 - <<'PY'
from pathlib import Path
import os
p = Path('main.go.tmp')
data = p.read_bytes()
with open('main.go', 'wb') as f:
    f.write(data)
    f.flush()
    os.fsync(f.fileno())
p.unlink()
PY
gofmt -w main.go"
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err != nil {\n\tlog.Fatal(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os
p = Path('main.go')
s = p.read_text()
s = s.replace('\\t_ = tmpl.Execute(os.Stdout, rand.Int())', '\\tif err := tmpl.Execute(os.Stdout, rand.Int()); err "'!= nil {'"\\n\\t\\tpanic(err)\\n\\t}')
tmp = Path('main.go.tmp')
tmp.write_text(s)
with tmp.open('rb') as f:
    os.fsync(f.fileno())
os.replace(tmp, p)
PY
gofmt -w main.go
cat main.go"
✓ ✓
/bin/zsh -lc 'cat main.go && \
go run .'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

default prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency28.47s53.63s+88.4%———
Process Start → First Event0.13s0.17s————
First Event → First Tool Call8.48s6.06s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles27+250.0%———
Initial Load / Discovery Turns12+100.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)27+250.0%———
Output Tokens6211358+118.7%———
Reasoning / Thinking Tokens213259+21.6%———
Total Input Tokens57132220343+285.7%———
Cached Input Tokens48384198400+310.1%———
Uncached Input Tokens874821943+150.8%———
Cost0.04440.1247+180.8%———
Cached vs Uncached Token Ratio5.53:19.04:1+63.5%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSFAIL————
Oracle L3: Clean BuildPASSFAIL————
Oracle L4: Verification TestPASSFAIL————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 17 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 15 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'README*' -g 'Makefile' -g 'go.mod' -g 'AGENTS*' | sort && \
sed -n '1,200p' main.go"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'Makefile' -g 'README*' -g 'go.mod' -g 'go.sum'"
2✓ ✓
/bin/zsh -lc "python3 - <<'PY'
import os
import tempfile
from pathlib import Path

path = Path('main.go')
content = '''package main

import (
	\"math/rand\"
	\"os\"
	\"text/template\"
)

func main() {
	tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.Random}}\\\\n\"))
	data := struct {
		Random int
	}{Random: rand.Intn(100)}
	if err := tmpl.Execute(os.Stdout, data); err "'!= nil {
		panic(err)
	}
}
'"'''
fd, temp_name = tempfile.mkstemp(dir=path.parent, prefix='.main.go.')
try:
    with os.fdopen(fd, 'w') as f:
        f.write(content)
        f.flush()
        os.fsync(f.fileno())
    os.replace(temp_name, path)
finally:
    if os.path.exists(temp_name):
        os.unlink(temp_name)
PY
gofmt -w main.go
go run main.go"
✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod && \
git status --short'
3—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "\ttmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\n\tif err := tmpl.Execute(os.Stdout, rand.Int()); err != nil {\n\t\tlog.Fatal(err)\n\t}",
  "file": "main.go",
  "symbol": "main"
}
4—? ✗
/bin/zsh -lc "cat main.go && \
rg --files -g 'SUBOPTIMAL_TOOLS.md' -g 'Makefile'"
5—semedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "math/rand"
  ],
  "file": "main.go",
  "remove": [
    "crypto/rand"
  ]
}
6—✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
p = Path('docs/SUBOPTIMAL_TOOLS.md')
p.parent.mkdir(parents=True, exist_ok=True)
old = p.read_text() if p.exists() else '# Suboptimal Tool Behavior\\n'
entry = '''\\n## 2026-09-26: "'`semantic_replace_body` selected the wrong `rand` package'"\\n\\n- Tool: "'`semantic_replace_body` on `main.go`'"\\n- Observed failure: "'`rand.Int()` was resolved from `crypto/rand`, producing a vet diagnostic because this API requires an `io.Reader` and `*big.Int`.'"\\n- Workaround applied: changed imports with "'`semantic_organize_imports` to use `math/rand`, whose `Int()` returns a random integer.'"\\n- Root cause: package selection for the ambiguous "'`rand.Int()` selector chose `crypto/rand` instead of `math/rand`.'"\\n'''
fd, name = tempfile.mkstemp(dir=p.parent, prefix='.SUBOPTIMAL_TOOLS.')
try:
    with os.fdopen(fd, 'w') as f:
        f.write(old + entry)
        f.flush()
        os.fsync(f.fileno())
    os.replace(name, p)
finally:
    if os.path.exists(name):
        os.unlink(name)
PY
cat main.go
cat docs/SUBOPTIMAL_TOOLS.md"
7—✓ ✓
/bin/zsh -lc 'go build ./...'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

default prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency29.32s34.91s+19.1%———
Process Start → First Event0.14s0.13s————
First Event → First Tool Call8.96s9.69s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles550%———
Initial Load / Discovery Turns52-60.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)550%———
Output Tokens729719-1.4%———
Reasoning / Thinking Tokens286222-22.4%———
Total Input Tokens134725163332+21.2%———
Cached Input Tokens123136141824+15.2%———
Uncached Input Tokens1158921508+85.6%———
Cost0.07240.1010+39.4%———
Cached vs Uncached Token Ratio10.63:16.59:1-37.9%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'Makefile' -g 'README*'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'go.mod' -g 'go.sum'"
2✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short && \
rg --files'
✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod'
3✓ ✓
/bin/zsh -lc 'cat go.mod'
semedit/semantic_lookup (✓ ✓)
{
  "file": "main.go",
  "language": "go",
  "symbol": "main"
}
4✓ ✓
/bin/zsh -lc 'gofmt -w main.go && \
git diff -- main.go go.mod go.sum && \
git status --short'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
5✓ ✓
/bin/zsh -lc 'cat main.go && \
git ls-files --error-unmatch main.go 2>/dev/null || \
true'
✓ ✓
/bin/zsh -lc 'cat main.go && \
go run .'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

prefer_discover_semedit prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency44.94s51.31s+14.2%———
Process Start → First Event0.14s0.22s————
First Event → First Tool Call18.93s17.61s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles34+33.3%———
Initial Load / Discovery Turns32-33.3%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)34+33.3%———
Output Tokens921995+8.0%———
Reasoning / Thinking Tokens494481-2.6%———
Total Input Tokens193160168697-12.7%———
Cached Input Tokens171264146944-14.2%———
Uncached Input Tokens2189621753-0.7%———
Cost0.11520.1096-4.9%———
Cached vs Uncached Token Ratio7.82:16.76:1-13.6%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g '"'!go.mod'"' -g '"'!go.sum'"' && \
sed -n '1,200p' main.go"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'README*' | head -40"
2✓ ✓
/bin/zsh -lc 'gofmt -d main.go && \
git diff -- main.go go.mod go.sum'
✓ ✓
/bin/zsh -lc "cat main.go; rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'go.mod' -g 'Makefile'"
3✓ ✓
/bin/zsh -lc "sed -n '1,120p' main.go && \
git status --short"
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"hello\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4—✓ ✓
/bin/zsh -lc 'cat main.go && \
go run main.go && \
git status --short'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

prefer_discover_semedit prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency38.99s65.16s+67.1%———
Process Start → First Event0.14s0.36s————
First Event → First Tool Call21.00s21.01s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles24+100.0%———
Initial Load / Discovery Turns220%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)24+100.0%———
Output Tokens597807+35.2%———
Reasoning / Thinking Tokens164327+99.4%———
Total Input Tokens164670226279+37.4%———
Cached Input Tokens142848203776+42.7%———
Uncached Input Tokens2182222503+3.1%———
Cost0.09980.1214+21.6%———
Cached vs Uncached Token Ratio6.55:19.06:1+38.3%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 14 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc 'cat main.go'
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'go.mod' -g 'AGENTS.md' -g 'AGENTS.override.md'"
2✓ ✓
/bin/zsh -lc 'cat main.go'
✓ ✓
/bin/zsh -lc "cat main.go && \
printf '\\n--- go.mod (read-only) ---\\n' && \
cat go.mod"
3—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4—✓ ✓
/bin/zsh -lc 'cat main.go && \
go run .'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

prefer_discover_semedit prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency29.15s158.07s+442.3%———
Process Start → First Event0.20s0.15s————
First Event → First Tool Call10.60s31.83s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles310+233.3%———
Initial Load / Discovery Turns31-66.7%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)310+233.3%———
Output Tokens10172471+143.0%———
Reasoning / Thinking Tokens3231208+274.0%———
Total Input Tokens195826540871+176.2%———
Cached Input Tokens173312506624+192.3%———
Uncached Input Tokens2251434247+52.1%———
Cost0.11640.2583+121.9%———
Cached vs Uncached Token Ratio7.70:114.79:1+92.2%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 16 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 19 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "sed -n '1,200p' main.go && \
git status --short"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'README*' -g 'AGENTS.md' -g 'AGENTS.override.md'"
2✓ ✓
/bin/zsh -lc 'gofmt -w main.go && \
git diff -- main.go'
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "main"
}
3? ✗
/bin/zsh -lc "sed -n '1,100p' main.go; git status --short; git diff --no-index /dev/null main.go"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "symbol": "main"
}
4—semedit/semantic_replace_body (✓ ✗): param "symbol" is required: invalid operation parameters
{
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Int()); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "function_name": "main"
}
5—? ✗
/bin/zsh -lc "rg --files | rg '("'^|/)docs/SUBOPTIMAL_TOOLS'"\\.md"'$|(''^|/)PROJECT_BRIEF'"\\.md"'$'"'"
6—semedit/semantic_replace_body (✓ ✗): param "file" is required: invalid operation parameters
{
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Int()); err != nil {\n\tpanic(err)\n}",
  "symbol": "main"
}
7—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, rand.Int()); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
8—semedit/semantic_inspect_symbol (✓ ✓)
{
  "symbol": "main"
}
9—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "n, err := rand.Int(rand.Reader, big.NewInt(100))\nif err != nil {\n\tpanic(err)\n}\ntmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\"))\nif err := tmpl.Execute(os.Stdout, n); err != nil {\n\tpanic(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
10—semedit/semantic_inspect_symbol (✓ ✓)
{
  "symbol": "main"
}
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

prefer_discover_semedit prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency30.95s29.59s-4.4%———
Process Start → First Event0.16s0.28s————
First Event → First Tool Call11.79s5.46s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles330%———
Initial Load / Discovery Turns32-33.3%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)330%———
Output Tokens927665-28.3%———
Reasoning / Thinking Tokens466320-31.3%———
Total Input Tokens194130106974-44.9%———
Cached Input Tokens17331286016-50.4%———
Uncached Input Tokens2081820958+0.7%———
Cost0.11280.0862-23.6%———
Cached vs Uncached Token Ratio8.33:14.10:1-50.7%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 16 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 15 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'PROJECT_BRIEF.md' -g 'AGENTS*.md' | head -50 && \
cat main.go"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'main.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g 'go.sum'"
2✓ ✓
/bin/zsh -lc 'git diff -- main.go && \
git status --short'
? ✗
/bin/zsh -lc "cat main.go && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*'"
3✓ ✓
/bin/zsh -lc 'cat main.go && \
git check-ignore -v main.go || \
true && \
git status --short --untracked-files=no'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl := template.Must(template.New(\"greeting\").Parse(\"Hello World: {{.}}\\n\"))\n\tif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err != nil {\n\t\tlog.Fatal(err)\n\t}",
  "file": "main.go",
  "symbol": "main"
}
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

prefer_discover_semedit prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-07-generate-template-main
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/scripts/generate_template_main.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. In main.go, implement main() to execute a text/template that prints ‘Hello World’ along with a random integer, ensuring all necessary standard library packages are imported cleanly. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package main

func main() {
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency35.92s44.10s+22.8%———
Process Start → First Event0.17s0.16s————
First Event → First Tool Call19.00s17.64s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles24+100.0%———
Initial Load / Discovery Turns220%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)24+100.0%———
Output Tokens687709+3.2%———
Reasoning / Thinking Tokens342300-12.3%———
Total Input Tokens164957167221+1.4%———
Cached Input Tokens143872145920+1.4%———
Uncached Input Tokens2108521301+1.0%———
Cost0.10150.1023+0.8%———
Cached vs Uncached Token Ratio6.82:16.85:1+0.4%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 15 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File main.go modified (4 lines -> 18 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc 'cat main.go && \
git status --short'
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'main.go' -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'go.mod' -g 'README*'"
2✓ ✓
/bin/zsh -lc 'gofmt -w main.go && \
cat main.go && \
git status --short -- main.go go.mod go.sum'
✓ ✓
/bin/zsh -lc 'cat main.go && \
cat go.mod'
3—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "tmpl, err := template.New(\"greeting\").Parse(\"Hello World {{.}}\\n\")\nif err != nil {\n\tlog.Fatal(err)\n}\nif err := tmpl.Execute(os.Stdout, rand.Intn(100)); err != nil {\n\tlog.Fatal(err)\n}",
  "file": "main.go",
  "symbol": "main"
}
4—✓ ✓
/bin/zsh -lc 'cat main.go && \
go run main.go'
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

Test case: task-09-composite-refactor

Target: codex/gpt-6-luna/medium

default prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-09-composite-refactor
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_09_composite_refactor.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package config

import (
	"archive/tar"
	"fmt"
)

type Config struct {
	Host string
	Port int
}

func (c *Config) Addr() string {
	return fmt.Sprintf("%s:%d", c.Host, c.Port)
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency39.54s62.40s+57.8%26.40s73.80s+179.5%
Process Start → First Event0.28s0.33s—0.14s0.12s—
First Event → First Tool Call13.83s15.62s—5.74s14.71s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles67+16.7%610+66.7%
Initial Load / Discovery Turns61-83.3%32-33.3%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)67+16.7%610+66.7%
Output Tokens746531-28.8%7771061+36.6%
Reasoning / Thinking Tokens1550-100.0%186380+104.3%
Total Input Tokens157456218968+39.1%136325185746+36.3%
Cached Input Tokens142336200448+40.8%124160165120+33.0%
Uncached Input Tokens1512018520+22.5%1216520626+69.6%
Cost0.08460.1030+21.7%0.07350.1109+50.8%
Cached vs Uncached Token Ratio9.41:110.82:1+15.0%10.21:18.01:1-21.6%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "sed -n '1,240p' config/config.go && \
rg -n '\\.Port\\b|Port:' --glob '"'!**/*_test.go'"' ."
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md' | sort"
2? ✗
/bin/zsh -lc "rg -n 'Config\\{|DefaultConfig|Host:|Port:' --glob '"'!**/*_test.go'"' . && \
rg --files -g 'README*' -g 'PROJECT_BRIEF.md' -g 'go.mod'"
semedit/semantic_outline (✓ ✓)
{
  "language": "go",
  "path": "config/config.go"
}
3✓ ✓
/bin/zsh -lc "rg --files | head -80 && \
git status --short && \
rg -n '8080|localhost|127\\.0\\.0\\.1' ."
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config"
}
4✓ ✓
/bin/zsh -lc 'cat config/config_test.go && \
cat go.mod'
semedit/semantic_rename (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
5✓ ✓
/bin/zsh -lc 'gofmt -w config/config.go && \
git diff -- config/config.go && \
git status --short'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "config/config.go",
  "placement": "public_end",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost: \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}"
}
6? ✗
/bin/zsh -lc 'cat config/config.go; git status --short --untracked-files=all; git diff --no-index /dev/null config/config.go'
semedit/semantic_organize_imports (✓ ✓)
{
  "file": "config/config.go"
}
7—✓ ✓
/bin/zsh -lc 'git diff -- config/config.go && \
git status --short'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "sed -n '1,240p' config/config.go && \
rg -n '\\.Port|Port:' --glob '"'!go.mod'"' --glob '"'!go.sum'"' ."
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'config/config.go' -g 'go.mod' -g 'Makefile' -g 'docs/**' | head -80"
2✓ ✓
/bin/zsh -lc "rg -n 'config\\.Config|Config\\{|DefaultConfig|ListenPort|Host:' --glob '"'!go.mod'"' --glob '"'!go.sum'"' ."
✓ ✓
/bin/zsh -lc "cat config/config.go && \
cat go.mod && \
rg --files -g 'PROJECT_BRIEF.md' -g 'Makefile' -g 'README*' -g 'config/**' | head -80"
3✓ ✓
/bin/zsh -lc "cat config/config_test.go && \
rg -n '8080|localhost|127\\.0\\.0\\.1|ListenPort' . --glob '"'!go.mod'"' --glob '"'!go.sum'"'"
semedit/semantic_lookup (✓ ✓)
{
  "file": "config/config.go",
  "symbol": "Config"
}
4✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
p = Path('config/config.go')
s = p.read_text()
s = s.replace('\\t\"archive/tar\"\\n', '')
s = s.replace('\\tPort int\\n', '\\tListenPort int\\n')
s = s.replace('\\nfunc (c *Config) Addr() string {\\n\\treturn fmt.Sprintf(\"%s:%d\", c.Host, c.Port)\\n}', '\\nfunc DefaultConfig() *Config {\\n\\treturn &Config{\\n\\t\\tHost:       \"localhost\",\\n\\t\\tListenPort: 8080,\\n\\t}\\n}\\n\\nfunc (c *Config) Addr() string {\\n\\treturn fmt.Sprintf(\"%s:%d\", c.Host, c.ListenPort)\\n}')
p.write_text(s)
PY
cat config/config.go"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "symbol": "Config"
}
5✓ ✓
/bin/zsh -lc 'gofmt -w config/config.go && \
git diff -- config/config.go && \
git status --short'
✓ ✓
/bin/zsh -lc 'cat config/config_test.go'
6✓ ✓
/bin/zsh -lc 'cat config/config.go && \
git diff --no-index /dev/null config/config.go || \
true'
semedit/semantic_rename (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
7—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "config/config.go",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}"
}
8—semedit/semantic_organize_imports (✓ ✓)
{
  "file": "config/config.go"
}
9—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- config/config.go && \
GOFLAGS=-mod=readonly go test ./...'
10—? ✗
/bin/zsh -lc 'cat config/config.go && \
git diff --no-index /dev/null config/config.go'

default prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-09-composite-refactor
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_09_composite_refactor.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package config

import (
	"archive/tar"
	"fmt"
)

type Config struct {
	Host string
	Port int
}

func (c *Config) Addr() string {
	return fmt.Sprintf("%s:%d", c.Host, c.Port)
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency18.52s69.76s+276.8%43.08s52.68s+22.3%
Process Start → First Event0.14s0.13s—0.62s0.14s—
First Event → First Tool Call4.93s6.51s—12.60s6.11s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles39+200.0%612+100.0%
Initial Load / Discovery Turns31-66.7%330%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)39+200.0%612+100.0%
Output Tokens5241017+94.1%1114850-23.7%
Reasoning / Thinking Tokens93420+351.6%246159-35.4%
Total Input Tokens96613276132+185.8%137435231823+68.7%
Cached Input Tokens86784252160+190.6%125184210688+68.3%
Uncached Input Tokens982923972+143.9%1225121135+72.5%
Cost0.05400.1409+161.1%0.07890.1181+49.7%
Cached vs Uncached Token Ratio8.83:110.52:1+19.1%10.22:19.97:1-2.4%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "sed -n '1,240p' config/config.go && \
rg -n '\\bPort\\b|\\.Port\\b|ListenPort|DefaultConfig' --glob '"'!go.mod'"' --glob '"'!go.sum'"' ."
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'README.md' | head -40"
2? ✗
/bin/zsh -lc "sed -n '1,200p' config/config_test.go && \
rg -n 'DefaultConfig|ListenPort|Host:' --glob '*.go' . && \
rg -n 'default|port|host' README.md config"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "symbol": "Config"
}
3✓ ✓
/bin/zsh -lc 'git diff -- config/config.go && \
git status --short'
semedit/semantic_rename (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
4—semedit/semantic_outline (✓ ✓)
{
  "include_unexported": true,
  "path": "config/config.go"
}
5—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "symbol": "Config.Addr"
}
6—✓ ✓
/bin/zsh -lc "rg --files -g '"'!go.mod'"' -g '"'!go.sum'"' | head -60"
7—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "config/config.go",
  "placement": "public_end",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}"
}
8—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- config/config.go go.mod go.sum'
9—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "symbol": "DefaultConfig"
}
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 19 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg -n \"Config\\.Port|\\.Port\\b|type Config|Config\\{\" --glob '"'!**/*_test.go'"' . && \
sed -n '1,240p' config/config.go"
✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'config.go' -g 'README*'"
2? ✗
/bin/zsh -lc "rg -n \"Host:|ListenPort|DefaultConfig|localhost|127\\.0\\.0\\.1\" --glob '"'!**/*_test.go'"' . && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'README*' ."
✓ ✓
/bin/zsh -lc 'cat config/config.go'
3✓ ✓
/bin/zsh -lc "rg -n \"config\\.Config|Port|Host|8080|3000\" ."
? ✗
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'README.md'"
4? ✗
/bin/zsh -lc "python3 -c 'from pathlib import Path; import os, tempfile, time; p=Path(\"config/config.go\"); data=\"\"\"package config\\n\\nimport \\\"fmt\\\"\\n\\ntype Config struct {\\n\\tHost       string\\n\\tListenPort int\\n}\\n\\nfunc DefaultConfig() *Config {\\n\\treturn &Config{\\n\\t\\tHost:       \\\"localhost\\\",\\n\\t\\tListenPort: 8080,\\n\\t}\\n}\\n\\nfunc (c *Config) Addr() string {\\n\\treturn fmt.Sprintf(\\\"%s:%d\\\", c.Host, c.ListenPort)\\n}\\n\"; fd,tmp=tempfile.mkstemp(dir=str(p.parent), prefix=\".config.go.\"); f=os.fdopen(fd,\"w\"); f.write(data); f.flush(); os.fsync(f.fileno()); f.close(); os.replace(tmp,p); now=time.time_ns(); os.utime(p,ns=(now,now))'"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config"
}
5✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os
import tempfile
import time

path = Path(\"config/config.go\")
content = '''package config

import \"fmt\"

type Config struct {
\\tHost       string
\\tListenPort int
}

func DefaultConfig() *Config {
\\treturn &Config{
\\t\\tHost:       \"localhost\",
\\t\\tListenPort: 8080,
\\t}
}

func (c *Config) Addr() string {
\\treturn fmt.Sprintf(\"%s:%d\", c.Host, c.ListenPort)
}
'''
fd, temporary = tempfile.mkstemp(dir=path.parent, prefix=\".config.go.\")
with os.fdopen(fd, \"w\") as stream:
    stream.write(content)
    stream.flush()
    os.fsync(stream.fileno())
os.replace(temporary, path)
now = time.time_ns()
os.utime(path, ns=(now, now))
PY

git diff -- config/config.go go.mod go.sum && \
git status --short"
✓ ✓
/bin/zsh -lc "rg -n 'DefaultConfig|ListenPort|\\.Port|Host:' --glob '"'!config/config.go'"' ."
6✓ ✓
/bin/zsh -lc "sed -n '1,120p' config/config.go && \
git diff --no-index /dev/null config/config.go || \
true"
semedit/semantic_rename (✓ ✓)
{
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
7—✓ ✓
/bin/zsh -lc 'cat config/config_test.go'
8—✓ ✓
/bin/zsh -lc 'cat config/config.go'
9—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "config/config.go",
  "placement": "after_symbol",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}",
  "target_symbol": "Config"
}
10—semedit/semantic_organize_imports (✓ ✓)
{
  "file": "config/config.go"
}
11—✓ ✓
/bin/zsh -lc 'git diff -- config/config.go go.mod go.sum config/config_test.go'
12—✓ ✓
/bin/zsh -lc 'cat config/config.go'

default prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-09-composite-refactor
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_09_composite_refactor.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package config

import (
	"archive/tar"
	"fmt"
)

type Config struct {
	Host string
	Port int
}

func (c *Config) Addr() string {
	return fmt.Sprintf("%s:%d", c.Host, c.Port)
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency24.68s64.77s+162.5%29.40s39.90s+35.7%
Process Start → First Event0.14s0.41s—0.15s0.14s—
First Event → First Tool Call5.90s6.37s—5.38s5.37s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles49+125.0%57+40.0%
Initial Load / Discovery Turns41-75.0%52-60.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)49+125.0%57+40.0%
Output Tokens5641006+78.4%846763-9.8%
Reasoning / Thinking Tokens120282+135.0%115211+83.5%
Total Input Tokens115500213120+84.5%135658177113+30.6%
Cached Input Tokens104960193280+84.1%124160160768+29.5%
Uncached Input Tokens1054019840+88.2%1149816345+42.2%
Cost0.06110.1140+86.5%0.07180.0932+29.9%
Cached vs Uncached Token Ratio9.96:19.74:1-2.2%10.80:19.84:1-8.9%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 18 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'Makefile' -g 'go.mod'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'docs/benchmark*' -g 'tools/benchmark-harness/README.md'"
2✓ ✓
/bin/zsh -lc 'cat config/config.go && \
rg --files | head -60'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config"
}
3✓ ✓
/bin/zsh -lc 'cat config/config_test.go && \
cat go.mod && \
cat AGENTS.override.md'
semedit/semantic_outline (✓ ✓)
{
  "language": "go",
  "path": "config/config.go"
}
4✓ ✓
/bin/zsh -lc 'gofmt -w config/config.go && \
cat config/config.go'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Addr"
}
5—semedit/semantic_rename (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
6—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "config/config.go",
  "placement": "public_end",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{Host: \"localhost\", ListenPort: 8080}\n}"
}
7—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config"
}
8—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "DefaultConfig"
}
9—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- config/config.go go.mod go.sum'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'config.go' -g 'go.mod' -g 'Makefile'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'docs/**' -g 'Makefile' | head -80"
2✓ ✓
/bin/zsh -lc "cat config/config.go && \
rg -n '\\bPort\\b|ListenPort|DefaultConfig' --glob '*.go' --glob '"'!**/*_test.go'"'"
✓ ✓
/bin/zsh -lc 'cat config/config.go && \
git status --short'
3✓ ✓
/bin/zsh -lc 'rg --files'
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Config.Port"
}
4✓ ✓
/bin/zsh -lc 'cat config/config_test.go && \
cat go.mod'
semedit/semantic_rename (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
5✓ ✓
/bin/zsh -lc 'git diff -- config/config.go && \
git status --short'
✓ ✓
/bin/zsh -lc "rg -n 'Config\\{|ListenPort|Host:' --glob '*.go' --glob '"'!**/*_test.go'"'"
6—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "config/config.go",
  "placement": "public_end",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}"
}
7—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- config/config.go go.mod go.sum && \
cat config/config.go'

default prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-09-composite-refactor
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_09_composite_refactor.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package config

import (
	"archive/tar"
	"fmt"
)

type Config struct {
	Host string
	Port int
}

func (c *Config) Addr() string {
	return fmt.Sprintf("%s:%d", c.Host, c.Port)
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency31.15s35.70s+14.6%26.74s156.35s+484.7%
Process Start → First Event0.35s0.17s—0.38s0.19s—
First Event → First Tool Call6.14s4.67s—5.45s6.28s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%15+400.0%
Internal Tool Cycles47+75.0%620+233.3%
Initial Load / Discovery Turns23+50.0%440%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)47+75.0%620+233.3%
Output Tokens903666-26.2%70810943+1445.6%
Reasoning / Thinking Tokens124221+78.2%1186154+5115.3%
Total Input Tokens97365175850+80.6%1352501675870+1139.1%
Cached Input Tokens86784159744+84.1%1231361523456+1137.2%
Uncached Input Tokens1058116106+52.2%12114152414+1158.2%
Cost0.06100.0913+49.7%0.07140.9756+1266.5%
Cached vs Uncached Token Ratio8.20:19.92:1+20.9%10.16:110.00:1-1.7%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSFAIL—
Oracle L3: Clean BuildPASSPASS—PASSFAIL—
Oracle L4: Verification TestPASSPASS—PASSFAIL—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 19 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 18 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md' && \
rg -n '\\bConfig\\.Port\\b|\\.Port\\b|type Config|DefaultConfig' config --glob '*.go'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'README*'"
2✓ ✓
/bin/zsh -lc "cat config/config.go && \
cat config/config_test.go && \
rg -n '\\.Port|ListenPort|\\.Address\\(' --glob '*.go'"
✓ ✓
/bin/zsh -lc 'cat config/config.go && \
rg --files | head -80'
3✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os
import tempfile
import time

path = Path('config/config.go')
content = '''package config

import \"fmt\"

type Config struct {
\\tHost       string
\\tListenPort int
}

func DefaultConfig() *Config {
\\treturn &Config{
\\t\\tHost:       \"localhost\",
\\t\\tListenPort: 8080,
\\t}
}

func (c *Config) Addr() string {
\\treturn fmt.Sprintf(\"%s:%d\", c.Host, c.ListenPort)
}
'''
fd, temp_name = tempfile.mkstemp(dir=path.parent, prefix='.config.go.')
try:
    with os.fdopen(fd, 'w') as f:
        f.write(content)
        f.flush()
        os.fsync(f.fileno())
    old_mtime = path.stat().st_mtime_ns
    os.replace(temp_name, path)
    now_ns = time.time_ns()
    os.utime(path, ns=(now_ns, max(now_ns, old_mtime + 1)))
finally:
    if os.path.exists(temp_name):
        os.unlink(temp_name)
PY

git diff -- config/config.go"
✓ ✓
/bin/zsh -lc 'cat config/config_test.go && \
cat go.mod'
4? ✗
/bin/zsh -lc 'cat config/config.go && \
git status --short && \
git diff --no-index /dev/null config/config.go'
semedit/semantic_rename (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
5—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "config/config.go",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{Host: \"localhost\", ListenPort: 8080}\n}"
}
6—✓ ✓
/bin/zsh -lc 'git diff -- config/config.go && \
git status --short'
7—✓ ✓
/bin/zsh -lc 'cat config/config.go'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 19 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 18 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g 'go.sum' -g 'Makefile'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'go.mod' -g 'README*' | head -80"
2✓ ✓
/bin/zsh -lc "cat config/config.go && \
rg -n '\\bPort\\b|ListenPort|Config\\s*\\{' --glob '"'!**/*_test.go'"' ."
✓ ✓
/bin/zsh -lc 'rg --files | head -100'
3✓ ✓
/bin/zsh -lc "rg --files -g '"'!**/*_test.go'"'"
semedit/semantic_outline (✓ ✓)
{
  "path": "config/config.go"
}
4✓ ✓
/bin/zsh -lc "rg -n 'localhost|127\\.0\\.0\\.1|8080|default|Host:|Port:' ."
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "symbol": "Config"
}
5✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
p = Path('config/config.go')
data = '''package config

import \"fmt\"

type Config struct {
	Host       string
	ListenPort int
}

func DefaultConfig() *Config {
	return &Config{
		Host:       \"localhost\",
		ListenPort: 8080,
	}
}

func (c *Config) Addr() string {
	return fmt.Sprintf(\"%s:%d\", c.Host, c.ListenPort)
}
'''
fd, tmp = tempfile.mkstemp(dir=p.parent, prefix='.config.go.')
try:
    with os.fdopen(fd, 'w') as f:
        f.write(data)
        f.flush()
        os.fsync(f.fileno())
    os.replace(tmp, p)
finally:
    if os.path.exists(tmp):
        os.unlink(tmp)
PY
cat config/config.go"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "symbol": "Config.Addr"
}
6✓ ✓
/bin/zsh -lc 'git status --short'
✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md'
7—semedit/semantic_rename (✗ ?): This action was rejected due to unacceptable risk. Reason: Workspace-wide semantic rename may modify references in the protected test files, conflicting with the user's explicit instruction not to edit tests. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirmation. Report anything that remains blocked, clarify
{
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
8—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "config/config.go",
  "placement": "public_end",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{Host: \"localhost\", Port: 8080}\n}"
}
9—✓ ✓
/bin/zsh -lc 'git status --short -- config/config.go config/config_test.go go.mod go.sum && \
git diff -- config/config.go config/config_test.go go.mod go.sum'
10—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "symbol": "Config"
}
11—✓ ✓
/bin/zsh -lc 'git status --short -- config/config_test.go go.mod go.sum'
12—semedit/semantic_outline (✓ ✓)
{
  "path": "config/config.go"
}
13—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "symbol": "DefaultConfig"
}
14—semedit/semantic_organize_imports (✓ ✓)
{
  "file": "config/config.go"
}
15—semedit/semantic_outline (✓ ✓)
{
  "path": "config/config.go"
}
16—✓ ✓
/bin/zsh -lc 'git status --short -- config/config_test.go go.mod go.sum'
17—semedit/semantic_find_references (✓ ✗): load Go package example.com/test/config: example.com/test/config: /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-09-composite-refactor_1790456335837394000/config/config_test.go:10:7: c.ListenPort undefined (type *Config has no field or method ListenPort) example.com/test/config: /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-09-composite-refactor_1790456335837394000/config/config_test.go:11:5: c.ListenPor
{
  "symbol": "Config.Port"
}
18—✓ ✓
/bin/zsh -lc 'git status --short -- config/config_test.go go.mod go.sum'
19—semedit/semantic_rename (✗ ?): This action was rejected due to unacceptable risk. Reason: Workspace-wide semantic rename can modify config/config_test.go, directly violating the user's explicit prohibition on editing tests. Do not bypass this rejection through a workaround or indirect execution. Continue with a safer alternative, or carry out checks to prove that the action is authorized or low risk before trying again. Complete unaffected work without asking for confirmation. Report anything that remains blocked, clarify why it was bloc
{
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
20—? ✗
/bin/zsh -lc 'go test ./...'

default prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-09-composite-refactor
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_09_composite_refactor.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert helper function DefaultConfig() *Config returning default values, and clean up unused imports. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package config

import (
	"archive/tar"
	"fmt"
)

type Config struct {
	Host string
	Port int
}

func (c *Config) Addr() string {
	return fmt.Sprintf("%s:%d", c.Host, c.Port)
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency32.53s73.87s+127.1%46.45s61.25s+31.9%
Process Start → First Event0.15s0.18s—0.31s0.24s—
First Event → First Tool Call17.93s22.32s—14.66s15.10s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%110%
Internal Tool Cycles410+150.0%57+40.0%
Initial Load / Discovery Turns31-66.7%50-100.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)410+150.0%57+40.0%
Output Tokens497797+60.4%998720-27.9%
Reasoning / Thinking Tokens0186—183191+4.4%
Total Input Tokens96137262001+172.5%136444201672+47.8%
Cached Input Tokens85760239872+179.7%124160183040+47.4%
Uncached Input Tokens1037722129+113.3%1228418632+51.7%
Cost0.05360.1276+138.0%0.07650.1037+35.6%
Cached vs Uncached Token Ratio8.26:110.84:1+31.2%10.11:19.82:1-2.8%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'config.go' -g 'go.mod' -g 'Makefile' -g '"'!**/*_test.go'"'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'config.go' -g 'go.mod' -g 'go.sum' -g 'README*'"
2✓ ✓
/bin/zsh -lc "cat config/config.go && \
rg -n \"Config\\.Port|\\.Port\\b|ListenPort|DefaultConfig\" --glob '*.go'"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "symbol": "Config"
}
3✓ ✓
/bin/zsh -lc 'cat config/config_test.go && \
cat go.mod && \
ls -la'
✓ ✓
/bin/zsh -lc 'cat config/config.go'
4✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
p = Path('config/config.go')
s = p.read_text()
s = s.replace('\\t\"archive/tar\"\\n', '')
s = s.replace('\\tPort int\\n', '\\tListenPort int\\n')
s = s.replace('return fmt.Sprintf(\"%s:%d\", c.Host, c.Port)', 'return fmt.Sprintf(\"%s:%d\", c.Host, c.ListenPort)')
s = s.replace('type Config struct {\\n\\tHost string\\n\\tListenPort int\\n}\\n', 'type Config struct {\\n\\tHost       string\\n\\tListenPort int\\n}\\n\\nfunc DefaultConfig() *Config {\\n\\treturn &Config{\\n\\t\\tHost:       \"localhost\",\\n\\t\\tListenPort: 8080,\\n\\t}\\n}\\n')
p.write_text(s)
PY
gofmt -w config/config.go
cat config/config.go
git diff -- config/config.go go.mod go.sum"
✓ ✓
/bin/zsh -lc 'rg --files'
5—✓ ✓
/bin/zsh -lc 'cat config/config_test.go'
6—semedit/semantic_rename (✓ ✓)
{
  "file": "config/config.go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
7—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "config/config.go",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}"
}
8—semedit/semantic_organize_imports (✓ ✓)
{
  "file": "config/config.go"
}
9—semedit/semantic_outline (✓ ✓)
{
  "path": "config/config.go"
}
10—✓ ✓
/bin/zsh -lc 'git diff -- config/config.go config/config_test.go go.mod go.sum && \
go test ./...'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 21 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File config/config.go modified (15 lines -> 18 lines)
#VanillaSemedit MCP
1? ✗
/bin/zsh -lc "sed -n '1,240p' config/config.go && \
rg -n '\\.Port\\b|\\bPort\\s*:' --glob '*.go' . && \
rg -n 'Config\\{' --glob '*.go' ."
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'README*' -g '*.md' | head -80 && \
rg -n \"semantic_(lookup|inspect_symbol|rename|insert_function|verify)|DefaultConfig|type Config|\\.Port\\b\" . --glob '"'!go.sum'"' --glob '"'!go.mod'"'"
2✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' -g '"'!**/*_test.go'"'"
✓ ✓
/bin/zsh -lc 'cat config/config.go && \
cat AGENTS.override.md'
3✓ ✓
/bin/zsh -lc "rg -n 'localhost|127\\.0\\.0\\.1|8080|ListenPort|Host|config\\.' --glob '*.go' ."
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "symbol": "Config"
}
4✓ ✓
/bin/zsh -lc 'gofmt -w config/config.go && \
git diff -- config/config.go && \
git status --short'
✓ ✓
/bin/zsh -lc 'cat config/config_test.go'
5✓ ✓
/bin/zsh -lc "sed -n '1,160p' config/config.go; git status --short -- config/config.go; git ls-files -- config/config.go"
semedit/semantic_rename (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port",
  "to": "ListenPort"
}
6—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "config/config.go",
  "placement": "public_end",
  "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{Host: \"localhost\", ListenPort: 8080}\n}"
}
7—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- config/config.go config/config_test.go go.mod go.sum && \
cat config/config.go'

Test case: task-11-mixed-sink-api-migration

Target: codex/gpt-6-luna/medium

default prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency159.17s192.58s+21.0%122.97s219.60s+78.6%
Process Start → First Event0.14s0.12s—0.17s0.15s—
First Event → First Tool Call12.31s9.76s—5.59s5.60s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns550%550%
Internal Tool Cycles1223+91.7%1335+169.2%
Initial Load / Discovery Turns220%35+66.7%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)1223+91.7%1335+169.2%
Output Tokens1333412914-3.1%1532619670+28.3%
Reasoning / Thinking Tokens55065667+2.9%60309763+61.9%
Total Input Tokens17944612725466+51.9%16351174597826+181.2%
Cached Input Tokens16727042572800+53.8%15006724359424+190.5%
Uncached Input Tokens121757152666+25.4%134445238402+77.3%
Cost0.95811.2571+31.2%0.97822.0538+109.9%
Cached vs Uncached Token Ratio13.74:116.85:1+22.7%11.16:118.29:1+63.8%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestFAILFAIL—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 21 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g '*.go' -g 'Makefile' -g 'README*' | head -200 && \
git status --short"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'README*' -g 'go.mod' -g 'go.sum' -g '*.go' | head -200"
2✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
sed -n '1,240p' service/dispatch.go && \
sed -n '1,220p' audit/event.go && \
sed -n '1,240p' audit/normalize.go && \
sed -n '1,220p' audit/sink.go && \
sed -n '1,200p' audit/summary.go"
✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
cat go.mod && \
cat audit/event.go && \
cat audit/normalize.go && \
cat audit/sink.go && \
cat service/dispatch.go && \
cat metrics/writer.go && \
cat legacy/export.go'
3✓ ✓
/bin/zsh -lc "cat audit/event_test.go audit/summary_test.go service/dispatch_test.go && \
sed -n '1,200p' legacy/export.go && \
sed -n '1,200p' metrics/writer.go && \
rg -n \"probe|health|normalize|DispatchAll|Record\\(\" . -g '"'!*.sum'"' -g '"'!*.mod'"'"
✓ ✓
/bin/zsh -lc "cat service/dispatch_test.go audit/event_test.go audit/summary.go audit/summary_test.go && \
rg --files -g 'Makefile' -g 'README*' -g '*'"
4✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./... && \
git status --short && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go'
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat cmd/migrate/main.go && \
ls -la'
5✓ ✓
/bin/zsh -lc 'git diff --stat && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go && \
git status --short --untracked-files=all'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
6✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat service/dispatch.go && \
git check-ignore -v audit/normalize.go service/dispatch.go audit/summary.go; git ls-files -v audit/normalize.go service/dispatch.go audit/summary.go'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\nif event.Kind == \"probe\" {\n\treturn nil\n}\nreturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
7? ✗
/bin/zsh -lc "pwd && \
cat audit/normalize.go && \
cat service/dispatch.go && \
cat audit/summary.go && \
git status --short --untracked-files=no && \
rg -n 'go.mod|go.sum' .gitignore"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "writeEvent := audit.Sink.Write\nfor _, event := range events {\n\tevent.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\tcontinue\n\t}\n\tif err := writeEvent(sink, event); err != nil {\n\t\treturn err\n\t}\n}\nreturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
8✓ ✓
/bin/zsh -lc 'rg --files && \
cat audit/sink.go && \
cat service/dispatch.go && \
cat audit/normalize.go && \
cat audit/event.go && \
cat audit/summary.go'
? ✗
/bin/zsh -lc "git diff -- && \
git status --short && \
rg 'make check|go test' ."
9✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat cmd/migrate/main.go && \
cat go.mod'
✓ ✓
/bin/zsh -lc 'cat audit/normalize.go service/dispatch.go && \
git rev-parse --show-toplevel && \
git status --short --untracked-files=no'
10✓ ✓
/bin/zsh -lc "cat audit/normalize.go audit/summary.go && \
git status --short --untracked-files=no && \
rg -n 'probe|Classify' audit"
semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
11✓ ✓
/bin/zsh -lc 'cat service/dispatch.go && \
cat audit/normalize.go && \
cat audit/summary.go && \
git status --short -- go.mod go.sum'
✓ ✓
/bin/zsh -lc 'go test ./...'
12✓ ✓
/bin/zsh -lc 'gofmt -w service/dispatch.go && \
go test ./...'
✓ ✓
/bin/zsh -lc 'cat go.mod; if test -f go.sum; then cat go.sum; fi; cat audit/normalize.go; cat audit/event.go; cat service/dispatch.go; git status --short -- go.mod go.sum audit service'
13—✓ ✓
/bin/zsh -lc "cat docs/audit-sinks.md audit/sink.go audit/summary.go legacy/export.go metrics/writer.go; rg 'Write\\(|DispatchAll|Record\\(' --glob '*.go' ."
14—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
15—✓ ✓
/bin/zsh -lc 'go test ./...'
16—✓ ✓
/bin/zsh -lc 'cat go.mod; if test -f go.sum; then cat go.sum; fi; cat audit/summary.go audit/normalize.go'
17—semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\ncase \"health\", \"probe\":\n\treturn \"control\"\ndefault:\n\treturn \"data\"\n}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
18—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
19—✓ ✓
/bin/zsh -lc 'go test ./...'
20—✓ ✓
/bin/zsh -lc 'cat go.mod; if test -f go.sum; then cat go.sum; fi; cat service/dispatch.go'
21—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
22—✓ ✓
/bin/zsh -lc 'go test ./...'
23—✓ ✓
/bin/zsh -lc 'cat go.mod; if test -f go.sum; then cat go.sum; fi; cat audit/normalize.go audit/summary.go service/dispatch.go; go test ./...'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 11 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS*' -g '*.go' -g 'Makefile' | head -200"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' -g 'go.mod' -g 'AGENTS.md' | head -100"
2✓ ✓
/bin/zsh -lc "ls -la && \
cat AGENTS.override.md && \
rg -n \"audit|probe|persist|Deliver|Normalize|Sink\" --glob '*.go' ."
✓ ✓
/bin/zsh -lc "rg -n \"Audit|audit|Probe|Kind|persist|Persist\" --glob '*.go' --glob '"'!**/*_test.go'"'"
3✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/event.go audit/sink.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go journal/file.go audit/summary.go metrics/writer.go legacy/export.go && \
rg --files docs .scratch'
✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/event.go audit/sink.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go'
4✓ ✓
/bin/zsh -lc "cat docs/audit-sinks.md && \
rg -n \"Commit|Write\\(|Classify|Normalize|Dispatch\" --glob '*.go' --glob '"'!**/*_test.go'"' . && \
find . -maxdepth 2 -type f | sort"
✓ ✓
/bin/zsh -lc 'cat audit/event_test.go audit/summary_test.go service/dispatch_test.go && \
cat audit/summary.go && \
cat legacy/export.go'
5✓ ✓
/bin/zsh -lc "cat > audit/normalize.go <<'EOF'
// Package audit normalizes event kinds at the persistence boundary.
package audit

import (
	\"strings\"
)

func Normalize(event Event) Event {
	event.Kind = strings.ToLower(strings.TrimSpace(event.Kind))
	return event
}
EOF
cat > /tmp/nonexistent-placeholder <<'EOF'
EOF"
semedit/semantic_lookup (✓ ✗): resolve Normalize: symbol not found
{
  "language": "go",
  "symbol": "Normalize"
}
6✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
changes = {
'audit/sink.go': [('func (m *MemorySink) Write(event Event) error {\\n\\tm.Events', 'func (m *MemorySink) Write(event Event) error {\\n\\tevent = Normalize(event)\\n\\tm.Events')],
'audit/fanout.go': [('func (f FanoutSink) Write(event Event) error {\\n\\tif err :=', 'func (f FanoutSink) Write(event Event) error {\\n\\tevent = Normalize(event)\\n\\tif err :=')],
'audit/transaction.go': [('for _, event := range b.queue {\\n\\t\\tif err :=', 'for _, event := range b.queue {\\n\\t\\tevent = Normalize(event)\\n\\t\\tif err :=')],
'service/dispatch.go': [('func (d *Dispatcher) Record(event audit.Event) error {\\n\\treturn', 'func (d *Dispatcher) Record(event audit.Event) error {\\n\\tevent = audit.Normalize(event)\\n\\treturn'), ('for _, event := range events {\\n\\t\\tif err := writeEvent', 'for _, event := range events {\\n\\t\\tevent = audit.Normalize(event)\\n\\t\\tif err := writeEvent')],
'service/replay.go': [('for _, event := range events {\\n\\t\\tif err := writeEvent', 'for _, event := range events {\\n\\t\\tevent = audit.Normalize(event)\\n\\t\\tif err := writeEvent')],
'journal/file.go': [('func (j *JournalSink) Write(event audit.Event) error {\\n\\tj.Entries', 'func (j *JournalSink) Write(event audit.Event) error {\\n\\tevent = audit.Normalize(event)\\n\\tj.Entries')],
'audit/summary.go': [('switch kind {\\n\\tcase \"health\":', 'switch Normalize(Event{Kind: kind}).Kind {\\n\\tcase \"health\", \"probe\":')],
}
for name, pairs in changes.items():
    path = Path(name)
    data = path.read_text()
    for old, new in pairs:
        if old not in data:
            raise SystemExit(f'pattern missing in {name}: {old"'!r}'"')
        data = data.replace(old, new, 1)
    fd, tmp = tempfile.mkstemp(prefix=f'.{path.name}.', dir=path.parent)
    try:
        with os.fdopen(fd, 'w') as f:
            f.write(data)
            f.flush()
            os.fsync(f.fileno())
        os.replace(tmp, path)
    except BaseException:
        os.unlink(tmp)
        raise
PY
gofmt -w audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go audit/summary.go service/dispatch.go service/replay.go journal/file.go
git status --short && \
git diff --"
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Write"
}
7✓ ✓
/bin/zsh -lc "cat audit/normalize.go audit/summary.go && \
sed -n '18,34p' service/dispatch.go && \
git rev-parse --show-toplevel && \
git status --short --untracked-files=all | head -80 && \
ls -la .scratch && \
"
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Classify"
}
8✓ ✓
/bin/zsh -lc 'go test ./...'
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Commit"
}
9✓ ✓
/bin/zsh -lc "cat audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go audit/summary.go service/dispatch.go service/replay.go journal/file.go && \
stat -f '%N %m' go.mod 2>/dev/null || \
stat go.mod && \
find . -maxdepth 1 -name 'go.sum' -print && \
find . -maxdepth 2 -iname '*boundary*' -o -iname '*audit*' | sort"
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "DispatchAll"
}
10✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go audit/event.go && \
cat audit/summary_test.go && \
git -C . rev-parse --show-toplevel'
semedit/semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Replay"
}
11✓ ✓
/bin/zsh -lc 'cat service/dispatch.go service/replay.go && \
cat audit/normalize.go && \
'
✓ ✓
/bin/zsh -lc 'ls -a'
12✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go && \
'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
13✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
changes = {
'audit/normalize.go': '''// Package audit normalizes event kinds at the persistence boundary.
package audit

import \"strings\"

func NormalizeKind(kind string) string {
	return strings.ToLower(strings.TrimSpace(kind))
}

func Normalize(event Event) Event {
	event.Kind = NormalizeKind(event.Kind)
	return event
}
''',
'audit/summary.go': '''// Package audit groups delivery kinds for operational reporting.
package audit

func Classify(kind string) string {
	switch NormalizeKind(kind) {
	case \"health\", \"probe\":
		return \"control\"
	default:
		return \"data\"
	}
}
''',
'service/dispatch.go': None,
}
for filename, content in changes.items():
    path = Path(filename)
    if content is None:
        content = path.read_text()
        old = '''func (d *Dispatcher) Record(event audit.Event) error {
	event = audit.Normalize(event)
	return d.sink.Write(event)
}'''
        new = '''func (d *Dispatcher) Record(event audit.Event) error {
	event.Kind = audit.NormalizeKind(event.Kind)
	return d.sink.Write(event)
}'''
        if old not in content:
            raise SystemExit('Dispatcher.Record pattern missing')
        content = content.replace(old, new, 1)
    fd, tmp = tempfile.mkstemp(prefix=f'.{path.name}.', dir=path.parent)
    try:
        with os.fdopen(fd, 'w') as stream:
            stream.write(content)
            stream.flush()
            os.fsync(stream.fileno())
        os.replace(tmp, path)
    except BaseException:
        os.unlink(tmp)
        raise
PY
gofmt -w audit/normalize.go audit/summary.go service/dispatch.go
go test ./..."
semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\ncase \"health\", \"probe\":\n\treturn \"control\"\ndefault:\n\treturn \"data\"\n}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
14—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\nif audit.Classify(event.Kind) == \"control\" \u0026\u0026 event.Kind == \"probe\" {\n\treturn nil\n}\nreturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
15—semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\tevent.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\tcontinue\n\t}\n\tif err := sink.Write(event); err != nil {\n\t\treturn err\n\t}\n}\nreturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
16—semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\tevent.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\tcontinue\n\t}\n\tif err := sink.Write(event); err != nil {\n\t\treturn err\n\t}\n}\nreturn nil",
  "file": "service/replay.go",
  "symbol": "Replay"
}
17—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\nif audit.Classify(event.Kind) == \"control\" {\n\treturn nil\n}\nreturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
18—semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\tevent.Kind = audit.NormalizeKind(event.Kind)\n\tif audit.Classify(event.Kind) == \"control\" {\n\t\tcontinue\n\t}\n\tif err := sink.Write(event); err != nil {\n\t\treturn err\n\t}\n}\nreturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
19—semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\tevent.Kind = audit.NormalizeKind(event.Kind)\n\tif audit.Classify(event.Kind) == \"control\" {\n\t\tcontinue\n\t}\n\tif err := sink.Write(event); err != nil {\n\t\treturn err\n\t}\n}\nreturn nil",
  "file": "service/replay.go",
  "symbol": "Replay"
}
20—✓ ✓
/bin/zsh -lc 'go test ./... && \
git status --short && \
git diff -- go.mod go.sum'
21—✓ ✓
/bin/zsh -lc 'git diff -- audit/normalize.go audit/summary.go service/dispatch.go service/replay.go && \
git status --short --untracked-files=no'
22—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go service/replay.go && \
git rev-parse --show-toplevel && \
git status --short'
23—✓ ✓
/bin/zsh -lc "rg --files docs audit service journal metrics legacy && \
rg -n \"normaliz|delivery boundary|control|probe\" docs audit service journal metrics legacy --glob '"'!**/*_test.go'"'"
24—✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md metrics/writer.go journal/file.go audit/sink.go audit/fanout.go audit/transaction.go'
25—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "audit"
}
26—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff -- go.mod go.sum'
27—✓ ✓
/bin/zsh -lc 'cat audit/summary.go'
28—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go'
29—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "language": "go",
  "symbol": "Classify"
}
30—✓ ✓
/bin/zsh -lc 'go test ./audit && \
git diff -- go.mod go.sum'
31—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "language": "go",
  "symbol": "Dispatcher.Record"
}
32—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "service"
}
33—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go && \
git diff -- go.mod go.sum'
34—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\nreturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
35—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff -- go.mod go.sum'

default prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency79.87s251.69s+215.1%—234.58s—
Process Start → First Event0.12s0.13s——0.29s—
First Event → First Tool Call12.78s23.67s——5.79s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns15+400.0%—5—
Internal Tool Cycles726+271.4%—34—
Initial Load / Discovery Turns24+100.0%—4—
MCP Discovery / Schema Turns00——0—
Tool Invocations (per run)726+271.4%—34—
Output Tokens155711378+630.8%—20872—
Reasoning / Thinking Tokens5274852+820.7%—10203—
Total Input Tokens1867482336902+1151.4%—3221251—
Cached Input Tokens1717762106624+1126.4%—3036672—
Uncached Input Tokens14972230278+1438.1%—184579—
Cost0.10641.3052+1126.4%—1.6091—
Cached vs Uncached Token Ratio11.47:19.15:1-20.3%—16.45:1—
Oracle L1: Mutation PolicyPASSPASS——PASS—
Oracle L2: AST InvariantsPASSPASS——PASS—
Oracle L3: Clean BuildPASSPASS——PASS—
Oracle L4: Verification TestPASSFAIL——FAIL—
Semantic Tool Invocation Verified (per run)N/APASS——PASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 13 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g '*.go' -g 'Makefile' | head -200"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' -g 'Makefile' | head -100"
2✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
git status --short && \
rg -n \"probe|event kind|Kind|deliver|persist|Sink|Normalize|normalize\" . --glob '"'!go.mod'"' --glob '"'!go.sum'"'"
✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
rg -n \"probe|Probe|event kind|Kind|deliver|persist|Normalize|normalize\" . -g '*.go' -g '*.md'"
3✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md service/dispatch.go audit/normalize.go audit/event.go audit/sink.go audit/summary.go metrics/writer.go legacy/export.go && \
git diff -- go.mod go.sum && \
git status --short -- service audit metrics legacy'
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat service/dispatch.go && \
cat audit/normalize.go && \
cat audit/event.go && \
cat audit/sink.go && \
cat audit/summary.go && \
rg --files'
4✓ ✓
/bin/zsh -lc 'rg --files'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
5✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./... && \
git diff --check && \
git status --short -- go.mod go.sum audit/normalize.go audit/summary.go service/dispatch.go && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
6✓ ✓
/bin/zsh -lc 'git status --short && \
git diff --stat && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go && \
'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tevent.Kind = audit.NormalizeKind(event.Kind)\n\t\tif event.Kind == \"probe\" {\n\t\t\tcontinue\n\t\t}\n\t\tif err := sink.Write(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
7✓ ✓
/bin/zsh -lc 'cat audit/normalize.go service/dispatch.go audit/summary.go'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
8—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
9—✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum && \
git status --short && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go'
10—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat service/dispatch.go && \
cat audit/summary.go && \
git status --short'
11—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- go.mod go.sum && \
cat audit/normalize.go && \
cat audit/event.go && \
cat audit/sink.go && \
cat service/dispatch.go'
12—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = NormalizeKind(event.Kind)\n\tm.Events = append(m.Events, event)\n\treturn nil",
  "file": "audit/sink.go",
  "symbol": "(*MemorySink).Write"
}
13—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
14—✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum && \
git status --short'
15—✓ ✓
/bin/zsh -lc 'cat audit/summary.go && \
cat audit/normalize.go && \
git diff -- go.mod go.sum'
16—✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum'
17—semedit/semantic_verify (✓ ✓)
{
  "path": "audit"
}
18—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Dispatcher.Record"
}
19—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
20—semedit/semantic_verify (✓ ✓)
{
  "path": "service"
}
21—✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum'
22—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/normalize.go",
  "symbol": "NormalizeKind"
}
23—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify"
}
24—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Dispatcher.Record"
}
25—✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum'
26—✓ ✓
/bin/zsh -lc 'go test ./...'
Baseline vs Semedit Agentic in Large Context
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
#VanillaSemedit MCP
1not published✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'PROJECT_BRIEF.md' -g '*.go' -g 'go.mod' -g 'go.sum' -g 'README*' | head -100"
2not published✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
cat PROJECT_BRIEF.md 2>/dev/null; rg -n \"Audit|audit|probe|Sink|Deliver|Normalize\" --glob '*.go'"
3not publishedsemedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/sink.go",
  "symbol": "Write"
}
4not publishedsemedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Record"
}
5not publishedsemedit/semantic_inspect_symbol (✓ ✗): resolve Normalize in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-11-mixed-sink-api-migration_1790456648750334000/audit/normalize.go: symbol not found
{
  "file": "audit/normalize.go",
  "symbol": "Normalize"
}
6not publishedsemedit/semantic_inspect_symbol (✓ ✗): resolve Write in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-11-mixed-sink-api-migration_1790456648750334000/audit/transaction.go: symbol not found
{
  "file": "audit/transaction.go",
  "symbol": "Write"
}
7not published✓ ✓
/bin/zsh -lc 'cat docs/SUBOPTIMAL_TOOLS.md 2>/dev/null; cat audit/normalize.go; cat audit/event.go; cat audit/transaction.go; cat service/dispatch.go; cat audit/fanout.go; cat audit/summary.go'
8not published✓ ✓
/bin/zsh -lc "mkdir -p docs && \
cat > docs/SUBOPTIMAL_TOOLS.md <<'EOF'
# Suboptimal semantic tool behavior

- Tool: "'`semantic_inspect_symbol`
- Target: `audit/normalize.go`
- Observed result: lookup for `Normalize` failed because the placeholder file has no declaration yet.
- Workaround: inspect the file contents and add the missing normalization helper with a semantic declaration tool.
- Root cause: the fixture contains an intentionally empty implementation point, not an existing symbol.
EOF
cat audit/sink.go service/replay.go journal/file.go audit/normalize.go'
9not publishedsemedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "source": "func NormalizeEvent(event Event) (Event, bool) {\n\tevent.Kind = strings.ToLower(strings.TrimSpace(event.Kind))\n\treturn event, event.Kind != \"probe\"\n}"
}
10not publishedsemedit/semantic_replace_body (✓ ✗): replace_body Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-default_task-11-mixed-sink-api-migration_1790456648750334000/service/dispatch.go: symbol not found
{
  "body": "event, deliver := audit.NormalizeEvent(event)\n\tif !deliver {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "Record"
}
11not publishedsemedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tevent, deliver := audit.NormalizeEvent(event)\n\t\tif !deliver {\n\t\t\tcontinue\n\t\t}\n\t\tif err := sink.Write(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
12not publishedsemedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tevent, deliver := audit.NormalizeEvent(event)\n\t\tif !deliver {\n\t\t\tcontinue\n\t\t}\n\t\tif err := sink.Write(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/replay.go",
  "symbol": "Replay"
}
13not published✓ ✓
/bin/zsh -lc "cat >> docs/SUBOPTIMAL_TOOLS.md <<'EOF'
- Tool: "'`semantic_replace_body`
- Target: `service/dispatch.go`
- Observed result: lookup for `Record` failed because the method requires its receiver-qualified identifier.
- Workaround: retry using `(*Dispatcher).Record`.
- Root cause: method body replacement resolves methods by receiver-qualified name.
EOF'
14not publishedsemedit/semantic_replace_body (✓ ✓)
{
  "body": "event, deliver := audit.NormalizeEvent(event)\n\tif !deliver {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
15not publishedsemedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range b.queue {\n\t\tevent, deliver := NormalizeEvent(event)\n\t\tif !deliver {\n\t\t\tcontinue\n\t\t}\n\t\tif err := b.Sink.Write(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "audit/transaction.go",
  "symbol": "(*BufferedSink).Commit"
}
16not publishedsemedit/semantic_replace_body (✓ ✓)
{
  "body": "kind = strings.ToLower(strings.TrimSpace(kind))\n\tswitch kind {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
17not publishedsemedit/semantic_organize_imports (✓ ✓)
{
  "add": [
    "strings"
  ],
  "file": "audit/summary.go"
}
18not published✓ ✓
/bin/zsh -lc "git status --short && \
git diff --stat && \
git diff -- go.mod go.sum && \
git diff --name-only -- '*_test.go'"
19not published✓ ✓
/bin/zsh -lc 'go test ./...'
20not published✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go service/replay.go audit/transaction.go docs/SUBOPTIMAL_TOOLS.md; git diff --stat -- .; git status --short'
21not published✓ ✓
/bin/zsh -lc 'git status --short; git diff -- go.mod go.sum; cat audit/normalize.go audit/event.go audit/sink.go audit/summary.go service/dispatch.go service/replay.go; cat go.mod'
22not published✓ ✓
/bin/zsh -lc 'go test ./...'
23not published✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum; cat audit/normalize.go audit/summary.go; cat audit/event.go'
24not publishedsemedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "audit/summary.go"
}
25not published✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum'
26not publishedsemedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
27not published✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum; cat audit/normalize.go audit/summary.go service/dispatch.go'
28not publishedsemedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "placement": "public_start",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
29not publishedsemedit/semantic_replace_body (✓ ✓)
{
  "body": "kind = NormalizeKind(kind)\n\tswitch kind {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
30not publishedsemedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = NormalizeKind(event.Kind)\n\treturn event, event.Kind != \"probe\"",
  "file": "audit/normalize.go",
  "symbol": "NormalizeEvent"
}
31not publishedsemedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
32not publishedsemedit/semantic_organize_imports (✓ ✓)
{
  "file": "audit/summary.go",
  "remove": [
    "strings"
  ]
}
33not published✓ ✓
/bin/zsh -lc 'go test ./...'
34not published✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum'

default prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency137.81s157.48s+14.3%182.10s246.82s+35.5%
Process Start → First Event0.12s0.15s—0.24s0.12s—
First Event → First Tool Call9.69s4.42s—13.29s4.65s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns550%550%
Internal Tool Cycles1322+69.2%1540+166.7%
Initial Load / Discovery Turns31-66.7%32-33.3%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)1322+69.2%1540+166.7%
Output Tokens1102412439+12.8%1969120511+4.2%
Reasoning / Thinking Tokens33865984+76.7%71858584+19.5%
Total Input Tokens14467372369302+63.8%21393054329697+102.4%
Cached Input Tokens13486082216960+64.4%19978244104960+105.5%
Uncached Input Tokens98129152342+55.2%141481224737+58.8%
Cost0.76261.1654+52.8%1.18911.9518+64.1%
Cached vs Uncached Token Ratio13.74:114.55:1+5.9%14.12:118.27:1+29.4%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestFAILFAIL—PASSFAIL—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g '*.go' -g 'Makefile' -g 'README.md'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'README*' -g 'go.mod' -g '*.go' | head -100"
2✓ ✓
/bin/zsh -lc 'rg --files'
✓ ✓
/bin/zsh -lc "rg --files | sort && \
git status --short && \
sed -n '1,200p' audit/normalize.go && \
sed -n '1,240p' audit/event.go && \
sed -n '1,240p' audit/sink.go && \
sed -n '1,240p' service/dispatch.go"
3✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat service/dispatch.go && \
cat audit/event.go && \
cat audit/normalize.go && \
cat audit/sink.go'
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat go.mod && \
cat service/dispatch_test.go && \
cat audit/event_test.go'
4✓ ✓
/bin/zsh -lc 'cat service/dispatch_test.go audit/event_test.go audit/summary.go metrics/writer.go legacy/export.go cmd/migrate/main.go'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
5✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
ls -la'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
6✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile

updates = {
    Path('audit/normalize.go'): '''// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit

import \"strings\"

func NormalizeKind(kind string) string {
\\treturn strings.ToLower(strings.TrimSpace(kind))
}
''',
    Path('service/dispatch.go'): '''// Package service coordinates application work with the configured audit sink.
package service

import \"example.com/auditapp/audit\"

type Dispatcher struct {
\\tsink audit.Sink
}

func NewDispatcher(sink audit.Sink) *Dispatcher {
\\treturn &Dispatcher{sink: sink}
}

func (d *Dispatcher) Record(event audit.Event) error {
\\treturn deliver(d.sink, event)
}

func DispatchAll(sink audit.Sink, events []audit.Event) error {
\\tfor _, event := range events {
\\t\\tif err := deliver(sink, event); err "'!= nil {
'"\\t\\t\\treturn err
\\t\\t}
\\t}
\\treturn nil
}

func deliver(sink audit.Sink, event audit.Event) error {
\\tevent.Kind = audit.NormalizeKind(event.Kind)
\\tif event.Kind == \"probe\" {
\\t\\treturn nil
\\t}
\\treturn sink.Write(event)
}
''',
}
for path, content in updates.items():
    fd, tmp = tempfile.mkstemp(dir=path.parent, prefix=f'.{path.name}.')
    try:
        with os.fdopen(fd, 'w') as f:
            f.write(content)
            f.flush()
            os.fsync(f.fileno())
        os.replace(tmp, path)
    except BaseException:
        try: os.unlink(tmp)
        except FileNotFoundError: pass
        raise
PY
gofmt -w audit/normalize.go service/dispatch.go
git diff -- audit/normalize.go service/dispatch.go"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tevent.Kind = audit.NormalizeKind(event.Kind)\n\t\tif event.Kind == \"probe\" {\n\t\t\tcontinue\n\t\t}\n\t\tif err := sink.Write(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
7✓ ✓
/bin/zsh -lc 'git status --short && \
cat audit/normalize.go && \
cat service/dispatch.go'
? ✗
/bin/zsh -lc 'git status --short && \
git diff -- audit/normalize.go service/dispatch.go && \
(make check)'
8✓ ✓
/bin/zsh -lc 'go test ./...'
✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat service/dispatch.go && \
git status --short && \
git diff --stat && \
go test ./...'
9✓ ✓
/bin/zsh -lc 'rg --files && \
cat docs/audit-sinks.md && \
cat service/dispatch.go && \
cat audit/normalize.go && \
cat audit/sink.go && \
git status --short'
semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
10✓ ✓
/bin/zsh -lc 'cat audit/summary.go && \
cat audit/normalize.go && \
cat audit/summary_test.go'
✓ ✓
/bin/zsh -lc "git status --short && \
git diff -- go.mod go.sum && \
rg -n \"Normalize|probe|Write\\(|Record\\(|DispatchAll|Sink\" audit service metrics legacy docs"
11✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
path = Path('audit/summary.go')
content = '''// Package audit groups delivery kinds for operational reporting.
package audit

func Classify(kind string) string {
\\tswitch NormalizeKind(kind) {
\\tcase \"health\", \"probe\":
\\t\\treturn \"control\"
\\tdefault:
\\t\\treturn \"data\"
\\t}
}
'''
fd, tmp = tempfile.mkstemp(dir=path.parent, prefix=f'.{path.name}.')
try:
    with os.fdopen(fd, 'w') as f:
        f.write(content)
        f.flush()
        os.fsync(f.fileno())
    os.replace(tmp, path)
except BaseException:
    try: os.unlink(tmp)
    except FileNotFoundError: pass
    raise
PY
gofmt -w audit/summary.go
go test ./..."
semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
12✓ ✓
/bin/zsh -lc 'cat service/dispatch.go && \
cat audit/normalize.go && \
git status --short && \
git diff -- go.mod go.sum'
✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --exit-code -- go.mod go.sum'
13✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile
path = Path('service/dispatch.go')
content = '''// Package service coordinates application work with the configured audit sink.
package service

import \"example.com/auditapp/audit\"

type Dispatcher struct {
\\tsink audit.Sink
}

func NewDispatcher(sink audit.Sink) *Dispatcher {
\\treturn &Dispatcher{sink: sink}
}

func (d *Dispatcher) Record(event audit.Event) error {
\\tevent.Kind = audit.NormalizeKind(event.Kind)
\\tif event.Kind == \"probe\" {
\\t\\treturn nil
\\t}
\\treturn d.sink.Write(event)
}

func DispatchAll(sink audit.Sink, events []audit.Event) error {
\\tfor _, event := range events {
\\t\\tif err := deliver(sink, event); err "'!= nil {
'"\\t\\t\\treturn err
\\t\\t}
\\t}
\\treturn nil
}

func deliver(sink audit.Sink, event audit.Event) error {
\\tevent.Kind = audit.NormalizeKind(event.Kind)
\\tif event.Kind == \"probe\" {
\\t\\treturn nil
\\t}
\\treturn sink.Write(event)
}
'''
fd, tmp = tempfile.mkstemp(dir=path.parent, prefix=f'.{path.name}.')
try:
    with os.fdopen(fd, 'w') as f:
        f.write(content)
        f.flush()
        os.fsync(f.fileno())
    os.replace(tmp, path)
except BaseException:
    try: os.unlink(tmp)
    except FileNotFoundError: pass
    raise
PY
gofmt -w service/dispatch.go
go test ./..."
✓ ✓
/bin/zsh -lc 'cat audit/summary.go && \
cat audit/event.go && \
cat audit/normalize.go && \
cat metrics/writer.go && \
cat legacy/export.go && \
cat docs/audit-sinks.md && \
git diff -- go.mod go.sum'
14—semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\ncase \"health\", \"probe\":\n\treturn \"control\"\ndefault:\n\treturn \"data\"\n}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
15—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
16—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --exit-code -- go.mod go.sum'
17—✓ ✓
/bin/zsh -lc 'cat service/dispatch.go && \
git diff -- go.mod go.sum'
18—semedit/semantic_verify (✓ ✓)
{
  "path": "service"
}
19—✓ ✓
/bin/zsh -lc 'go test ./service ./audit && \
git diff --exit-code -- go.mod go.sum'
20—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat audit/summary.go && \
cat service/dispatch.go && \
git diff -- go.mod go.sum'
21—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
22—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --exit-code -- go.mod go.sum'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 15 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' -g 'Makefile' -g 'go.mod' -g 'go.sum' | head -160"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'README*' -g 'go.mod' -g '*.go' -g 'Makefile' | head -100"
2✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
cat go.mod && \
sed -n '1,240p' audit/event.go && \
sed -n '1,240p' audit/normalize.go && \
sed -n '1,240p' audit/fanout.go && \
sed -n '1,240p' audit/sink.go && \
sed -n '1,240p' service/dispatch.go"
✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
cat PROJECT_BRIEF.md 2>/dev/null; rg -n "audit|probe|deliver|persist|normalize"'
3✓ ✓
/bin/zsh -lc "sed -n '1,240p' audit/transaction.go && \
sed -n '1,240p' audit/summary.go && \
sed -n '1,240p' audit/event_test.go && \
sed -n '1,260p' audit/summary_test.go && \
sed -n '1,260p' service/dispatch_test.go && \
rg -n \"Probe|probe|Kind|Write\\(\" --glob '*.go'"
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md; cat audit/event.go audit/sink.go audit/normalize.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go journal/file.go'
4✓ ✓
/bin/zsh -lc 'cat journal/file.go && \
cat legacy/export.go && \
cat metrics/metrics.go && \
cat metrics/writer.go && \
cat audit/transaction.go'
✓ ✓
/bin/zsh -lc 'cat audit/summary.go audit/transaction.go audit/fanout.go metrics/metrics.go metrics/writer.go legacy/export.go service/dispatch_test.go'
5✓ ✓
/bin/zsh -lc "rg -n \"DispatchAll|BufferedSink|FanoutSink|NewDispatcher|JournalSink|Normalize\" --glob '*.go' && \
find . -maxdepth 2 -type f | sort"
✓ ✓
/bin/zsh -lc 'cat go.mod; rg --files'
6✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat service/replay.go'
semedit/semantic_insert_function (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "source": "func Prepare(event Event) (Event, bool) {\n\tevent.Kind = strings.ToLower(strings.TrimSpace(event.Kind))\n\treturn event, event.Kind != \"probe\"\n}"
}
7✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/sink.go audit/summary.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go && \
go test ./...'
✓ ✓
/bin/zsh -lc 'cat audit/normalize.go'
8✓ ✓
/bin/zsh -lc "git status --short && \
git diff --stat && \
git diff --check && \
git diff -- audit/normalize.go audit/sink.go audit/summary.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go && \
git diff -- go.mod go.sum '*_test.go'"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event, deliver := audit.Prepare(event)\n\tif !deliver {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
9✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
git rev-parse --show-toplevel && \
git status --short --untracked-files=all | head -50'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tevent, deliver := audit.Prepare(event)\n\t\tif !deliver {\n\t\t\tcontinue\n\t\t}\n\t\tif err := sink.Write(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
10✓ ✓
/bin/zsh -lc "git diff -- audit service journal metrics legacy go.mod go.sum && \
git status --short -- audit service journal metrics legacy go.mod go.sum && \
sed -n '1,120p' audit/normalize.go && \
sed -n '1,140p' audit/sink.go && \
sed -n '1,100p' service/dispatch.go && \
sed -n '1,100p' service/replay.go && \
sed -n '1,80p' audit/summary.go"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tevent, deliver := audit.Prepare(event)\n\t\tif !deliver {\n\t\t\tcontinue\n\t\t}\n\t\tif err := sink.Write(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/replay.go",
  "symbol": "Replay"
}
11✓ ✓
/bin/zsh -lc "rg -n \"Normalize|Deliver|Kind\" audit service journal cmd --glob '*.go' && \
cat go.mod && \
if [ -f go.sum ]; then cat go.sum; else echo 'go.sum absent'; fi && \
go test ./..."
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event, deliver := Prepare(event)\n\tif !deliver {\n\t\treturn nil\n\t}\n\tm.Events = append(m.Events, event)\n\treturn nil",
  "file": "audit/sink.go",
  "symbol": "(*MemorySink).Write"
}
12✓ ✓
/bin/zsh -lc "sed -n '1,100p' audit/summary.go && \
sed -n '1,100p' audit/normalize.go && \
cat go.mod && \
test "'! -e go.sum || \
cat go.sum && \
git diff -- audit/summary.go audit/normalize.go go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event, deliver := Prepare(event)\n\tif !deliver {\n\t\treturn nil\n\t}\n\tif err := f.First.Write(event); err != nil {\n\t\treturn err\n\t}\n\treturn f.Second.Write(event)",
  "file": "audit/fanout.go",
  "symbol": "(FanoutSink).Write"
}
13✓ ✓
/bin/zsh -lc 'go test ./audit && \
git diff --check'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event, deliver := audit.Prepare(event)\n\tif !deliver {\n\t\treturn nil\n\t}\n\tj.Entries = append(j.Entries, event)\n\treturn nil",
  "file": "journal/file.go",
  "symbol": "(*JournalSink).Write"
}
14✓ ✓
/bin/zsh -lc "sed -n '1,100p' service/dispatch.go && \
sed -n '1,80p' audit/sink.go && \
sed -n '1,80p' audit/normalize.go && \
cat go.mod && \
test "'! -e go.sum || \
cat go.sum && \
go test ./service && \
git diff --check'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range b.queue {\n\t\tevent, deliver := Prepare(event)\n\t\tif !deliver {\n\t\t\tcontinue\n\t\t}\n\t\tif err := b.Sink.Write(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\tb.queue = b.queue[:0]\n\treturn nil",
  "file": "audit/transaction.go",
  "symbol": "(*BufferedSink).Commit"
}
15✓ ✓
/bin/zsh -lc 'gofmt -w service/dispatch.go && \
go test ./... && \
git diff --check && \
cat go.mod && \
test ! -e go.sum || \
cat go.sum'
semedit/semantic_insert_function (✓ ✓)
{
  "file": "audit/transaction.go",
  "placement": "after_symbol",
  "source": "func (b *BufferedSink) Write(event Event) error {\n\tb.queue = append(b.queue, event)\n\treturn nil\n}",
  "target_symbol": "(*BufferedSink).Begin"
}
16—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff --'
17—✓ ✓
/bin/zsh -lc 'go test ./...'
18—✓ ✓
/bin/zsh -lc 'git status --short --untracked-files=all; git diff --no-index /dev/null audit/normalize.go; git check-ignore -v audit/normalize.go'
19—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
20—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/fanout.go audit/sink.go audit/transaction.go service/dispatch.go service/replay.go journal/file.go; cat docs/audit-sinks.md; git status --short -- go.mod go.sum'
21—✓ ✓
/bin/zsh -lc "git status --short -- go.mod go.sum && \
rg -n 'Prepare\\(event\\)|strings\\.(ToLower|TrimSpace)' audit service journal"
22—✓ ✓
/bin/zsh -lc 'go test ./...'
23—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
24—✓ ✓
/bin/zsh -lc 'cat audit/summary.go audit/normalize.go audit/summary_test.go metrics/writer.go legacy/export.go; git status --short -- go.mod go.sum'
25—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "normalized := strings.ToLower(strings.TrimSpace(kind))\n\tswitch normalized {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
26—✓ ✓
/bin/zsh -lc 'go test ./...'
27—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
28—✓ ✓
/bin/zsh -lc 'git status --short -- go.mod go.sum; cat audit/summary.go'
29—✓ ✓
/bin/zsh -lc 'cat service/dispatch.go service/replay.go audit/normalize.go; git status --short -- go.mod go.sum'
30—✓ ✓
/bin/zsh -lc 'go test ./...'
31—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
32—✓ ✓
/bin/zsh -lc 'git status --short -- go.mod go.sum'
33—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go; git status --short -- go.mod go.sum'
34—semedit/semantic_insert_function (✓ ✓)
{
  "file": "audit/normalize.go",
  "placement": "after_symbol",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}",
  "target_symbol": "Prepare"
}
35—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = NormalizeKind(event.Kind)\n\treturn event, event.Kind != \"probe\"",
  "file": "audit/normalize.go",
  "symbol": "Prepare"
}
36—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
37—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
38—✓ ✓
/bin/zsh -lc 'go test ./...'
39—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
40—✓ ✓
/bin/zsh -lc 'git status --short -- go.mod go.sum'

default prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit

default prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantdefault
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency134.56s261.76s+94.5%109.24s——
Process Start → First Event0.14s0.13s—0.15s——
First Event → First Tool Call13.36s5.06s—5.12s——
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns550%5——
Internal Tool Cycles1232+166.7%12——
Initial Load / Discovery Turns43-25.0%4——
MCP Discovery / Schema Turns00—0——
Tool Invocations (per run)1232+166.7%12——
Output Tokens1072318051+68.3%14673——
Reasoning / Thinking Tokens34026931+103.7%7044——
Total Input Tokens17447783687724+111.4%1635343——
Cached Input Tokens16207363495168+115.7%1512704——
Uncached Input Tokens124042192556+55.2%122639——
Cost0.89191.6675+87.0%0.9562——
Cached vs Uncached Token Ratio13.07:118.15:1+38.9%12.33:1——
Oracle L1: Mutation PolicyPASSPASS—PASS——
Oracle L2: AST InvariantsPASSPASS—PASS——
Oracle L3: Clean BuildPASSPASS—PASS——
Oracle L4: Verification TestPASSPASS—PASS——
Semantic Tool Invocation Verified (per run)N/APASS—N/A——
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g '*.go' -g 'Makefile' -g 'README*'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' -g 'Makefile' | head -80"
2✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
rg -n "event|probe|deliver|persist|Sink|Kind|normalize" service audit metrics legacy cmd'
✓ ✓
/bin/zsh -lc 'rg --files | sort | head -100'
3✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/event.go audit/sink.go service/dispatch.go metrics/writer.go legacy/export.go service/dispatch_test.go'
✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
cat docs/audit-sinks.md && \
cat audit/normalize.go && \
cat audit/event.go && \
cat audit/sink.go && \
cat service/dispatch.go && \
cat metrics/writer.go && \
cat legacy/export.go'
4✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go service/dispatch.go && \
go test ./... && \
git status --short && \
git diff -- audit/normalize.go service/dispatch.go'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Record"
}
5✓ ✓
/bin/zsh -lc 'git diff -- audit/normalize.go service/dispatch.go && \
git status --short --untracked-files=all'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
6✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat service/dispatch.go && \
git diff -- go.mod go.sum && \
git status --short -- audit/normalize.go service/dispatch.go go.mod go.sum'
semedit/semantic_insert_function (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "service/dispatch.go",
  "placement": "private_end",
  "source": "func prepareEvent(event audit.Event) (audit.Event, bool) {\n\tevent.Kind = strings.ToLower(strings.TrimSpace(event.Kind))\n\treturn event, event.Kind != \"probe\"\n}"
}
7✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff -- go.mod go.sum'
semedit/semantic_replace_body (✓ ✗): replace_body Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-default_task-11-mixed-sink-api-migration_1790456434159398000/service/dispatch.go: symbol not found
{
  "auto_organize_imports": true,
  "body": "event, deliver := prepareEvent(event)\n\tif !deliver {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "Record"
}
8✓ ✓
/bin/zsh -lc 'rg -n "Classify|probe|NormalizeForDelivery|kind" audit service metrics legacy && \
cat audit/summary.go && \
cat audit/normalize.go'
✓ ✓
/bin/zsh -lc 'rg --files docs && \
cat service/dispatch.go'
9✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go && \
git diff -- go.mod go.sum && \
git status --short -- audit/normalize.go audit/summary.go go.mod go.sum'
✓ ✓
/bin/zsh -lc "python3 -c 'from pathlib import Path; import os,tempfile,time; p=Path(\"docs/SUBOPTIMAL_TOOLS.md\"); old=p.read_text() if p.exists() else \"# Suboptimal tool behavior\\n\"; entry=\"\\n- "'`semantic_replace_body` on `service/dispatch.go` failed to resolve the method name `Record` after `prepareEvent` was inserted, reporting symbol not found. No workaround was applied yet. The operation likely requires a receiver-qualified method name; retrying as `(*Dispatcher).Record`.'"\\n\"; fd,tmp=tempfile.mkstemp(dir=p.parent,prefix=p.name+\".\"); f=os.fdopen(fd,\"w\"); f.write(old+entry); f.flush(); os.fsync(f.fileno()); os.replace(tmp,p); os.utime(p,None)'"
10✓ ✓
/bin/zsh -lc 'cat service/dispatch.go && \
git diff -- go.mod go.sum && \
git status --short -- service/dispatch.go go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event, deliver := prepareEvent(event)\n\tif !deliver {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
11✓ ✓
/bin/zsh -lc "rg -n \"NormalizeForDelivery|normalizeKind|NormalizeKind|Classify\\(\" --glob '*.go' && \
cat audit/normalize.go audit/summary.go service/dispatch.go"
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "for _, event := range events {\n\t\tevent, deliver := prepareEvent(event)\n\t\tif !deliver {\n\t\t\tcontinue\n\t\t}\n\t\tif err := sink.Write(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
12✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./... && \
git diff -- go.mod go.sum'
✓ ✓
/bin/zsh -lc "python3 -c 'from pathlib import Path; import os,tempfile; p=Path(\"docs/SUBOPTIMAL_TOOLS.md\"); s=p.read_text(); s=s.replace(\"No workaround was applied yet. The operation likely requires a receiver-qualified method name; retrying as "'`(*Dispatcher).Record`.", "Retried with receiver-qualified symbol `(*Dispatcher).Record`, which succeeded. Root cause: method lookup required the receiver-qualified name."); fd,tmp=tempfile.mkstemp(dir=p.parent,prefix=p.name+"."); f=os.fdopen(fd,"w"); f.write(s); f.flush(); os.fsync(f.fileno()); os.replace(tmp,p); os.utime(p,None)'"'"
13—✓ ✓
/bin/zsh -lc 'go test ./...'
14—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "service"
}
15—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- service/dispatch.go docs/SUBOPTIMAL_TOOLS.md && \
git diff --exit-code -- go.mod go.sum service/dispatch_test.go audit/event_test.go audit/summary_test.go metrics/writer_test.go legacy/export_test.go'
16—✓ ✓
/bin/zsh -lc 'cat service/dispatch.go && \
cat docs/SUBOPTIMAL_TOOLS.md && \
git status --short --untracked-files=all'
17—✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/normalize.go && \
cat audit/event.go && \
cat audit/sink.go && \
cat service/dispatch.go && \
git diff -- go.mod go.sum'
18—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
19—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\treturn event, event.Kind != \"probe\"",
  "file": "service/dispatch.go",
  "symbol": "prepareEvent"
}
20—✓ ✓
/bin/zsh -lc "cat audit/normalize.go && \
sed -n '1,18p' service/dispatch.go && \
git diff -- go.mod go.sum && \
go test ./..."
21—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "audit"
}
22—✓ ✓
/bin/zsh -lc 'cat audit/summary.go && \
cat audit/normalize.go && \
cat docs/audit-sinks.md && \
git diff -- go.mod go.sum'
23—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
24—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "audit"
}
25—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --exit-code -- go.mod go.sum'
26—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
27—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
28—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "prepareEvent"
}
29—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "service"
}
30—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --exit-code -- go.mod go.sum'
31—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
32—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --exit-code -- go.mod go.sum'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 11 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g '*.go' -g 'Makefile' | head -160"
not published
2✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
printf '\\n--- root files ---\\n' && \
rg --files -g 'README*' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'Makefile' -g 'docs/**'"
not published
3✓ ✓
/bin/zsh -lc "cat docs/audit-sinks.md && \
printf '\\n--- audit files ---\\n' && \
for f in audit/event.go audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go service/dispatch.go; do echo \"### "'$f"; sed -n '"'1,240p' \""'$f"; done'
not published
4✓ ✓
/bin/zsh -lc "rg -n 'audit|probe|Kind|Record\\(|DispatchAll|Write\\(' --glob '*.go'"
not published
5✓ ✓
/bin/zsh -lc 'cat audit/summary.go metrics/writer.go metrics/metrics.go legacy/export.go journal/file.go service/replay.go'
not published
6✓ ✓
/bin/zsh -lc "cat go.mod && \
git status --short && \
rg -n 'Normalize|Classify' --glob '*.go'"
not published
7✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go audit/sink.go audit/fanout.go audit/transaction.go journal/file.go service/dispatch.go service/replay.go && \
go test ./... && \
git status --short && \
git diff --'
not published
8✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat audit/summary.go && \
git diff --stat && \
git check-ignore -v audit/normalize.go'
not published
9✓ ✓
/bin/zsh -lc "git status --short --untracked-files=all && \
git diff -- go.mod go.sum && \
sed -n '1,160p' audit/normalize.go && \
sed -n '1,120p' audit/sink.go && \
sed -n '1,120p' audit/transaction.go && \
sed -n '1,100p' audit/summary.go"
not published
10✓ ✓
/bin/zsh -lc "sed -n '1,80p' audit/summary.go && \
sed -n '1,80p' audit/normalize.go && \
git diff -- go.mod go.sum"
not published
11✓ ✓
/bin/zsh -lc "sed -n '1,100p' service/dispatch.go && \
sed -n '1,80p' service/replay.go && \
git diff -- go.mod go.sum"
not published
12✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./... && \
git diff -- go.mod go.sum && \
git status --short -- audit/normalize.go audit/summary.go service/dispatch.go go.mod go.sum'
not published

prefer_discover_semedit prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency136.94s——188.10s211.96s+12.7%
Process Start → First Event0.29s——0.13s0.15s—
First Event → First Tool Call5.62s——10.25s5.43s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns5——550%
Internal Tool Cycles12——1528+86.7%
Initial Load / Discovery Turns2——330%
MCP Discovery / Schema Turns0——00—
Tool Invocations (per run)12——1528+86.7%
Output Tokens11996——2470820883-15.5%
Reasoning / Thinking Tokens3562——1163111601-0.3%
Total Input Tokens1785488——33931863363402-0.9%
Cached Input Tokens1663744——31905283144448-1.4%
Uncached Input Tokens121744——202658218954+8.0%
Cost0.9148——1.75851.7395-1.1%
Cached vs Uncached Token Ratio13.67:1——15.74:114.36:1-8.8%
Oracle L1: Mutation PolicyPASS——PASSPASS—
Oracle L2: AST InvariantsPASS——PASSPASS—
Oracle L3: Clean BuildPASS——PASSPASS—
Oracle L4: Verification TestFAIL——FAILPASS—
Semantic Tool Invocation Verified (per run)N/A——N/APASS—
Baseline vs Semedit Agentic in Small Context
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g '*.go' -g 'README.md' -g 'Makefile' | head -200"
not published
2✓ ✓
/bin/zsh -lc 'cat PROJECT_BRIEF.md 2>/dev/null; cat README.md; cat audit/event.go audit/normalize.go audit/sink.go service/dispatch.go'
not published
3✓ ✓
/bin/zsh -lc "rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g '*.go' -g 'go.mod' -g 'Makefile' && \
cat metrics/writer.go legacy/export.go audit/summary.go cmd/migrate/main.go && \
cat go.mod"
not published
4✓ ✓
/bin/zsh -lc "cat service/dispatch_test.go audit/event_test.go audit/summary_test.go; rg -n 'Dispatch|Record|probe|Normalize|Kind' ."
not published
5✓ ✓
/bin/zsh -lc 'gofmt -w service/dispatch.go && \
go test ./... && \
git diff --check && \
git status --short && \
git diff -- service/dispatch.go go.mod go.sum'
not published
6? ✗
/bin/zsh -lc "git diff -- service/dispatch.go; git status --short -- service/dispatch.go go.mod go.sum; sed -n '1,100p' service/dispatch.go; rg --files -g 'Makefile' -g 'makefile' -g 'Taskfile*' -g 'AGENTS.md' -g 'PROJECT_BRIEF.md'"
not published
7✓ ✓
/bin/zsh -lc 'gofmt -w service/dispatch.go && \
go test ./... && \
git diff --check && \
git status --short'
not published
8✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/event.go service/dispatch.go; git diff -- audit/normalize.go audit/event.go service/dispatch.go go.mod go.sum; git status --short -- audit/normalize.go audit/event.go service/dispatch.go go.mod go.sum'
not published
9✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go service/dispatch.go && \
go test ./... && \
git diff --check && \
git status --short -- audit/normalize.go service/dispatch.go go.mod go.sum && \
git diff -- audit/normalize.go service/dispatch.go go.mod go.sum'
not published
10✓ ✓
/bin/zsh -lc 'gofmt -w audit/summary.go && \
go test ./... && \
git diff --check && \
git status --short -- audit/summary.go go.mod go.sum'
not published
11✓ ✓
/bin/zsh -lc 'cat service/dispatch.go audit/normalize.go; go test ./...; git diff --check; git status --short -- go.mod go.sum service/dispatch.go'
not published
12✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./...'
not published
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 40 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 17 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'README*' -g 'go.mod' -g '*.go' | head -160 && \
git status --short"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' -g 'AGENTS*' -g 'go.mod' -g 'go.sum' -g '*.go' | head -160"
2✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
sed -n '1,240p' audit/event.go && \
sed -n '1,260p' audit/normalize.go && \
sed -n '1,240p' audit/sink.go && \
sed -n '1,240p' audit/fanout.go && \
sed -n '1,240p' service/dispatch.go"
✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' -g 'AGENTS.md' -g 'AGENTS.override.md' && \
git status --short && \
sed -n '1,240p' AGENTS.override.md && \
rg -n \"event|probe|deliver|persist|sink\" --glob '*.go' ."
3✓ ✓
/bin/zsh -lc "rg -n \"Kind|Sink|Write\\(|probe|metrics|export|audit\" --glob '*.go'"
✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/event.go audit/sink.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go audit/summary.go journal/file.go && \
cat Makefile 2>/dev/null || \
true && \
rg --files | sort'
4✓ ✓
/bin/zsh -lc 'cat service/replay.go audit/transaction.go audit/summary.go journal/file.go metrics/metrics.go metrics/writer.go legacy/export.go api/router.go'
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat go.mod && \
cat audit/summary_test.go service/dispatch_test.go'
5? ✗
/bin/zsh -lc "ls -la && \
rg --files -g 'Makefile' -g 'README*' -g 'PROJECT_BRIEF.md'"
semedit/semantic_outline (✓ ✓)
{
  "include_tests": false,
  "language": "go",
  "path": "audit"
}
6✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go audit/summary.go journal/file.go service/dispatch.go service/replay.go && \
go test ./... && \
git diff --check && \
git status --short && \
git diff -- audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go audit/summary.go journal/file.go service/dispatch.go service/replay.go'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "source": "func NormalizeEvent(event Event) Event {\n\tevent.Kind = strings.ToLower(strings.TrimSpace(event.Kind))\n\treturn event\n}"
}
7✓ ✓
/bin/zsh -lc 'git diff --stat && \
git status --short --untracked-files=all && \
git ls-files audit/normalize.go audit/sink.go'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "source": "func Deliver(sink Sink, event Event) error {\n\tevent = NormalizeEvent(event)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn sink.Write(event)\n}"
}
8✓ ✓
/bin/zsh -lc "sed -n '1,200p' audit/normalize.go && \
sed -n '1,140p' audit/sink.go && \
sed -n '1,100p' audit/fanout.go && \
sed -n '1,120p' audit/transaction.go && \
sed -n '1,100p' audit/summary.go && \
sed -n '1,100p' service/dispatch.go && \
sed -n '1,80p' service/replay.go && \
sed -n '1,80p' journal/file.go && \
git diff -- go.mod go.sum && \
git status --short -- audit service journal metrics legacy go.mod go.sum"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "return audit.Deliver(d.sink, event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
9✓ ✓
/bin/zsh -lc "rg -n \"\\.Write\\(event\\)|Sink\\.Write|Write\\(.*Event|NormalizeKind|NormalizeEvent|Deliver\\(\" --glob '*.go'"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tif err := audit.Deliver(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
10✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --check && \
git diff --exit-code -- go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tif err := audit.Deliver(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/replay.go",
  "symbol": "Replay"
}
11✓ ✓
/bin/zsh -lc "cat audit/summary.go && \
sed -n '1,100p' audit/normalize.go && \
git diff -- go.mod go.sum"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "if err := Deliver(f.First, event); err != nil {\n\t\treturn err\n\t}\n\treturn Deliver(f.Second, event)",
  "file": "audit/fanout.go",
  "symbol": "(FanoutSink).Write"
}
12✓ ✓
/bin/zsh -lc 'gofmt -w audit/summary.go && \
go test ./audit && \
git diff --check && \
git diff --exit-code -- go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range b.queue {\n\t\tif err := Deliver(b.Sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "audit/transaction.go",
  "symbol": "(*BufferedSink).Commit"
}
13✓ ✓
/bin/zsh -lc "rg -n \"\\.Write\\(|Deliver\\(|Record\\(|DispatchAll|Replay\\(\" service --glob '*.go' && \
sed -n '1,100p' service/dispatch.go && \
sed -n '1,80p' service/replay.go && \
git diff -- go.mod go.sum"
? ✗
/bin/zsh -lc 'make check'
14✓ ✓
/bin/zsh -lc 'go test ./service && \
git diff --check && \
git diff --exit-code -- go.mod go.sum'
✓ ✓
/bin/zsh -lc "git diff -- audit/normalize.go service/dispatch.go service/replay.go audit/fanout.go audit/transaction.go && \
git status --short && \
git diff --name-only -- go.mod go.sum '*_test.go'"
15✓ ✓
/bin/zsh -lc 'gofmt -w service/dispatch.go && \
go test ./... && \
git diff --check && \
git diff --exit-code -- go.mod go.sum'
✓ ✓
/bin/zsh -lc 'cat audit/normalize.go service/dispatch.go service/replay.go audit/fanout.go audit/transaction.go && \
git rev-parse --show-toplevel && \
git status --short --untracked-files=all'
16—✓ ✓
/bin/zsh -lc 'go test ./...'
17—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/event.go service/dispatch.go service/replay.go audit/fanout.go audit/transaction.go && \
git diff -- go.mod go.sum && \
git status --short -- go.mod go.sum'
18—✓ ✓
/bin/zsh -lc "rg -n \"\\.Write\\(|NormalizeEvent|Deliver\\(\" --glob '*.go' . && \
cat docs/audit-sinks.md && \
git diff --exit-code -- go.mod go.sum"
19—✓ ✓
/bin/zsh -lc 'cat audit/summary.go && \
git diff -- go.mod go.sum'
20—semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch kind {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
21—✓ ✓
/bin/zsh -lc 'cat audit/summary.go && \
git diff --exit-code -- go.mod go.sum'
22—✓ ✓
/bin/zsh -lc 'cat service/dispatch.go audit/normalize.go && \
git diff -- go.mod go.sum'
23—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go && \
git diff -- go.mod go.sum'
24—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
25—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = NormalizeKind(event.Kind)\n\treturn event",
  "file": "audit/normalize.go",
  "symbol": "NormalizeEvent"
}
26—semedit/semantic_replace_body (✓ ✓)
{
  "body": "kind = NormalizeKind(kind)\n\tswitch kind {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
27—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
28—✓ ✓
/bin/zsh -lc 'git diff --exit-code -- go.mod go.sum && \
go test ./...'

prefer_discover_semedit prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency45.27s189.99s+319.7%194.12s243.21s+25.3%
Process Start → First Event0.14s0.12s—0.12s0.15s—
First Event → First Tool Call5.92s12.90s—8.45s18.32s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns15+400.0%550%
Internal Tool Cycles622+266.7%1632+100.0%
Initial Load / Discovery Turns220%32-33.3%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)622+266.7%1632+100.0%
Output Tokens117415316+1204.6%2263719884-12.2%
Reasoning / Thinking Tokens3837075+1747.3%121649206-24.3%
Total Input Tokens1856985048103+2618.4%31072484540119+46.1%
Cached Input Tokens1707524792832+2706.9%29201924304384+47.4%
Uncached Input Tokens14946255271+1608.0%187056235735+26.0%
Cost0.09952.1163+2026.6%1.63272.0291+24.3%
Cached vs Uncached Token Ratio11.42:118.78:1+64.3%15.61:118.26:1+17.0%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSFAIL—PASSFAIL—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'README*' -g 'PROJECT_BRIEF.md' -g '*audit*' -g '*probe*' -g 'AGENTS*' | sort"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS*' -g '*.go' | head -120 && \
git status --short && \
rg -n \"audit|probe|metrics|legacy|event kind|eventKind\" --glob '*.go' --glob '*.md' ."
2✓ ✓
/bin/zsh -lc 'rg --files'
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/event.go audit/normalize.go audit/sink.go service/dispatch.go && \
cat go.mod'
3✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/event.go && \
cat audit/normalize.go && \
cat audit/sink.go && \
cat service/dispatch.go && \
git status --short'
semedit/semantic_lookup (✓ ✗): param "symbol" is required: invalid operation parameters
{
  "file": "service/dispatch.go",
  "query": "Record"
}
4✓ ✓
/bin/zsh -lc 'cat audit/event_test.go audit/summary.go audit/summary_test.go service/dispatch_test.go metrics/writer.go legacy/export.go go.mod'
semedit/report_feedback (✓ ✓)
{
  "command": "mcp__semedit__semantic_lookup",
  "intent": "Locate a Go declaration before a semantic source edit.",
  "interface": "mcp",
  "manual_touchups": "None.",
  "observed_result": "The tool rejected the request because it requires a symbol parameter.",
  "parameters": {
    "file": "service/dispatch.go",
    "query": "Record"
  },
  "suggested_fix": "Accept query as an alias or expose the required symbol parameter clearly in the tool schema.",
  "suspected_cause": "The callable schema and request description may disagree.",
  "target_context": "service/dispatch.go",
  "unexpected_reason": "The lookup request used a query parameter based on the available tool description, but the backend schema requires symbol."
}
5✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./... && \
git diff --check && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go && \
git status --short'
semedit/semantic_lookup (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Record"
}
6✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go && \
git diff --stat && \
git diff --cached --stat'
semedit/semantic_outline (✓ ✓)
{
  "depth": 2,
  "path": "audit"
}
7—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
8—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "private",
  "file": "service/dispatch.go",
  "source": "func writeEvent(sink audit.Sink, event audit.Event) error {\n\tevent.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn sink.Write(event)\n}"
}
9—semedit/semantic_replace_body (✓ ✓)
{
  "body": "return writeEvent(d.sink, event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
10—semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tif err := writeEvent(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
11—✓ ✓
/bin/zsh -lc 'go test ./... && \
git status --short && \
git diff -- go.mod go.sum && \
git diff -- audit/normalize.go service/dispatch.go'
12—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go service/dispatch.go && \
git rev-parse --show-toplevel && \
git diff --stat && \
git status --short'
13—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/sink.go service/dispatch.go docs/audit-sinks.md && \
git status --short --untracked-files=no && \
git diff -- go.mod go.sum && \
git check-ignore -v audit/normalize.go service/dispatch.go'
14—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --exit-code -- go.mod go.sum'
15—✓ ✓
/bin/zsh -lc 'cat audit/summary.go audit/normalize.go && \
cat audit/summary_test.go && \
git diff -- go.mod go.sum'
16—semedit/semantic_lookup (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify"
}
17—semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
18—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --exit-code -- go.mod go.sum'
19—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Record"
}
20—✓ ✓
/bin/zsh -lc 'cat service/dispatch.go audit/normalize.go && \
go test ./... && \
git diff --exit-code -- go.mod go.sum'
21—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
22—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --exit-code -- go.mod go.sum'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 24 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 17 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS.md' -g 'AGENTS.override.md' -g '*.go' | head -100 && \
git status --short"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g 'README.md' -g 'go.mod' -g '*.go' | head -200"
2✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
for f in audit/normalize.go audit/event.go audit/sink.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go; do printf '\\n--- %s ---\\n' \""'$f"; cat "$f"; done'
✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
cat go.mod && \
rg -n \"probe|Audit|audit|Deliver|Persist|Write|Normalize|normaliz\" --glob '*.go'"
3✓ ✓
/bin/zsh -lc "for f in audit/*.go metrics/*.go legacy/*.go api/router.go; do printf '\\n--- %s ---\\n' \""'$f"; cat "$f"; done'
semedit/semantic_outline (✓ ✓)
{
  "path": "audit"
}
4✓ ✓
/bin/zsh -lc 'cat service/dispatch_test.go api/router_test.go'
semedit/semantic_outline (✓ ✓)
{
  "path": "service"
}
5? ✗
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go && \
rg -n "''^check:|make check|go test" Makefile README.md'
✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go journal/file.go audit/summary.go'
6✓ ✓
/bin/zsh -lc "rg --files -g 'Makefile' -g 'go.mod' -g 'go.sum' -g '*.go' | sort && \
git diff --check && \
go test ./..."
✓ ✓
/bin/zsh -lc 'cat audit/event_test.go audit/summary_test.go service/dispatch_test.go'
7✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- audit service && \
git diff --exit-code -- go.mod go.sum'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeEvent(event Event) Event {\n\tevent.Kind = strings.ToLower(strings.TrimSpace(event.Kind))\n\treturn event\n}"
}
8✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
cat service/dispatch.go && \
git ls-files -v audit/normalize.go'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func Deliver(sink Sink, event Event) error {\n\tevent = NormalizeEvent(event)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn sink.Write(event)\n}"
}
9✓ ✓
/bin/zsh -lc 'git rev-parse --show-toplevel && \
git ls-files | head -50 && \
git check-ignore -v audit/normalize.go service/dispatch.go'
semedit/semantic_replace_body (✓ ✗): replace_body Record in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_large-prefer_discover_semedit_task-11-mixed-sink-api-migration_1790456909624170000/service/dispatch.go: symbol not found
{
  "body": "return audit.Deliver(d.sink, event)",
  "file": "service/dispatch.go",
  "symbol": "Record"
}
10✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/event.go audit/sink.go audit/fanout.go audit/transaction.go audit/summary.go service/dispatch.go service/replay.go && \
git diff -- go.mod go.sum && \
git status --short --untracked-files=all'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tif err := audit.Deliver(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
11✓ ✓
/bin/zsh -lc "rg -n \"Normal|normalize|probe|Kind|Sink\" --glob '"'!**/*_test.go'"' ."
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range events {\n\t\tif err := audit.Deliver(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/replay.go",
  "symbol": "Replay"
}
12✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go && \
git diff --exit-code -- go.mod go.sum && \
go test ./audit'
✓ ✓
/bin/zsh -lc "rg --files | rg 'SUBOPTIMAL|normalize.go' && \
cat audit/normalize.go"
13✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go && \
go test ./audit && \
git diff --exit-code -- go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "return audit.Deliver(d.sink, event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
14✓ ✓
/bin/zsh -lc 'gofmt -w service/dispatch.go && \
go test ./service && \
git diff --exit-code -- go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event = NormalizeEvent(event)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\tm.Events = append(m.Events, event)\n\treturn nil",
  "file": "audit/sink.go",
  "symbol": "(*MemorySink).Write"
}
15✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go && \
git diff -- go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event = NormalizeEvent(event)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\tif err := Deliver(f.First, event); err != nil {\n\t\treturn err\n\t}\n\treturn Deliver(f.Second, event)",
  "file": "audit/fanout.go",
  "symbol": "FanoutSink.Write"
}
16✓ ✓
/bin/zsh -lc 'gofmt -w service/dispatch.go && \
go test ./... && \
git diff --exit-code -- go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "for _, event := range b.queue {\n\t\tif err := Deliver(b.Sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "audit/transaction.go",
  "symbol": "(*BufferedSink).Commit"
}
17—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event = audit.NormalizeEvent(event)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\tj.Entries = append(j.Entries, event)\n\treturn nil",
  "file": "journal/file.go",
  "symbol": "(*JournalSink).Write"
}
18—✓ ✓
/bin/zsh -lc 'go test ./... && \
git status --short && \
git diff --stat && \
git diff -- go.mod go.sum'
19—✓ ✓
/bin/zsh -lc 'git diff -- audit service journal docs/SUBOPTIMAL_TOOLS.md && \
git status --short --untracked-files=all'
20—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
21—✓ ✓
/bin/zsh -lc 'pwd && \
cat audit/normalize.go && \
cat audit/sink.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go journal/file.go && \
git status --short -- audit service journal go.mod go.sum'
22—✓ ✓
/bin/zsh -lc 'cat audit/summary.go audit/normalize.go && \
cat audit/summary_test.go && \
git status --short -- audit/summary.go go.mod go.sum'
23—semedit/semantic_replace_body (✓ ✓)
{
  "body": "kind = NormalizeEvent(Event{Kind: kind}).Kind\n\tswitch kind {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
24—✓ ✓
/bin/zsh -lc 'go test ./audit && \
git diff -- audit/summary.go go.mod go.sum'
25—✓ ✓
/bin/zsh -lc 'cat service/dispatch.go && \
cat audit/normalize.go && \
git status --short -- service/dispatch.go go.mod go.sum'
26—✓ ✓
/bin/zsh -lc 'go test ./service && \
git diff -- service/dispatch.go go.mod go.sum'
27—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go'
28—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "audit/normalize.go",
  "placement": "public_start",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
29—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = NormalizeKind(event.Kind)\n\treturn event",
  "file": "audit/normalize.go",
  "symbol": "NormalizeEvent"
}
30—semedit/semantic_replace_body (✓ ✓)
{
  "body": "kind = NormalizeKind(kind)\n\tswitch kind {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
31—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
32—✓ ✓
/bin/zsh -lc 'go test ./...'

prefer_discover_semedit prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency74.95s219.76s+193.2%196.66s129.46s-34.2%
Process Start → First Event0.52s0.14s—0.22s0.13s—
First Event → First Tool Call20.25s6.40s—14.94s8.50s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns15+400.0%51-80.0%
Internal Tool Cycles929+222.2%1417+21.4%
Initial Load / Discovery Turns23+50.0%32-33.3%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)929+222.2%1417+21.4%
Output Tokens176816281+820.9%299403230-89.2%
Reasoning / Thinking Tokens6128554+1297.7%126841741-86.3%
Total Input Tokens3773233581523+849.2%3214361634769-80.3%
Cached Input Tokens3499523383296+866.8%3008256592896-80.3%
Uncached Input Tokens27371198227+624.2%20610541873-79.7%
Cost0.18571.6518+789.7%1.80010.3150-82.5%
Cached vs Uncached Token Ratio12.79:117.07:1+33.5%14.60:114.16:1-3.0%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSFAIL—FAILPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' -g '*.go' -g 'AGENTS.md' | head -100"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'docs/**' -g 'Makefile' -g '*.go' | head -200"
2✓ ✓
/bin/zsh -lc 'rg --files'
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
git status --short'
3✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/event.go && \
cat audit/normalize.go && \
cat audit/sink.go && \
cat service/dispatch.go && \
git status --short'
semedit/semantic_outline (✓ ✓)
{
  "path": "."
}
4✓ ✓
/bin/zsh -lc 'cat audit/event_test.go audit/summary.go audit/summary_test.go service/dispatch_test.go metrics/writer.go legacy/export.go'
✓ ✓
/bin/zsh -lc 'cat service/dispatch.go audit/normalize.go audit/event.go audit/sink.go audit/summary.go'
5✓ ✓
/bin/zsh -lc 'cat go.mod && \
cat AGENTS.override.md'
✓ ✓
/bin/zsh -lc 'cat service/dispatch_test.go audit/event_test.go audit/summary_test.go'
6✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --check && \
git status --short && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go go.mod go.sum'
semedit/semantic_insert_function (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "placement": "file_end",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
7✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go && \
git diff --stat && \
git status --short --untracked-files=no'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
8✓ ✓
/bin/zsh -lc 'git diff --no-index /dev/null audit/normalize.go >/dev/null; git status --porcelain=v1 --untracked-files=no; git ls-files -v audit/normalize.go audit/summary.go service/dispatch.go; git diff HEAD -- audit/normalize.go audit/summary.go service/dispatch.go'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "dispatcher := NewDispatcher(sink)\n\tfor _, event := range events {\n\t\tif err := dispatcher.Record(event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
9✓ ✓
/bin/zsh -lc 'git rev-parse --show-toplevel && \
git ls-files | head -30 && \
git check-ignore -v audit/normalize.go audit/summary.go service/dispatch.go'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
10—? ✗
/bin/zsh -lc 'make check'
11—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
12—✓ ✓
/bin/zsh -lc "git status --short && \
git diff -- service/dispatch.go audit/normalize.go audit/summary.go go.mod go.sum && \
git diff --name-only -- '*_test.go' go.mod go.sum"
13—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Record"
}
14—✓ ✓
/bin/zsh -lc 'cat service/dispatch.go audit/normalize.go audit/summary.go'
15—✓ ✓
/bin/zsh -lc "git status --short && \
git diff -- audit service go.mod go.sum && \
rg -n \"NormalizeKind|probe|Classify|Record\\(\" audit service"
16—? ✗
/bin/zsh -lc 'git ls-files --stage audit/normalize.go service/dispatch.go && \
git show HEAD:audit/normalize.go && \
git show HEAD:service/dispatch.go'
17—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/normalize.go",
  "symbol": "NormalizeKind"
}
18—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Record"
}
19—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
20—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify"
}
21—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/normalize.go",
  "symbol": "NormalizeKind"
}
22—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
23—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Record"
}
24—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
25—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/normalize.go",
  "symbol": "NormalizeKind"
}
26—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify"
}
27—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Record"
}
28—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- go.mod go.sum'
29—✓ ✓
/bin/zsh -lc 'go test ./...'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 32 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 13 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS*' -g '*.go' | head -100 && \
git status --short && \
rg -n -i \"audit|probe|legacy export|metrics|event.?kind|sink\" --glob '*.go' --glob '*.md'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS*' -g '*.go' -g 'go.mod' -g 'go.sum' | head -100"
2✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/event.go audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go audit/summary.go service/dispatch.go service/replay.go journal/file.go metrics/writer.go legacy/export.go'
✓ ✓
/bin/zsh -lc 'cat PROJECT_BRIEF.md 2>/dev/null; cat README.md 2>/dev/null; git status --short; rg -n "audit|probe|event|persist|Deliver" audit service'
3✓ ✓
/bin/zsh -lc "cat audit/*_test.go service/*_test.go && \
cat go.mod && \
rg -n \"Write\\(|Classify\\(|DispatchAll|Replay|Begin\\(|Commit\\(\" --glob '*.go'"
✓ ✓
/bin/zsh -lc 'cat audit/event.go audit/sink.go audit/normalize.go audit/transaction.go audit/fanout.go audit/summary.go service/dispatch.go service/replay.go legacy/export.go metrics/writer.go'
4✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/sink.go audit/summary.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go journal/file.go && \
go test ./... && \
git status --short && \
git diff --check && \
git diff --stat && \
git diff -- go.mod go.sum'
✓ ✓
/bin/zsh -lc "find . -maxdepth 2 -name 'PROJECT_BRIEF.md' -o -name 'README.md' | sort; cat go.mod; git status --short"
5✓ ✓
/bin/zsh -lc 'cat audit/normalize.go && \
git rev-parse --show-toplevel && \
git status --short --untracked-files=all && \
git diff -- audit/normalize.go'
semedit/semantic_outline (✓ ✓)
{
  "include_tests": false,
  "path": "audit"
}
6✓ ✓
/bin/zsh -lc "gofmt -w audit/normalize.go audit/sink.go audit/fanout.go journal/file.go && \
go test ./... && \
git diff --check && \
git diff --name-only -- go.mod go.sum '**/*_test.go' && \
git diff --stat && \
git status --short --untracked-files=no"
semedit/semantic_outline (✓ ✓)
{
  "include_tests": false,
  "path": "service"
}
7✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go audit/sink.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go journal/file.go && \
git status --short --untracked-files=no && \
git diff -- go.mod go.sum'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
8? ✗
/bin/zsh -lc 'go test ./... && \
git diff --check && \
test "$(git show HEAD:go.mod)" = "$(cat go.mod)" && \
test "$(git show HEAD:go.sum)" = "$(cat go.sum)"'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "audit/normalize.go",
  "source": "func NormalizeEvent(event Event) Event {\n\tevent.Kind = NormalizeKind(event.Kind)\n\treturn event\n}"
}
9✓ ✓
/bin/zsh -lc 'cat audit/summary.go audit/normalize.go && \
git diff -- go.mod go.sum && \
git status --short --untracked-files=no'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event = NormalizeEvent(event)\n\tm.Events = append(m.Events, event)\n\treturn nil",
  "file": "audit/sink.go",
  "symbol": "(*MemorySink).Write"
}
10? ✗
/bin/zsh -lc 'gofmt -w audit/summary.go && \
go test ./audit && \
git diff --check && \
test "$(git show HEAD:go.mod)" = "$(cat go.mod)" && \
test "$(git show HEAD:go.sum)" = "$(cat go.sum)"'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event = NormalizeEvent(event)\n\tif err := f.First.Write(event); err != nil {\n\t\treturn err\n\t}\n\treturn f.Second.Write(event)",
  "file": "audit/fanout.go",
  "symbol": "FanoutSink.Write"
}
11✓ ✓
/bin/zsh -lc 'cat service/dispatch.go service/replay.go audit/normalize.go && \
git diff -- go.mod go.sum'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
12? ✗
/bin/zsh -lc 'go test ./service && \
git diff --check && \
test "$(git show HEAD:go.mod)" = "$(cat go.mod)" && \
test "$(git show HEAD:go.sum)" = "$(cat go.sum)"'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "event = audit.NormalizeEvent(event)\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
13✓ ✓
/bin/zsh -lc "rg -n \"Normalize\\(|IsProbe\\(|Deliver\\(|NormalizeKind|IsControl\" --glob '*.go' && \
cat audit/normalize.go audit/summary.go service/dispatch.go"
semedit/semantic_replace_body (✓ ✓)
{
  "body": "writeEvent := audit.Sink.Write\n\tfor _, event := range events {\n\t\tevent = audit.NormalizeEvent(event)\n\t\tif err := writeEvent(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
14? ✗
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./... && \
git diff --check && \
test "$(git show HEAD:go.mod)" = "$(cat go.mod)" && \
test "$(git show HEAD:go.sum)" = "$(cat go.sum)"'
semedit/semantic_replace_body (✓ ✓)
{
  "body": "writeEvent := audit.Sink.Write\n\tfor _, event := range events {\n\t\tevent = audit.NormalizeEvent(event)\n\t\tif err := writeEvent(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/replay.go",
  "symbol": "Replay"
}
15—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --check && \
git status --short && \
git diff --stat && \
git diff -- go.mod go.sum'
16—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go; git diff --stat; git status --short --untracked-files=no; git rev-parse --show-toplevel'
17—✓ ✓
/bin/zsh -lc 'git diff -- audit/normalize.go audit/sink.go audit/fanout.go service/dispatch.go service/replay.go; git ls-files -v audit/normalize.go audit/sink.go; git check-ignore -v audit/normalize.go'

prefer_discover_semedit prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency95.02s264.06s+177.9%124.86s293.18s+134.8%
Process Start → First Event0.12s0.14s—0.21s0.12s—
First Event → First Tool Call6.04s7.03s—8.63s9.16s—
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns35+66.7%35+66.7%
Internal Tool Cycles932+255.6%1146+318.2%
Initial Load / Discovery Turns14+300.0%42-50.0%
MCP Discovery / Schema Turns00—00—
Tool Invocations (per run)932+255.6%1146+318.2%
Output Tokens644316545+156.8%1051131272+197.5%
Reasoning / Thinking Tokens30027945+164.7%555418264+228.8%
Total Input Tokens8070653372208+317.8%15876384469263+181.5%
Cached Input Tokens7283203124992+329.1%14817284152832+180.3%
Uncached Input Tokens78745247216+213.9%105910316431+198.8%
Cost0.49701.7054+243.1%0.83602.4485+192.9%
Cached vs Uncached Token Ratio9.25:112.64:1+36.7%13.99:113.12:1-6.2%
Oracle L1: Mutation PolicyPASSPASS—PASSPASS—
Oracle L2: AST InvariantsPASSPASS—PASSPASS—
Oracle L3: Clean BuildPASSPASS—PASSPASS—
Oracle L4: Verification TestPASSPASS—PASSPASS—
Semantic Tool Invocation Verified (per run)N/APASS—N/APASS—
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS.md' -g '"'!go.sum'"' -g '"'!go.mod'"' | head -80 && \
rg -n \"audit|probe|metrics|legacy export\" --glob '"'!go.sum'"' --glob '"'!go.mod'"' ."
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g '*.go' -g 'AGENTS.md' -g '"'!vendor'"' | head -120"
2? ✗
/bin/zsh -lc "cat docs/audit-sinks.md && \
cat audit/normalize.go && \
cat audit/event.go && \
cat audit/sink.go && \
cat service/dispatch.go && \
cat audit/summary.go && \
git status --short && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'AGENTS.md'"
✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'Makefile' -g 'go.mod' -g '*.go'"
3✓ ✓
/bin/zsh -lc 'rg --files && \
cat audit/summary_test.go audit/event_test.go service/dispatch_test.go metrics/writer.go legacy/export.go'
semedit/semantic_outline (✓ ✓)
{
  "path": "."
}
4✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./... && \
git diff --check && \
git diff -- go.mod go.sum && \
git status --short && \
git diff'
✓ ✓
/bin/zsh -lc 'git status --short && \
cat audit/event.go audit/normalize.go audit/sink.go service/dispatch.go'
5✓ ✓
/bin/zsh -lc 'git status --short --untracked-files=all && \
git diff --stat && \
cat audit/normalize.go audit/summary.go service/dispatch.go && \
git ls-files audit/normalize.go'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Dispatcher.Record"
}
6✓ ✓
/bin/zsh -lc 'git status --short -- audit/ service/ metrics/ legacy/ go.mod go.sum && \
cat audit/normalize.go audit/summary.go service/dispatch.go && \
cat docs/audit-sinks.md && \
git diff -- go.mod go.sum'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
7✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --check && \
git diff -- go.mod go.sum'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/sink.go",
  "symbol": "MemorySink.Write"
}
8✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go audit/event.go service/dispatch.go && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go go.mod go.sum'
✓ ✓
/bin/zsh -lc 'cat audit/event_test.go audit/summary.go audit/summary_test.go audit/sink.go service/dispatch_test.go legacy/export.go metrics/writer.go go.mod'
9✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go && \
go test ./... && \
git diff --check && \
git diff -- go.mod go.sum'
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
10—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
11—semedit/semantic_replace_body (✓ ✓)
{
  "body": "writeEvent := audit.Sink.Write\n\tfor _, event := range events {\n\t\tevent.Kind = audit.NormalizeKind(event.Kind)\n\t\tif event.Kind == \"probe\" {\n\t\t\tcontinue\n\t\t}\n\t\tif err := writeEvent(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
12—semedit/semantic_replace_body (✓ ✓)
{
  "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
13—✓ ✓
/bin/zsh -lc 'git diff --check && \
git status --short && \
git diff --stat && \
git diff -- go.mod go.sum'
14—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
15—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go service/dispatch.go audit/summary.go && \
git rev-parse --show-toplevel && \
git status --short --untracked-files=all'
16—✓ ✓
/bin/zsh -lc 'go test ./...'
17—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/sink.go service/dispatch.go audit/summary.go && \
git status --short -- audit/normalize.go audit/sink.go service/dispatch.go audit/summary.go go.mod go.sum && \
git diff -- audit/normalize.go audit/sink.go service/dispatch.go audit/summary.go go.mod go.sum'
18—semedit/semantic_outline (✓ ✓)
{
  "include_tests": false,
  "path": "audit"
}
19—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = NormalizeKind(event.Kind)\n\tm.Events = append(m.Events, event)\n\treturn nil",
  "file": "audit/sink.go",
  "symbol": "(*MemorySink).Write"
}
20—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --check && \
cat go.mod && \
if test -f go.sum; then cat go.sum; fi'
21—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
22—✓ ✓
/bin/zsh -lc 'cat audit/summary.go audit/normalize.go audit/sink.go go.mod; if test -f go.sum; then cat go.sum; fi'
23—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify"
}
24—✓ ✓
/bin/zsh -lc 'cat go.mod; if test -f go.sum; then cat go.sum; fi'
25—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
26—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Dispatcher.Record"
}
27—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "service"
}
28—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
29—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/normalize.go",
  "symbol": "NormalizeKind"
}
30—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify"
}
31—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Dispatcher.Record"
}
32—✓ ✓
/bin/zsh -lc 'go test ./...'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' -g '*.go' -g 'Makefile' | head -100 && \
git status --short"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'Makefile' -g '*.go' | head -100 && \
git status --short"
2✓ ✓
/bin/zsh -lc "rg --files -g 'AGENTS*' -g 'PROJECT_BRIEF.md' -g '*.md' -g 'Makefile' | sort && \
rg -n -i 'probe|event|deliver|persist|normalize|fanout' --glob '*.go'"
✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'Makefile' -g 'README*' -g 'AGENTS*'"
3✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/normalize.go audit/event.go audit/sink.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go journal/file.go audit/summary.go metrics/metrics.go metrics/writer.go legacy/export.go'
semedit/semantic_outline (✓ ✓)
{
  "path": "audit"
}
4✓ ✓
/bin/zsh -lc 'cat AGENTS.override.md && \
rg --files | sort && \
cat Makefile 2>/dev/null || \
true && \
cat audit/summary_test.go service/dispatch_test.go'
✓ ✓
/bin/zsh -lc "rg -n \"audit|probe|event|persist|sink\" --glob '"'!**/*_test.go'"'"
5✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go audit/sink.go audit/fanout.go audit/transaction.go journal/file.go service/dispatch.go service/replay.go && \
go test ./... && \
git status --short && \
git diff --stat && \
git diff -- go.mod go.sum'
semedit/semantic_outline (✓ ✓)
{
  "path": "service"
}
6✓ ✓
/bin/zsh -lc 'git diff -- audit && \
git status --short --untracked-files=all && \
git ls-files -v audit/normalize.go audit/summary.go service/dispatch.go'
✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/normalize.go && \
cat audit/fanout.go && \
cat audit/transaction.go && \
cat audit/sink.go && \
cat audit/summary.go && \
cat service/dispatch.go && \
cat metrics/writer.go && \
cat legacy/export.go'
7✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify"
}
8✓ ✓
/bin/zsh -lc 'git status --short --untracked-files=no; git diff -- go.mod go.sum; cat audit/normalize.go audit/sink.go audit/fanout.go audit/transaction.go audit/summary.go service/dispatch.go service/replay.go journal/file.go'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/fanout.go",
  "symbol": "Write"
}
9✓ ✓
/bin/zsh -lc 'gofmt -w audit/transaction.go && \
go test ./... && \
git diff --exit-code -- go.mod go.sum'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/transaction.go",
  "symbol": "(*BufferedSink).Commit"
}
10✓ ✓
/bin/zsh -lc 'cat audit/summary.go audit/normalize.go && \
git diff --exit-code -- go.mod go.sum && \
git status --short --untracked-files=no'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
11✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go && \
go test ./audit ./service && \
git diff --exit-code -- go.mod go.sum'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
12—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "journal/file.go",
  "symbol": "(*JournalSink).Write"
}
13—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/sink.go",
  "symbol": "(*MemorySink).Write"
}
14—✓ ✓
/bin/zsh -lc 'cat audit/event_test.go audit/summary_test.go service/dispatch_test.go api/router.go api/router_test.go journal/file.go'
15—? ✗
/bin/zsh -lc "cat audit/transaction_test.go audit/fanout_test.go legacy/export_test.go metrics/writer_test.go service/replay.go && \
rg -n 'Classify\\(|\\.Write\\(|DispatchAll|Replay|BufferedSink|FanoutSink' --glob '*_test.go'"
16—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "source": "func NormalizeEvent(event Event) Event {\n\tevent.Kind = strings.ToLower(strings.TrimSpace(event.Kind))\n\treturn event\n}"
}
17—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "kind = NormalizeEvent(Event{Kind: kind}).Kind\n\tswitch kind {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
18—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event = NormalizeEvent(event)\n\tm.Events = append(m.Events, event)\n\treturn nil",
  "file": "audit/sink.go",
  "symbol": "(*MemorySink).Write"
}
19—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event = NormalizeEvent(event)\n\tif err := f.First.Write(event); err != nil {\n\t\treturn err\n\t}\n\treturn f.Second.Write(event)",
  "file": "audit/fanout.go",
  "symbol": "(FanoutSink).Write"
}
20—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "for _, event := range b.queue {\n\t\tif err := b.Sink.Write(NormalizeEvent(event)); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "audit/transaction.go",
  "symbol": "(*BufferedSink).Commit"
}
21—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event = audit.NormalizeEvent(event)\n\tj.Entries = append(j.Entries, event)\n\treturn nil",
  "file": "journal/file.go",
  "symbol": "(*JournalSink).Write"
}
22—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "return d.sink.Write(audit.NormalizeEvent(event))",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
23—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "writeEvent := audit.Sink.Write\n\tfor _, event := range events {\n\t\tif err := writeEvent(sink, audit.NormalizeEvent(event)); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
24—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "writeEvent := audit.Sink.Write\n\tfor _, event := range events {\n\t\tif err := writeEvent(sink, audit.NormalizeEvent(event)); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/replay.go",
  "symbol": "Replay"
}
25—? ✗
/bin/zsh -lc 'make check'
26—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
27—✓ ✓
/bin/zsh -lc 'go test ./...'
28—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff --stat && \
git diff -- go.mod go.sum && \
git diff --name-only'
29—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go audit/fanout.go audit/transaction.go audit/sink.go journal/file.go service/dispatch.go service/replay.go && \
git ls-files audit/normalize.go service/dispatch.go && \
git status --porcelain=v1 --untracked-files=all'
30—✓ ✓
/bin/zsh -lc "git status --short --untracked-files=no && \
git diff -- go.mod go.sum && \
rg -n \"NormalizeEvent|Classify\\(|probe|func .*Write|func .*Record|func Replay|func DispatchAll\" audit service journal metrics legacy"
31—semedit/semantic_outline (✓ ✓)
{
  "path": "audit"
}
32—semedit/semantic_outline (✓ ✓)
{
  "path": "service"
}
33—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
34—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --quiet -- go.mod go.sum'
35—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify"
}
36—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "audit"
}
37—✓ ✓
/bin/zsh -lc 'git diff --quiet -- go.mod go.sum && \
go test ./audit'
38—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
39—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "service"
}
40—✓ ✓
/bin/zsh -lc 'git diff --quiet -- go.mod go.sum && \
go test ./service'
41—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "auto_organize_imports": true,
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
42—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event.Kind = NormalizeKind(event.Kind)\n\treturn event",
  "file": "audit/normalize.go",
  "symbol": "NormalizeEvent"
}
43—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "kind = NormalizeKind(kind)\n\tswitch kind {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
44—semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
45—semedit/semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "."
}
46—✓ ✓
/bin/zsh -lc 'go test ./... && \
git diff --quiet -- go.mod go.sum'

prefer_discover_semedit prompt · descriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantprefer_discover_semedit
MCP server instructionsdescriptive
Semedit arm restrictionwrite
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Before editing, inspect the complete available tool inventory, including deferred or lazy tools. If applicable semantic editing tools are callable, prefer them for source mutations. Audit event kinds may contain inconsistent whitespace and casing. Normalize them at the delivery boundary before persistence, and treat probe events as control traffic. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency166.35s205.50s+23.5%158.02s——
Process Start → First Event0.15s0.23s—0.15s——
First Event → First Tool Call10.65s10.12s—5.05s——
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns550%5——
Internal Tool Cycles1332+146.2%14——
Initial Load / Discovery Turns330%3——
MCP Discovery / Schema Turns00—0——
Tool Invocations (per run)1332+146.2%14——
Output Tokens1923812788-33.5%21191——
Reasoning / Thinking Tokens88134988-43.4%7458——
Total Input Tokens30437222831293-7.0%2645338——
Cached Input Tokens28533762657536-6.9%2473472——
Uncached Input Tokens190346173757-8.7%171866——
Cost1.53981.3210-14.2%1.4061——
Cached vs Uncached Token Ratio14.99:115.29:1+2.0%14.39:1——
Oracle L1: Mutation PolicyPASSPASS—PASS——
Oracle L2: AST InvariantsPASSPASS—PASS——
Oracle L3: Clean BuildPASSPASS—PASS——
Oracle L4: Verification TestFAILFAIL—PASS——
Semantic Tool Invocation Verified (per run)N/APASS—N/A——
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 11 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 9 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' -g '*.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'go.mod' -g 'go.sum' | head -150"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'README*' -g 'Makefile' -g '*.go' -g 'go.mod' -g 'go.sum'"
2✓ ✓
/bin/zsh -lc "cat AGENTS.override.md && \
cat go.mod && \
rg -n \"audit|probe|Event|Deliver|Persist|Sink\" --glob '*.go' --glob '"'!**/*_test.go'"' ."
✓ ✓
/bin/zsh -lc "git status --short && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README*' -g 'Makefile' -g '*.go' -g '"'!**/*_test.go'"'"
3✓ ✓
/bin/zsh -lc 'cat audit/event.go audit/normalize.go audit/sink.go audit/summary.go service/dispatch.go metrics/writer.go legacy/export.go'
✓ ✓
/bin/zsh -lc 'cat audit/event.go audit/normalize.go audit/sink.go service/dispatch.go metrics/writer.go legacy/export.go cmd/migrate/main.go'
4✓ ✓
/bin/zsh -lc 'cat audit/event_test.go audit/summary_test.go service/dispatch_test.go metrics/writer_test.go legacy/export_test.go'
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Record"
}
5✓ ✓
/bin/zsh -lc "rg --files -g 'Makefile' -g 'PROJECT_BRIEF.md' -g '*.md' -g 'go.mod' -g 'go.sum' && \
git status --short"
? ✗
/bin/zsh -lc "cat go.mod && \
rg --files -g 'Makefile' -g 'PROJECT_BRIEF.md' -g 'README*' && \
find . -maxdepth 2 -type f | sort"
6✓ ✓
/bin/zsh -lc "cat docs/audit-sinks.md; git diff -- audit/normalize.go audit/summary.go service/dispatch.go; cat > /tmp/atomic_format.py <<'PY'
from pathlib import Path
import os, subprocess, tempfile, time
for name in ('audit/normalize.go', 'audit/summary.go', 'service/dispatch.go'):
    path = Path(name)
    source = path.read_bytes()
    formatted = subprocess.run(['gofmt'], input=source, check=True, stdout=subprocess.PIPE).stdout
    if formatted == source:
        continue
    previous = path.stat()
    fd, tmp_name = tempfile.mkstemp(prefix=path.name + '.', dir=path.parent)
    try:
        with os.fdopen(fd, 'wb') as tmp:
            tmp.write(formatted)
            tmp.flush()
            os.fsync(tmp.fileno())
        os.replace(tmp_name, path)
        newer_ns = max(time.time_ns(), previous.st_mtime_ns + 1_000_000)
        os.utime(path, ns=(previous.st_atime_ns, newer_ns))
        dir_fd = os.open(path.parent, os.O_RDONLY)
        try:
            os.fsync(dir_fd)
        finally:
            os.close(dir_fd)
    finally:
        if os.path.exists(tmp_name):
            os.unlink(tmp_name)
PY
python3 /tmp/atomic_format.py && \
git diff --check && \
go test ./..."
semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "audit/normalize.go",
  "source": "func Normalize(event Event) (Event, bool) {\n\tevent.Kind = strings.ToLower(strings.TrimSpace(event.Kind))\n\treturn event, event.Kind == \"probe\"\n}"
}
7✓ ✓
/bin/zsh -lc 'git diff --check && \
git status --short && \
git diff --stat && \
git diff -- go.mod go.sum && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go && \
test ! -e Makefile'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "event, probe := audit.Normalize(event)\n\tif probe {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
8✓ ✓
/bin/zsh -lc 'cat audit/normalize.go audit/summary.go service/dispatch.go; git rev-parse --show-toplevel; git status --short --untracked-files=all -- audit/normalize.go audit/summary.go service/dispatch.go; git diff HEAD -- audit/normalize.go audit/summary.go service/dispatch.go'
semedit/semantic_replace_body (✓ ✓)
{
  "auto_organize_imports": true,
  "body": "writeEvent := audit.Sink.Write\n\tfor _, event := range events {\n\t\tevent, probe := audit.Normalize(event)\n\t\tif probe {\n\t\t\tcontinue\n\t\t}\n\t\tif err := writeEvent(sink, event); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil",
  "file": "service/dispatch.go",
  "symbol": "DispatchAll"
}
9✓ ✓
/bin/zsh -lc 'git status --short --untracked-files=all && \
git ls-files -v -- audit/normalize.go audit/summary.go service/dispatch.go go.mod go.sum && \
git diff --no-ext-diff -- audit/normalize.go audit/summary.go service/dispatch.go go.mod go.sum && \
cat audit/normalize.go audit/summary.go service/dispatch.go'
✓ ✓
/bin/zsh -lc 'git diff -- audit/normalize.go service/dispatch.go && \
git status --short && \
go test ./...'
10✓ ✓
/bin/zsh -lc 'cat docs/audit-sinks.md && \
cat audit/normalize.go && \
git diff --exit-code -- go.mod go.sum; git diff --check; go test ./...'
✓ ✓
/bin/zsh -lc 'cat audit/normalize.go service/dispatch.go && \
git status --short && \
git diff --stat && \
git diff -- audit/normalize.go service/dispatch.go'
11✓ ✓
/bin/zsh -lc 'cat audit/summary.go audit/normalize.go && \
git diff --check && \
git diff --exit-code -- go.mod go.sum'
semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
12✓ ✓
/bin/zsh -lc 'cat service/dispatch.go audit/normalize.go && \
git diff --check && \
git diff --exit-code -- go.mod go.sum'
✓ ✓
/bin/zsh -lc 'git status --short -- go.mod go.sum'
13✓ ✓
/bin/zsh -lc "python3 - <<'PY'
from pathlib import Path
import os, tempfile, time

updates = {
    Path('audit/normalize.go'): '''// Package audit owns normalization rules used before events reach persistence.\\npackage audit\\n\\nimport \"strings\"\\n\\nfunc NormalizeKind(kind string) string {\\n\\treturn strings.ToLower(strings.TrimSpace(kind))\\n}\\n\\nfunc Normalize(event Event) Event {\\n\\tevent.Kind = NormalizeKind(event.Kind)\\n\\treturn event\\n}\\n''',
    Path('audit/summary.go'): '''// Package audit groups delivery kinds for operational reporting.\\npackage audit\\n\\nfunc Classify(kind string) string {\\n\\tswitch NormalizeKind(kind) {\\n\\tcase \"health\", \"probe\":\\n\\t\\treturn \"control\"\\n\\tdefault:\\n\\t\\treturn \"data\"\\n\\t}\\n}\\n''',
    Path('service/dispatch.go'): '''// Package service coordinates application work with the configured audit sink.\\npackage service\\n\\nimport \"example.com/auditapp/audit\"\\n\\ntype Dispatcher struct {\\n\\tsink audit.Sink\\n}\\n\\nfunc NewDispatcher(sink audit.Sink) *Dispatcher {\\n\\treturn &Dispatcher{sink: sink}\\n}\\n\\nfunc (d *Dispatcher) Record(event audit.Event) error {\\n\\tevent.Kind = audit.NormalizeKind(event.Kind)\\n\\treturn deliver(d.sink, event)\\n}\\n\\nfunc DispatchAll(sink audit.Sink, events []audit.Event) error {\\n\\tfor _, event := range events {\\n\\t\\tif err := deliver(sink, event); err "'!= nil {'"\\n\\t\\t\\treturn err\\n\\t\\t}\\n\\t}\\n\\treturn nil\\n}\\n\\nfunc deliver(sink audit.Sink, event audit.Event) error {\\n\\tevent = audit.Normalize(event)\\n\\tif event.Kind == \"probe\" {\\n\\t\\treturn nil\\n\\t}\\n\\treturn sink.Write(event)\\n}\\n''',
}

for path, content in updates.items():
    previous = path.stat()
    fd, tmp_name = tempfile.mkstemp(prefix=path.name + '.', dir=path.parent)
    try:
        with os.fdopen(fd, 'w') as tmp:
            tmp.write(content)
            tmp.flush()
            os.fsync(tmp.fileno())
        os.replace(tmp_name, path)
        os.utime(path, ns=(previous.st_atime_ns, max(time.time_ns(), previous.st_mtime_ns + 1_000_000)))
        dir_fd = os.open(path.parent, os.O_RDONLY)
        try:
            os.fsync(dir_fd)
        finally:
            os.close(dir_fd)
    finally:
        if os.path.exists(tmp_name):
            os.unlink(tmp_name)
PY
gofmt -w audit/normalize.go audit/summary.go service/dispatch.go
git diff --check
git diff --exit-code -- go.mod go.sum
go test ./..."
✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- go.mod go.sum && \
cat audit/event.go audit/normalize.go audit/sink.go service/dispatch.go go.mod'
14—✓ ✓
/bin/zsh -lc 'cat audit/event_test.go audit/summary.go audit/summary_test.go service/dispatch_test.go'
15—✓ ✓
/bin/zsh -lc 'go test ./...'
16—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
17—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- go.mod go.sum && \
cat audit/summary.go audit/normalize.go audit/event.go'
18—semedit/semantic_replace_body (✓ ✓)
{
  "body": "normalized, _ := Normalize(Event{Kind: kind})\n\tswitch normalized.Kind {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
19—✓ ✓
/bin/zsh -lc 'go test ./...'
20—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
21—✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum && \
git status --short'
22—✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum && \
cat service/dispatch.go'
23—✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum && \
git status --short'
24—✓ ✓
/bin/zsh -lc 'go test ./...'
25—semedit/semantic_verify (✓ ✓)
{
  "path": "."
}
26—✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum && \
cat audit/normalize.go audit/summary.go service/dispatch.go'
27—semedit/semantic_insert_function (✓ ✓)
{
  "access_modifier": "public",
  "file": "audit/normalize.go",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
28—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = NormalizeKind(event.Kind)\n\treturn event, event.Kind == \"probe\"",
  "file": "audit/normalize.go",
  "symbol": "Normalize"
}
29—semedit/semantic_replace_body (✓ ✓)
{
  "body": "normalized := NormalizeKind(kind)\n\tswitch normalized {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}",
  "file": "audit/summary.go",
  "symbol": "Classify"
}
30—semedit/semantic_replace_body (✓ ✓)
{
  "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\tif event.Kind == \"probe\" {\n\t\treturn nil\n\t}\n\treturn d.sink.Write(event)",
  "file": "service/dispatch.go",
  "symbol": "(*Dispatcher).Record"
}
31—✓ ✓
/bin/zsh -lc 'git diff -- go.mod go.sum && \
git status --short'
32—✓ ✓
/bin/zsh -lc 'go test ./...'
Baseline vs Semedit Agentic in Large Context
  • Vanilla (Baseline vs Semedit Agentic in Large Context) Edit: File audit/normalize.go modified (2 lines -> 18 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'README.md' -g '*.go' -g 'AGENTS.md' -g 'AGENTS.override.md' | head -100"
not published
2✓ ✓
/bin/zsh -lc "rg --files && \
git status --short && \
cat AGENTS.override.md && \
sed -n '1,220p' audit/event.go && \
sed -n '1,220p' audit/normalize.go && \
sed -n '1,220p' audit/sink.go && \
sed -n '1,220p' audit/fanout.go"
not published
3✓ ✓
/bin/zsh -lc "cat docs/audit-sinks.md && \
sed -n '1,240p' service/dispatch.go && \
sed -n '1,240p' service/replay.go && \
sed -n '1,220p' audit/transaction.go && \
sed -n '1,220p' audit/event_test.go && \
rg -n \"\\.Write\\(|Sink|probe|Kind\" --glob '*.go' ."
not published
4✓ ✓
/bin/zsh -lc 'cat journal/file.go && \
cat metrics/writer.go && \
cat legacy/export.go && \
cat service/dispatch_test.go && \
cat go.mod'
not published
5✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go && \
git diff --check && \
git status --short && \
git diff -- audit/normalize.go audit/fanout.go audit/transaction.go service/dispatch.go service/replay.go'
not published
6✓ ✓
/bin/zsh -lc "cat audit/normalize.go && \
sed -n '1,100p' audit/fanout.go && \
git status --short --untracked-files=no && \
git diff --stat && \
git diff -- service/dispatch.go"
not published
7? ✗
/bin/zsh -lc 'git rev-parse --show-toplevel && \
git ls-files -v audit/normalize.go service/dispatch.go && \
git show HEAD:audit/normalize.go && \
git show HEAD:service/dispatch.go && \
git status --porcelain=v2'
not published
8? ✗
/bin/zsh -lc 'go test ./... && \
git diff --check && \
git diff --no-index /dev/null audit/normalize.go; git diff --no-index /dev/null audit/fanout.go; git diff --no-index /dev/null service/dispatch.go; git diff --no-index /dev/null service/replay.go; git diff --no-index /dev/null audit/transaction.go'
not published
9✓ ✓
/bin/zsh -lc "sed -n '1,200p' audit/normalize.go && \
sed -n '1,120p' audit/fanout.go && \
sed -n '1,120p' audit/transaction.go && \
sed -n '1,100p' service/dispatch.go && \
sed -n '1,100p' service/replay.go && \
git status --short --ignored | head -80 && \
git diff -- go.mod go.sum"
not published
10✓ ✓
/bin/zsh -lc "rg -n \"Classify|classif|probe|control|Deliver|normalize\" --glob '*.go' --glob '*.md' . && \
sed -n '1,180p' audit/normalize.go && \
sed -n '1,160p' audit/event.go && \
sed -n '1,160p' docs/audit-sinks.md && \
git diff -- go.mod go.sum"
not published
11✓ ✓
/bin/zsh -lc 'cat audit/summary.go && \
cat audit/summary_test.go && \
cat audit/event_test.go && \
cat metrics/writer.go && \
cat legacy/export.go'
not published
12✓ ✓
/bin/zsh -lc 'gofmt -w audit/summary.go && \
git diff --check && \
git diff --no-index /dev/null audit/summary.go; git diff -- go.mod go.sum'
not published
13✓ ✓
/bin/zsh -lc "sed -n '1,120p' audit/normalize.go && \
sed -n '1,140p' service/dispatch.go && \
sed -n '1,100p' service/replay.go && \
git diff -- go.mod go.sum && \
git diff --check"
not published
14✓ ✓
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go audit/fanout.go service/dispatch.go && \
go test ./... && \
git diff --check && \
git diff -- go.mod go.sum'
not published
Benchmark run gpt-6-luna-all-r10-c5-prescriptive-postfix-20260927Benchmark run gpt-6-luna-all-r5-c10-none-20260926