Skip to content
Documentation
Benchmarks
Benchmark runs
Benchmark run hard-smoke-luna6-medium-20261011T105400Z

Benchmark run hard-smoke-luna6-medium-20261011T105400Z

This page contains exactly the publishable benchmark observations recorded in run hard-smoke-luna6-medium-20261011T105400Z; it does not select or aggregate them. Return to the run index or view aggregate metrics.

Interactive browser

Rows are pre-aggregated by task, target, prompt variant, MCP-instruction mode, context, arm, and restriction. Numeric values show median (p50), IQR, p90, maximum, and sample count. Boolean checks show pass counts and success rates. Filters apply to these summary groups; per-run observations remain available on their run pages.

Loading benchmark results…

Baseline vs semedit

Comparison evaluates baseline and semedit values within each restriction policy, target, and context group. Negative changes mark improvements for these lower-is-better metrics; positive changes mark degradations.

Detailed observations

Empirical telemetry measuring wall-clock latency, token usage, tool turns, and compiler correctness when AI coding agents perform code modifications.

Every benchmark pairs Vanilla LLM (standard file editing tools) against Semedit MCP (deterministic AST compiler operations).

Benchmark Methodology & Transparency

  • Reproducibility & Provenance: Self-contained txtar archives define test scenarios. Links below resolve to commit-anchored GitHub source files for published commits, or cryptographic SHA-256 fingerprints for uncommitted local fixtures.
  • Model Cost: Tables report model-specific credit consumption where a target model has a declared schedule: (uncached input × input credits + cached input × cached credits + (reasoning + visible output) × output credits) / 1,000,000. Thinking tokens use the output rate. ADR-0043 defines the per-million-token credit rates.
  • Multi-Level Correctness Oracle: Four validation levels evaluate each trial:
    1. Level 1 (Mutation Policy): Restricts file modifications strictly to authorized paths.
    2. Level 2 (AST Invariants): Compiler AST verification of required symbols, imports, and relative declaration ordering.
    3. Level 3 (Clean Build): Strict compilation validation with zero compiler or typecheck errors.
    4. Level 4 (Test Suite): Unit and integration test suite execution.
  • Isolated Sandboxing: Each trial executes in an ephemeral directory with pristine git and module state.

Test case: task-09-composite-refactor

Target: codex/gpt-6-luna/medium

explicit prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-09-composite-refactor
Targetcodex/gpt-6-luna/medium
Prompt variantexplicit
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Run provenanceprovider_transport=codex_app_server_v2
Fixturetestdata/bench/task_09_composite_refactor.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert DefaultConfig() *Config returning Host localhost and ListenPort 8080, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Initial Workspace State (Before Edit)
package config

import (
	"archive/tar"
	"fmt"
)

type Config struct {
	Host string
	Port int
}

func (c *Config) Addr() string {
	return fmt.Sprintf("%s:%d", c.Host, c.Port)
}
MetricVanilla (Small)Semedit Explicit (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency10.62s34.28s+222.9%———
Process Start → First Event0.18s0.03s————
First Event → First Tool Call4.04s19.37s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles24+100.0%———
Initial Load / Discovery Turns10-100.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)24+100.0%———
Output Tokens4011060+164.3%———
Reasoning / Thinking Tokens49306+524.5%———
Total Input Tokens62444276584+342.9%———
Cached Input Tokens42240237312+461.8%———
Uncached Input Tokens2020439272+94.4%———
Cost0.06670.1746+161.8%———
Cached vs Uncached Token Ratio2.09:16.04:1+189.0%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Explicit in Small Context
  • Vanilla (Baseline vs Semedit Explicit in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
  • Semedit MCP (Baseline vs Semedit Explicit in Small Context) Edit: File config/config.go modified (15 lines -> 16 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "sed -n '1,240p' config/config.go && \
git status --short"
semedit/semantic_outline (✓ ✓)
{
  "path": "config"
}
2file_change (✓ ✓)semedit/semantic_lookup (✓ ✓)
{
  "file": "config/config.go",
  "symbol": "Config.Port",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-09-composite-refactor_1791716025587377000"
}
3—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "symbol": "Config.Addr",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-09-composite-refactor_1791716025587377000"
}
4—semedit/semantic_batch (✓ ✓)
{
  "edits": [
    {
      "params": {
        "auto_organize_imports": false,
        "file": "config/config.go",
        "symbol": "Config.Port",
        "to": "ListenPort"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "file": "config/config.go",
        "placement": "public_end",
        "source": "func DefaultConfig() *Config { return \u0026Config{Host: \"localhost\", ListenPort: 8080} }"
      },
      "tool": "semantic_insert_function"
    },
    {
      "params": {
        "file": "config/config.go"
      },
      "tool": "semantic_organize_imports"
    }
  ],
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-09-composite-refactor_1791716025587377000"
}

Semedit integration feedback (agent opinion, excluded from measurements):

Integration feedback unavailable: codex follow-up changed the original workspace, target, or arm

Baseline vs Semedit Explicit in Large Context
#VanillaSemedit MCP
1not publishednot published

implicit prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-09-composite-refactor
Targetcodex/gpt-6-luna/medium
Prompt variantimplicit
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Run provenanceprovider_transport=codex_app_server_v2
Fixturetestdata/bench/task_09_composite_refactor.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert DefaultConfig() *Config returning Host localhost and ListenPort 8080, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert DefaultConfig() *Config returning Host localhost and ListenPort 8080, and clean up unused imports. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package config

import (
	"archive/tar"
	"fmt"
)

type Config struct {
	Host string
	Port int
}

func (c *Config) Addr() string {
	return fmt.Sprintf("%s:%d", c.Host, c.Port)
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency10.62s46.62s+339.2%———
Process Start → First Event0.18s0.02s————
First Event → First Tool Call4.04s3.49s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles210+400.0%———
Initial Load / Discovery Turns110%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)210+400.0%———
Output Tokens4013215+701.7%———
Reasoning / Thinking Tokens491485+2930.6%———
Total Input Tokens62444637777+921.4%———
Cached Input Tokens42240584960+1284.8%———
Uncached Input Tokens2020452817+161.4%———
Cost0.06670.3370+405.3%———
Cached vs Uncached Token Ratio2.09:111.08:1+429.7%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 21 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 18 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "sed -n '1,240p' config/config.go && \
git status --short"
✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'config/config.go' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'SUBOPTIMAL_TOOLS.md'"
2file_change (✓ ✓)semedit/semantic_batch (✓ ✗): Critical result (JSON): {"application_state":"unknown","error":"semantic_batch requires non-empty 'edits' list"} semantic_batch requires non-empty 'edits' list
{
  "edits": [],
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-implicit_task-09-composite-refactor_1791716014919993000"
}
3—semedit/semantic_outline (✓ ✓)
{
  "path": "config",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-implicit_task-09-composite-refactor_1791716014919993000"
}
4—semedit/semantic_batch (✓ ✓)
{
  "edits": [
    {
      "params": {
        "file": "config/config.go",
        "symbol": "Config.Port",
        "to": "ListenPort"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "file": "config/config.go",
        "kind": "function",
        "placement": "file_end",
        "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{Host: \"localhost\", ListenPort: 8080}\n}"
      },
      "tool": "semantic_insert_construct"
    }
  ],
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-implicit_task-09-composite-refactor_1791716014919993000"
}
5—semedit/semantic_batch (✓ ✓)
{
  "edits": [
    {
      "params": {
        "file": "config/config.go"
      },
      "tool": "semantic_organize_imports"
    }
  ],
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-implicit_task-09-composite-refactor_1791716014919993000"
}
6—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "symbol": "Config",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-implicit_task-09-composite-refactor_1791716014919993000"
}
7—file_change (✓ ✓)
8—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- config/config.go docs/SUBOPTIMAL_TOOLS.md'
9—✓ ✓
/bin/zsh -lc 'pwd; git rev-parse --show-toplevel; git status --short -- config/config.go docs/SUBOPTIMAL_TOOLS.md; git diff -- config/config.go'
10—✗ ?: # internal/msan compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/unsafeheader compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/asan compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/profilerecord compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/runtime/pprof/label compile: version "go1.27.2" does not match go tool version "go1.27.1" # unicode/utf8 compile: version "go1.27.2" does no
/bin/zsh -lc 'go build ./config'

Semedit integration feedback (agent opinion, excluded from measurements):

I’d ask my owner to add semedit: its semantic rename and insertion tools made the refactor straightforward, though I had to use a separate import organizer when automatic cleanup left an unused import.
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

Test case: task-11-mixed-sink-api-migration

Target: codex/gpt-6-luna/medium

explicit prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantexplicit
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Run provenanceprovider_transport=codex_app_server_v2
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)Semedit Explicit (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency23.93s31.60s+32.0%———
Process Start → First Event0.04s0.04s————
First Event → First Tool Call3.38s17.06s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles880%———
Initial Load / Discovery Turns20-100.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)880%———
Output Tokens12851379+7.3%———
Reasoning / Thinking Tokens341374+9.7%———
Total Input Tokens191378294957+54.1%———
Cached Input Tokens171776228096+32.8%———
Uncached Input Tokens1960266861+241.1%———
Cost0.11230.2461+119.2%———
Cached vs Uncached Token Ratio8.76:13.41:1-61.1%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Explicit in Small Context
  • Vanilla (Baseline vs Semedit Explicit in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
  • Semedit MCP (Baseline vs Semedit Explicit in Small Context) Edit: File audit/normalize.go modified (2 lines -> 6 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'go.sum' -g 'audit/**' -g 'Makefile' -g 'README*'"
semedit/semantic_outline (✓ ✓)
{
  "path": ".",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-11-mixed-sink-api-migration_1791716085756993000"
}
2✓ ✓
/bin/zsh -lc "git status --short && \
rg -n \"Record\\(|health|probe|Kind|NormalizeKind\" audit"
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Dispatcher.Record",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-11-mixed-sink-api-migration_1791716085756993000"
}
3✓ ✓
/bin/zsh -lc "cat audit/normalize.go && \
cat audit/summary.go && \
cat audit/sink.go && \
cat audit/event.go && \
rg -n \"type Dispatcher|func \\(.*Dispatcher.*Record|Dispatcher\" . -g '*.go'"
semedit/semantic_lookup (✓ ✗): Critical result (JSON): {"application_state":"unknown","error":"resolve NormalizeKind: symbol not found"} resolve NormalizeKind: symbol not found
{
  "symbol": "NormalizeKind",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-11-mixed-sink-api-migration_1791716085756993000"
}
4✓ ✓
/bin/zsh -lc 'cat service/dispatch.go && \
cat service/dispatch_test.go && \
cat Makefile 2>/dev/null || \
true && \
cat go.mod'
semedit/semantic_lookup (✓ ✓)
{
  "symbol": "Classify",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-11-mixed-sink-api-migration_1791716085756993000"
}
5file_change (✓ ✓)semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-11-mixed-sink-api-migration_1791716085756993000"
}
6✗ ?: # internal/godebugs compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/trace/tracev2 compile: version "go1.27.2" does not match go tool version "go1.27.1" # unicode/utf8 compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/unsafeheader compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/goarch compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/profilerecord compile: version "go1.27.2" does no
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./... && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go && \
git status --short -- go.mod go.sum audit service'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Dispatcher",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-11-mixed-sink-api-migration_1791716085756993000"
}
7✓ ✓
/bin/zsh -lc 'git diff -- audit/normalize.go audit/summary.go service/dispatch.go; git status --short -- go.mod go.sum audit service'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Dispatcher.Record",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-11-mixed-sink-api-migration_1791716085756993000"
}
8✗ ?: // Package audit leaves room for normalization rules owned by the delivery boundary. package audit import "strings" func NormalizeKind(kind string) string { return strings.ToLower(strings.TrimSpace(kind)) } // Package audit groups delivery kinds for operational reporting. package audit func Classify(kind string) string { switch NormalizeKind(kind) { case "health", "probe": return "control" default: return "data" } } func NewDispatcher(sink audit.Sink) *Dispatcher { return &Dispatcher{sink: sink} } func (d *
/bin/zsh -lc "cat audit/normalize.go && \
cat audit/summary.go && \
sed -n '10,22p' service/dispatch.go && \
git status --short --untracked-files=all . && \
git diff --no-index /dev/null audit/normalize.go"
semedit/semantic_batch (✓ ✓)
{
  "edits": [
    {
      "params": {
        "file": "audit/normalize.go",
        "source": "func NormalizeKind(kind string) string { return strings.ToLower(strings.TrimSpace(kind)) }"
      },
      "tool": "semantic_insert_function"
    },
    {
      "params": {
        "body": "event.Kind = audit.NormalizeKind(event.Kind)\nreturn d.sink.Write(event)",
        "expected_revision": "550fd5283aef8cc729610e7e44cbc24e1e152985732114c581e243ab32319079",
        "file": "service/dispatch.go",
        "symbol": "Dispatcher.Record"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "body": "switch NormalizeKind(kind) {\ncase \"health\", \"probe\":\n\treturn \"control\"\ndefault:\n\treturn \"data\"\n}",
        "expected_revision": "7daa017bf7a2e91b43352a940b8f60d124deb68710bd1cefb59460432ce88f2e",
        "file": "audit/summary.go",
        "symbol": "Classify"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "file": "audit/normalize.go"
      },
      "tool": "semantic_organize_imports"
    }
  ],
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-11-mixed-sink-api-migration_1791716085756993000"
}

Semedit integration feedback (agent opinion, excluded from measurements):

Integration feedback unavailable: codex follow-up changed the original workspace, target, or arm

Baseline vs Semedit Explicit in Large Context
#VanillaSemedit MCP
1not publishednot published

implicit prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetcodex/gpt-6-luna/medium
Prompt variantimplicit
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Run provenanceprovider_transport=codex_app_server_v2
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency23.93s46.76s+95.4%———
Process Start → First Event0.04s0.02s————
First Event → First Tool Call3.38s3.96s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles817+112.5%———
Initial Load / Discovery Turns23+50.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)817+112.5%———
Output Tokens12852689+109.3%———
Reasoning / Thinking Tokens3411058+210.3%———
Total Input Tokens191378566873+196.2%———
Cached Input Tokens171776520448+203.0%———
Uncached Input Tokens1960246425+136.8%———
Cost0.11230.2930+161.0%———
Cached vs Uncached Token Ratio8.76:111.21:1+27.9%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'go.sum' -g 'audit/**' -g 'Makefile' -g 'README*'"
✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'audit/**' -g 'AGENTS.md' -g 'AGENTS.override.md' -g 'Makefile' | sort"
2✓ ✓
/bin/zsh -lc "git status --short && \
rg -n \"Record\\(|health|probe|Kind|NormalizeKind\" audit"
✓ ✓
/bin/zsh -lc "git status --short && \
rg --files -g 'PROJECT_BRIEF.md' -g 'Makefile' -g 'go.mod' -g 'audit/*.go' | sort"
3✓ ✓
/bin/zsh -lc "cat audit/normalize.go && \
cat audit/summary.go && \
cat audit/sink.go && \
cat audit/event.go && \
rg -n \"type Dispatcher|func \\(.*Dispatcher.*Record|Dispatcher\" . -g '*.go'"
✓ ✓
/bin/zsh -lc "cat audit/normalize.go && \
sed -n '1,240p' audit/event.go && \
sed -n '1,240p' audit/sink.go"
4✓ ✓
/bin/zsh -lc 'cat service/dispatch.go && \
cat service/dispatch_test.go && \
cat Makefile 2>/dev/null || \
true && \
cat go.mod'
✓ ✓
/bin/zsh -lc "rg -n 'type Dispatcher|func .*Record|health|probe|control' audit"
5file_change (✓ ✓)semedit/semantic_outline (✓ ✓)
{
  "path": "."
}
6✗ ?: # internal/godebugs compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/trace/tracev2 compile: version "go1.27.2" does not match go tool version "go1.27.1" # unicode/utf8 compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/unsafeheader compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/goarch compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/profilerecord compile: version "go1.27.2" does no
/bin/zsh -lc 'gofmt -w audit/normalize.go audit/summary.go service/dispatch.go && \
go test ./... && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go && \
git status --short -- go.mod go.sum audit service'
✓ ✓
/bin/zsh -lc "rg -n 'Dispatcher|Record\\(' --glob '*.go' ."
7✓ ✓
/bin/zsh -lc 'git diff -- audit/normalize.go audit/summary.go service/dispatch.go; git status --short -- go.mod go.sum audit service'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "symbol": "Dispatcher.Record"
}
8✗ ?: // Package audit leaves room for normalization rules owned by the delivery boundary. package audit import "strings" func NormalizeKind(kind string) string { return strings.ToLower(strings.TrimSpace(kind)) } // Package audit groups delivery kinds for operational reporting. package audit func Classify(kind string) string { switch NormalizeKind(kind) { case "health", "probe": return "control" default: return "data" } } func NewDispatcher(sink audit.Sink) *Dispatcher { return &Dispatcher{sink: sink} } func (d *
/bin/zsh -lc "cat audit/normalize.go && \
cat audit/summary.go && \
sed -n '10,22p' service/dispatch.go && \
git status --short --untracked-files=all . && \
git diff --no-index /dev/null audit/normalize.go"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "symbol": "Classify"
}
9—semedit/semantic_insert_construct (✓ ✓)
{
  "file": "audit/normalize.go",
  "kind": "function",
  "placement": "file_end",
  "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
}
10—✓ ✓
/bin/zsh -lc 'cat audit/normalize.go'
11—semedit/semantic_insert_construct (✓ ✓)
{
  "anchor": "start",
  "file": "service/dispatch.go",
  "in_function": "Dispatcher.Record",
  "kind": "statement/expression",
  "source": "event.Kind = audit.NormalizeKind(event.Kind)"
}
12—semedit/semantic_replace_construct (✓ ✓)
{
  "file": "audit/summary.go",
  "kind": "function",
  "source": "func Classify(kind string) string {\n\tswitch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n\t}\n}",
  "symbol": "Classify"
}
13—✓ ✓
/bin/zsh -lc "rg --files -g 'Makefile' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'go.sum' -g 'README.md' | sort && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go go.mod go.sum && \
git status --short -- audit/normalize.go audit/summary.go service/dispatch.go go.mod go.sum audit/*_test.go"
14—✗ ?
/bin/zsh -lc 'go test ./... && \
git diff --check && \
git status --short -- audit/normalize.go audit/summary.go service/dispatch.go go.mod go.sum audit/event_test.go audit/summary_test.go service/dispatch_test.go && \
git diff -- audit/normalize.go audit/summary.go service/dispatch.go'
15—✓ ✓
/bin/zsh -lc 'git diff --check; git status --short -- audit/normalize.go audit/summary.go service/dispatch.go go.mod go.sum audit/event_test.go audit/summary_test.go service/dispatch_test.go; git diff -- audit/normalize.go audit/summary.go service/dispatch.go; go version'
16—✓ ✓
/bin/zsh -lc "git rev-parse --show-toplevel && \
git status --short --untracked-files=all | rg 'audit/|service/|go\\.mod|go\\.sum' || \
true && \
cat audit/normalize.go && \
cat service/dispatch.go && \
cat audit/summary.go && \
go env GOROOT GOTOOLDIR"
17—✗ ?: ls: .scratch/go/mod/golang.org/toolchain@v0.0.1-go1.27.2.darwin-arm64/bin/go: No such file or directory
/bin/zsh -lc 'ls -l .scratch/go/mod/golang.org/toolchain@v0.0.1-go1.27.2.darwin-arm64/bin/go && \
.scratch/go/mod/golang.org/toolchain@v0.0.1-go1.27.2.darwin-arm64/bin/go test ./...'

Semedit integration feedback (agent opinion, excluded from measurements):

I’d ask my owner to add semedit’s tools: they let me make targeted Go changes through semantic operations, though I regret missing the chance to batch them and couldn’t verify the result because of a Go toolchain mismatch.
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

Test case: task-12-request-identity-migration

Target: codex/gpt-6-luna/medium

explicit prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-12-request-identity-migration
Targetcodex/gpt-6-luna/medium
Prompt variantexplicit
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Run provenanceprovider_transport=codex_app_server_v2
Fixturetestdata/bench/task_12_request_identity_migration.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Initial Workspace State (Before Edit)
package request

import (
	"errors"
	"strings"
)

type Request struct {
	ID      string
	Kind    string
	Payload []byte
}

func (r Request) Validate() error {
	if r.ID == "" {
		return errors.New("invalid request")
	}
	return nil
}

func Normalize(kind string) string {
	kind = strings.TrimSpace(kind)
	kind = strings.ToLower(kind)
	return kind
}

func FindByID(requests []Request, id string) (Request, bool) {
	for _, r := range requests {
		if r.ID == id {
			return r, true
		}
	}
	return Request{}, false
}
MetricVanilla (Small)Semedit Explicit (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency33.65s157.23s+367.2%———
Process Start → First Event0.05s0.05s————
First Event → First Tool Call3.31s12.03s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles941+355.6%———
Initial Load / Discovery Turns30-100.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)941+355.6%———
Output Tokens219111104+406.8%———
Reasoning / Thinking Tokens6204963+700.5%———
Total Input Tokens2349332458723+946.6%———
Cached Input Tokens2083842335744+1020.9%———
Uncached Input Tokens26549122979+363.2%———
Cost0.15361.0922+611.1%———
Cached vs Uncached Token Ratio7.85:118.99:1+142.0%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Explicit in Small Context
  • Vanilla (Baseline vs Semedit Explicit in Small Context) Edit: File request/request.go modified (34 lines -> 41 lines)
  • Semedit MCP (Baseline vs Semedit Explicit in Small Context) Edit: File request/request.go modified (34 lines -> 41 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g '*.go' -g 'Makefile' -g 'go.mod' -g 'go.sum' | head -200"
semedit/semantic_outline (✓ ✓)
{
  "path": "request",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
2✓ ✓
/bin/zsh -lc "cat go.mod && \
rg -n '\\bRequest\\b|\\.ID\\b|Normalize|Validate|ErrInvalid' --glob '*.go'"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
3✓ ✓
/bin/zsh -lc 'cat request/request.go gateway/handler.go notifications/notifier.go queue/queue.go review/review.go storage/requests.go scheduler/jobs.go correlation/identities.go response/response.go reporting/report.go'
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Validate",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
4file_change (✓ ✓)semedit/semantic_find_references (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request.ID",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
5✓ ✓
/bin/zsh -lc "rg -n '\\.ID\\b|Request\\s*\\{["'^}]*ID:|'"\\.RequestID\\b|normalizeKind|ErrInvalidRequest' --glob '*.go' && \
git status --short && \
gofmt -w request/request.go gateway/handler.go notifications/notifier.go queue/queue.go storage/requests.go scheduler/jobs.go correlation/identities.go reporting/report.go"
semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Normalize",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
6✗ ?: # internal/byteorder compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/coverage/rtcov compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/goarch compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/godebugs compile: version "go1.27.2" does not match go tool version "go1.27.1" # math/bits compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/msan compile: version "go1.27.2" does not match go too
/bin/zsh -lc 'go build ./...'
semedit/semantic_rename (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request.ID",
  "to": "RequestID",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
7✗ ?: # internal/unsafeheader compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/coverage/rtcov compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/godebugs compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/byteorder compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/profilerecord compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/goos compile: version "go1.27.2" do
/bin/zsh -lc 'go test ./...'
semedit/semantic_batch (✓ ✗): Batch child results are intermediate and do not provide final continuation targets. Critical result (JSON): {"result":{"status":"error","results":[{"status":"error","application_state":"unknown","error":"edit 1 params: unknown parameter \"source\" for \"semantic_replace_body\"; supported keys are [auto-organize-imports, auto_organize_imports, body, expected-revision, expected_revision, file, symbol] Required parameters: [body, file, symbol]","tool":"semantic_replace_body"}],"workspace_root":"/Users/alessand
{
  "edits": [
    {
      "params": {
        "file": "request/request.go",
        "placement": "public_end",
        "source": "var ErrInvalidRequest = errors.New(\"invalid request\")"
      },
      "tool": "semantic_insert_decl"
    },
    {
      "params": {
        "file": "request/request.go",
        "source": "if r.RequestID == \"\" {\n\treturn fmt.Errorf(\"%w\", ErrInvalidRequest)\n}\nreturn nil",
        "symbol": "Request.Validate"
      },
      "tool": "semantic_replace_body"
    }
  ],
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
8✓ ✓
/bin/zsh -lc 'git diff -- request/request.go gateway/handler.go notifications/notifier.go queue/queue.go storage/requests.go scheduler/jobs.go correlation/identities.go reporting/report.go && \
git status --short -- go.mod go.sum request/request.go gateway/handler.go notifications/notifier.go queue/queue.go storage/requests.go scheduler/jobs.go correlation/identities.go reporting/report.go'
semedit/semantic_inspect_symbol (✓ ✗): Critical result (JSON): {"application_state":"unknown","error":"resolve ErrInvalidRequest in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000/request/request.go: symbol not found"} resolve ErrInvalidRequest in /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000/request/request.go: symbol not fo
{
  "file": "request/request.go",
  "symbol": "ErrInvalidRequest",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
9✓ ✓
/bin/zsh -lc 'git rev-parse --show-toplevel && \
git status --short && \
git diff --no-index /dev/null request/request.go | head -40'
✓ ✓
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'SUBOPTIMAL_TOOLS.md' -g 'AGENTS*' -g 'request.go' -g 'go.mod' -g 'go.sum'"
10—file_change (✓ ✓)
11—semedit/semantic_batch (✓ ✓)
{
  "edits": [
    {
      "params": {
        "file": "request/request.go",
        "placement": "public_end",
        "source": "var ErrInvalidRequest = errors.New(\"invalid request\")"
      },
      "tool": "semantic_insert_decl"
    },
    {
      "params": {
        "body": "if r.RequestID == \"\" {\n\treturn fmt.Errorf(\"request ID is required: %w\", ErrInvalidRequest)\n}\nreturn nil",
        "file": "request/request.go",
        "symbol": "Request.Validate"
      },
      "tool": "semantic_replace_body"
    }
  ],
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
12—semedit/semantic_batch (✓ ✗): Batch child results are intermediate and do not provide final continuation targets. Critical result (JSON): {"result":{"status":"error","results":[{"status":"error","application_state":"unknown","error":"edit 0 params: unknown parameter \"name\" for \"semantic_extract_function\"; supported keys are [end-line, end_line, file, in-function, in_function, kind, start-line, start_line, symbol] Required parameters: [end_line, file, in_function, kind, start_line, symbol]","tool":"semantic_extract_function"}],"works
{
  "edits": [
    {
      "params": {
        "file": "request/request.go",
        "in_function": "Normalize",
        "name": "normalizeKind",
        "source": "kind = strings.TrimSpace(kind)\nkind = strings.ToLower(kind)\nreturn kind"
      },
      "tool": "semantic_extract_function"
    }
  ],
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
13—file_change (✓ ✓)
14—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Normalize",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
15—semedit/semantic_batch (✓ ✗): Batch child results are intermediate and do not provide final continuation targets. Critical result (JSON): {"result":{"status":"error","results":[{"status":"error","application_state":"unknown","error":"line range 21-23 does not select a contiguous top-level statement region in Normalize","tool":"semantic_extract_function"}],"workspace_root":"/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"}} { "s
{
  "edits": [
    {
      "params": {
        "end_line": 23,
        "file": "request/request.go",
        "in_function": "Normalize",
        "kind": "function",
        "start_line": 21,
        "symbol": "normalizeKind"
      },
      "tool": "semantic_extract_function"
    }
  ],
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
16—file_change (✓ ✓)
17—semedit/semantic_batch (✓ ✗): Batch child results are intermediate and do not provide final continuation targets. Critical result (JSON): {"result":{"status":"error","results":[{"status":"error","application_state":"unknown","error":"cannot extract Normalize: only one leading := declaration with matching named results is supported","tool":"semantic_extract_function"}],"workspace_root":"/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_17917161516619
{
  "edits": [
    {
      "params": {
        "end_line": 24,
        "file": "request/request.go",
        "in_function": "Normalize",
        "kind": "function",
        "start_line": 22,
        "symbol": "normalizeKind"
      },
      "tool": "semantic_extract_function"
    }
  ],
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
18—file_change (✓ ✓)
19—semedit/semantic_batch (✓ ✓)
{
  "edits": [
    {
      "params": {
        "body": "normalized := strings.ToLower(strings.TrimSpace(kind))\nreturn normalized",
        "file": "request/request.go",
        "symbol": "Normalize"
      },
      "tool": "semantic_replace_body"
    }
  ],
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
20—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Normalize",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
21—semedit/semantic_batch (✓ ✗): Batch child results are intermediate and do not provide final continuation targets. Critical result (JSON): {"result":{"status":"error","results":[{"status":"error","application_state":"unknown","error":"cannot extract Normalize: selected statement on line 23 is not a call or supported local declaration","tool":"semantic_extract_function"}],"workspace_root":"/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_179171615166
{
  "edits": [
    {
      "params": {
        "end_line": 23,
        "file": "request/request.go",
        "in_function": "Normalize",
        "kind": "function",
        "start_line": 22,
        "symbol": "normalizeKind"
      },
      "tool": "semantic_extract_function"
    }
  ],
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
22—file_change (✓ ✓)
23—semedit/semantic_batch (✓ ✗): Batch child results are intermediate and do not provide final continuation targets. Critical result (JSON): {"result":{"status":"error","results":[{"status":"error","application_state":"unknown","error":"cannot extract Normalize: cannot infer a safe return type for local declaration expression \"strings.ToLower(strings.TrimSpace(kind))\"","tool":"semantic_extract_function"}],"workspace_root":"/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-requ
{
  "edits": [
    {
      "params": {
        "end_line": 22,
        "file": "request/request.go",
        "in_function": "Normalize",
        "kind": "function",
        "start_line": 22,
        "symbol": "normalizeKind"
      },
      "tool": "semantic_extract_function"
    }
  ],
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
24—file_change (✓ ✓)
25—semedit/semantic_batch (✓ ✓)
{
  "edits": [
    {
      "params": {
        "body": "normalized := kind\nreturn normalized",
        "file": "request/request.go",
        "symbol": "Normalize"
      },
      "tool": "semantic_replace_body"
    }
  ],
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
26—semedit/semantic_batch (✓ ✗): Batch child results are intermediate and do not provide final continuation targets. Critical result (JSON): {"result":{"status":"error","results":[{"status":"error","application_state":"unknown","error":"cannot extract Normalize: selected statement on line 22 is not a call or supported local declaration","tool":"semantic_extract_function"}],"workspace_root":"/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_179171615166
{
  "edits": [
    {
      "params": {
        "end_line": 22,
        "file": "request/request.go",
        "in_function": "Normalize",
        "kind": "function",
        "start_line": 22,
        "symbol": "normalizeKind"
      },
      "tool": "semantic_extract_function"
    }
  ],
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
27—file_change (✓ ✓)
28—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Normalize",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
29—semedit/semantic_batch (✓ ✗): Batch child results are intermediate and do not provide final continuation targets. Critical result (JSON): {"result":{"status":"error","results":[{"status":"error","application_state":"unknown","error":"line range 20-20 does not select a contiguous top-level statement region in Normalize","tool":"semantic_extract_function"}],"workspace_root":"/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"}} { "s
{
  "edits": [
    {
      "params": {
        "end_line": 20,
        "file": "request/request.go",
        "in_function": "Normalize",
        "kind": "function",
        "start_line": 20,
        "symbol": "normalizeKind"
      },
      "tool": "semantic_extract_function"
    }
  ],
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
30—file_change (✓ ✓)
31—semedit/semantic_batch (✓ ✗): Batch child results are intermediate and do not provide final continuation targets. Critical result (JSON): {"result":{"status":"error","results":[{"status":"error","application_state":"unknown","error":"resolve extracted result types: load Go package for extraction: /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000/request/request.go:15:10: undefined: fmt","tool":"semantic_extract_function"}],"works
{
  "edits": [
    {
      "params": {
        "end_line": 21,
        "file": "request/request.go",
        "in_function": "Normalize",
        "kind": "function",
        "start_line": 21,
        "symbol": "normalizeKind"
      },
      "tool": "semantic_extract_function"
    }
  ],
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
32—file_change (✓ ✓)
33—semedit/semantic_batch (✓ ✓)
{
  "edits": [
    {
      "params": {
        "body": "if r.RequestID == \"\" {\n\treturn ErrInvalidRequest\n}\nreturn nil",
        "file": "request/request.go",
        "symbol": "Request.Validate"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "end_line": 21,
        "file": "request/request.go",
        "in_function": "Normalize",
        "kind": "function",
        "start_line": 21,
        "symbol": "normalizeKind"
      },
      "tool": "semantic_extract_function"
    }
  ],
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
34—semedit/semantic_batch (✓ ✓)
{
  "edits": [
    {
      "params": {
        "body": "kind = strings.TrimSpace(kind)\nreturn strings.ToLower(kind)",
        "file": "request/request.go",
        "symbol": "normalizeKind"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "body": "if r.RequestID == \"\" {\n\treturn fmt.Errorf(\"request ID is required: %w\", ErrInvalidRequest)\n}\nreturn nil",
        "file": "request/request.go",
        "symbol": "Request.Validate"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "file": "request/request.go"
      },
      "tool": "semantic_organize_imports"
    }
  ],
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
35—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request.Validate",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
36—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
37—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Normalize",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
38—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "normalizeKind",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
39—semedit/semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "ErrInvalidRequest",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}
40—✓ ✓
/bin/zsh -lc 'git status --short && \
git diff -- go.mod go.sum'
41—semedit/semantic_find_references (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request.RequestID",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit-explicit_small-explicit_task-12-request-identity-migration_1791716151661946000"
}

Semedit integration feedback (agent opinion, excluded from measurements):

Integration feedback unavailable: codex follow-up changed the original workspace, target, or arm

Baseline vs Semedit Explicit in Large Context
#VanillaSemedit MCP
1not publishednot published

implicit prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-12-request-identity-migration
Targetcodex/gpt-6-luna/medium
Prompt variantimplicit
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Run provenanceprovider_transport=codex_app_server_v2
Fixturetestdata/bench/task_12_request_identity_migration.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package request

import (
	"errors"
	"strings"
)

type Request struct {
	ID      string
	Kind    string
	Payload []byte
}

func (r Request) Validate() error {
	if r.ID == "" {
		return errors.New("invalid request")
	}
	return nil
}

func Normalize(kind string) string {
	kind = strings.TrimSpace(kind)
	kind = strings.ToLower(kind)
	return kind
}

func FindByID(requests []Request, id string) (Request, bool) {
	for _, r := range requests {
		if r.ID == id {
			return r, true
		}
	}
	return Request{}, false
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency33.65s47.29s+40.5%———
Process Start → First Event0.05s0.02s————
First Event → First Tool Call3.31s8.32s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles910+11.1%———
Initial Load / Discovery Turns330%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)910+11.1%———
Output Tokens21911846-15.7%———
Reasoning / Thinking Tokens620525-15.3%———
Total Input Tokens234933530110+125.6%———
Cached Input Tokens208384477184+129.0%———
Uncached Input Tokens2654952926+99.4%———
Cost0.15360.2812+83.1%———
Cached vs Uncached Token Ratio7.85:19.02:1+14.9%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 41 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File request/request.go modified (34 lines -> 41 lines)
#VanillaSemedit MCP
1✓ ✓
/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g '*.go' -g 'Makefile' -g 'go.mod' -g 'go.sum' | head -200"
✓ ✓
/bin/zsh -lc 'rg --files'
2✓ ✓
/bin/zsh -lc "cat go.mod && \
rg -n '\\bRequest\\b|\\.ID\\b|Normalize|Validate|ErrInvalid' --glob '*.go'"
✓ ✓
/bin/zsh -lc 'cat request/request.go'
3✓ ✓
/bin/zsh -lc 'cat request/request.go gateway/handler.go notifications/notifier.go queue/queue.go review/review.go storage/requests.go scheduler/jobs.go correlation/identities.go response/response.go reporting/report.go'
✓ ✓
/bin/zsh -lc "rg '\\.ID|\\bID\\b|Normalize|invalid request' --glob '*.go'"
4file_change (✓ ✓)semedit/semantic_rename (✓ ✓)
{
  "file": "request/request.go",
  "symbol": "Request.ID",
  "to": "RequestID",
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-implicit_task-12-request-identity-migration_1791716119194531000"
}
5✓ ✓
/bin/zsh -lc "rg -n '\\.ID\\b|Request\\s*\\{["'^}]*ID:|'"\\.RequestID\\b|normalizeKind|ErrInvalidRequest' --glob '*.go' && \
git status --short && \
gofmt -w request/request.go gateway/handler.go notifications/notifier.go queue/queue.go storage/requests.go scheduler/jobs.go correlation/identities.go reporting/report.go"
semedit/semantic_batch (✓ ✗): Batch child results are intermediate and do not provide final continuation targets. Critical result (JSON): {"result":{"status":"error","results":[{"status":"error","application_state":"unknown","error":"edit 0 params: unknown parameter \"source\" for \"semantic_replace_body\"; supported keys are [auto-organize-imports, auto_organize_imports, body, expected-revision, expected_revision, file, symbol] Required parameters: [body, file, symbol]","tool":"semantic_replace_body"}],"workspace_root":"/Users/alessand
{
  "edits": [
    {
      "file": "request/request.go",
      "source": "if r.RequestID == \"\" {\n\t\treturn fmt.Errorf(\"invalid request: %w\", ErrInvalidRequest)\n\t}\n\treturn nil",
      "symbol": "Request.Validate",
      "tool": "semantic_replace_body"
    },
    {
      "file": "request/request.go",
      "source": "return normalizeKind(kind)",
      "symbol": "Normalize",
      "tool": "semantic_replace_body"
    },
    {
      "file": "request/request.go",
      "placement": "file_end",
      "source": "var ErrInvalidRequest = errors.New(\"invalid request\")",
      "tool": "semantic_insert_decl"
    },
    {
      "file": "request/request.go",
      "placement": "file_end",
      "source": "func normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}",
      "tool": "semantic_insert_function"
    }
  ],
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-implicit_task-12-request-identity-migration_1791716119194531000"
}
6✗ ?: # internal/byteorder compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/coverage/rtcov compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/goarch compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/godebugs compile: version "go1.27.2" does not match go tool version "go1.27.1" # math/bits compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/msan compile: version "go1.27.2" does not match go too
/bin/zsh -lc 'go build ./...'
✓ ✓
/bin/zsh -lc 'rg --files'
7✗ ?: # internal/unsafeheader compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/coverage/rtcov compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/godebugs compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/byteorder compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/profilerecord compile: version "go1.27.2" does not match go tool version "go1.27.1" # internal/goos compile: version "go1.27.2" do
/bin/zsh -lc 'go test ./...'
✓ ✓
/bin/zsh -lc 'cat request/request.go'
8✓ ✓
/bin/zsh -lc 'git diff -- request/request.go gateway/handler.go notifications/notifier.go queue/queue.go storage/requests.go scheduler/jobs.go correlation/identities.go reporting/report.go && \
git status --short -- go.mod go.sum request/request.go gateway/handler.go notifications/notifier.go queue/queue.go storage/requests.go scheduler/jobs.go correlation/identities.go reporting/report.go'
semedit/semantic_batch (✓ ✓)
{
  "edits": [
    {
      "body": "if r.RequestID == \"\" {\n\t\treturn fmt.Errorf(\"invalid request: %w\", ErrInvalidRequest)\n\t}\n\treturn nil",
      "file": "request/request.go",
      "symbol": "Request.Validate",
      "tool": "semantic_replace_body"
    },
    {
      "body": "return normalizeKind(kind)",
      "file": "request/request.go",
      "symbol": "Normalize",
      "tool": "semantic_replace_body"
    },
    {
      "file": "request/request.go",
      "placement": "file_end",
      "source": "var ErrInvalidRequest = errors.New(\"invalid request\")",
      "tool": "semantic_insert_decl"
    },
    {
      "file": "request/request.go",
      "placement": "file_end",
      "source": "func normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}",
      "tool": "semantic_insert_function"
    }
  ],
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-implicit_task-12-request-identity-migration_1791716119194531000"
}
9✓ ✓
/bin/zsh -lc 'git rev-parse --show-toplevel && \
git status --short && \
git diff --no-index /dev/null request/request.go | head -40'
semedit/semantic_batch (✓ ✓)
{
  "auto_organize_imports": true,
  "edits": [
    {
      "file": "request/request.go",
      "tool": "semantic_organize_imports"
    }
  ],
  "workspace_root": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_codex_semedit_small-implicit_task-12-request-identity-migration_1791716119194531000"
}
10—✓ ✓
/bin/zsh -lc "rg '\\.ID|RequestID|ErrInvalidRequest|normalizeKind' --glob '*.go' && \
git status --short"

Semedit integration feedback (agent opinion, excluded from measurements):

Semedit’s semantic rename and edit tools gave me a clear, low-stress way to update request identities across packages while keeping response and review IDs intact, so I’d ask my owner to add them.
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published
Benchmark run gpt-6-luna-all-r5-c10-prescriptive-20260926Benchmark run hard-smoke-saluki-20261011T113200Z