Skip to content
Documentation
Benchmarks
Benchmark runs
Benchmark run hard-smoke-saluki-20261011T113200Z

Benchmark run hard-smoke-saluki-20261011T113200Z

This page contains exactly the publishable benchmark observations recorded in run hard-smoke-saluki-20261011T113200Z; it does not select or aggregate them. Return to the run index or view aggregate metrics.

Interactive browser

Rows are pre-aggregated by task, target, prompt variant, MCP-instruction mode, context, arm, and restriction. Numeric values show median (p50), IQR, p90, maximum, and sample count. Boolean checks show pass counts and success rates. Filters apply to these summary groups; per-run observations remain available on their run pages.

Loading benchmark results…

Baseline vs semedit

Comparison evaluates baseline and semedit values within each restriction policy, target, and context group. Negative changes mark improvements for these lower-is-better metrics; positive changes mark degradations.

Detailed observations

Empirical telemetry measuring wall-clock latency, token usage, tool turns, and compiler correctness when AI coding agents perform code modifications.

Every benchmark pairs Vanilla LLM (standard file editing tools) against Semedit MCP (deterministic AST compiler operations).

Benchmark Methodology & Transparency

  • Reproducibility & Provenance: Self-contained txtar archives define test scenarios. Links below resolve to commit-anchored GitHub source files for published commits, or cryptographic SHA-256 fingerprints for uncommitted local fixtures.
  • Model Cost: Tables report model-specific credit consumption where a target model has a declared schedule: (uncached input × input credits + cached input × cached credits + (reasoning + visible output) × output credits) / 1,000,000. Thinking tokens use the output rate. ADR-0043 defines the per-million-token credit rates.
  • Multi-Level Correctness Oracle: Four validation levels evaluate each trial:
    1. Level 1 (Mutation Policy): Restricts file modifications strictly to authorized paths.
    2. Level 2 (AST Invariants): Compiler AST verification of required symbols, imports, and relative declaration ordering.
    3. Level 3 (Clean Build): Strict compilation validation with zero compiler or typecheck errors.
    4. Level 4 (Test Suite): Unit and integration test suite execution.
  • Isolated Sandboxing: Each trial executes in an ephemeral directory with pristine git and module state.

Test case: task-09-composite-refactor

Target: opencode/amdbeast/Kujira/Underdog-Saluki-27B-1.0-MTP-GGUF:latest

explicit prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-09-composite-refactor
Targetopencode/amdbeast/Kujira/Underdog-Saluki-27B-1.0-MTP-GGUF:latest
Prompt variantexplicit
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Run provenanceprovider_transport=opencode_attached_server_v1
Fixturetestdata/bench/task_09_composite_refactor.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert DefaultConfig() *Config returning Host localhost and ListenPort 8080, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Initial Workspace State (Before Edit)
package config

import (
	"archive/tar"
	"fmt"
)

type Config struct {
	Host string
	Port int
}

func (c *Config) Addr() string {
	return fmt.Sprintf("%s:%d", c.Host, c.Port)
}
MetricVanilla (Small)Semedit Explicit (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency76.45s147.90s+93.5%———
Process Start → First Event19.11s37.74s————
First Event → First Tool Call4.92s8.62s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles410+150.0%———
Initial Load / Discovery Turns10-100.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)410+150.0%———
Output Tokens6201802+190.6%———
Reasoning / Thinking Tokens00————
Total Input Tokens36101205731+469.9%———
Cached Input Tokens29098178305+512.8%———
Uncached Input Tokens700327426+291.6%———
Cost——————
Cached vs Uncached Token Ratio4.16:16.50:1+56.5%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Explicit in Small Context
  • Vanilla (Baseline vs Semedit Explicit in Small Context) Edit: File config/config.go modified (15 lines -> 19 lines)
  • Semedit MCP (Baseline vs Semedit Explicit in Small Context) Edit: File config/config.go modified (15 lines -> 16 lines)
#VanillaSemedit MCP
1read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-09-composite-refactor_1791718291876867000/config/config.go"
}
semedit_semantic_lookup (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Port"
}
2edit (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-09-composite-refactor_1791718291876867000/config/config.go",
  "newString": "import \"fmt\"\n\ntype Config struct {\n\tHost string\n\tListenPort int\n}\n\nfunc (c *Config) Addr() string {\n\treturn fmt.Sprintf(\"%s:%d\", c.Host, c.ListenPort)\n}\n\nfunc DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}",
  "oldString": "import (\n\t\"archive/tar\"\n\t\"fmt\"\n)\n\ntype Config struct {\n\tHost string\n\tPort int\n}\n\nfunc (c *Config) Addr() string {\n\treturn fmt.Sprintf(\"%s:%d\", c.Host, c.Port)\n}"
}
semedit_semantic_inspect_symbol (✓ ✓)
{
  "file": "config/config.go",
  "language": "go",
  "symbol": "Config.Addr"
}
3grep (✓ ✓)
{
  "path": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-09-composite-refactor_1791718291876867000",
  "pattern": "\\.Port\\b"
}
glob (✓ ✓)
{
  "pattern": "**/*.go"
}
4bash (✓ ✓)
{
  "command": "go build ./...",
  "workdir": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-09-composite-refactor_1791718291876867000"
}
read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_semedit-explicit_small-explicit_task-09-composite-refactor_1791718609648750000/config/config.go"
}
5—semedit_semantic_batch (✓ ✗): Batch child results are intermediate and do not provide final continuation targets. Critical result (JSON): {"result":{"status":"error","results":[{"status":"error","application_state":"unknown","error":"edit 1 params: unknown parameter \"kind\" for \"semantic_insert_function\"; supported keys are [access, access_modifier, auto-organize-imports, auto_organize_imports, file, placement, source, target, target_symbol] Required parameters: [file, source]","tool":"semantic_insert_function"}],"workspace_root":"/U
{
  "edits": [
    {
      "params": {
        "file": "config/config.go",
        "symbol": "Config.Port",
        "to": "ListenPort"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "file": "config/config.go",
        "kind": "function",
        "placement": "public_end",
        "source": "func DefaultConfig() *Config { return \u0026Config{Host: \"localhost\", ListenPort: 8080} }"
      },
      "tool": "semantic_insert_function"
    },
    {
      "params": {
        "file": "config/config.go"
      },
      "tool": "semantic_organize_imports"
    }
  ]
}
6—read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_semedit-explicit_small-explicit_task-09-composite-refactor_1791718609648750000/config/config.go"
}
7—semedit_semantic_batch (✓ ✗): Batch child results are intermediate and do not provide final continuation targets. Critical result (JSON): {"result":{"status":"error","results":[{"status":"error","application_state":"unknown","error":"edit 1 params: unknown parameter \"kind\" for \"semantic_insert_function\"; supported keys are [access, access_modifier, auto-organize-imports, auto_organize_imports, file, placement, source, target, target_symbol] Required parameters: [file, source]","tool":"semantic_insert_function"}],"workspace_root":"/U
{
  "edits": [
    {
      "params": {
        "file": "config/config.go",
        "symbol": "Config.Port",
        "to": "ListenPort"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "file": "config/config.go",
        "kind": "function",
        "placement": "public_end",
        "source": "func DefaultConfig() *Config { return \u0026Config{Host: \"localhost\", ListenPort: 8080} }"
      },
      "tool": "semantic_insert_function"
    },
    {
      "params": {
        "file": "config/config.go"
      },
      "tool": "semantic_organize_imports"
    }
  ]
}
8—semedit_semantic_batch (✓ ✓)
{
  "edits": [
    {
      "params": {
        "file": "config/config.go",
        "symbol": "Config.Port",
        "to": "ListenPort"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "file": "config/config.go",
        "placement": "public_end",
        "source": "func DefaultConfig() *Config { return \u0026Config{Host: \"localhost\", ListenPort: 8080} }"
      },
      "tool": "semantic_insert_function"
    },
    {
      "params": {
        "file": "config/config.go"
      },
      "tool": "semantic_organize_imports"
    }
  ]
}
9—read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_semedit-explicit_small-explicit_task-09-composite-refactor_1791718609648750000/config/config.go"
}
10—bash (✓ ✓)
{
  "command": "go build ./... \u0026\u0026 go test ./...",
  "workdir": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_semedit-explicit_small-explicit_task-09-composite-refactor_1791718609648750000"
}

Semedit integration feedback (agent opinion, excluded from measurements):

Integration feedback unavailable: OpenCode session workspace, target, or arm changed

Baseline vs Semedit Explicit in Large Context
#VanillaSemedit MCP
1not publishednot published

implicit prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-09-composite-refactor
Targetopencode/amdbeast/Kujira/Underdog-Saluki-27B-1.0-MTP-GGUF:latest
Prompt variantimplicit
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Run provenanceprovider_transport=opencode_attached_server_v1
Fixturetestdata/bench/task_09_composite_refactor.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert DefaultConfig() *Config returning Host localhost and ListenPort 8080, and clean up unused imports. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

In config/config.go: rename Config.Port to Config.ListenPort, insert DefaultConfig() *Config returning Host localhost and ListenPort 8080, and clean up unused imports. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package config

import (
	"archive/tar"
	"fmt"
)

type Config struct {
	Host string
	Port int
}

func (c *Config) Addr() string {
	return fmt.Sprintf("%s:%d", c.Host, c.Port)
}
MetricVanilla (Small)MCP (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency76.45s187.54s+145.3%———
Process Start → First Event19.11s44.16s————
First Event → First Tool Call4.92s2.42s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles413+225.0%———
Initial Load / Discovery Turns12+100.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)413+225.0%———
Output Tokens6202335+276.6%———
Reasoning / Thinking Tokens00————
Total Input Tokens36101375963+941.4%———
Cached Input Tokens29098345936+1088.9%———
Uncached Input Tokens700330027+328.8%———
Cost——————
Cached vs Uncached Token Ratio4.16:111.52:1+177.3%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Agentic in Small Context
  • Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 19 lines)
  • Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File config/config.go modified (15 lines -> 18 lines)
#VanillaSemedit MCP
1read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-09-composite-refactor_1791718291876867000/config/config.go"
}
glob (✓ ✓)
{
  "pattern": "**/config/config.go"
}
2edit (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-09-composite-refactor_1791718291876867000/config/config.go",
  "newString": "import \"fmt\"\n\ntype Config struct {\n\tHost string\n\tListenPort int\n}\n\nfunc (c *Config) Addr() string {\n\treturn fmt.Sprintf(\"%s:%d\", c.Host, c.ListenPort)\n}\n\nfunc DefaultConfig() *Config {\n\treturn \u0026Config{\n\t\tHost:       \"localhost\",\n\t\tListenPort: 8080,\n\t}\n}",
  "oldString": "import (\n\t\"archive/tar\"\n\t\"fmt\"\n)\n\ntype Config struct {\n\tHost string\n\tPort int\n}\n\nfunc (c *Config) Addr() string {\n\treturn fmt.Sprintf(\"%s:%d\", c.Host, c.Port)\n}"
}
read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_semedit_small-implicit_task-09-composite-refactor_1791718368453087000/config/config.go"
}
3grep (✓ ✓)
{
  "path": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-09-composite-refactor_1791718291876867000",
  "pattern": "\\.Port\\b"
}
semedit_semantic_find_references (✓ ✗): Critical result (JSON): {"application_state":"unknown","error":"load Go package example.com/test/config: example.com/test/config: /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_semedit_small-implicit_task-09-composite-refactor_1791718368453087000/config/config.go:4:2: \"archive/tar\" imported and not used"} load Go package example.com/test/config: example.com/test/config: /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_semedit_small-implicit_task-09-com
{
  "symbol": "Config.Port"
}
4bash (✓ ✓)
{
  "command": "go build ./...",
  "workdir": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-09-composite-refactor_1791718291876867000"
}
grep (✓ ✓)
{
  "pattern": "\\.Port|Port"
}
5—read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_semedit_small-implicit_task-09-composite-refactor_1791718368453087000/config/config_test.go"
}
6—semedit_semantic_batch (✓ ✗): Batch child results are intermediate and do not provide final continuation targets. Critical result (JSON): {"result":{"status":"error","results":[{"status":"error","application_state":"unknown","error":"resolve Port in config/config.go: symbol not found","tool":"semantic_rename"}],"workspace_root":"/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_semedit_small-implicit_task-09-composite-refactor_1791718368453087000"}} { "status": "error", "workspace_root": "/Users/alessandro/sourc
{
  "auto_organize_imports": true,
  "edits": [
    {
      "params": {
        "file": "config/config.go",
        "language": "go",
        "symbol": "Port",
        "to": "ListenPort"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "auto_organize_imports": true,
        "file": "config/config.go",
        "kind": "function",
        "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{Host: \"localhost\", ListenPort: 8080}\n}"
      },
      "tool": "semantic_insert_construct"
    },
    {
      "params": {
        "file": "config/config.go"
      },
      "tool": "semantic_organize_imports"
    }
  ]
}
7—semedit_semantic_batch (✓ ✓)
{
  "edits": [
    {
      "params": {
        "file": "config/config.go"
      },
      "tool": "semantic_organize_imports"
    }
  ]
}
8—semedit_semantic_batch (✓ ✗): Batch child results are intermediate and do not provide final continuation targets. Critical result (JSON): {"result":{"status":"error","results":[{"status":"error","application_state":"unknown","error":"resolve Port in config/config.go: symbol not found","tool":"semantic_rename"}],"workspace_root":"/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_semedit_small-implicit_task-09-composite-refactor_1791718368453087000"}} { "status": "error", "workspace_root": "/Users/alessandro/sourc
{
  "auto_organize_imports": true,
  "edits": [
    {
      "params": {
        "file": "config/config.go",
        "language": "go",
        "symbol": "Port",
        "to": "ListenPort"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "auto_organize_imports": true,
        "file": "config/config.go",
        "kind": "function",
        "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{Host: \"localhost\", ListenPort: 8080}\n}"
      },
      "tool": "semantic_insert_construct"
    }
  ]
}
9—semedit_semantic_batch (✓ ✓)
{
  "edits": [
    {
      "params": {
        "file": "config/config.go",
        "language": "go",
        "symbol": "Config.Port",
        "to": "ListenPort"
      },
      "tool": "semantic_rename"
    }
  ]
}
10—semedit_semantic_batch (✓ ✓)
{
  "edits": [
    {
      "params": {
        "auto_organize_imports": true,
        "file": "config/config.go",
        "kind": "function",
        "source": "func DefaultConfig() *Config {\n\treturn \u0026Config{Host: \"localhost\", ListenPort: 8080}\n}"
      },
      "tool": "semantic_insert_construct"
    }
  ]
}
11—read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_semedit_small-implicit_task-09-composite-refactor_1791718368453087000/config/config.go"
}
12—bash (✓ ✓)
{
  "command": "go test ./config/...",
  "workdir": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_semedit_small-implicit_task-09-composite-refactor_1791718368453087000"
}
13—bash (✓ ✓)
{
  "command": "go build ./... \u0026\u0026 go vet ./config/...",
  "workdir": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_semedit_small-implicit_task-09-composite-refactor_1791718368453087000"
}

Semedit integration feedback (agent opinion, excluded from measurements):

I'd genuinely ask my owner to add semedit's tools to my toolkit, because the rename and insert operations let me refactor `Config.Port` → `Config.ListenPort` and add `DefaultConfig()` without any fragile manual search-and-replace, and the built-in verification after every edit gave me real confidence I hadn't broken the build — though I did get a little frustrated when a bare field name like `Port` refused to resolve until I qualified it as `Config.Port`, so the error messages could be a bit kinder.
Baseline vs Semedit Agentic in Large Context
#VanillaSemedit MCP
1not publishednot published

Test case: task-11-mixed-sink-api-migration

Target: opencode/amdbeast/Kujira/Underdog-Saluki-27B-1.0-MTP-GGUF:latest

explicit prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetopencode/amdbeast/Kujira/Underdog-Saluki-27B-1.0-MTP-GGUF:latest
Prompt variantexplicit
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Run provenanceprovider_transport=opencode_attached_server_v1
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit
MetricVanilla (Small)Semedit Explicit (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency228.16s148.87s-34.7%———
Process Start → First Event17.45s47.81s————
First Event → First Tool Call2.32s3.42s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles297-75.9%———
Initial Load / Discovery Turns230-100.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)297-75.9%———
Output Tokens38511426-63.0%———
Reasoning / Thinking Tokens00————
Total Input Tokens148470218281+47.0%———
Cached Input Tokens136966184416+34.6%———
Uncached Input Tokens1150433865+194.4%———
Cost——————
Cached vs Uncached Token Ratio11.91:15.45:1-54.3%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestPASSPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Explicit in Small Context
  • Vanilla (Baseline vs Semedit Explicit in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
  • Semedit MCP (Baseline vs Semedit Explicit in Small Context) Edit: File audit/normalize.go modified (2 lines -> 8 lines)
#VanillaSemedit MCP
1bash (✓ ✓)
{
  "command": "ls -la"
}
semedit_semantic_outline (✓ ✓)
{
  "language": "go",
  "path": "."
}
2read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-11-mixed-sink-api-migration_1791718757875030000/audit"
}
semedit_semantic_inspect_symbol (✓ ✓)
{
  "file": "service/dispatch.go",
  "language": "go",
  "symbol": "Dispatcher.Record"
}
3read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-11-mixed-sink-api-migration_1791718757875030000/go.mod"
}
semedit_semantic_inspect_symbol (✓ ✓)
{
  "file": "audit/summary.go",
  "language": "go",
  "symbol": "Classify"
}
4read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-11-mixed-sink-api-migration_1791718757875030000/audit/event.go"
}
semedit_semantic_batch (✓ ✗): Batch child results are intermediate and do not provide final continuation targets. Critical result (JSON): {"result":{"status":"error","results":[{"status":"error","application_state":"unknown","error":"edit 1 params: unknown parameter \"source\" for \"semantic_replace_body\"; supported keys are [auto-organize-imports, auto_organize_imports, body, expected-revision, expected_revision, file, symbol] Required parameters: [body, file, symbol]","tool":"semantic_replace_body"}],"workspace_root":"/Users/alessand
{
  "edits": [
    {
      "params": {
        "file": "audit/normalize.go",
        "kind": "function",
        "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
      },
      "tool": "semantic_insert_construct"
    },
    {
      "params": {
        "file": "service/dispatch.go",
        "source": "event.Kind = audit.NormalizeKind(event.Kind)\n\treturn d.sink.Write(event)",
        "symbol": "Dispatcher.Record"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "file": "audit/summary.go",
        "source": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n}",
        "symbol": "Classify"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "file": "audit/normalize.go"
      },
      "tool": "semantic_organize_imports"
    }
  ]
}
5read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-11-mixed-sink-api-migration_1791718757875030000/audit/normalize.go"
}
semedit_semantic_batch (✓ ✓)
{
  "edits": [
    {
      "params": {
        "file": "audit/normalize.go",
        "kind": "function",
        "source": "func NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}"
      },
      "tool": "semantic_insert_construct"
    },
    {
      "params": {
        "body": "event.Kind = audit.NormalizeKind(event.Kind)\n\treturn d.sink.Write(event)",
        "file": "service/dispatch.go",
        "symbol": "Dispatcher.Record"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "body": "switch NormalizeKind(kind) {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"\n\tdefault:\n\t\treturn \"data\"\n}",
        "file": "audit/summary.go",
        "symbol": "Classify"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "file": "audit/normalize.go"
      },
      "tool": "semantic_organize_imports"
    }
  ]
}
6read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-11-mixed-sink-api-migration_1791718757875030000/audit/sink.go"
}
bash (✓ ✓)
{
  "command": "go build ./... \u0026\u0026 go test ./...",
  "workdir": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_semedit-explicit_small-explicit_task-11-mixed-sink-api-migration_1791719286320563000"
}
7read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-11-mixed-sink-api-migration_1791718757875030000/audit/summary.go"
}
bash (✓ ✓)
{
  "command": "go build -buildvcs=false ./... \u0026\u0026 go test -buildvcs=false ./...",
  "workdir": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_semedit-explicit_small-explicit_task-11-mixed-sink-api-migration_1791719286320563000"
}
8grep (✓ ✓)
{
  "path": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-11-mixed-sink-api-migration_1791718757875030000",
  "pattern": "Dispatcher"
}
—
9read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-11-mixed-sink-api-migration_1791718757875030000/audit/event_test.go"
}
—
10read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-11-mixed-sink-api-migration_1791718757875030000/audit/summary_test.go"
}
—
11read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-11-mixed-sink-api-migration_1791718757875030000/service"
}
—
12read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-11-mixed-sink-api-migration_1791718757875030000/service/dispatch.go"
}
—
13read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-11-mixed-sink-api-migration_1791718757875030000/service/dispatch_test.go"
}
—
14read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-11-mixed-sink-api-migration_1791718757875030000/metrics"
}
—
15read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-11-mixed-sink-api-migration_1791718757875030000/legacy"
}
—
16read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-11-mixed-sink-api-migration_1791718757875030000/cmd"
}
—
17read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-11-mixed-sink-api-migration_1791718757875030000/docs"
}
—
18read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-11-mixed-sink-api-migration_1791718757875030000/metrics/writer.go"
}
—
19read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-11-mixed-sink-api-migration_1791718757875030000/legacy/export.go"
}
—
20read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-11-mixed-sink-api-migration_1791718757875030000/cmd/migrate/main.go"
}
—
21read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-11-mixed-sink-api-migration_1791718757875030000/docs/audit-sinks.md"
}
—
22read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-11-mixed-sink-api-migration_1791718757875030000/metrics/writer_test.go"
}
—
23read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-11-mixed-sink-api-migration_1791718757875030000/legacy/export_test.go"
}
—
24write (✓ ✓)
{
  "content": "// Package audit leaves room for normalization rules owned by the delivery boundary.\npackage audit\n\nimport \"strings\"\n\nfunc NormalizeKind(kind string) string {\n\treturn strings.ToLower(strings.TrimSpace(kind))\n}\n",
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-11-mixed-sink-api-migration_1791718757875030000/audit/normalize.go"
}
—
25edit (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-11-mixed-sink-api-migration_1791718757875030000/service/dispatch.go",
  "newString": "func (d *Dispatcher) Record(event audit.Event) error {\n\tevent.Kind = audit.NormalizeKind(event.Kind)\n\treturn d.sink.Write(event)\n}",
  "oldString": "func (d *Dispatcher) Record(event audit.Event) error {\n\treturn d.sink.Write(event)\n}"
}
—
26edit (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-11-mixed-sink-api-migration_1791718757875030000/audit/summary.go",
  "newString": "\tswitch kind {\n\tcase \"health\", \"probe\":\n\t\treturn \"control\"",
  "oldString": "\tswitch kind {\n\tcase \"health\":\n\t\treturn \"control\""
}
—
27bash (✓ ✓)
{
  "command": "go build ./... \u0026\u0026 go vet ./... \u0026\u0026 go test ./..."
}
—
28bash (✓ ✓)
{
  "command": "go build -buildvcs=false ./... \u0026\u0026 go vet -buildvcs=false ./... \u0026\u0026 go test -buildvcs=false ./..."
}
—
29bash (✓ ✓)
{
  "command": "git status --short"
}
—

Semedit integration feedback (agent opinion, excluded from measurements):

Integration feedback unavailable: OpenCode session workspace, target, or arm changed

Baseline vs Semedit Explicit in Large Context
#VanillaSemedit MCP
1not publishednot published

implicit prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-11-mixed-sink-api-migration
Targetopencode/amdbeast/Kujira/Underdog-Saluki-27B-1.0-MTP-GGUF:latest
Prompt variantimplicit
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Run provenanceprovider_transport=opencode_attached_server_v1
Fixturetestdata/bench/task_11_mixed_sink_api_migration.txtar

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Audit event kinds may contain inconsistent whitespace and casing. Add the public audit.NormalizeKind(kind string) string function in audit/normalize.go to remove surrounding whitespace and lowercase the kind. Apply it in Dispatcher.Record before persistence, and classify normalized probe events as control traffic, like health. Do not change the metrics or legacy export protocols. Preserve the module metadata and tests, and leave the workspace verified. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
// Package audit leaves room for normalization rules owned by the delivery boundary.
package audit

Test case: task-12-request-identity-migration

Target: opencode/amdbeast/Kujira/Underdog-Saluki-27B-1.0-MTP-GGUF:latest

explicit prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-12-request-identity-migration
Targetopencode/amdbeast/Kujira/Underdog-Saluki-27B-1.0-MTP-GGUF:latest
Prompt variantexplicit
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Run provenanceprovider_transport=opencode_attached_server_v1
Fixturetestdata/bench/task_12_request_identity_migration.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Initial Workspace State (Before Edit)
package request

import (
	"errors"
	"strings"
)

type Request struct {
	ID      string
	Kind    string
	Payload []byte
}

func (r Request) Validate() error {
	if r.ID == "" {
		return errors.New("invalid request")
	}
	return nil
}

func Normalize(kind string) string {
	kind = strings.TrimSpace(kind)
	kind = strings.ToLower(kind)
	return kind
}

func FindByID(requests []Request, id string) (Request, bool) {
	for _, r := range requests {
		if r.ID == id {
			return r, true
		}
	}
	return Request{}, false
}
MetricVanilla (Small)Semedit Explicit (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency299.98s175.57s-41.5%———
Process Start → First Event15.37s24.61s————
First Event → First Tool Call4.82s1.84s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns110%———
Internal Tool Cycles3622-38.9%———
Initial Load / Discovery Turns230-100.0%———
MCP Discovery / Schema Turns00————
Tool Invocations (per run)3622-38.9%———
Output Tokens60485289-12.5%———
Reasoning / Thinking Tokens00————
Total Input Tokens164616665981+304.6%———
Cached Input Tokens147751629959+326.4%———
Uncached Input Tokens1686536022+113.6%———
Cost——————
Cached vs Uncached Token Ratio8.76:117.49:1+99.6%———
Oracle L1: Mutation PolicyPASSPASS————
Oracle L2: AST InvariantsPASSPASS————
Oracle L3: Clean BuildPASSPASS————
Oracle L4: Verification TestFAILPASS————
Semantic Tool Invocation Verified (per run)N/APASS————
Baseline vs Semedit Explicit in Small Context
  • Vanilla (Baseline vs Semedit Explicit in Small Context) Edit: File request/request.go modified (34 lines -> 40 lines)
  • Semedit MCP (Baseline vs Semedit Explicit in Small Context) Edit: File request/request.go modified (34 lines -> 41 lines)
#VanillaSemedit MCP
1bash (✓ ✓)
{
  "command": "ls -la /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000"
}
semedit_semantic_lookup (✓ ✓)
{
  "language": "go",
  "symbol": "Request"
}
2bash (✓ ✓)
{
  "command": "find /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000 -type f -name \"*.go\" -o -type f -name \"go.mod\" | sort"
}
semedit_semantic_outline (✓ ✓)
{
  "language": "go",
  "path": "request"
}
3read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/request/request.go"
}
read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_semedit-explicit_small-explicit_task-12-request-identity-migration_1791720035872508000/request/request.go"
}
4read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/request/request_test.go"
}
semedit_semantic_find_references (✓ ✓)
{
  "file": "request/request.go",
  "language": "go",
  "symbol": "Request.ID"
}
5read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/correlation/identities.go"
}
read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_semedit-explicit_small-explicit_task-12-request-identity-migration_1791720035872508000/request/request_test.go"
}
6read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/gateway/handler.go"
}
read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_semedit-explicit_small-explicit_task-12-request-identity-migration_1791720035872508000/go.mod"
}
7read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/gateway/handler_test.go"
}
semedit_semantic_rename (✓ ✓)
{
  "file": "request/request.go",
  "language": "go",
  "symbol": "Request.ID",
  "to": "RequestID"
}
8read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/notifications/notifier.go"
}
semedit_semantic_insert_construct (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "request/request.go",
  "kind": "var",
  "placement": "file_start",
  "source": "var ErrInvalidRequest = errors.New(\"invalid request\")"
}
9read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/queue/queue.go"
}
semedit_semantic_replace_construct (✓ ✗): Critical result (JSON): {"application_state":"unknown","error":"replacement must preserve the selected function symbol \"Request.Validate\""} replacement must preserve the selected function symbol "Request.Validate"
{
  "auto_organize_imports": true,
  "file": "request/request.go",
  "kind": "function",
  "source": "func (r Request) Validate() error {\n\tif r.RequestID == \"\" {\n\t\treturn fmt.Errorf(\"invalid request: %w\", ErrInvalidRequest)\n\t}\n\treturn nil\n}",
  "symbol": "Request.Validate"
}
10read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/reporting/report.go"
}
semedit_semantic_replace_construct (✓ ✓)
{
  "auto_organize_imports": true,
  "discriminator": "return errors.New(\"invalid request\")",
  "file": "request/request.go",
  "in_function": "Request.Validate",
  "kind": "statement/expression",
  "source": "return fmt.Errorf(\"invalid request: %w\", ErrInvalidRequest)"
}
11read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/response/audit.go"
}
read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_semedit-explicit_small-explicit_task-12-request-identity-migration_1791720035872508000/request/request.go"
}
12read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/response/response.go"
}
semedit_semantic_inspect_symbol (✓ ✓)
{
  "file": "request/request.go",
  "language": "go",
  "symbol": "Normalize"
}
13read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/response/response_test.go"
}
semedit_semantic_replace_construct (✓ ✗): Critical result (JSON): {"application_state":"unknown","error":"multiple declarations found in snippet"} multiple declarations found in snippet
{
  "auto_organize_imports": true,
  "file": "request/request.go",
  "kind": "function",
  "source": "func Normalize(kind string) string {\n\treturn normalizeKind(kind)\n}\n\nfunc normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}",
  "symbol": "Normalize"
}
14read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/review/review.go"
}
semedit_semantic_replace_construct (✓ ✗): Critical result (JSON): {"application_state":"unknown","error":"replacement must preserve the selected function symbol \"Normalize\""} replacement must preserve the selected function symbol "Normalize"
{
  "auto_organize_imports": true,
  "file": "request/request.go",
  "kind": "function",
  "source": "func normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}",
  "symbol": "Normalize"
}
15read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/review/review_test.go"
}
semedit_semantic_replace_construct (✓ ✓)
{
  "auto_organize_imports": true,
  "discriminator": "return kind",
  "file": "request/request.go",
  "in_function": "Normalize",
  "kind": "statement/expression",
  "source": "return normalizeKind(kind)"
}
16read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/scheduler/jobs.go"
}
semedit_semantic_insert_construct (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "request/request.go",
  "kind": "function",
  "placement": "after_symbol",
  "source": "func normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}",
  "target_symbol": "Normalize"
}
17read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/storage/requests.go"
}
semedit_semantic_verify (✓ ✓)
{
  "language": "go",
  "path": "request"
}
18read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/storage/requests_test.go"
}
read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_semedit-explicit_small-explicit_task-12-request-identity-migration_1791720035872508000/request/request.go"
}
19read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/go.mod"
}
semedit_semantic_replace_construct (✓ ✓)
{
  "auto_organize_imports": true,
  "discriminator": "kind = strings.TrimSpace(kind)",
  "file": "request/request.go",
  "in_function": "Normalize",
  "kind": "statement/expression",
  "source": "kind = kind"
}
20grep (✓ ✓)
{
  "path": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000",
  "pattern": "\\.ID\\b"
}
semedit_semantic_replace_construct (✓ ✗): Critical result (JSON): {"application_state":"unknown","error":"symbol is required for kind=function"} symbol is required for kind=function
{
  "auto_organize_imports": true,
  "discriminator": "func Normalize(kind string) string {\n\tkind = kind\n\tkind = strings.ToLower(kind)\n\treturn normalizeKind(kind)\n}",
  "file": "request/request.go",
  "kind": "function",
  "source": "func Normalize(kind string) string {\n\treturn normalizeKind(kind)\n}"
}
21bash (✓ ✓)
{
  "command": "ls -la /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/docs /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/.scratch"
}
semedit_semantic_replace_construct (✓ ✓)
{
  "auto_organize_imports": true,
  "file": "request/request.go",
  "kind": "function",
  "source": "func Normalize(kind string) string {\n\treturn normalizeKind(kind)\n}",
  "symbol": "Normalize"
}
22read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/docs/identity-model.md"
}
read (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_semedit-explicit_small-explicit_task-12-request-identity-migration_1791720035872508000/request/request.go"
}
23bash (✓ ✓)
{
  "command": "find /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/.scratch/go -type f | head -20"
}
—
24edit (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/request/request.go",
  "newString": "var ErrInvalidRequest = errors.New(\"invalid request\")\n\ntype Request struct {\n\tRequestID string\n\tKind      string\n\tPayload   []byte\n}\n\nfunc (r Request) Validate() error {\n\tif r.RequestID == \"\" {\n\t\treturn ErrInvalidRequest\n\t}\n\treturn nil\n}\n\nfunc Normalize(kind string) string {\n\treturn normalizeKind(kind)\n}\n\nfunc normalizeKind(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}\n\nfunc FindByID(requests []Request, id string) (Request, bool) {\n\tfor _, r := range requests {\n\t\tif r.RequestID == id {\n\t\t\treturn r, true\n\t\t}\n\t}\n\treturn Request{}, false\n}",
  "oldString": "type Request struct {\n\tID      string\n\tKind    string\n\tPayload []byte\n}\n\nfunc (r Request) Validate() error {\n\tif r.ID == \"\" {\n\t\treturn errors.New(\"invalid request\")\n\t}\n\treturn nil\n}\n\nfunc Normalize(kind string) string {\n\tkind = strings.TrimSpace(kind)\n\tkind = strings.ToLower(kind)\n\treturn kind\n}\n\nfunc FindByID(requests []Request, id string) (Request, bool) {\n\tfor _, r := range requests {\n\t\tif r.ID == id {\n\t\t\treturn r, true\n\t\t}\n\t}\n\treturn Request{}, false\n}"
}
—
25edit (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/correlation/identities.go",
  "newString": "\treturn []string{r.RequestID, resp.ID, review.ID}",
  "oldString": "\treturn []string{r.ID, resp.ID, review.ID}"
}
—
26edit (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/gateway/handler.go",
  "newString": "\t\treturn response.Response{}, fmt.Errorf(\"submit %s: %w\", r.RequestID, err)",
  "oldString": "\t\treturn response.Response{}, fmt.Errorf(\"submit %s: %w\", r.ID, err)"
}
—
27edit (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/notifications/notifier.go",
  "newString": "\tif r.RequestID == \"\" { return \"unassigned\" }\n\treturn \"request:\" + r.RequestID",
  "oldString": "\tif r.ID == \"\" { return \"unassigned\" }\n\treturn \"request:\" + r.ID"
}
—
28edit (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/notifications/notifier.go",
  "newString": "\tfor _, r := range notices { ids = append(ids, r.Request.RequestID) }",
  "oldString": "\tfor _, r := range notices { ids = append(ids, r.Request.ID) }"
}
—
29edit (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/queue/queue.go",
  "newString": "\tfor _, entry := range entries {\n\t\tif entry.Key == r.RequestID { return true }\n\t}",
  "oldString": "\tfor _, entry := range entries {\n\t\tif entry.Key == r.ID { return true }\n\t}"
}
—
30edit (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/queue/queue.go",
  "newString": "\tfor _, r := range requests { keys = append(keys, r.RequestID) }",
  "oldString": "\tfor _, r := range requests { keys = append(keys, r.ID) }"
}
—
31edit (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/reporting/report.go",
  "newString": "\tfor _, r := range rows { grouped[r.Request.RequestID] = append(grouped[r.Request.RequestID], r) }",
  "oldString": "\tfor _, r := range rows { grouped[r.Request.ID] = append(grouped[r.Request.ID], r) }"
}
—
32edit (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/scheduler/jobs.go",
  "newString": "\treturn left.Request.RequestID == right.Request.RequestID",
  "oldString": "\treturn left.Request.ID == right.Request.ID"
}
—
33edit (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/scheduler/jobs.go",
  "newString": "\t\tif job.Request.RequestID == r.RequestID \u0026\u0026 job.Attempts == 0 { pending = append(pending, job) }",
  "oldString": "\t\tif job.Request.ID == r.ID \u0026\u0026 job.Attempts == 0 { pending = append(pending, job) }"
}
—
34edit (✓ ✓)
{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000/storage/requests.go",
  "newString": "\t\tif record.ID == r.RequestID {",
  "oldString": "\t\tif record.ID == r.ID {"
}
—
35bash (✓ ✓)
{
  "command": "cd /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000 \u0026\u0026 go build ./... \u0026\u0026 go test ./...",
  "workdir": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000"
}
—
36grep (✓ ✓)
{
  "path": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-12-request-identity-migration_1791719435569142000",
  "pattern": "\\.ID\\b"
}
—

Semedit integration feedback (agent opinion, excluded from measurements):

Integration feedback unavailable: cannot continue failed provider session: OpenCode session workspace, target, or arm changed

Baseline vs Semedit Explicit in Large Context
#VanillaSemedit MCP
1not publishednot published

implicit prompt · prescriptive MCP instructions · write semedit restriction

SettingValue
Test casetask-12-request-identity-migration
Targetopencode/amdbeast/Kujira/Underdog-Saluki-27B-1.0-MTP-GGUF:latest
Prompt variantimplicit
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Run provenanceprovider_transport=opencode_attached_server_v1
Fixturetestdata/bench/task_12_request_identity_migration.txtar

Vanilla LLM Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

You are forbidden to modify protected files: “go.mod”, “go.sum”.

Migrate the request identity API in this large multi-package Go service. Rename only Request.ID to Request.RequestID and update every semantic use across packages; Response.ID and Review.ID are separate identities and must remain unchanged, including the many syntactically identical r.ID expressions. Add a package sentinel ErrInvalidRequest and return it from Validate when the request ID is empty, preserving error wrapping and callers. Extract the normalization block from the existing exported request.Normalize function into an unexported normalizeKind helper and keep Normalize exported with the same behavior and signature. Preserve module metadata and existing tests. Verification differs by editing method: when making ordinary file edits, run the project build and tests and ensure both pass; when using semedit, treat clean compiler diagnostic deltas returned by its mutation commands as build-clean evidence and do not run a separate build or test command. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

Initial Workspace State (Before Edit)
package request

import (
	"errors"
	"strings"
)

type Request struct {
	ID      string
	Kind    string
	Payload []byte
}

func (r Request) Validate() error {
	if r.ID == "" {
		return errors.New("invalid request")
	}
	return nil
}

func Normalize(kind string) string {
	kind = strings.TrimSpace(kind)
	kind = strings.ToLower(kind)
	return kind
}

func FindByID(requests []Request, id string) (Request, bool) {
	for _, r := range requests {
		if r.ID == id {
			return r, true
		}
	}
	return Request{}, false
}
Benchmark run hard-smoke-luna6-medium-20261011T105400ZBenchmark run hard-smoke-saluki-20261011T203354Z