Benchmark run saluki-opencode-terminal-runtime-20261011T110500Z

This page contains exactly the publishable benchmark observations recorded in run saluki-opencode-terminal-runtime-20261011T110500Z; it does not select or aggregate them. Return to the run index or view aggregate metrics.

Interactive browser

Detailed observations

Empirical telemetry measuring wall-clock latency, token usage, tool turns, and compiler correctness when AI coding agents perform code modifications.

Every benchmark pairs Vanilla LLM (standard file editing tools) against Semedit MCP (deterministic AST compiler operations).

Benchmark Methodology & Transparency

• Reproducibility & Provenance: Self-contained txtar archives define test scenarios. Links below resolve to commit-anchored GitHub source files for published commits, or cryptographic SHA-256 fingerprints for uncommitted local fixtures.
• Model Cost: Tables report model-specific credit consumption where a target model has a declared schedule: (uncached input × input credits + cached input × cached credits + (reasoning + visible output) × output credits) / 1,000,000. Thinking tokens use the output rate. ADR-0043 defines the per-million-token credit rates.
• Multi-Level Correctness Oracle: Four validation levels evaluate each trial:
  1. Level 1 (Mutation Policy): Restricts file modifications strictly to authorized paths.
  2. Level 2 (AST Invariants): Compiler AST verification of required symbols, imports, and relative declaration ordering.
  3. Level 3 (Clean Build): Strict compilation validation with zero compiler or typecheck errors.
  4. Level 4 (Test Suite): Unit and integration test suite execution.
• Isolated Sandboxing: Each trial executes in an ephemeral directory with pristine git and module state.

Test case: task-01-rename-local

Target: opencode/amdbeast/Kujira/Underdog-Saluki-27B-1.0-MTP-GGUF:latest

explicit prompt · prescriptive MCP instructions · write semedit restriction

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Rename the unexported method Server.oldName to Server.newName and update all call-sites. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

package api

type Server struct{}

func (s *Server) oldName() string {
	return "ok"
}

func Run() string {
	s := &Server{}
	return s.oldName()
}

Baseline vs Semedit Explicit in Small Context

• Vanilla (Baseline vs Semedit Explicit in Small Context) Edit: File api/server.go modified (12 lines -> 12 lines)
• Semedit MCP (Baseline vs Semedit Explicit in Small Context) Edit: File api/server.go modified (12 lines -> 12 lines)

{
  "pattern": "oldName"
}

{
  "file": "api/server.go",
  "language": "go",
  "symbol": "Server.oldName"
}

{
  "pattern": "**/*.go"
}

{
  "file": "api/server.go",
  "language": "go",
  "symbol": "Server.oldName",
  "to": "newName"
}

{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-01-rename-local_1791714483942844000/api/server.go"
}

{
  "command": "grep -rn \"oldName\" . 2\u003e/dev/null; go build ./... 2\u003e\u00261 | head -20"
}

{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-01-rename-local_1791714483942844000/api/server_test.go"
}

{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-01-rename-local_1791714483942844000/api/server.go",
  "newString": "func (s *Server) newName() string {",
  "oldString": "func (s *Server) oldName() string {"
}

{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-01-rename-local_1791714483942844000/api/server.go",
  "newString": "\treturn s.newName()",
  "oldString": "\treturn s.oldName()"
}

{
  "command": "cd /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-01-rename-local_1791714483942844000 \u0026\u0026 go build ./... \u0026\u0026 go test ./..."
}

Semedit integration feedback (agent opinion, excluded from measurements):

Integration feedback unavailable: cannot continue failed provider session: OpenCode session workspace, target, or arm changed

Baseline vs Semedit Explicit in Large Context

implicit prompt · prescriptive MCP instructions · write semedit restriction

Vanilla LLM Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Rename the unexported method Server.oldName to Server.newName and update all call-sites. Do not use semantic editing MCP tools; use standard file editing. When done, output DONE.

Semedit MCP Prompt:

Do not edit tests. You are forbidden to modify protected files: “go.mod”, “go.sum”.

Rename the unexported method Server.oldName to Server.newName and update all call-sites. Use semedit semantic tools for supported source code modifications. Shell commands for builds and tests are allowed. When done, output DONE.

package api

type Server struct{}

func (s *Server) oldName() string {
	return "ok"
}

func Run() string {
	s := &Server{}
	return s.oldName()
}

Baseline vs Semedit Agentic in Small Context

• Vanilla (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (12 lines -> 12 lines)
• Semedit MCP (Baseline vs Semedit Agentic in Small Context) Edit: File api/server.go modified (12 lines -> 12 lines)

{
  "pattern": "oldName"
}

{
  "path": "."
}

{
  "pattern": "**/*.go"
}

{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_semedit_small-implicit_task-01-rename-local_1791714515532303000/api/server.go"
}

{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-01-rename-local_1791714483942844000/api/server.go"
}

{
  "file": "api/server.go",
  "language": "go",
  "symbol": "Server.oldName",
  "to": "newName"
}

{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-01-rename-local_1791714483942844000/api/server_test.go"
}

{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_semedit_small-implicit_task-01-rename-local_1791714515532303000/api/server.go"
}

{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-01-rename-local_1791714483942844000/api/server.go",
  "newString": "func (s *Server) newName() string {",
  "oldString": "func (s *Server) oldName() string {"
}

{
  "command": "cd /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_semedit_small-implicit_task-01-rename-local_1791714515532303000 \u0026\u0026 go build ./... \u0026\u0026 go vet ./... \u0026\u0026 go test ./... 2\u003e\u00261 || true"
}

{
  "filePath": "/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-01-rename-local_1791714483942844000/api/server.go",
  "newString": "\treturn s.newName()",
  "oldString": "\treturn s.oldName()"
}

{
  "command": "cd /Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_baseline-diff_small_task-01-rename-local_1791714483942844000 \u0026\u0026 go build ./... \u0026\u0026 go test ./..."
}

Semedit integration feedback (agent opinion, excluded from measurements):

I was genuinely impressed by how the semantic rename tool handled the declaration and call-site in a single, clean atomic operation, and the outline gave me a perfect structural map before I touched anything—I'd absolutely ask my owner to add semedit's tools to my toolkit.

Baseline vs Semedit Agentic in Large Context