Skip to content
Benchmark run swe-smoke-saluki-20261011T120900Z

Benchmark run swe-smoke-saluki-20261011T120900Z

This page contains exactly the publishable benchmark observations recorded in run swe-smoke-saluki-20261011T120900Z; it does not select or aggregate them. Return to the run index or view aggregate metrics.

Interactive browser

Rows are pre-aggregated by task, target, prompt variant, MCP-instruction mode, context, arm, and restriction. Numeric values show median (p50), IQR, p90, maximum, and sample count. Boolean checks show pass counts and success rates. Filters apply to these summary groups; per-run observations remain available on their run pages.

Loading benchmark results…

Baseline vs semedit

Comparison evaluates baseline and semedit values within each restriction policy, target, and context group. Negative changes mark improvements for these lower-is-better metrics; positive changes mark degradations.

Detailed observations

Empirical telemetry measuring wall-clock latency, token usage, tool turns, and compiler correctness when AI coding agents perform code modifications.

Every benchmark pairs Vanilla LLM (standard file editing tools) against Semedit MCP (deterministic AST compiler operations).

Benchmark Methodology & Transparency

  • Reproducibility & Provenance: Self-contained txtar archives define test scenarios. Links below resolve to commit-anchored GitHub source files for published commits, or cryptographic SHA-256 fingerprints for uncommitted local fixtures.
  • Model Cost: Tables report model-specific credit consumption where a target model has a declared schedule: (uncached input × input credits + cached input × cached credits + (reasoning + visible output) × output credits) / 1,000,000. Thinking tokens use the output rate. ADR-0043 defines the per-million-token credit rates.
  • Multi-Level Correctness Oracle: Four validation levels evaluate each trial:
    1. Level 1 (Mutation Policy): Restricts file modifications strictly to authorized paths.
    2. Level 2 (AST Invariants): Compiler AST verification of required symbols, imports, and relative declaration ordering.
    3. Level 3 (Clean Build): Strict compilation validation with zero compiler or typecheck errors.
    4. Level 4 (Test Suite): Unit and integration test suite execution.
  • Isolated Sandboxing: Each trial executes in an ephemeral directory with pristine git and module state.

Test case: grpc__grpc-go-2735

Target: opencode/amdbeast/Kujira/Underdog-Saluki-27B-1.0-MTP-GGUF:latest

explicit prompt · prescriptive MCP instructions · write semedit restriction

LocalityContextAmbiguityCompositionInvariantsScaleStaging
Task complexity across locality, context, ambiguity, composition, invariants, edit scale, and staging (1–5).
SettingValue
Test casegrpc__grpc-go-2735
Targetopencode/amdbeast/Kujira/Underdog-Saluki-27B-1.0-MTP-GGUF:latest
Prompt variantexplicit
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Run provenanceprovider_transport=opencode_attached_server_v1
Fixturetestdata/bench/grpc__grpc-go-2735.txtar

Vanilla LLM Prompt:

Use case(s) - what problem will this feature solve?

In some cases we want to retry connections more aggressively compared to the default behavior (e.g. for health checkers in tests in order to detect that server is up faster).

Proposed Solution

Add tunable parameters as described in the documentation.

It can be done by extending grpc.BackoffConfig struct (and therefore is not a breaking change).

MetricVanilla (Small)Semedit Explicit (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency—176.93s————
Process Start → First Event—35.50s————
First Event → First Tool Call—125.61s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns—1————
Internal Tool Cycles—1————
Initial Load / Discovery Turns—0————
MCP Discovery / Schema Turns—0————
Tool Invocations (per run)—1————
Output Tokens—4181————
Reasoning / Thinking Tokens—0————
Total Input Tokens—63622————
Cached Input Tokens—29459————
Uncached Input Tokens—34163————
Cost——————
Cached vs Uncached Token Ratio—0.86:1————
Oracle L1: Mutation Policy—PASS————
Oracle L2: AST Invariants—PASS————
Oracle L3: Clean Build—PASS————
Oracle L4: Verification Test—FAIL————
Semantic Tool Invocation Verified (per run)—PASS————
Baseline vs Semedit Explicit in Small Context
#VanillaSemedit MCP
1not publishedsemedit_semantic_batch (✓ ✗): Critical result (JSON): {"application_state":"unknown","error":"batch execution error: batch diagnostics before in \"/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_semedit-explicit_small-explicit_grpc__grpc-go-2735_1791720824982408000\": check Go diagnostics with gopls: load Go diagnostics session scope: err: exit status 1: stderr: go: cloud.google.com/go@v0.26.0: Get \"https://proxy.golang.org/cloud.google.com/go/@v/v0.26.0.mod\": dial tcp: lookup proxy.golang.org: no such host\
{
  "auto_organize_imports": false,
  "edits": [
    {
      "params": {
        "auto_organize_imports": false,
        "file": "internal/backoff/backoff.go",
        "language": "go",
        "symbol": "factor",
        "to": "multiplier"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "file": "internal/backoff/backoff.go",
        "group": "append",
        "kind": "const",
        "source": "const maxDelay = 120 * time.Second"
      },
      "tool": "semantic_insert_construct"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "file": "internal/backoff/backoff.go",
        "kind": "type",
        "source": "type Exponential struct {\n\t// baseDelay is the amount of time to backoff after the first connection\n\t// failure.\n\tbaseDelay time.Duration\n\t// multiplier is the factor with which to multiply backoffs after a failed\n\t// retry.\n\tmultiplier float64\n\t// jitter is the factor with which backoffs are randomized.\n\tjitter float64\n\t// maxDelay is the upper bound of backoff delay.\n\tmaxDelay time.Duration\n}",
        "symbol": "Exponential"
      },
      "tool": "semantic_replace_construct"
    },
    {
      "params": {
        "access_modifier": "public",
        "auto_organize_imports": false,
        "file": "internal/backoff/backoff.go",
        "kind": "type",
        "source": "// StrategyBuilder helps build an implementation of the Strategy interface,\n// with setters provided to set individual parameters of the backoff strategy.\ntype StrategyBuilder interface {\n\t// BaseDelay sets the amount of time to backoff after the first connection\n\t// failure.\n\tBaseDelay(time.Duration) StrategyBuilder\n\t// Multiplier sets the the factor with which to multiply backoffs after a\n\t// failed retry.\n\tMultiplier(float64) StrategyBuilder\n\t// Jitter sets the factor with which backoffs are randomized.\n\tJitter(float64) StrategyBuilder\n\t// MaxDelay sets the upper bound of backoff delay.\n\tMaxDelay(time.Duration) StrategyBuilder\n\t// Build builds an implementation of the Strategy interface configured using\n\t// the above setter methods.\n\tBuild() Strategy\n}"
      },
      "tool": "semantic_insert_construct"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "file": "internal/backoff/backoff.go",
        "kind": "type",
        "source": "// exponentialBuilder implements the StrategyBuilder interface for an\n// exponential backoff strategy.\ntype exponentialBuilder struct {\n\tbd     time.Duration\n\tmltpr  float64\n\tjitter float64\n\tmd     time.Duration\n}"
      },
      "tool": "semantic_insert_construct"
    },
    {
      "params": {
        "access_modifier": "public",
        "auto_organize_imports": false,
        "file": "internal/backoff/backoff.go",
        "kind": "function",
        "source": "// NewExponentialBuilder returns an implementation of the StrategyBuilder\n// interface used to build an exponential backoff strategy implementation, as\n// defined in\n// https://github.com/grpc/grpc/blob/master/doc/connection-backoff.md.\nfunc NewExponentialBuilder() StrategyBuilder {\n\treturn \u0026exponentialBuilder{\n\t\tbd:     baseDelay,\n\t\tmltpr:  multiplier,\n\t\tjitter: jitter,\n\t\tmd:     maxDelay,\n\t}\n}"
      },
      "tool": "semantic_insert_construct"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "file": "internal/backoff/backoff.go",
        "kind": "method",
        "source": "// BaseDelay helps implement StrategyBuilder.\nfunc (eb *exponentialBuilder) BaseDelay(bd time.Duration) StrategyBuilder {\n\teb.bd = bd\n\treturn eb\n}"
      },
      "tool": "semantic_insert_construct"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "file": "internal/backoff/backoff.go",
        "kind": "method",
        "source": "// BaseDelay helps implement StrategyBuilder.\nfunc (eb *exponentialBuilder) Multiplier(m float64) StrategyBuilder {\n\teb.mltpr = m\n\treturn eb\n}"
      },
      "tool": "semantic_insert_construct"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "file": "internal/backoff/backoff.go",
        "kind": "method",
        "source": "// BaseDelay helps implement StrategyBuilder.\nfunc (eb *exponentialBuilder) Jitter(j float64) StrategyBuilder {\n\teb.jitter = j\n\treturn eb\n}"
      },
      "tool": "semantic_insert_construct"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "file": "internal/backoff/backoff.go",
        "kind": "method",
        "source": "// BaseDelay helps implement StrategyBuilder.\nfunc (eb *exponentialBuilder) MaxDelay(md time.Duration) StrategyBuilder {\n\teb.md = md\n\treturn eb\n}"
      },
      "tool": "semantic_insert_construct"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "file": "internal/backoff/backoff.go",
        "kind": "method",
        "source": "// BaseDelay helps implement StrategyBuilder.\nfunc (eb *exponentialBuilder) Build() Strategy {\n\treturn Exponential{\n\t\tbaseDelay:  eb.bd,\n\t\tmultiplier: eb.mltpr,\n\t\tjitter:     eb.jitter,\n\t\tmaxDelay:   eb.md,\n\t}\n}"
      },
      "tool": "semantic_insert_construct"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "file": "internal/backoff/backoff.go",
        "kind": "method",
        "source": "// Backoff returns the amount of time to wait before the next retry given the\n// number of retries.\n// Helps implement Strategy.\nfunc (bc Exponential) Backoff(retries int) time.Duration {\n\tif retries == 0 {\n\t\treturn bc.baseDelay\n\t}\n\tbackoff, max := float64(bc.baseDelay), float64(bc.maxDelay)\n\tfor backoff \u003c max \u0026\u0026 retries \u003e 0 {\n\t\tbackoff *= bc.multiplier\n\t\tretries--\n\t}\n\tif backoff \u003e max {\n\t\tbackoff = max\n\t}\n\t// Randomize backoff delays so that if a cluster of requests start at\n\t// the same time, they won't operate in lockstep.\n\tbackoff *= 1 + bc.jitter*(grpcrand.Float64()*2-1)\n\tif backoff \u003c 0 {\n\t\treturn 0\n\t}\n\treturn time.Duration(backoff)\n}",
        "symbol": "Exponential.Backoff"
      },
      "tool": "semantic_replace_construct"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "file": "backoff.go",
        "kind": "type",
        "source": "// BackoffConfig defines the parameters for the default gRPC backoff strategy.\n// Deprecated: use ConnectParams instead.\ntype BackoffConfig struct {\n\t// MaxDelay is the upper bound of backoff delay.\n\tMaxDelay time.Duration\n}",
        "symbol": "BackoffConfig"
      },
      "tool": "semantic_replace_construct"
    },
    {
      "params": {
        "access_modifier": "public",
        "auto_organize_imports": false,
        "file": "backoff.go",
        "kind": "type",
        "source": "// ConnectParams defines the parameters for connecting and retrying. Users\n// are encouraged to use this instead of BackoffConfig.\n// https://github.com/grpc/grpc/blob/master/doc/connection-backoff.md.\n//\n// This API is EXPERIMENTAL.\ntype ConnectParams struct {\n\t// BackoffBaseDelay is the amount of time to backoff after the first\n\t// connection failure.\n\tBackoffBaseDelay time.Duration\n\t// BackoffMultiplier is the factor with which to multiply backoffs after a\n\t// failed retry. Should ideally be greater than 1.\n\tBackoffMultiplier float64\n\t// BackoffJitter is the factor with which backoffs are randomized.\n\tBackoffJitter float64\n\t// BackoffMaxDelay is the upper bound of backoff delay.\n\tBackoffMaxDelay time.Duration\n\t// MinConnectTimeout is the minimum amount of time we are willing to give a\n\t// connection to complete.\n\tMinConnectTimeout time.Duration\n}"
      },
      "tool": "semantic_insert_construct"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "file": "backoff.go",
        "kind": "var",
        "source": "// DefaultBackoffConfig uses values specified for backoff in\n// https://github.com/grpc/grpc/blob/master/doc/connection-backoff.md.\n// Deprecated: use ConnectParams instead.\nvar DefaultBackoffConfig = BackoffConfig{\n\tMaxDelay: 120 * time.Second,\n}",
        "symbol": "DefaultBackoffConfig"
      },
      "tool": "semantic_replace_construct"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "file": "dialoptions.go",
        "in_type": "dialOptions",
        "kind": "field",
        "placement": "type_end",
        "source": "mct time.Duration"
      },
      "tool": "semantic_insert_construct"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "file": "clientconn.go",
        "kind": "const",
        "symbol": "minConnectTimeout"
      },
      "tool": "semantic_remove_construct"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "file": "dialoptions.go",
        "group": "append",
        "kind": "const",
        "source": "const (\n\t// Minimum time to give a connection to complete.\n\tminConnectTimeout = 20 * time.Second\n)"
      },
      "tool": "semantic_insert_construct"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "file": "dialoptions.go",
        "source": "// WithConnectParams configures the dialer to use the provided backoff\n// parameters for the algorithm defined in\n// https://github.com/grpc/grpc/blob/master/doc/connection-backoff.md.\n// This will override all the default values with the ones provided here. So,\n// use with caution.\n//\n// This API is EXPERIMENTAL.\nfunc WithConnectParams(p ConnectParams) DialOption {\n\tb := backoff.NewExponentialBuilder()\n\tb.BaseDelay(p.BackoffBaseDelay).Multiplier(p.BackoffMultiplier).Jitter(p.BackoffJitter).MaxDelay(p.BackoffMaxDelay)\n\treturn newFuncDialOption(func(o *dialOptions) {\n\t\to.bs = b.Build()\n\t\to.mct = p.MinConnectTimeout\n\t})\n}"
      },
      "tool": "semantic_insert_function"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "file": "dialoptions.go",
        "kind": "function",
        "source": "// WithBackoffMaxDelay configures the dialer to use the provided maximum delay\n// when backing off after failed connection attempts.\n//\n// Deprecated: use WithConnectParams instead.\nfunc WithBackoffMaxDelay(md time.Duration) DialOption {\n\treturn WithBackoffConfig(BackoffConfig{MaxDelay: md})\n}",
        "symbol": "WithBackoffMaxDelay"
      },
      "tool": "semantic_replace_construct"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "file": "dialoptions.go",
        "kind": "function",
        "source": "// WithBackoffConfig configures the dialer to use the provided backoff\n// parameters after connection failures.\n//\n// Deprecated: use WithConnectParams instead.\nfunc WithBackoffConfig(b BackoffConfig) DialOption {\n\treturn withBackoff(backoff.NewExponentialBuilder().MaxDelay(b.MaxDelay).Build())\n}",
        "symbol": "WithBackoffConfig"
      },
      "tool": "semantic_replace_construct"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "body": "return dialOptions{\n\tdisableRetry:    !envconfig.Retry,\n\treqHandshake:    envconfig.RequireHandshake,\n\thealthCheckFunc: internal.HealthCheckFunc,\n\tcopts: transport.ConnectOptions{\n\t\tWriteBufferSize: defaultWriteBufSize,\n\t\tReadBufferSize:  defaultReadBufSize,\n\t},\n\tmct: minConnectTimeout,\n}",
        "file": "dialoptions.go",
        "symbol": "defaultDialOptions"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "discriminator": "cc.dopts.bs == nil",
        "file": "clientconn.go",
        "in_function": "DialContext",
        "kind": "if",
        "source": "if cc.dopts.bs == nil {\n\tcc.dopts.bs = backoff.NewExponentialBuilder().Build()\n}"
      },
      "tool": "semantic_replace_construct"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "discriminator": "dialDuration := minConnectTimeout",
        "file": "clientconn.go",
        "in_function": "addrConn.resetTransport",
        "kind": "statement/expression",
        "source": "dialDuration := ac.dopts.mct"
      },
      "tool": "semantic_replace_construct"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "file": "balancer/grpclb/grpclb.go",
        "language": "go",
        "symbol": "defaultBackoffConfig",
        "to": "defaultBackoffStrategy"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "file": "balancer/grpclb/grpclb.go",
        "source": "// defaultBackoffStrategy configures the backoff strategy that's used when the\n// init handshake in the RPC is unsuccessful. It's not for the clientconn\n// reconnect backoff.\n//\n// It has the same value as the default grpc.DefaultBackoffConfig.\n//\n// TODO: make backoff configurable.\nvar defaultBackoffStrategy = backoff.NewExponentialBuilder().Build()",
        "symbol": "defaultBackoffStrategy"
      },
      "tool": "semantic_replace_decl"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "file": "balancer/xds/xds_client.go",
        "language": "go",
        "symbol": "defaultBackoffConfig",
        "to": "defaultBackoffStrategy"
      },
      "tool": "semantic_rename"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "file": "balancer/xds/xds_client.go",
        "kind": "var",
        "source": "var defaultBackoffStrategy = backoff.NewExponentialBuilder().Build()",
        "symbol": "defaultBackoffStrategy"
      },
      "tool": "semantic_replace_construct"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "file": "health/client.go",
        "kind": "var",
        "source": "var backoffStrategy = backoff.NewExponentialBuilder().Build()",
        "symbol": "backoffStrategy"
      },
      "tool": "semantic_replace_construct"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "file": "health/client.go",
        "kind": "const",
        "safe_delete": false,
        "symbol": "maxDelay"
      },
      "tool": "semantic_remove_construct"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "file": "resolver/dns/dns_resolver.go",
        "kind": "type",
        "source": "// dnsResolver watches for the name resolution update for a non-IP target.\ntype dnsResolver struct {\n\tfreq       time.Duration\n\tbackoff    backoff.Strategy\n\tretryCount int\n\thost       string\n\tport       string\n\tresolver   netResolver\n\tctx        context.Context\n\tcancel     context.CancelFunc\n\tcc         resolver.ClientConn\n\t// rn channel is used by ResolveNow() to force an immediate resolution of the target.\n\trn chan struct{}\n\tt  *time.Timer\n\t// wg is used to enforce Close() to return after the watcher() goroutine has finished.\n\t// Otherwise, data race will be possible. [Race Example] in dns_resolver_test we\n\t// replace the real lookup functions with mocked ones to facilitate testing.\n\t// If Close() doesn't wait for watcher() goroutine finishes, race detector sometimes\n\t// will warns lookup (READ the lookup function pointers) inside watcher() goroutine\n\t// has data race with replaceNetFunc (WRITE the lookup function pointers).\n\twg                   sync.WaitGroup\n\tdisableServiceConfig bool\n}",
        "symbol": "dnsResolver"
      },
      "tool": "semantic_replace_construct"
    },
    {
      "params": {
        "auto_organize_imports": false,
        "body": "\thost, port, err := parseTarget(target.Endpoint, defaultPort)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\t// IP address.\n\tif net.ParseIP(host) != nil {\n\t\thost, _ = formatIP(host)\n\t\taddr := []resolver.Address{{Addr: host + \":\" + port}}\n\t\ti := \u0026ipResolver{\n\t\t\tcc: cc,\n\t\t\tip: addr,\n\t\t\trn: make(chan struct{}, 1),\n\t\t\tq:  make(chan struct{}),\n\t\t}\n\t\tcc.NewAddress(addr)\n\t\tgo i.watcher()\n\t\treturn i, nil\n\t}\n\n\t// DNS address (non-IP).\n\tctx, cancel := context.WithCancel(context.Background())\n\td := \u0026dnsResolver{\n\t\tfreq:                 b.minFreq,\n\t\tbackoff:              backoff.NewExponentialBuilder().MaxDelay(b.minFreq).Build(),\n\t\thost:                 host,\n\t\tport:                 port,\n\t\tctx:                  ctx,\n\t\tcancel:               cancel,\n\t\tcc:                   cc,\n\t\tt:                    time.NewTimer(0),\n\t\trn:                   make(chan struct{}, 1),\n\t\tdisableServiceConfig: opts.DisableServiceConfig,\n\t}\n\n\tif target.Authority == \"\" {\n\t\td.resolver = defaultResolver\n\t} else {\n\t\td.resolver, err = customAuthorityResolver(target.Authority)\n\t\tif err != nil {\n\t\t\treturn nil, err\n\t\t}\n\t}\n\n\td.wg.Add(1)\n\tgo d.watcher()\n\treturn d, nil",
        "file": "resolver/dns/dns_resolver.go",
        "symbol": "(*dnsBuilder).Build"
      },
      "tool": "semantic_replace_body"
    }
  ],
  "trust_workspace": true
}
Baseline vs Semedit Explicit in Large Context
#VanillaSemedit MCP
1not publishednot published

Test case: zeromicro__go-zero-964

Target: opencode/amdbeast/Kujira/Underdog-Saluki-27B-1.0-MTP-GGUF:latest

explicit prompt · prescriptive MCP instructions · write semedit restriction

LocalityContextAmbiguityCompositionInvariantsScaleStaging
Task complexity across locality, context, ambiguity, composition, invariants, edit scale, and staging (1–5).
SettingValue
Test casezeromicro__go-zero-964
Targetopencode/amdbeast/Kujira/Underdog-Saluki-27B-1.0-MTP-GGUF:latest
Prompt variantexplicit
MCP server instructionsprescriptive
Semedit arm restrictionwrite
Run provenanceprovider_transport=opencode_attached_server_v1
Fixturetestdata/bench/zeromicro__go-zero-964.txtar

Vanilla LLM Prompt:

如题。 etcd打开了rbac authentication。但discov.EtcdConf中没有user等字段,无法进行身份认证。连接etcd时会报错: etcdserver: user name is empty

MetricVanilla (Small)Semedit Explicit (Small)Δ (Small)Vanilla (Large)MCP (Large)Δ (Large)
Wall-Clock Latency—114.52s————
Process Start → First Event—29.50s————
First Event → First Tool Call—54.75s————
MCP Initialize → First Semantic Call——————
MCP Server Start → Initialize——————
Top-Level User Turns—1————
Internal Tool Cycles—1————
Initial Load / Discovery Turns—0————
MCP Discovery / Schema Turns—0————
Tool Invocations (per run)—1————
Output Tokens—2928————
Reasoning / Thinking Tokens—0————
Total Input Tokens—55641————
Cached Input Tokens—26441————
Uncached Input Tokens—29200————
Cost——————
Cached vs Uncached Token Ratio—0.91:1————
Oracle L1: Mutation Policy—PASS————
Oracle L2: AST Invariants—PASS————
Oracle L3: Clean Build—PASS————
Oracle L4: Verification Test—FAIL————
Semantic Tool Invocation Verified (per run)—PASS————
Baseline vs Semedit Explicit in Small Context
#VanillaSemedit MCP
1not publishedsemedit_semantic_batch (✓ ✗): Critical result (JSON): {"application_state":"unknown","error":"batch execution error: batch diagnostics before in \"/Users/alessandro/sources/semantic-editor/.scratch/benchmarks/run_opencode_semedit-explicit_small-explicit_zeromicro__go-zero-964_1791724188437716000\": check Go diagnostics with gopls: load Go diagnostics session scope: err: exit status 1: stderr: go: github.com/ClickHouse/clickhouse-go@v1.4.3: Get \"https://proxy.golang.org/github.com/%21click%21house/clickhouse-go/@v/v1.4.3.mod\": dial tcp
{
  "auto_organize_imports": true,
  "edits": [
    {
      "params": {
        "file": "core/discov/config.go",
        "kind": "type",
        "source": "type EtcdConf struct {\n\tHosts []string\n\tKey   string\n\tUser  string `json:\",optional\"`\n\tPass  string `json:\",optional=User\"`\n}",
        "symbol": "EtcdConf"
      },
      "tool": "semantic_replace_construct"
    },
    {
      "params": {
        "body": "if len(c.Hosts) == 0 {\n\treturn errors.New(\"empty etcd hosts\")\n}\nif len(c.Key) == 0 {\n\treturn errors.New(\"empty etcd key\")\n}\nif (c.User == \"\") != (c.Pass == \"\") {\n\treturn errors.New(\"etcd user and pass must be paired\")\n}\nreturn nil",
        "file": "core/discov/config.go",
        "symbol": "EtcdConf.Validate"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "file": "core/discov/config.go",
        "source": "func (c EtcdConf) EnableAuth() bool {\n\treturn c.User != \"\" \u0026\u0026 c.Pass != \"\"\n}"
      },
      "tool": "semantic_insert_function"
    },
    {
      "params": {
        "file": "core/discov/internal/registry.go",
        "kind": "type",
        "source": "type cluster struct {\n\tendpoints  []string\n\tuser       string\n\tpass       string\n\tkey        string\n\tvalues     map[string]map[string]string\n\tlisteners  map[string][]UpdateListener\n\twatchGroup *threading.RoutineGroup\n\tdone       chan lang.PlaceholderType\n\tlock       sync.Mutex\n}",
        "symbol": "cluster"
      },
      "tool": "semantic_replace_construct"
    },
    {
      "params": {
        "file": "core/discov/internal/registry.go",
        "kind": "function",
        "source": "func DialClient(endpoints []string, user, pass string) (EtcdClient, error) {\n\treturn clientv3.New(clientv3.Config{\n\t\tEndpoints:            endpoints,\n\t\tAutoSyncInterval:     autoSyncInterval,\n\t\tDialTimeout:          DialTimeout,\n\t\tDialKeepAliveTime:    dialKeepAliveTime,\n\t\tDialKeepAliveTimeout: DialTimeout,\n\t\tRejectOldCluster:     true,\n\t\tUsername:             user,\n\t\tPassword:             pass,\n\t})\n}",
        "symbol": "DialClient"
      },
      "tool": "semantic_replace_construct"
    },
    {
      "params": {
        "body": "cli, err := NewClient(c.endpoints, c.user, c.pass)\nif err != nil {\n\treturn nil, err\n}\n\ngo c.watchConnState(cli)\n\nreturn cli, nil",
        "file": "core/discov/internal/registry.go",
        "symbol": "cluster.newClient"
      },
      "tool": "semantic_replace_body"
    },
    {
      "params": {
        "file": "core/discov/internal/registry.go",
        "source": "func newClusterWithAuth(endpoints []string, user, pass string) *cluster {\n\tc := newCluster(endpoints)\n\tc.user = user\n\tc.pass = pass\n\treturn c\n}"
      },
      "tool": "semantic_insert_function"
    },
    {
      "params": {
        "file": "core/discov/internal/registry.go",
        "source": "func (r *Registry) getClusterWithAuth(endpoints []string, user, pass string) *cluster {\n\tclusterKey := fmt.Sprintf(\"%s|%q|%q\", getClusterKey(endpoints), user, pass)\n\tr.lock.Lock()\n\tdefer r.lock.Unlock()\n\tc, ok := r.clusters[clusterKey]\n\tif !ok {\n\t\tc = newClusterWithAuth(endpoints, user, pass)\n\t\tr.clusters[clusterKey] = c\n\t}\n\treturn c\n}"
      },
      "tool": "semantic_insert_function"
    },
    {
      "params": {
        "file": "core/discov/internal/registry.go",
        "source": "func (r *Registry) GetConnWithAuth(endpoints []string, user, pass string) (EtcdClient, error) {\n\treturn r.getClusterWithAuth(endpoints, user, pass).getClient()\n}"
      },
      "tool": "semantic_insert_function"
    },
    {
      "params": {
        "file": "core/discov/internal/registry.go",
        "source": "func (r *Registry) MonitorWithAuth(endpoints []string, user, pass, key string, l UpdateListener) error {\n\treturn r.getClusterWithAuth(endpoints, user, pass).monitor(key, l)\n}"
      },
      "tool": "semantic_insert_function"
    },
    {
      "params": {
        "file": "core/discov/publisher.go",
        "kind": "type",
        "source": "// A Publisher can be used to publish the value to an etcd cluster on the given key.\ntype Publisher struct {\n\tendpoints  []string\n\tuser       string\n\tpass       string\n\tkey        string\n\tfullKey    string\n\tid         int64\n\tvalue      string\n\tlease      clientv3.LeaseID\n\tquit       *syncx.DoneChan\n\tpauseChan  chan lang.PlaceholderType\n\tresumeChan chan lang.PlaceholderType\n}",
        "symbol": "Publisher"
      },
      "tool": "semantic_replace_construct"
    },
    {
      "params": {
        "file": "core/discov/publisher.go",
        "source": "func NewPublisherWithAuth(endpoints []string, user, pass, key, value string, opts ...PublisherOption) *Publisher {\n\tpublisher := \u0026Publisher{\n\t\tendpoints:  endpoints,\n\t\tuser:       user,\n\t\tpass:       pass,\n\t\tkey:        key,\n\t\tvalue:      value,\n\t\tquit:       syncx.NewDoneChan(),\n\t\tpauseChan:  make(chan lang.PlaceholderType),\n\t\tresumeChan: make(chan lang.PlaceholderType),\n\t}\n\tfor _, opt := range opts {\n\t\topt(publisher)\n\t}\n\treturn publisher\n}"
      },
      "tool": "semantic_insert_function"
    },
    {
      "params": {
        "file": "core/discov/publisher.go",
        "source": "func (p *Publisher) KeepAliveWithAuth() error {\n\tcli, err := internal.GetRegistry().GetConnWithAuth(p.endpoints, p.user, p.pass)\n\tif err != nil {\n\t\treturn err\n\t}\n\tp.lease, err = p.register(cli)\n\tif err != nil {\n\t\treturn err\n\t}\n\tproc.AddWrapUpListener(func() {\n\t\tp.Stop()\n\t})\n\treturn p.keepAliveAsync(cli)\n}"
      },
      "tool": "semantic_insert_function"
    },
    {
      "params": {
        "file": "zrpc/internal/rpcpubserver.go",
        "kind": "function",
        "source": "func NewRpcPubServer(etcdEndpoints []string, etcdKey, listenOn string, opts ...ServerOption) (Server, error) {\n\tregisterEtcd := func() error {\n\t\tpubListenOn := figureOutListenOn(listenOn)\n\t\tpubClient := discov.NewPublisher(etcdEndpoints, etcdKey, pubListenOn)\n\t\treturn pubClient.KeepAlive()\n\t}\n\treturn newKeepAliveServer(registerEtcd, listenOn, opts...), nil\n}",
        "symbol": "NewRpcPubServer"
      },
      "tool": "semantic_replace_construct"
    },
    {
      "params": {
        "file": "zrpc/internal/rpcpubserver.go",
        "source": "func NewRpcPubServerWithEtcdAuth(etcdEndpoints []string, user, pass, etcdKey, listenOn string, opts ...ServerOption) (Server, error) {\n\tregisterEtcd := func() error {\n\t\tpubListenOn := figureOutListenOn(listenOn)\n\t\tpubClient := discov.NewPublisherWithAuth(etcdEndpoints, user, pass, etcdKey, pubListenOn)\n\t\treturn pubClient.KeepAliveWithAuth()\n\t}\n\treturn newKeepAliveServer(registerEtcd, listenOn, opts...), nil\n}"
      },
      "tool": "semantic_insert_function"
    },
    {
      "params": {
        "file": "zrpc/internal/rpcpubserver.go",
        "source": "func newKeepAliveServer(registerEtcd func() error, listenOn string, opts ...ServerOption) keepAliveServer {\n\treturn keepAliveServer{\n\t\tregisterEtcd: registerEtcd,\n\t\tServer:       NewRpcServer(listenOn, opts...),\n\t}\n}"
      },
      "tool": "semantic_insert_function"
    },
    {
      "params": {
        "file": "zrpc/server.go",
        "kind": "function",
        "source": "func NewServer(c RpcServerConf, register internal.RegisterFn) (*RpcServer, error) {\n\tvar err error\n\tif err = c.Validate(); err != nil {\n\t\treturn nil, err\n\t}\n\n\tvar server internal.Server\n\tmetrics := stat.NewMetrics(c.ListenOn)\n\tserverOptions := []internal.ServerOption{internal.WithMetrics(metrics), internal.WithMaxRetries(c.MaxRetries)}\n\n\tif c.HasEtcd() {\n\t\tif c.Etcd.EnableAuth() {\n\t\t\tserver, err = internal.NewRpcPubServerWithEtcdAuth(c.Etcd.Hosts, c.Etcd.User, c.Etcd.Pass, c.Etcd.Key, c.ListenOn, serverOptions...)\n\t\t} else {\n\t\t\tserver, err = internal.NewRpcPubServer(c.Etcd.Hosts, c.Etcd.Key, c.ListenOn, serverOptions...)\n\t\t}\n\t\tif err != nil {\n\t\t\treturn nil, err\n\t\t}\n\t} else {\n\t\tserver = internal.NewRpcServer(c.ListenOn, serverOptions...)\n\t}\n\n\tserver.SetName(c.Name)\n\tif err = setupInterceptors(server, c, metrics); err != nil {\n\t\treturn nil, err\n\t}\n\n\trpcServer := \u0026RpcServer{\n\t\tserver:   server,\n\t\tregister: register,\n\t}\n\tif err = c.SetUp(); err != nil {\n\t\treturn nil, err\n\t}\n\n\treturn rpcServer, nil\n}",
        "symbol": "NewServer"
      },
      "tool": "semantic_replace_construct"
    }
  ],
  "trust_workspace": true
}
Baseline vs Semedit Explicit in Large Context
#VanillaSemedit MCP
1not publishednot published