Benchmark run luna6-multiswe-go-smoke-20261007-r3

This page contains exactly the publishable benchmark observations recorded in run luna6-multiswe-go-smoke-20261007-r3; it does not select or aggregate them. Return to the run index or view aggregate metrics.

Interactive browser

Rows are pre-aggregated by task, target, prompt variant, MCP-instruction mode, context, arm, and restriction. Numeric values show median (p50), IQR, p90, maximum, and sample count. Boolean checks show pass counts and success rates. Filters apply to these summary groups; per-run observations remain available on their run pages.

Requirement met Expected semantic tools used Model Prompt variant Semedit restriction MCP instructions

Loading benchmark results…

Baseline vs semedit

Comparison evaluates baseline and semedit values within each restriction policy, target, and context group. Negative changes mark improvements for these lower-is-better metrics; positive changes mark degradations.

Requirement met Expected semantic tools used Model Prompt variant Semedit restriction MCP instructions

Detailed observations

Empirical telemetry measuring wall-clock latency, token usage, tool turns, and compiler correctness when AI coding agents perform code modifications.

Every benchmark pairs Vanilla LLM (standard file editing tools) against Semedit MCP (deterministic AST compiler operations).

Benchmark Methodology & Transparency

Reproducibility & Provenance: Self-contained txtar archives define test scenarios. Links below resolve to commit-anchored GitHub source files for published commits, or cryptographic SHA-256 fingerprints for uncommitted local fixtures.
Model Cost: Tables report model-specific credit consumption where a target model has a declared schedule: (uncached input × input credits + cached input × cached credits + (reasoning + visible output) × output credits) / 1,000,000. Thinking tokens use the output rate. ADR-0043 defines the per-million-token credit rates.
Multi-Level Correctness Oracle: Four validation levels evaluate each trial:
1. Level 1 (Mutation Policy): Restricts file modifications strictly to authorized paths.
2. Level 2 (AST Invariants): Compiler AST verification of required symbols, imports, and relative declaration ordering.
3. Level 3 (Clean Build): Strict compilation validation with zero compiler or typecheck errors.
4. Level 4 (Test Suite): Unit and integration test suite execution.
Isolated Sandboxing: Each trial executes in an ephemeral directory with pristine git and module state.

Test case: `grpc__grpc-go-2735`

Target: `codex/gpt-6-luna/high`

default prompt · none MCP instructions · write semedit restriction

Task complexity across locality, context, ambiguity, composition, invariants, edit scale, and staging (1–5).

Setting	Value
Test case	`grpc__grpc-go-2735`
Target	`codex/gpt-6-luna/high`
Prompt variant	`default`
MCP server instructions	`none`
Semedit arm restriction	`write`
Run provenance	`coding_network=disabled,oracle_patch_installation=checked`
Fixture	testdata/bench/grpc__grpc-go-2735.txtar

LLM Prompt:

Use case(s) - what problem will this feature solve?
In some cases we want to retry connections more aggressively compared to the default behavior (e.g. for health checkers in tests in order to detect that server is up faster).
Proposed Solution
Add tunable parameters as described in the documentation.
It can be done by extending grpc.BackoffConfig struct (and therefore is not a breaking change).

Target: `codex/gpt-6-luna/medium`

default prompt · none MCP instructions · write semedit restriction

Task complexity across locality, context, ambiguity, composition, invariants, edit scale, and staging (1–5).

Setting	Value
Test case	`grpc__grpc-go-2735`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`none`
Semedit arm restriction	`write`
Run provenance	`coding_network=disabled,oracle_patch_installation=checked`
Fixture	testdata/bench/grpc__grpc-go-2735.txtar

LLM Prompt:

Use case(s) - what problem will this feature solve?
In some cases we want to retry connections more aggressively compared to the default behavior (e.g. for health checkers in tests in order to detect that server is up faster).
Proposed Solution
Add tunable parameters as described in the documentation.
It can be done by extending grpc.BackoffConfig struct (and therefore is not a breaking change).

default prompt · none MCP instructions · write semedit restriction

Task complexity across locality, context, ambiguity, composition, invariants, edit scale, and staging (1–5).

Setting	Value
Test case	`grpc__grpc-go-2735`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`none`
Semedit arm restriction	`write`
Run provenance	`coding_network=disabled,oracle_patch_installation=checked`
Fixture	testdata/bench/grpc__grpc-go-2735.txtar

LLM Prompt:

Use case(s) - what problem will this feature solve?
In some cases we want to retry connections more aggressively compared to the default behavior (e.g. for health checkers in tests in order to detect that server is up faster).
Proposed Solution
Add tunable parameters as described in the documentation.
It can be done by extending grpc.BackoffConfig struct (and therefore is not a breaking change).

Metric	Vanilla (Small)	⚠ Semedit Agentic (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	43.86s	69.75s	N/A	—	—	—
Process Start → First Event	0.76s	0.15s	—	—	—	—
First Event → First Tool Call	3.23s	4.93s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	1	1	N/A	—	—	—
Internal Tool Cycles	11	9	N/A	—	—	—
Initial Load / Discovery Turns	5	3	N/A	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	11	9	N/A	—	—	—
Output Tokens	2321	2576	N/A	—	—	—
Reasoning / Thinking Tokens	787	720	N/A	—	—	—
Total Input Tokens	296682	272839	N/A	—	—	—
Cached Input Tokens	268288	243968	N/A	—	—	—
Uncached Input Tokens	28394	28871	N/A	—	—	—
Cost	0.1769	0.1744	N/A	—	—	—
Cached vs Uncached Token Ratio	9.45:1	8.45:1	N/A	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	FAIL	FAIL	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	FAIL	—	—	—	—

Baseline vs Semedit Agentic in Small Context

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'backoff' -g '*.go' \| head -80"`	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'backoff' -g '*.go' \| head -100"`
2	✓ ✓ `/bin/zsh -lc "cat PROJECT_BRIEF.md 2>/dev/null; rg -n 'BackoffConfig\|backoff' --glob '*.go' \| head -100; find .. -name AGENTS.md -print"`	✓ ✓ `/bin/zsh -lc "rg -n \"BackoffConfig\|backoff\" --glob '*.go' \| head -160; find .. -name AGENTS.md -o -name PROJECT_BRIEF.md"`
3	✓ ✓ `/bin/zsh -lc "cat backoff.go; sed -n '30,100p' internal/backoff/backoff.go; sed -n '220,280p' dialoptions.go; cat go.mod \| head -8"`	✓ ✓ `/bin/zsh -lc "cat backoff.go && \ cat internal/backoff/backoff.go && \ sed -n '240,275p' dialoptions.go && \ sed -n '810,865p' clientconn_test.go && \ rg -n 'BackoffConfig\\{' --glob '*.go'"`
4	✓ ✓ `/bin/zsh -lc "rg -n 'minConnectTimeout\|MIN_CONNECT_TIMEOUT\|baseDelay\|factor\|jitter' clientconn.go internal/backoff backoff.go"`	`file_change` (✓ ✓)
5	✓ ✓ `/bin/zsh -lc ''`	`file_change` (✓ ✓)
6	`file_change` (✓ ✓)	`file_change` (✓ ✓)
7	`file_change` (✓ ✓)	`file_change` (✓ ✓)
8	`file_change` (✓ ✓)	✓ ✓ `/bin/zsh -lc 'gofmt -w backoff.go dialoptions.go internal/backoff/backoff.go && \ git diff -- backoff.go dialoptions.go internal/backoff/backoff.go'`
9	`file_change` (✓ ✓)	`file_change` (✓ ✓)
10	`file_change` (✓ ✓)	—
11	✓ ✓ `/bin/zsh -lc 'gofmt -w backoff.go dialoptions.go internal/backoff/backoff.go && \ git diff -- backoff.go dialoptions.go internal/backoff/backoff.go'`	—

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

Test case: `grpc__grpc-go-2951`

Target: `codex/gpt-6-luna/medium`

default prompt · none MCP instructions · write semedit restriction

Task complexity across locality, context, ambiguity, composition, invariants, edit scale, and staging (1–5).

Setting	Value
Test case	`grpc__grpc-go-2951`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`none`
Semedit arm restriction	`write`
Run provenance	`coding_network=disabled,oracle_patch_installation=checked`
Fixture	testdata/bench/grpc__grpc-go-2951.txtar

LLM Prompt:

If a resolver could not resolve a name, there’s currently no way to report error back to ClientConn, and RPCs on the ClientConn will fail with timeout.
This method allows resolver to report error to ClientConn, and failfast RPCs should fail with this error (or RPC errors that contain info from this error).

default prompt · none MCP instructions · write semedit restriction

Task complexity across locality, context, ambiguity, composition, invariants, edit scale, and staging (1–5).

Setting	Value
Test case	`grpc__grpc-go-2951`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`none`
Semedit arm restriction	`write`
Run provenance	`coding_network=disabled,oracle_patch_installation=checked`
Fixture	testdata/bench/grpc__grpc-go-2951.txtar

LLM Prompt:

If a resolver could not resolve a name, there’s currently no way to report error back to ClientConn, and RPCs on the ClientConn will fail with timeout.
This method allows resolver to report error to ClientConn, and failfast RPCs should fail with this error (or RPC errors that contain info from this error).

default prompt · none MCP instructions · write semedit restriction

Task complexity across locality, context, ambiguity, composition, invariants, edit scale, and staging (1–5).

Setting	Value
Test case	`grpc__grpc-go-2951`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`none`
Semedit arm restriction	`write`
Run provenance	`coding_network=disabled,oracle_patch_installation=checked`
Fixture	testdata/bench/grpc__grpc-go-2951.txtar

LLM Prompt:

If a resolver could not resolve a name, there’s currently no way to report error back to ClientConn, and RPCs on the ClientConn will fail with timeout.
This method allows resolver to report error to ClientConn, and failfast RPCs should fail with this error (or RPC errors that contain info from this error).

Test case: `grpc__grpc-go-3361`

Target: `codex/gpt-6-luna/high`

default prompt · none MCP instructions · write semedit restriction

Task complexity across locality, context, ambiguity, composition, invariants, edit scale, and staging (1–5).

Setting	Value
Test case	`grpc__grpc-go-3361`
Target	`codex/gpt-6-luna/high`
Prompt variant	`default`
MCP server instructions	`none`
Semedit arm restriction	`write`
Run provenance	`coding_network=disabled,oracle_patch_installation=checked`
Fixture	testdata/bench/grpc__grpc-go-3361.txtar

LLM Prompt:

What version of gRPC are you using?
1.26
What version of Go are you using (go version)?
1.13.4
What operating system (Linux, Windows, …) and version?
Linux / Kubernetes / Docker - various.
What did you do?
When using grpc.WithBalancerName(roundrobin.Name), with a dns:/// Dial target, errors do not trigger re-resolve.
Detail: a client starts with dns:///my-service as a dial target and the round_robin load balancer. my-service DNS returns an IP 192.168.0.19. Due to an operational change, my-service is moved and DNS is updated to return 192.168.0.20. The service stops running on 192.168.0.19, refusing connections.
This is approximately the workflow that would occur when using Kubernetes and a headless service if a pod is replaced. This happens for a variety of reasons, including routine updates and rescheduling.
What did you expect to see?
(a) Upon receiving a connection refused, DNS should be re-queried for an updated endpoint list (b) Without receiving an error from an existing endpoint, the load balancer should be repopulated occasionally with the latest records in DNS as endpoint sets change
This was the behavior previous to 1.26. By rolling code back to grpc 1.25.1, we were able to restore this behavior.
https://github.com/grpc/grpc-go/pull/3165 is likely the culprit.
What did you see instead?
(a) Endpoint list is never re-queried and the client simply throws a persistent error:
rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 10.1.5.253:443: connect: connection refused"
(b) Since every subsequent request fails, the endpoint list does not seem to getting updated with new records in DNS at all.
Additional
I recognize the API around DNS and load balancing may have changed recently (and are evolving) but I’m not seeing any clear indication of how to retain the expected behavior.

Metric	Vanilla (Small)	⚠ Semedit Agentic (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	720.41s	679.28s	N/A	—	—	—
Process Start → First Event	0.82s	0.57s	—	—	—	—
First Event → First Tool Call	7.78s	4.60s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	1	1	N/A	—	—	—
Internal Tool Cycles	14	18	N/A	—	—	—
Initial Load / Discovery Turns	10	12	N/A	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	14	18	N/A	—	—	—
Output Tokens	7410	6701	N/A	—	—	—
Reasoning / Thinking Tokens	5048	4375	N/A	—	—	—
Total Input Tokens	500637	680260	N/A	—	—	—
Cached Input Tokens	461056	622592	N/A	—	—	—
Uncached Input Tokens	39581	57668	N/A	—	—	—
Cost	0.3699	0.4383	N/A	—	—	—
Cached vs Uncached Token Ratio	11.65:1	10.80:1	N/A	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	FAIL	FAIL	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	FAIL	—	—	—	—

Baseline vs Semedit Agentic in Small Context


/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g '*dns*' -g '*roundrobin*' | head -100"
/bin/zsh -lc "sed -n '1,240p' resolver/dns/dns_resolver.go && \
sed -n '1,220p' balancer/roundrobin/roundrobin.go"
/bin/zsh -lc "rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'README.md' | head -100"
/bin/zsh -lc 'rg -n "ResolveNow|ResolveNow|ResolveNow|ReportError|ExitIdle|UpdateState|NewAddress|StateListener|resolver" balancer resolver internal/resolver naming'
/bin/zsh -lc 'git status --short && \
git log -5 --oneline'
/bin/zsh -lc "sed -n '35,235p' internal/resolver/dns/dns_resolver.go && \
sed -n '300,445p' balancer/balancer.go && \
sed -n '1,240p' balancer/base/base.go && \
git show --stat --oneline 648cf9b0"
/bin/zsh -lc "git log --oneline --all -- internal/resolver/dns/dns_resolver.go | head -20 && \
git log --oneline --all --grep='3165\\|ResolveNow\\|DNS.*resolve' | head -30 && \
sed -n '80,230p' balancer/base/balancer.go"
/bin/zsh -lc 'git show --format=fuller --stat e5e980f2 && \
git show e5e980f2 -- internal/resolver/dns/dns_resolver.go'
/bin/zsh -lc 'git log --oneline -12 && \
git branch -a --contains e5e980f2 | head'
/bin/zsh -lc "rg -n \"ResolveNow\\(\" --glob='*.go' | head -60 && \
git show e5e980f2:internal/resolver/dns/dns_resolver.go | sed -n '1,240p'"
/bin/zsh -lc "sed -n '1,210p' pickfirst.go && \
sed -n '180,225p' balancer_conn_wrappers.go && \
sed -n '180,245p' resolver_conn_wrapper.go"
/bin/zsh -lc "sed -n '830,920p' internal/resolver/dns/dns_resolver_test.go && \
sed -n '1210,1325p' internal/resolver/dns/dns_resolver_test.go && \
rg -n \"minDNSResRate\""
/bin/zsh -lc 'gofmt -w internal/resolver/dns/dns_resolver.go && \
git diff -- internal/resolver/dns/dns_resolver.go && \
git status --short'
/bin/zsh -lc "git diff --check && \
git status --short && \
nl -ba internal/resolver/dns/dns_resolver.go | sed -n '194,250p'"

/bin/zsh -lc 'gofmt -w internal/resolver/dns/dns_resolver.go balancer/base/balancer.go && \
git diff --check && \
git diff -- internal/resolver/dns/dns_resolver.go balancer/base/balancer.go'

/bin/zsh -lc 'git diff --check && \
git diff --stat && \
git status --short && \
git diff -- internal/resolver/dns/dns_resolver.go balancer/base/balancer.go'

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'dns' -g 'roundrobin' \| head -100 && \ git status --short && \ git log -5 --oneline"`	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'dns' -g 'roundrobin' \| head -100"`
2	✓ ✓ `/bin/zsh -lc "cat go.mod && \ sed -n '1,260p' internal/resolver/dns/dns_resolver.go && \ sed -n '1,220p' balancer/roundrobin/roundrobin.go && \ rg -n \"ResolveNow\|MinResolutionInterval\|Refresh\|resolve\" internal/resolver/dns resolver/dns balancer/roundrobin"`	✓ ✓ `/bin/zsh -lc "sed -n '1,240p' resolver/dns/dns_resolver.go && \ sed -n '1,220p' balancer/roundrobin/roundrobin.go"`
3	✓ ✓ `/bin/zsh -lc "sed -n '250,380p' internal/resolver/dns/dns_resolver.go && \ rg -n \"ResolveNow\\(\|ReportError\|SubConnState\|TRANSIENT_FAILURE\|UpdateState\" clientconn.go balancer \| head -160 && \ git log --oneline --all --grep='3165' && \ git remote -v"`	✓ ✓ `/bin/zsh -lc "rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'README.md' \| head -100"`
4	? ✗ `/bin/zsh -lc "git show --stat --oneline e5e980f2 && \ git show --format=fuller --no-ext-diff e5e980f2 -- internal/resolver/dns/dns_resolver.go clientconn.go balancer \| head -260 && \ sed -n '650,735p' clientconn.go && \ sed -n '150,235p' balancer/base/balancer.go"`	✓ ✓ `/bin/zsh -lc 'rg -n "ResolveNow\|ResolveNow\|ResolveNow\|ReportError\|ExitIdle\|UpdateState\|NewAddress\|StateListener\|resolver" balancer resolver internal/resolver naming'`
5	✓ ✓ `/bin/zsh -lc "git show --format=fuller --no-ext-diff e5e980f2 -- internal/resolver/dns/dns_resolver.go clientconn.go balancer \| head -300 && \ sed -n '690,725p' clientconn.go && \ rg -n \"ResolveNow\|handleSubConnStateChange\" balancer/base/balancer.go clientconn.go"`	✓ ✓ `/bin/zsh -lc 'git status --short && \ git log -5 --oneline'`
6	✓ ✓ `/bin/zsh -lc "git show --name-status --format=oneline e5e980f2 \| head -100 && \ sed -n '930,975p' clientconn.go && \ sed -n '1050,1110p' clientconn.go && \ sed -n '1,260p' balancer/base/balancer.go"`	✓ ✓ `/bin/zsh -lc "sed -n '35,235p' internal/resolver/dns/dns_resolver.go && \ sed -n '300,445p' balancer/balancer.go && \ sed -n '1,240p' balancer/base/base.go && \ git show --stat --oneline 648cf9b0"`
7	? ✗ `/bin/zsh -lc "sed -n '1090,1195p' clientconn.go && \ git show e5e980f2:internal/resolver/dns/dns_resolver.go \| rg -n \"ticker\|Timer\|ResolveNow\|watcher\|for \\{\" && \ git cat-file -p e5e980f2 \| head -8 && \ rg -n \"poll\|periodic\|30 seconds\|30s\|ResolveNow\" internal/resolver/dns/dns_resolver_test.go"`	✓ ✓ `/bin/zsh -lc "git log --oneline --all -- internal/resolver/dns/dns_resolver.go \| head -20 && \ git log --oneline --all --grep='3165\\\|ResolveNow\\\|DNS.*resolve' \| head -30 && \ sed -n '80,230p' balancer/base/balancer.go"`
8	? ✗ `/bin/zsh -lc "rg -n \"func \\(.resolverWrapper.resolveNow\|resolveNow\\(\" resolver_wrapper.go clientconn.go && \ sed -n '120,250p' resolver_wrapper.go && \ rg -n \"dns\|ResolveNow\" internal/resolver/dns/dns_resolver_test.go \| tail -35"`	? ✗ `/bin/zsh -lc 'git show --format=fuller --stat e5e980f2 && \ git show e5e980f2 -- internal/resolver/dns/dns_resolver.go'`
9	✓ ✓ `/bin/zsh -lc "rg -n \"type ccResolverWrapper\|func \\(.*resolveNow\|ResolverWrapper\""`	✓ ✓ `/bin/zsh -lc 'git log --oneline -12 && \ git branch -a --contains e5e980f2 \| head'`
10	✓ ✓ `/bin/zsh -lc "sed -n '38,145p' resolver_conn_wrapper.go"`	✓ ✓ `/bin/zsh -lc "rg -n \"ResolveNow\\(\" --glob='*.go' \| head -60 && \ git show e5e980f2:internal/resolver/dns/dns_resolver.go \| sed -n '1,240p'"`
11	`file_change` (✓ ✓)	✓ ✓ `/bin/zsh -lc "sed -n '1,210p' pickfirst.go && \ sed -n '180,225p' balancer_conn_wrappers.go && \ sed -n '180,245p' resolver_conn_wrapper.go"`
12	`file_change` (✓ ✓)	✓ ✓ `/bin/zsh -lc "sed -n '830,920p' internal/resolver/dns/dns_resolver_test.go && \ sed -n '1210,1325p' internal/resolver/dns/dns_resolver_test.go && \ rg -n \"minDNSResRate\""`
13	✓ ✓ `/bin/zsh -lc 'gofmt -w internal/resolver/dns/dns_resolver.go && \ git diff -- internal/resolver/dns/dns_resolver.go && \ git status --short'`	`file_change` (✓ ✓)
14	✓ ✓ `/bin/zsh -lc "git diff --check && \ git status --short && \ nl -ba internal/resolver/dns/dns_resolver.go \| sed -n '194,250p'"`	`file_change` (✓ ✓)
15	—	`file_change` (✓ ✓)
16	—	✓ ✓ `/bin/zsh -lc 'gofmt -w internal/resolver/dns/dns_resolver.go balancer/base/balancer.go && \ git diff --check && \ git diff -- internal/resolver/dns/dns_resolver.go balancer/base/balancer.go'`
17	—	`file_change` (✓ ✓)
18	—	✓ ✓ `/bin/zsh -lc 'git diff --check && \ git diff --stat && \ git status --short && \ git diff -- internal/resolver/dns/dns_resolver.go balancer/base/balancer.go'`

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

default prompt · none MCP instructions · write semedit restriction

Task complexity across locality, context, ambiguity, composition, invariants, edit scale, and staging (1–5).

Setting	Value
Test case	`grpc__grpc-go-3361`
Target	`codex/gpt-6-luna/high`
Prompt variant	`default`
MCP server instructions	`none`
Semedit arm restriction	`write`
Run provenance	`coding_network=disabled,oracle_patch_installation=checked`
Fixture	testdata/bench/grpc__grpc-go-3361.txtar

LLM Prompt:

What version of gRPC are you using?
1.26
What version of Go are you using (go version)?
1.13.4
What operating system (Linux, Windows, …) and version?
Linux / Kubernetes / Docker - various.
What did you do?
When using grpc.WithBalancerName(roundrobin.Name), with a dns:/// Dial target, errors do not trigger re-resolve.
Detail: a client starts with dns:///my-service as a dial target and the round_robin load balancer. my-service DNS returns an IP 192.168.0.19. Due to an operational change, my-service is moved and DNS is updated to return 192.168.0.20. The service stops running on 192.168.0.19, refusing connections.
This is approximately the workflow that would occur when using Kubernetes and a headless service if a pod is replaced. This happens for a variety of reasons, including routine updates and rescheduling.
What did you expect to see?
(a) Upon receiving a connection refused, DNS should be re-queried for an updated endpoint list (b) Without receiving an error from an existing endpoint, the load balancer should be repopulated occasionally with the latest records in DNS as endpoint sets change
This was the behavior previous to 1.26. By rolling code back to grpc 1.25.1, we were able to restore this behavior.
https://github.com/grpc/grpc-go/pull/3165 is likely the culprit.
What did you see instead?
(a) Endpoint list is never re-queried and the client simply throws a persistent error:
rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 10.1.5.253:443: connect: connection refused"
(b) Since every subsequent request fails, the endpoint list does not seem to getting updated with new records in DNS at all.
Additional
I recognize the API around DNS and load balancing may have changed recently (and are evolving) but I’m not seeing any clear indication of how to retain the expected behavior.

Metric	Vanilla (Small)	⚠ Semedit Agentic (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	720.51s	720.88s	N/A	—	—	—
Process Start → First Event	0.76s	0.83s	—	—	—	—
First Event → First Tool Call	6.23s	6.48s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	1	1	N/A	—	—	—
Internal Tool Cycles	9	14	N/A	—	—	—
Initial Load / Discovery Turns	7	11	N/A	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	9	14	N/A	—	—	—
Output Tokens	5633	4894	N/A	—	—	—
Reasoning / Thinking Tokens	4072	3260	N/A	—	—	—
Total Input Tokens	660234	624103	N/A	—	—	—
Cached Input Tokens	570624	567040	N/A	—	—	—
Uncached Input Tokens	89610	57063	N/A	—	—	—
Cost	0.4880	0.3863	N/A	—	—	—
Cached vs Uncached Token Ratio	6.37:1	9.94:1	N/A	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	FAIL	FAIL	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	FAIL	—	—	—	—

Baseline vs Semedit Agentic in Small Context


/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g '*dns*' -g '*roundrobin*' | head -100"
/bin/zsh -lc "git status --short && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'README.md' | head -60 && \
sed -n '1,240p' resolver/dns/dns_resolver.go && \
sed -n '1,240p' balancer/roundrobin/roundrobin.go"
/bin/zsh -lc "sed -n '1,300p' internal/resolver/dns/dns_resolver.go && \
rg -n 'ResolveNow|ReportError|UpdateState|ExitIdle' resolver balancer -g '*.go' | head -120 && \
git log -6 --oneline --"
/bin/zsh -lc "sed -n '145,255p' resolver/resolver.go && \
rg -n 'ResolveNow\\(' clientconn.go balancer -g '*.go' && \
rg -n 'minDNSResRate|resolveNow|ResolveNow' internal/resolver/dns/dns_resolver_test.go resolver"
/bin/zsh -lc "rg -n 'ReportError|ResolveNow|NewSubConn|connectivity.TransientFailure|resolverWrapper|resolverClientConn' --glob '*.go' | head -150 && \
sed -n '380,465p' balancer/balancer.go"
/bin/zsh -lc "sed -n '1060,1160p' clientconn.go && \
sed -n '1,250p' balancer/base/balancer.go && \
sed -n '1,220p' resolver_conn_wrapper.go"
/bin/zsh -lc "git log --all --oneline -- internal/resolver/dns/dns_resolver.go | head -25 && \
git status --short --untracked-files=no && \
rg -n 'ResolveNow|periodic|30 seconds|DNS' doc"
/bin/zsh -lc "git log --all --oneline --grep='3165\\|resolver' | head -30 && \
git log --all --oneline --reverse -- internal/resolver/dns/dns_resolver.go | tail -20"
/bin/zsh -lc "git log --all --oneline --grep='DNS resolver' | head -30 && \
git show 1c598a11 --stat && \
git show 1c598a11"'^:internal/resolver/dns/dns_resolver.go | sed -n '"'160,225p'"
/bin/zsh -lc 'git blame -L 155,220 internal/resolver/dns/dns_resolver.go && \
git branch --show-current && \
git rev-parse --short HEAD'
/bin/zsh -lc "git show -s --format='%H%n%P%n%s' HEAD && \
sed -n '1,170p' internal/resolver/dns/dns_resolver_test.go && \
sed -n '1230,1375p' internal/resolver/dns/dns_resolver_test.go"

/bin/zsh -lc 'gofmt -w internal/resolver/dns/dns_resolver.go && \
git diff -- internal/resolver/dns/dns_resolver.go && \
git status --short --untracked-files=no'
/bin/zsh -lc "sed -n '1120,1195p' clientconn.go && \
git diff --check"

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'go.mod' -g 'dns' -g 'roundrobin' -g 'balancer' \| head -80 && \ git status --short"`	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'dns' -g 'roundrobin' \| head -100"`
2	✓ ✓ `/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' . .. \| head -40; rg -n \"ResolveNow\|ReportError\|UpdateState\|NewAddress\|ResolveNow\|round_robin\|roundrobin\|WithBalancerName\" resolver internal balancer naming \| head -150; sed -n '1,240p' resolver/dns/dns_resolver.go; sed -n '1,200p' balancer/roundrobin/roundrobin.go"`	✓ ✓ `/bin/zsh -lc "git status --short && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'README.md' \| head -60 && \ sed -n '1,240p' resolver/dns/dns_resolver.go && \ sed -n '1,240p' balancer/roundrobin/roundrobin.go"`
3	? ✗ `/bin/zsh -lc "sed -n '130,250p' internal/resolver/dns/dns_resolver.go; sed -n '1,260p' balancer/base/balancer.go; sed -n '130,210p' resolver/resolver.go; rg -n \"ReportError\|ResolveNow\\(\" balancer/base balancer.go clientconn.go"`	✓ ✓ `/bin/zsh -lc "sed -n '1,300p' internal/resolver/dns/dns_resolver.go && \ rg -n 'ResolveNow\|ReportError\|UpdateState\|ExitIdle' resolver balancer -g '*.go' \| head -120 && \ git log -6 --oneline --"`
4	✓ ✓ `/bin/zsh -lc "rg -n \"ResolveNow\|ResolverError\|ReportError\|TransientFailure\|connection error\" clientconn.go resolver balancer \| head -200; git log -8 --oneline --decorate; sed -n '360,430p' balancer/balancer.go; rg -n \"func \\(.*ResolveNow\|resolveNow\|resolveNowLocked\" clientconn.go"`	✓ ✓ `/bin/zsh -lc "sed -n '145,255p' resolver/resolver.go && \ rg -n 'ResolveNow\\(' clientconn.go balancer -g '*.go' && \ rg -n 'minDNSResRate\|resolveNow\|ResolveNow' internal/resolver/dns/dns_resolver_test.go resolver"`
5	✓ ✓ `/bin/zsh -lc "sed -n '780,875p' clientconn.go; sed -n '930,975p' clientconn.go; sed -n '1080,1160p' clientconn.go; git show --stat --oneline HEAD"`	✓ ✓ `/bin/zsh -lc "rg -n 'ReportError\|ResolveNow\|NewSubConn\|connectivity.TransientFailure\|resolverWrapper\|resolverClientConn' --glob '*.go' \| head -150 && \ sed -n '380,465p' balancer/balancer.go"`
6	✓ ✓ `/bin/zsh -lc "sed -n '1,145p' internal/resolver/dns/dns_resolver.go; sed -n '1,140p' naming/dns_resolver.go; rg -n \"minDNSResRate\|ResolveNow\|30 \\* time.Second\|time.After\" internal/resolver/dns naming/dns_resolver.go"`	✓ ✓ `/bin/zsh -lc "sed -n '1060,1160p' clientconn.go && \ sed -n '1,250p' balancer/base/balancer.go && \ sed -n '1,220p' resolver_conn_wrapper.go"`
7	✓ ✓ `/bin/zsh -lc "sed -n '1,280p' resolver_conn_wrapper.go; sed -n '260,330p' clientconn.go; sed -n '1,180p' pickfirst.go"`	? ✗ `/bin/zsh -lc "git log --all --oneline -- internal/resolver/dns/dns_resolver.go \| head -25 && \ git status --short --untracked-files=no && \ rg -n 'ResolveNow\|periodic\|30 seconds\|DNS' doc"`
8	`file_change` (✓ ✓)	✓ ✓ `/bin/zsh -lc "git log --all --oneline --grep='3165\\\|resolver' \| head -30 && \ git log --all --oneline --reverse -- internal/resolver/dns/dns_resolver.go \| tail -20"`
9	✓ ✓ `/bin/zsh -lc 'gofmt -w internal/resolver/dns/dns_resolver.go && \ git diff -- internal/resolver/dns/dns_resolver.go && \ git status --short'`	? ✗ `/bin/zsh -lc "git log --all --oneline --grep='DNS resolver' \| head -30 && \ git show 1c598a11 --stat && \ git show 1c598a11"'^:internal/resolver/dns/dns_resolver.go \| sed -n '"'160,225p'"`
10	—	✓ ✓ `/bin/zsh -lc 'git blame -L 155,220 internal/resolver/dns/dns_resolver.go && \ git branch --show-current && \ git rev-parse --short HEAD'`
11	—	✓ ✓ `/bin/zsh -lc "git show -s --format='%H%n%P%n%s' HEAD && \ sed -n '1,170p' internal/resolver/dns/dns_resolver_test.go && \ sed -n '1230,1375p' internal/resolver/dns/dns_resolver_test.go"`
12	—	`file_change` (✓ ✓)
13	—	✓ ✓ `/bin/zsh -lc 'gofmt -w internal/resolver/dns/dns_resolver.go && \ git diff -- internal/resolver/dns/dns_resolver.go && \ git status --short --untracked-files=no'`
14	—	✓ ✓ `/bin/zsh -lc "sed -n '1120,1195p' clientconn.go && \ git diff --check"`

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

default prompt · none MCP instructions · write semedit restriction

Task complexity across locality, context, ambiguity, composition, invariants, edit scale, and staging (1–5).

Setting	Value
Test case	`grpc__grpc-go-3361`
Target	`codex/gpt-6-luna/high`
Prompt variant	`default`
MCP server instructions	`none`
Semedit arm restriction	`write`
Run provenance	`coding_network=disabled,oracle_patch_installation=checked`
Fixture	testdata/bench/grpc__grpc-go-3361.txtar

LLM Prompt:

What version of gRPC are you using?
1.26
What version of Go are you using (go version)?
1.13.4
What operating system (Linux, Windows, …) and version?
Linux / Kubernetes / Docker - various.
What did you do?
When using grpc.WithBalancerName(roundrobin.Name), with a dns:/// Dial target, errors do not trigger re-resolve.
Detail: a client starts with dns:///my-service as a dial target and the round_robin load balancer. my-service DNS returns an IP 192.168.0.19. Due to an operational change, my-service is moved and DNS is updated to return 192.168.0.20. The service stops running on 192.168.0.19, refusing connections.
This is approximately the workflow that would occur when using Kubernetes and a headless service if a pod is replaced. This happens for a variety of reasons, including routine updates and rescheduling.
What did you expect to see?
(a) Upon receiving a connection refused, DNS should be re-queried for an updated endpoint list (b) Without receiving an error from an existing endpoint, the load balancer should be repopulated occasionally with the latest records in DNS as endpoint sets change
This was the behavior previous to 1.26. By rolling code back to grpc 1.25.1, we were able to restore this behavior.
https://github.com/grpc/grpc-go/pull/3165 is likely the culprit.
What did you see instead?
(a) Endpoint list is never re-queried and the client simply throws a persistent error:
rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 10.1.5.253:443: connect: connection refused"
(b) Since every subsequent request fails, the endpoint list does not seem to getting updated with new records in DNS at all.
Additional
I recognize the API around DNS and load balancing may have changed recently (and are evolving) but I’m not seeing any clear indication of how to retain the expected behavior.

Metric	Vanilla (Small)	⚠ Semedit Agentic (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	678.80s	712.90s	N/A	—	—	—
Process Start → First Event	0.68s	0.78s	—	—	—	—
First Event → First Tool Call	3.85s	6.80s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	1	1	N/A	—	—	—
Internal Tool Cycles	21	13	N/A	—	—	—
Initial Load / Discovery Turns	7	9	N/A	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	21	13	N/A	—	—	—
Output Tokens	5931	4266	N/A	—	—	—
Reasoning / Thinking Tokens	3707	2664	N/A	—	—	—
Total Input Tokens	673055	459355	N/A	—	—	—
Cached Input Tokens	605696	379392	N/A	—	—	—
Uncached Input Tokens	67359	79963	N/A	—	—	—
Cost	0.4403	0.3814	N/A	—	—	—
Cached vs Uncached Token Ratio	8.99:1	4.74:1	N/A	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	FAIL	FAIL	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	FAIL	—	—	—	—

Baseline vs Semedit Agentic in Small Context


/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g '*dns*' -g '*roundrobin*' | head -80"
/bin/zsh -lc "rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'README.md' | head -60 && \
git status --short && \
sed -n '1,240p' internal/resolver/dns/dns_resolver.go && \
sed -n '1,260p' balancer/roundrobin/roundrobin.go"
/bin/zsh -lc "sed -n '240,540p' internal/resolver/dns/dns_resolver.go && \
rg -n 'ResolveNow|ReportError|UpdateState|connection refused|TransientFailure' balancer clientconn.go resolver | head -180 && \
sed -n '1,180p' go.mod && \
sed -n '1,200p' README.md"
/bin/zsh -lc "sed -n '790,880p' clientconn.go && \
sed -n '930,980p' clientconn.go && \
sed -n '1070,1160p' clientconn.go && \
sed -n '1,250p' balancer/base/balancer.go && \
git log --oneline -10"
/bin/zsh -lc "rg -n 'type resolverWrapper|func \\(.*resolveNow|ResolveNow' resolver_wrapper.go clientconn.go && \
sed -n '1080,1125p' clientconn.go && \
rg -n 'minDNSResRate|watcher|periodic' internal/resolver/dns/*test.go"
/bin/zsh -lc "rg --files | rg 'resolver.*wrapper|resolver_wrapper|resolver.*go"'$'"' | head -60 && \
rg -n 'func \\(.*resolveNow|ResolveNow\\(' --glob '*.go' | head -100 && \
rg -n 'minDNSResRate|watcher' internal/resolver/dns/*test.go"
/bin/zsh -lc "sed -n '70,135p' resolver_conn_wrapper.go && \
sed -n '565,590p' dialoptions.go && \
git show --stat --oneline HEAD && \
git log --all --oneline -- internal/resolver/dns/dns_resolver.go | head"
/bin/zsh -lc "git log --all --oneline --grep='3165\\|re-resol\\|re-resolv\\|DNS' -i | head -60 && \
git show 648cf9b0"'^:internal/resolver/dns/dns_resolver.go 2>/dev/null | sed -n '"'170,230p'"
/bin/zsh -lc 'git show --stat e5e980f2 && \
git show e5e980f2 -- internal/resolver/dns/dns_resolver.go && \
git show 6f8ce092 -- internal/resolver/dns/dns_resolver.go'
/bin/zsh -lc 'git show --format=fuller --stat 6f8ce092 && \
git show --format= --find-renames 6f8ce092 -- internal/resolver/dns/dns_resolver.go'
/bin/zsh -lc 'git log --oneline --all --decorate --max-count=25 | head -25 && \
git show --stat 9dc72d1d && \
git show --format= --stat 6f8ce092'
/bin/zsh -lc 'gofmt -w internal/resolver/dns/dns_resolver.go && \
git diff -- internal/resolver/dns/dns_resolver.go && \
git status --short'
/bin/zsh -lc "sed -n '1210,1325p' internal/resolver/dns/dns_resolver_test.go && \
rg -n 'TestRateLimitedResolve|RateLimited' internal/resolver/dns/dns_resolver_test.go"
/bin/zsh -lc "git tag -l 'v1.25*' | head && \
git show v1.25.1:internal/resolver/dns/dns_resolver.go 2>/dev/null | sed -n '150,235p' | head -85 && \
rg -n \"resolver.*poll|DNS.*poll|polling\" internal/resolver/dns/dns_resolver_test.go"
/bin/zsh -lc "git ls-tree -r --name-only v1.25.1 | rg 'dns_resolver.go|resolver/dns' | head -20"
/bin/zsh -lc "git show v1.25.1:internal/resolver/dns/dns_resolver.go | sed -n '155,235p'"
/bin/zsh -lc "sed -n '1220,1320p' internal/resolver/dns/dns_resolver_test.go && \
sed -n '830,975p' internal/resolver/dns/dns_resolver_test.go && \
sed -n '1,180p' internal/resolver/dns/dns_resolver_test.go"


/bin/zsh -lc 'git diff -- internal/resolver/dns/dns_resolver.go && \
git status --short'


#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'dns' -g 'roundrobin' -g 'balancer' \| head -120"`	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'dns' -g 'roundrobin' \| head -80"`
2	✓ ✓ `/bin/zsh -lc "rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'README.md' \| head -60 && \ git status --short && \ git log -5 --oneline"`	✓ ✓ `/bin/zsh -lc "rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'README.md' \| head -60 && \ git status --short && \ sed -n '1,240p' internal/resolver/dns/dns_resolver.go && \ sed -n '1,260p' balancer/roundrobin/roundrobin.go"`
3	✓ ✓ `/bin/zsh -lc "sed -n '1,260p' resolver/dns/dns_resolver.go && \ sed -n '1,260p' balancer/roundrobin/roundrobin.go && \ rg -n \"ResolveNow\|ReportError\|ExitIdle\|connection refused\|dns\" resolver/dns internal/resolver/dns balancer/roundrobin clientconn.go"`	✓ ✓ `/bin/zsh -lc "sed -n '240,540p' internal/resolver/dns/dns_resolver.go && \ rg -n 'ResolveNow\|ReportError\|UpdateState\|connection refused\|TransientFailure' balancer clientconn.go resolver \| head -180 && \ sed -n '1,180p' go.mod && \ sed -n '1,200p' README.md"`
4	✓ ✓ `/bin/zsh -lc "sed -n '95,230p' internal/resolver/dns/dns_resolver.go && \ sed -n '920,990p' clientconn.go && \ sed -n '1060,1120p' clientconn.go && \ rg -n \"ResolveNow\|UpdateState\|ReportError\" balancer/base balancer_v1_wrapper.go balancer/balancer.go clientconn.go"`	✓ ✓ `/bin/zsh -lc "sed -n '790,880p' clientconn.go && \ sed -n '930,980p' clientconn.go && \ sed -n '1070,1160p' clientconn.go && \ sed -n '1,250p' balancer/base/balancer.go && \ git log --oneline -10"`
5	✓ ✓ `/bin/zsh -lc "git log --oneline --all -- internal/resolver/dns/dns_resolver.go \| head -20 && \ sed -n '1060,1180p' clientconn.go && \ sed -n '1,110p' internal/resolver/dns/dns_resolver.go && \ rg -n \"minDNSResRate\|dnsResRate\|ResolveNow\" internal/resolver/dns/dns_resolver_test.go"`	? ✗ `/bin/zsh -lc "rg -n 'type resolverWrapper\|func \\(.resolveNow\|ResolveNow' resolver_wrapper.go clientconn.go && \ sed -n '1080,1125p' clientconn.go && \ rg -n 'minDNSResRate\|watcher\|periodic' internal/resolver/dns/test.go"`
6	✓ ✓ `/bin/zsh -lc "git log --all --oneline --grep='#3165' \| head && \ git log --oneline --all -- clientconn.go \| head -15 && \ rg -n \"resolveNow\\(resolver.ResolveNowOptions\" --glob '*.go' \| head -30 && \ cat go.mod \| head -12"`	✓ ✓ `/bin/zsh -lc "rg --files \| rg 'resolver.wrapper\|resolver_wrapper\|resolver.go"'$'"' \| head -60 && \ rg -n 'func \\(.resolveNow\|ResolveNow\\(' --glob '.go' \| head -100 && \ rg -n 'minDNSResRate\|watcher' internal/resolver/dns/*test.go"`
7	? ✗ `/bin/zsh -lc 'git show --stat --oneline e5e980f2 && \ git show e5e980f2 --'`	✓ ✓ `/bin/zsh -lc "sed -n '70,135p' resolver_conn_wrapper.go && \ sed -n '565,590p' dialoptions.go && \ git show --stat --oneline HEAD && \ git log --all --oneline -- internal/resolver/dns/dns_resolver.go \| head"`
8	? ✗ `/bin/zsh -lc 'git show --format=fuller --no-patch e5e980f2 && \ git show --format= --stat e5e980f2'`	✓ ✓ `/bin/zsh -lc "git log --all --oneline --grep='3165\\\|re-resol\\\|re-resolv\\\|DNS' -i \| head -60 && \ git show 648cf9b0"'^:internal/resolver/dns/dns_resolver.go 2>/dev/null \| sed -n '"'170,230p'"`
9	✓ ✓ `/bin/zsh -lc "sed -n '100,175p' resolver_conn_wrapper.go && \ git show e5e980f2:internal/resolver/dns/dns_resolver.go \| sed -n '150,230p' && \ git show e5e980f2"'^:internal/resolver/dns/dns_resolver.go 2>&1 \| head -10'`	? ✗ `/bin/zsh -lc 'git show --stat e5e980f2 && \ git show e5e980f2 -- internal/resolver/dns/dns_resolver.go && \ git show 6f8ce092 -- internal/resolver/dns/dns_resolver.go'`
10	✓ ✓ `/bin/zsh -lc "git log --all --oneline -S 'time.After' -- internal/resolver/dns/dns_resolver.go \| head -20 && \ git log --all --oneline --ancestry-path e5e980f2..HEAD -- internal/resolver/dns/dns_resolver.go \| head -20 && \ git branch --contains HEAD \| head"`	? ✗ `/bin/zsh -lc 'git show --format=fuller --stat 6f8ce092 && \ git show --format= --find-renames 6f8ce092 -- internal/resolver/dns/dns_resolver.go'`
11	? ✗ `/bin/zsh -lc 'git log --oneline --all --decorate --max-count=25 \| head -25 && \ git show --stat 9dc72d1d && \ git show --format= --stat 6f8ce092'`	`file_change` (✓ ✓)
12	✓ ✓ `/bin/zsh -lc 'git blame -L 1095,1105 clientconn.go && \ git blame -L 185,225 internal/resolver/dns/dns_resolver.go && \ git log --oneline --all -- clientconn.go \| tail -20'`	✓ ✓ `/bin/zsh -lc 'gofmt -w internal/resolver/dns/dns_resolver.go && \ git diff -- internal/resolver/dns/dns_resolver.go && \ git status --short'`
13	✓ ✓ `/bin/zsh -lc 'git show --format= -- internal/resolver/dns/dns_resolver.go 6f8ce092 \| head'`	✓ ✓ `/bin/zsh -lc "sed -n '1210,1325p' internal/resolver/dns/dns_resolver_test.go && \ rg -n 'TestRateLimitedResolve\|RateLimited' internal/resolver/dns/dns_resolver_test.go"`
14	? ✗ `/bin/zsh -lc "git tag -l 'v1.25' \| head && \ git show v1.25.1:internal/resolver/dns/dns_resolver.go 2>/dev/null \| sed -n '150,235p' \| head -85 && \ rg -n \"resolver.poll\|DNS.*poll\|polling\" internal/resolver/dns/dns_resolver_test.go"`	—
15	✓ ✓ `/bin/zsh -lc "git ls-tree -r --name-only v1.25.1 \| rg 'dns_resolver.go\|resolver/dns' \| head -20"`	—
16	✓ ✓ `/bin/zsh -lc "git show v1.25.1:internal/resolver/dns/dns_resolver.go \| sed -n '155,235p'"`	—
17	✓ ✓ `/bin/zsh -lc "sed -n '1220,1320p' internal/resolver/dns/dns_resolver_test.go && \ sed -n '830,975p' internal/resolver/dns/dns_resolver_test.go && \ sed -n '1,180p' internal/resolver/dns/dns_resolver_test.go"`	—
18	`file_change` (✓ ✓)	—
19	`file_change` (✓ ✓)	—
20	✓ ✓ `/bin/zsh -lc 'git diff -- internal/resolver/dns/dns_resolver.go && \ git status --short'`	—
21	`file_change` (✓ ✓)	—

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

Target: `codex/gpt-6-luna/medium`

default prompt · none MCP instructions · write semedit restriction

Task complexity across locality, context, ambiguity, composition, invariants, edit scale, and staging (1–5).

Setting	Value
Test case	`grpc__grpc-go-3361`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`none`
Semedit arm restriction	`write`
Run provenance	`coding_network=disabled,oracle_patch_installation=checked`
Fixture	testdata/bench/grpc__grpc-go-3361.txtar

LLM Prompt:

What version of gRPC are you using?
1.26
What version of Go are you using (go version)?
1.13.4
What operating system (Linux, Windows, …) and version?
Linux / Kubernetes / Docker - various.
What did you do?
When using grpc.WithBalancerName(roundrobin.Name), with a dns:/// Dial target, errors do not trigger re-resolve.
Detail: a client starts with dns:///my-service as a dial target and the round_robin load balancer. my-service DNS returns an IP 192.168.0.19. Due to an operational change, my-service is moved and DNS is updated to return 192.168.0.20. The service stops running on 192.168.0.19, refusing connections.
This is approximately the workflow that would occur when using Kubernetes and a headless service if a pod is replaced. This happens for a variety of reasons, including routine updates and rescheduling.
What did you expect to see?
(a) Upon receiving a connection refused, DNS should be re-queried for an updated endpoint list (b) Without receiving an error from an existing endpoint, the load balancer should be repopulated occasionally with the latest records in DNS as endpoint sets change
This was the behavior previous to 1.26. By rolling code back to grpc 1.25.1, we were able to restore this behavior.
https://github.com/grpc/grpc-go/pull/3165 is likely the culprit.
What did you see instead?
(a) Endpoint list is never re-queried and the client simply throws a persistent error:
rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 10.1.5.253:443: connect: connection refused"
(b) Since every subsequent request fails, the endpoint list does not seem to getting updated with new records in DNS at all.
Additional
I recognize the API around DNS and load balancing may have changed recently (and are evolving) but I’m not seeing any clear indication of how to retain the expected behavior.

Metric	Vanilla (Small)	⚠ Semedit Agentic (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	673.14s	638.53s	N/A	—	—	—
Process Start → First Event	0.66s	0.15s	—	—	—	—
First Event → First Tool Call	5.63s	3.50s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	1	1	N/A	—	—	—
Internal Tool Cycles	11	11	N/A	—	—	—
Initial Load / Discovery Turns	9	9	N/A	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	11	11	N/A	—	—	—
Output Tokens	2469	1864	N/A	—	—	—
Reasoning / Thinking Tokens	1436	736	N/A	—	—	—
Total Input Tokens	318037	310343	N/A	—	—	—
Cached Input Tokens	263424	275200	N/A	—	—	—
Uncached Input Tokens	54613	35143	N/A	—	—	—
Cost	0.2512	0.1892	N/A	—	—	—
Cached vs Uncached Token Ratio	4.82:1	7.83:1	N/A	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	FAIL	FAIL	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	FAIL	—	—	—	—

Baseline vs Semedit Agentic in Small Context


/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g '*dns*' -g '*roundrobin*' -g '*resolver*' | head -100"
/bin/zsh -lc "rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'Makefile' | head -60 && \
git status --short && \
git log -8 --oneline --all"
/bin/zsh -lc "sed -n '1,260p' resolver/dns/dns_resolver.go && \
sed -n '1,260p' balancer/roundrobin/roundrobin.go && \
rg -n 'ResolveNow|ReportError|UpdateState|connection refused|re.?resolv' resolver/dns balancer/roundrobin"
/bin/zsh -lc "sed -n '1,300p' internal/resolver/dns/dns_resolver.go && \
rg -n 'ResolveNow|ResolveNow\\(' balancer internal resolver"
/bin/zsh -lc "git log --oneline -- internal/resolver/dns/dns_resolver.go | head -15 && \
sed -n '140,200p' balancer/balancer.go && \
sed -n '370,430p' balancer/balancer.go && \
rg -n 'ResolveNow\\(' clientconn.go balancer_conn_wrappers.go"
/bin/zsh -lc "sed -n '175,225p' balancer_conn_wrappers.go && \
git show --stat --oneline 3165 2>/dev/null || \
true && \
sed -n '820,930p' internal/resolver/dns/dns_resolver_test.go"
/bin/zsh -lc 'git show --stat 3165 && \
git show 3165 --'
/bin/zsh -lc "rg -n 'TransientFailure|ConnectionError|ResolveNow' balancer/base clientconn.go"
/bin/zsh -lc "sed -n '1080,1110p' clientconn.go && \
git show 3165 --stat --oneline"

/bin/zsh -lc 'git diff -- internal/resolver/dns/dns_resolver.go && \
git status --short'

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'dns' -g 'roundrobin' \| head -80"`	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'dns' -g 'roundrobin' -g 'resolver' \| head -100"`
2	✓ ✓ `/bin/zsh -lc "git status --short && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'README*' \| head -60 && \ sed -n '1,240p' resolver/dns/dns_resolver.go && \ sed -n '1,260p' balancer/roundrobin/roundrobin.go"`	✓ ✓ `/bin/zsh -lc "rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'Makefile' \| head -60 && \ git status --short && \ git log -8 --oneline --all"`
3	✓ ✓ `/bin/zsh -lc "sed -n '1,320p' internal/resolver/dns/dns_resolver.go && \ rg -n \"ResolveNow\|ResolveNowOptions\|ReportError\|UpdateState\|NotifyError\|connection refused\" resolver internal balancer"`	✓ ✓ `/bin/zsh -lc "sed -n '1,260p' resolver/dns/dns_resolver.go && \ sed -n '1,260p' balancer/roundrobin/roundrobin.go && \ rg -n 'ResolveNow\|ReportError\|UpdateState\|connection refused\|re.?resolv' resolver/dns balancer/roundrobin"`
4	? ✗ `/bin/zsh -lc "sed -n '320,500p' internal/resolver/dns/dns_resolver.go && \ sed -n '380,430p' balancer/balancer.go && \ rg -n 'ResolveNow\\(' balancer/base clientconn.go"`	✓ ✓ `/bin/zsh -lc "sed -n '1,300p' internal/resolver/dns/dns_resolver.go && \ rg -n 'ResolveNow\|ResolveNow\\(' balancer internal resolver"`
5	✓ ✓ `/bin/zsh -lc "git log -8 --oneline && \ rg -n 'min.DNS\|30 \\ time.Second\|dns.*re-resolv\|periodic' internal/resolver/dns/dns_resolver_test.go"`	✓ ✓ `/bin/zsh -lc "git log --oneline -- internal/resolver/dns/dns_resolver.go \| head -15 && \ sed -n '140,200p' balancer/balancer.go && \ sed -n '370,430p' balancer/balancer.go && \ rg -n 'ResolveNow\\(' clientconn.go balancer_conn_wrappers.go"`
6	✓ ✓ `/bin/zsh -lc "rg -n 'ResolveNow\|HandleSubConnStateChange\|TransientFailure' clientconn.go balancer.go"`	✓ ✓ `/bin/zsh -lc "sed -n '175,225p' balancer_conn_wrappers.go && \ git show --stat --oneline 3165 2>/dev/null \|\| \ true && \ sed -n '820,930p' internal/resolver/dns/dns_resolver_test.go"`
7	✓ ✓ `/bin/zsh -lc "sed -n '1080,1160p' clientconn.go && \ sed -n '920,975p' clientconn.go && \ rg -n 'HandleSubConnStateChange' balancer"`	✓ ✓ `/bin/zsh -lc 'git show --stat 3165 && \ git show 3165 --'`
8	✓ ✓ `/bin/zsh -lc ''`	✓ ✓ `/bin/zsh -lc "rg -n 'TransientFailure\|ConnectionError\|ResolveNow' balancer/base clientconn.go"`
9	✓ ✓ `/bin/zsh -lc "sed -n '1,150p' internal/resolver/dns/dns_resolver_test.go && \ sed -n '1320,1370p' internal/resolver/dns/dns_resolver_test.go"`	✓ ✓ `/bin/zsh -lc "sed -n '1080,1110p' clientconn.go && \ git show 3165 --stat --oneline"`
10	`file_change` (✓ ✓)	`file_change` (✓ ✓)
11	✓ ✓ `/bin/zsh -lc 'gofmt -w internal/resolver/dns/dns_resolver.go && \ git diff -- internal/resolver/dns/dns_resolver.go'`	✓ ✓ `/bin/zsh -lc 'git diff -- internal/resolver/dns/dns_resolver.go && \ git status --short'`

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

default prompt · none MCP instructions · write semedit restriction

Task complexity across locality, context, ambiguity, composition, invariants, edit scale, and staging (1–5).

Setting	Value
Test case	`grpc__grpc-go-3361`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`none`
Semedit arm restriction	`write`
Run provenance	`coding_network=disabled,oracle_patch_installation=checked`
Fixture	testdata/bench/grpc__grpc-go-3361.txtar

LLM Prompt:

What version of gRPC are you using?
1.26
What version of Go are you using (go version)?
1.13.4
What operating system (Linux, Windows, …) and version?
Linux / Kubernetes / Docker - various.
What did you do?
When using grpc.WithBalancerName(roundrobin.Name), with a dns:/// Dial target, errors do not trigger re-resolve.
Detail: a client starts with dns:///my-service as a dial target and the round_robin load balancer. my-service DNS returns an IP 192.168.0.19. Due to an operational change, my-service is moved and DNS is updated to return 192.168.0.20. The service stops running on 192.168.0.19, refusing connections.
This is approximately the workflow that would occur when using Kubernetes and a headless service if a pod is replaced. This happens for a variety of reasons, including routine updates and rescheduling.
What did you expect to see?
(a) Upon receiving a connection refused, DNS should be re-queried for an updated endpoint list (b) Without receiving an error from an existing endpoint, the load balancer should be repopulated occasionally with the latest records in DNS as endpoint sets change
This was the behavior previous to 1.26. By rolling code back to grpc 1.25.1, we were able to restore this behavior.
https://github.com/grpc/grpc-go/pull/3165 is likely the culprit.
What did you see instead?
(a) Endpoint list is never re-queried and the client simply throws a persistent error:
rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 10.1.5.253:443: connect: connection refused"
(b) Since every subsequent request fails, the endpoint list does not seem to getting updated with new records in DNS at all.
Additional
I recognize the API around DNS and load balancing may have changed recently (and are evolving) but I’m not seeing any clear indication of how to retain the expected behavior.

Metric	Vanilla (Small)	⚠ Semedit Agentic (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	638.59s	720.58s	N/A	—	—	—
Process Start → First Event	0.67s	0.79s	—	—	—	—
First Event → First Tool Call	4.36s	6.29s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	1	1	N/A	—	—	—
Internal Tool Cycles	8	15	N/A	—	—	—
Initial Load / Discovery Turns	8	7	N/A	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	8	15	N/A	—	—	—
Output Tokens	1820	4078	N/A	—	—	—
Reasoning / Thinking Tokens	1060	2068	N/A	—	—	—
Total Input Tokens	247489	416938	N/A	—	—	—
Cached Input Tokens	212736	388096	N/A	—	—	—
Uncached Input Tokens	34753	28842	N/A	—	—	—
Cost	0.1761	0.2460	N/A	—	—	—
Cached vs Uncached Token Ratio	6.12:1	13.46:1	N/A	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	FAIL	FAIL	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	FAIL	—	—	—	—

Baseline vs Semedit Agentic in Small Context

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc 'pwd && \ rg -n "ResolveNow\|WithBalancerName\|roundrobin\|DNS" resolver balancer'`	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'dns' -g 'roundrobin' \| head -100"`
2	✓ ✓ `/bin/zsh -lc "cat PROJECT_BRIEF.md 2>/dev/null; sed -n '1,260p' resolver/dns/dns_resolver.go; rg -n \"ResolveNow\\(\|resolver.*ResolveNow\|connectivity.TransientFailure\""`	✓ ✓ `/bin/zsh -lc 'git status --short && \ rg -n "3165\|ResolveNow\|connection refused\|ReportError\|Refresh\|reresolv\|re-resolv" resolver balancer internal'`
3	✓ ✓ `/bin/zsh -lc "sed -n '1,230p' balancer/roundrobin/roundrobin.go; sed -n '1,260p' balancer/base/balancer.go; sed -n '1,260p' internal/resolver/dns/dns_resolver.go; sed -n '840,920p' clientconn.go"`	✓ ✓ `/bin/zsh -lc "sed -n '1,250p' internal/resolver/dns/dns_resolver.go && \ sed -n '1,220p' balancer/roundrobin/roundrobin.go && \ rg -n \"ResolveNow\\(\" balancer"`
4	✓ ✓ `/bin/zsh -lc "sed -n '1070,1160p' clientconn.go; git log -5 --oneline; rg -n \"resolveNow\" --glob '*.go' \| head -50"`	✓ ✓ `/bin/zsh -lc "rg -n \"minDNSResRate\|ResolveNow\|periodic\|30 \\* time.Second\|watcher\" internal/resolver/dns/dns_resolver_test.go"`
5	✓ ✓ `/bin/zsh -lc "git show --stat --oneline 3165 2>/dev/null; git status --short; sed -n '120,170p' resolver_conn_wrapper.go; git log --oneline --all -- internal/resolver/dns/dns_resolver.go \| head"`	✓ ✓ `/bin/zsh -lc 'rg -n "TransientFailure\|Shutdown\|UpdateSubConnState\|Connect" balancer/base clientconn.go'`
6	✓ ✓ `/bin/zsh -lc 'rg -n "HandleSubConnStateChange\|UpdateSubConnState\|TransientFailure" balancer_v1_wrapper.go'`	✓ ✓ `/bin/zsh -lc "sed -n '165,225p' balancer/base/balancer.go && \ rg -n \"type ClientConn interface\|type ClientConn struct\" balancer"`
7	✓ ✓ `/bin/zsh -lc "sed -n '210,245p' balancer_v1_wrapper.go; sed -n '300,335p' balancer_v1_wrapper.go"`	✓ ✓ `/bin/zsh -lc "sed -n '130,170p' balancer/balancer.go && \ sed -n '120,155p' balancer/base/balancer.go"`
8	✓ ✓ `/bin/zsh -lc "sed -n '945,975p' clientconn.go; sed -n '1,125p' resolver_conn_wrapper.go; git show --stat 648cf9b0"`	`file_change` (✓ ✓)
9	—	`file_change` (✓ ✓)
10	—	`file_change` (✓ ✓)
11	—	`file_change` (✓ ✓)
12	—	✓ ✓ `/bin/zsh -lc "sed -n '20,48p' balancer/base/balancer.go && \ gofmt -w internal/resolver/dns/dns_resolver.go balancer/base/balancer.go && \ git diff -- internal/resolver/dns/dns_resolver.go balancer/base/balancer.go"`
13	—	✓ ✓ `/bin/zsh -lc 'cat resolver/dns/dns_resolver.go && \ rg -n "baseBalancer\|UpdateSubConnState" balancer/base/*test.go \| head'`
14	—	`file_change` (✓ ✓)
15	—	✓ ✓ `/bin/zsh -lc 'git diff --check && \ git status --short && \ git diff --stat && \ git diff -- balancer/base/balancer.go internal/resolver/dns/dns_resolver.go'`

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

default prompt · none MCP instructions · write semedit restriction

Task complexity across locality, context, ambiguity, composition, invariants, edit scale, and staging (1–5).

Setting	Value
Test case	`grpc__grpc-go-3361`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`none`
Semedit arm restriction	`write`
Run provenance	`coding_network=disabled,oracle_patch_installation=checked`
Fixture	testdata/bench/grpc__grpc-go-3361.txtar

LLM Prompt:

What version of gRPC are you using?
1.26
What version of Go are you using (go version)?
1.13.4
What operating system (Linux, Windows, …) and version?
Linux / Kubernetes / Docker - various.
What did you do?
When using grpc.WithBalancerName(roundrobin.Name), with a dns:/// Dial target, errors do not trigger re-resolve.
Detail: a client starts with dns:///my-service as a dial target and the round_robin load balancer. my-service DNS returns an IP 192.168.0.19. Due to an operational change, my-service is moved and DNS is updated to return 192.168.0.20. The service stops running on 192.168.0.19, refusing connections.
This is approximately the workflow that would occur when using Kubernetes and a headless service if a pod is replaced. This happens for a variety of reasons, including routine updates and rescheduling.
What did you expect to see?
(a) Upon receiving a connection refused, DNS should be re-queried for an updated endpoint list (b) Without receiving an error from an existing endpoint, the load balancer should be repopulated occasionally with the latest records in DNS as endpoint sets change
This was the behavior previous to 1.26. By rolling code back to grpc 1.25.1, we were able to restore this behavior.
https://github.com/grpc/grpc-go/pull/3165 is likely the culprit.
What did you see instead?
(a) Endpoint list is never re-queried and the client simply throws a persistent error:
rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 10.1.5.253:443: connect: connection refused"
(b) Since every subsequent request fails, the endpoint list does not seem to getting updated with new records in DNS at all.
Additional
I recognize the API around DNS and load balancing may have changed recently (and are evolving) but I’m not seeing any clear indication of how to retain the expected behavior.

Metric	Vanilla (Small)	⚠ Semedit Agentic (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	640.91s	634.26s	N/A	—	—	—
Process Start → First Event	0.14s	0.23s	—	—	—	—
First Event → First Tool Call	3.46s	4.08s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	1	1	N/A	—	—	—
Internal Tool Cycles	8	8	N/A	—	—	—
Initial Load / Discovery Turns	6	8	N/A	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	8	8	N/A	—	—	—
Output Tokens	2103	1364	N/A	—	—	—
Reasoning / Thinking Tokens	1121	599	N/A	—	—	—
Total Input Tokens	256357	226278	N/A	—	—	—
Cached Input Tokens	231168	199424	N/A	—	—	—
Uncached Input Tokens	25189	26854	N/A	—	—	—
Cost	0.1611	0.1415	N/A	—	—	—
Cached vs Uncached Token Ratio	9.18:1	7.43:1	N/A	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	FAIL	FAIL	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	FAIL	—	—	—	—

Baseline vs Semedit Agentic in Small Context


/bin/zsh -lc "pwd && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g '*dns*' -g '*roundrobin*' -g 'go.mod' | head -80 && \
git status --short --branch"
/bin/zsh -lc "cat go.mod | head -10; rg -n \"3165|ResolveNow|MinResolution|resolveNow|ResolveNow\" resolver balancer internal | head -160; sed -n '1,260p' resolver/dns/dns_resolver.go; sed -n '1,240p' balancer/roundrobin/roundrobin.go"
/bin/zsh -lc "sed -n '1,290p' internal/resolver/dns/dns_resolver.go; rg -n \"ResolveNow|ExitIdle|connection.*error|TransientFailure\" balancer/base clientconn.go"
/bin/zsh -lc "sed -n '380,430p' balancer/balancer.go; sed -n '180,250p' balancer/base/balancer.go; rg -n \"ResolveNow\\(\" balancer/base"
/bin/zsh -lc 'git log --oneline -8; git log --oneline --all -- internal/resolver/dns/dns_resolver.go | head'
/bin/zsh -lc "rg -n \"ResolveNow|resolveNow\" balancer/base clientconn.go; git show --stat --oneline 648cf9b0; git log --oneline --all --grep='3165' --grep='re-resolv' | head -20"
/bin/zsh -lc "git show --stat e5e980f2 && \
git show e5e980f2 -- internal/resolver/dns/dns_resolver.go | head -160; sed -n '1,130p' balancer/base/balancer.go"
/bin/zsh -lc "sed -n '1080,1110p' clientconn.go; git show e5e980f2"'^:internal/resolver/dns/dns_resolver.go 2>/dev/null | rg -n "ticker|ResolveNow|watcher"'

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'dns' -g 'roundrobin' \| head -100 && \ git status --short && \ rg -n \"WithBalancerName\|re-resolv\|ResolveNow\|round_robin\|roundrobin\""`	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'dns' -g 'roundrobin' -g 'go.mod' \| head -80 && \ git status --short --branch"`
2	✓ ✓ `/bin/zsh -lc "find .. -name AGENTS.md -o -name PROJECT_BRIEF.md \| head -30 && \ sed -n '1,240p' internal/resolver/dns/dns_resolver.go && \ sed -n '1,210p' resolver_conn_wrapper.go && \ sed -n '1,150p' balancer/roundrobin/roundrobin.go"`	✓ ✓ `/bin/zsh -lc "cat go.mod \| head -10; rg -n \"3165\|ResolveNow\|MinResolution\|resolveNow\|ResolveNow\" resolver balancer internal \| head -160; sed -n '1,260p' resolver/dns/dns_resolver.go; sed -n '1,240p' balancer/roundrobin/roundrobin.go"`
3	✓ ✓ `/bin/zsh -lc 'rg -n "ResolveNow\|TransientFailure\|Connect.Error\|UpdateSubConnState\|Report.Error" balancer/base balancer_conn_wrappers.go clientconn.go; rg -n "minDNSResRate\|watcher\|ResolveNow" internal/resolver/dns/dns_resolver_test.go; git log -5 --oneline --all -- . \| head -10'`	✓ ✓ `/bin/zsh -lc "sed -n '1,290p' internal/resolver/dns/dns_resolver.go; rg -n \"ResolveNow\|ExitIdle\|connection.*error\|TransientFailure\" balancer/base clientconn.go"`
4	✓ ✓ `/bin/zsh -lc "sed -n '160,235p' balancer/base/balancer.go; sed -n '1060,1160p' clientconn.go; sed -n '1,120p' resolver/dns/dns_resolver.go; git log --all --oneline --grep='DNS.re-resolve\\\|ResolveNow.round\\\|round_robin.*DNS' \| head"`	? ✗ `/bin/zsh -lc "sed -n '380,430p' balancer/balancer.go; sed -n '180,250p' balancer/base/balancer.go; rg -n \"ResolveNow\\(\" balancer/base"`
5	? ✗ `/bin/zsh -lc 'git show 3165 --stat --oneline 2>/dev/null \|\| \ true; git log --oneline --all -- internal/resolver/dns/dns_resolver.go \| head -20; rg -n "minDNSResRate\|DNS.*refresh\|periodic\|re-resolv\|re-resolve" doc internal/resolver/dns naming'`	✓ ✓ `/bin/zsh -lc 'git log --oneline -8; git log --oneline --all -- internal/resolver/dns/dns_resolver.go \| head'`
6	? ✗ `/bin/zsh -lc 'git show 9b970fdd -- internal/resolver/dns/dns_resolver.go internal/resolver/dns/dns_resolver_test.go'`	✓ ✓ `/bin/zsh -lc "rg -n \"ResolveNow\|resolveNow\" balancer/base clientconn.go; git show --stat --oneline 648cf9b0; git log --oneline --all --grep='3165' --grep='re-resolv' \| head -20"`
7	`file_change` (✓ ✓)	✓ ✓ `/bin/zsh -lc "git show --stat e5e980f2 && \ git show e5e980f2 -- internal/resolver/dns/dns_resolver.go \| head -160; sed -n '1,130p' balancer/base/balancer.go"`
8	✓ ✓ `/bin/zsh -lc 'git diff -- internal/resolver/dns/dns_resolver.go && \ git status --short'`	? ✗ `/bin/zsh -lc "sed -n '1080,1110p' clientconn.go; git show e5e980f2"'^:internal/resolver/dns/dns_resolver.go 2>/dev/null \| rg -n "ticker\|ResolveNow\|watcher"'`

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

Test case: `zeromicro__go-zero-2463`

Target: `codex/gpt-6-luna/high`

default prompt · none MCP instructions · write semedit restriction

Task complexity across locality, context, ambiguity, composition, invariants, edit scale, and staging (1–5).

Setting	Value
Test case	`zeromicro__go-zero-2463`
Target	`codex/gpt-6-luna/high`
Prompt variant	`default`
MCP server instructions	`none`
Semedit arm restriction	`write`
Run provenance	`coding_network=disabled,oracle_patch_installation=checked`
Fixture	testdata/bench/zeromicro__go-zero-2463.txtar

LLM Prompt:

logx能否支持一下全局添加field功能，或默认加一个serviceName，收集日志想根据服务分索引很不方便

Metric	Vanilla (Small)	⚠ Semedit Agentic (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	94.39s	142.52s	N/A	—	—	—
Process Start → First Event	0.39s	0.51s	—	—	—	—
First Event → First Tool Call	3.75s	5.72s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	1	1	N/A	—	—	—
Internal Tool Cycles	22	15	N/A	—	—	—
Initial Load / Discovery Turns	5	3	N/A	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	22	15	N/A	—	—	—
Output Tokens	7080	5927	N/A	—	—	—
Reasoning / Thinking Tokens	3500	3460	N/A	—	—	—
Total Input Tokens	790428	512587	N/A	—	—	—
Cached Input Tokens	716800	473088	N/A	—	—	—
Uncached Input Tokens	73628	39499	N/A	—	—	—
Cost	0.4955	0.3344	N/A	—	—	—
Cached vs Uncached Token Ratio	9.74:1	11.98:1	N/A	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	FAIL	FAIL	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	FAIL	—	—	—	—

Baseline vs Semedit Agentic in Small Context

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'logx' -g '*.go' \| head -100"`	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'logx' -g 'go.mod' \| head -100"`
2	? ✗ `/bin/zsh -lc "rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'logx/**' \| head -100 && \ rg -n \"type (Config\|Logger\|logEntry)\|WithFields\|Fields\|ServiceName\|serviceName\" logx"`	? ✗ `/bin/zsh -lc "rg --files \| rg '("'^\|/)logx(/\|$)\|logx'"' \| head -160; git status --short; rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md'"`
3	✓ ✓ `/bin/zsh -lc "ls -la \| head -80; rg --files \| rg '("'^\|/)logx/'"' \| head -100; find .. -name AGENTS.md -o -name PROJECT_BRIEF.md"`	✓ ✓ `/bin/zsh -lc "sed -n '1,240p' core/logx/config.go && \ sed -n '1,240p' core/logx/fields.go && \ sed -n '1,220p' core/logx/logs.go && \ sed -n '1,220p' core/logx/readme-cn.md"`
4	✓ ✓ `/bin/zsh -lc "sed -n '1,240p' core/logx/config.go && \ sed -n '1,240p' core/logx/fields.go && \ sed -n '1,280p' core/logx/logger.go && \ sed -n '1,220p' core/logx/readme-cn.md"`	✓ ✓ `/bin/zsh -lc "sed -n '220,520p' core/logx/logs.go && \ sed -n '1,300p' core/logx/logger.go && \ sed -n '1,300p' core/logx/richlogger.go && \ sed -n '1,260p' core/logx/logwriter.go && \ sed -n '1,220p' core/logx/vars.go"`
5	✓ ✓ `/bin/zsh -lc "sed -n '1,300p' core/logx/logs.go && \ sed -n '1,280p' core/logx/richlogger.go && \ sed -n '1,220p' core/logx/vars.go && \ rg -n \"ServiceName\" core/logx"`	✓ ✓ `/bin/zsh -lc "rg 'ServiceName\|LogConf\\{' core/logx -n; rg 'type .LogConf\|LogConf' -g '.go' \| head -80; sed -n '1,260p' core/logx/richlogger_test.go; sed -n '1,220p' core/logx/fields_test.go; sed -n '1,220p' core/logx/writer.go"`
6	✓ ✓ `/bin/zsh -lc "sed -n '280,500p' core/logx/logs.go && \ sed -n '1,240p' core/logx/logwriter.go && \ sed -n '1,260p' core/logx/writer.go"`	✓ ✓ `/bin/zsh -lc "rg -n 'func output\|func build\|fields' core/logx/writer.go core/logx/*.go \| tail -60 && \ cat go.mod \| head -20 && \ sed -n '1,90p' core/logx/readme.md"`
7	✓ ✓ `/bin/zsh -lc "rg -n \"func output\|addCaller\\(\" core/logx && \ sed -n '260,440p' core/logx/writer.go && \ rg -n \"LogConf\\{\""`	✓ ✓ `/bin/zsh -lc "sed -n '250,315p' core/logx/writer.go; rg -n 'func TestSetUp\|func TestWrite\|func Test.*Info\|SetUp\\(' core/logx/logs_test.go \| head -60;"`
8	✓ ✓ `/bin/zsh -lc "head -12 go.mod; sed -n '1,120p' core/logx/logs_test.go; sed -n '760,830p' core/logx/logs_test.go"`	`file_change` (✓ ✓)
9	✓ ✓ `/bin/zsh -lc "sed -n '14,40p' core/logx/logs.go; sed -n '450,465p' core/logx/logs.go"`	`file_change` (✓ ✓)
10	`file_change` (✓ ✓)	`file_change` (✓ ✓)
11	`file_change` (✓ ✓)	`file_change` (✓ ✓)
12	`file_change` (✓ ✓)	`file_change` (✓ ✓)
13	✓ ✓ `/bin/zsh -lc "sed -n '20,55p' core/logx/readme.md"`	✓ ✓ `/bin/zsh -lc 'gofmt -w core/logx/config.go core/logx/logs.go core/logx/richlogger.go && \ git diff --check && \ git diff -- core/logx/config.go core/logx/logs.go core/logx/richlogger.go core/logx/readme.md core/logx/readme-cn.md'`
14	`file_change` (✓ ✓)	`file_change` (✓ ✓)
15	`file_change` (✓ ✓)	✓ ✓ `/bin/zsh -lc 'gofmt -w core/logx/writer.go && \ git diff --check && \ git status --short && \ git diff --stat'`
16	`file_change` (✓ ✓)	—
17	`file_change` (✓ ✓)	—
18	`file_change` (✓ ✓)	—
19	✓ ✓ `/bin/zsh -lc 'gofmt -w core/logx/logs.go core/logx/richlogger.go core/logx/writer.go && \ git diff --check && \ git diff -- core/logx/logs.go core/logx/richlogger.go core/logx/writer.go core/logx/readme-cn.md core/logx/readme.md'`	—
20	`file_change` (✓ ✓)	—
21	`file_change` (✓ ✓)	—
22	✓ ✓ `/bin/zsh -lc 'git status --short && \ git diff --check && \ rg -n "AddGlobalField\|getGlobalFields" core/logx'`	—

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

default prompt · none MCP instructions · write semedit restriction

Task complexity across locality, context, ambiguity, composition, invariants, edit scale, and staging (1–5).

Setting	Value
Test case	`zeromicro__go-zero-2463`
Target	`codex/gpt-6-luna/high`
Prompt variant	`default`
MCP server instructions	`none`
Semedit arm restriction	`write`
Run provenance	`coding_network=disabled,oracle_patch_installation=checked`
Fixture	testdata/bench/zeromicro__go-zero-2463.txtar

LLM Prompt:

logx能否支持一下全局添加field功能，或默认加一个serviceName，收集日志想根据服务分索引很不方便

Metric	Vanilla (Small)	⚠ Semedit Agentic (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	72.57s	126.71s	N/A	—	—	—
Process Start → First Event	0.65s	0.64s	—	—	—	—
First Event → First Tool Call	4.14s	5.69s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	1	1	N/A	—	—	—
Internal Tool Cycles	13	14	N/A	—	—	—
Initial Load / Discovery Turns	3	5	N/A	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	13	14	N/A	—	—	—
Output Tokens	5699	5524	N/A	—	—	—
Reasoning / Thinking Tokens	3540	3204	N/A	—	—	—
Total Input Tokens	427362	410651	N/A	—	—	—
Cached Input Tokens	380416	374528	N/A	—	—	—
Uncached Input Tokens	46946	36123	N/A	—	—	—
Cost	0.3280	0.2930	N/A	—	—	—
Cached vs Uncached Token Ratio	8.10:1	10.37:1	N/A	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	FAIL	FAIL	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	FAIL	—	—	—	—

Baseline vs Semedit Agentic in Small Context


/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'README*' | head -60 && \
rg -n \"WithFields|AddField|serviceName|logx\" logx"
/bin/zsh -lc "ls -a && \
rg --files | rg '("'^|/)logx|AGENTS'"\\.md|PROJECT_BRIEF\\.md' | head -100 && \
git status --short"
/bin/zsh -lc "sed -n '1,240p' core/logx/config.go && \
sed -n '1,240p' core/logx/fields.go && \
sed -n '1,260p' core/logx/logs.go"
/bin/zsh -lc "sed -n '1,220p' core/logx/readme.md && \
sed -n '1,220p' core/logx/readme-cn.md"
/bin/zsh -lc "rg -n \"NewLogger|WithFields|Fields|Config\\{\" core/logx"
/bin/zsh -lc "sed -n '1,260p' core/logx/richlogger.go && \
sed -n '1,330p' core/logx/writer.go && \
rg -n \"ServiceName\""
/bin/zsh -lc "sed -n '260,470p' core/logx/logs.go && \
sed -n '1,120p' core/logx/vars.go && \
sed -n '1,130p' core/logx/fields_test.go && \
sed -n '210,280p' core/logx/richlogger_test.go"
/bin/zsh -lc "sed -n '1,100p' core/service/serviceconf.go && \
rg 'SetUp\\(sc.Log|SetUp\\(.*Log' -n core/service rest zrpc"

/bin/zsh -lc 'gofmt -w core/logx/logs.go core/logx/vars.go core/logx/writer.go && \
git diff --check && \
git diff -- core/logx/logs.go core/logx/vars.go core/logx/writer.go core/logx/readme-cn.md core/logx/readme.md && \
git status --short'
/bin/zsh -lc 'go build ./core/logx'
/bin/zsh -lc 'gofmt -w core/logx/fields.go core/logx/logs.go core/logx/richlogger.go && \
git diff --check && \
git diff -- core/logx/fields.go core/logx/logs.go core/logx/richlogger.go core/logx/readme.md core/logx/readme-cn.md'
/bin/zsh -lc 'git diff --check && \
git status --short && \
git diff --stat'
/bin/zsh -lc 'git diff --check && \
git status --short'

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'logx' -g '*.go' \| head -120"`	? ✗ `/bin/zsh -lc "pwd && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'README*' \| head -60 && \ rg -n \"WithFields\|AddField\|serviceName\|logx\" logx"`
2	✓ ✓ `/bin/zsh -lc "rg --files \| rg '("'^\|/)(AGENTS'"\\.md\|PROJECT_BRIEF\\.md)"'$\|logx/'"'"`	✓ ✓ `/bin/zsh -lc "ls -a && \ rg --files \| rg '("'^\|/)logx\|AGENTS'"\\.md\|PROJECT_BRIEF\\.md' \| head -100 && \ git status --short"`
3	✓ ✓ `/bin/zsh -lc "git status --short && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g '"'!vendor'"' .. \| head -60 && \ sed -n '1,240p' core/logx/config.go && \ sed -n '1,240p' core/logx/fields.go && \ sed -n '1,200p' core/logx/readme-cn.md"`	✓ ✓ `/bin/zsh -lc "sed -n '1,240p' core/logx/config.go && \ sed -n '1,240p' core/logx/fields.go && \ sed -n '1,260p' core/logx/logs.go"`
4	✓ ✓ `/bin/zsh -lc "sed -n '1,280p' core/logx/logs.go && \ sed -n '1,220p' core/logx/vars.go && \ sed -n '1,260p' core/logx/logwriter.go && \ sed -n '1,220p' core/logx/fields_test.go && \ rg 'ServiceName\|serviceName\|Set.Field\|Add.Field' core/logx"`	✓ ✓ `/bin/zsh -lc "sed -n '1,220p' core/logx/readme.md && \ sed -n '1,220p' core/logx/readme-cn.md"`
5	✓ ✓ `/bin/zsh -lc "sed -n '220,560p' core/logx/logs.go && \ sed -n '1,300p' core/logx/logger.go && \ sed -n '1,260p' core/logx/richlogger.go && \ sed -n '1,220p' core/logx/writer.go && \ sed -n '1,200p' core/logx/readme.md"`	✓ ✓ `/bin/zsh -lc "rg -n \"NewLogger\|WithFields\|Fields\|Config\\{\" core/logx"`
6	✓ ✓ `/bin/zsh -lc "rg 'func output\|func (write\|marshal)' core/logx -n && \ rg 'LogConf\\{' --glob '*.go' \| head -80 && \ rg 'ServiceName =' rest zrpc -n \| head -60"`	✓ ✓ `/bin/zsh -lc "sed -n '1,260p' core/logx/richlogger.go && \ sed -n '1,330p' core/logx/writer.go && \ rg -n \"ServiceName\""`
7	? ✗ `/bin/zsh -lc "sed -n '250,330p' core/logx/writer.go && \ rg 'ServiceName' rest zrpc core/service -n \| head -80 && \ rg 'logx.SetUp\|Log.SetUp\|SetUp\\(.*Log' -n rest zrpc"`	✓ ✓ `/bin/zsh -lc "sed -n '260,470p' core/logx/logs.go && \ sed -n '1,120p' core/logx/vars.go && \ sed -n '1,130p' core/logx/fields_test.go && \ sed -n '210,280p' core/logx/richlogger_test.go"`
8	✓ ✓ `/bin/zsh -lc "sed -n '1,100p' core/service/serviceconf.go && \ rg 'SetUp\\(sc.Log\|SetUp\\(.*Log' -n core/service rest zrpc"`	`file_change` (✓ ✓)
9	`file_change` (✓ ✓)	`file_change` (✓ ✓)
10	✓ ✓ `/bin/zsh -lc 'gofmt -w core/logx/logs.go core/logx/vars.go core/logx/writer.go && \ git diff --check && \ git diff -- core/logx/logs.go core/logx/vars.go core/logx/writer.go core/logx/readme-cn.md core/logx/readme.md && \ git status --short'`	`file_change` (✓ ✓)
11	? ✗ `/bin/zsh -lc 'go build ./core/logx'`	`file_change` (✓ ✓)
12	`file_change` (✓ ✓)	✓ ✓ `/bin/zsh -lc 'gofmt -w core/logx/fields.go core/logx/logs.go core/logx/richlogger.go && \ git diff --check && \ git diff -- core/logx/fields.go core/logx/logs.go core/logx/richlogger.go core/logx/readme.md core/logx/readme-cn.md'`
13	✓ ✓ `/bin/zsh -lc 'git diff --check && \ git status --short && \ git diff --stat'`	`file_change` (✓ ✓)
14	—	✓ ✓ `/bin/zsh -lc 'git diff --check && \ git status --short'`

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

default prompt · none MCP instructions · write semedit restriction

Task complexity across locality, context, ambiguity, composition, invariants, edit scale, and staging (1–5).

Setting	Value
Test case	`zeromicro__go-zero-2463`
Target	`codex/gpt-6-luna/high`
Prompt variant	`default`
MCP server instructions	`none`
Semedit arm restriction	`write`
Run provenance	`coding_network=disabled,oracle_patch_installation=checked`
Fixture	testdata/bench/zeromicro__go-zero-2463.txtar

LLM Prompt:

logx能否支持一下全局添加field功能，或默认加一个serviceName，收集日志想根据服务分索引很不方便

Metric	Vanilla (Small)	⚠ Semedit Agentic (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	184.42s	102.88s	N/A	—	—	—
Process Start → First Event	0.23s	0.52s	—	—	—	—
First Event → First Tool Call	7.24s	5.41s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	1	1	N/A	—	—	—
Internal Tool Cycles	17	12	N/A	—	—	—
Initial Load / Discovery Turns	6	4	N/A	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	17	12	N/A	—	—	—
Output Tokens	7887	4005	N/A	—	—	—
Reasoning / Thinking Tokens	4032	2305	N/A	—	—	—
Total Input Tokens	649536	384575	N/A	—	—	—
Cached Input Tokens	588544	330496	N/A	—	—	—
Uncached Input Tokens	60992	54079	N/A	—	—	—
Cost	0.4486	0.2967	N/A	—	—	—
Cached vs Uncached Token Ratio	9.65:1	6.11:1	N/A	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	FAIL	FAIL	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	FAIL	—	—	—	—

Baseline vs Semedit Agentic in Small Context


/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g '*logx*' -g '*.go' | head -200"
/bin/zsh -lc "rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g '*.go' logx"
/bin/zsh -lc "rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' | head -100 && \
rg --files | rg '("'^|/)logx/'"' | head -100"
/bin/zsh -lc "sed -n '1,240p' core/logx/config.go && \
sed -n '1,280p' core/logx/logs.go && \
sed -n '1,240p' core/logx/fields.go && \
sed -n '1,220p' core/logx/vars.go && \
sed -n '1,200p' core/logx/readme-cn.md"
/bin/zsh -lc 'rg -n "ServiceName|serviceName|logOptions|logStackCooldown|writeInfo|writeDebug|ContextWithFields" core/logx'
/bin/zsh -lc "sed -n '260,480p' core/logx/logs.go && \
sed -n '1,300p' core/logx/richlogger.go && \
sed -n '1,200p' core/logx/writer.go && \
rg -n \"SetUp\\(|LogConf\" core/logx/logs_test.go"
/bin/zsh -lc "sed -n '200,400p' core/logx/writer.go && \
sed -n '1,180p' core/logx/logwriter.go && \
sed -n '1,120p' core/logx/logs_test.go && \
sed -n '620,730p' core/logx/logs_test.go && \
sed -n '1,160p' core/logx/readme.md"
/bin/zsh -lc "rg -n \"ServiceName.*SetUp|LogConf\\{|logx.SetUp|SetUp\\(.*Log\" rest zrpc core | head -100 && \
rg -n \"func .*buildFields|addCaller\\(\" core/logx/logs_test.go"
/bin/zsh -lc "sed -n '1,100p' core/service/serviceconf.go && \
rg -n \"ServiceName\" rest zrpc core/service"
/bin/zsh -lc "sed -n '218,255p' core/logx/logs.go; sed -n '430,450p' core/logx/logs.go; sed -n '115,155p' core/logx/richlogger.go; sed -n '35,58p' core/logx/vars.go"
/bin/zsh -lc "sed -n '560,630p' core/logx/logs_test.go && \
sed -n '770,815p' core/logx/logs_test.go && \
git diff -- core/logx/logs.go core/logx/vars.go core/logx/writer.go core/logx/readme-cn.md core/logx/readme.md"
/bin/zsh -lc 'gofmt -w core/logx/logs.go core/logx/vars.go core/logx/writer.go && \
git diff --check && \
git status --short'

/bin/zsh -lc 'gofmt -w core/logx/fields.go core/logx/logs.go core/logx/richlogger.go core/logx/vars.go && \
git diff -- core/logx'
/bin/zsh -lc 'git diff --check && \
git status --short'

/bin/zsh -lc 'gofmt -w core/logx/writer.go core/logx/logs.go && \
git diff --check && \
git diff --stat'

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'logx' \| head -100"`	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'logx' -g '*.go' \| head -200"`
2	✓ ✓ `/bin/zsh -lc 'pwd; rg --files \| head -80'`	? ✗ `/bin/zsh -lc "rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g '*.go' logx"`
3	? ✗ `/bin/zsh -lc "rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'logx' -g 'go.mod' \| head -100; rg -n 'package logx\|logx\\.' logx"`	✓ ✓ `/bin/zsh -lc "rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' \| head -100 && \ rg --files \| rg '("'^\|/)logx/'"' \| head -100"`
4	✓ ✓ `/bin/zsh -lc "rg --files \| rg '("'^\|/)logx(/\|'"\\.)\|logx'; rg -n 'package logx\|type LogConf\|With.Field\|Add.Field'"`	✓ ✓ `/bin/zsh -lc "sed -n '1,240p' core/logx/config.go && \ sed -n '1,280p' core/logx/logs.go && \ sed -n '1,240p' core/logx/fields.go && \ sed -n '1,220p' core/logx/vars.go && \ sed -n '1,200p' core/logx/readme-cn.md"`
5	✓ ✓ `/bin/zsh -lc "rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md'; cat core/logx/config.go core/logx/fields.go core/logx/vars.go"`	✓ ✓ `/bin/zsh -lc 'rg -n "ServiceName\|serviceName\|logOptions\|logStackCooldown\|writeInfo\|writeDebug\|ContextWithFields" core/logx'`
6	✓ ✓ `/bin/zsh -lc 'cat core/logx/logger.go core/logx/logs.go core/logx/richlogger.go core/logx/lesslogger.go'`	✓ ✓ `/bin/zsh -lc "sed -n '260,480p' core/logx/logs.go && \ sed -n '1,300p' core/logx/richlogger.go && \ sed -n '1,200p' core/logx/writer.go && \ rg -n \"SetUp\\(\|LogConf\" core/logx/logs_test.go"`
7	✓ ✓ `/bin/zsh -lc "cat core/logx/writer.go core/logx/logwriter.go core/logx/richlogger.go core/logx/readme-cn.md; rg -n 'ServiceName\|SetUp\\(\|Reset\\(' core/logx"`	✓ ✓ `/bin/zsh -lc "sed -n '200,400p' core/logx/writer.go && \ sed -n '1,180p' core/logx/logwriter.go && \ sed -n '1,120p' core/logx/logs_test.go && \ sed -n '620,730p' core/logx/logs_test.go && \ sed -n '1,160p' core/logx/readme.md"`
8	✓ ✓ `/bin/zsh -lc "head -30 go.mod; rg -n 'writeSevere\|writeStack\|Alert\\(' core/logx"`	? ✗ `/bin/zsh -lc "rg -n \"ServiceName.SetUp\|LogConf\\{\|logx.SetUp\|SetUp\\(.Log\" rest zrpc core \| head -100 && \ rg -n \"func .*buildFields\|addCaller\\(\" core/logx/logs_test.go"`
9	`file_change` (✓ ✓)	✓ ✓ `/bin/zsh -lc "sed -n '1,100p' core/service/serviceconf.go && \ rg -n \"ServiceName\" rest zrpc core/service"`
10	✓ ✓ `/bin/zsh -lc "sed -n '218,255p' core/logx/logs.go; sed -n '430,450p' core/logx/logs.go; sed -n '115,155p' core/logx/richlogger.go; sed -n '35,58p' core/logx/vars.go"`	`file_change` (✓ ✓)
11	`file_change` (✓ ✓)	✓ ✓ `/bin/zsh -lc "sed -n '560,630p' core/logx/logs_test.go && \ sed -n '770,815p' core/logx/logs_test.go && \ git diff -- core/logx/logs.go core/logx/vars.go core/logx/writer.go core/logx/readme-cn.md core/logx/readme.md"`
12	✓ ✓ `/bin/zsh -lc "rg -n 'WithDuration\|WithContext' core/logx/readme.md core/logx/readme-cn.md"`	✓ ✓ `/bin/zsh -lc 'gofmt -w core/logx/logs.go core/logx/vars.go core/logx/writer.go && \ git diff --check && \ git status --short'`
13	`file_change` (✓ ✓)	—
14	✓ ✓ `/bin/zsh -lc 'gofmt -w core/logx/fields.go core/logx/logs.go core/logx/richlogger.go core/logx/vars.go && \ git diff -- core/logx'`	—
15	✓ ✓ `/bin/zsh -lc 'git diff --check && \ git status --short'`	—
16	`file_change` (✓ ✓)	—
17	✓ ✓ `/bin/zsh -lc 'gofmt -w core/logx/writer.go core/logx/logs.go && \ git diff --check && \ git diff --stat'`	—

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

Target: `codex/gpt-6-luna/medium`

default prompt · none MCP instructions · write semedit restriction

Task complexity across locality, context, ambiguity, composition, invariants, edit scale, and staging (1–5).

Setting	Value
Test case	`zeromicro__go-zero-2463`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`none`
Semedit arm restriction	`write`
Run provenance	`coding_network=disabled,oracle_patch_installation=checked`
Fixture	testdata/bench/zeromicro__go-zero-2463.txtar

LLM Prompt:

logx能否支持一下全局添加field功能，或默认加一个serviceName，收集日志想根据服务分索引很不方便

Metric	Vanilla (Small)	⚠ Semedit Agentic (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	53.10s	54.02s	N/A	—	—	—
Process Start → First Event	1.15s	0.20s	—	—	—	—
First Event → First Tool Call	5.50s	3.76s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	1	1	N/A	—	—	—
Internal Tool Cycles	13	15	N/A	—	—	—
Initial Load / Discovery Turns	4	5	N/A	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	13	15	N/A	—	—	—
Output Tokens	2496	3190	N/A	—	—	—
Reasoning / Thinking Tokens	695	896	N/A	—	—	—
Total Input Tokens	403840	415648	N/A	—	—	—
Cached Input Tokens	372224	382976	N/A	—	—	—
Uncached Input Tokens	31616	32672	N/A	—	—	—
Cost	0.2120	0.2285	N/A	—	—	—
Cached vs Uncached Token Ratio	11.77:1	11.72:1	N/A	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	FAIL	FAIL	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	FAIL	—	—	—	—

Baseline vs Semedit Agentic in Small Context

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'logx' -g '*.go' \| head -200"`	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'logx' \| head -80"`
2	✓ ✓ `/bin/zsh -lc "rg --files \| rg '("'^\|/)AGENTS'"\\.md"'$\|PROJECT_BRIEF'"\\.md"'$\|logx/'"'"`	✓ ✓ `/bin/zsh -lc 'rg --files \| head -80; rg -n "package logx\|type Logger\|WithFields\|AddField"'`
3	✓ ✓ `/bin/zsh -lc "cat core/logx/config.go && \ cat core/logx/fields.go && \ sed -n '1,240p' core/logx/logger.go && \ cat core/logx/vars.go && \ sed -n '1,180p' core/logx/readme-cn.md"`	✓ ✓ `/bin/zsh -lc "sed -n '1,240p' core/logx/config.go; sed -n '1,220p' core/logx/fields.go; sed -n '1,260p' core/logx/richlogger.go; sed -n '1,220p' core/logx/vars.go; sed -n '1,130p' core/logx/readme-cn.md"`
4	✓ ✓ `/bin/zsh -lc "sed -n '1,280p' core/logx/logs.go && \ sed -n '1,260p' core/logx/richlogger.go && \ rg 'ServiceName\|LogField\|fieldsContextKey\|WithFields' core/logx"`	✓ ✓ `/bin/zsh -lc 'rg -n "ServiceName\|Set.*Log\|Reset\|buildFields\|func (Must\|With\|Set)" core/logx'`
5	✓ ✓ `/bin/zsh -lc "sed -n '280,540p' core/logx/logs.go; sed -n '1,180p' core/logx/writer.go; cat go.mod \| head"`	? ✗ `/bin/zsh -lc "sed -n '180,330p' core/logx/logs.go; sed -n '1,180p' core/logx/logs.go; sed -n '1,120p' core/logx/fields_test.go; sed -n '220,275p' core/logx/richlogger_test.go; sed -n '1,90p' core/logx/logfield.go"`
6	✓ ✓ `/bin/zsh -lc "rg 'func output\|func buildFields' -n core/logx/writer.go && \ git status --short && \ "`	✓ ✓ `/bin/zsh -lc "cat go.mod \| head -8; rg -n \"func writeInfo\|buildFields\\(\" core/logx/logs.go"`
7	✓ ✓ `/bin/zsh -lc "sed -n '15,40p' core/logx/logs.go"`	✓ ✓ `/bin/zsh -lc "sed -n '400,460p' core/logx/logs.go"`
8	`file_change` (✓ ✓)	✓ ✓ `/bin/zsh -lc 'rg -n "func addCaller" core/logx'`
9	`file_change` (✓ ✓)	✓ ✓ `/bin/zsh -lc "sed -n '335,355p' core/logx/logs.go; rg -n \"getWriter\$\$\\.(Alert\|Severe\|Stack\|Stat)\" core/logx"`
10	`file_change` (✓ ✓)	`file_change` (✓ ✓)
11	`file_change` (✓ ✓)	`file_change` (✓ ✓)
12	`file_change` (✓ ✓)	`file_change` (✓ ✓)
13	✓ ✓ `/bin/zsh -lc 'gofmt -w core/logx/logs.go core/logx/richlogger.go && \ git diff --check && \ git diff -- core/logx/logs.go core/logx/richlogger.go core/logx/readme-cn.md core/logx/readme.md'`	`file_change` (✓ ✓)
14	—	`file_change` (✓ ✓)
15	—	✓ ✓ `/bin/zsh -lc 'gofmt -w core/logx/fields.go core/logx/logs.go core/logx/richlogger.go && \ git diff --check && \ git diff -- core/logx'`

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

default prompt · none MCP instructions · write semedit restriction

Task complexity across locality, context, ambiguity, composition, invariants, edit scale, and staging (1–5).

Setting	Value
Test case	`zeromicro__go-zero-2463`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`none`
Semedit arm restriction	`write`
Run provenance	`coding_network=disabled,oracle_patch_installation=checked`
Fixture	testdata/bench/zeromicro__go-zero-2463.txtar

LLM Prompt:

logx能否支持一下全局添加field功能，或默认加一个serviceName，收集日志想根据服务分索引很不方便

Metric	Vanilla (Small)	⚠ Semedit Agentic (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	69.19s	68.45s	N/A	—	—	—
Process Start → First Event	0.15s	0.15s	—	—	—	—
First Event → First Tool Call	5.54s	5.54s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	1	1	N/A	—	—	—
Internal Tool Cycles	11	11	N/A	—	—	—
Initial Load / Discovery Turns	4	3	N/A	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	11	11	N/A	—	—	—
Output Tokens	2068	2403	N/A	—	—	—
Reasoning / Thinking Tokens	756	781	N/A	—	—	—
Total Input Tokens	281112	355682	N/A	—	—	—
Cached Input Tokens	257024	315392	N/A	—	—	—
Uncached Input Tokens	24088	40290	N/A	—	—	—
Cost	0.1598	0.2194	N/A	—	—	—
Cached vs Uncached Token Ratio	10.67:1	7.83:1	N/A	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	FAIL	FAIL	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	FAIL	—	—	—	—

Baseline vs Semedit Agentic in Small Context

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'logx' -g '*.go' \| head -80"`	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'logx' \| head -100 && \ rg -n \"package logx\|WithFields\|Field\|serviceName\|ServiceName\" --glob '*.go' . \| head -200"`
2	✓ ✓ `/bin/zsh -lc "rg --files \| rg '("'^\|/)logx(/\|$)\|PROJECT_BRIEF.md\|(''^\|/)AGENTS.md'"' \| head -100"`	✓ ✓ `/bin/zsh -lc "git status --short && \ rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'README.md' \| head -80 && \ sed -n '1,220p' core/logx/logs.go && \ sed -n '1,230p' core/logx/richlogger.go && \ sed -n '1,110p' core/logx/vars.go && \ sed -n '1,100p' core/logx/fields.go"`
3	? ✗ `/bin/zsh -lc "cat core/logx/fields.go core/logx/vars.go core/logx/config.go && \ rg -n 'WithFields\|With.Field\|serviceName\|SetLogger\|New.Logger' core/logx *.go"`	✓ ✓ `/bin/zsh -lc "sed -n '190,480p' core/logx/logs.go && \ sed -n '1,180p' core/logx/fields_test.go && \ sed -n '1,180p' core/logx/logs_test.go && \ rg -n \"SetGlobal\|SetUp\\(\|LogConf\" core/logx"`
4	✓ ✓ `/bin/zsh -lc 'cat core/logx/logger.go core/logx/richlogger.go core/logx/logs.go'`	`file_change` (✓ ✓)
5	✓ ✓ `/bin/zsh -lc "rg -n 'func .write\|fields\|LogConf\|SetUp' core/logx/logs.go core/logx/writer.go core/logx/readme-cn.md"`	`file_change` (✓ ✓)
6	? ✗ `/bin/zsh -lc "cat core/logx/fields_test.go; sed -n '250,310p' core/logx/writer.go; rg -n 'func Reset\|setupOnce' core/logx/*test.go"`	✓ ✓ `/bin/zsh -lc 'gofmt -w core/logx/logs.go core/logx/richlogger.go && \ git diff --check && \ git diff -- core/logx/logs.go core/logx/richlogger.go'`
7	✓ ✓ `/bin/zsh -lc "sed -n '10,35p' core/logx/writer.go; rg -n 'func \\(.Alert\|func \\(.Severe\|func \\(.*Stack' core/logx/writer.go"`	✓ ✓ `/bin/zsh -lc "sed -n '1,180p' core/logx/readme.md && \ git status --short && \ rg -n \"WithContext\|ContextWithFields\|Infow\" core/logx/readme*"`
8	`file_change` (✓ ✓)	`file_change` (✓ ✓)
9	`file_change` (✓ ✓)	✓ ✓ `/bin/zsh -lc "sed -n '1,110p' core/logx/readme-cn.md"`
10	`file_change` (✓ ✓)	`file_change` (✓ ✓)
11	✓ ✓ `/bin/zsh -lc 'gofmt -w core/logx/logs.go core/logx/richlogger.go && \ git diff -- core/logx/logs.go core/logx/richlogger.go'`	✓ ✓ `/bin/zsh -lc 'git diff --check && \ git status --short && \ git diff --stat'`

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

default prompt · none MCP instructions · write semedit restriction

Task complexity across locality, context, ambiguity, composition, invariants, edit scale, and staging (1–5).

Setting	Value
Test case	`zeromicro__go-zero-2463`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`none`
Semedit arm restriction	`write`
Run provenance	`coding_network=disabled,oracle_patch_installation=checked`
Fixture	testdata/bench/zeromicro__go-zero-2463.txtar

LLM Prompt:

logx能否支持一下全局添加field功能，或默认加一个serviceName，收集日志想根据服务分索引很不方便

Metric	Vanilla (Small)	⚠ Semedit Agentic (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	92.99s	54.75s	N/A	—	—	—
Process Start → First Event	0.60s	0.14s	—	—	—	—
First Event → First Tool Call	6.22s	3.70s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	1	1	N/A	—	—	—
Internal Tool Cycles	14	13	N/A	—	—	—
Initial Load / Discovery Turns	5	6	N/A	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	14	13	N/A	—	—	—
Output Tokens	3157	3314	N/A	—	—	—
Reasoning / Thinking Tokens	1095	1147	N/A	—	—	—
Total Input Tokens	498457	430528	N/A	—	—	—
Cached Input Tokens	456704	399360	N/A	—	—	—
Uncached Input Tokens	41753	31168	N/A	—	—	—
Cost	0.2717	0.2335	N/A	—	—	—
Cached vs Uncached Token Ratio	10.94:1	12.81:1	N/A	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	FAIL	FAIL	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	FAIL	—	—	—	—

Baseline vs Semedit Agentic in Small Context


/bin/zsh -lc "pwd && \
rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g '*logx*' -g '*.go' | head -100"
/bin/zsh -lc "find .. -name AGENTS.md -o -name PROJECT_BRIEF.md; rg -n \"package logx|type .*Logger|WithFields|Fields\\(\" logx"
/bin/zsh -lc "rg --files | rg '("'^|/)logx|PROJECT_BRIEF|AGENTS.md'"' | head -80"
/bin/zsh -lc 'cat core/logx/config.go && \
cat core/logx/fields.go && \
cat core/logx/logger.go && \
rg -n "WithFields|fields|ServiceName" core/logx'
/bin/zsh -lc "sed -n '1,220p' core/logx/logs.go; sed -n '220,470p' core/logx/logs.go; sed -n '1,180p' core/logx/richlogger.go; sed -n '1,100p' core/logx/vars.go; sed -n '1,100p' core/logx/readme-cn.md"
/bin/zsh -lc "sed -n '15,45p' core/logx/logs.go"
/bin/zsh -lc "sed -n '1,180p' core/logx/richlogger.go; sed -n '1,220p' core/logx/writer.go; rg -n 'func.*(Debug|Info|Error|Stat|Severe|Stack)' core/logx/*writer*"
/bin/zsh -lc "rg -n 'func output|range fields|logEntry' core/logx"
/bin/zsh -lc "rg -n '"'^##|ServiceName'"' core/logx/readme.md"
/bin/zsh -lc "sed -n '20,48p' core/logx/logs.go"

/bin/zsh -lc 'gofmt -w core/logx/logs.go core/logx/richlogger.go && \
git diff -- core/logx/logs.go core/logx/richlogger.go core/logx/readme.md core/logx/readme-cn.md'
/bin/zsh -lc "rg -n 'SetUp\\(|MustSetup\\(' --glob '*_test.go' | head -30; git status --short"
/bin/zsh -lc 'gofmt -w core/logx/logs.go core/logx/writer.go && \
git diff -- core/logx/logs.go core/logx/writer.go core/logx/readme-cn.md && \
git status --short'

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

Test case: `zeromicro__go-zero-964`

Target: `codex/gpt-6-luna/high`

default prompt · none MCP instructions · write semedit restriction

Task complexity across locality, context, ambiguity, composition, invariants, edit scale, and staging (1–5).

Setting	Value
Test case	`zeromicro__go-zero-964`
Target	`codex/gpt-6-luna/high`
Prompt variant	`default`
MCP server instructions	`none`
Semedit arm restriction	`write`
Run provenance	`coding_network=disabled,oracle_patch_installation=checked`
Fixture	testdata/bench/zeromicro__go-zero-964.txtar

LLM Prompt:

如题。 etcd打开了rbac authentication。但discov.EtcdConf中没有user等字段，无法进行身份认证。连接etcd时会报错： etcdserver: user name is empty

default prompt · none MCP instructions · write semedit restriction

Task complexity across locality, context, ambiguity, composition, invariants, edit scale, and staging (1–5).

Setting	Value
Test case	`zeromicro__go-zero-964`
Target	`codex/gpt-6-luna/high`
Prompt variant	`default`
MCP server instructions	`none`
Semedit arm restriction	`write`
Run provenance	`coding_network=disabled,oracle_patch_installation=checked`
Fixture	testdata/bench/zeromicro__go-zero-964.txtar

LLM Prompt:

如题。 etcd打开了rbac authentication。但discov.EtcdConf中没有user等字段，无法进行身份认证。连接etcd时会报错： etcdserver: user name is empty

Metric	Vanilla (Small)	⚠ Semedit Agentic (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	—	206.30s	—	—	—	—
Process Start → First Event	—	0.27s	—	—	—	—
First Event → First Tool Call	—	5.86s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	—	1	—	—	—	—
Internal Tool Cycles	—	19	—	—	—	—
Initial Load / Discovery Turns	—	8	—	—	—	—
MCP Discovery / Schema Turns	—	0	—	—	—	—
Tool Invocations (per run)	—	19	—	—	—	—
Output Tokens	—	8238	—	—	—	—
Reasoning / Thinking Tokens	—	3058	—	—	—	—
Total Input Tokens	—	572191	—	—	—	—
Cached Input Tokens	—	531456	—	—	—	—
Uncached Input Tokens	—	40735	—	—	—	—
Cost	—	0.3759	—	—	—	—
Cached vs Uncached Token Ratio	—	13.05:1	—	—	—	—
Oracle L1: Mutation Policy	—	PASS	—	—	—	—
Oracle L2: AST Invariants	—	PASS	—	—	—	—
Oracle L3: Clean Build	—	PASS	—	—	—	—
Oracle L4: Verification Test	—	FAIL	—	—	—	—
Semantic Tool Invocation Verified (per run)	—	FAIL	—	—	—	—

Baseline vs Semedit Agentic in Small Context

#	Vanilla	Semedit MCP
1	not published	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'etcd' -g 'discov' \| head -100"`
2	not published	✓ ✓ `/bin/zsh -lc "rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g 'etcd' -g 'discov' \| head -200 && \ rg -n 'type EtcdConf\|EtcdConf\|NewClient\|Username\|Password\|user name is empty' core/discov"`
3	not published	✓ ✓ `/bin/zsh -lc "ls -la \| head -50 && \ cat core/discov/config.go && \ cat core/discov/internal/etcdclient.go && \ sed -n '1,100p' core/discov/config_test.go && \ cat go.mod \| head -30"`
4	not published	✓ ✓ `/bin/zsh -lc "rg -n 'DialClient\|NewClient =\|endpoints\|EtcdConf' core/discov"`
5	not published	✓ ✓ `/bin/zsh -lc "sed -n '1,120p' core/discov/internal/registry.go && \ sed -n '220,325p' core/discov/internal/registry.go && \ rg -n 'discov\\.EtcdConf\|EtcdConf:\|NewSubscriber\\(\|NewPublisher\\(' --glob '*.go' \| head -100"`
6	not published	✓ ✓ `/bin/zsh -lc "cat zrpc/config.go && \ sed -n '1,120p' zrpc/server.go && \ sed -n '1,100p' zrpc/client.go && \ sed -n '1,100p' zrpc/internal/resolver/discovbuilder.go && \ sed -n '1,100p' core/discov/subscriber.go && \ sed -n '1,90p' core/discov/publisher.go"`
7	not published	✓ ✓ `/bin/zsh -lc "rg -n 'BuildDiscovTarget\|DiscovScheme\|EndpointSepChar\|BuildDirectTarget' zrpc"`
8	not published	✓ ✓ `/bin/zsh -lc "cat zrpc/internal/target.go && \ cat zrpc/internal/rpcpubserver.go && \ rg -n 'NewRpcPubServer' zrpc"`
9	not published	`file_change` (✓ ✓)
10	not published	✓ ✓ `/bin/zsh -lc "cat core/discov/internal/vars.go && \ sed -n '1,80p' core/discov/internal/registry_test.go && \ sed -n '1,50p' core/discov/clients_test.go"`
11	not published	`file_change` (✓ ✓)
12	not published	`file_change` (✓ ✓)
13	not published	`file_change` (✓ ✓)
14	not published	`file_change` (✓ ✓)
15	not published	`file_change` (✓ ✓)
16	not published	✓ ✓ `/bin/zsh -lc 'gofmt -w core/discov/config.go core/discov/internal/vars.go core/discov/internal/registry.go core/discov/subscriber.go core/discov/publisher.go zrpc/client.go zrpc/config.go zrpc/internal/target.go zrpc/internal/resolver/discovbuilder.go zrpc/internal/rpcpubserver.go zrpc/server.go && \ git diff --stat && \ git diff -- core/discov/internal/registry.go zrpc/internal/target.go zrpc/internal/rpcpubserver.go'`
17	not published	`file_change` (✓ ✓)
18	not published	✓ ✓ `/bin/zsh -lc "cat core/discov/internal/vars.go && \ sed -n '1,65p' zrpc/config.go && \ sed -n '1,90p' core/discov/subscriber.go && \ git diff --check && \ git status --short"`
19	not published	✓ ✓ `/bin/zsh -lc "nl -ba core/discov/config.go \| head -22 && \ nl -ba core/discov/internal/registry.go \| sed -n '35,100p;250,275p;330,370p' && \ nl -ba zrpc/internal/target.go \| sed -n '12,40p' && \ nl -ba zrpc/server.go \| sed -n '38,50p'"`

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

default prompt · none MCP instructions · write semedit restriction

Task complexity across locality, context, ambiguity, composition, invariants, edit scale, and staging (1–5).

Setting	Value
Test case	`zeromicro__go-zero-964`
Target	`codex/gpt-6-luna/high`
Prompt variant	`default`
MCP server instructions	`none`
Semedit arm restriction	`write`
Run provenance	`coding_network=disabled,oracle_patch_installation=checked`
Fixture	testdata/bench/zeromicro__go-zero-964.txtar

LLM Prompt:

如题。 etcd打开了rbac authentication。但discov.EtcdConf中没有user等字段，无法进行身份认证。连接etcd时会报错： etcdserver: user name is empty

Metric	Vanilla (Small)	⚠ Semedit Agentic (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	278.83s	129.45s	N/A	—	—	—
Process Start → First Event	0.50s	0.54s	—	—	—	—
First Event → First Tool Call	5.32s	4.41s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	1	1	N/A	—	—	—
Internal Tool Cycles	28	32	N/A	—	—	—
Initial Load / Discovery Turns	9	8	N/A	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	28	32	N/A	—	—	—
Output Tokens	11934	11307	N/A	—	—	—
Reasoning / Thinking Tokens	5954	4244	N/A	—	—	—
Total Input Tokens	964808	1309415	N/A	—	—	—
Cached Input Tokens	914176	1250048	N/A	—	—	—
Uncached Input Tokens	50632	59367	N/A	—	—	—
Cost	0.5787	0.6553	N/A	—	—	—
Cached vs Uncached Token Ratio	18.06:1	21.06:1	N/A	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	FAIL	FAIL	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	FAIL	—	—	—	—

Baseline vs Semedit Agentic in Small Context

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'etcd' \| head -100"`	✓ ✓ `/bin/zsh -lc "pwd && \ rg -n \"type EtcdConf\|EtcdConf\|clientv3\\.Config\|New\\(\" --glob '.go' . \| head -200 && \ rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'go.mod' -g 'etcd*'"`
2	✓ ✓ `/bin/zsh -lc "ls -a && \ rg -n \"type EtcdConf\|EtcdConf struct\|NewEtcd\|etcd\\.New\|clientv3.Config\|Username\|Password\" core/discov"`	✓ ✓ `/bin/zsh -lc "rg --files core/discov \| sort && \ rg -n \"EtcdConf\|Etcd.Conf\|clientv3.New\|Username\|Password\" core/discov core --glob '.go' \| head -180 && \ git status --short"`
3	✓ ✓ `/bin/zsh -lc "sed -n '1,200p' core/discov/config.go && \ sed -n '250,335p' core/discov/internal/registry.go && \ rg -n \"EtcdConf\""`	? ✗ `/bin/zsh -lc "cat core/discov/config.go && \ sed -n '260,340p' core/discov/internal/registry.go && \ cat core/discov/config_test.go && \ sed -n '1,180p' core/discov/clients.go && \ rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md'"`
4	✓ ✓ `/bin/zsh -lc "sed -n '1,200p' core/discov/etcd.go 2>/dev/null; rg -n \"DialClient\\(\|NewEtcd\|etcdclient\" core/discov"`	✓ ✓ `/bin/zsh -lc "rg -n \"DialClient\\(\|New.Cluster\|NewEtcd\|NewSubscriber\|NewPublisher\|Etcd\\.Hosts\|\\.Etcd\" core/discov zrpc --glob '.go' && \ sed -n '1,220p' core/discov/internal/registry.go && \ sed -n '1,220p' core/discov/subscriber.go && \ sed -n '1,200p' core/discov/publisher.go"`
5	✓ ✓ `/bin/zsh -lc "rg --files core/discov \| sort && \ rg -n \"DialClient\|cluster.New\|NewCluster\|NewSubscriber\|NewPublisher\|internal\" core/discov --glob '*.go'"`	✓ ✓ `/bin/zsh -lc "sed -n '1,120p' zrpc/client.go && \ sed -n '1,100p' zrpc/server.go && \ cat zrpc/config.go && \ cat zrpc/internal/resolver/discovbuilder.go && \ sed -n '1,120p' zrpc/internal/rpcpubserver.go && \ rg -n \"GetRegistry\$\$\\.(GetConn\|Monitor)\|DialClient\|BuildDiscovTarget\" --glob '*.go'"`
6	✓ ✓ `/bin/zsh -lc "cat core/discov/internal/vars.go && \ sed -n '1,130p' core/discov/internal/registry.go && \ sed -n '1,100p' core/discov/publisher.go && \ sed -n '1,90p' core/discov/subscriber.go && \ sed -n '1,120p' zrpc/config.go && \ cat core/discov/config_test.go"`	✓ ✓ `/bin/zsh -lc "rg -n \"Register\|discovBuilder\|BuildDiscovTarget\|resolver\" zrpc/internal --glob '*.go' && \ cat zrpc/internal/target.go && \ sed -n '1,200p' zrpc/internal/client.go && \ cat core/discov/internal/vars.go && \ sed -n '1,80p' core/discov/internal/etcdclient.go"`
7	✓ ✓ `/bin/zsh -lc "rg -n \"NewSubscriber\|NewPublisher\|Etcd\\.Hosts\|Etcd\\.Key\|EtcdConf\|discov\" zrpc rest core --glob '*.go'"`	✓ ✓ `/bin/zsh -lc 'rg -n "type Target struct" $(go env GOPATH 2>/dev/null)/pkg/mod/google.golang.org/grpc* -g '"'*.go' \| head"`
8	✓ ✓ `/bin/zsh -lc "cat zrpc/internal/resolver/discovbuilder.go && \ sed -n '1,130p' zrpc/client.go && \ sed -n '1,110p' zrpc/internal/rpcpubserver.go && \ sed -n '1,100p' zrpc/server.go && \ cat core/discov/clients.go"`	✓ ✓ `/bin/zsh -lc "cat core/discov/clients_test.go && \ rg -n \"GetConn\\(\|Monitor\\(\|newCluster\\(\" core/discov/internal --glob '*_test.go'"`
9	✓ ✓ `/bin/zsh -lc "rg -n \"newClient\|NewClient\\(\" core/discov/internal/registry.go && \ sed -n '155,225p' core/discov/internal/registry.go && \ cat zrpc/internal/target.go"`	`file_change` (✓ ✓)
10	`file_change` (✓ ✓)	`file_change` (✓ ✓)
11	✓ ✓ `/bin/zsh -lc "rg -n \"NewClient =\|getClusterKey\\(\|newCluster\\(\|GetConn\\(\|Monitor\\(\" core/discov/internal"`	`file_change` (✓ ✓)
12	`file_change` (✓ ✓)	`file_change` (✓ ✓)
13	`file_change` (✓ ✓)	`file_change` (✓ ✓)
14	`file_change` (✓ ✓)	`file_change` (✓ ✓)
15	✓ ✓ `/bin/zsh -lc "sed -n '32,95p' core/discov/internal/registry.go && \ sed -n '225,255p' core/discov/internal/registry.go && \ sed -n '1,65p' core/discov/subscriber.go && \ sed -n '1,60p' core/discov/publisher.go && \ sed -n '1,75p' zrpc/internal/resolver/discovbuilder.go"`	`file_change` (✓ ✓)
16	✓ ✓ `/bin/zsh -lc "rg -n \"func \$.Target\$.Endpoint\|type Target struct\" "'$(go env GOPATH)/pkg/mod/google.golang.org/grpc* -g '"'.go' \| head -15 && \ rg -n \"BuildDiscovTarget\" zrpc/internal/test.go"`	`file_change` (✓ ✓)
17	✓ ✓ `/bin/zsh -lc "rg 'google.golang.org/grpc' go.mod && \ go env GOMODCACHE"`	`file_change` (✓ ✓)
18	? ✗ `/bin/zsh -lc "rg -n \"type Target struct\|func \$.Target\$.Endpoint\|func \$.Target\$.String\" .scratch/go/mod/google.golang.org/grpc@v1.41.0/resolver -g '*.go'"`	? ✗ /bin/zsh -lc 'gofmt -w core/discov/config.go core/discov/internal/registry.go core/discov/publisher.go core/discov/subscriber.go zrpc/client.go zrpc/server.go zrpc/internal/target.go zrpc/internal/resolver/discovbuilder.go zrpc/internal/rpcpubserver.go && \ git diff --check && \ git diff --stat && \ git diff -- core/discov/config.go core/discov/internal/registry.go core/discov/publisher.go core/discov/subscriber.go zrpc/client.go zrpc/server.go zrpc/internal/target.go zrpc/internal/resolver/discovbuilder.go zrpc/internal/rpcpubserver.go'
19	✓ ✓ `/bin/zsh -lc 'gofmt -w core/discov/config.go core/discov/internal/registry.go core/discov/internal/vars.go core/discov/publisher.go core/discov/subscriber.go zrpc/internal/target.go zrpc/internal/resolver/discovbuilder.go zrpc/server.go zrpc/client.go zrpc/internal/rpcpubserver.go && \ git diff --stat && \ git diff -- core/discov/config.go core/discov/internal/registry.go'`	✓ ✓ `/bin/zsh -lc "sed -n '290,350p' core/discov/internal/registry.go"`
20	✓ ✓ `/bin/zsh -lc "find .scratch/go -path 'grpcresolver*' -type f 2>/dev/null \| head -20"`	`file_change` (✓ ✓)
21	`file_change` (✓ ✓)	✓ ✓ `/bin/zsh -lc 'gofmt -w core/discov/config.go core/discov/internal/registry.go core/discov/publisher.go core/discov/subscriber.go zrpc/client.go zrpc/server.go zrpc/internal/target.go zrpc/internal/resolver/discovbuilder.go zrpc/internal/rpcpubserver.go && \ git diff --check && \ git diff --stat && \ git status --short'`
22	`file_change` (✓ ✓)	✓ ✓ `/bin/zsh -lc "rg -n \"getCluster\\(\|newCluster\\(\|GetConn\\(\|\\.Monitor\\(\" core/discov --glob '*.go' && \ sed -n '20,100p' core/discov/publisher.go && \ sed -n '20,65p' core/discov/subscriber.go && \ sed -n '24,65p' core/discov/internal/registry.go && \ sed -n '1,60p' zrpc/internal/resolver/discovbuilder.go && \ git diff --check"`
23	✓ ✓ `/bin/zsh -lc 'gofmt -w zrpc/internal/target.go zrpc/internal/resolver/discovbuilder.go && \ git diff --check && \ git status --short && \ git diff -- zrpc/internal/target.go zrpc/internal/resolver/discovbuilder.go zrpc/internal/rpcpubserver.go zrpc/server.go zrpc/client.go'`	`file_change` (✓ ✓)
24	`file_change` (✓ ✓)	✓ ✓ `/bin/zsh -lc "rg -n \"func \$c \\cluster\$ newClient\|getClient\$\" core/discov/internal/registry.go && \ sed -n '190,225p' core/discov/internal/registry.go && \ rg -n \"func \\(r \\Registry\$\" core/discov/internal/registry.go"`
25	`file_change` (✓ ✓)	✓ ✓ `/bin/zsh -lc "sed -n '96,122p' core/discov/internal/registry.go && \ sed -n '246,265p' core/discov/internal/registry.go"`
26	✓ ✓ `/bin/zsh -lc 'gofmt -w core/discov/internal/registry.go zrpc/config.go && \ git diff --check && \ git diff --stat && \ git status --short'`	`file_change` (✓ ✓)
27	`file_change` (✓ ✓)	`file_change` (✓ ✓)
28	✓ ✓ `/bin/zsh -lc 'gofmt -w core/discov/publisher.go core/discov/subscriber.go zrpc/internal/resolver/discovbuilder.go zrpc/internal/rpcpubserver.go && \ rg -n "WithEtcdAuth\|WithPublisherEtcdAuth\|WithSubscriberEtcdAuth\|NewClientWithAuth\|func getClusterKey" core/discov zrpc && \ git diff --check'`	✓ ✓ `/bin/zsh -lc "gofmt -w core/discov/config.go core/discov/internal/registry.go core/discov/publisher.go core/discov/subscriber.go zrpc/client.go zrpc/server.go zrpc/internal/target.go zrpc/internal/resolver/discovbuilder.go zrpc/internal/rpcpubserver.go && \ git diff --check && \ git diff --stat && \ rg -n \"WithEtcdAuth\|WithPublisherEtcdAuth\|DialClientWithAuth\|Username\|Password\" core/discov zrpc --glob '*.go'"`
29	—	✓ ✓ `/bin/zsh -lc 'git diff -- core/discov/internal/registry.go zrpc/internal/target.go zrpc/internal/resolver/discovbuilder.go zrpc/internal/rpcpubserver.go core/discov/config.go core/discov/publisher.go core/discov/subscriber.go zrpc/client.go zrpc/server.go'`
30	—	✓ ✓ `/bin/zsh -lc "head -8 go.mod && \ rg -n \"strings\\.Cut\\(\" --glob '*.go' \| head"`
31	—	`file_change` (✓ ✓)
32	—	✓ ✓ `/bin/zsh -lc 'gofmt -w zrpc/internal/resolver/discovbuilder.go core/discov/internal/registry.go && \ git diff --check && \ git status --short'`

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

Target: `codex/gpt-6-luna/medium`

default prompt · none MCP instructions · write semedit restriction

Task complexity across locality, context, ambiguity, composition, invariants, edit scale, and staging (1–5).

Setting	Value
Test case	`zeromicro__go-zero-964`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`none`
Semedit arm restriction	`write`
Run provenance	`coding_network=disabled,oracle_patch_installation=checked`
Fixture	testdata/bench/zeromicro__go-zero-964.txtar

LLM Prompt:

如题。 etcd打开了rbac authentication。但discov.EtcdConf中没有user等字段，无法进行身份认证。连接etcd时会报错： etcdserver: user name is empty

Metric	Vanilla (Small)	⚠ Semedit Agentic (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	63.43s	67.00s	N/A	—	—	—
Process Start → First Event	0.58s	0.15s	—	—	—	—
First Event → First Tool Call	5.36s	4.97s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	1	1	N/A	—	—	—
Internal Tool Cycles	9	11	N/A	—	—	—
Initial Load / Discovery Turns	8	9	N/A	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	9	11	N/A	—	—	—
Output Tokens	1360	1511	N/A	—	—	—
Reasoning / Thinking Tokens	486	443	N/A	—	—	—
Total Input Tokens	220315	270066	N/A	—	—	—
Cached Input Tokens	181760	243712	N/A	—	—	—
Uncached Input Tokens	38555	26354	N/A	—	—	—
Cost	0.1649	0.1512	N/A	—	—	—
Cached vs Uncached Token Ratio	4.71:1	9.25:1	N/A	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	FAIL	FAIL	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	FAIL	—	—	—	—

Baseline vs Semedit Agentic in Small Context


/bin/zsh -lc "pwd && \
rg -n \"type EtcdConf|EtcdConf|etcdserver|Username|username\" --glob '*.go' --glob 'PROJECT_BRIEF.md' --glob 'AGENTS.md' ."
/bin/zsh -lc "rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'go.mod' -g '*etcd*' core/discov && \
cat core/discov/config.go && \
rg -n 'NewEtcd|clientv3.Config|Endpoints:|discov.EtcdConf|etcd.New' . --glob '*.go'"
/bin/zsh -lc "cat core/discov/internal/registry.go && \
cat core/discov/config_test.go && \
rg -n 'NewEtcdClient|EtcdConf|Hosts:' core/discov"
/bin/zsh -lc "rg -n 'DialClient|NewClient\\(' core/discov && \
rg --files | rg 'PROJECT_BRIEF|AGENTS\\.md' | head -20 && \
rg -n 'DialClient\\('"
/bin/zsh -lc "ls core/discov && \
cat core/discov/etcd.go 2>/dev/null; cat core/discov/clients.go; cat core/discov/internal/vars.go; rg -n 'Registry|GetConn|Monitor\\(' core/discov"
/bin/zsh -lc 'cat core/discov/publisher.go core/discov/subscriber.go'
/bin/zsh -lc "rg -n 'NewSubscriber|NewPublisher' zrpc core --glob '*.go'"
/bin/zsh -lc 'cat zrpc/internal/resolver/discovbuilder.go zrpc/internal/rpcpubserver.go zrpc/config.go'
/bin/zsh -lc ''

/bin/zsh -lc 'git diff -- core/discov/config.go'

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

default prompt · none MCP instructions · write semedit restriction

Task complexity across locality, context, ambiguity, composition, invariants, edit scale, and staging (1–5).

Setting	Value
Test case	`zeromicro__go-zero-964`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`none`
Semedit arm restriction	`write`
Run provenance	`coding_network=disabled,oracle_patch_installation=checked`
Fixture	testdata/bench/zeromicro__go-zero-964.txtar

LLM Prompt:

如题。 etcd打开了rbac authentication。但discov.EtcdConf中没有user等字段，无法进行身份认证。连接etcd时会报错： etcdserver: user name is empty

Metric	Vanilla (Small)	⚠ Semedit Agentic (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	65.01s	115.28s	N/A	—	—	—
Process Start → First Event	0.15s	0.24s	—	—	—	—
First Event → First Tool Call	4.76s	5.06s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	1	1	N/A	—	—	—
Internal Tool Cycles	15	18	N/A	—	—	—
Initial Load / Discovery Turns	15	7	N/A	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	15	18	N/A	—	—	—
Output Tokens	1461	3966	N/A	—	—	—
Reasoning / Thinking Tokens	413	933	N/A	—	—	—
Total Input Tokens	365064	483687	N/A	—	—	—
Cached Input Tokens	310528	452864	N/A	—	—	—
Uncached Input Tokens	54536	30823	N/A	—	—	—
Cost	0.2374	0.2515	N/A	—	—	—
Cached vs Uncached Token Ratio	5.69:1	14.69:1	N/A	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	FAIL	FAIL	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	FAIL	—	—	—	—

Baseline vs Semedit Agentic in Small Context


/bin/zsh -lc "pwd && \
rg -n \"type EtcdConf|EtcdConf|discov\" --glob '*.go' | head -200 && \
rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'go.mod'"
/bin/zsh -lc 'cat core/discov/config.go && \
rg -n "NewPublisher|NewSubscriber|EtcdConf|etcd.New|clientv3.Config|Endpoints" core/discov core/discov/internal'
/bin/zsh -lc "sed -n '260,340p' core/discov/internal/registry.go && \
sed -n '1,180p' core/discov/internal/registry.go && \
cat core/discov/publisher.go && \
cat core/discov/subscriber.go && \
cat core/discov/config_test.go"
/bin/zsh -lc "cat zrpc/internal/resolver/discovbuilder.go; cat zrpc/config.go; rg -n \"DialClient\\(|GetConn\\(|\\.Monitor\\(\" --glob '*.go' core zrpc"
/bin/zsh -lc 'cat zrpc/client.go; cat zrpc/internal/rpcpubserver.go && \
cat core/discov/internal/clients.go 2>/dev/null'
/bin/zsh -lc "cat zrpc/internal/target.go; rg -n \"NewRpcPubServer\\(\""
/bin/zsh -lc ''
/bin/zsh -lc "rg -n \"GetConn|Monitor\\(\" core/discov --glob '*.go' | tail -20; cat core/discov/discov.go 2>/dev/null"
/bin/zsh -lc 'cat core/discov/publisher.go; cat core/discov/subscriber.go'
/bin/zsh -lc "rg -n 'DialClient|newClient' core/discov/internal/registry.go"
/bin/zsh -lc "sed -n '235,260p' core/discov/internal/registry.go"
/bin/zsh -lc "rg -n 'NewClient' core/discov/internal"
/bin/zsh -lc "sed -n '55,90p' zrpc/client.go"
/bin/zsh -lc "sed -n '1,55p' core/discov/internal/vars.go; sed -n '1,50p' zrpc/internal/target.go"
/bin/zsh -lc ''


/bin/zsh -lc 'gofmt -w core/discov/config.go core/discov/internal/registry.go core/discov/publisher.go core/discov/subscriber.go zrpc/server.go zrpc/internal/rpcpubserver.go zrpc/internal/target.go zrpc/client.go zrpc/internal/resolver/discovbuilder.go && \
git diff --stat && \
git diff -- core/discov/internal/registry.go zrpc/internal/resolver/discovbuilder.go'

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

default prompt · none MCP instructions · write semedit restriction

Task complexity across locality, context, ambiguity, composition, invariants, edit scale, and staging (1–5).

Setting	Value
Test case	`zeromicro__go-zero-964`
Target	`codex/gpt-6-luna/medium`
Prompt variant	`default`
MCP server instructions	`none`
Semedit arm restriction	`write`
Run provenance	`coding_network=disabled,oracle_patch_installation=checked`
Fixture	testdata/bench/zeromicro__go-zero-964.txtar

LLM Prompt:

如题。 etcd打开了rbac authentication。但discov.EtcdConf中没有user等字段，无法进行身份认证。连接etcd时会报错： etcdserver: user name is empty

Metric	Vanilla (Small)	⚠ Semedit Agentic (Small)	Δ (Small)	Vanilla (Large)	MCP (Large)	Δ (Large)
Wall-Clock Latency	70.06s	138.13s	N/A	—	—	—
Process Start → First Event	0.61s	0.14s	—	—	—	—
First Event → First Tool Call	5.39s	6.19s	—	—	—	—
MCP Initialize → First Semantic Call	—	—	—	—	—	—
MCP Server Start → Initialize	—	—	—	—	—	—
Top-Level User Turns	1	1	N/A	—	—	—
Internal Tool Cycles	10	26	N/A	—	—	—
Initial Load / Discovery Turns	8	11	N/A	—	—	—
MCP Discovery / Schema Turns	0	0	—	—	—	—
Tool Invocations (per run)	10	26	N/A	—	—	—
Output Tokens	1750	4296	N/A	—	—	—
Reasoning / Thinking Tokens	674	1132	N/A	—	—	—
Total Input Tokens	277076	725728	N/A	—	—	—
Cached Input Tokens	247040	662784	N/A	—	—	—
Uncached Input Tokens	30036	62944	N/A	—	—	—
Cost	0.1671	0.3909	N/A	—	—	—
Cached vs Uncached Token Ratio	8.22:1	10.53:1	N/A	—	—	—
Oracle L1: Mutation Policy	PASS	PASS	—	—	—	—
Oracle L2: AST Invariants	PASS	PASS	—	—	—	—
Oracle L3: Clean Build	PASS	PASS	—	—	—	—
Oracle L4: Verification Test	FAIL	FAIL	—	—	—	—
Semantic Tool Invocation Verified (per run)	N/A	FAIL	—	—	—	—

Baseline vs Semedit Agentic in Small Context

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "pwd && \ rg -n \"type EtcdConf\|EtcdConf\|etcdserver\|clientv3.Config\|Username\|Password\" --glob '*.go' . \| head -250 && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod'"`	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'go.mod' -g '*.go' \| head -200"`
2	✓ ✓ `/bin/zsh -lc "cat core/discov/config.go && \ sed -n '260,330p' core/discov/internal/registry.go && \ cat core/discov/config_test.go && \ cat go.mod \| head -25 && \ find .. -name AGENTS.md -o -name PROJECT_BRIEF.md"`	? ✗ `/bin/zsh -lc 'find .. -name AGENTS.md -o -name PROJECT_BRIEF.md; rg -n "type EtcdConf\|EtcdConf\|NewEtcd\|clientv3.Config" discov'`
3	✓ ✓ `/bin/zsh -lc "rg -n \"DialClient\\(\|newCluster\|EtcdConf\" core/discov"`	✓ ✓ `/bin/zsh -lc 'ls -a \| head -60 && \ rg -n "EtcdConf\|etcdserver\|NewEtcd"'`
4	✓ ✓ `/bin/zsh -lc "rg -n \"DialClient\|NewEtcd\|endpoints\" core/discov --glob '*.go'"`	✓ ✓ `/bin/zsh -lc 'cat core/discov/config.go; rg -n "clientv3.Config\|Endpoints:\|New.*Etcd\|EtcdConf" core/discov'`
5	✓ ✓ `/bin/zsh -lc "head -90 core/discov/internal/registry.go; sed -n '1,90p' core/discov/clients.go; rg -n \"NewSubscriber\|NewPublisher\|Etcd\""`	✓ ✓ `/bin/zsh -lc "sed -n '1,180p' core/discov/internal/registry.go; sed -n '270,335p' core/discov/internal/registry.go; rg -n \"NewClient\\(\" core/discov"`
6	✓ ✓ `/bin/zsh -lc "sed -n '1,120p' core/discov/subscriber.go; sed -n '1,90p' core/discov/publisher.go; sed -n '1,100p' zrpc/client.go; sed -n '1,80p' zrpc/server.go; sed -n '1,100p' zrpc/internal/resolver/discovbuilder.go; sed -n '1,60p' zrpc/internal/rpcpubserver.go; sed -n '220,250p' core/discov/internal/registry.go"`	✓ ✓ `/bin/zsh -lc "rg -n \"GetConn\|Monitor\\(\|NewPublisher\|NewSubscriber\|DialClient\|Etcd\" core/discov/*.go zrpc"`
7	✓ ✓ `/bin/zsh -lc "sed -n '1,120p' zrpc/internal/target.go; rg -n \"BuildDiscovTarget\|DiscovScheme\" zrpc/internal"`	✓ ✓ `/bin/zsh -lc "cat core/discov/subscriber.go core/discov/publisher.go \| head -100; sed -n '1,100p' core/discov/internal/registry_test.go; cat zrpc/internal/resolver/discovbuilder.go"`
8	✓ ✓ `/bin/zsh -lc ''`	✓ ✓ `/bin/zsh -lc ''`
9	`file_change` (✓ ✓)	✓ ✓ `/bin/zsh -lc "sed -n '1,75p' core/discov/publisher.go; cat zrpc/internal/rpcpubserver.go"`
10	`file_change` (✓ ✓)	✓ ✓ `/bin/zsh -lc "sed -n '1,80p' zrpc/config.go"`
11	—	✓ ✓ `/bin/zsh -lc ''`
12	—	`file_change` (✓ ✓)
13	—	✓ ✓ `/bin/zsh -lc "sed -n '55,100p' zrpc/client.go; rg -n \"BuildDiscovTarget\""`
14	—	`file_change` (✓ ✓)
15	—	`file_change` (✓ ✓)
16	—	`file_change` (✓ ✓)
17	—	`file_change` (✓ ✓)
18	—	✓ ✓ `/bin/zsh -lc "sed -n '32,54p' zrpc/server.go"`
19	—	`file_change` (✓ ✓)
20	—	✓ ✓ `/bin/zsh -lc 'cat zrpc/internal/target.go'`
21	—	`file_change` (✓ ✓)
22	—	`file_change` (✓ ✓)
23	—	`file_change` (✓ ✓)
24	—	✓ ✓ `/bin/zsh -lc 'gofmt -w core/discov/config.go core/discov/internal/registry.go core/discov/subscriber.go core/discov/publisher.go zrpc/internal/rpcpubserver.go zrpc/internal/target.go zrpc/client.go zrpc/internal/resolver/discovbuilder.go && \ git diff --stat && \ git diff --check && \ git status --short'`
25	—	✓ ✓ `/bin/zsh -lc 'git diff -- core/discov/internal/registry.go core/discov/subscriber.go zrpc/internal/target.go'`
26	—	`file_change` (✓ ✓)

Baseline vs Semedit Agentic in Large Context

#	Vanilla	Semedit MCP
1	not published	not published

Benchmark run luna6-multiswe-go-hints-smoke-20261007-r1 Benchmark run opencode-qwen36-iq4-xs-hard-20261004-1635

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'dns' -g 'roundrobin' \| head -100 && \ git status --short && \ rg -n \"WithBalancerName\|re-resolv\|ResolveNow\|round_robin\|roundrobin\""`	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'dns' -g 'roundrobin' -g 'go.mod' \| head -80 && \ git status --short --branch"`
2	✓ ✓ `/bin/zsh -lc "find .. -name AGENTS.md -o -name PROJECT_BRIEF.md \| head -30 && \ sed -n '1,240p' internal/resolver/dns/dns_resolver.go && \ sed -n '1,210p' resolver_conn_wrapper.go && \ sed -n '1,150p' balancer/roundrobin/roundrobin.go"`	✓ ✓ `/bin/zsh -lc "cat go.mod \| head -10; rg -n \"3165\|ResolveNow\|MinResolution\|resolveNow\|ResolveNow\" resolver balancer internal \| head -160; sed -n '1,260p' resolver/dns/dns_resolver.go; sed -n '1,240p' balancer/roundrobin/roundrobin.go"`
3	✓ ✓ `/bin/zsh -lc 'rg -n "ResolveNow\|TransientFailure\|Connect.Error\|UpdateSubConnState\|Report.Error" balancer/base balancer_conn_wrappers.go clientconn.go; rg -n "minDNSResRate\|watcher\|ResolveNow" internal/resolver/dns/dns_resolver_test.go; git log -5 --oneline --all -- . \| head -10'`	✓ ✓ `/bin/zsh -lc "sed -n '1,290p' internal/resolver/dns/dns_resolver.go; rg -n \"ResolveNow\|ExitIdle\|connection.*error\|TransientFailure\" balancer/base clientconn.go"`
4	✓ ✓ `/bin/zsh -lc "sed -n '160,235p' balancer/base/balancer.go; sed -n '1060,1160p' clientconn.go; sed -n '1,120p' resolver/dns/dns_resolver.go; git log --all --oneline --grep='DNS.re-resolve\\\|ResolveNow.round\\\|round_robin.*DNS' \| head"`	? ✗ `/bin/zsh -lc "sed -n '380,430p' balancer/balancer.go; sed -n '180,250p' balancer/base/balancer.go; rg -n \"ResolveNow\\(\" balancer/base"`
5	? ✗ `/bin/zsh -lc 'git show 3165 --stat --oneline 2>/dev/null \|\| \ true; git log --oneline --all -- internal/resolver/dns/dns_resolver.go \| head -20; rg -n "minDNSResRate\|DNS.*refresh\|periodic\|re-resolv\|re-resolve" doc internal/resolver/dns naming'`	✓ ✓ `/bin/zsh -lc 'git log --oneline -8; git log --oneline --all -- internal/resolver/dns/dns_resolver.go \| head'`
6	? ✗ `/bin/zsh -lc 'git show 9b970fdd -- internal/resolver/dns/dns_resolver.go internal/resolver/dns/dns_resolver_test.go'`	✓ ✓ `/bin/zsh -lc "rg -n \"ResolveNow\|resolveNow\" balancer/base clientconn.go; git show --stat --oneline 648cf9b0; git log --oneline --all --grep='3165' --grep='re-resolv' \| head -20"`
7	`file_change` (✓ ✓)	✓ ✓ `/bin/zsh -lc "git show --stat e5e980f2 && \ git show e5e980f2 -- internal/resolver/dns/dns_resolver.go \| head -160; sed -n '1,130p' balancer/base/balancer.go"`
8	✓ ✓ `/bin/zsh -lc 'git diff -- internal/resolver/dns/dns_resolver.go && \ git status --short'`	? ✗ `/bin/zsh -lc "sed -n '1080,1110p' clientconn.go; git show e5e980f2"'^:internal/resolver/dns/dns_resolver.go 2>/dev/null \| rg -n "ticker\|ResolveNow\|watcher"'`

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'logx' -g '*.go' \| head -120"`	✓ ✓ `/bin/zsh -lc "pwd && \ rg --files -g 'AGENTS.md' -g 'PROJECT_BRIEF.md' -g 'logx' -g '*.go' \| head -100"`
2	? ✗ `/bin/zsh -lc 'cat PROJECT_BRIEF.md 2>/dev/null; find . -name AGENTS.md -print; rg -n "package logx\|type LogConf\|Add.Field\|With.Field\|serviceName" logx'`	? ✗ `/bin/zsh -lc "find .. -name AGENTS.md -o -name PROJECT_BRIEF.md; rg -n \"package logx\|type .*Logger\|WithFields\|Fields\\(\" logx"`
3	✓ ✓ `/bin/zsh -lc ls`	✓ ✓ `/bin/zsh -lc "rg --files \| rg '("'^\|/)logx\|PROJECT_BRIEF\|AGENTS.md'"' \| head -80"`
4	✓ ✓ `/bin/zsh -lc "rg --files \| rg 'log\|logger' \| head -100; rg -n \"package logx\|type LogConf\|func (With\|Add\|Set).*Field\|logx\\.\""`	✓ ✓ `/bin/zsh -lc 'cat core/logx/config.go && \ cat core/logx/fields.go && \ cat core/logx/logger.go && \ rg -n "WithFields\|fields\|ServiceName" core/logx'`
5	✓ ✓ `/bin/zsh -lc "cat core/logx/config.go; cat core/logx/fields.go; sed -n '1,240p' core/logx/logs.go; sed -n '1,220p' core/logx/logger.go; sed -n '1,160p' core/logx/vars.go"`	✓ ✓ `/bin/zsh -lc "sed -n '1,220p' core/logx/logs.go; sed -n '220,470p' core/logx/logs.go; sed -n '1,180p' core/logx/richlogger.go; sed -n '1,100p' core/logx/vars.go; sed -n '1,100p' core/logx/readme-cn.md"`
6	✓ ✓ `/bin/zsh -lc "sed -n '220,520p' core/logx/logs.go; rg -n 'func.write\|new.Logger\|LogField\|fields' core/logx/richlogger.go core/logx/lesslogger.go core/logx/logwriter.go; sed -n '1,180p' core/logx/readme-cn.md"`	✓ ✓ `/bin/zsh -lc "sed -n '15,45p' core/logx/logs.go"`
7	✓ ✓ `/bin/zsh -lc "sed -n '1,180p' core/logx/richlogger.go; sed -n '1,220p' core/logx/writer.go; rg -n 'func.(Debug\|Info\|Error\|Stat\|Severe\|Stack)' core/logx/writer*"`	`file_change` (✓ ✓)
8	✓ ✓ `/bin/zsh -lc "rg -n 'func output\|range fields\|logEntry' core/logx"`	`file_change` (✓ ✓)
9	✓ ✓ `/bin/zsh -lc "sed -n '250,310p' core/logx/writer.go; sed -n '1,45p' core/logx/logs_test.go"`	✓ ✓ `/bin/zsh -lc "rg -n '"'^##\|ServiceName'"' core/logx/readme.md"`
10	✓ ✓ `/bin/zsh -lc "sed -n '20,48p' core/logx/logs.go"`	`file_change` (✓ ✓)
11	`file_change` (✓ ✓)	`file_change` (✓ ✓)
12	`file_change` (✓ ✓)	✓ ✓ `/bin/zsh -lc 'gofmt -w core/logx/logs.go core/logx/richlogger.go && \ git diff -- core/logx/logs.go core/logx/richlogger.go core/logx/readme.md core/logx/readme-cn.md'`
13	`file_change` (✓ ✓)	✓ ✓ `/bin/zsh -lc "rg -n 'SetUp\\(\|MustSetup\\(' --glob '*_test.go' \| head -30; git status --short"`
14	✓ ✓ `/bin/zsh -lc 'gofmt -w core/logx/logs.go core/logx/writer.go && \ git diff -- core/logx/logs.go core/logx/writer.go core/logx/readme-cn.md && \ git status --short'`	—

#	Vanilla	Semedit MCP
1	✓ ✓ `/bin/zsh -lc 'rg -n "type EtcdConf\|EtcdConf\|etcdserver\|clientv3.Config\|Username\|Password"'`	✓ ✓ `/bin/zsh -lc "pwd && \ rg -n \"type EtcdConf\|EtcdConf\|discov\" --glob '*.go' \| head -200 && \ rg --files -g 'PROJECT_BRIEF.md' -g 'AGENTS.md' -g 'go.mod'"`
2	✓ ✓ `/bin/zsh -lc "cat PROJECT_BRIEF.md 2>/dev/null; cat core/discov/config.go; sed -n '250,340p' core/discov/internal/registry.go; cat core/discov/config_test.go; rg -n \"EtcdConf\" --glob '.yaml' --glob '.yml' --glob '.json' --glob '.go' \| head -70"`	✓ ✓ `/bin/zsh -lc 'cat core/discov/config.go && \ rg -n "NewPublisher\|NewSubscriber\|EtcdConf\|etcd.New\|clientv3.Config\|Endpoints" core/discov core/discov/internal'`
3	✓ ✓ `/bin/zsh -lc "rg -n \"DialClient\\(\|NewEtcd\|etcdClient\|Hosts\" core/discov; rg -n \"DialClient\\(\""`	✓ ✓ `/bin/zsh -lc "sed -n '260,340p' core/discov/internal/registry.go && \ sed -n '1,180p' core/discov/internal/registry.go && \ cat core/discov/publisher.go && \ cat core/discov/subscriber.go && \ cat core/discov/config_test.go"`
4	✓ ✓ `/bin/zsh -lc "rg -n \"DialClient\|clientv3.New\|EtcdConf\" core/discov --glob '"'!*/test*'"';"`	✓ ✓ `/bin/zsh -lc "cat zrpc/internal/resolver/discovbuilder.go; cat zrpc/config.go; rg -n \"DialClient\\(\|GetConn\\(\|\\.Monitor\\(\" --glob '*.go' core zrpc"`
5	✓ ✓ `/bin/zsh -lc "cat core/discov/vars.go; rg -n \"NewClient\\(\" core/discov"`	? ✗ `/bin/zsh -lc 'cat zrpc/client.go; cat zrpc/internal/rpcpubserver.go && \ cat core/discov/internal/clients.go 2>/dev/null'`
6	✓ ✓ `/bin/zsh -lc 'rg -n "NewClient\|type cluster\|func .*Etcd" core/discov/internal'`	✓ ✓ `/bin/zsh -lc "cat zrpc/internal/target.go; rg -n \"NewRpcPubServer\\(\""`
7	? ✗ `/bin/zsh -lc "sed -n '1,125p' core/discov/internal/registry.go; rg -n \"internal.NewRegistry\|NewRegistry\\(\" core/discov"`	✓ ✓ `/bin/zsh -lc ''`
8	? ✗ `/bin/zsh -lc "rg -n \"GetConn\|Monitor\\(\" core/discov --glob '*.go' \| tail -20; cat core/discov/discov.go 2>/dev/null"`	`file_change` (✓ ✓)
9	✓ ✓ `/bin/zsh -lc 'cat core/discov/publisher.go; cat core/discov/subscriber.go'`	`file_change` (✓ ✓)
10	✓ ✓ `/bin/zsh -lc 'rg -n "NewPublisher\|NewSubscriber" zrpc core'`	✓ ✓ `/bin/zsh -lc "rg -n 'DialClient\|newClient' core/discov/internal/registry.go"`
11	✓ ✓ `/bin/zsh -lc "cat zrpc/internal/rpcpubserver.go; cat zrpc/internal/resolver/discovbuilder.go; sed -n '1,100p' zrpc/config.go; rg -n \"Register\\(\|NewPublisher\" zrpc/internal"`	✓ ✓ `/bin/zsh -lc "sed -n '235,260p' core/discov/internal/registry.go"`
12	✓ ✓ `/bin/zsh -lc "rg -n \"Etcd\\.Hosts\|Etcd.Key\|DiscovScheme\" zrpc"`	✓ ✓ `/bin/zsh -lc "rg -n 'NewClient' core/discov/internal"`
13	✓ ✓ `/bin/zsh -lc "sed -n '55,90p' zrpc/client.go"`	`file_change` (✓ ✓)
14	✓ ✓ `/bin/zsh -lc "sed -n '1,55p' core/discov/internal/vars.go; sed -n '1,50p' zrpc/internal/target.go"`	`file_change` (✓ ✓)
15	✓ ✓ `/bin/zsh -lc ''`	`file_change` (✓ ✓)
16	—	`file_change` (✓ ✓)
17	—	`file_change` (✓ ✓)
18	—	✓ ✓ `/bin/zsh -lc 'gofmt -w core/discov/config.go core/discov/internal/registry.go core/discov/publisher.go core/discov/subscriber.go zrpc/server.go zrpc/internal/rpcpubserver.go zrpc/internal/target.go zrpc/client.go zrpc/internal/resolver/discovbuilder.go && \ git diff --stat && \ git diff -- core/discov/internal/registry.go zrpc/internal/resolver/discovbuilder.go'`

Benchmark run luna6-multiswe-go-smoke-20261007-r3

Interactive browser

Baseline vs semedit

Detailed observations

Benchmark Methodology & Transparency

Test case: grpc__grpc-go-2735

Target: codex/gpt-6-luna/high

default prompt · none MCP instructions · write semedit restriction

Use case(s) - what problem will this feature solve?

Proposed Solution

Target: codex/gpt-6-luna/medium

default prompt · none MCP instructions · write semedit restriction

Use case(s) - what problem will this feature solve?

Proposed Solution

default prompt · none MCP instructions · write semedit restriction

Use case(s) - what problem will this feature solve?

Proposed Solution

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

Test case: grpc__grpc-go-2951

Target: codex/gpt-6-luna/medium

default prompt · none MCP instructions · write semedit restriction

default prompt · none MCP instructions · write semedit restriction

default prompt · none MCP instructions · write semedit restriction

Test case: grpc__grpc-go-3361

Target: codex/gpt-6-luna/high

default prompt · none MCP instructions · write semedit restriction

What version of gRPC are you using?

What version of Go are you using (go version)?

What operating system (Linux, Windows, …) and version?

What did you do?

What did you expect to see?

What did you see instead?

Additional

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · none MCP instructions · write semedit restriction

What version of gRPC are you using?

What version of Go are you using (go version)?

What operating system (Linux, Windows, …) and version?

What did you do?

What did you expect to see?

What did you see instead?

Additional

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · none MCP instructions · write semedit restriction

What version of gRPC are you using?

What version of Go are you using (go version)?

What operating system (Linux, Windows, …) and version?

What did you do?

What did you expect to see?

What did you see instead?

Additional

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

Target: codex/gpt-6-luna/medium

default prompt · none MCP instructions · write semedit restriction

What version of gRPC are you using?

What version of Go are you using (go version)?

What operating system (Linux, Windows, …) and version?

What did you do?

What did you expect to see?

What did you see instead?

Additional

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · none MCP instructions · write semedit restriction

What version of gRPC are you using?

What version of Go are you using (go version)?

What operating system (Linux, Windows, …) and version?

What did you do?

What did you expect to see?

What did you see instead?

Additional

Baseline vs Semedit Agentic in Small Context

Baseline vs Semedit Agentic in Large Context

default prompt · none MCP instructions · write semedit restriction

What version of gRPC are you using?

What version of Go are you using (go version)?

Test case: `grpc__grpc-go-2735`

Target: `codex/gpt-6-luna/high`

Target: `codex/gpt-6-luna/medium`

Test case: `grpc__grpc-go-2951`

Target: `codex/gpt-6-luna/medium`

Test case: `grpc__grpc-go-3361`

Target: `codex/gpt-6-luna/high`

What version of Go are you using (`go version`)?

What version of Go are you using (`go version`)?

What version of Go are you using (`go version`)?

Target: `codex/gpt-6-luna/medium`

What version of Go are you using (`go version`)?

What version of Go are you using (`go version`)?

What version of Go are you using (`go version`)?

Test case: `zeromicro__go-zero-2463`

Target: `codex/gpt-6-luna/high`

Target: `codex/gpt-6-luna/medium`

Test case: `zeromicro__go-zero-964`

Target: `codex/gpt-6-luna/high`

Target: `codex/gpt-6-luna/medium`