Use repository rules and a hold-out benchmark to review agent patches

A coding agent can pass every test it can see and still break a repository-specific compatibility expectation. For maintainers, the question is whether repository rules and a workload withheld from the agent support accepting the change.

On 13 October 2026, Google described an overnight optimisation exercise on its Agent Development Kit (ADK), an open-source agent framework. Its Python repository has roughly 205,000 lines across nearly 800 files and nearly 17,000 tests. Google says the agent proposed six optimisations, of which maintainers shipped two after review. This is one vendor-reported case, not an independent comparison of coding agents or a prediction of results in another codebase. (Google Developers Blog, 13 October 2026)

A passing suite can miss the repository’s contract

For the second run, the ADK team put maintainer rules in repository guidance files. These covered conventions such as private module visibility and lazy imports. The agent read relevant guidance before editing and ran the open-source test suite. Yet one proposed import change passed all 16,892 open-source tests while failing 25 internal Google test targets. Those tests patched attributes at the import location the change had removed. Google revised the shipped approach so heavy modules load when first needed while the expected attributes remain patchable. (Google’s account of the import change and test result)

That result does not make public tests inadequate. It shows the published suite did not cover every compatibility expectation held by ADK maintainers. Tests show only the cases they exercise. Repository rules describe conventions, while targeted tests check whether a change preserves them.

Across modules, an edit can preserve public signatures yet change where consumers find an attribute, where mocks must replace it, or when a dependency loads. Owners should name and test supported consumer contracts. Otherwise neither an agent nor reviewer can reliably check them.

Withhold one workload from the change author

Google also describes a benchmark written before either coding-agent run and withheld from the agent. To isolate ADK’s own overhead, the benchmark used an instant model and tools that returned immediately. For the comparison, Google reports byte-identical requests to the model before and after the code change and medians over 6–12 runs. These controls make the reported comparison more inspectable within that setup. They do not provide an external replication of the result. (Google Developers Blog benchmark method) The merged ADK commit and benchmark results provide the change record.

The gains applied to particular workloads. Google reports that importing google.adk became 21% faster, while a new process’s time to its first model request stayed effectively flat at 3.07 seconds before and 3.09 seconds after. The heavy modules still load on first use. It also reports about 40% less preparation overhead for long in-memory sessions. Across one 158-turn interaction, the full run fell from 65.6 to 56.1 seconds. That result concerns ADK’s in-memory session service with an instant model and tools. The post says external database, SQLite, Gemini platform, Firestore and Redis session stores do not receive the same copying change. It is not a general 40% reduction in coding-agent runtime or a forecast of savings for another organisation. (Google’s workload scope and results) The commit’s benchmark summary records the same medians.

The hold-out benchmark helped distinguish a repeatable change in the tested framework overhead from an improvement that appeared only in a benchmark the agent had constructed. It did not measure a real model’s response time, because the no-op model deliberately removed that delay. A team should choose a test according to the decision: isolate framework overhead when that is the question, then measure the intended service path when deciding whether users will see a benefit.

Review the proposals that do not survive

Four of the six proposed changes were rejected. Google reports that one cache could return stale data if Python reused an object’s memory address after collection. A telemetry optimisation read a private OpenTelemetry attribute that might change across a library update. Another rewrite made a central path harder to read without improving the hold-out latency. A workflow replay index helped synthetic tests but not everyday agent runs. (Google’s description of the four rejected proposals)

These decisions illustrate why generated tests and benchmark gains should enter review as evidence, not as approval. The maintainers considered state lifetime, dependency contracts, complexity and whether the measured workload represented ordinary use. The merged change also added 17 test functions, according to the published account. The agent supplied proposals and code. Repository owners decided which trade-offs the project would accept. (Google Developers Blog)

Define acceptance evidence before delegating

For a proposed coding-agent change in an existing repository, the owner should set the compatibility conditions, representative workload and review decision before the agent starts:

illustrative fixture

Review an agent patch against independent evidence

What evidence should decide whether an agent-written patch ships?

Fixture

An existing repository, a proposed performance change, compatibility expectations and a representative workload kept from the agent.

Procedure

1. Record the supported interfaces, import locations, state behaviour and dependency rules that the change could affect.
2. Set the baseline and acceptance rule before the agent starts, then retain the public tests and agent-produced benchmark as exploratory evidence.
3. Run the candidate against a representative workload the agent could not inspect or change, keeping inputs and comparison conditions fixed.
4. Review the diff for callers, mocks, state lifetime, dependency contracts, complexity and failure cases.

Result to check

Proceed only when compatibility checks pass, the pre-set workload supports the change and the repository owner accepts the remaining trade-offs.

Evidence limit

This method is inferred from one Google-authored case. Its withheld benchmark isolates ADK overhead with an instant model and tools, not real model response time or results in another codebase.

Next test

Measure the intended service path with its actual model and tools before claiming a user-visible improvement.

The ADK report compares two runs with several changes to the instructions, run duration and task setup. It therefore does not isolate how much repository guidance alone changed the outcome. Nor does the internal test result show that every organisation needs Google’s private test suite. The practical lesson is narrower: teams should make important compatibility expectations testable and keep the acceptance workload outside the change author’s control.

The repository owner should define acceptance evidence before delegation. If compatibility checks or a representative hold-out are missing, establish them before asking an agent to change code.