Hold the model constant when evaluating geospatial agents

A geospatial agent should earn its place by improving a real analysis workflow, not by producing a convincing map or a fluent explanation. A recent Google Research preprint offers buyers a useful way to separate two sources of capability: specialist tools that find and process geospatial data, and the orchestration layer that plans and executes multi-step work with those tools.

The distinction matters when an organisation considers a domain-specific agent for work such as flood exposure screening, renewable-energy site selection or monitoring land-use change. The choice is not simply whether a language model can answer a question. It is whether the specialist data and analysis tools improve the work enough to justify the agent layer, its operating cost and the review it still needs.

Hold the model constant and vary the system

The Google Research authors’ arXiv preprint, dated 11 October 2026, evaluates its Geospatial Reasoning Agent (GRA), a Gemini-powered research system that combines geospatial datasets, models and tools behind a natural-language interface. The paper describes 23 loaded skills and 62 distinct tools in the evaluated runs. Its architecture uses upfront planning, on-demand skill loading and external tools to work with data that does not fit in the model’s context. (Google Research preprint, sections 3.1 and 3.2)

To attribute performance, the authors compare four configurations with the same underlying Gemini model: a standalone model, the model with basic search, maps and code tools, the same model with GRA’s geospatial skills and tools but without its planning and execution guidance, and full GRA. They repeat the comparison with Gemini 3.1 Pro and Gemini 3.8 Flash. GeoExpertQueries has 76 test tasks withheld from the agent. For each query, three independent geospatial experts, blind to GRA’s tools, wrote workflows. Four queries had incompatible expert approaches, so the paper retains multiple alternatives and scores a response against the closest one. A human-validated, tool-agnostic autorater assesses whether a system follows a sound workflow without requiring a fixed sequence of tool calls. (Google Research preprint, sections 3.2 and 4.1)

For Gemini 3.8 Flash, adding GRA’s specialist tools to the basic-tools configuration raised the mean workflow score from 3.3 to 52.2 out of 100, with full GRA at 62.7. The paper reports standard deviations of 0.8, 5.4 and 2.5 points respectively across three runs. With Gemini 3.1 Pro, the means were 2.3 for basic tools, 50.8 for specialist tools and 47.6 for full GRA, with standard deviations of 0.8, 1.7 and 2.7. These author-reported results suggest that specialist geospatial capabilities account for the larger gain in this comparison, while planning and execution guidance add value on the open-ended tasks for one tested model and not the other. (Google Research preprint, section 5.2)

The pattern changed on GeoKTools and CBGB, where each query has a more determinate answer. GeoKTools uses query templates designed around operations supported by GRA’s tools, which favours systems with those tools. The tools-only and full-agent configurations were within a few points, and the tools-only system sometimes scored slightly higher. Read this benchmark as a test of multi-tool orchestration, not neutral evidence that GRA’s tool suite covers every buyer’s needs. A buyer should ask which system layer changes the task at hand, rather than assume a more autonomous harness will always improve it. (Google Research preprint, sections 4.2 and 6.2)

Score the method as well as the answer

For an estate portfolio, a hypothetical pilot might ask which sites warrant a closer flood or heat assessment. An answer listing sites is not enough. The reviewer needs to know which hazard and population datasets were used, how boundaries were resolved, whether the analysis used current observations or projections, what was omitted, and how uncertain locations should be treated. Those checks determine whether the output supports triage or could be mistaken for a complete assessment.

The Google benchmark gives workflow alignment an explicit score, which is valuable when tasks allow several defensible analyses. But the paper states that matching workflow steps establishes process alignment rather than outcome correctness. It treats the system’s responses as first drafts for expert review. A workflow score should therefore sit beside checks on the result itself, such as comparisons with known data, independent calculations or a qualified geospatial analyst’s review. (Google Research paper, section 6.2)

Before a supplier evaluation, the system owner should agree with domain experts on a small set of realistic tasks, required evidence and acceptance thresholds.

illustrative fixture

Set the geospatial agent acceptance rule

Baseline

The same underlying model with search, maps and code tools, or the current human-led workflow.

Outcome

Expert-defined task completion, evidence trail, independently checked result quality and reviewer effort.

Guardrails

Named data sources, valid spatial and temporal assumptions, uncertainty, external data and compute costs, and human review where a result affects a consequential decision.

Decision rule

Fund a bounded pilot only when specialist tools or orchestration improve the buyer-defined workflow enough to meet agreed quality and total operating-cost thresholds.

Decision owner

Geospatial service owner

Population

Tasks and data representative of the proposed service, held out from supplier development.

Then compare the same base model with a general-purpose toolset, the proposed specialist capabilities and the full agent. Keep the test tasks and expected evidence outside the supplier’s development loop. Record the data source, analysis path, transformations, result quality, exceptions and reviewer effort for each run. The owner can then decide whether the extra tools or orchestration support a defined workflow, and where a human decision remains necessary.

Include the cost of the data path

The paper also estimates model-token inference cost. Its figures exclude external data and compute services such as Google Earth Engine, so they are not the cost of running a geospatial service. The buyer’s estimate should include data access, imagery or model inference where charged, geospatial compute, orchestration, monitoring, human review and the cost of correcting a poor analysis. The quoted model cost alone cannot establish whether an agent is economic for a particular workload. (Google Research paper, section 5.6)

The study is a useful design reference, not an independent validation of a commercial product. The authors evaluate Google’s tools with Google’s Gemini models. GeoKTools deliberately varies the number of tool types in queries built for that suite, and the autorater remains part of the score. Although GeoExpertQueries uses experts who were blind to the tool suite and tests tasks withheld from the agent, its benchmark is still a research team’s selection of geospatial work. The paper does not establish how another organisation’s data, workflow or acceptance rules would perform.

The practical decision is to fund specialist geospatial capability only after separating its contribution from the model and the agent harness. Ask the people who will own the analysis to set the tasks, evidence and review threshold, then test each layer against them. If a simpler configuration meets the threshold, the organisation can retain the expert workflow without buying autonomy it does not need. If full orchestration earns its additional cost on the held-out work, it has a clearer case for a bounded pilot.