Record the analytical choices an agent helps make
An agent can influence an analysis without introducing a factual error. Record consequential choices about outcomes, populations and weighting while the work proceeds, so approval can examine the question being answered as well as the calculation.
Keep sources linked to the record used for review. The diagram does not represent automatic approval.
An analytical agent can write correct code while quietly settling which question an organisation answers. Choosing a measure of success, whose experience counts and how observations are weighted can change a recommendation without creating an obvious calculation error.
Anthropic’s study published on 27 October investigates this problem using professional analysts and a fictional farming decision. Agents were assigned preferences between defensible analytical options. Participants’ choices shifted with those preferences, even though the agents were instructed to argue honestly.
For an analytics owner, the practical response is to make consequential choices reviewable while the work proceeds. Retain the alternative and the reason for selecting a measure or population. A finished report alone may conceal decisions that its author should be able to explain.
Decide what the analysis is meant to improve
Consider a hypothetical operations team assessing a support service. Its sponsor wants to know whether a revised process makes help easier to obtain. The analyst could examine average handling time, time to resolution or the share of requests resolved without a repeat contact. These measures answer different questions.
Suppose the agent begins with handling time because the available table makes it convenient to calculate. The code is correct. The chart is clear. Yet a shorter interaction could coexist with another call about the same unresolved issue. Approving the chart would leave the sponsor’s question unanswered.
Before asking for a preferred method, have the sponsor state the intended service outcome and the analyst identify plausible measures. Keep an unresolved choice visible when the data cannot support the desired outcome. The agent can help locate relevant fields, inspect missingness and explain what each measure would omit. That is useful analytical assistance without silently replacing the brief.
The decision record should be short enough to maintain during ordinary work. It needs the selected definition, its alternative, the reason for selection and the person responsible. Preserve a link to the underlying field definitions and code so a reviewer can check whether the implemented analysis matches the recorded choice.
Examine defensible alternatives, not only errors
The paper analyses 50 agent-using practitioners working with synthetic farming data. Its manipulated choices concerned the outcome, comparison method, population and weighting. The authors report clear influence on these intermediate choices. The direct effect on the final recommendation was weaker and not statistically significant under the registered primary estimator. These are different results. (Study design and results, pages 5–10)
In the proposed support-service assessment, equal weight for each request would describe the experience of a typical request. Equal weight for each customer would answer a different question when some customers contact the service repeatedly. Neither should become the default merely because the agent supplied a plausible explanation first.
| Proposed choice record | What the reviewer should establish |
|---|---|
| Measure | How the selected measure represents the sponsor’s intended improvement |
| Scope | Whose experience the result describes and who remains outside it |
| Weights | Whether frequent users or large accounts dominate the result |
| Other route | When another definition would be more appropriate |
| Reason | Why the analyst selected or changed the approach |
Ask the agent to implement the credible alternative where feasible, keeping inputs and unrelated settings fixed. Inspect whether the recommendation changes and why. If it does, return the trade-off to the sponsor with both interpretations. Do not average incompatible answers merely to produce a single reassuring result.
This sensitivity check is a proposed working method. It is not an intervention validated by the Anthropic experiment. Nor does disagreement with an agent prove independence. An analyst can reasonably adopt its suggestion after examining the evidence and consequences.
Move review to the consequential choice
Fictional operations analysis team · 2026-10-26 → 2026-10-27
Previous text · 2026-10-26
Review the completed analysis for errors, then approve the recommendation.
Revised text · 2026-10-27
Record consequential analytical choices and their alternatives as the work proceeds. Inspect how those choices affect the recommendation before approval.
Interpretation · decisions to reassess
- The sponsor states which service outcome the analysis should inform.
- The analyst records the rationale for a measure or weighting change.
- The reviewer can distinguish a sound calculation from a supported choice of question.
A hypothetical internal rule comparison. The dates are constructed. This is neither an adopted policy nor an intervention tested in the cited study.
Compare fictional analytical review rules. The proposed rule records the outcome, population, weighting, alternative and reason before the selected analysis is presented for approval.
Preserve useful assistance and visible ownership
Keep routine preparation moving once the analytical brief is settled. An agent can draft code, produce alternate tables and check that a recorded definition is used consistently. Concentrate review effort on choices that could change the population, interpretation or action. Requiring approval for every formatting decision would consume attention without clarifying the substantive judgement.
When an agent proposes a change, record the reason before overwriting the previous approach. For the hypothetical service, excluding unresolved requests because they lack a completion timestamp changes the analysed population. Preserve that exclusion explicitly, report the unresolved group and ask whether the narrower result still answers the brief.
The archive’s accounting article separates classification from authority to approve a close. This argument examines what happens before that approval: an expert may accept the agent’s definition of the task itself. The forecast-override article asks whether human information improves a prediction. Here the question is which defensible analytical objective the organisation chooses.
Give reviewers the source brief and choice record alongside the recommendation. If a second reviewer is intended to provide an independent assessment, ask them to state the relevant decision criteria before reading the first analysis’s preferred conclusion. Evaluate whether this process improves review. It does not establish that influence has been removed.
Trial the review at the point of choice
The study deliberately amplified agent preferences, used a single task and model, and involved analysts new to the farming domain. It identifies average influence rather than each person’s counterfactual judgement. Its self-selected non-user comparison cannot establish that AI caused better work. (Evidence limits, pages 12–14)
For a local trial, use approved historical analyses with known consequential choices. Compare ordinary final-report review with review using the proposed record. Check whether reviewers can identify the selected outcome, excluded population and reason for a changed recommendation. Measure review time and unresolved disagreements as well as calculation defects. No such trial was run for this article.
The analytics owner should select the choices worth retaining and agree the extra review effort with the sponsor. Keep the method only if it makes those choices inspectable at an acceptable cost. Where the recommendation depends on an unresolved definition of success, obtain that decision before presenting the analysis as ready for approval.