Randomise the capacity pool before testing the AI

Consider a constructed appointment-booking trial. Customers use either an AI assistant or the existing page, but both draw on the same calendar. Assisted customers may book desirable slots sooner, leaving fewer options for others even if the number of completed appointments remains unchanged.

The assistant may improve its users’ access to scarce capacity without increasing total work sold. When treatment changes what control can obtain, customer-level randomisation alone does not identify the effect of giving the policy to everyone. The experiment needs to account for the shared resource.

This example is illustrative, not a client result. The buying decision concerns whether a customer-level result is sufficient or whether the trial needs assignment at the capacity-pool level. Similar questions arise when policies allocate stock, delivery capacity or specialist staff.

A holdout can inherit the treatment

The earlier personalisation article asks buyers to retain a customer holdout and measure incremental margin. That remains useful when one customer’s treatment leaves another’s opportunity substantially unchanged. Here the treatment can alter that opportunity directly. A faster booking consumes a slot. A different dispatch policy changes which worker is available for the next request.

Researchers tested this measurement problem in an Airbnb pricing experiment. They compared individual listing assignment with assignment of groups of similar listings. Their joint analysis found that grouping reduced interference bias in the estimated booking effect. The intervention changed platform fees, rather than introducing an AI assistant. It provides evidence that experiment boundaries can change the estimated effect in a connected market, not a forecast of gains for another business. Holtz and colleagues, published in Management Science

Interference can also spread benefits into control. DoorDash’s engineering account describes customers sharing a delivery fleet: changing demand for some customers can affect service for the rest. Its team used randomised regional time windows to compare dispatch policies. This is an account of that team’s implementation, not a universal recipe for enterprise experiments. DoorDash’s switchback design

The direction therefore needs investigation. Competition can make a treatment look better by worsening control. Shared improvements can make a useful policy look weaker because control benefits too. A narrow confidence interval does not settle either problem if the comparison answers the wrong rollout question.

Follow a booking through the shared calendar

Before choosing a test, run a small deterministic exercise with the booking team. Give the assistant and incumbent page access to the same test calendar. Record the slots visible to an untreated customer. Let an assisted customer reserve a desirable slot, then repeat the untreated customer’s search. The change in available choices demonstrates a path by which treatment can affect control. It does not measure how often that path matters in production.

Repeat with separate calendars and with ample spare capacity. The purpose is to distinguish the existence of competition from its likely commercial size. Review historical availability, abandoned searches and requests that moved to another branch or date. An apparently separate branch is a poor boundary if customers routinely substitute across it or staff are moved between branches.

Write the intended outcome before selecting the randomisation unit. For this business it could be contribution from completed appointments across a capacity pool, with cancellations, waiting time and unmet demand alongside it. Count customers who could not book. Measuring only successful bookings discards the people most likely to reveal displacement.

Unchanged appointment volume does not rule out value. The policy might reduce cancellations, change the work mix or lower handling cost. Treat these as separate hypotheses and measure them. The design should distinguish such benefits from a change in which customers secure the same slots.

Choose a boundary the service can defend

Where substitution is mostly local, consider assigning entire capacity pools to the candidate or incumbent policy. Grouping branches by administrative region is insufficient on its own. Use search, booking and staff-allocation records to show why competition mostly stays inside the proposed group, and record the cross-boundary traffic that remains. The analyst must account for assignment by group when calculating uncertainty.

Where a resource can return to a comparable state, a switchback can be useful: apply one policy to the whole pool during a randomised time block, then use the other in later assigned blocks. Agree the schedule before seeing outcomes and account for recurring demand patterns. A simple before-and-after comparison can confuse the policy with a staffing change or a seasonal peak.

The difficult condition is persistence. An appointment booked during treatment may occupy capacity far into a control period. Switching the interface does not reset the calendar. Research on switchback design and analysis explicitly models how long a treatment continues to affect later outcomes. Its design results depend on those assumptions. They do not supply a safe switching interval for an appointment business.

For long-lived reservations, separate pools may be more credible than rapid switching. If the business cannot identify enough independent pools, or cannot wait for effects to clear, the buyer may need a narrower claim or a different trial. A larger count of individual customer visits cannot repair a missing independent comparison.

Baseline

The incumbent booking policy, with demand, available appointments, completed work and cancellations recorded by capacity pool.

Outcome

Contribution from completed appointments across each assigned pool and observation window, including operating costs.

Guardrails

Unmet demand, waiting time, cancellations, staff load, cross-pool substitution and reservations carried into later periods.

Decision rule

Expand only when a comparison that accounts for shared capacity supports an economically worthwhile gain without breaching service limits. Redesign the test if displacement or carryover makes the rollout effect unresolved.

Put the experiment design in the buying decision

The evidence pack should contain the resource map, assignment rule, treatment exposure, available capacity at each decision, completed outcomes and unresolved spillovers. Ask the analyst to estimate uncertainty using the actual assignment structure and assess whether the proposed run can distinguish a worthwhile change. Agree how to handle outages and incomplete periods before launch.

The cited evidence concerns marketplace experiments and statistical design. Applying it to an enterprise booking assistant is an inference from the shared capacity mechanism. Neither a rehearsal nor a favourable trial proves a lasting return after competitors, customers or staffing respond to a full rollout.

Approve a trial whose assignment and outcome match the investment decision. For a calendar-based service, follow available capacity and completed work across the relevant pool. Report unresolved displacement and carryover before using a treatment-group gain to justify expansion.