Test the questions your documents cannot answer

A policy assistant’s successful answers establish only part of its service capability. The acceptance review also needs questions that the approved guidance cannot answer. The service owner must know what happens to those requests before relying on the assistant to absorb help-desk work.

Consider an illustrative appeal question where the policy gives a deadline only for the original claim. Reusing that deadline would produce a plausible but unsupported instruction. Test that distinction explicitly. The service implications include correction work, repeated contacts and a handoff to the person authorised to settle the missing guidance.

Evaluate answerability against the evidence the requester is permitted to receive. Report unsupported answers, supported answers and unnecessary abstentions separately. Connect each outcome to the work it creates for the service rather than converting a combined score into a staffing assumption.

Make the missing fact plausible

Our earlier article on the evaluation set already calls for refusal cases. The extension here is to define what makes a question answerable from a particular evidence set, and deliberately change that evidence while holding the question fixed. A general instruction to include awkward questions leaves this distinction to whoever assembled the test.

The SQuAD unanswerable-question paper constructed questions that remained relevant to their passages while having no supported answer. Plausible distractors made the task harder than recognising an unrelated question. The study concerns extractive reading comprehension and the models it examined. It does not establish a current error rate for an enterprise assistant or a recommended proportion of unsupported questions.

Ask the policy owner to adjudicate each expected answer against the approved material. Record its revision, effective context and requester entitlement. Distinguish an established negative answer from absent evidence: a source saying a benefit is unavailable can support a clear answer, even when the requester would prefer another outcome.

Keep missing evidence separate from access denial, prohibited requests and conflicting guidance. They may lead to different messages and different owners. An assistant must not reveal restricted material while explaining its refusal. The retrieval access-control boundary remains a prerequisite for any answerability test.

Change the evidence and keep the question

Use a constructed policy exercise before the substantive acceptance run. A fictional passage says expense claims must arrive before the next payroll cut-off. Another describes permitted hotel categories. Neither establishes an appeal deadline or a meal allowance. These are invented test materials, not advice about an employer’s policy.

Ask when to submit an expense claim. The expected response follows the claim passage. Ask when to appeal a rejected claim. The expected response states that the approved material does not establish that deadline and follows the agreed handoff. Reusing the submission deadline would answer a different question. Ask about meals and check that the hotel passage does not become its authority.

Now remove the answering passage and repeat the original claim question. The assistant should stop giving the deadline. Restore the passage and repeat again. It should resume giving the supported answer. The paired change exposes both an unsupported answer and an assistant that has learned to decline regardless of available evidence.

This exercise specifies expected behaviour. A dictionary-presence rehearsal can check that the fixture marks the intended evidence as available or absent. It cannot measure a language model, validate retrieval or establish service quality. The real acceptance run must execute the actual system, preserve its retrieved passages and have the outputs adjudicated against those passages.

Separate a missing source from a retrieval miss

Keep a corpus-level label and a supplied-context label. The permitted policy collection may contain the answer while the retrieval step fails to return it. Alternatively, retrieval may work perfectly against a collection that never contained the fact. Both situations require the assistant to withhold an unsupported instruction, but their repairs differ.

The first directs attention to retrieval, filtering or document processing. The second needs a policy decision, new approved evidence or a service handoff. Buying a larger model does not supply the missing organisational decision. Record the relevant trace so the service owner can assign the work to the right team instead of asking for an undifferentiated accuracy improvement.

Check the actual support behind an answer. The ALCE citation research measures correctness and citation quality separately. Its distinction is useful here: a citation marker does not itself establish support for every instruction in a response. Its automated measures and experimental datasets do not provide a universal acceptance threshold for internal policy advice.

Give the service owner separate outcomes

Give the service owner separate counts and denominators for supported correct answers, wrong answers despite available support, unsupported answers and unnecessary abstentions. An assistant that declines every request can avoid unsupported instructions while leaving nearly all the work with staff. Acceptance needs both evidence discipline and useful service coverage.

Deliberately difficult cases are useful for finding failures. Their share of an acceptance set need not match live demand. Before estimating queue effects, measure the frequency of relevant request types in the intended service and keep that operating view separate from the stress-test view. Mark contested interpretations for review rather than manufacturing a clean expected answer.

illustrative fixture

An answer needs evidence as well as a valid shape

When must an employee appeal a rejected expense claim?

Text responseTyped decision

Swipe or scroll for the full diagram →

{"action":"handoff","reason":"appeal_deadline_not_established"}

Eligibility

The permitted passages must establish the appeal deadline, rather than only the original claim deadline.

Confidence

A confidence score or a valid JSON object does not supply the missing policy fact.

Bounded route

Route the unresolved question to the policy owner through the agreed service handoff.

Abstention

State that the approved material does not establish the appeal deadline.

Evidence limit

Fictional policy example. The interaction illustrates a decision gate and does not call or evaluate a language model.

illustrative fixture

Accept answers and handoffs separately

Baseline

Current resolution, repeat contacts and staff effort for the same request types.

Outcome

Supported resolutions and appropriate handoffs, reported by evidence availability.

Guardrails

Unsupported instructions, unnecessary abstention, queue delay and correction work.

Decision rule

Approve a bounded scope only when answer quality and handoff capacity meet the service owner's agreed conditions.

Decision owner

Policy and service owners

Population

The permitted evidence and request types proposed for release.

Agree who receives an unresolved request, what information accompanies it and what response the employee should expect. Measure review effort, repeated contacts and time to resolution in a bounded trial alongside answer support. A well-founded refusal still consumes operational capacity when somebody must resolve the underlying question.

Approve only request types with demonstrated support and a workable handoff. Assign missing guidance to its policy owner and keep unresolved uses outside the release scope. Record the decision and the conditions for expanding it after further evidence becomes available.