Write the event before generating its paperwork
A synthetic workplace can test how an assistant reconciles records. Establish the underlying events and expected answers before generating the documents used to score it.
Keep development and test groups separate before comparing performance. No measured results are shown.
A synthetic workplace can supply something a folder of example documents cannot. It can give a meeting, a message and a task record the same underlying history. That makes it possible to test whether an assistant can reconcile records created by different people at different points in the work. For an evaluation team, the design decision is which events must be established before the supporting documents are generated.
Microsoft Research describes generating simulated Microsoft Copilot environments with staff roles, communications and work artefacts that evolve over time. The team uses public information about roles and organisations to inform those environments and generates questions grounded in their contents. Its examples include pharmaceutical, capital markets and software workflows. The work describes an internal evaluation system. The published account gives buyers a design to examine without supplying evidence of their own workflow performance. (Microsoft Research, 30 October 2026)
Start with the event that decides the answer
Consider a hypothetical evaluation for an engineering programme. A supplier proposes moving a delivery date in a message. The planning meeting records the proposed date and assigns someone to check whether it is possible. The scheduling owner later approves a different date in the planning system. The assistant is asked what date the programme is working towards and what happened to the supplier’s proposal.
Generate the messages and notes around that sequence. Write the proposal, assignment and approval into a separate test record first, with their responsible roles and effective times. The evaluation team can then inspect each generated document against the event it is supposed to represent. A meeting note that turns the proposal into an approval is a fault in the fixture unless that conflict was explicitly introduced as part of the test.
Include plausible disagreement where it belongs. The supplier can still expect the proposed date after the owner has changed it. The answer key should explain which source establishes the effective schedule, which source only records a view and what remains uncertain. A test designer needs to know whether a contradiction was deliberately placed in the case or introduced accidentally by generation.
A proposal and an approval have different effects
Swipe or scroll for the full diagram →
Schedule approval: excluded from earlier inputs
The scheduling owner approves a different date. The record appears after the reference time.
Event: 2026-10-28T11:30:00Z. Available: 2026-10-28T14:00:00Z.
- Prediction boundary
- 2026-10-28T12:00:00Z. Availability after this boundary excludes a value from the earlier input, even if its event date is earlier.
- Supplier proposal
- Event 2026-10-28T09:00:00Z; available 2026-10-28T09:00:00Z. A supplier proposes a date in a message. The message establishes the proposal.
- Planning note
- Event 2026-10-28T10:00:00Z; available 2026-10-28T10:30:00Z. The meeting assigns a person to check the proposed date.
- Schedule approval
- Event 2026-10-28T11:30:00Z; available 2026-10-28T14:00:00Z. The scheduling owner approves a different date. The record appears after the reference time.
Hypothetical fixture. At the reference time the approval has occurred but its record has yet to appear in the workspace.
The proposal and planning note are available at the reference time. The later approval has occurred but is not yet available, so the expected answer can report the proposal without asserting approval.
Set an answer expectation before running the assistant
Write an expected result from the event record and the evidence actually exposed in the workspace. In one version, the approved date and the decision record are present. A satisfactory answer gives the date, identifies the superseded proposal and shows where the change was approved. In a second version, remove the approval and all its derived references. A satisfactory answer can report the proposed date and say that it cannot establish approval.
Keeping these paired cases close helps diagnose what changed. If the assistant gives the same approval in both, the tester can inspect whether a stray generated record still implies it or the assistant supplied it without evidence. If the assistant gives a correct result only when handed the relevant records directly, the team can investigate search separately from interpretation. In each case retain enough detail to reproduce the discrepancy.
Use a reviewer who understands the domain workflow. An automated generator that writes the workspace and the question can also produce a plausible answer key based on an invented relationship. A separate review should confirm that the artefacts support the expected outcome, including deliberate ambiguity. Style and realism checks cannot settle which event a particular record establishes.
Keep the fixture stable enough to compare releases
Store the event record, rendered artefacts, expected answer and setup instructions as one versioned fixture. If the generator changes, compare the actual artefacts and their references before replacing the baseline. A repeated instruction to the generator does not establish that a new fixture contains the same evidence. When the intended test changes, keep the earlier case available to explain any difference in results.
This is a construction method for the evaluation evidence the organisation maintains. The archive’s correction article asks how a live search system should retrieve revisions. Here the test designer creates a known sequence and checks that the artefacts preserve it before using the fixture to score the system.
The evaluation owner should start with a consequential event that usually spans several records. Establish the evidence for that event, generate an intact and an incomplete case, and have the workflow owner review both. Use them to compare assistant releases within the simulated workspace. Evidence about the frequency and outcomes of these cases in real work still needs an appropriately scoped service evaluation.