Research #001Published September 2026

An AI patient is easy now.
An examiner you can trust isn’t.

Kent TanVibeServe Research·Kent Tan·17 min read

Abstract

Simulating the patient in an OSCE became easy; trusting the marking did not. We describe an architecture where a single model call plays both patient and examiner, the patient is never shown the marking criteria, the examiner’s questions are chosen by code, and the pass/fail verdict is arithmetic over individual judgements rather than a model’s opinion. We then report what we measured when we tried to trust that marking — including a prompt-stability experiment that our own preferred prompt failed.

One model callPatientExaminerjudgementsCode computes the verdictMark schemenever enters the roomcode only

An OSCE — an objective structured clinical examination — is the part of a licensing exam where nobody asks you what the right answer is. You are put in a room with a patient, given a task and a clock, and marked on what you actually did.

Simulating the patient stopped being hard some time in the last two years. Any competent team can put a model behind a chat box and have it cough when you ask about its chest. What is hard is everything the patient must not know, everything the examiner must say and no more, and the part nobody wants to talk about: whether the marking gives the same candidate the same answer twice.

This is how we build that, what we measured, and which of our own ideas the measurements killed.

30
Published stations
609
Marking criteria
451
Of those, essential
209
Red flags
0.7s
Mean turn latency
01

What the exam actually looks like

Two regulators, two shapes, one common property: the thing being measured is behaviour, not recall.

United Kingdom — NMC Test of Competence

Ten stations. Four of them are one scenario taken in order — assessment, planning, implementation, evaluation — so you meet the same patient throughout, and what you found in the assessment is what you are expected to plan and act on. Four are practical skills on manikins and equipment, set as two pairs. Two cover professional values and behaviours, and evidence-based practice.

  • Attempts per application3
  • Minimum wait between sittings10 days
  • After a third fail, reapply in6 months
  • Fee£794 · £397 resit

The four APIE stations are what a conversation can assess, so they are what we simulate. The manikin stations we leave alone.

Canada — NAC Examination

Twelve stations, eleven minutes each, marked against seven competencies rather than a single blueprint of topics. A different exam culture: shorter encounters, a broader sweep of presentations, and feedback that is organised by competency rather than by station.

  • Stations12
  • Minutes per station11
  • Competencies assessed7
  • Our build statusIn progress

An independent third-party tool with no MCC affiliation — a sentence a test asserts verbatim, so nobody can quietly soften it into an implied endorsement.

02

Two people in the room, one model call

The candidate types. One call comes back carrying both the patient’s words and the examiner’s.

A label that runs on is a mis-attributed sentence

Splitting the reply into two speakers sounds trivial until you see a real one. This came back from a probe run: 

PATIENT: ...thank you.EXAMINER: The doctor has been bleeped.

One line. The examiner’s words were stored as something the patient said — and marking reads the transcript, so the candidate was about to be credited for eliciting information they had been handed. The splitter anchors a label to the start of a line, and separately recovers a run-on label only when what precedes it is neither a newline nor an asterisk, so an emphasised *EXAMINER:* is left alone. The probe still reports the formatting fault even though the splitter now recovers from it, because a recovered fault is still a fault.

Why both roles come from one call

Two calls per turn would double the latency the candidate watches with a clock running, and would need a third decision about which of the two speaks first. One call, two labelled lines, and a deterministic splitter is cheaper and easier to reason about — the cost is exactly the run-on above, which is a parsing problem with a parsing fix.

What the examiner may say when the candidate acts is bounded on purpose: a value from the data it was given, or a plain confirmation that the action was done. Never an invented result, never an arrival, never a change in the patient’s condition nobody scripted.

03

The patient never sees the mark scheme

This is the whole difference between a station and a roleplay. Get it wrong and the exam measures nothing.

A patient that knows the criteria is helpful, and helpful is fatal. It says “you haven’t asked about my allergies yet”, the candidate asks, the criterion is met, and the station has just tested the model’s prompting rather than the nurse’s clinical reasoning. So the grid is not redacted from the prompt. It is not a parameter of it. There is nothing to forget to strip out.

Goes into the actor prompt
The scenario
Where the patient is, why, and what the shift has recorded so far.
The patient brief
Who they are, how they speak, what hurts, what they say only if asked, what settles them and what does not.
Clinical data
A set of vitals, plus a findings map the examiner may read from only when the candidate asks for that thing by name.
At most one examiner question
Already chosen by code, with an instruction to ask it in exactly those words.
Not even a parameter
The criteria grid
Not redacted — not a parameter. There is nothing to forget to strip out.
Which criteria are essential
The patient has no idea which omissions are fatal, so it cannot steer you around them.
The red flags
A patient who knows the unsafe acts can avoid provoking them. Then the station stops testing safety.
The examiner question’s model answer
An examiner holding the answer hands over the answer to the question it is asking.
The test that keeps it true

A rule nobody checks is a rule that lasts until the first “just for context”. So a test renders the real prompt and asserts it contains none of criteri, essential, red flag, marking or rubric — and separately that an examiner question’s expected answer never appears either. It is a smell test, not a proof. It fails loudly the moment someone passes the grid in for a good reason, which is the only failure mode that has ever actually happened.

04

Two disclosure rules that pull opposite ways

And a prompt that has to say, in as many words, which rule governs which speaker.

The patient withholds

Answers one question at a time. Pushes back on the same open question asked twice. Volunteers nothing.

Because marking reads the transcript, not the list of questions the candidate asked. A patient who volunteers hands over criteria nobody elicited, and the candidate is credited for information they were given.

The examiner discloses

Answers at the scope it was asked at, all of it. “A full set of observations” returns all six in one reply. Several things named in one message all get answered.

Because a real examiner does. Withhold here and you mark the candidate on how many turns they spent extracting a blood pressure, which is a test of the clock, not of their nursing.

Why it has to be spelled out

Left to reconcile the two, a model applies the patient’s rule to a request for observations and hands over one vital per turn. We watched it happen. The prompt now names which rule belongs to which role rather than stating both and hoping — a small thing that took a measured run to find, and that no unit test would ever have caught, because the rules were both correctly present in the prompt.

05

The examiner asks its own questions — and code decides when

58 authored questions across the bank, and not one of them is the model’s idea.

Real examiners interrupt. Near the end of a station they ask the thing that separates a candidate who followed a script from one who understood the patient: “What is her NEWS2 score, and what does that score require?” Our stations have carried questions like that since they were first authored — validated, house-style checked, verified, and for a while completely inert, because nothing asked them.

They are asked now, and the choice of when is made in code: at most one per turn, selected from the station clock and the examiner lines already in the transcript. The prompt receives one question, already chosen, with an instruction to ask it in exactly those words.

Why not let the model decide

A model asked to watch its own clock and remember what it has already asked fails in both directions — asking the same question every turn until time runs out, or never asking at all. Neither is recoverable from inside the same prompt that caused it.

The “in exactly those words” instruction is load-bearing for a reason worth noticing: the already-asked check works by looking for the question’s text in the transcript. Loosen the wording instruction to make the examiner sound more natural and the same question fires every turn, because the check can no longer find it.

06

The model judges. The code decides.

The model is never asked whether the candidate passed.

Why the split

Ask a model to judge every criterion and reach a verdict and it reconciles the two: it softens a criterion it was about to mark unmet, because “met all but one, still essential, still fails the station” is an uncomfortable thing to say twice in one breath. Splitting the decision out removes the incentive — and makes the outcome explicable without a model in the loop at all.

The rule is the regulator’s

Every essential criterion met, no red flag raised. Not a weighted average, not a percentage. A red flag fails the station on its own rather than being diluted by twenty things done well, which is what makes it worth calling a red flag. Two candidates who did the same thing get told the same thing, because the last step is arithmetic.

Iterate the grid, not the model

Reconciliation walks the criteria grid and looks up a match in what the model returned — never the other way round. A criterion the model’s reply omits comes back unmet, not silently dropped. Iterate the model’s list instead and a truncated response quietly passes everything it never got around to grading, which is the failure you would never see.

A report is a record, not a view

Marking rows snapshot the criterion text they were judged against rather than joining live to the grid. It costs duplication and it is the right call: an author tightening a criterion’s wording next month would otherwise silently rewrite what every earlier candidate was told they were marked on.

07

What we measured, including what we’d rather not report

A marking probe checks the three things that need no answer key: was every criterion judged, does every quoted piece of evidence actually appear in the transcript, and do repeat marks of one transcript agree.

Clean

No fabricated evidence in eight runs

Every criterion marked met quoted something genuinely present in the transcript. That was the biggest risk going in, and it is the one that came back clean.

Clean

Full coverage, every time

23 of 23 criteria judged on every run — nowhere near the one-third threshold at which we reject a marking response as truncated.

Clean

A pass is actually reachable

Nobody had checked. Every earlier probe run returned FAIL — correct for the transcripts given, and silent on whether the other outcome could happen at all. A station with an unsatisfiable essential criterion is unpassable and no candidate would ever find out why. A transcript written against the grid marks PASS on five runs of five, with no criterion judged differently between them.

Not clean

Five runs at temperature 0 disagreed

On a real 99-turn transcript, five identical marking runs disagreed about two essential criteria and one red flag. The verdict held at FAIL all five times only because that candidate was failing on other grounds too. For a borderline candidate, an unstable essential criterion is the verdict.

Then we tried to fix it, and made it worse four times

A separate defect: marking occasionally cited the answer key as its evidence — quoting a sentence out of the prompt’s own reference data rather than out of the transcript. The judgement was right; its stated evidence was the answer sheet. Three prompt edits were tried, each measured over five runs of the transcript that motivated it.

PromptVerdicts over 5 runsCriteria unstableEvidence problems
Shipped prompt, unchangedPASS ×505, all essential
Bar quoting the chartPASS ×514, now met-with-no-evidence
That, restructured and emphaticPASS FAIL FAIL PASS PASS60
Two added sentences onlyPASS FAIL PASS PASS PASS20

Every version that fixed the citation destabilised the verdict, and the smallest edit still flipped it. A verdict that flips tells two candidates who did identical work different things — far worse than a correct judgement with a badly sourced quote. All of it was reverted. The probe still reports five evidence problems against the shipped prompt, and that is the check working, not a number to silence.

Voting doesn’t fix it. Measured.

The obvious response to an unstable judgement is to mark three times and take the majority. It was built, wired, measured, and turned back off: three passes over five runs left the same items unstable as a single pass, for three times the tokens.

The within-run numbers say why. The three passes disagreed with each other about three or four items every single run. Voting only helps when one vote beats chance, and on these items it doesn’t — the instability is ambiguity in the judgement, not noise around a settled answer. Averaging a coin toss three times gives you a coin toss. The mechanism stays in the codebase behind a constant set to 1, in case a future model beats chance on the hard rows.

The wobbly items have something in common

All three are negatively phrased. They ask the model to judge an absence, which is materially harder than spotting a presence: “Does not settle on urinary infection from a dipstick alone” requires reading the whole transcript and finding nothing, rather than finding something.

Rewriting them is barely available. When this was measured, 50 of the 80 red flags in the bank were negative by nature — because a red flag describes an unsafe act or an omission. “Does not check her identity before starting” is what a red flag is, not sloppy authoring, and inverting it changes its meaning. Of the criteria, only a small minority are negative, and those are the tractable ones.

So red-flag stability is tracked separately from criterion stability rather than folded into one number. The first version of that check looked only at criteria and would have missed the flag entirely — and a red flag is the most consequential item on the grid.

08

A reasoning model is the wrong shape for this call

Same station, same prompt, same settings, three runs each. “Findings” are prompt-rule violations the probe detected; “clean turns” are turns with none.

Actor modelFindingsClean turnsOutput that was thinkingMean turnWorst turn
gemini-3.5-flash-lite shipped135 / 36none0.7s1.1s
gemini-3.6-flash 236 / 3691%10.5s73.7s
gemini-3.6-flash, thinking off 533 / 36none6.4s22.6s
deepseek-v4-flash, 4k budget 111 / 1291%4.0s10.5s

Nothing beat the small model at the thing this call is for. The bigger models are not meaningfully better at following the prompt and are six to fifteen times slower on a turn the candidate watches with a clock running — one of them took 73.7 seconds. Switching thinking off made rule-following worse, not better, and still only got it to 6.4 seconds, so the latency is the model rather than the reasoning.

The durable finding: the actor’s job is to withhold, to stay in scope, and to keep two labels straight. Reasoning helps with none of those, while every thinking token is dead air in the room.

Caveat we have to state: the latency figures were taken during a window when two sibling models were returning sustained 503s, so the slower numbers may carry capacity pressure a calmer day would not show.

Where the calls are allowed to go

A probe run puts invented candidate turns and a seeded station to whatever model we want to compare. A real attempt carries a named student’s live exam performance, and sending that outside the UK or EEA is a restricted transfer under UK GDPR — which is a paperwork exercise, not a config flag. So the second client exists, and nothing in the request path calls it.

The same reasoning sets the API tier. A free tier that may train on submitted content is not somewhere a candidate’s exam performance goes, in any environment, including the one where it is convenient.

09

Nothing generated gets published unverified

And the approval is bound to the exact words it approved.

Stations are part authored and part generated, and every field carries which it is. A generated field is put to three deliberately different reviewers — clinical accuracy, applicability to the country’s practice, and exam format — and what each said is stored. Two of three must agree, and any single safety objection withholds the field regardless of the majority. A generated field with no verification record at all cannot be published by any path.

The mechanism that keeps a record honest over time is a content hash stored beside it. Rewrite a verified scenario and its record no longer matches the text, so the station arrives at the publication gate as unverified and blocks. Without that comparison, a verification record is just JSON sitting next to prose that has since changed.

We keep a station whose criteria are deliberately unsafe outside the seeded directory, so the veto can be watched working end to end. A safety check nobody has seen fire is not a safety check.

The one prompt that should see the grid

The reviewer prompt gets the whole station, criteria included — the exact opposite of the actor prompt. The two are not in tension. The actor streams to a candidate who must not be told the mark scheme; the reviewer runs in a seed script with no attempt, no session and no candidate anywhere in the process. A reviewer who cannot see a criterion cannot review it.

And in Canada

The same discipline, one layer deeper: every field of every adapted station is marked as written by the source physician or generated by us, and a model-returned value can never overwrite a physician’s. Adaptations are committed as JSON so a reviewer reads a diff rather than a database, and only title, slug, duration and competencies cross into the public catalogue — real case content does not reach a public page.

10

What we will not claim

Each of these is a figure the regulator does not publish — which is exactly what makes it tempting, and why a copy test fails the build if one creeps in.

01

A pass mark, or how stations combine

The NMC does not publish either. The resit fee is banded by how many stations you resit, which implies stations are marked individually — but "implies" is not "published", and an inference does not belong in copy a candidate plans around.

02

Pass rates. Ours or anybody’s

Not published, not measurable from our own traffic in any way that would mean anything, and the single most tempting number to invent.

03

How long a station lasts

The NMC’s OSCE page gives no timings, so neither do we. Our own stations carry a clock because a station needs one; that clock is ours, not a claim about the exam.

04

That we predict how you would be marked

The result is a simulated verdict against criteria we wrote. It is not a prediction of how an NMC assessor would mark you, and the report says so on the page it is delivered on.

05

That we cover the whole exam

Four of the ten stations are clinical skills performed on manikins and equipment. A conversation cannot assess them, so we do not simulate them and every page in the cluster has to say so.

A candidate deciding whether to book a £794 exam is the person these numbers would mislead. That is the whole reason the rule is enforced by a test rather than by good intentions — marketing copy erodes, and it erodes in one direction.

11

Where this goes

The engine is not specific to nursing, or to the UK.

Live

NMCMATE — United Kingdom

The NMC Test of Competence, both halves: an 839-question bank generating papers from the official test specification, and the OSCE simulator described on this page. 30 stations across seven scenario families, each family running the full assessment → planning → implementation → evaluation arc, plus professional values and evidence-based practice.

nmcmate.com
In build

NACMATE — Canada

Canada’s NAC Examination for international medical graduates. 154 stations adapted from UK to Canadian form with per-field provenance, the report layer landed, the practice engine in progress. A different exam shape — twelve short stations, seven competencies — which is exactly the test of whether the engine generalises or merely fits one regulator.

Next

Clinical certification, more broadly

Anywhere competence is assessed by watching someone work rather than by asking what they know: care certificates, competency sign-off, supervised practice records, revalidation. The hard parts transfer intact — a mark scheme the simulated patient must never see, an examiner bounded to what it was actually given, and a verdict a human can re-derive by hand from the individual judgements.

Building an assessment your regulator will look at?
Let’s talk.

Tell us what you assess, who assesses it today, and what you wish the record looked like afterwards. We’ll come back within two business days with an honest read on whether an AI examiner helps you at all — including the case where the answer is no.

hello@vibeserve.dev or talk to the founder directly: kent@vibeserve.dev
See all five products
Cite this report

VibeServe Research (2026). An AI clinical examiner you can audit. Research #001. https://vibeserve.dev/research/ai-osce-examiner