Deploying clinical AI without losing the audit trail

Clinical AI fails in deployment far more often than it fails in validation. The model performs on the test set and then meets a population it was not trained on, a scanner it has not seen, or a workflow that gives clinicians no way to disagree with it.

The engineering around the model decides whether it is safe. Three things matter most.

1. Shadow mode is not optional

A new model runs alongside the existing workflow, producing outputs that nobody sees and nobody acts on, while its performance is measured against your own population and your own ground truth.

This costs you weeks. It buys you the only evidence that matters: how the model behaves on your data, not on the vendor's. Every deployment we have run has surfaced something in shadow mode — a subgroup with degraded performance, a scanner protocol that shifts the distribution, a preprocessing assumption that does not hold.

Only when the shadow numbers are measured and accepted does an output surface to a clinician.

2. Explainability is a workflow requirement

"The model scored this 0.87" is not decision support. It is a number a clinician must either trust blindly or ignore entirely, and both are bad outcomes.

Every output should arrive attached to:

• the inputs it was computed from
• the model version that produced it
• the reasoning surface appropriate to the modality — a localisation, a set of contributing features, the rule that fired
• the confidence and its limits, stated in terms a clinician can act on

If a clinician cannot interrogate a suggestion in a few seconds, they will learn to dismiss it, and the system's value goes to zero regardless of its accuracy.

3. Log the overrides — they are the signal

When a clinician disagrees with the model, that is the most valuable event in the system. Record it as a first-class object: what was suggested, what was decided, by whom, and where available, why.

Override patterns tell you about drift before your monitoring does, identify subgroups where the model underperforms, and give you an evidence base when a decision is questioned months later.

What the audit trail must reconstruct

Assume that at some point, someone will ask what the system saw and said on a specific date. The log must be able to reconstruct, immutably:

• the exact inputs, including data version
• the model and threshold configuration in force at that moment
• the output as presented to the clinician
• the action taken, and by whom

If any of those are missing or mutable, you cannot answer the question, and the deployment cannot be defended.

The clinician overrules the model

Every time. The system's job is to make a good decision faster and better-informed — never to make the decision. Build every interface on that assumption and the governance follows naturally.