So you’re sitting opposite your GP. It’s been months since your last appointment. This time, though, your doctor isn’t hunched over a computer screen double-checking Dr Google.

She’s looking directly at you.

Emboldened, you explain that your symptoms first appeared “in the morning”.

Instead of this, the AI scribe records: “every morning”.

It may not seem like much, but there’s a real problem here. A medical symptom appearing “in the morning” versus “every morning” can be the difference between a situational issue and a chronic one. Treatment then would differ, as would outcomes.

This transcription error actually happened, reminding us that even apparently low-risk AI tools need careful oversight.

Of course, if you’ve ever asked AI to summarise a meeting, you’ll recognise the problem: it gets most things right – until it doesn’t. In healthcare, small differences really matter. For example, a thunderclap headache that peaks in 60 seconds might be mistaken for a migraine; in fact, it can signal a ruptured brain aneurysm. Taking aspirin instead of calling an ambulance could delay urgent treatment, potentially leading to coma or even death.

New Zealand’s health system needs help. As our population ages, demand is rising. Workforce shortages leave clinicians carrying heavier patient loads. Long waits for treatment and ageing hospital infrastructure continue to place pressure on patients and staff alike.

That’s one reason medical AI’s promise is so compelling.

Used well, it can reduce paperwork, improve access to care, and make better use of scarce clinical expertise. Every emergency department now has access to an AI scribe, and early reports show doctors averaging one additional patient per shift. In South Auckland, AI-assisted retinal screening is helping identify diabetes-related eye disease. And Health NZ is preparing to use AI for one of the two reads required for mammograms.

The opportunities are real, and they’re multiplying. However, medical AI is moving into everyday clinical practice faster than we are designing systems to govern it. So what do we do?

A public national register of medical AI systems is a practical first step. Health leaders should know which AI tools are being used for what, how they have been validated for New Zealand populations, and who is responsible for monitoring their performance.

Even if a model functions well overseas, it may not here. Its accuracy depends on the data used to train it, so the data needs to reflect the people receiving care. Populations with different genetic, environmental, and behavioural profiles do not look the same when they are reduced to data points. Patients who are least well served by the health system now are at the greatest risk of being poorly served by AI tools.

And like their human counterparts, AI tools continue “learning” and so must be constantly evaluated. If performance is allowed to drift, problems could remain hidden until patients are affected.

A national register should tell us where AI is being used, who is accountable for overseeing each system’s performance, and when they should be reviewed. It will allow health leaders to decide when to pause, replace, or withdraw systems. That isn’t bureaucracy; it’s good governance.

Crucially, we also have to decide what we want AI tools to achieve. If AI scribes save clinicians time, we shouldn’t just assume that extra time will be used to see more patients or have longer consultations. Free time gets absorbed quickly if it is not directed towards something intentionally. Let’s aim for better conversations between physicians and patients, more time for careful diagnosis and less burnout of medical staff.

The same principle applies to screening and diagnostic tools. Earlier detection and wider access are genuine gains, but patients are not collections of data moving through an increasingly efficient pipeline. Good medicine requires trust, careful judgment, and relationships built on careful listening.

AI can strengthen these aspects of care. It can reduce paperwork, identify patterns clinicians might otherwise miss, and help professionals use their expertise where it matters most.

It can also quietly reshape healthcare around what is easiest to measure: speed, throughput, and cost.

New Zealand is already embracing medical AI. Let’s ensure we test it before deployment, monitor it throughout its life, and retire systems that don’t deserve our confidence.

This isn’t an argument to shackle technology. It’s an argument to pursue technology with a purpose: ensuring that humans remain at the heart of health care.

This op-ed was originally published in The Post on 21 August 2026.