The Digital Government Target State is coming.

That’s not a scary name I invented; it’s the name of an official government strategy. And recent announcements suggest the Government’s ambition is already taking shape.

Nearly 9,000 public service jobs are set to be cut over the next three years, driven by anticipated AI efficiency gains.

The Social Security (Modernisation) Amendment Bill is the most recent example. Having passed under urgency, the bill allows the use of automated electronic systems to “make any decision, exercise any power, comply with any obligation, or take any other related action under any specified provision” under the statute. The stated intention is simple enough: use AI for routine eligibility checks in cases that are “straightforward and do not require discretion”. Fair enough.

But the legislation is not constrained to simple, rules-based decisions.

Many have raised concerns about whether automated decisions can effectively evaluate benefit eligibility.

The ghost of the Robodebt scandal looms large. Beginning in 2016, Australia used an automated debt assessment tool that miscalculated debts owed to the government, leaving a trail of stress and hardship in its wake. No one wants that here, but technical problems are not the only kind we face. There are political and constitutional questions as well.

What kinds of tasks are we comfortable automating? Are there things we think only a human should do?

Imagine three different scenarios.

In an understaffed emergency room, AI triages who gets care first.

Traffic chokes roads. AI decides not only congestion pricing, but recommends where new roads should be built.

The crime rate spikes. Widespread surveillance uses AI tools to process information and direct where police resources are deployed.

Each of these scenarios is already being trialled overseas. In every case, a subtle shift is being made. AI is moving from providing analysis to making judgements—about priority, trade-offs, and the allocation of resources.

How would you feel sitting in a hospital waiting room knowing a human hasn’t yet thought about you? Or receiving a letter informing you that your land is being acquired for a new road because an algorithm says so? Or knowing the police may or may not visit your neighbourhood based on datasets and risk probabilities?

Every time AI tools step into the realm of important political decisions, the typical response is “get a human in the loop”. The Social Security (Modernisation) Amendment Bill still requires a “specified person” to be accountable for the tool. All well and good.

But let’s be honest. Humans take the path of least resistance. If the decision-making machine keeps making reasonable recommendations, at some point, we will all fall into the trap of saying “yes” by default. The effects of automation bias are well known, especially when we are busy and under pressure.

Do this for long enough and we lose the capability to challenge automated decisions. We forget what independent judgement feels like. The mental muscle weakens.

That doesn’t mean we should reject AI entirely. Used well, AI could enhance government capability, speed up services for people who need support, and help keep rising costs under control.

Will we have government with algorithms or government by algorithms? The difference matters, and we need a strategy that embraces the former without falling prey to the latter.

The first step is to implement “cognitive forcing” in decision-making systems.

Put simply, a human reviewer should record an initial judgement before seeing an AI recommendation. Once the recommendation is revealed, the human can accept or reject it, but any decision must be logged with a justification for the final decision.

Where AI processes decisions at scale, we should also use blind testing. Take a sample of cases and have them assessed by humans without seeing the AI output. Then compare the results.

Transparency matters too. New Zealanders should be able to check a single public register to see where AI is being used across government, how much autonomy it has, and who is accountable for it.

Most people will be happy with the AI notetaker at the GP.

I am happy with AI triaging the emergency room, but that might make some people a bit queasy.

I’m not sure I want a Ministry of Health GPT choosing funding allocations or where to build a hospital. But it’s hard to explain why that’s different.

We will inevitably disagree about where to draw various lines, but let’s have a debate about where those lines should be.

To govern is to choose. If we let those choices migrate quietly into systems we cannot see, question, or hold to account, we could lose democratic oversight. Worse, we might not even notice.

This op-ed was originally published in The Post on 16 June 2026.

go back