Adding artificial intelligence to a piece of software has become almost a reflex. A competitor announces an AI assistant, and within weeks there is pressure to ship one too. Yet a feature that works perfectly on a technical level can still make people’s jobs harder, and when that happens they quietly stop using it.
Take an insurance company that built a tool to draft complete written decisions on benefit appeals. The specialists who handle those appeals soon abandoned it. Those who persevered were slower than colleagues who ignored it, because checking an AI-written decision thoroughly enough to put their name to it took about as long as writing one from scratch, and they also had to hunt for mistakes they would never have made themselves.
The fix was not a better model. A second version simply found the relevant policy clauses and earlier rulings for each case, then left the writing to the specialist. The tedious searching vanished, review time fell, and the skilled part of the job stayed with the person who was good at it. Same technology, different slice of the task.
Every new feature has a price for the user
Any change to an interface asks something of the people using it before it gives anything back. They have to relearn the layout, update their mental picture of how the tool behaves and decide how far to trust it. Working memory is small, and all three of those jobs compete for it.
AI asks for more than most changes, because it alters the work itself. Let it pull information together and the user turns into a proof-reader. Let it make the recommendation too, and the user is reduced to signing things off. For experienced professionals, that can feel less like help and more like a demotion.
So the useful test is a balance sheet for one person doing one task. On one side sits what the feature costs them in attention. On the other sit the three ways it can pay back:
- Effort saved – genuinely less thinking, not just fewer clicks.
- Errors caught – spotting problems the person would have missed, as a fraud-screening tool does.
- Consistency – getting a whole team to the same standard.
If none of those outweighs the cost, the feature sits below the break-even line, however impressive the demo.
The four questions
These are best worked through together, early, while the idea is still a proposal, and ideally with someone who does the job every day.
1. Does it cut the total effort for this particular user?
Count the checking the user inherits, not just the clicks removed. Pre-filling a reference number that took two seconds to type swaps a trivial action for one that needs attention. Meanwhile, the same AI summary might be a real head start for a junior analyst and a pointless four-minute read for a senior one, so decide who the feature is for and give everyone else a way out.
2. Is it better than how they work today?
AI tends to win where a problem is fuzzy, large, repetitive across a team or full of things people routinely miss. It loses where someone already has a quick, practised route to a known answer. Suggesting unexpected candidates for a promotion from thousands of staff records is a good fit; drafting a two-minute email the sender has firm views about is not.
3. Will they notice when it is wrong?
Some errors are obvious, such as a connecting flight that leaves before the first one lands. Others look entirely plausible, like a museum recommended for the day it is closed. Decades of research show people tend to defer to automated output, so the invisible, high-stakes mistakes need designing for. Showing where each answer came from, and when the data was gathered, lets people judge how far to trust it.
4. Will they still be able to tell in a year?
Handing routine chores to software is fine. The judgement that spots the machine’s mistakes is different, because it fades with disuse. Simple habits keep it sharp: asking people to form their own view before seeing the AI’s, or routing a share of cases through without assistance.
By the numbers: how a wrong suggestion sways experts
A 2023 experiment with radiologists shows why that fourth question matters. Readers were given a simulated AI system whose suggestions were deliberately wrong in some cases.
- 27 radiologists took part.
- 40 mammograms were assessed after a training set, with incorrect suggestions planted in 12 of them.
- The most experienced readers, averaging around 15 years, were right 82.3% of the time with a correct suggestion and just 45.5% with a wrong one.
- The least experienced dropped from 79.7% to 19.8%.
Experience softened the effect but did not stop it.
What this means for your website or business
You do not need to be building enterprise software for this to apply. Chatbots, AI-written product descriptions, automated replies and smart form-filling are now easy to bolt on to an ordinary business website. Before adding one, it is worth weighing it up honestly.
Where AI usually earns its place:
- Sorting or searching through large amounts of content quickly.
- Giving newer staff or first-time visitors a starting point.
- Flagging things a busy person might overlook.
Where it tends to cost more than it saves:
- Tasks that are already quick and well understood.
- Answers customers cannot easily verify, such as prices, opening hours or policies, unless the source is shown.
- Jobs where staff gradually lose the knack of spotting when something is off.
The amount of attention a person can give in a day does not grow just because the software gets cleverer. Spend an hour asking these four questions before you build, and the AI you add will be the kind people actually keep using.
