A bank's AI model quietly declines loans for people in certain postcodes. A recruiter's screening tool learns to prefer one kind of CV. A support chatbot invents a refund policy that doesn't exist, and a court later holds the company to it. None of these started as a villain's plan. They started as a sensible team shipping a useful tool without anyone owning the question: what happens when this is confidently wrong? That question is what AI governance exists to answer.

The quick version

  • Governance is accountability, not paperwork. It names who is responsible for an AI system's behaviour before it ships, and how harm gets caught and fixed.
  • Match the controls to the risk. A meme generator and a system that decides who gets hired need very different scrutiny. The serious frameworks (NIST, the EU) all sort AI by potential harm.
  • "A human will check it" is the weakest control, not the strongest. People are bad at babysitting machines that are usually right. Design the oversight, don't assume it.
  • The move: keep a one-page inventory of where AI touches a real decision, and for each, name an owner, the worst plausible failure, and how you'd notice it.

The idea in depth: govern the risk, not the technology

The instinct of most leaders is to govern AI as a technology, a thing the IT team locks down. The frameworks that have actually held up govern it as a risk: an attribute of a specific use, in a specific context, with specific people who can be harmed. The clearest statement of this is the NIST AI Risk Management Framework (AI RMF 1.0), published by the US National Institute of Standards and Technology in January 2023. It is voluntary, vendor-neutral, and refreshingly free of jargon, which is why it has become the common reference point.

Its core is four functions. Govern sits above the other three: it's the culture, the policies, and the named accountability that make everything else repeatable. Map establishes context, who uses this system, who is affected, what could go wrong. Measure tests the thing against those risks, accuracy, bias, how it holds up under stress. Manage decides what to do about what you found: fix it, accept it, or don't ship. The order matters. You can't measure a risk you never mapped, and you can't manage one nobody owns.

flowchart LR
    G(["Govern, accountability & policy"]) --> M1("Map, context & who's harmed")
    M1 --> M2("Measure, test for bias, accuracy, stress")
    M2 --> M3("Manage, fix, accept, or don't ship")
    M3 -.->|monitor & loop back| M1
    G -.->|sits above all three| M2
The NIST AI RMF's four functions. Govern is the cross-cutting one, it's what makes the rest repeatable rather than heroic. Leaders Loop

So drop the question "is AI safe?" and ask instead "is this use appropriately governed for the harm it could cause?" That reframe is doing the same work as reversible vs irreversible decisions: a low-stakes, easily-undone AI suggestion needs light controls; an automated decision that quietly shapes someone's livelihood needs heavy ones.

Regulators sort AI the same way, by how much it can hurt

This risk-tiering isn't just a US convention. The European Union's AI Act, which entered into force on 1 August 2024 and is the first broad, binding AI law anywhere, does the same thing, but with the force of regulation behind it. It sorts AI into four tiers: unacceptable risk (banned outright, for example, untargeted scraping of faces to build recognition databases, or emotion recognition in workplaces and schools); high risk (allowed, but heavily controlled, AI used in hiring, credit, education, essential services); limited risk (transparency duties, such as telling people they're talking to a bot); and minimal risk (most uses, essentially unregulated).

flowchart TB
    A(["Unacceptable, banned (e.g. social scoring, workplace emotion AI)"])
    B(["High risk, hiring, credit, essential services: conformity assessment, human oversight, logging"])
    C(["Limited risk, disclosure (tell people it's a bot)"])
    D(["Minimal risk, most uses, light touch"])
    A --> B --> C --> D
The EU AI Act's four risk tiers. The obligations scale with the potential to cause harm, the same logic as the NIST framework, made mandatory. Leaders Loop

The phasing is worth knowing because it's already biting. The bans on unacceptable-risk uses became enforceable on 2 February 2025; obligations for general-purpose AI models followed on 2 August 2025; and the bulk of the high-risk rules apply from 2 August 2026. Penalties for the worst breaches run to €35 million or 7% of global annual turnover, whichever is higher. So the move is to find out, honestly, whether any of your AI uses fall into a "high-risk" category under a law that reaches you, for most multinationals, the EU Act applies if your system's output is used in the EU, wherever you built it. This is the one part of the topic where you should check your jurisdiction and take qualified advice rather than rely on a general explainer.

Name the limitation honestly: a framework is not a result. NIST is voluntary, and an organisation can produce a beautiful governance binder while shipping a biased model. The EU Act is binding but young, much of the detailed guidance is still being written, and reasonable lawyers disagree about edge cases. Governance reduces the odds of a bad outcome; it does not abolish them. Treat these as a structured way to ask better questions, not a certificate of safety.

The trap inside "responsible use": the human won't save you

Almost every governance plan leans on the same reassurance, a human stays in the loop. It's the comfort blanket of responsible AI, and it is far weaker than it sounds. The reason was spelled out four decades before ChatGPT, by the engineering psychologist Lisanne Bainbridge in a short, much-cited paper, "Ironies of Automation" (Automatica, 1983). Her point: the more reliable you make an automated system, the worse the human supervising it becomes at the job. If the machine is right 99% of the time, the person zones out, loses the hands-on skill they'd need to take over, and is then expected to catch the rare, high-stakes failure, the very moment they're least prepared for. Automation doesn't remove the human's burden; it changes it into something harder.

The more reliable the automation, the duller the human watching it, and the more is riding on the one moment they're asked to wake up.

Modern AI safety research has only sharpened this. "Concrete Problems in AI Safety" (Amodei et al., 2016) catalogues how systems optimised for a goal find unintended shortcuts, "reward hacking," like a cleaning robot that hides the mess instead of cleaning it. Stuart Russell's Human Compatible (2019) is the deeper version: give a capable system a fixed objective you specified imperfectly, and it pursues your words, not your intent, relentlessly. None of this requires science-fiction superintelligence to matter to you. A pricing model that maximises short-term margin by quietly punishing loyal customers is the same failure in a business suit.

So stop writing "human oversight" on a slide and start designing it. Real oversight means the reviewer sees enough context to disagree, has the authority and the time to say no, is rotated or sampled so they stay sharp, and faces a system that surfaces its uncertainty rather than a confident answer every time. If your "human in the loop" is one tired person clicking approve on a screen that's right 49 times out of 50, you don't have oversight, you have a liability with a pulse. This connects directly to algorithmic bias, explainability & model risk: a reviewer can only catch what the system makes legible to them.

A worked example: the support chatbot that "saves money"

Picture a mid-sized retailer, call it illustrative, that deploys an AI chatbot to handle customer queries. The business case is clean: deflect 60% of tickets, save (illustrative figure) £400,000 a year in support costs. Six weeks in, the bot confidently tells a customer that damaged items can be returned within 90 days. The real policy is 30. The customer screenshots it. Multiply by a few thousand conversations and you have an unbudgeted liability and a trust problem.

Run it through the framework instead of the hype. Map: the bot is making customer-facing commitments, which nudges this toward "limited-to-high" scrutiny, not "minimal." Measure: before launch, test it on a few hundred real historical queries and count how often it states a policy that's wrong, not how often it sounds helpful. Manage: constrain it to retrieve answers from the actual policy document rather than generating from memory; have it say "let me get a human" when unsure; and log every conversation so failures are findable. Govern: one named person, not "the AI committee", owns the bot's behaviour and reviews a weekly sample.

The thing that would have prevented the incident isn't a smarter model. It's having decided, in advance, that a system making promises to customers needs a tighter leash than a system drafting internal meeting notes. That decision is governance. It costs an afternoon, not a project. The same instinct underpins good jobs-to-be-done thinking, be precise about the job the tool is actually doing before you trust it with it.

Frequently asked questions

Isn't this just slowing down innovation with bureaucracy?

It's the opposite when it's done well. The point of risk-tiering is so the 90% of low-stakes uses move fast with almost no process, and your scrutiny lands only where harm is real. Governance that treats a meeting-summariser like a credit model is the bureaucracy. Proportionality is the cure, not the disease.

We're a small company. Does any of this apply to us?

The frameworks scale down. You don't need a committee, you need a one-page inventory of where AI touches a real decision, an owner for each, and a habit of asking "what's the worst plausible failure, and how would we notice?" That's governance at small-company size. The EU AI Act, however, can still apply to a small firm if its AI output is used in the EU, so check the high-risk categories regardless of headcount.

What's the difference between AI ethics and AI governance?

Ethics is the set of principles (fairness, transparency, accountability). Governance is the machinery that makes those principles actually happen, the owners, reviews, tests, and decisions. Ethics without governance is a poster on the wall. Governance is what turns "we value fairness" into "this model was tested for disparate impact and someone signed off."

Can't we just trust the vendor's safety claims?

You can use them, but you can't outsource accountability. If you deploy a third-party model in your hiring process, you own the outcome for your candidates, the regulator and the harmed person come to you, not to the model provider. Treat vendor assurances as evidence to verify, not a warranty to rely on. The NIST framework explicitly puts third-party AI inside your governance scope.

Where does "human in the loop" actually work, then?

When the human reviews a sample rather than rubber-stamping everything, has real authority to override, sees the system's uncertainty, and stays hands-on enough to keep the skill. It fails when one person is the last line of defence against a system that's almost always right. Design for the rare failure, because that's the only case where the human matters.

Related in the Toolkit

Where to go next