The governance gap in agentic AI: What happens when your AI agent makes a bad decision?
From the blog this month: Nobody wants to answer the question until they have to. Then everyone asks it at once: "Who owns the decision when AI is wrong?”
It's the question surfacing in boardrooms, compliance meetings, and regulatory inquiries across insurance, financial services, and healthcare. And for most organizations deploying agentic AI today, there is no clean answer. Not because the governance principles are unknown, but because they have not been extended to cover the architecture that now needs to be governed.
Grant Thornton's 2026 AI Impact Survey found that 52% of insurers report AI-driven revenue growth while only 24% are very confident they could pass an independent AI governance review within 90 days, according to Grant Thornton's report. Insurance Journal and other coverage note 44% of insurance executives say governance or compliance challenges contributed to AI project failure or underperformance, and 62% rate AI maturity as scaling across functions (Grant Thornton). Complete AI Training and Insurance Business coverage add that roughly three in four leaders lack confidence in passing a governance audit and that regulatory guidance from the National Association of Insurance Commissioners is increasing scrutiny. Editorial analysis: Industry patterns show rapid AI adoption often outpaces operational controls, which raises monitoring, explainability, and compliance burdens for insurers.
Given the rapidly evolving insurance response to AI risk, organizations should not treat coverage as static or incidental. At a minimum, boards and risk teams should actively map how AI is being used across the organization, including where it influences decisions, content, or outcomes rather than merely supporting workflow. In an environment where insurers are increasingly signalling that AI related losses may be foreseeable and avoidable or expressly excluded (absent specific endorsement / standalone cover) proactive oversight of both AI risk and insurance response is quickly becoming a key component of effective corporate governance.
Restrictions on high-risk uses of artificial intelligence in the EU will be postponed by more than a year under a deal agreed by EU legislators on Thursday morning.
The plan to delay a key part of the bloc’s flagship AI law received support from European Parliament lawmakers and EU countries after heavy pressure from industry and capitals. The deal struck early on Thursday also largely exempts the use of AI in industrial applications from the scope of the law — a big win for Germany after top officials including Chancellor Friedrich Merz pushed for the change to keep tech heavyweights Siemens and Bosch competitive.
AI agents, systems that independently pursue complex goals with limited human oversight, have entered the mainstream. They are now widely used to produce software, conduct business activities, and automate everyday personal tasks. But they also introduce unique risks. The EU AI Act, the most comprehensive AI regulation to date, was not written with agents in mind. In a new paper, we analyze five governance challenges they pose, covering performance, misuse, privacy, equity, and oversight, and identify critical gaps for the EU to address.
Artificial intelligence is becoming astonishingly capable. It can write essays, identify patterns, summarize research, generate images, reason through problems, and hold conversations that feel increasingly human. Yet a troubling question remains: Does an intelligent system understand us, or is it simply getting better at predicting what we want to hear? This question matters because the challenge of advanced AI is not simply whether machines will become smarter. It is whether they will become aligned (called the alignment problem) with human life and values.
Artificial intelligence—and the subsequent technologies it may underpin, including biological research, robotics, and more—present a shock that may be akin to the next Industrial Revolution. If models improve at the rate they have for the past three years, they could accelerate scientific research, reshape the economy, and more. In this view, AI will certainly present another critical juncture.
Forrester does not endorse any company, product, brand, or service included in its research publications and does not advise any person to select the products or services of any company or brand based on the ratings included in such publications. Information is based on the best available resources. Opinions reflect judgment at the time and are subject to change. For more information, read about Forrester’s objectivity here.
