AI governance’s real gap is accountability, not technology

Most enterprises govern AI the way they would any other IT rollout: deploy dashboards, institute policies, and conduct periodic reviews. Having the right AI governance tools in place and addressing gaps is important, but IT executives should be more concerned about rethinking workflows and establishing ownership.

“Accountability usually breaks down first,” says Beena Ammanath, executive director of the Global Deloitte AI Institute. “Teams deploy AI faster than their organizations can define who owns risk, approvals, monitoring, and outcomes,” she adds, which results in inconsistent use, fragmented controls, and limitations in scale.

Enterprises need to redesign jobs and workflows, not just the tooling, yet 84% of organizations the Institute surveyed haven’t adjusted jobs for AI, and just 21% have a mature AI governance framework in place despite rapid adoption, according to Deloitte’s 2026 State of AI in the Enterprise report.

“Technology can provide visibility and controls, but it can’t replace leadership,” says Rajesh Arora, chief data and analytics officer at Principal Financial Group. “The organizations that scale AI most effectively are the ones that treat accountability as a business responsibility, not a technology responsibility.”

One year ago Principal was focused on building its ethical and responsible AI framework, its inventory, and governance processes to support responsible adoption. Today the firm has assessed nearly 200 use cases. “Governance is increasingly embedded in those investment and operating decisions, rather than applied as a separate review at the end,” he says.

Costly, compounding mistakes

Some organizations have already run into issues with respect to inadequate governance. A 2024 Stanford RegLab study found that AI-assisted research tools offered by LexisNexis (Lexis+ AI) and Thomson Reuters (Westlaw AI-Assisted Research and Ask Practical Law AI) hallucinated between 17% and 33% of the time when working on legal queries, while the Damien Charlotin AI Hallucination Cases Database has identified some 2,041cases of fabrications so far. In at least one legal case (Couvrette v. Wisnovsky) this resulted in nearly $95,000 in sanctions.

Cases like this one are likely the result of an accountability failure. It may be that no one actually owned the job of verifying the AI’s output before it went to the court. But if a single instance of a static AI output could get past a review today, how will organizations deal with autonomous, multi-step agentic AI systems going forward?

Most use cases at Principal are still human-led, with AI providing decision support only, says Arora. “But we’re preparing for a future where agentic and more autonomous capabilities become increasingly common, and those systems raise new questions around accountability, monitoring, and control,” he says. “The goal isn’t to predict every future use case, but to build adaptable governance processes that can support innovation while maintaining trust and transparency.”

Agentic automation raises the stakes

When things go south with agentic AI today, the consequences usually aren’t devastating because in most cases, the outputs are designed to provide insights, rather than take automated actions. That’s already starting to change, however.

Today, many AI failures are like the one described above where someone failed to catch a single bad output. But agentic AI will gradually take more control of workflow tasks in the enterprise, according to Forrester Research. As those agents act autonomously, crossing multiple process steps along the way, small errors will compound before any periodic review can catch them, so governance must be built into every step in the process.

According to Microsoft Research’s VeriTrail project, catching hallucinations in a single final output isn’t sufficient in multi-step AI workflows. The provenance must be traced back through the intermediate outputs, because an agent chains one call into the next, and an error early in that chain can propagate through several steps before a human sees the finished result. Traditional review, built to check outputs rather than the chains that produced them, isn’t designed to catch that.

So business teams can’t be accountable for agent deployments if they can’t verify performance at runtime. “If the agent is goal-oriented, there could be errors in the workflow process that indicate unwanted behavior or skipped steps that the business wouldn’t be able to detect if they only look at the outcome,” says Lauren Kornutick, senior director analyst, AI governance at Gartner.

Power management company Eaton, for instance, has already deployed multi-step agentic workflows without a human in the loop. “We tend to be risk-conscious, not risk-averse, about autonomous workflows,” says Ross Schalmo, the company’s VP and chief data and AI officer. For example, one autonomous agent responds to supplier payment status inquiries by authenticating the requestor, matching purchase orders and invoices, and returning payment information.

Schalmo’s team engineers such applications to be as deterministic as possible, while the non-deterministic, LLM parts of the program are isolated and continuously evaluated against expected outputs to catch model drift. “When there’s a human in the loop, we can be a lot more permissive,” he says. “Anything autonomous, without a follow-on human step takes longer to develop because we have to think about where to use the generative portion of AI versus where we can still use deterministic tooling.”

Breakthru Beverage Group is still early in its AI journey, but EVP and CIO Glenn Remoreras is well aware of the issues around autonomous agentic AI. “It’s people, process, and culture first — to me that’s the bigger play,” he says. The beverage wholesaler has updated use policies, instituted mandatory AI e-learning programs, and is blocking use of unsanctioned AI tools.

But to successfully manage active, agentic AI, where the AI takes actions autonomously across multi-step workflows, users will need more than basic training in AI, says Craig Le Clair, VP and principal analyst at Forrester Research. “AI will take on more of the work and move the human out of duct tape roles that patch around vague systems, poorly designed interfaces, and poorly integrated data,” he says. “The end game is the agent managing and orchestrating the process — that’s the redesigned operating model.”

Humans, of course, will still need to play a role, although today many employees don’t yet have the required skill sets, and currently that point isn’t getting a lot of attention.

AI supervision skills will be needed for agentic oversight, threshold management, confidence intervals, and root cause analysis, for instance. “Figuring out why an agent hallucinated or got stuck in an execution loop — whether due to poor input data or contradictory rules — and how to refine instructions, that’s the direction,” Le Clair says.

Who’s governing anyway?

AI governance is also complicated by the fact that AI projects exist both inside and outside the IT organization. That leaves governance work distributed across CIOs, data and analytics leads, AI leads, IT security, legal, and other players, with no consistent owner, says Kornutick. “It becomes a multi-role orchestration problem of how to govern this, who owns this, and who’s responsible for which component,” she says.

In response, some organizations have brought in a chief AI officer. Twenty-one percent of organizations now have already done so, says Le Clair, adding that roughly 24% of enterprises will move AI and automation governance outside the CIO’s office over the next year to deal with the risk of fragmentation.

Arora, who reports to CIO Kathy Kay, is the focal point at Principal. “My team establishes the standards, oversight, and governance process, but AI is too embedded in the business to be owned by a single function,” he adds. They collaborate with leaders from the business as well as the legal, compliance, risk, and technology organizations.

The challenge, therefore, lies in deciding who’s responsible for controls, model sprawl, and token economics. “We’re seeing a cooperative, co-model where IT focuses on the hard rails — cybersecurity, performance management, infrastructure reliability, and availability,” Le Clair says, while the CAIO focuses on soft rails, such as context engineering, the effects on employees as agents move to the center of the process and start taking over the middleware function humans used to perform, and connecting disparate systems and poorly designed interfaces.

At Eaton, governance is built on a three-legged stool between Schalmo’s organization, cybersecurity, and legal, while enterprise architecture weighs in on tech foundation decisions. The company also assigns ownership of an AI project only after a successful pilot, while IT performs a custodian or stewardship role. “Once that agent is live and operational, it shows up in an org chart, reporting to someone in the organization,” he says.

At Breakthru Beverage, Remoreras formed and leads an AI governance team that reports to the board and co-leads with the VP of data and AI a governance council that reviews and approves AI projects from SVP- and VP-level product owners. “I’m seen as the owner of our AI portfolio, but it’s co-leadership with my peers on the executive team,” he says.

Closing the tools gap

With regard to AI management platforms, says Le Clair, none of the platforms building AI agents have an adequate control plane today to give the visibility needed to move to truly agentic solutions. The most advanced planes are walled gardens. For example, Microsoft’s agent registry will show every agent built across its own platforms, says Le Clair, but not agents built on Salesforce or AWS.

“You’ve got maybe 50 vendors trying to be pure-play control-plane providers, plus the hyperscalers, portfolio companies, and startups building agentic frameworks,” he adds. “There’s no consensus yet on what’s even most important to control, and there’s confusion about requirements.”

Most AI governance tools available today are limited, Schalmo says. “We’ve looked at four or five different external vendors for AI governance and AI control-tower-type solutions, and a lot of them fall down on discovery of AI built by someone else’s solutions,” he says. “That’s a huge gap for me. Either I have to buy governance solutions from almost every piece of software we interact with, which gets really expensive, or I have major gaps in my portfolio I just don’t have visibility into.”

Currently, he’s looking at using a cyber vendor’s AI component for discoverability, going with at least one major firm that provides control-tower capability, and using AI to build another governance layer on top of that.

Organizations clearly need agent registries and agent lifecycle management but it’s less clear what they require beyond that. “How deep does explainability need to go when decisions are made at runtime that weren’t articulated at design time?” Le Clair asks. “There’s real vagueness in the requirements. Vendors have very well-tuned pitches that promise to do anything you want, but buyers often don’t know what they need in the first place, and there are real tool gaps. This is the major problem the industry is actively trying to solve. It’ll just take time.”

But those aren’t Arora’s biggest concerns. “Most organizations can acquire tools,” he says. “Far fewer have clear ownership, decision rights, and accountability embedded across the business.”

Speed versus governance

There’s a perception that governance slows down AI projects because it can’t keep up with the rapid pace of adoption and advancement. A 2026 GitLab AI accountability survey of developers and technology buyers found that 92% of organizations reported some form of governance challenge with respect to governing AI-generated code, and 34% of organizations that experienced an AI-related incident couldn’t determine after the fact whether AI-generated code was the cause.

Similarly, Faros AI’s telemetry analysis of 22,000 developers in its 2026 Engineering Report discovered that developers with high AI adoption rates completed 34% more tasks and 66% more epics, but the median pull-request review time rose by 441%, a nearly five-fold increase.

Is it the case that governance and speed are inherently at odds, and that human review simply can’t keep up? That’s a false choice, says Ammanath. Rather, it shows what happens when both aren’t designed together. “When done well, governance accelerates scaling by giving teams clear guardrails from the start,” she says. “What slows organizations down is retrofitting governance after broad deployment.”

Arora adds that controls don’t slow down innovation but protect the conditions that allow innovation to scale responsibly. “Our approach has always been to move with urgency, but not at the expense of trust,” he says.

How to proceed

For Principal, investing in workforce readiness was one of its biggest investments, helping employees understand both the opportunities and responsibilities that come with these tools, Arora says.

But organizations shouldn’t stop there. Redesigning workflows and decision-making structures is essential before scaling, says Ammanath. “Many organizations have policies and technology in place but lack clear decision rights or governance embedded into daily work,” she says.

Operating model changes take more time than training, and require executive sponsorship and cross-functional coordination. “Without workflow redesign, a workforce may understand AI but lack clarity on when to use it, how to govern it, and how daily responsibilities change,” she adds. “The organizations seeing the strongest results pair broad AI education with role-specific training, workflow redesign, and clear accountability for outcomes.”

Make sure you’ve thoroughly considered the organizational change management aspect of agentic AI projects, says Schalmo at Eaton. His team developed an automated workflow for processing complex bid requests to save the sales team time, but piloted it in a country with lower labor costs than the US. That challenged the economics of the project, and those individuals only dealt with higher-volume, lower-complexity cases.

The lesson: “Lead from a people-first standpoint, and understand the workflow and what’s impacted — whether it’s relieving drudgery and freeing someone for higher-order thinking or customer time, or impacting an actual support function’s role,” he says. “And make sure there’s a viable path to business value.”

An adaptive governance process is key to keeping up with the technology, Kornutick at Gartner adds. She recommends categorizing AI projects following a stoplight approach. Green light projects, such as for personal assistants, present acceptable risks. “We know there’s risk associated with it, but we’re willing to take it,” she says. Yellow-light projects need guardrails in place around such things as PII, or for those triggering a privacy-by-design process, or a security review of a new application. Red light projects need an architectural review before proceeding.

“Think about those use cases as business-transformation processes where you’re fully redesigning workflows, or considering replacing tasks individuals do and shifting them to supervising the AI doing those tasks instead,” she says.

Schalmo’s team does risk tiering based on the EU AI Act’s four-tier risk framework to determine the appropriate level of governance scrutiny for each project. Low-risk requests are automatically provisioned after the requestor fills out a questionnaire, while high-risk cases get a full architectural review.

“Treat governance as the foundation for scaling, not a constraint on speed,” says Ammanath. “Before expanding agentic AI, organizations should define what decisions agents can make, where human oversight is required, who’s accountable, and how performance and risk will be monitored.”

The hardest part, says Arora, is helping an entire organization evolve at the pace of technology. “Successful AI adoption is ultimately a business transformation challenge,” he says. “It requires leadership alignment, clear accountability, workforce readiness, strong governance, and ongoing change management. Long-term trust will belong to organizations that can translate technology into sustainable business value.”