The hype around AI agents and the models that power them is warranted. But while they can give organizations a real competitive advantage, in reality, how much value they deliver ultimately depends on something much less glamorous: the middleware and integrations that connect agents to systems across enterprise workflows.
Enterprises have used AI in various forms for decades, but agents represent a significant shift in how we put AI to work and what we ask it to do. The first phase of AI largely focused on generating, summarizing, analyzing and recommending information. Agents go a step further by executing actions and this is where the middle layer often gets overlooked.
Without the right APIs, MCPs, data streams, tools, deterministic logic, identity management and systems of record, agents and models essentially operate in a sandbox. As enterprises adopt agents into real-world workflows, the question CIOs need to ask isn’t simply, “Which AI agent should we deploy next?” It’s, “Can our enterprise faithfully support software action on our behalf?”
The model is becoming the commodity layer
After decades of leading a technology organization and countless implementations, I’ve seen firsthand how important it is to align technology with the business problems it is meant to solve. When it comes to AI agents, one of the biggest mistakes I see organizations make is setting unrealistic expectations for how a model will operate within their existing enterprise environment.
AI agents and models are often touted for what they can accomplish, which naturally focuses attention on their reasoning and execution capabilities. But we cannot simply layer them onto existing workflows and expect them to make decisions or take consistent, repeatable action reliably.
Financial services make for a great example of this. Think about an investing agent who evaluates companies using financial data. When they need it to calculate something precise, like a company’s cash burn ratio or runway, they don’t ask the LLM to do the math. Instead, they use a modern coding language such as python or node.js to perform deterministic analysis, then pass the results to the model to interpret their meaning in the broader context of the company.
As models continue to improve, the real differentiator isn’t the model itself, but what we engineer around them to turn their reasoning into trustworthy action.
The middle layer is where agents become enterprise software
For agents and models to deliver value outside a sandbox environment, they need connectivity to the enterprise systems supporting workflows. They also need to retrieve information from these systems, pass data between them and regularly execute actions within business-critical applications.
Think of the middle layer as the connective tissue between reasoning and execution. The difference between an agent that can answer questions using information you’ve provided versus one that can retrieve a customer’s information, determine the appropriate next step and execute that action in a CRM or ERP system is fundamental. Building, managing and maintaining these connections across systems at scale requires a mature approach to integration.
Integration maturity varies across the enterprise landscape today. While some organizations have “point-to-point” technical debt, others have already developed mature AI integration infrastructure, investing in platforms such as Workato, MuleSoft or Apigee and building the engineering practices and dedicated teams needed to operate these connections. From what I’ve seen working with enterprise technology teams, it’s the latter that will find adding agents exponentially easier.
Another “gotcha” is how prominent MCP’s (Model Context Protocol) are becoming. MCP’s are quickly becoming the new standard for how Agents communicate with systems. There are hundreds already available within Enterprise AI tools such as Claude, Gemini or ChatGPT, but that still does not replace the need for an API/MCP management tool. B2B transactions will soon be performed via MCP or custom vibe-coded apps will need their own MCP to expose capabilities. Regardless, the technical requirement stays the same.
For organizations without a purpose-built foundation for connecting Agents with core systems, what starts as an AI initiative can actually turn into an integration modernization initiative. Before agents can reliably interact with systems, organizations need to connect them.
Agents will expose the technical debt we’ve learned to work around
Modernizing the connections between enterprise systems to prepare for agent integration can help us uncover critical gaps in our existing environments. We see where humans commonly pick up the slack between disconnected systems through manual processes like data unification or handoffs. Agents, however, cannot compensate for these inconsistencies in the same way.
Regardless of the type of AI model, if data is inaccessible, APIs are unreliable, business logic is undocumented or systems can’t communicate with each other, an agent’s capabilities are hindered. This doesn’t mean agents create new architectural problems; instead, they help us expose technical debt we’ve grown accustomed to. The danger is trying to recreate those human workarounds for individual agents rather than fixing the underlying problem.
As organizations deploy more agents, one-off fixes can snowball into novel integrations, data access patterns and workflows that IT teams have to maintain. Instead of reducing complexity, we risk adding another fragmented layer to the enterprise architecture.
Enterprise-grade agents require a different engineering discipline
Beyond these architectural challenges, Agentic AI integrations also inform software engineering. Unlike enterprise software, which is built around predictable inputs, deterministic logic and defined failure status, agents introduce probabilistic behavior into enterprise environments. This means that the same request can produce different outputs or actions depending on the context and information available to the model.
I’ve seen this play out in the enterprise environments we work with. Even relatively simple agentic workflows can become unreliable when connections are faulty, schedules are inconsistent or prompts drift over time. What looks straightforward from the outside can require significant engineering and oversight to operate reliably.
If a relatively simple, low-stakes workflow like this example is easily derailed, consider the engineering required when an agent is expected to take consequential action across ERP, CRM or financial systems. This is why real software engineering is necessary when integrating agents.
Enterprises can’t simply buy an agent and expect stable automation in the same way they might configure and deploy other platforms. Agents must operate as part of the whole system, often spanning multiple apps and requiring ongoing testing, monitoring and maintenance. This requires a team of dedicated engineers and developers to make agent behavior observable, recoverable and reliable enough for enterprise use.
CIOs need to build the organization around AI Agents
For CIOs, preparing your organization to scale agents requires looking beyond the technology itself. Mature integration practices and dedicated software engineering teams are important, but those capabilities need to be matched by strong technical leadership and a culture that understands software, data and calculated risk.
Strong technical leadership is especially important because, remember, not every workflow should be agentic. We need leaders to recognize where AI reasoning and adaptability create value and where conventional automation or deterministic software is the better choice. At the same time, teams need to be comfortable experimenting with probabilistic systems, learning from failures and pushing innovation without sacrificing the engineering discipline required for enterprise environments.
For CIOs, preparing your organization to scale agents requires four capabilities: mature integration practices, dedicated software engineering teams, strong technical leadership and a culture that understands software, data and calculated risk. These capabilities reinforce each other; having two or three isn’t enough. Organizations need all four to move beyond individual AI experiments and scale agents across the enterprise.
Enterprise connectivity has a new purpose
Enterprise connectivity used to enable information to move between systems. In the agentic era, it increasingly enables software to reason and act across them. Organizations can see improvement in operations and service delivery, but it depends on infrastructure that supports connectivity between systems.
With the right connections in place, agents can eliminate manual handoffs across complex workflows, accelerate decision-making and help organizations scale expertise and services without adding another layer to existing operations. That’s where I see the real competitive advantage emerging.
As agents and models become more capable and widely available, access to the technology itself will become less of a differentiator. What will separate organizations is their ability to build systems around Agentic AI, enabling it to work dependably across the systems that run their business.
The organizations that gain the greatest advantage from Agentic AI may not be those with the most agents deployed or even the best models, but those that have invested in the infrastructure, integration practices and engineering capabilities around them.