To get the most out of generative and agentic AI systems, enterprises are giving them broad access to business data. This process, however, is not simply a matter of opening the data floodgates and letting AI do its thing; it raises some very important governance questions that CIOs need to address.
For instance, if an employee does not have permission to open a particular document, should they be able to obtain its contents by asking an AI tool? After all, conventional access controls govern how people access information at its original location, but AI systems can draw from information copied from source systems and stored in data lakes and vector databases.
That can create a gap between the permissions applied to the source data and the information an AI system can retrieve in response to a user’s question. What about the issues related to information such as HR records, payroll data, patient health information and intellectual property, for example?
It’s perfectly plausible that an employee may not normally have access to the company’s salary spreadsheet. But if that data has been added to the data used by an internal AI system without those access rules being carried forward, that same employee may be able to ask the system for salary information and receive an answer regardless.
How AI draws on enterprise data
To understand how this can happen, it helps to look at the way AI systems access enterprise data. Typically, LLMs have a training cutoff date, and therefore may have no automatic access to information created beyond that point.
Yet, contemporary organizations are highly data-driven, and AI systems need to be constantly updated. The way around this potential information bottleneck is to make newer or internal information available by converting it into numerical representations and storing it in a vector database.
When a user asks a question related to that data, retrieval-augmented generation (RAG) can search it for relevant information and incorporate it into the AI’s response. This allows an internal AI system to appear informed about recent or organization-specific information without the underlying model being retrained. Crucially, decisions about which data enters the vector database can shape subsequent AI output.
In our scenario, the employee does not need to open the salary spreadsheet itself. They only need to ask a question that prompts the AI system to retrieve relevant information from it and generate an answer. This is why controls applied only to the original file may no longer be enough.
This doesn’t mean the model itself is unsafe; the issue lies in the pipeline that feeds it. The AI system does not “know” who is allowed to see what; it simply retrieves whatever content sits in the vector database that appears relevant to a query. Done well, this process should carry the original access permissions forward at every stage: when data is ingested from source systems, when it’s indexed for retrieval, and when the AI generates a response, so that only content the requesting user is authorized to see is ever returned. When that permission mapping is done properly, an AI system is no more of a governance risk than a well-configured search index. The problem CIOs need to solve is ensuring that mapping happens consistently, rather than assuming it happens automatically.
Building governance into the AI pipeline
Another big issue is that data supplied to an AI system can affect the accuracy of its answers and, by definition, the information it exposes. In one case in Canada, a tribunal found Air Canada liable after its website chatbot gave a passenger incorrect advice about a bereavement fare.
The chatbot told the passenger that he could apply for the discount after buying their ticket and completing their travel. Air Canada’s written bereavement-travel policy said that requests could not be made after travel had been completed. The passenger relied on the chatbot’s information, then brought a claim when Air Canada refused the discount and won. As the American Bar Association noted at the time, “companies remain liable for the actions of their AI tools and should put into place adequate internal policies for their accuracy and other considerations.”
The solution to these various important issues lies in better governance. The starting point is the source data, along with the rules and processes that apply before information is copied into a data lake or made available to an AI system. This is only possible if there is a clear inventory of the files and objects held across relevant source systems. Data can then be selected based on factors such as location, file type, date of creation or its relevance to the intended AI use case, among other considerations.
At the same time, older information or file types that are irrelevant to the intended AI use case can be excluded before data is copied onward. The selected data can be hosted in a designated area of the data lake for further processing. Maintaining a log of those movements creates a record of which information has entered the downstream AI pipeline. If, at some point, an explanation or proof of activity is required, that record can help in investigating how a particular document became available to an AI system.
Ideally, and before approving a new AI initiative, CIOs need to ask some important questions. For example, what information will the system be able to access, and where has it come from? How has the organization decided that it is relevant to the intended use case? Is it current? Has information that is outdated or unnecessary been excluded?
The aim must be to ensure that the permissions applied to the original document are reflected when an AI system searches for and returns information. Without these controls, an internal AI tool could quite easily become an unintended route around existing and very important access controls.