Deploying AI agents lawfully: fom hallucination risks to privacy by design

What are AI agents and why are they not ordinary AI tools?

An AI agent is an autonomous system that independently performs tasks, makes decisions or carries out actions in other systems. This fundamentally distinguishes it from a traditional chatbot, which merely generates text.

However, the distinction is fluid. What matters is a functional assessment: as soon as a system does not merely respond, but takes action, it moves into the realm of agents. Typical agent actions include:

• modifying or deleting data in IT systems or databases
• independently sending emails or messages
• triggering or controlling business processes
• accessing external data sources and consolidating results

Whether a particularly powerful chatbot already qualifies as an agent cannot be answered with absolute precision. There is currently no statutory definition — the EU AI Act refers to “AI systems” without separately addressing the term “agent”.

The MCP protocol as a technical enabler

From a technical perspective, the Model Context Protocol (MCP) in particular significantly facilitates the development of AI agents. This open standard from Anthropic connects AI models with external data sources, tools and systems in a standardised way. Instead of building an individual adapter for each connection, MCP servers provide published API information. The result: machines can communicate with one another and act in a coordinated manner — a key prerequisite for the increasingly autonomous AI applications that are currently making headlines.

As autonomy increases, so does the need for compliance.

What risks arise when AI agents act autonomously?

The greatest risks associated with AI agents are uncontrolled actions, which can result in data loss, legally binding false statements and reputational damage. There have been numerous documented incidents showing that these scenarios are not merely theoretical. Four risk categories stand out.

Data loss and information security risks

Not only did an AI coding agent delete the production database of the software company PocketOS, it also deleted all backups within nine seconds. While carrying out a routine task, the agent independently decided to 'fix' a problem by deleting the database, despite not being programmed to take such action. It then apologised politely, but was unable to explain its behaviour.

The data could be partially restored after two days, but the oldest available backup was three months old. For PocketOS's customers, who are small car rental companies that rely on the software, this meant more than 30 hours of downtime and the loss of three months' worth of booking and customer data.

Civil liability

It has now been confirmed by several decisions that companies are liable for statements made by their AI systems. In the Air Canada case, a chatbot promised a customer a refund that did not actually exist. The competent Canadian tribunal, the British Columbia Civil Resolution Tribunal, held that the airline was bound by the statement — the chatbot was part of the website and not a separate legal entity.

A clear line is also emerging in Germany: at the end of 2025, the Hamburg Regional Court held, in case no. 324 O 461/25, that the operator of the AI chatbot Grok on the platform X was liable for false factual statements publicly disseminated by the bot. In the specific case, Grok had falsely claimed that a German association was receiving substantial federal funding. The court made clear that it does not matter whether a false statement originates from a human being or from an AI system.

Infringements of personality rights

AI agents can also act directly against individuals. In one documented case, an agent attempted to contribute to an open-source project, was rejected — and subsequently launched a public smear campaign against the developer.

Studies also show that large AI models may resort to blackmail and threats under certain conditions.

Reputational damage

The risk of AI systems becoming radicalised on the open internet has been known since Microsoft’s chatbot Tay in 2016. Ten years later, the same pattern repeated itself with Grok, Elon Musk’s AI bot, which spread Holocaust scepticism and attributed this to a programming error.

Clearly, there has been no meaningful improvement, this is an inherent systemic risk.

The common factor in all these cases is that hallucinations and uncontrolled behaviour cannot be entirely eliminated in AI agents. Anyone granting an agent the right to act must understand and actively manage the residual risks.

Why is Privacy by Design so important for AI agents?

How can these risks be addressed structurally? The key lies in a principle that has long remained somewhat neglected in data protection practice.

Privacy by Design means taking data protection into account from the very conception of a system and with AI agents, this is becoming practically relevant for many companies for the first time. The reason is that no-code and low-code platforms are turning companies themselves into AI developers.

Why Article 25 GDPR has often had little practical effect for AI chatbots so far

Article 25(1) GDPR requires the controller to take data protection principles into account already when determining the means of processing. In practice, however, this provision has often had little practical effect in the context of AI chatbots, because:

• the typical controller does not develop the AI software itself, but purchases or licenses it
• Article 25 is addressed to the controller, not to the software manufacturer
• manufacturers are subject to other rules, such as the provider obligations under Article 16 of the EU AI Act

The controller has had little influence over the product design of AI chatbots and therefore often had no real point of leverage for Privacy by Design in that context

No-code platforms change the starting point

Platforms such as Microsoft Copilot Studio, Google Gems and Anthropic Teams fundamentally change this situation. Companies can build and deploy their own AI agents without programming skills. At that moment, they themselves become designers of the means of processing and Article 25 GDPR applies directly. This makes AI agent compliance a mandatory task.

Shift left instead of last-minute review

The underlying principle can be summarised as “shift left”: on an imagined timeline, data protection and information security should be involved as early as possible. Anyone who involves the data protection officer only once the agent has already been completed risks precisely the scenarios described in the previous section.

No-code platforms democratise AI development and thereby also transfer responsibility for Privacy by Design to the companies that build and deploy these agents.

Which specific measures belong in the development process?

Companies that want to use AI agents in a legally compliant manner need a structured process that integrates data protection, information security and AI governance from the outset. The model for this is provided by the Secure Software Development Lifecycle (SSDLC) from the field of information security.

The classic SSDLC comprises phases ranging from planning, design, development and testing through to operation, monitoring and decommissioning. In each phase, data protection and information security are involved, identify risks and design protective measures. For low-code and no-code agents, however, this full lifecycle may be too extensive depending on the use case. A practical adaptation reduces it to the essential phases: analysis and design, testing and verification, approval and deployment, monitoring and decommissioning.

What matters is not the number of phases, but that certain core activities take place:

• Have management determine the organisation’s risk appetite. Anyone who knows that AI agents can act unpredictably and nevertheless grants them far-reaching action rights must make this risk decision consciously, rather than accepting it silently.
• Involve data protection, information security and AI governance at an early stage. Not as downstream review bodies, but as co-designers of the use case.
• Develop appropriate use cases. A more limited scope of functionality with fewer rights may be sufficient to achieve the intended benefit while reducing the risk.
• Conduct data protection impact assessments and AI impact assessments. This is particularly important for agents with access to personal data or far-reaching action rights.
• Implement technical and organisational measures, especially protection against uncontrolled actions through human oversight. This may mean that the agent must obtain human approval before every critical action instead of acting independently.

The database incident described at the beginning could have been prevented by a single measure: a confirmation loop before destructive actions. These are precisely the kinds of measures that a structured development process identifies, provided it begins early enough.

Even for simple agents, a structured review should take place before approval. The intensity of the review depends on the risk associated with the respective use case.

Practical example: Microsoft Copilot Studio: what matters?

Microsoft Copilot Studio makes it possible to create AI agents without programming skills but a data-protection-compliant configuration requires targeted settings. Precisely because the barrier to entry is so low, the compliance configuration deserves particular attention.

What Copilot Studio can do

The platform can be used to create chatbots or telephone bots as well as automation processes. Internal or external knowledge can be connected via API, for example through RAG/grounding. The agents integrate seamlessly into Microsoft 365 (for example as a Teams app) and can also be published on websites. Control and access restrictions are managed through the admin center.

Contractual basis and its limits

In principle, Copilot Studio is also subject to Microsoft’s data processing agreement, the Data Processing Addendum, and the EU Data Boundary, meaning that data processing takes place within the EU or the EEA. However, these two safeguards do not apply in all cases.

Function / configuration	DPA valid	EU Data Boundary
Standard use, Azure OpenAI	Yes	Yes
Web search / Grounding with Bing	No, but separate safeguards apply under the terms of use	No
External content and sources	No	No
Certain AI models, e.g. Anthropic	Yes	No
Certain AI models, e.g. xAI	No	No

Note: These classifications are based on the status as of April 2026. Microsoft’s contractual terms change regularly, before implementation, the current Microsoft documentation should be reviewed.

Companies that connect external sources, activate web search or enable alternative AI models must therefore assess on a case-by-case basis which contractual and data protection rules still apply.

Interaction with Microsoft 365

Copilot Studio uses Azure OpenAI as its AI model by default. Data is connected (for example to documents, emails or Teams posts) via Microsoft Graph. Chat histories are stored in Exchange mailboxes.

In principle, Copilot agents follow existing access restrictions: users can only access data for which they have read permissions. However, additionally connected internal or external sources expand this access and are not automatically subject to the same restrictions. Clear rules are therefore needed as to which content may be connected as a source in Copilot Studio.

Specific configuration recommendations

For data-protection-compliant operation of Copilot and Copilot Studio, settings in five areas are decisive:

• Identity and access control. Who can access agents and how authentication takes place has a direct impact on the applicable contractual terms and the scope of data protection safeguards. Access to confidential company content must also be controlled in a targeted manner.
• Data sources and knowledge integration. Which internal and external sources an agent may use should be defined and approved in advance. Unreviewed sources increase liability risks and the risk of incorrect outputs.
• Publication and reach. The channels through which an agent is accessible determine its risk profile. Internal channels are generally lower-risk than external publication, the latter requires a separate review.
• Monitoring, tracking and usage analytics. Functions for personalisation and usage analytics that are activated by default may be problematic from an employee data protection perspective. A restrictive starting point is therefore recommended.
• AI model selection and retention periods. Not all AI models available in Copilot Studio are subject to the same contractual safeguards. The retention period for chat histories should also be configured deliberately.

The specific implementation of these measures depends on the respective company environment, the use cases deployed and the existing governance structures.

Companies that want to operate Copilot Studio in compliance with data protection requirements need clear governance rules, so that agents are not created in an uncontrolled manner without anyone having reviewed them.

Conclusion: using AI agents in compliance with the law means considering compliance from the outset

AI agents are no longer a topic for the future. They are already being used in production, and the risks are real and documented. At the same time, no-code and low-code platforms such as Microsoft Copilot Studio offer enormous opportunities if companies approach their use in a structured way.

The decisive lever is Privacy by Design: data protection, information security and AI governance must be involved from the very first idea, not merely as downstream approval bodies. An adapted development process, based on the SSDLC, but scaled according to the risk of the use case, creates the necessary foundation. Specific technical measures such as human oversight before critical actions, restrictive access rights and proper contractual reviews reduce the risks to an acceptable level.

ISiCO supports companies on three levels: with the data protection review of AI providers and contracts, with the compliance assessment of specific use cases including DPIAs and risk analyses, and with the creation of AI agent policies that structure the entire process from initial idea to approval.

Companies that take this path do not need to fear AI agents — they can use their benefits without giving up control.

Frequently Asked Questions (FAQ)

#1 Who is liable if an AI agent independently causes damage?

As a general rule, the company deploying the AI agent is liable for its actions. In the Air Canada case, the Canadian Civil Resolution Tribunal held that a chatbot forms part of the website and is not a separate legal entity. At the end of 2025, the Hamburg Regional Court also confirmed that AI-generated false statements can be attributed to the operator. Whether the platform provider may also be liable depends on the individual case and the contractual arrangements in place.

#2 When do I need a data protection impact assessment for an AI agent?

A data protection impact assessment, or DPIA, is required whenever the processing is likely to result in a high risk to the rights and freedoms of natural persons. For AI agents with access to personal data, far-reaching action rights or external publication, this threshold will regularly be met. In cases of doubt, at least a threshold assessment is recommended.

#3 Does Microsoft’s data processing agreement also apply to Copilot Studio?

Yes, in principle, Microsoft’s Data Processing Addendum also applies to Copilot Studio. However, there are relevant exceptions: when web search, external sources or certain AI models, such as xAI, are used, the DPA does not apply. Companies should therefore assess for each individual agent which contractual safeguards actually apply.

#4 How can I protect against hallucinations in AI agents?

The most effective safeguard is human oversight before critical actions are taken. In practical terms, this means that the agent should obtain human approval before carrying out any action with a potentially high risk of harm, such as deleting data, sending messages or changing configurations. Restrictive access rights and a clearly limited scope of action provide additional protection.

#5 What is an AI agent policy, and do I need one?

An AI agent policy defines binding requirements, processes and responsibilities for the development and operation of AI agents within the company. It creates structure where chaos could otherwise arise: from the initial idea and compliance review through to approval. Especially where several departments are able to build agents independently, as platforms such as Copilot Studio make possible, such a policy helps prevent agents from going live without proper review.