AI & Security6 min read
How to hack AI – and how companies protect themselves
Chatbots can be tricked with plain language: prompt injection, jailbreaks, data leaks. What is behind it and how to secure your bot.

“Hacking” a chatbot often takes no code at all, just a cleverly worded sentence. That makes AI assistants on websites vulnerable in a way classic IT security never had to deal with. This article explains the principle and the defence at management level. It deliberately contains no instructions for attacks.
Why language models can be tricked
A language model receives the operator's rules and the user's input as one single text. It cannot reliably tell them apart. Researchers described back in 2023 that AI applications thereby “blur the line between data and instructions” [3].
In its list of the ten biggest risks for LLM applications (2025 edition), the security organisation OWASP puts prompt injection first [1]. It means that an input changes the model's behaviour in unintended ways, sometimes through content that humans cannot see but the model reads [2]. According to OWASP, a jailbreak is a form of this that aims to disable the model's safety mechanisms altogether [2]. Important: OWASP itself notes that no countermeasure is completely foolproof [2].
No. 1
For companies running a chatbot, three further entries on the OWASP list are especially relevant: sensitive information disclosure (LLM02), system prompt leakage (LLM07) and misinformation (LLM09) [1]. In other words, a bot can reveal internals, expose its own instructions or simply state falsehoods, and each of these problems ends up with you. That is why a well-written system text alone is not protection. You need several layers of technology, organisation and testing.

Direct and indirect: two routes to the same goal
In the direct variant, the user's own input changes the model's behaviour, deliberately or by accident. In the indirect variant, the model reads external content such as a web page or a file that contains hidden instructions [2]. The second variant is treacherous because the attacker never has to talk to the bot. The researchers around Kai Greshake demonstrated it in 2023, among other things against Bing's GPT-4 chat [3].
According to OWASP, the possible consequences range from data disclosure and unauthorised access to commands in connected systems and manipulated decisions [2]. Germany's Federal Office for Information Security (BSI) groups (indirect) prompt injections, jailbreaks and adversarial attacks as “evasion attacks”: they work during live operation, and the model itself stays unchanged [4].
The BSI puts it soberly: the countermeasures can make attacks harder or reduce the potential damage [4]. That is a realistic goal. Anyone promising complete immunity is selling hope, not security.
What went wrong in practice
In December 2023, users got the chatbot of a Chevrolet dealer in Watsonville, California, to “sell” a Tahoe for one dollar. The bot came from the third-party vendor Fullpath; GM said such tools are chosen by the dealers themselves [5]. Fullpath reported more than 3,000 manipulation attempts over one weekend and stressed that the bot had not revealed any dealership data [6]. Whether the dealer ever had to honour the “deal” is not stated in the sources; the bot disappeared from the site soon afterwards [5].
Another case was not an attack but a liability case. In February 2024, the Civil Resolution Tribunal of British Columbia ruled that Air Canada is responsible for wrong information its website chatbot gave about bereavement fares. Among other things, the airline had argued that the bot was a separate legal entity responsible for its own actions. The tribunal treated the bot as part of the website and awarded the customer about $650 plus interest and fees [7]. The amount is small, the principle is not: what your bot says is attributed to you.
approx. $650
And data leaks do not only come from attacks. In May 2023, Samsung banned generative AI on company devices after, according to an internal memo, employees had exposed internal data, including source code, through ChatGPT; three incidents were reported. The memo comes from a Bloomberg report, which TechRadar relays [8].

The real risk: bots that are allowed to act
A bot that only answers questions can embarrass you. A bot that triggers orders, sends emails or reads databases can do damage. OWASP calls this “excessive agency”: damaging actions in response to unexpected, ambiguous or manipulated model output. Triggers include hallucinations as well as direct and indirect prompt injection; the root causes are too much functionality, too broad permissions and too much autonomy [9]. The sensible principle is to treat the model like an untrusted user [2].
Checklist: how to protect your chatbot
- Least privilege: the bot gets only the tools and data access it truly needs, read-only wherever possible. Permissions are enforced by the system behind it, not by the model [9].
- Human in the loop: consequential actions such as commitments, payments, deletions or sending messages are confirmed by a person, or at least explicitly by the user [2][4][9].
- Check the answers: expected formats are validated in code, inputs and outputs are filtered [2]. Prices and legally relevant commitments never come from the bot alone.
- Separate foreign content: documents and web pages from third parties are labelled as untrusted and filtered [2][4].
- Test regularly: targeted adversarial tests before launch and at fixed intervals afterwards [2].
- Log and limit: logs, monitoring and rate limits do not prevent abuse, but they limit the damage [9].
- Clear rules for staff: which data may go into which external AI services? The Samsung case shows how quickly that goes wrong [8].
In November 2025, the BSI published “Evasion Attacks on LLMs – Countermeasures in Practice” for developers and IT managers, together with a checklist. It names, among other things, precise system prompts, filtering harmful content from third-party documents, and explicit user confirmation before the model executes functions [4]. This largely matches the list above.
Sources (9)
- [1]OWASP Top 10 for LLMs and Gen AI Apps 2025OWASP GenAI Security Project · January 1, 2025
- [2]LLM01:2025 Prompt InjectionOWASP GenAI Security Project · January 1, 2025
- [3]Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt InjectionGreshake et al., arXiv:2302.12173 · February 23, 2023
- [4]BSI presents measures against evasion attacks on large language models (press release, German)Federal Office for Information Security (BSI) · November 10, 2025
- [5]Chevy dealer's AI chatbot allegedly recommended Fords, gave free access to ChatGPTThe Autopian · December 18, 2023
- [6]There were more than 3,000 attempts to hack a dealership's AI chatbot this weekendThe Autopian · December 1, 2023
- [7]BC Tribunal confirms companies remain liable for AI chatbot-created information (Moffatt v. Air Canada, 2024 BCCRT 149)Torkin Manes LLP · February 1, 2024
- [8]Samsung bans ChatGPT use after employee leakTechRadar (citing Bloomberg) · May 2, 2023
- [9]LLM06:2025 Excessive AgencyOWASP GenAI Security Project · January 1, 2025
How much of this is in your day-to-day?
In a free initial consultation we look together at what AI can already do in your business today – and where it has no place.
Book an initial consultation

