A junior developer at a rapidly expanding fintech startup, rushing to meet a launch deadline, accidentally exposed an API key in a public GitHub repository. Within hours, the key was scraped, combined with others, and shared on Discord among a hidden group of digital mischief-makers.

By the time the CTO noticed unusual activity, the damage was already done: thousands of dollars in LLM computing expenses and potentially sensitive business data exposed to the public.

This isn’t a hypothetical scenario—it reflects a pattern of incidents that have occurred repeatedly in the first half of 2025.

Figure 1. AI at Risk: LLM Hijacking & Prompt Leaks Drive Data Breaches

In January, the AI community faced a new wave of cyberattacks that went beyond typical data leaks. DeepSeek, a popular Chinese LLM, had its keys stolen, resulting in 2 billion tokens being exploited by unknown attackers. Figure 1 shows AI at Risk: LLM Hijacking & Prompt Leaks Drive Data Breaches.

Shortly after, OmniGPT, a popular AI chatbot aggregator connecting users to multiple LLMs, experienced a major breach, exposing over 34 million user messages and thousands of API keys publicly.

If you’re relying on these AI systems with your data, you’re witnessing that trust being broken in real time.

The new strategy: hijack the mind, not just the data

For years, the main fear was hackers stealing files or demanding ransom for data [1]. But LLM hijacking is a different, more alarming threat — attackers target the very "brains" behind your apps, research, and business.

They scour GitHub, cloud settings, and even Slack channels for exposed API keys. Once found, they can create shadow networks, resell access, gather more data for further attacks, or rack up huge service bills.

In the DeepSeek case, attackers used reverse proxies to hide their activity, allowing multiple bad actors to abuse stolen keys without detection. This can lead to massive unauthorized AI usage charges and the exposure of private data online.

The threat grows worse with system prompt leaks. These prompts, which guide GPT behavior, should be secret, but attackers can trick models into revealing them. This exposes the AI’s rules and sensitive info, meaning the AI might be operating under someone else’s control without you realizing it.

Why This Should Concern Everyone

We’re embedding LLMs everywhere—from customer service bots to healthcare, legal research, and even coding systems. Each new integration expands the attack surface, yet our security practices haven’t kept pace. Meanwhile, an underground market for LLM exploits is booming, with stolen keys traded like collectibles and prompt-leak tools becoming increasingly advanced. Hackers are racing ahead, and as we grant these models more autonomy, the potential damage from breaches grows. This is a critical struggle for control, trust, and the future of automation.

Are We Racing Ahead at the Cost of Our Security?

Treating AI as “just another tool” is a serious error. You can’t simply integrate these systems and expect to add security afterward. Unlike predictable spreadsheets or file servers, LLMs are dynamic and increasingly autonomous—often making decisions that even their creators can’t fully understand.

If we don’t change direction, we’re on track for a reckoning—losing money and, more critically, trust. The future of AI adoption hinges on whether people see these systems as safe, reliable, and deserving of the power we grant them. Continuing to treat LLMs as black boxes only invites disaster.

What should have changed yesterday — and still needs to now

Here’s my take on what to do:

• Treat API keys like plutonium: rotate them regularly, limit their permissions, and never share them in code, chats, or logs. Stashing keys in Slack is a recipe for disaster.
• Monitor everything in real time. If your AI suddenly starts generating tons of tokens at 3 a.m., you want to catch it before your cloud bill skyrockets.
• Don’t rely solely on the model’s built-in safeguards. Add extra layers by filtering inputs and outputs, always assuming someone will try to manipulate your AI if it’s exposed.
• Red-team your AI—test it rigorously to find vulnerabilities before attackers do.
• Use strict access controls to segregate permissions. Don’t give your chatbot full access to your entire system.

Some vendors are stepping up, like Nexos.ai for centralized LLM monitoring, and WhyLabs and Lasso Security for detecting prompt injection and emerging threats. While not perfect, these tools mark a crucial move toward stronger security in generative AI.

Your AI’s “brain” is at risk—time to take control

LLM hijacking and system prompt leaks aren’t science fiction—they’re happening now, and your organization could be next. AI powers the core of your business, and if left unprotected, it’s vulnerable to exploitation. Relying on hope isn’t a strategy [2]. The future of AI is promising, but only if we address its risks seriously before the next breach turns optimism into regret.

Source:

References:

1. https://cybernews.com/security/llm-hijacking-prompt-leaks-data-breaches/
2. https://www.globalrailwayreview.com/article/203275/your-ai-isnt-safe-how-llm-hijacking-and-prompt-leaks-are-fueling-a-new-wave-of-data-breaches/

Cite this article:

Janani R (2025), Your AI at Risk: LLM Hijacking and Prompt Leaks Drive Rising Data Breaches, AnaTechMaz, pp. 234