Why AI Prompt Injection Is the New Social Engineering

Generative AI tools are becoming a major part of many business operations. Large language models (LLMs) like ChatGPT and Claude are often used as human-like agents, capable of interpreting complex prompts and delivering fast, intelligent responses. But as organizations rely more heavily on these systems, they face a new set of challenges that mirror those posed by attacks on human employees.

Just as phishing emails and social engineering tactics are designed to manipulate people, prompt injection attacks aim to deceive AI models. This creates a growing risk that must be addressed with the same level of scrutiny applied to other vulnerabilities businesses face.

AI prompt injection attacks work by feeding carefully crafted inputs into LLMs – either directly or indirectly – with the goal of manipulating their behavior. This means that instead of following intended instructions or safeguards, the model responds to the attacker’s commands. Hackers may use these inputs to extract sensitive data, bypass content filters or trigger unauthorized actions within connected systems.

Because LLMs are often trusted to handle internal processes or interface with other tools, the consequences of successful prompt hacking can be severe. Attackers could leak confidential business information, initiate malicious workflows or expose systems to further compromise. In some cases, compromised outputs may even be used to spread misinformation or assist with follow-up attacks like phishing or ransomware.

As these models become more embedded in enterprise networks, treating them as secure endpoints is essential. Failing to do so could leave serious gaps in a business’ overall cybersecurity posture.

AI systems – and particularly LLMs – are vulnerable to many of the same tactics used in traditional social engineering attacks, which are a threat every cybersecurity pro should be familiar with. Indeed, according to Avast, almost 90 percent of attacks blocked in the first quarter of 2024 used these techniques in some form.

Just as tactics like phishing emails are crafted to mislead or pressure human employees into taking risky actions, prompt injection relies on carefully worded inputs that aim to manipulate the behavior of the target – in this case, generative AI platforms.

However, while humans can be trained to be suspicious of messages they receive, it is harder for AI tools to spot these attacks. Unlike humans, LLMs cannot assess the intent behind a prompt or question. They interpret all inputs as legitimate instructions and respond accordingly, even when the input is designed to bypass safeguards or extract restricted information. If a prompt appears to contain conflicting or deceptive language, the model lacks the context or critical reasoning needed to identify it as suspicious.

This makes LLMs highly susceptible to manipulation. Whether through direct commands or more subtle phrasing, attackers can trick systems into violating their own rules. Without human-like judgment or a sense of intent, AI models remain exposed to a form of digital social engineering that is difficult to detect without strong oversight.

Because of this, it can be useful to treat these platforms as more like digital employees than software when planning an LLM cybersecurity strategy. After all, these tools do function in a human way as they receive instructions using natural language, process requests and generate responses in real-time.

Attackers often use familiar manipulation techniques to target both people and AI systems. Prompts may appear harmless or authoritative, encouraging the model to carry out unintended actions. While the target is a machine, the methods used rely on persuasion rather than exploiting software vulnerabilities.

To defend against this, organizations must apply the same mindset they use to protect employees. Just as phishing training helps staff recognize suspicious requests, LLMs require safeguards to detect deceptive prompts. Behavioral monitoring, usage restrictions and prompt validation tools are all critical in protecting these systems.

Protecting LLMs from prompt injection requires more than technical solutions – it needs an understanding of how they can be manipulated. Because these attacks rely on an almost psychological use of language, many of the same tactics used to defend against social engineering can be applied to AI systems.

By using adaptive security practices based on proven, human-focused training, firms can better defend against AI-specific threats. Key defense strategies to think about as part of this include:

• Monitoring prompt behavior: Track user interactions to identify abnormal patterns or repeated probing attempts.
• Filtering inputs and outputs: Use language models or rule-based systems to screen for suspicious commands and prevent data exfiltration by analyzing responses to see if they contain sensitive information.
• Testing AI responses: Regularly run penetration testing activities that simulate prompt injection attacks to identify weaknesses, much like phishing simulations that test user responses to suspicious emails.
• Restricting access: Limit who can interact with the model and under what conditions, following principles of zero trust and least privilege.
• Establishing audit trails: Log all interactions for forensic review to spot any patterns, identify how any data breaches have occurred and support continuous improvement.

LLMs are powerful tools, but they introduce new risks that resemble social engineering more than software flaws. Building safeguards against prompt injection is essential to ensure they are factored into modern cybersecurity strategies from the start. To do this, security teams must treat these systems as part of the human attack surface in order to fully protect them.