I have observed a profound shift in the cybersecurity landscape over the past few years, the threat landscape is changing rapidly. The integration of Artificial Intelligence (AI) into core business functions is no longer futuristic; it’s a present-day reality. However, with this incredible innovation comes a fresh wave of sophisticated risks. This is precisely why the demand for expertise in threat modelling and risk assessments specifically tailored to AI workloads is skyrocketing – and rightly so.

So what is the difference? Traditional threat modelling, while foundational, often falls short when applied directly to AI systems. Why? Because AI introduces entirely new attack vectors and vulnerabilities that simply don’t exist in conventional software. We’re not just dealing with misconfigurations or code injection; we’re contending with attacks that exploit the very learning process or the decision-making logic of the AI itself.

Let’s break down some of these AI-specific threats that demand our keenest attention during threat modelling and risk assessment:

1. Adversarial Attacks: The Art of Deception

This is perhaps the most widely discussed AI threat, and for good reason. Adversarial attacks involve subtly manipulating input data to trick an AI model into making incorrect predictions or classifications. These perturbations are often imperceptible to the human eye, yet devastatingly effective against a machine learning model.

• Examples: Adding carefully calculated noise to an image to make a self-driving car misidentify a stop sign, or subtly altering audio commands to bypass voice authentication.
• Threat Modelling Focus: Identifying critical input channels, understanding the model’s sensitivity to perturbations, and assessing the potential impact of misclassification in real-world scenarios. We need to think about where an attacker might inject such inputs and what the consequences of a wrong decision could be.

2. Model Poisoning: Tainting the Well

Model poisoning, also known as data poisoning, is a more insidious attack that occurs during the training phase of an AI model. Malicious actors inject corrupted or misleading data into the training dataset, causing the model to learn incorrect patterns or biases. The aim is to compromise the model’s integrity or performance over time, often to enable future attacks or degrade service.

• Examples: Injecting biased data into a loan application fraud detection system to allow specific fraudulent patterns to be overlooked, or adding seemingly benign but malicious samples to a malware detection dataset to create blind spots.
• Threat Modelling Focus: Analysing data provenance and integrity across the entire data pipeline, from collection to storage and processing. We must identify trust boundaries in data sourcing, assess the impact of data quality on model behaviour, and consider methods for detecting anomalous training data.

3. Prompt Injection: The New Code Injection for LLMs

With the explosive growth of Large Language Models (LLMs), prompt injection has emerged as a significant and novel threat. This involves crafting prompts (inputs) that manipulate the LLM into bypassing its intended safeguards, revealing sensitive information, or performing unintended actions. It’s akin to the SQL injection of the AI world.

• Examples: Getting a customer service chatbot to reveal internal system information, making an LLM generate harmful or biased content it was programmed to avoid, or forcing it to execute unauthorised commands via integrated tools.
• Threat Modelling Focus: Deeply understanding the LLM’s capabilities and integrations, identifying potential leakage vectors (e.g., through retrieval-augmented generation), assessing the impact of uncontrolled output, and devising robust input validation and sanitisation strategies for prompts.

4. Data Leakage (Inference Attacks): Unveiling the Secrets

AI models can inadvertently reveal information about the data they were trained on, leading to data leakage through various inference attacks. These attacks don’t directly compromise the model’s functionality but instead aim to extract sensitive information.

• Examples: A membership inference attack might determine if a specific individual’s data was part of the training set, or a model inversion attack could reconstruct aspects of training data (like faces from a facial recognition model’s output).
• Threat Modelling Focus: Assessing the sensitivity of the training data, understanding potential privacy implications of model outputs, and implementing privacy-preserving techniques like differential privacy or homomorphic encryption where appropriate.

The Way Forward: A Specialised Approach

To effectively manage these AI-specific risks, our threat modelling and risk assessment processes must evolve:

• AI-Specific Methodologies: While general frameworks like STRIDE or PASTA still provide a good starting point, they need augmentation with AI-specific considerations. Frameworks like MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) are invaluable here.
• Cross-Functional Expertise: This isn’t just a security team’s job. Successful AI threat modelling requires close collaboration with data scientists, machine learning engineers, and ethical AI specialists. Each brings a unique perspective on how the AI operates and can be exploited.
• Continuous Assessment: As AI models are often iterative and adaptive, threat modelling shouldn’t be a one-off event. It needs to be integrated into the MLOps lifecycle, regularly reviewed as models are updated, retrained, or deployed into new contexts.
• Quantifying AI Risk: Translating these novel technical risks into quantifiable business impacts (financial loss, reputational damage, regulatory fines) remains crucial for communicating effectively with C-level executives.

The growing demand for specialists in AI threat modelling and risk assessment highlights a critical need in our industry. For security architects, this represents both a significant challenge and a tremendous opportunity. By developing deep expertise in these areas, we can ensure that the transformative power of AI is harnessed securely, enabling innovation without compromising our organisations’ integrity. It’s about being proactive guardians of this new digital frontier.