If model inversion is the attack where you interrogate a system until it starts leaking what it learned, then pipeline compromise is the attack where you don’t bother interrogating the model at all. You simply change what gets built.

That’s the quiet truth about AI security: the model is only as trustworthy as the machinery that produces it. And the machinery is bigger than most people assume.

The model development pipeline is the full chain that turns raw data into deployed behaviour—collection, preprocessing, feature engineering, training, evaluation, packaging, deployment, and retraining. It looks like “engineering” on a diagram, but it behaves like a supply chain. It crosses teams. It crosses platforms. It crosses trust boundaries. And it’s full of artefacts attackers would love to control: datasets, labels, notebooks, feature stores, training code, weights, evaluation reports, container images, API keys, and deployment manifests.

Why AI pipelines are different

Traditional application security is built around a reassuring idea: code is the thing you ship, and you can reason about it. With AI, the behaviour you ship is a function of code and data and configuration and compute environment and a set of training-time decisions that often live in notebooks and tribal knowledge.

That data-driven nature changes the risk. A poisoned dataset isn’t just “bad input”; it becomes part of the product. A slightly biased label set isn’t just an analytics problem; it can become a compliance and harm problem. And the “build artefact” isn’t a binary you can diff easily—it’s a set of weights that encode decisions you may not be able to fully explain.

AI also raises the stakes because models increasingly sit in high-consequence workflows: finance, security decisions, healthcare, hiring, fraud, operational automation. The failure mode isn’t only downtime; it’s systematic wrongness at scale.

Then there’s the supply chain reality. AI pipelines tend to be stitched together from multiple tools and parties: data providers, SaaS platforms, open-source packages, model hubs, internal teams, external consultancies. That complexity means compromise doesn’t always look like a breach. Sometimes it looks like a dependency update. Or a dataset refresh. Or a “minor change” to an evaluation harness.

Finally, models are dynamic. Even if you “lock down deployment”, retraining, fine-tuning, and drift management mean your production behaviour can change over time. That makes pipeline security a continuous discipline, not a one-time release gate.

Where the real threats live

The most useful way to threat model the pipeline is to stop thinking about “the model” and start thinking about control points: where an attacker could change inputs, change transformations, change evaluation, or change what gets deployed.

Data poisoning is the obvious one, and you’ve covered it already. But it’s worth reframing it here as a pipeline problem: poisoning succeeds when provenance is weak, ingestion is blind, and dataset versioning is poor. If you can’t prove what data produced a model, you can’t prove anything else either.

Adversarial attacks often get described as “inputs crafted to fool the model”, but pipeline security has its own adversarial shape: attacks against the evaluation harness. If an attacker can manipulate tests, metrics, or benchmark datasets, they can make a degraded or backdoored model look “good enough” to ship. That’s a governance failure masquerading as model performance.

Unsecured storage is another silent killer. Pipelines generate intermediate artefacts that often get treated as temporary and therefore less protected: raw data extracts, feature matrices, cached embeddings, training checkpoints. In practice, “temporary” becomes “permanent enough to be stolen”. Those artefacts can contain sensitive data, proprietary insights, and—depending on the system—credentials and configuration that grant access elsewhere.

And then there’s the classic set of issues that don’t go away just because the system is “AI”: unvalidated input paths, excessive permissions, leaked secrets, weak access control, and inadequate monitoring. In a pipeline, those issues don’t just create an incident; they can create persistent compromise because the compromised artefact is reused and propagated across training and deployment.

The other failure mode is organisational: nobody owns the pipeline end-to-end. Data teams own ingestion. ML engineers own training. Platform teams own CI/CD. Security owns… “security”. That split ownership is where attacks hide, because each team assumes someone else is covering the boundary.

What “secure pipeline” looks like (in plain terms)

A secure model development pipeline is one where you can answer three questions quickly and confidently.

First: What produced this model? That means dataset lineage, dataset versioning, and reproducible training runs. Not for academic purity—for incident response. When something goes wrong, you need to know what changed.

Second: Who can change what? That means least privilege across data stores, feature stores, training environments, registries, and deployment mechanisms. It means separating duties so that no single actor can silently push a model from “trained” to “deployed” without review.

Third: How would you notice compromise? That means monitoring pipeline actions, not just model outputs. Alerts for unexpected dataset changes, unusual training runs, new dependencies, changed evaluation scripts, or model registry changes outside normal workflows. In other words: treat the pipeline like production, because it effectively is.

The practical controls follow from that: - Strong identity and access management for every stage of the pipeline, including service accounts used by automation. - Integrity controls on artefacts: signed training code, signed model packages, and controlled promotion between environments. - Secrets management designed for pipelines, because notebooks and build logs are where secrets go to die. - Evaluation that is hard to game: multiple metrics, holdout datasets, regression tests for known bad behaviours, and review gates that focus on behavioural change, not just “accuracy improved”. - A model registry and deployment path that behaves like a release pipeline, not a shared drive.

The point of all this

AI security conversations often gravitate toward the model as if it’s a mystical object. In reality, the model is an output of a pipeline. If the pipeline is weak, the model will be untrustworthy no matter how many safety layers you bolt on later.

Securing the model development pipeline is how you prevent the most damaging class of AI failure: the one where you deploy a system that looks legitimate, performs well enough to pass casual checks, and is quietly shaped by an attacker’s hand.

Until the next post, have fun.

Reference Links - OWASP Top 10 for Large Language Model Applications: https://owasp.org/www-project-top-10-for-large-language-model-applications/ - Adversarial Attacks and Defenses in Deep Learning: https://www.sciencedirect.com/science/article/pii/S209580991930503X - Threat Modeling LLM Applications: https://aivillage.org/large%20language%20models/threat-modeling-llm/ - Securing the AI Pipeline: https://www.mandiant.com/resources/blog/securing-ai-pipeline