PoisonGPT and the Tampered-Model Demonstration

By Satwik · May 25, 2026

In 2023, the security company Mithril Security published a demonstration they called PoisonGPT. They took an open-source model, surgically edited it so it would answer one specific factual question with false information while behaving normally on everything else, and then uploaded the tampered model to Hugging Face under a name resembling a legitimate project. Their point was that a user downloading the model would have no easy way to detect the implant, since standard benchmarks and casual testing would not surface the single planted falsehood.

The edit used a model-editing technique to change a targeted association inside the weights, leaving the rest of the model's behavior intact. The result was a model that would confidently state a specific piece of misinformation on cue.

Why it matters: PoisonGPT dramatized the model-supply-chain problem. Weights are opaque binary artifacts, and a maliciously modified model can pass ordinary evaluation while carrying a hidden payload. As organizations increasingly download pretrained models from public hubs, the integrity and provenance of those artifacts become a genuine security concern, not just a quality one.

The defensive lesson is provenance and verification. Obtain models from trusted, verifiable sources, check cryptographic hashes and signatures where available, prefer publishers with accountable identities, and evaluate models on your own held-out and adversarial tests rather than trusting a hub name. Treat a downloaded model like any other unverified binary in your supply chain, because that is what it is.