Multimodality: From CLIP to GPT-4o and the New Input Channels
By Satwik ยท May 29, 2026
CLIP aligned images and text in a shared embedding space, and native multimodal models like GPT-4o fused vision, audio, and text into a single real-time system. Every new input modality is also a new injection channel, and images in particular smuggle instructions past text-only defenses.
From alignment to fusion
CLIP, from Radford and colleagues at OpenAI in 2021, is the hinge. It trained an image encoder and a text encoder jointly on hundreds of millions of image-caption pairs using a contrastive objective: pull matching image-text pairs together in a shared embedding space and push mismatched pairs apart. The result was a model that could perform zero-shot classification by comparing an image's embedding to the embeddings of candidate text labels, with no task-specific training. More importantly, CLIP produced a shared representation where images and text live in the same space, which became the connective tissue for a generation of multimodal systems, from image generation to visual question answering.
The trajectory from there was toward fusion. Early vision-language models bolted a vision encoder onto a language model with a projection layer, letting the LLM read image features as if they were tokens. By 2024, native multimodal models like GPT-4o were trained across modalities from the start, processing text, images, and audio in a single model with a single set of weights. The "o" for omni signaled the shift: not a text model with vision attached, but one model whose input and output space spans modalities, fast enough for real-time voice conversation.
Why it mattered
Multimodality removed the requirement that the world be transcribed into text before a model could reason about it. A model that sees a chart reads the chart. A model that hears a voice hears tone and timing. A model that watches a screen can operate an interface. This is the enabling capability behind computer-use agents, visual assistants, and real-time voice interfaces, and it broadened the set of tasks a single model can do more than any single technique of the reasoning era.
Native fusion also improved grounding. A jointly trained model can reason about the relationship between what it sees and what it reads rather than shuttling between separate systems, which reduces the seams where information gets lost. The unified representation is why these models feel qualitatively more capable than pipelines of specialized components.
The security angle: every modality is an input channel
Our central claim is simple. Each modality a model accepts is an additional untrusted input channel, and defenses built for text do not automatically cover the others.
Images are the clearest case. A multimodal model reads text that appears inside an image. That means an attacker can place instructions in a picture, as visible text, faint low-contrast text, or text tucked in a corner, and the model may follow them. This is prompt injection through pixels, and it routes around every text-based input filter because the payload never exists as input text. A screenshot, a scanned document, a meme, or a photo of a sign can all carry an injected instruction. In agentic settings where a model looks at a web page or a screen to decide what to do, image-borne injection is a direct path to hijacking the agent.
The attack surface is richer than visible text. Because vision encoders map pixels to embeddings, adversarial perturbations, small, carefully computed pixel changes imperceptible to a human, can push an image's representation toward an attacker-chosen meaning. The lineage runs straight back to CLIP: contrastive image-text models are known to be manipulable in embedding space, and that manipulability is inherited by systems built on similar encoders. An image can look benign to a person and read as an instruction, or as a different image entirely, to the model.
Audio extends the same logic. A voice interface accepts spoken instructions, and speech can carry injected commands, potentially embedded in media a user did not knowingly submit as a command. Real-time audio raises the stakes because there is less opportunity for a human to inspect the input before the model acts on it.
There is also a cross-modal laundering effect. Content that a text filter would block can be reintroduced through another modality: rendered into an image, spoken as audio, and read back by the model as instructions. The safety tuning and input filtering invested in the text channel does not transfer for free to pixels and waveforms, so a defense-in-depth posture built only around text has blind spots exactly where the new channels enter.
Practical guidance
Extend the untrusted-input assumption to every modality. If the model can see it or hear it, treat it as potentially adversarial, not as neutral data. Apply the same provenance, isolation, and least-privilege discipline to image and audio inputs that you apply to retrieved text.
Be especially careful when a multimodal model drives an agent. An agent that reads screens or documents to decide actions inherits image-borne injection as a direct control path, so the action-boundary controls, confirmation, sandboxing, allowlists, matter even more.
Red-team across modalities. Test for text-in-image injection, low-contrast payloads, and adversarial perturbations, not just textual jailbreaks. The capability that makes these models useful, reading the world directly, is the same capability that lets the world instruct them, and the instruction can arrive in any channel the model can perceive.