Eclypsium: an AI model's weights can be altered in memory after passing integrity checks

Security firm Eclypsium has published a report warning that attackers can sabotage AI models by tampering with the infrastructure that supports them, such as the firmware and software of the machines they run on. According to the Cybersecurity Dive article by Eric Geller, this is an overlooked area of risk. It calls for safeguards from the major AI labs and oversight from the companies that use these systems.

The report argues that once a model is running, its runtime state is controlled by a hierarchy of software, firmware, memory controllers, management processors and input/output devices. Each of those layers is an opportunity to skew the model and its outputs if an attacker manages to tamper with it. Eclypsium gives an example: a subtle runtime change could nudge the model toward weakening certificate validation, choosing an insecure dependency or generating a specific insecure code pattern.

The easiest target is the host machine. It relies on an operating system, a kernel, graphics drivers and other hardware and software components. These machines are vulnerable to the same malware as any other computer, including rootkits that persist below the operating system level by accessing the device's firmware. Third-party components open up other avenues for tampering with models. These include firmware update services, AI orchestration agents and management controller hardware. The report notes that compromising the management plane can allow attackers to modify firmware, change isolation controls, directly control connected devices or gain other privileged access to the workload.

Chipmakers use data processing units (DPUs) to handle part of the work involved in running an AI model. Because they perform such important processing tasks, they can also give attackers a way to alter the model's operating state. Eclypsium, which carried out this research in collaboration with Nvidia, stresses that this same position is what makes a well-protected DPU valuable for independent monitoring and policy enforcement.

The report describes a demonstration of a tampering technique that alters the operating state after traditional security checks have been carried out. This leads the system to consider its integrity verified. The researchers explain that the model file does not need to change: the container, the model hash and the signed artifacts on disk can remain valid while certain weights are modified in memory at runtime. According to the report, systems from AMD, Intel and Arm, as well as Nvidia's, are vulnerable to these techniques.

As recommendations, Eclypsium urges chipmakers to follow Nvidia's lead and use DPUs to enforce security policies independently of the AI system itself, while also properly protecting the DPUs. It advises companies to maintain accurate inventories of all components of their AI systems, bring them into their vulnerability management processes and carefully scrutinize third-party software running on critical hardware such as DPUs.

The researchers sum up the underlying idea as follows: every AI infrastructure stack contains privileged components, each with its own firmware, software, update process and supply chain. That is why trustworthy AI requires being able to see and verify those components, rather than assuming the model's boundary begins at the container.

The practical takeaway for security teams is that verifying a model file's hash or signature is not enough. If weights can be changed in memory once verification has passed, the runtime state and the entire chain of components around it must also be monitored and verified. The article provides no further technical details about the demonstration or figures on real-world exposure.