Fly.io recounts patching its entire fleet in 10 hours over kernel bug CVE-2026-99999

Fly.io has published a post-incident analysis of October 16, 2026, when it patched and rebooted its entire server fleet in response to a Linux kernel flaw. The full rollout took 10 hours. The article, written by Daniel Botha and published on November 6, 2026, combines the technical account with lessons on incident communication and security recommendations for anyone deploying software with agents.

The flaw, identified as CVE-2026-99999, was announced on the morning of the 15th by the Linux kernel CVE team and by pwn.ai. Fly.io learned of it that night and began preparing mitigations. It stems from 'checksum offload', a Linux mechanism that avoids immediately computing checksums for protocols such as TCP and UDP. Instead, the kernel stores metadata indicating where to read from and where to write the checksum, and computes it only when needed. This works the same way with virtual machines: the virtual network card accepts the metadata and passes it to the host. The offload code included checks to reject invalid metadata, but for IPv4 they were wrong.

To exploit it, according to the article, an attacker must do three things: point the checksum at another header field, choose a message that produces the desired checksum, and force the host to compute it in software. As a result, the host receives the packet, parses it and decides where it goes, then applies the checksum in the wrong place. The attacker thus controls two bytes of an IPv4 header the host had already trusted.

Those two bytes open two avenues. The first is a firewall bypass: a rewrite of the source or destination IP happens after the host has already applied routing and firewall rules. The second is an out-of-bounds read: rewriting the header length from the smallest IPv4 header to the largest causes the host to treat adjacent memory as part of the packet and capture 32 bytes as options on the outgoing packet. That memory may contain host packets, kernel structures and packets from the neighboring virtual machine. Detection is hard: firewall-bypass packets can be well-formed, and out-of-bounds-read packets probably carry an incorrect header checksum, but by the time that is visible it is too late. Existing XDP rules were of no use because the rewrite happens after them. Disabling checksum offload on the host required a reboot, and if a reboot was needed, it would be done with a patched kernel.

As for potential impact, Fly.io considers it unlikely that customer data was exposed, based on network activity observed during the rollout. It acknowledges that, in theory, between disclosure and the last reboot a malicious application could have read small fragments of traffic from other applications on the same physical host, though it has no evidence that this happened. It recommends that anyone who ran workloads between October 15 and 17 rotate, as a precaution, any credentials sent over HTTP or another unencrypted protocol, including unencrypted traffic over the 6PN private network.

On the operational side, each host takes 15 to 20 minutes to reboot, and all the Fly Machines it hosts are down during that time. Because placement distributes machines randomly across hosts, they could not reboot more than one at a time without taking down every machine of some applications. Single-machine applications were going to go down regardless. They therefore planned around Managed Postgres, splitting each region into waves so that no cluster had both of its nodes in the same one. Even so, their largest region, iad, required 17 waves.

There were surprises: most hosts came back within the announced estimate, but some did not. Some rebooted without essential software and had to be reinstalled, others got stuck in the BIOS, and a four-node chassis in gru lost a power supply mid-incident with no spare on site. The machines on those hosts remained down until they were restored on the 20th. According to the company, no Fly Machines or customer data were lost.

Communication was the other sensitive issue. At 16:22 UTC they posted an emergency maintenance notice on the status page and began notifying their largest customers. The notice said 'maintenance', not security, and did not name the CVE. Customers voiced their frustration, which the company considers understandable: same-day emergency maintenance that reboots an entire application looks like a rushed, unexplained patch. The company explains that, because the flaw had been disclosed along with a detailed description of how to exploit it, it chose to avoid publicly confirming that its hosts were vulnerable at that point. As a result of the post-incident review, its head of security reworked the incident communication plan: previously, whoever was in the incident channel decided the approach; now that responsibility lies with the leadership team, with precise instructions on who says what, where and when.

The article closes with general recommendations for a world in which, according to the authors, there are more bugs, they are found faster and they are exploited more easily. First, encrypt traffic between your own services, even on a private network; the only advice they had to give after this CVE was to rotate whatever had been sent in plaintext. Second, give agents credentials scoped to the job and with an expiry: an agent's environment is the first thing a successful exploit reads, and a token that can only deploy one application for one hour is preferable to an organization-wide or non-expiring one. If possible, keep credentials out of agents' reach altogether. Third, assume every boundary leaks a little: hardware virtualization is the strongest barrier, but for a day and a half a guest could extract 32 bytes at a time, so it is best not to put an agent, its secrets and production data in the same place. Fourth, treat whatever the agent reads as hostile and give it only the permissions you would grant the author of the worst web page, issue or email it might read. Fifth, rehearse credential rotation: you need to know where your credentials are and be able to locate them quickly.

The authors acknowledge that none of this makes anyone invincible, but argue that these are firm constraints that help when something goes wrong. They close on a resigned note: none of this was fun, and they will probably have to do it again soon.