Same prompt, different answer: a study proves neither vLLM nor SGLang guarantees determinism in AI

By Zendoric · October 9, 2026.

A preprint published on arXiv on September 30 (Jianxing Qin, Alexander Du, Danfeng Zhang, Matthew Lentz and Danyang Zhuo) puts to the test a promise that several AI inference engines already advertise as solved: that the same model, with the same prompt and the same sampling configuration, should always return the same answer. According to the authors themselves, neither vLLM's 'batch-invariant mode' nor SGLang's 'deterministic mode' —two of the most widely used open-source engines for serving large models in production— delivers on it in every case: their tests found that both produce different logits (the raw scores the model assigns to each candidate word before converting them into probabilities) under certain execution variations.

The cause lies not in the model but in the infrastructure that serves it. To be efficient, an inference server groups requests from different users into 'batches' whose composition varies depending on who happens to be asking something at that moment; it splits long prompts in different ways; it decides dynamically when to process the input text (prefill) and when to generate word by word (decode); and it reuses, discards or recomputes the key-value cache (KV cache), the memory that avoids recomputing attention over all the preceding text at every step. None of these decisions should alter the result, but in practice they do, because floating-point operations on GPUs are not strictly associative: adding the same numbers in a different order —something that changes depending on how the batch is grouped— can give a slightly different result, and that margin propagates token by token.

In response, the authors present Vosti, an inference engine designed from scratch with two rules: it chooses which GPU 'kernel' to use (the specific routine that runs the computation) without looking at the engine's state at that moment, and it ties each cached value to its logical token prefix rather than to its physical position in memory. The underlying difference, however, is not one of engineering but of method: instead of merely testing the system with more test cases, they verify it formally. They use Verus —a deductive verification tool common in Rust systems projects— to prove by induction that task scheduling and the paged, prefix-shared KV cache preserve the output logits, and an analyzer over Triton (OpenAI's language for writing GPU kernels, widely used in the PyTorch ecosystem) to prove that the chosen kernels give bit-for-bit identical results regardless of batch size, query length or how the cache is laid out. The result, according to the paper itself, is bitwise determinism across all the variations tested, with performance on generation-heavy workloads comparable to vLLM's invariant mode.

It is an invisible piece of engineering —the paper reached Hacker News with a single point and zero comments, which gives the measure of how specialized the topic is even for a technical audience— but the problem it tackles is no small matter. Reproducibility is the foundation of any serious evaluation of a model: if the quality or safety benchmarks we use to compare models (something we at Zendoric keep repeating matters more than the marketing) can vary depending on how the server grouped requests that afternoon, the very notion of 'measuring' suffers. The same goes for debugging agents in production: if a system misbehaves once, you need to be able to reproduce that exact run to understand why, and non-determinism makes that impossible.

Our reading is that this kind of work —dry, headline-free, designed so that nobody has to think about it— is precisely what is missing as AI starts to move into areas where traceability is not optional: medical diagnosis, credit decisions, legal advice. We have pointed out before how in healthcare the discourse is already beginning to shift the burden of proof onto those who do not use AI; well, that responsibility can only be demanded seriously if the system behind it can be audited, and auditing requires that the same question produce the same answer, provably and not merely as observed across a set of tests. Vosti is, for now, an academic prototype, not something any cloud is going to deploy tomorrow. But the gap between what production engines advertise ('deterministic mode') and what they actually verify is exactly the kind of gap between narrative and evidence worth watching as the AI frontier advances toward uses where getting it wrong —or simply being unable to explain why it was right— carries a real cost.