Flide: engine claims 9.5x faster decision-model inference than vLLM

A team posting as unikcc published Flide, an Apache-2.0 inference engine for decision models, and introduced it on Hacker News on 9 November 2026 (73 points at the time of writing; the thread is at news.ycombinator.com, item 50421155). Decision models answer a list of categorical questions about one input, such as a support ticket, in a single pass. The README says Flide serves them up to 9.5 times faster than vLLM and SGLang while giving the same answer the model gives to each question alone.

What the authors claim

The README describes Flide as built for the Jev-style decision-model API and says it keeps one shared state for the input, then forks each question from it. The arXiv paper (Li, Song, Fei, Hsu and Lee, 2 November 2026) says it also tracks branches through the recurrent Gated DeltaNet layers used by hybrid models, and that its answers stay closest to a float32 reference in total variation distance.

The README's table lists speed-ups per model over the faster of vLLM and SGLang, as median latency on one H200. It ranges from about 1.0x to 28.1x depending on the model and workload; the 28.1x entry is Senova-4B on the MultiRC workload. Options include a session cache so an agent's next step reads only what is new, and an FP8 mode for Qwen3.5. The README lists more than 30 model families, including Perplexity's pplx-decider, and says it has been tested on Linux only.

Where the number comes from

In the thread, one of the authors said the 9.5x figure is Qwen3.5-0.8B on QuALITY, a long article of about 5.8k tokens with roughly 18 questions. The more questions share a context, the bigger the gain. With a single question they reported a range from a near tie, for a 27B model on a long input, to more than 3x for a small model on a short input, because the whole forward pass runs as one prerecorded CUDA graph.

Pushback in the thread

The strongest objection was about fit. A commenter said the reverse situation, the same question asked about many inputs, is much more useful in practice. The author agreed the current setting may not help much there and said the question could be placed first so states share it. Another commenter replied that a voice-assistant style workload, one input with many questions, does fit.

A second commenter, Tostino, said a whole new engine is not needed: they modified vLLM to reuse the shared context and reached roughly the same speed-up on the right workloads. The author replied that vLLM only caches Gated DeltaNet state at block boundaries, which is why they built their own, and that Flide keeps answers bit-identical at any load. Neither claim was independently tested in the thread.

What it leaves out

The README gives no cost comparison and no end-to-end task accuracy; it reports latency and closeness to a float32 reference. The benchmarks are the authors' own, run on a single H200. A GLiDE 9B+ model is not supported yet, and the author said its bidirectional attention would limit the gain.

What to do with it

If your team already runs a small decision model to route tickets or screen agent turns, count how many questions you ask per input. Many per input is the case Flide targets. One question per input is where the authors' own range is lowest. Run your workload on a stock vLLM setup first, then on Flide, and compare latency and answers on the same data before changing infrastructure.