DeepSeek opens its inference engine, and llm-d reaches 1.0
DeepSeek publishes the engine behind its API (offline inference only), llm-d 1.0 adds the serving layer around it, and Mistral prices a full voice-agent stack.
Two stories today are really one: the inference stack that runs big models is getting opened up, and the interesting question is which layer you want to own.
1 · Top story
DeepSeek open-sources DeepInfer, but only the offline half
DeepSeek published DeepInfer on GitHub this week. Its README calls it the framework that powers DeepSeek’s large-scale production services, under Apache-2.0. It runs DeepSeek-V4.1-Flash on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, with expert parallelism, speculative decoding and structured outputs.
The release is offline inference only. A KV cache system, prefill/decode disaggregation and load balancing across replicas are “not included”, and those are the parts that decide your cost per token in production. Batch-invariant inference and deterministic sampling are NVIDIA-only, which matters if you want reproducible evals or RL rollouts. The README shows benchmark charts but no numbers in text, so “faster than vLLM” is still a guess. Worth reading for what a second hardware target looks like in the code.
2 · Models & features
Mistral prices a whole voice-agent stack with Voxtral 3
Mistral’s Voxtral 3 page, listed as an October 20 announcement, bundles batch and realtime transcription, text-to-speech with voice cloning, and a voice agent SDK built on LiveKit that runs speech-in, reasoning and speech-out as one streaming pipeline over SIP and WebRTC. The agent API costs $0.04 per minute with Mistral Small 4 for reasoning and $0.08 with any other model; realtime transcription is $0.008 per minute. Mistral claims sub-500 ms realtime transcription and under one second per agent turn.
The price is what makes it testable. The page doesn’t say whether any weights are open, and its language counts disagree (35+, 40+, 50+), so check your languages before committing.
3 · Agents
Eon swaps ten local MCP servers for one gateway
Eon, a cloud backup company, wrote up how it moved its developers’ tools and its autonomous SRE agent onto a single MCP gateway on Amazon Bedrock AgentCore. SSO through Cognito handles identity, Cedar policies check every tool call, and production tools are read-only. The post reports local MCP processes per machine going from 10 to 0 and onboarding from two days to two minutes, across 100+ developers and about 10,000 sessions.
The useful part is the friction list. Pre-indexing tool schemas pushed Cedar policy toward its size quota, which forced a hybrid listing mode and cost some semantic tool search. AgentCore’s OAuth session binding didn’t match what standard MCP clients expect, so Eon wrote a shim. A gateway removes credential sprawl, but you inherit its limits.
4 · Research
An agent’s reaction to prompt injection is still being decided while it thinks
A preprint posted on 16 October replays agent episodes up to the moment an injection is read, then resamples the rest many times. With thinking off, Qwen3’s outcome is mostly fixed at read time (0.91 of the variance lies between states). With thinking on, that falls to 0.62, and single-run labels disagree with a rerun up to 35% of the time. Thinking lowered attack success for Qwen3 but raised it for two other models the abstract doesn’t name.
The consequence is for testing: a single pass over an injection benchmark can pass an agent that fails on the next run, and a monitor that judges only the read step misses the decision forming afterward. The results come from four AgentDojo suites, and the abstract doesn’t mention released code.
5 · People worth following
Hamel Husain on tuning the threshold of an eval classifier
Hamel Husain and Antaripa Saha tested cheap “decision models”, classifiers that return a probability for pass or fail, as replacements for LLM judges on 98 labelled traces from an apartment-leasing agent. Moving off the default 0.50 threshold, picked by maximising F1 in cross-validation, mattered. OpenAI’s option reached an F1 of 0.87 at 3.72 cents per 100 eval runs, about 4.5 times the cost of the cheapest candidate, which scored 0.79 at 0.82 cents.
The warning is about calibration: for one group of traces, the average predicted failure probability was about 8% while 60% of those traces contained failures. Treat the scores as rankings until you’ve checked them against your own labels. Ninety-eight traces is a small sample, so use it as a method more than a leaderboard.
6 · Architecture
llm-d 1.0 puts routing and cache management around your model servers
llm-d, a Kubernetes-native layer over vLLM and SGLang, hit 1.0 on 20 October. It is a CNCF Sandbox project co-founded by Google, Red Hat and IBM. Version 1.0 adds v1 APIs, multi-tenant priority and fairness routing, router high availability, zero-downtime upgrades, and session-aware routing and turn-priority scheduling for agents. Google reports that prefix-aware routing plus host-memory offload cut median time to first token by more than 85% and lifted throughput 2.4x on Ironwood TPUs for agentic coding, against a plain Kubernetes Service.
Those are Google’s benchmarks, tied to specific hardware and workloads. The portable part is the open-source stack; the managed gateway, multi-cluster routing and Lustre cache tier are GKE products. If your agents hold long sessions, cache-aware routing is the first thing to test, because it’s where DeepInfer’s missing half lives.
Also worth knowing
🆕 New on the radar
- Tencent Hy-Memory-30B-A3B — an open (Apache-2.0) 30B MoE with 3B active parameters that acts as the “writer” in agent memory systems such as Mem0 and MemOS; its card shows a 55.49 average against 54.62 for Claude Opus 5 on fixed 20% subsamples. Watch. Source
- Inception Mercury 3 — a diffusion-based language model on OpenRouter since 16 October, with a 1.0M-token context and $0.10 input / $0.35 output per 1M tokens (currently 50% off). Try. Source
🛡️ Security & governance
- vLLM 0.32.0 — drops Python 3.10, changes the default CORS origins from
*to none (set--allowed-originsfor browser clients), removes cascade attention and fixes several security advisories. Source
🧰 Models & tools
- TorchTPU — Google open-sourced its native PyTorch stack for TPUs, with vLLM, SGLang and TorchTitan integrations; “minimal code changes” is Google’s claim and the page doesn’t state a licence. Source
- AWS TorchNeuron — a public-beta open-source PyTorch backend for Trainium2 and Trainium3 that runs existing code by setting the device to
neuron. Source
🏢 Platforms & enterprise
- Databricks Agent Memory — in beta on Agent Bricks, with managed sessions, searchable memory and background consolidation on Lakebase; Databricks says its internal benchmark tripled pass rate with 31% fewer tokens. Source
- AWS Well-Architected Agent — a preview service that reviews cloud environments and infrastructure-as-code against Well-Architected pillars; it’s in three US regions and needs a Business Support+ plan. Source
Take what’s useful, leave the rest. The engines will still be open tomorrow.
Researched and drafted with AI, checked against the linked sources. How we use AI →