Tag: inference
TypeSafe's Diogo Almeida on Jev, a "System 1" model trained for calibrated decisions that software calls in a loop — three primitives, no refusals, no public benchmarks, intelligence per dollar. Also the identification of the author of the earlier KV-cache post, and his view of coding agents without a KV cache.
An agent re-reads its whole context on every tool call, so cache reads, not output tokens, are the largest line on the bill — and they cost the provider close to nothing to serve. What that means for routing, subagents, self-hosting, and what to measure.
Karpathy expects specialised models and does not see them arriving, and his explanation is that adjusting weights without losing capability is still an underdeveloped science while context windows just work. Plus where he puts the open-weight models — six to eight months behind, and better off there.
Two OpenAI people on what changed in computer use since this spring (writing code, accessibility trees, speed), then the API stack — async tool calls, mid-turn steering, 12-hour cache guarantees — and how the Decisions API came out of Jev's launch in about a week. The first answer here to "can a big lab just copy it".
OpenRouter's CEO and an early backer on the multi-model bet, why labs struggle to get a finished checkpoint into developers' hands, the Mistral price war, the roads not taken (tuning, memory, fusion), and why Stripe bought it — token fraud as the next payments problem. Plus a speaker-label problem in the capture.
Philip Kiely and Ali Taha on what actually happens between an open-weight checkpoint and a production endpoint — the four optimisations that stack to 2–4× on fixed hardware, why quantising more of a model can make it better, and the reliability failures that turn out to be a race condition in a kernel rather than anything about the weights.
Why video generation is stuck at five-second clips — quadratic attention over 35,000 tokens for five seconds of 480p — and why the open/closed gap that has nearly closed for text is night-and-day for video. Plus the argument that text diffusion is being marketed as the wrong product.
Inference engineers now write training pipelines, RL rollouts bottleneck on inference speed, and Baseten reports running GPU kernels inside their inference engine that GLM-5.2 wrote for itself from its own profiling traces. Plus the argument that continual learning belongs in the KV cache rather than in the weights.
Lab notes on turning a spare Linux desktop with a 12 GB GPU into an always-on, OpenAI-compatible HTTP endpoint for open-weight small models — VRAM budgeting, a dated model shortlist, llama.cpp vs. Ollama vs. vLLM, systemd, and not exposing the thing naked to the internet.