Y CombinatorWhy Inference Matters 12x More Than Training | YC Paper Club
Y Combinator
Inference costs and scale 0:01
Inference is presented as the real bottleneck in AI systems. Training matters, but for a weight file that lives for two years, inference can take about 12 times more flops than training. That gap grows as more users come online. The point is made with DeepSeek, whose reported numbers suggest heavy inference use and strong margins because of it.
Serving tokens efficiently 3:31
The talk then turns to how a request is served. A prompt first goes through prefill, where cached work should not be repeated, then through the decode loop that produces each next token. Time to first token, time per output token, throughput, goodput, cache hit rate, and KV cache retention all shape performance. Decode is far slower than prefill, so it is the main bottleneck.
Routing and token flow 16:30
Requests first move through routing, where the system picks a GPU replica and tries to keep the KV cache in the right place. That cache matters because sending the same work to the same GPU can avoid repeating earlier steps. The next stage is tokenization, where human text and tool calls are turned into model input. The model can ask to search files, run commands, or use the internet, so this step has to keep those choices correct and well formed.
Scheduling and optimization 19:31
A scheduler then groups requests for throughput or for faster replies, depending on the goal. After that come prefill and decode. Prefill is heavy compute. Decode is memory bound and generates tokens one by one. The main ways to speed things up here are better kernels, smarter parallelism, and quantization. Kernels improve the math work. Parallelism splits large models across GPUs. Quantization cuts weight precision while trying to keep user quality acceptable.
Evaluation and dogfooding 31:32
The work depends on a tight evaluation loop. Internal dogfooding catches broken tool calls, weak benchmark scores, and failures in end-to-end checks. A Flappy Bird loop runs all the time so the team can see whether quantization or another change has broken model behavior. The point is not only algorithmic work. It is careful benchmarking, agent tests, and constant in-house use.
Quantization and serving 33:01
A MiniMax model offered a clear example. Their attempt to quantize to NVFP4 did not match the original FP8 model, but a different recipe did. That version preserved performance, could be served on Nvidia GPUs, and produced more tokens for MiniMax. The result helped bridge training and profitable serving, so the model maker could earn revenue and fund the next model.
Serving quality gaps 46:30
Inference quality can vary for ordinary reasons like over-quantized models, faulty hardware, or different weight versions. Tool calling is a major source of trouble, because JSON parsing, schema checks, tokenizer-level parsing, and failure recovery can all go wrong. Different sampling methods and custom inference kernels also change results, especially on long contexts. A model may even be served under the wrong name, so a large customer base across regions and providers can see different behavior that is hard to trace.
Ways to monitor drift 50:30
The gap between the worst and best open-model provider can be larger than the open-versus-closed gap, and it grows worse for coding agents with many back-to-back calls. Evals are useful but expensive. Cheaper checks include distribution tests against a reference model, log-prob comparisons on one token, and simple request-shape monitoring for tool-call success, output length, or looping. Labs can help by publishing their own inference stacks, quantized weights, lower-level APIs, and trusted execution fingerprints.
Customer specific checks 1:00:31
Providers can game public tests by training to the benchmark. The safer approach is to build tests around one customer’s own traffic and use case. Those checks are private, so a provider cannot easily tune for them. When quality slips, the customer’s bespoke evals catch it and close the loop.
Local serving and compilers 1:02:00
Local models create the same serving problems as cloud models. You still need reliable testing for the stack that runs the model, whether it is on a laptop, a desk-side machine, or in a data center. The future looks mixed, with local and cloud models used together, and routing decisions spread across both.
Equality saturation 1:04:30
Luminal builds compilers for model inference and uses equality saturation to search for better code. The system reduces PyTorch models to about 14 or 15 primitive ops, such as add, multiply, and sum, then maps those through a small shared core to GPUs, TPUs, Trainium, and other chips. It avoids huge rewrite libraries and makes new back ends easier to add.
Multi level search space 1:15:32
The compiler treats hardware as layered. It can reason about threads, warps, thread blocks, whole GPUs, and even whole nodes. That lets it keep a multi-level IR and mix operations from community code and vendor libraries. cuBLAS, CUTLASS, CuBLAS DX, and ThunderKit can all sit in the same search space, and the compiler can fuse lower-level pieces into larger kernels when that helps.
Profiles guide the search 1:18:00
The team says a cost model is too hard to build for every GPU, so it measures candidates on real hardware instead. A profiling run can finish in under 10 milliseconds, and they prune with roofline checks before running everything. They search with genetic algorithms, target a strong local optimum, and accept long compile times because inference pays that cost once and then serves many runs.
Agent memory management 1:31:00
The talk starts with agent workloads from late 2023, when tool use and parallel subagents were only being imagined. Short prompts were reused many times, so the engine needed a way to share prefixes and avoid recomputing them. Radix attention did that by keeping shared context in a tree in memory. When memory filled, unused leaves were evicted. Prompt order was also important, since cache-aware batching could reuse context instead of mixing requests.
Unified serving system 1:35:04
The same idea grew into a general inference engine in 2024. Overlap scheduling delayed host sync so the GPU could keep running while the next batch was prepared, which mattered more as GPU compute sped up. High cache linked GPU, host, and storage memory, cutting TRFT by 56 percent. A unified radix tree and unified memory pools then tied together sliding windows, recurrent state, and full-token caches, with compaction keeping physical memory contiguous.
AI-generated summary. It can be wrong or incomplete - check anything that matters against the original.
