20VC with Harry StebbingsModal CEO: Has Open-Source LLM Inference Peaked? | What Happens to the Cost of Compute
20VC with Harry Stebbings
Early love of coding 1:32
He says he started coding around 1992, when he was eight, on a Mac Plus with 1 megabyte of RAM. He loved being able to control computers and spent his childhood making games and later entering programming competitions at 16 and 17. He treats that habit as luck as well as skill, because it turned a childhood hobby into the work that now pays him well.
Building Modal late 2:32
He says he did not start a company young because he lacked confidence, and a bit of money too. Modal began when he was 36 or 37, after he saw what seemed like an obvious market gap and wondered why no one else was doing it. He says he probably should have started earlier, but also thinks founders in their 20s may have too much confidence and not enough experience with hiring, firing, and people.
Margins and stack layers 13:32
There is room for strong gross margins in inference because the market can support more than raw model hosting. Sandboxes, agent tools, workflows, post-training, and reinforcement learning all add revenue around the core service. The speaker says the old 90% software margin world is gone because a customer dollar now passes through applications, models, infrastructure, cloud, and Nvidia, with each layer taking its cut. He thinks the business can still reach 40 to 60% margins over time.
Specialization and compute 15:00
The inference market is expected to fragment into many products. Different customers will need low-latency inference, batch inference, sandboxes, and custom training for specialized models. That is why the company wants to host many products under one roof and rebuild cloud for AI. GPU buying is described as frantic, with deals spread across many vendors and 25 Neo clouds. The firm builds its own file system, container runtime, scheduler, and capacity system so it can pool compute, smooth demand, and scale customers up fast.
Capital and open models 26:02
The discussion turns to NeoCloud finance, where the hard part is locking in customer contracts, raising money against them, buying chips, and paying debt back over time. Open-source models are still expected to matter, even as frontier models get cheaper. They let people fine-tune, control latency, and push throughput. There is also concern that too many Chinese models dominate, while Europe may struggle to gather the capital and talent needed for a leading model company.
Modal’s product bets 29:31
The focus shifts to model specialization. Jev is described as strong in classification and regression, and as the market grows, more narrow submarkets appear. Modal’s own growth story comes from brand and distribution, then from adding products. Sandboxes took years to matter, but later became a 300 million run-rate business. The next push is training, batch work, and reinforcement learning, with the same aim of building useful blocks rather than one small tool.
Growth and new bets 39:00
Revenue is growing fast, with 50x growth in the last 12 months and 30 to 40 percent growth every month now. He thinks that makes a 15 billion valuation cheap if the company can keep compounding. He points to training, post-training, batch jobs, databases, workflow orchestration, feature stores, and vector databases as areas that could become large in the next few years.
Software and model choice 49:00
He is very bullish on software engineering and thinks coding agents will raise demand rather than shrink it. Modal spends about half a million a month on coding agents for 150 engineers. On model ownership, he says some customers want to fine-tune open-source models and keep control, but many do not need to own their own intelligence. Latency, cost, performance, hackability, and adjacent services all matter too.
Product scope and timing 52:01
Standalone point products will likely need to sit inside a broader platform, because customers want evals, fine-tuning, and tighter control over latency and throughput. OpenRouter is praised as a strong business with a strong brand, even with many competitors, but the speaker would have told it to keep growing rather than sell. He also says seed investing as a slow, relationship-first model is basically dead. AI can still see a 20% to 30% stock pullback, but he expects the long-term run to continue.
AI-generated summary. It can be wrong or incomplete - check anything that matters against the original.
