20VC with Harry StebbingsHow Many Will Actually Get Built & Is Energy AI's BIGGEST Bottleneck? | Positron AI Co-founder
20VC with Harry Stebbings
Introducing Positron AI 0:00
Thomas Somas, co-founder and chairman of Positron AI, joins the show after his company raised an 875 million dollar Series C at a 5 billion dollar valuation. Positron builds semiconductor hardware for generative AI inference, meaning the chips, the software that talks directly to those chips, and full rack-scale systems that power applications like ChatGPT and Claude. He explains that Positron sits at the lowest level of the AI stack, focused specifically on making inference, the running of trained models, fast and efficient.
Training versus inference demands 2:02
Training a model is compute bound, meaning performance depends mainly on raw flops, the number of floating point operations a chip can do per second, and this is why export control rules focus on flops. Inference works differently. Because each output token has to be generated one after another without knowing future words, it cannot be massively parallelized the way training data can. This makes inference heavily memory bound, meaning speed depends on how fast you can read stored values, not just how much raw compute you have.
The memory wall explained 6:00
Between 2014 and 2024, a single Nvidia GPU saw roughly a 120-fold improvement in flops, but memory bandwidth improved only about 17-fold. This gap exists partly because SRAM, the basic memory cell built from six transistors, has barely shrunk in 30 or 40 years, unlike general-purpose transistors. It also happened because the dominant models of the 2010s, like AlexNet and ResNet, were compute bound and did not need much memory, so there was little incentive to improve memory technology until transformer models, especially after GPT-3 in 2020, made memory bandwidth the real constraint.
Hidden profits in cached tokens 8:30
Thomas points out that providers now charge different prices for cached versus uncached tokens, but few people realize how high the margin is on cached tokens, since reprocessing a cached token costs roughly one thousandth of what generating a new one costs. This is a major reason Anthropic reportedly runs about 80 points of gross margin on its API business. He argues the popular idea that these companies are unprofitable and burning through cash is wrong, and that if they stopped training new frontier models, they would be massively profitable overnight.
Concerns about pacing the frontier 12:00
Discussing Dario Amodei's call to slow AI development, Thomas says he is broadly opposed to that direction. He sees two big risks: first, that calls for a pause hand ammunition to people who want to stop the technology entirely, and second, and more worrying to him, that AI capability could become concentrated among a small number of companies or governments. He compares this concentration to a modern version of feudal serfdom, where ordinary people are locked out of a transformative technology while a few large players and regulators control it.
Global competition and China 18:00
Thomas argues that the U.S. and its companies underestimate how quickly rivals like China could catch up, comparing the false confidence to assuming military dominance guarantees safety against new threats like drone incursions. He believes China currently promotes open access to AI strategically, but expects that once it takes the lead, it will pull that access away, and that a CCP-controlled superintelligence would deny its own billion citizens the technology's benefits. He supports free trade and open exchange of ideas generally, but argues totalitarian regimes that close off their own systems while benefiting from open ones should not get the same access.
Anti-data center sentiment and false claims 22:30
Thomas calls it the scariest political trend that opposition to data centers has become a unifying issue across the left and right, and argues this sentiment is largely fueled by Chinese-linked misinformation. He pushes back on claims about water and power use, noting that a single In-N-Out restaurant uses more water than the largest U.S. data centers, and that new data centers typically bring their own generation capacity rather than draining power reserved for homes. He contrasts this with China's approach of bulldozing communities and adding largely dirty power generation without such public pushback, and argues that U.S. federal land, especially open desert in states like Nevada, could host far more data centers without displacing anyone.
Will planned data centers get built 28:30
Asked what percentage of planned data centers will actually be completed, Thomas says capacity may shift locations as some communities block projects, but he is not too worried about an existential shortfall, since builders are now investing more in educating communities and abundant land remains available elsewhere. On the idea of building data centers in space, championed by Elon Musk, he says he would never bet against Elon but currently sees building on land as cheaper and easier, while acknowledging promising alternative technologies exist.
Ocean data centers and space economy 30:01
The guest mentions a partner company, Panthalassa, which is building ocean-based data centers using a pumped hydro solution in the middle of the ocean, offering an alternative to space-based compute. He still believes in space data centers long term, not just as a practical solution but because he thinks humanity needs a space economy to reach its long-term potential.
Energy, economics, and the real bottleneck 32:33
Asked whether energy is a bottleneck for AI, he argues that Positron's goal is to deliver more compute per watt, but that efficiency gains will not shrink data center sizes, since operators will still build the maximum compute they can. He frames progress across human history as a track record of energy production and use, from fire to nuclear power, but says the real limiter now is economics, not raw energy capability. He points to sovereign debt as his bigger worry than AI company debt, arguing the United States government's ability to print currency and enforce tax collection masks deeper risks, and that he trusts companies like Oracle to execute more than he trusts government finances. He is not worried about AI companies missing revenue targets, since the past few years have shown accelerating revenue and productivity gains.
What tokens and KV caching actually are 34:30
He explains that a token is a chunk of text, often a prefix or suffix, that models are trained to recognize as a compression unit, roughly half to three quarters of a word in English. A sequence is the ordered string of tokens that builds up context. Originally, transformers recomputed everything for every new token, which was wasteful, so the KV cache was introduced to store two matrices, keys and values, from prior computation so it doesn't need to be redone. This trades expensive quadratic compute growth for linear memory storage growth, though managing a unique cache per user, and deciding how long to keep it, adds real complexity.
Compressing the cache through quantization 39:31
He describes quantization as shrinking the number of bits used to represent each value, moving from the old standard of 32-bit floating point down through 16-bit, 8-bit, and now 4-bit formats. Naive quantization causes real accuracy loss, sometimes worsening benchmark scores by 20 to 30 percent, but advanced techniques share a single higher-precision multiplier across a group of low-precision values, achieving roughly 75 percent size savings while keeping quality loss under about 1 percent compared to an unquantized model.
Why caching matters so much for agentic coding 43:01
He notes that KV caching is not strictly required, but that most valuable AI workloads today are highly cacheable, citing a SemiAnalysis benchmark of Claude Code sessions showing about 96 percent of tokens across long agentic sessions are cached. He gives scale numbers to make the memory problem vivid, an estimated 1.8 trillion parameter GPT-4 taking about 900 gigabytes at 4-bit quantization, and a hypothetical 10 trillion parameter model like Claude Fable needing around 5 terabytes, while individual long-context user sessions can reach 100 gigabytes, meaning just 50 users could require more memory than the model weights themselves. This creates a tradeoff between fast accelerator memory for weights and persistent storage for cached sessions, which mainly saves the service provider money rather than speeding things up for the user.
Tiered memory and growing context needs 48:31
He describes the common industry approach as a tiered memory hierarchy, keeping active sessions in fast accelerator memory, recent ones in host memory at four to ten times more capacity, and older sessions in slower NVMe flash or network storage, down to something like an old ChatGPT conversation sitting on a slow disk. He notes context lengths have grown fast, from 8,000 to 64,000 tokens a couple of years ago to a million tokens now, though even a million tokens can't hold a full large company codebase, which he sees as the real limiter for agents replacing whole programming teams, more than raw model capability.
Big frontier models versus small on-device models 53:33
Asked whether growing model sizes conflict with enterprises wanting their own smaller proprietary models, he says both trends continue at once. Frontier labs like OpenAI, Anthropic, and Google will keep pushing larger, more capable models, and he'd happily pay ten times more per token for ten times better output. He estimates the top four model companies handle 80 to 85 percent of all tokens today, with the next few companies handling another 5 to 10 percent, leaving a small remaining share for on-premises or local models. He argues local, on-device models won't reduce cloud token use, since a local assistant constantly scanning email and calendars will trigger far more lookups to larger cloud models than a person prompting ChatGPT occasionally ever would.
Scaling laws and why GPT6 felt like AGI 55:01
He describes belief in continued scaling from trillion-parameter models toward 50 or 100 trillion parameters as more gut feeling than proven law, since there's no mathematical guarantee the trend continues, only the absence of signs it's slowing. He compares his first 24 hours with GPT6 Astra to his first encounter with ChatGPT's GPT-3.5 launch at NeurIPS in November 2022, and to the excitement around Sora 2. For coding, he calls Astra's improvement a step function, solving hard problems other models looped on, and finding bugs and performance issues no one had noticed in his company's codebase. More striking to him was its general computer-use ability, including Blender animation and interior design from photos. Most notably for Positron, GPT6 finally handled the full chip design flow, writing Verilog for an encryption block and scripting through EDA tools despite their notoriously outdated, undocumented nature from the 1990s and early 2000s.
AI compresses chip design timelines 1:00:30
A chip design task that would normally take an experienced engineer two to three weeks was completed by an AI model in a little over fifty hours, running at over a gigahertz. This speed still feels almost unbelievable given what these models are trained on, and it reflects OpenAI's own in-house chip development work, alongside similar efforts from Anthropic and reportedly DeepSeek.
Every major AI lab builds its own chips 1:01:31
The commoditization of the chip layer, with labs building custom silicon like OpenAI's "Jalapino," is seen as good for the industry because it lowers cost and raises capability. Companies share the same broad goals but take very different architectural approaches, and the market ultimately decides which approaches create value and get rewarded.
Memory costs limit how far context can grow 1:04:00
Traditional attention mechanisms hit hardware limits as context windows grow, since memory cost expands quadratically. Positron's upcoming hardware aims for eight times the memory capacity of Nvidia's highest-memory chip. Chinese labs, working under export controls that restricted access to high-memory chips, drove algorithmic workarounds instead, including DeepSeek V3's multi-head latent attention and gated delta net, which can cut attention-related compute time by roughly 75 percent.
Compression trades capability for savings, and context needs testing 1:06:33
These compression techniques aren't free; multi-head latent attention reduces cache size but costs some model capability, which is reportedly why major US labs avoid it. Having a long context window also isn't the same as using it well. On the "ruler" needle-in-a-haystack test, where a hidden value is buried in a flood of text, GPT-5.6 found it correctly only about 70 percent of the time, while GPT-6 Astra succeeds over 95 percent of the time.
Token prices fell, but token value rose far more 1:09:34
The Silicon Data token price index dropped from $60 to under $1 per million tokens in five years, but a $60 token from five years ago, comparable to GPT-3.5 quality, would be worthless today. The value per token, measured by real economic impact, has likely grown by a hundred to a thousandfold, far outpacing the drop in price. Pricing may eventually shift from cost-per-token toward cost-per-useful-result, though token-based pricing will likely persist because it's simple to calculate margins on.
Where cracks might appear, and what stays optimistic 1:13:30
A possible warning sign for the AI boom would be frontier labs like Anthropic and OpenAI missing growth targets or shifting toward open-weight models, though rising local AI use should keep driving demand for frontier tokens as trust in autonomous local models grows. Data-labeling companies like Mercor and Surge, already near three billion in revenue, could plausibly reach $200 billion in value serving both labs and enterprises, unless frontier labs eventually replace that work with their own AI agents. The final point raised is that AI alignment depends on human alignment, on getting governance, regulation, and energy policy right, since economic self-interest will ultimately drive real-world decisions.
AI-generated summary. It can be wrong or incomplete - check anything that matters against the original.
