Y CombinatorGoing In Deep On Data | YC Paper Club
Y Combinator
Why Data Deserves Its Own Night 0:00
The host opens by recalling that in 2016, when he left his PhD program to start Focal Systems, the common belief in venture capital was that data had no lasting value, since anyone could already download ImageNet. He recalls that aside from one investor at Excel who backed Scale's Series A, most VCs thought the terminal value of data businesses was zero. Since then, he notes, that view has produced roughly a hundred billion dollars in market cap, proving the skepticism wrong.
The Interview Question That Reveals Everything 1:00
He describes the interview question he used at Focal for PhD candidates from Stanford or MIT: if your model hits 85% F1 score on a hot-dog-versus-not-hot-dog task, what do you do next. Candidates who answered by talking about reading papers, trying ReLU versus GELU, or adding more layers were rejected on the spot. The correct answer, he explains, is to look at the data itself, examine false positives and false negatives, sort them into buckets, and find the biggest recurring issue, such as fog on a fridge door or a person blocking the camera in his in-stock/out-of-stock detection work.
Data Now Matters More Than Architecture 2:00
He points to Andrej Karpathy's famous effort to measure human-level performance on ImageNet by manually trying to label it himself, discovering that humans struggle to distinguish things like Eskimo sheepdogs from Siberian huskies, and that roughly 30 to 40 percent of ImageNet is dog breeds most people can't tell apart. He recounts that during a PhD, someone might spend 95 percent of their time on architecture and 5 percent on data, but in production that ratio flips, and today, with transformers working so well, it's more like 97 percent data and 3 percent architecture. The core problem is that clean training distributions like ImageNet don't match the messy reality models face once deployed.
Expert Data and Environments Drive the Economy 3:31
He argues that automating the economy requires both expert data and expert reinforcement-learning environments, the RL kind providing verifiable rewards and the preference-based kind capturing subjective judgments, like whether a design is more usable or whether two doctors disagree on a diagnosis. He stresses that datasets and environments should be treated as products, citing the craftsmanship Greg at ARC-AGI puts into each game, and describing the real difficulty Focal faced deciding how to classify tricky cases like "product pushers" that look like out-of-stock shelves but technically aren't.
Data Is Never Truly Finished 5:00
Using the example of automating Salesforce workflows, he explains that even with a hundred billion hours of screen captures, a single UI change from Salesforce can make that data obsolete overnight, meaning you always need more data no matter how much you already have. He cites Brendan Emery's recent slide at Sequoia arguing that data, not architecture or GPUs or power, is the current bottleneck, and notes his own finding that no AI model currently beats random chance at predicting next-seven-day S&P returns, because there's no dataset of traces from traders who reliably beat the market. The same gap applies to accountants, doctors, lawyers, and other experts whose knowledge hasn't been captured in usable form.
Data Companies Are Like Apps 7:00
He proposes thinking of expert data and RL environments the way you'd think about apps on a phone: just as Apple wouldn:t try to run its own Instacart delivery network, AI labs won't each independently build the deep, specialized data infrastructure needed to make a model a great doctor, lawyer, or carpenter. Instead, he expects a landscape of specialized companies, each focused on curating data and environments for one domain, working with major labs rather than being absorbed by them, which sets up the introduction of the panel: Vincent Chen of Snorkel, Vololo of Inception Labs, and Shane from Anthropic's data provenance initiative.
Snorkel's Decade Long Bet on Expertise 10:00
Vincent Chen introduces Snorkel's core thesis, developed since its roots at the Stanford AI Lab: the real bottleneck in building effective datasets is scaling expertise, meaning getting the judgment of doctors, clinicians, journalists, and other specialists out of their heads and into usable data. He describes Snorkel's evolution from what he calls Data 1.0, simple thumbs up or down labels taking about 30 seconds each, to today's Data 2.0 landscape, where building a single training environment can take single or triple digit hours because it now involves full task rubrics, verifiers, and Docker-packaged simulated worlds.
The Limits of Manual Labeling 12:30
Vincent explains why manual labeling doesn't scale: it costs new cognitive effort for every single data point, it's fragile to noise unless you add redundancy through repeated relabeling, it has to restart from scratch whenever the task definition changes, and it leaves no record of the reasoning behind each label. These problems worsen in expert domains, where getting a cardiologist to hand-label large volumes of MRI scans or health records becomes essentially intractable.
Turning Expert Judgment Into Software 14:00
Snorkel's response, developed with CEO Alex Ratner during his graduate work, was called data programming: encoding expert supervision as software rather than one-off labels. This approach models expertise as labeling functions, essentially encoded heuristics capturing why someone labels something spam or not spam, which gives the scale and adaptability of software along with an auditable record of reasoning. Because these labeling sources overlap and are individually inaccurate, Snorkel built a weak supervision theory to estimate each source's reliability without any ground truth, comparable to grading students' answers by comparing their votes against each other rather than an answer key, then using that denoised signal to train models that generalize beyond the original labeling functions' coverage.
Evaluations Must Keep Pace With Complexity 17:00
Vincent turns to what he calls Data 2.0, illustrated through coding benchmarks, where he plots task complexity against sequence length to show that even as basic evaluations like HumanEval become saturated, agents are being trusted with far longer, more open-ended work, so data challenges keep growing rather than shrinking. He points to Programbench, where he says the latest model just cracked 1 percent, as evidence that current benchmarks fail to capture the real difficulty of modern agent tasks.
Building SeniorSWEBench With Real Experts 18:30
He describes a project built with the original SWEBench team from Princeton called SeniorSWEBench, motivated by the fact that coding agents are now trusted to make architectural decisions like senior engineers, yet are still evaluated like junior engineers. Instead of the overly specified, step-by-step tasks typical of older coding benchmarks, the team designed instructions to resemble realistic inputs such as a Slack message, a dump of logs, or a rough set of user stories, requiring real expert judgment to translate into evaluable tasks.
Balancing Reliable Tests and Flexible Judgment 20:30
Vincent lays out a tradeoff between traditional unit-test style verifiers, which are highly reliable but inflexible and can wrongly penalize valid alternative solutions, and LLM judges, which are flexible but unreliable since they don't actually execute the code. Snorkel's middle path was a validation agent: an expert first writes a short validation spec describing the user stories and requirements for a task, and then the validation agent turns that spec into deterministic test scripts that actually run against the code, with judges layered in afterward to catch collusion or reward hacking, allowing expert intuition to scale without requiring senior engineers to hand-grade every single agent trajectory.
A New Metric Called Tasteful Pass 25:31
The speaker describes a benchmark metric introduced to measure something beyond correctness or mergeability: whether code actually matches how a senior engineer would write it, and whether it keeps patch size minimal rather than bloated. Combining this with correctness scores let the team rank models, and as of last week Fable, Opus, and Sonnet were all tied for first place. The dashboard is live and continuously updated, and the speaker credits co-founder Henry as lead, along with collaborators from Princeton's SweetBench team and the Wisconsin Madison lab.
Entering Data 2.0 26:30
The talk closes by framing a shift called data 2.0, where environments, outputs, and agent autonomy are all growing more complex at once. Correctness is becoming harder to verify, outputs are becoming more subjective, and agents are being given more independence, and each of these axes compounds the difficulty of building good data. The core claim is that this is a research problem, not something solved by simply adding more human labelers, and it requires giving domain experts like software engineers, doctors, and lawyers real leverage to exercise judgment. The speaker announces open benchmark grants, citing collaborations with teams behind Agents Exam at Berkeley, a continual learning benchmark, and Terminal Bench, offering this as a resource to accelerate community benchmark development.
Diffusion Language Models From Inception 28:30
Volo, from Inception, introduces diffusion language models built with co-founders Stephano and Aditya. Instead of generating tokens one at a time left to right, a diffusion model starts from a noisy guess at the whole sequence and refines all tokens in parallel over multiple steps. The main advantage is speed, with their Mercury 2 model exceeding a thousand tokens per second, far beyond standard autoregressive generation.
Speeding Up Real Time Voice 30:01
Voice agents used in customer support and education typically run on a cascaded pipeline of speech to text, an LLM, and text to speech, and the LLM is usually the latency bottleneck. Running Mercury 2 at 1,000 tokens per second lets voice interactions feel more seamless, or lets developers deploy a larger model, or let the model reason longer to improve quality. On voice benchmarks Mercury 2 sets a new frontier of quality versus latency, beating baselines including a 120 billion parameter GPT-OSS model running on Cerebras hardware, achieving both higher quality on tobench and faster speed, using smarter software on ordinary GPUs rather than specialized chips.
Building Better Data With Tao Forge 32:01
Good voice models need good data, and existing benchmarks like tbench fall short since they contain far fewer specifications and tools than real business settings and are heavily benchmaxed, with models scoring in the 90s on leaderboards but performing much worse on real production logs. Inception's answer is a system called Tao Forge, an agentic harness that synthesizes realistic RL environments from real user interaction data or from a crawled business knowledge graph when real data isn't available. Given a target domain like banking, it generates a policy, tools, and a database, then produces tasks drawn from a library of abstract scenarios and personas, calibrated through hardening steps so tasks are neither trivially easy nor unsolvably hard. In one experiment, training the Mercury 2.5 preview model on Tao Forge tasks raised accuracy from about 50 percent to a 23 percent improvement on a held-out set of businesses, matching state of the art open and closed source models, and the models are now available commercially with $500,000 in credits offered to YC startups.
Multilingual Pretraining And Transfer 40:32
Shane, presenting PhD work done at Google and MIT, examines how training data from different languages interact during pretraining, a problem he says generalizes to any mix of data sources, not just languages. Most scaling law research focuses almost entirely on English, leaving a gap for understanding languages like Thai, which makes up only 0.6 percent of the tokens in the Madlad 400 corpus compared to English. Training a model only on Thai causes overfitting because there's so little data, so the better approach mixes Thai with synergistic languages such as Indonesian, Malay, Lao, and Croatian, determined empirically rather than by assumed language family similarity.
Measuring Synergy And Interference 45:30
The team measured how helpful one language is to another by comparing training curves, finding that Indonesian helps Thai more than English does, which helps more than Telugu. This produces a cross-lingual transfer matrix where red marks synergy and blue marks interference, revealing for instance that Portuguese, Italian, and French help Spanish while Japanese hurts it. The relationship isn't symmetric, since Portuguese helping Spanish doesn't mean Spanish helps Portuguese equally, and shared script turned out to matter slightly more than shared language family, likely due to tokenization effects. Bigger models show more synergy across languages, while smaller models show more interference as languages compete for limited capacity.
Turning The Matrix Into Scaling Laws 50:01
To make this practical, the team extended Chinchilla-style scaling laws, which normally predict loss from model size and data amount, by breaking the data term into monolingual data, closely related transfer languages from the matrix, and a bucket for everything else, while accounting for diminishing returns from repeated epochs. This approach produced state of the art predictions for language mixtures. The findings help practitioners decide whether to fine-tune an existing multilingual model or pretrain from scratch for a target language based on available compute, and they quantify the curse of multilinguality, showing how much more data and model size is needed to maintain performance when expanding from four languages to eight.
AI-generated summary. It can be wrong or incomplete - check anything that matters against the original.
