Y CombinatorNew LLMs Are Unlocking Robot-Use Agents
Y Combinator
Robot Use Agents Emerge 0:00
Coding agents have turned out to generalize across many domains, and researchers are now showing this extends to controlling robots. MIT professor Philip Isola wrote a viral essay suggesting we may be entering an era of robot use agents, where general purpose language models make different robots more capable. The hosts bring in the founders of Wad Labs and Robocurve, two startups working at the frontier of using large language models to control robots. Wad Labs builds a harness that lets language models control robots effectively and collects data to train better models, while Robocurve acts as an evaluation company for physical AI, measuring robots, models, and vision language action approaches across embodiments like hands, grippers, arms, humanoids, and quadrupeds. Recent viral videos showed language models unscrewing caps, uncapping pens, and coordinating between multiple robots.
From RT2 to Chain of Thought 2:00
The discussion traces back to the RT2 paper, one of the earliest successful uses of AI on robots, which took a language model pretrained on web text and images and fine tuned it to output effector poses, coordinates that translate into joint commands, instead of English words. This is compared to the early chain of thought moment in math problems like GSM8K, where models moved from just outputting an answer to reasoning through steps first. Vision language action models, or VLAs, originally worked like the earlier non chain of thought models, forced to output an action immediately with no room to allocate more compute to harder tasks, whereas newer approaches let the model think or write code before acting. The guests link this to the bitter lesson, the idea that giving models more autonomy and resources lets them do more of what they are fine tuned for, and argue the real question is not architecture but what data is most useful, since robotics has long suffered from a data bottleneck while language and code data are abundant and well understood.
Code as Policies and In Context Limits 7:01
The conversation turns to code as policies research, exemplified by Voyager, an early Minecraft agent that used built in tools like sort, if, and for loops to assemble new tools and compress experience into reusable functions. A related Google DeepMind approach gave coding agents libraries of robot functions, such as pick up an object or move to a pose, letting the agent write code to complete complex tasks in one shot without extra robot specific data, because it already understood sequencing from its coding training. The hosts then describe an unpublished experiment on in context learning, feeding state, action, and reward examples back into a model's context. Improvement is cheap but non monotonic, getting better and worse repeatedly, and it saturates after roughly twenty to forty examples, capping out entirely once the model's context window, tied to its training length, is exceeded, at which point performance actually degrades.
Compressing in-context learning into skills 14:00
The discussion turns to how a robot harness works almost like a form of domain specificity. When a robot is deployed in a new setting, say a wet lab where it must handle test tubes repeatedly, it can learn a lot of this in context, but that learning can then be consolidated into reusable programs, skills, or memories for future agents to draw on. This mirrors older ideas from meta-learning, where a bigger model effectively programs a smaller, faster model to carry out a task once the bigger model has specialized the harness well. The open theoretical question raised is how far in-context learning alone can go, especially since some research suggests these models can approximate gradient descent while learning in context, blurring the line between adjusting weights and manipulating symbols.
Astra picks up a block 16:00
A demo shows Astra controlling robot arms to pick a block off a table and place it into a bowl, using camera feeds and tool calls rather than written code. Directly commanding a robot's pose this way is not always the optimal approach: for repetitive tasks it is better to write fast-running code, and for tasks done before it is better to call a previously compiled skill that handles edge cases more reliably. The demo's latency is currently bottlenecked by Astra's response speed, but frontier language model latency is improving roughly twofold per month, suggesting real-time robot control could arrive by the end of the year. Breaking down the coding agent's process, the routine parts of a task, like approaching and picking up an object, are fairly deterministic and can be automated as code, while a vision-language model is kept in the loop at points of variation, such as detecting objects or responding to failures, to allow the system to generalize.
Why computer-use data helps robots 19:00
A key insight, credited partly to conversations with Francois Chollet, distinguishes transduction, learning a direct mapping from inputs to outputs, from program induction, generating code that performs that mapping, with code being far more data-efficient when examples are scarce. This reframes why models like Astra, described through Philip Ola's platonic representation hypothesis, may be converging toward a shared, consistent representation of the world as they train on more data, meaning a strong language model and a strong robotics model might end up representing reality similarly. Astra's leap in spatial intelligence is attributed largely to heavy pretraining on computer-use data, such as dragging a cursor to orbit a CAD object in Blender, which teaches concepts like top-down, left-right spatial reasoning transferable to robot control, alongside egocentric video data beyond simple teleoperation. The speakers note the irony that graphical interfaces were originally designed to resemble the physical world, and now that same interface data is what teaches models to act in the physical world. They close by predicting general-purpose robots, able to follow natural language instructions as well as a competent teenager, within roughly two years, comparing it to a ChatGPT-like moment for robotics, while acknowledging that latency remains a major unsolved barrier to economic usefulness.
Sleep as memory compression 27:31
The speaker compares fast in-context learning to how intelligent systems seem to need periodic rest. He notes that almost every intelligent system sleeps, and during sleep the hippocampus sends sharp-wave ripples to the brain's two hemispheres, compressing the day's experiences into updated long-term memory.
Updating weights like Dagger 28:31
He suggests a similar cycle for robots and models: collect data, reflect on it, then fold it back into an updated weight file, comparing this to the Dagger data aggregation framework from classic reinforcement learning, and noting it echoes DreamCoder's approach of refactoring a skill library during a sleep phase into more compact form.
Managing growing skill libraries 29:02
He points out that many robotic code as policy papers now build a growing library of skills, yet nobody has settled how to prune, organize, or refactor them. Managing this expanding library of skills and deployment data is expected to be an important open problem going forward, closing an excellent discussion that anticipates major robotics advances ahead.
AI-generated summary. It can be wrong or incomplete - check anything that matters against the original.
