How AI Agents Use Tools: Function Calling & MCP | Stanford CS329Z Lecture 4 with Diyi Yang
Stanford Online
RAG limits 0:05
Retrieval helps by bringing in outside evidence, but it also fails in familiar ways. Permission leaks can expose information a model should not see. Similar but irrelevant text can distract it. Conflicting articles can pull it in opposite directions, and prompt injection can plant malicious text in the retrieved material. A retrieval, read, and generate loop can also let early mistakes spread forward.
Middle positions fail 4:31
The lost-in-the-middle effect shows that the right answer can be in the context and still be ignored. In one setup, models did well when the correct document came first or last, but fell off when it sat in the middle. More documents did not fix this. The chance of finding the answer rose with about 50 retrieved items, yet reader accuracy flattened after 10 to 20 and sometimes dropped.
Why tools are needed 19:31
Models get worse as tasks get longer or more exact. Multiplication, puzzles, and dynamic programming all show the same drop. They also struggle with weather, post-cutoff facts, private data, and non-text systems like databases and APIs. Tools help by handing those jobs to something else. A calculator can do arithmetic, retrieval can reach web or company data, and browsers, robots, and APIs can extend what the model can do.
What a tool is 23:32
A tool is an external function interface. The model does not do the task itself. It writes a function name and arguments in text, and an outside program carries out the work. This can cover calculators, weather lookups, calendars, email, image generation, translation systems, and even other models. The key point is that the model learns when to call a tool, which tool to choose, and how to use the result.
Loss-based tool choice 38:30
A tool call is kept only if it lowers the weighted cross-entropy loss compared with simple baselines. The model compares no call, a call without output, and a call with output. In the Pittsburgh example, the QA call drops loss from 2.5 to 0.8, so it is kept. This works by executing the external tool, fetching the result, and judging usefulness from what comes back, even if the server is down or the format is wrong.
Scaling to many tools 50:30
The same idea later scales to many APIs through ToolLM. It crawls public APIs, asks ChatGPT to invent user requests, tool inputs, outputs, and even solution trees, then fine-tunes a smaller model on that synthetic data. This can support over 16,000 APIs. The same section also notes failures: models sometimes hallucinate tool names or pick the wrong API, as in speech-to-text cases where translation or nonexistent parameters were chosen.
Gorilla and retrieval 57:31
Gorilla connects LLMs to many APIs, including 1.6 thousand modern AI tools such as Hugging Face, PyTorch Hub, and TensorFlow Hub. It uses GPT-4 to make synthetic instruction data, then trains with both correct and incorrect retrieval so the model learns to tell right from wrong before it picks a call. At inference, a user query like cats dancing can lead it to choose Stable Diffusion and return only the final execution result.
Standard interfaces 1:03:00
The next move is to stop retraining for every new API and instead change how tools are described so any model can use them at runtime. REST APIs are human-facing and use readable requests like get weather ct=SF. Function calling gives tools as JSON schema, with typed arguments and a checked output. MCP goes further and gives one protocol between AI apps and data sources, so the same server can be reused across hosts.
Benchmarks and metrics 1:15:31
The Berkeley function call leaderboard is a live place to check tool use models. It scores models with syntax tree matching, execution checks, and state-based tests for multi-step calls. ToolBench uses synthetic customer-agent chats like flight changes. For these settings, pass@k is not enough; pass^k requires every try to work.
AI-generated summary. It can be wrong or incomplete - check anything that matters against the original.

