Andrej Karpathy — AGI is still a decade away

Hi everyone, my name is Keon Vafa and in this video I'm going to go over some research that tries to answer the question, what are the implicit world models inside of generative models in AI. Now to see what I mean by this, let's go over some functionalities we may want from AI systems like large language models or LLMs. One exciting functionality is that they can synthesize concepts. We're also excited they can apply concepts to new domains such as in Fshot or in context learning. Lately, we focused on reasoning, and it's amazing that just with the right prompting, Gemini can solve math olympiad problems. Today's models can also be creative while being grounded in reality, as you can see, not only in text, but also in video models. These may all seem like different uses, but they all have a shared kind of functionality. They could all be performed by a model that has understood the world to some capacity, or in other words, a model that has learned the correct world model. Now this raises a couple of questions like what would it mean to have a correct world model and how would we even be able to evaluate or measure if a model has understood the world. It may seem that one approach is to evaluate understanding like how we do most other kinds of evaluation in AI using benchmark test questions. These benchmarks such as the questions on AP exams are used to signify understanding in people. But do they signify understanding in AI? Benchmarks test a very narrow kind of understanding, the ability to answer test questions. If a person aces an AP math exam, we'd say they understand math, but it requires strong assumptions to extend this logic to LLMs. And the evidence we have is that LLMs learn things in nonhumanlike ways. GBD5 does incredibly well on an Amy math competition, but it thinks that 4.11 is larger than 4.9. To give a bit of a crude analogy, using human tests to evaluate the understanding of LLMs is a bit like evaluating the vision capacities of an AI model using a vision exam like the one we get at an optometrist. It's just not what it's designed for. In this video, I'm going to take a step back and focus on the question of how to evaluate a generative models implicit world model. I'll start by discussing some general strategies for evaluation and then I'll focus on a couple of notions of world models for my own research. For each notion, I'll walk through what a world model means in that setting. I'll discuss evaluation metrics that can be used to test it, and I'll go over empirical results. At the end of this video, I'll discuss related ideas and offer some ways forward. So, let's start with some background. It may seem like evaluating understanding is challenging and it is but there's been a general strategy that different groups have used and the strategy is to take constrained problems for which we know true world models and use those as test beds to answer important questions like what is a world model. One of the early test beds was Eth. For those of you who don't know, Ethell is a board game that involves players taking turns placing black and white tiles on a board a little bit like Go. Every time a piece is put down, it results in a cascade of other changes in the board. To make a test bed, researchers collected sequences of transcripts from Athell games and trained a transformer model to predict next tokens on these sequences. The model never saw the true world, which in this case is the Athell board, but it was trained to predict the moves that were made in games. And the evaluation question here is did this transformer uncover the implicit structure of the Athell board? Now, even though the other domains are test beds, they're used to create general procedures that can be applied elsewhere. In this video, I'm going to focus on two research projects I've worked on which form evaluation metrics for world models in two kinds of settings. One where we care about a world model for a single task and one where we care about a world model for many tasks. So I'm going to start with a single test setting. And here our test bed is going to be Manhattan. Specifically, we collected a data set of taxi rides that took place in Manhattan. And we tokenized each taxi ride into a sequence of directions, sort of like a language for Google Maps directions. We took a transformer and only trained it on these directions. We never allowed it to see the map of Manhattan. But if a model has succeeded at this task, you can imagine using it to generate new rides. It could provide the functionality of Google Maps or Ways without ever requiring hard-coded maps or navigation algorithms, just sequences of trips. So when we ran this exercise, the model looked good. It proposed legal terms nearly 100% of the time and it could find valid routes between new points 98% of the time. So had the model discovered the world model of Manhattan? Answering this question first requires defining a world model. And there's a natural definition here. The map of Manhattan can be described by a structure known as a deterministic finite automaton or DFA. A DFA is pretty much a way to represent a set of rules. And it's made up of two things. A set of states. Here, each state is an intersection in Manhattan. And transition rules, the legal turns at each intersection and where they take you. For each taxi ride being taken, its trajectory can be tracked by a DFA, which tells you where each turn ends up. And while the DFA here is a map of Manhattan, DFAs are a general way to describe structure and can be used to model many other kinds of tasks. This notion gives rise to a definition which is that a generative model like a transformer recovers a DFA if every sequence that generates is valid in the DFA and vice versa. Now, there's a little bit I'm glossing over here, which is what does valid or invalid mean in the context of a transformer? But you can imagine this being determined by some probability threshold that a sequence is valid according to a transformer if all of a token probabilities are above 1% or something. Now, while it may seem intractable to compare these two enormous sets, this definition gives rise to a nice result, which is that if a model always predicts legal single next tokens, it must have recovered the DFA. This connects to how most LLMs are trained, which is to make accurate single next token predictions. And it also suggests a test. Measure how often a model's predicted single next tokens are valid. But there's a problem with this test. To see, let's consider a simplified game called cumulative connect 4. This is like the real game of Connect 4 where players take turns placing tiles in columns and moves are allowed in unfilled columns except there are n rows and people keep placing tiles in the board even if there are four in a row. The next token test here works by providing a model with the beginning of a connect four game and seeing what percent of the time it predicts legal next move. But it turns out there's a very simple model that gets 99% accuracy for a large enough board. always predict everything is legal. The model has clearly understood none of the structure of the world, but it does well because many states have the same possible next tokens. For example, these two boards are clearly different, but the set of legal next moves allowed all of them is the same. So, while perfect next token prediction implies world model recovery, nearperfect next token prediction doesn't mean much. While single next tokens aren't enough to differentiate states, there's a classic result from language theory that's here to help. The mile and the road theorem states that for every pair of states, there's some number k where there's a continuation of length k that's allowed by one state and not the other. So for example, a and b here are two separate intersections or states in Manhattan. They have the same legal next turns, so they would look the same if you just looked at next token prediction. They also have the same set of legal next two turns. But crucially they don't have the same set of legal next three turns. So here the myhole neurode boundary is three. There exists some k that differentiates states but it doesn't necessarily need to be one. Now this result motivates new metrics for testing world models which we call compression and distinction. At a high level, these metrics go beyond next token prediction and instead go to the full boundary defined by the myel road theorem. There's more details in the paper which is linked below, but briefly compression tests whether a model recognizes that the same state can be reached in different ways like there are multiple ways to get from 14th Street to Time Square and a model should provide the same continuations no matter how you got to where you are now. distinction is a little more general and it says that if two sequences lead to distinct states, a model shouldn't distinguish their length K continuations where K is defined by the my Holden road boundary. So how about some results? We train models on multiple kinds of data, not just shortest paths, but also simulated traffic and random walks and show that all models have greater than 99.9% next token accuracy. But they perform poorly on these metrics. They fail to compress sequences that lead to the same state and they incorrectly differentiate sequences that lead to different states. Now, at this point, you may be thinking, hold up, why should we care about world models? After all, I just told you that the model could find shortest paths. The reason to care is that not having the right world model means a model can perform poorly on different but related tasks. I'll show you one such task, which is detours. When we add detours by forcing models to take certain turns, they often fail to reroute. Models that navigate well without detours perform poorly once they're there precisely because they haven't learned a coherent world model. We also try to visualize each model's implicit map of Manhattan. I I won't go into the details here, but we did this by trying to reconstruct each model's map of Manhattan in a way that would be generous to the model. As a sanity check, we tried doing graph reconstruction on the true world model and found that it reconstructed the true map. We also tried adding transcription errors to sequences from the true model to match the transformers error rate and found the reconstructed map to be imperfect but largely sensible. But when we tried to reconstruct the transformers map, we found nonsense. The model assumed many roads that didn't exist existed and also added physically impossible flyovers. And this was all despite being generous to the model in the way we reconstructed the graph. So what's happening here? The most helpful way for me to think about it is that you and I have a single map of Manhattan. If I'm on 14th Street and I go a block north, I don't throw out my map. I just shift my eyes because I know I'm on 15th Street. Meanwhile, transformers have to reconstruct a map at every turn they make. If these reconstructions are inconsistent, the world model is incoherent. We also extended these metrics to large language models focusing on logic puzzles. And most LLMs we tried incorrectly differentiated between two ways of arriving at equivalent states indicating poor world models for these kinds of logic problems. So to summarize, we've looked at a definition of world model based on DFAS and evaluation metrics inspired by the myhill road theorem. On both our Manhattan test beds and other applications, models could achieve good predictive performance while forming poor world models. While these definitions and tests are specific to today's generative models, we've been here before. These results relate to the idea of the Rashimon effect which was coined by Leo Briman in his 2001 paper the two cultures. The effect describes the fact that two separate regression or classification models can have similar performance in dramatically different ways. And it's relevant here because it shows that a model here can achieve nearperfect prediction without recovering structure from the true world model. But what if a model gets perfect predictions? All of these results have been for one notion of world model, but sometimes we may want another notion. To see why, it's helpful to look at an example. So consider the problem of predicting how planets move in the night sky. Physicists and astronomers have worked on this problem for centuries, and a breakthrough model was offered by the Dutch astronomer Johannes Kepler in the 17th century. He used geometric properties to pinpoint the future locations of planets in the night sky. These properties couldn't explain why the planets move the way that they did, but they offered nearperfect predictions. A little later, Isaac Newton built off this progress to develop rules, which we now know as Newtonian mechanics, to predict orbits. These mechanics could not only predict orbits, but they could also explain their movement. So, who was right? In one sense, both were right. They both made perfect orbital predictions. We'd be fine with both if we only cared about predicting future movements, which is what the definitions so far have addressed. But Newton provided more generality. The same laws he developed could be used to solve new problems. Anything ranging from pendulums to cannonballs to rockets. And many of the uses of foundation models we're excited by involve this kind of generality, such as fuchter and context learning. So, I'll now shift and focus on how we may measure world models if we care about performing well at many different tasks that involve shared structure. It's helpful to think about a foundation model as a learning algorithm. It takes in a small amount of data from a new task and gives us a new predictive model for that task. Now, I'm not going to specify the way it adapts. We can think about it generally, but we can think about this as being via maybe fine-tuning or end context learning. And again, it's helpful to go back to a theoretical result, the no launch theorem for learning algorithms. Loosely, this theorem states that every learning algorithm has an inductive bias towards some set of functions, or in other words, problems that's better at solving from a limited amount of data. This gives rise to a natural notion of a world model, a restriction over functions described by a state space. In the illustration to the right, every row is a function, and the shadings describe different values we'd allow the function to take, which all obey a state structure. With these tools, we can see that a foundation model's inductive bias reveals its world model. In other words, how a model behaves when it extrapolates to small amounts of data reveals its structure. We came up with a method for testing this, which we call an inductive bias probe. The probe has two steps. Given a foundation model, we apply it to many small synthetic tasks that obey some world model. We then look at statistical patterns in the functions it learns. We look at how it extrapolates to see if it follows patterns that would be dictated by the true world model. So as an example, let's consider the case where the state space is discrete. Here two metrics pop out. One is whether a model's learned functions respect state. The picture in the middle shows what happens when the learn functions don't respect state. Another is the opposite, whether a model's learn functions successfully distinguish state. And a failure of this is illustrated on the right. So these metrics test both kinds of failures and they're analogous to type one and type two errors in classification. As an example, we consider a 1D state tracking problem. Essentially a very small map with K states in a line. And while we find good inductive bias for small states, we find that inductive biases worsen quickly for many different kinds of models, we also find that generally states space models like RNN's and Mamba are consistently better than transformers. So we also try these metrics with models trained on planetary orbits. We train a transformer to predict the future locations of planets across many solar systems. And we find that like Kepler, the model makes good predictions. But has it learned Newtonian mechanics? When we use the inductive bias probe, we find a low inductive bias toward Newtonian mechanics. The model makes similar predictions for orbits with different states and different predictions for orbits with similar states. To illustrate this, we try fine-tuning the model to predict the force vectors between planets using a small amount of data. Force vectors are a cornerstone of Newtonian mechanics. So, a model that's using Newtonian mechanics should easily pick this up. But we find that the transformer struggles to learn force. It learns something nonsensical. When we use a symbolic regression to try to estimate the implied force law of the model, we find the law to not only be nonsensical but also fickle. It changes depending on the galaxy it's applied to. And it's not just our domain specific transformer. We find that LLM which have surely been trained on the text of Newton's laws struggle at this too. So if inductive biases aren't toward the true world models, what are they toward? One possibility is that models which are trained to predict next tokens conflate sequences that have similar legal next tokens even if they correspond to very different states. As an example, two different boards can have the same allowed set of legal next tokens. When we fine-tune a model trained on a moves to predict the boards, we find that it often reconstructs boards incorrectly but reconstructs them well enough so that the legal next moves from the reconstructed board are correct. This suggests that foundation models may only recover enough of state to calculate next tokens. So in this multiple task setting, we've use a definition of a world model to be functions that obey a state space. We developed inductive bias probes as evaluation metrics and used planetary orbits as a test bed that also extended to other problems. Now I want to take a few moments to go over some related ideas. The metrics I've described have taken a functional approach to evaluation. They've evaluated models by their performance on input output pairs. But there's another possible approach that's mechanistic. Evaluating a model by its inner workings. The field of mechanistic interpretability works on developing tools for understanding the inner workings of neural networks. There are a few goals, but a big one is using this understanding to steer model performance in some way. For example, the Claude team at Enthropic released a demo of a Claude model, which would always divert conversations to be about the Golden Gate Bridge. Of course, you wouldn't actually want a model that always talks about the Golden Gate Bridge, but you can use similar tools to steer models so that they're used safely. Now, there are many interesting results which I don't have time to get to about adapting these methods to study world models. For example, the researchers from the Athell paper I discussed earlier show that you can intervene on the model's activations so that the model predictably acts like a different board. Now, if we comprehensively understand the inner workings of a model, we could use this understanding to evaluate if a model has understood the world. But how feasible is comprehensive understanding? It turns out it's quite challenging. Chris Ola, one of the leaders of the field, has called this the dark matter of neural networks, a large fraction that cannot be easily understood or interpreted. Neil Nandanda, another leader of the field who's now a deep mind, had another way to say it. If you're aiming to explain 99.9% of a model's performance, there's probably going to be a long tale of random crap you need to care about. We can see why this makes it challenging to evaluate world models mechanistically. The original GBT paper found an emergent nonlinear internal representation of board state. A follow-up study found that there was actually a linear representation and that the reason for the discrepancy was that a fellow boards can be represented to humans in different but equivalent ways. But another follow-up found that a GBT actually learned a bag of huristics instead. not a coherent board, but a bundle of rules like if the move A4 was just played and B4 is occupied and C4 is occupied, update B4 plus C4 plus D4. A rule that doesn't generalize across the board. And I think this makes clear where mechanistic tools are useful because fortunately these tools can still be used to edit models to make specific improvements even if we don't have comprehensive understanding. But evaluating world models requires comprehensive measurement. We want to be able to take any general procedure and see if it improves a world model. If we only understand part of a model, we can't compare two models by only looking at the parts that we understand. Now, another related idea is to study model architectures by looking at their theoretic capacities. For example, a really interesting literature tries to understand which formal languages can theoretically be recognized by different architectures. And these results can guide the kinds of ways we use models. Another related idea is the use of world models in reinforcement learning or RL in RL. A world model has a technical definition that's somewhat different from how I've been using it. A world model in RL is a predictive model of an environment's dynamics. For example, if we want to train an agent to play a video game, we may have it develop a world model of the game or how its actions result in different outcomes. These world models are trained on state explicitly which differs from our goal of evaluating implicit states. And moreover, the goal in RL isn't necessarily recovering structure. It's primarily about making better predictions about an environment or improving an agent's planning capabilities. So, we've seen that generative models like transformers can do amazing things with incoherent world models, but this incoherence makes them fragile for other tasks. So, where should we go from here? One possibility is to accept the fact that our world models are imperfect. Fortunately, models don't need to have correct world models to be useful. So, one approach is to zoom in and evaluate models in specific places such as based on the places where people use them. But of course, we should also work on improving world models. And there are many ways of doing this. One is to design new architectures that are designed to have better world models. And we've seen some promising results from state space models. Another approach is a neurosy symbolic one, combining neural networks with formal reasoning modules like probabilistic programs. We don't need to stop at new architectures. We can also think about new training procedures such as those that go beyond next token prediction. Another possibility is to find better ways to incorporate human feedback into world model training. Or we can also try training models by taking inspiration from ideas from causality and causal representation learning. Overall, there are many promising ways we can work on improving world models, and evaluation metrics will help get us there.