How a Large Language Model Works

You send a prompt, and a second later a response starts appearing in the terminal. It doesn't arrive all at once. It grows piece by piece, as though someone were typing it right now. Sometimes Thinking hangs on the screen first; the answer takes noticeably longer, but is often more useful.

All of this comes from one mechanism. Let's unpack how a large language model works: what it does with your text, how it measures its length, where its knowledge comes from, and what happens during those seconds while it's thinking. You don't need any of these facts just to press Enter. Together, though, they can change what you expect from the response.

A language model estimates what comes next

Let's start with the basics. A large language model, or LLM, isn't an application with windows and buttons, and it isn't a chat. It's a mechanism that receives the text of a conversation and returns a continuation. Claude Code, a web chat, and suggestions in an editor are all wrappers around that call, and each wrapper works differently.

The model itself does one thing: given the beginning of a text, it estimates which continuation is most likely. It doesn't search for the correct answer. It assigns probabilities, not to one option but to every option in its vocabulary.

A short phrase is the easiest way to get a feel for this. Complete this out loud: "I drink my coffee without..." "Sugar" probably comes to mind first, then "milk." "Internet" doesn't fit at all, even though the grammar is fine. You made that judgment in a fraction of a second without thinking about it. The model does something similar, but formally and exhaustively: it goes through its vocabulary and gives every candidate a score.

Notice what the diagram doesn't show. No option is marked as correct; there are only more and less likely continuations. These scores don't come from counting how often the exact phrase appeared in a collection of texts. That was a much simpler kind of technology, abandoned long before today's models. The probability is calculated from patterns captured inside the model during training. That's why it can score a sentence no one has ever written before.

How an LLM differs from a chat and an agent

These three terms are often used interchangeably, even though they describe different layers of a system.

• LLM - the language model itself. It receives context and generates the next message.
• Chat - the interface and application around an LLM. It preserves the conversation history, assembles the request, sends it to the model, and displays the response.
• Agent - an LLM with a goal, a work loop, and tools. It can ask the application to read a file, perform a search, or run a test, inspect the result, and choose the next step.

In short, an LLM proposes a solution, a chat organizes the conversation, and an agent connects the model's decisions to actions in the outside world.

Tokens, not words

The diagram above needs one correction: it shows whole words only to make the idea easy to see. In reality, a model breaks text into tokens, pieces from its own vocabulary. A token can be a whole word, part of a word, or a punctuation mark. A space is usually bundled into the following token rather than treated as a separate unit.

The vocabulary is built in advance from a large body of text. The process is gradual: an algorithm finds frequently occurring character sequences and merges them into larger pieces. Common sequences get their own tokens. That's why the finished vocabulary contains a mixture of complete words and word fragments.

Every token has a number in the vocabulary, so text enters the model as numbers: first the string is split into pieces, then each piece is replaced with its number. The response is converted back into readable text with the same vocabulary.

Two details are worth remembering. First, every model family has its own vocabulary, so models don't split the same string in exactly the same way. Rules of thumb such as "one token is about three quarters of a word" are only rough estimates for English. Second, a language that appeared less often in the training data tends to be split into smaller pieces. A German word may become two or three tokens where its English equivalent fits into one.

This is easier to see for yourself. Open The Tokenizer Playground, paste in a couple of your everyday prompts, then add the same text in another language. You'll immediately see how the string breaks apart and how many pieces it produces. One caveat: the tokenizer there belongs to a specific model, and the model you use may split the text differently. Visit it to see the effect, not to get an exact number for every model.

This has an unexpected practical consequence. The model sees pieces rather than letters, so questions about individual letters are noticeably harder for it than questions about meaning. Ask it to count how many times "o" appears in a long word, and the answer may well be wrong. Not because the task is difficult, but because the word may have reached the model as two pieces and the individual letters inside them aren't directly visible to it.

How a response is built from tokens

Now let's put the mechanism together. It's simpler than it might seem. The model scores the options, one is selected and appended to the text, and then the process starts again with the same text plus one more token.

The finished response isn't stored anywhere or fetched from somewhere else. It's built in as many steps as it contains tokens. Nor does it stop on a timer: at some point, a special token meaning "the end is here" becomes the chosen continuation, and generation stops.

The same diagram reveals another detail. The model can't erase what it has written: once a token enters the text, it stays there and becomes part of every following step's input. The model can correct itself in exactly the same way a person can in conversation - with the next sentence.

Here's a consequence that surprises many people: the same question asked twice can produce two different answers. At each step, the selected token isn't necessarily the most likely one, and one divergent token can pull everything that follows in a new direction. Most of the time only the wording changes, but sometimes the proposed solution changes too.

Why the model is called large

The word "large" doesn't describe the size of a single file sitting somewhere. It refers to two entirely different kinds of scale, and both are useful to understand.

The first is the amount of text the model encountered during training: books, articles, documentation, forums, code, and more, in quantities no person could read in several lifetimes. The second is the number of parameters. A parameter is a number inside the model, one of the settings adjusted during training. There are billions of them. Open models often put the count right in the name: 7B, 14B, or 70B, where B means billion.

And here's where it's easy to make a mistake. Parameter count isn't a quality score. Results also depend on the training data, the model's architecture, the tasks it was tuned for, and how it was compressed to fit on particular hardware. A smaller model can easily outperform a larger one on a specific job. A big number in the name promises nothing by itself.

At this scale, one more capability seems to emerge almost by itself. To continue a sentence about a failed test coherently, knowing which words usually appear together isn't enough. The model also has to capture something about how tests work. It learned to predict words, while facts and relationships described by those words settled into its parameters along the way. That's where its apparent knowledge comes from, even though the underlying mechanism hasn't changed.

Where the model's knowledge comes from

There are exactly two sources, and confusing them can be costly.

The first is the parameters. They hold the patterns learned during training. This isn't a card catalog where every entry carries a source: knowledge is distributed across many numbers, and you can't retrieve an exact quotation from them. That's why a model can produce something almost correct: the form looks right and the name sounds familiar, but a specific detail is wrong.

The second source is whatever is in the conversation right now: your prompt, everything said before it, the contents of open files, and the output of commands that have been run. This source is precise but short-lived. It doesn't move into the parameters, and the next conversation knows nothing about it.

Everything else that might look like a separate source - a memory file, documentation found online, or a response from an external service - arrives by the same route. It first enters the conversation and only then affects the response. The model itself has no private doorway into your project or the internet.

The first source has a date, known as the knowledge cutoff. It's a rough boundary for how current the model's built-in knowledge usually is. For a library released after that date, the model knows only what has been placed into the current conversation.

The caveat matters more than the date itself. A cutoff doesn't mean the model knows everything before that day and nothing after it. There can be gaps inside the covered period: perhaps little was written about a subject, sources contradicted one another, or the relevant material never entered the training data. The date sets an upper boundary, not a promise of completeness below it.

One small but useful point. Asking the model for its own cutoff isn't a great plan: it'll generate the answer just like any other, which means it may choose something plausible. The exact date belongs in the documentation for that specific model version, and nearby models from the same family can have different cutoffs.

What Thinking means

When Thinking appears on the screen, the model hasn't switched into some other state of consciousness. It's simply been given room to reason before it starts answering. Technically, this is still generation: the model first works through a draft for itself, breaking down the task, considering options, and challenging its own ideas, then produces the final response. Modern models largely decide how much reasoning to use. For a simple question, the draft may be short or absent; for a tangled problem, it may be longer.

An example shows why this helps. An agent is looking for the cause of a failing test. The first plausible explanation points to the client for an external service, but a neighboring test and the error message suggest something else: the problem is in data parsing. With enough room to reason, the model can notice the contradiction and change direction before proposing an edit. The gain isn't the length of the reasoning. It's that the check changed the next step.

The first explanation is quite often the wrong one. It's plausible, but plausibility was the very criterion the mechanism used to select it.

So when is extra reasoning worth it? It costs time, and there's no need to use it for every small task.

The rule of thumb is simple: the more branches and dependencies a task contains, the more useful it is to give the model room to work through them. When renaming a variable, you can inspect the finished result faster than you can read the reasoning about it.

One honest caveat: displayed reasoning isn't a precise transcript of what happened inside the model. It's often a summary prepared for the reader. A long, tidy chain can grow from a false premise and look more reliable than it is. The opposite happens too: the answer has already been found, but the model keeps second-guessing itself, goes in circles, and becomes confused. There is even a name for this - overthinking.

What the model cannot do

The list is short, and it's better to know it now than discover it in the middle of a real task.

• Guarantee that a fact is correct. The mechanism chooses a likely continuation; it doesn't verify each claim against a source. Invented but plausible details are called hallucinations.
• Ignore the wording. Reordering words or adding a clarification changes the response because everything depends on the text that comes before it.
• Learn new information by itself. Built-in knowledge doesn't update without external sources. The model knows about yesterday's library release only if someone shows it the relevant information.
• Repeat itself word for word. The same prompt can produce a different result. That's a property of the mechanism, not a malfunction.
• Escape biases in the source material. If something was described one-sidedly or not described at all, that imbalance will echo in the answers.

Almost everything built around language models is a response to one item or another on this list. Search and documentation tools exist because built-in knowledge becomes outdated. Tools and agents exist because a model on its own can only produce language.

What this explains about an agent's behavior

You can see the effects of this mechanism directly in Claude Code:

• a response appears piece by piece because that's how it's created. This is the process itself, not a visual effect;
• two identical prompts produce different answers - that's expected because generation doesn't repeat itself exactly. If you need an earlier result, returning to the one you already have is more reliable than asking again;
• Thinking provides room for a draft, not a guarantee of correctness. It pays off when untangling a multi-step task and often just slows down a variable rename;
• until the agent reads a file, anything it says about that file rests on general patterns. Information placed in the conversation has more direct influence than knowledge stored in the parameters;
• by default, the model knows little about recent library versions. The simplest fix is to bring current documentation into the conversation;
• if a response goes in the wrong direction, adding the missing information works better than repeating the same request. The mechanism relies on the text before the answer, not on how insistently you ask.

Understanding the mechanism doesn't make the model any less useful. It simply clarifies what to expect: not an infallible reference book, but a remarkably good text-completion system that continues the beginning you give it.

Video: the same mechanics in pictures

3Blue1Brown has a short video that brings the essentials together in one clear animation: tokens, next-word probabilities, attention, and where a model's parameters come from. It's called Large Language Models explained briefly.

The original is in English, but the video also has several translated audio tracks that you can select in the player settings. If the language you need isn't listed, open the video on YouTube itself, where the web player may offer more options.

That isn't the channel's only video on the subject. It also has a full series on neural networks, transformers, and attention - the same ideas in greater depth and with mathematics. If this article leaves you wanting to understand not just "what happens" but "why it works at all," that series is a good place to start.