How a large language model works

A primer on modern LLMs

A large language model does one thing. Given some text, it works out how likely each possible next word is, and picks one. A chatbot, a coding assistant and a “reasoning” model are all that one step, repeated.

This post goes through that step with a model small enough to check by hand. It carries three numbers per word where a real model carries a thousand or more, it has one layer where a real model has dozens, and it has fewer than a hundred weights where a real model has billions. Every operation in it is the real one. The prompt throughout is “The cat sat on the”, and the question is how the model gets to “mat”.

The figures are live. Each number in them is worked out in your browser from the toy model’s tables, so the values in the first figure carry through to the last. They are rounded to two decimals for display, which means a sum you check by hand can be off by 0.01.

Colour means the same thing in every figure:

  • Stored in the model. It never changes while the model runs.
  • Worked out fresh for this text.
  • Worked out once, then kept in the KV cache.

A network is a table of numbers

A neural network is tables of numbers plus one rule: multiply, then add. Here is a tiny one, drawn two ways.

Figure 1. Each line is one number in the table, and one neuron’s lines are one column. Thicker lines are bigger numbers and dashed lines are negative ones. Pick a neuron, or point at a number, to find it in both pictures.

The two halves are the same object. The left is how people draw a network. The right is what is stored on disk.

Each number says how strongly one neuron pushes one output. The 0.8 in the top-left corner means “when neuron 1 fires, push output 1 up by 0.8 times however hard it is firing”. These stored numbers are the model’s weights. When someone says a model has seven billion parameters, they are counting the cells in tables like this one.

To run the network, each neuron is given a number: how hard it is firing right now. The table stays fixed. The firing strengths change with every input. Suppose neuron 1 fires at 2, neuron 2 fires at 1, and neuron 3 is silent.

Figure 2. Multiply each column by how hard its neuron is firing, then add across each row. Drag a slider to change a firing strength and watch its column rescale.

There are two steps. Scale each column by its neuron’s firing strength, then add across each row. The four totals, 1.2, 0.5, 1.3 and −0.4, are the output.

That is the entire operation, with no hidden extra step. Every table in a language model is used exactly like this, and the results are handed to the next table.

For now the firing strengths 2, 1, 0 are made up. Keep them in mind. By the end of this post the model will have produced those three numbers on its own.

The same table can choose a word

Give the four outputs names, and the same table can pick a word. Say the last word of the text so far is “the”, and suppose it has been turned into the numbers 2, 1, 0.

Figure 3. The network from Figure 1 with the same twelve weights, now choosing the word that follows “the”. Select a word to see its sum. The grey dashed lines come from the neuron firing at 0, so they add nothing.

Reading it left to right:

  1. Token to numbers. “the” becomes 2, 1, 0. How that happens is most of this post.
  2. Multiply and add. Those numbers travel along the lines. The scores are the row totals from Figure 2: 1.2, 0.5, 1.3, −0.4.
  3. One score per word. In this toy, each of the four outputs stands for one word the model could say next.
  4. Scores to chances. Raise e (about 2.718) to the power of each score, then divide by the total. That gives 3.32, 1.65, 3.67 and 0.67, which add up to 9.31, so “mat” gets 3.67 ÷ 9.31 = 39%. The step makes every chance positive and makes them add up to 100%. Its name is softmax.
  5. Pick and repeat. The model does not always take the favourite. It draws at random according to these odds, adds the word it drew to the text, and starts again with the longer text.

That loop is the whole product. What is missing is step 1: how the text so far became three numbers that favour “mat” and “sofa” over “banana”. A real model does not get there with one table.

The parts of a language model

Here are the parts in the order the data passes through them, with what the data looks like at each point for our prompt.

Figure 4. One trip through the model, top to bottom. The violet boxes are the model itself, which is tables of numbers. The white boxes are bookkeeping around it. A violet box links to the figure that opens it up.
  • Tokenizer. A fixed dictionary that splits text into pieces called tokens and gives each one an ID number. A token is a word or part of a word.
  • Embedding table. One row per token. Looking up a token’s row gives its starting list of numbers.
  • Layer. Attention followed by an MLP. Each one adds its result onto the token’s list. The model is this same pair stacked many times, and every copy has its own numbers.
  • Output table. Turns the final list into one score for every possible token. It is the table from Figure 1.
  • Pick one token. Scores become odds, and one token is drawn.

One idea runs through all of it. Each token carries a list of numbers, and that list is only ever added to. The embedding table creates it, every attention step and every MLP adds something to it, and the output table reads what is there at the end. In the toy the list has three numbers, so it can stand in for the three neurons of Figure 1.

This design is called a Transformer. For a sense of scale I will use one small real model, Qwen3.5-0.8B: its lists have 1,024 numbers, it has 24 layers, it knows 248,320 tokens, and its weights fill a 1.75 GB file.

Figure 5. The model is a stack of layers, and a layer is a handful of tables. Open a different layer to see the same structure filled with different numbers.

Zoom in and there are no new kinds of object, only more tables. The attention part of a layer holds four tables: query, key, value and write-back. A real layer runs several attention steps side by side, each with its own four tables looking for something different. Each of those is called a head. The toy has one head. The MLP part holds two larger tables with a squash between them. The MLP is the multiply-and-add of Figure 2, used twice.

Here is one whole pass of the toy model over the prompt, a stage at a time. A pass like this, which takes in a whole prompt at once and ends by producing one token, is called prefill. Producing the tokens after that one works a little differently. That is called decode, and it is explained further down.

Figure 6. Prefill: one pass over the whole prompt, through the toy model’s single layer, ending with one token. The stage you are on is drawn at full strength and earlier stages are faded. Decode, which writes the tokens after this one, comes later.

The sections that follow take these stages one at a time, with the numbers.

From tokens to lists

The lists have to come from somewhere, and that is the embedding table.

Figure 7. The embedding stage is a lookup. Choose a token, or any row of the table, to see its row copied out.

There is no arithmetic at this stage.

  1. The tokenizer splits the text and assigns IDs. “The” is 1, “cat” is 2, “sat” is 5, “on” is 4 and “the” is 6. “The” and “the” are different tokens, with different rows.
  2. The ID is a row number. The model goes to that row of the embedding table and copies it out.
  3. Five tokens give five lists. Each token now carries three numbers.

Nothing about the other tokens is involved. “cat” gets the same list in every sentence it appears in. Getting the rest of the sentence into a token’s list is attention’s job.

The numbers in the rows are weights, learned in training like every other table. Tokens that are used in similar ways end up with similar rows. Compare “dog” and “cat”.

Keys and values

Layer 1 begins with every token preparing two things from its list: a key and a value. Each is made by one table, with the multiply-and-add you already know.

Figure 8. A token’s list is multiplied by the key table and by the value table, and both results are stored in that token’s slot of the KV cache. The sums underneath show where each stored number comes from.

The key table has two rows, so a key has two numbers. The value table has three rows, so a value has three.

  • The key is the token’s label. It is what the token advertises to other tokens that come looking.
  • The value is the token’s package. It is what the token hands over if it is chosen.

I have given the three value slots names so the numbers can be read: animal, sitting, and noun due, meaning “a noun should come next”. The package for “cat” says animal. Try “sat”, whose package says sitting, and “the”, whose package says a noun is due. Real models have no such labels. A slot means whatever training made it mean.

Both results are saved for every token, in every layer. That store is the KV cache: K for key, V for value. The list itself is not what gets cached. A token’s key and value are saved because they will never change once made, and every later token is going to need them.

Queries, and where their table comes from

A third table, the query table, turns a token’s list into a query: what that token is looking for. For “the”, whose list is 0.4, 0.6, −0.8, the two rows of the query table give

  • 3.0 × 0.4 + (−2.0) × 0.6 + (−1.5) × (−0.8) = 1.2
  • 0.0 × 0.4 + 0.5 × 0.6 + (−2.0) × (−0.8) = 1.9

A query has as many numbers as a key, because the two are about to be multiplied together slot by slot. In the toy, the first slot is roughly “a thing” and the second is roughly “an action”. The key for “cat” is 1.7, 0.2, which says “I am a thing”. The key for “sat” is −0.9, 1.8, which says “I am an action”. The query from “the” is 1.2, 1.9, which says “I want both”.

So where does a table that asks useful questions come from? Nobody writes it. It starts as random numbers and is shaped by training. The model makes a prediction, the error is measured, and every number in every table is nudged slightly in the direction that would have made the error smaller. Repeat that an enormous number of times and the random grid becomes a useful one.

Here is a miniature version. The query table starts out random, so “the” looks more or less anywhere. Each nudge changes its six numbers so that a little more of the attention lands on “cat” and “sat”.

Figure 9. A query table being trained. The six numbers on the left drift with each nudge, the query they produce changes with them, and the attention moves towards “cat” and “sat”.

Two things are simplified here. In real training nobody tells the table where to look. The only signal is whether the final prediction was right, and the nudge for these six numbers is found by tracing that error backwards through the arithmetic, which comes back in the last section. And every table is nudged at once, the key and value tables included, so they are shaped to work with each other.

For comparison, the query table the toy uses everywhere else puts 82% of the attention from “the” on “cat” and “sat”.

Matching and blending

Now the query meets the keys. Take “sat”, the third token, as the example. It can see three keys, which makes a sum small enough to read in full.

Figure 10. The attention step for “sat”, written out as sums. Keys meet the query to set the shares, then values meet the shares to make the blend.

Read each row from left to right.

  • Left half, keys. The query from “sat” is multiplied by each key, slot by slot, and added up. That gives one score per token. The scores become shares of 100% in the same way scores became chances in Figure 3. The key for “cat” matches best, so “cat” gets 78%.
  • Right half, values. Each token’s value is multiplied by its share, and the results are added up column by column. That sum is the blend: 0.73, 0.18, 0.22. It is mostly the package from “cat”, with a little of the other two mixed in.

Keys decide how much to take from each token. Values are what gets taken. A key is never part of the blend.

Nothing is looked up by key here, the way a dictionary works. Every token in view is always included, and the scores only set the proportions.

Why can “sat” see only three tokens? A token may only attend to itself and the tokens before it. When the model is writing, later tokens do not exist yet, and holding to the same rule everywhere means a token’s numbers come out identical whether or not anything follows it. That fact is what will make the cache work.

All five tokens do this at the same moment, each with its own query.

Figure 11. Every token’s shares and blend. Each row is one token doing what “sat” did in Figure 10, and “later” marks the tokens it is not allowed to see. The green tags are each token’s value. Select a row to see its sums.

“The” can only see itself, so it takes 100% of its own value. Now select the last row. “the” takes 43% from “cat” and 39% from “sat”, and its blend is 0.41, 0.41, 0.36: an animal, sitting, and a noun is due. That is the sentence so far, squeezed into three numbers and delivered to the one token that has to predict what comes next.

One step of attention is left. The blend is written in the value table’s terms, not the list’s. A fourth table, the write-back table, converts it, and the result is added to the token’s own list. For “the”, the blend becomes the update 0.56, −0.28, 0.36, and its list goes from 0.40, 0.60, −0.80 to 0.96, 0.32, −0.44.

The MLP

Attention moved information between tokens. The MLP that follows works on each token’s list alone, with no reference to any other token. It is the network from Figure 1, used twice, with a squash in between. (MLP is short for multi-layer perceptron, an old name for that arrangement.)

Figure 12. The MLP for the last token. Select any row of either table to see the sum behind the number it produces.
  1. Expand. The list of three is multiplied by the first table, which gives five neurons. Each neuron responds to one pattern in the list. In the small real model this step goes from 1,024 numbers to 3,584.
  2. Squash. Any neuron below zero is set to zero. Two of the five are silenced here. The squash matters more than it looks. Without it, two tables in a row could be replaced by a single table, and the MLP would be no more than Figure 2 again. Real models use a smoother version of the same idea.
  3. Compress and add. The surviving neurons are multiplied by the second table, back down to three numbers, and that update is added to the list.

For “the”, the list goes from 0.96, 0.32, −0.44 to 2.00, 1.00, 0.00.

If attention is where a token collects what it needs from the rest of the text, the MLP is where it works out what follows from that.

Choosing the word

After the last layer, one list matters for the prediction: the last token’s.

Figure 13. The final list is multiplied by the output table to give a score for each word, the scores become odds, and a word is drawn at random according to the odds.

This is where the post closes its loop. The final list for “the” is 2, 1, 0, the firing strengths that were made up for Figure 2. The output table is the twelve weights of Figure 1, with its outputs given their names. So the rest is what you have already seen: scores of 1.20, 0.50, 1.30 and −0.40, and odds of 36%, 18%, 39% and 7%.

Then the model draws. Press “Draw a word” a few times. “mat” is the favourite, “sofa” comes up nearly as often, and now and then you get “banana”. Draw 100 times and the tally settles close to the odds. This randomness is why the same prompt can give different answers on different runs.

A real output table has one row for every token the model knows, which is 248,320 rows in the small model. Every pass produces a chance for every one of them.

The other four tokens finished with lists of their own. During this pass, nothing reads them, because only the last list goes to the output table. Their work was not wasted. Their keys and values are in the cache.

One pass, start to finish

We followed one token. In reality all five went through each step together. Here is the whole attention step in one picture, in six stages, for whichever token you choose to follow.

Figure 14. The attention step of layer 1 in six stages. Line thickness shows how much the chosen token takes from each of the others. Tokens to its right are greyed out because it cannot see them.

Tokens do not wait for each other. Within a layer, every token makes its key, value and query from the list it arrived with, so all five can be computed side by side. The rule about earlier tokens is enforced by leaving the later keys out of the match, which is why “The” can run at the same moment as “the”.

The toy skips one thing here. Real models also work each token’s position into the numbers, so that “the cat” and “cat the” do not look the same.

Now the full pass again. Figure 6 drew it with one layer, which is all the toy has and where the worked numbers above came from. Here it is with two, so you can see the second layer repeat the first with its own tables.

Figure 15. The prefill pass of Figure 6 with a second layer added. The stage you are on is drawn at full strength and earlier stages are faded.

Three things to notice as you click through:

  • Each layer keeps its own cache. Layer 2’s keys and values are made from the lists that layer 1 finished with, using layer 2’s tables. They are different numbers.
  • Attention is the only stage with wires between columns. Everything else happens inside one token’s column.
  • Only the last column reaches the output table.

However many layers there are, prefill does the same three things: it processes the whole prompt in one go, fills every layer’s KV cache, and produces one token.

Prefill, decode and the KV cache

Having drawn “mat”, the model needs the token after it. The text is now six tokens long. The obvious thing would be to run all six through the model again. The KV cache makes that unnecessary.

A token’s list, at any layer, depends only on itself and the tokens before it. Adding “mat” on the end changes nothing for “The”, “cat”, “sat”, “on” or “the”. Computing them again would give the same keys and values to the last decimal place. So the model keeps what it has and computes one new column. That is a decode pass.

Figure 16. A prefill pass and the decode pass that follows it, drawn the same way so the differences line up. The notes on the right compare them row by row.

The new token does everything a token did during prefill. It gets its list from the embedding table, makes a key and a value that are added to the cache, makes a query, matches it against all six keys, blends the values, goes through the MLP, and at the end reaches the output table. The five older tokens do nothing. Their cached keys are read, their cached values are blended, and that is all.

Prefill handles any number of tokens in one pass, because they all exist already. Decode handles one token per pass, because a token does not exist until the pass before it has picked it. This is why a model reads a long prompt quickly and writes its answer one token at a time.

Two clarifications about the cache:

  • It is not a dictionary. Despite the name, nothing is looked up by key. It is a list for each layer that grows by one entry per token, and attention reads all of it every time. The speed comes from not recomputing things, not from fast lookup.
  • It costs memory. There is a key and a value for every token, in every layer. A long conversation carries a large cache.
Figure 17. A nine-token prompt and a six-token answer. Step through the passes, then turn the cache off and step through again. Each square in a cache row stands for one saved key-value pair.

With the cache, prefill computes nine columns and each decode pass computes one, for 15 in total. Without it, every pass starts over: 9 + 10 + 11 + 12 + 13 + 14 + 15 = 84. The gap widens as the text grows.

That is a complete language model: tables, lists that are added to, attention to move information between tokens, an MLP to work on each token, an output table, and a random draw. The design comes from the 2017 paper Attention Is All You Need. The KV cache is not in that paper. It is an engineering consequence of attention only ever looking backwards.

Reasoning

A reasoning model is the model you have just seen, used in a particular way. Before it answers, it writes out its working in a scratchpad, one token at a time, and then it reads that working back while it writes the answer. This is called chain of thought. It needs no new parts. Every token of the working is an ordinary decode pass.

Nobody programs that behaviour in. It is trained in, with reinforcement learning. The model is given problems whose answers can be checked, it tries each one many times, and it is nudged towards whatever it wrote on the tries that came out right. Tries that worked through the problem step by step came out right more often, so working step by step became the habit.

First, why writing out the working helps. Each token gets one trip through the stack. That is a fixed amount of arithmetic. Ask a model for 17 × 24 and demand the answer at once, and it has to get from the question to 408 in a single pass, with no room to work.

The fix is to let the model write its working first.

<think> 17 × 20 = 340. 17 × 4 = 68. 340 + 68 = 408. </think>
The answer is 408.

Every one of those tokens is produced by an ordinary decode pass. But once “340” has been written, it is in the text. Its key and value are in the cache, and every later token can attend to it. The scratchpad turns one hard step into several easy ones, each of which fits in a single pass. In the products you use, the scratchpad is often hidden, or shown only as a summary.

The costs follow from the mechanism. Reasoning tokens are decoded one at a time like any others, so thinking takes time and fills up the context.

One caution. The scratchpad is text the model generated, not a recording of its arithmetic. It usually tracks what the model did, but researchers have found cases where the written reasoning is not what actually drove the answer.

If putting tokens back in is all it takes, why could models not always do this? Mechanically, they could. Nothing in the architecture had to change. What was missing was the habit. A model trained to continue ordinary text has seen few examples of someone patiently writing out every small step, so it tends to blurt out an answer. Asking it to “think step by step” helped. Making the habit reliable took training, and since nobody has millions of correct scratchpads to train on, we needed a different kind of training, reinforcement learning.

Training a model to reason

Ordinary training shows the model the right next token and nudges the tables towards it. For reasoning there is no right next token to show. There are many good ways to work through a problem, and nobody has written them down. What can be checked is the final answer.

Reinforcement learning, or RL, trains on exactly that. Let the model try, score the result, and make the tries that scored well more likely. It uses the model you now know, placed inside a loop of supporting parts.

Part What it is
Task bank A large collection of problems, each stored with what is needed to check it: a known answer, or a set of tests
Environment A task plus the rules around it: how the prompt is built, what the model may do, when the attempt ends, and how it is scored
Policy The model being trained. Same tables, same prefill and decode. “Policy” is the RL word for it
Rollout One complete attempt at one task: scratchpad and answer
Verifier A program that checks an attempt and returns a number. No AI is involved
Judge A second AI model that scores an attempt, used where no program can
Reward The number the verifier or judge returns
Advantage The reward compared with the other attempts at the same task
Reference copy A frozen earlier copy of the model, used as a leash so training cannot drift too fast
Trainer The code that turns advantages into nudges to the tables

One trip around the loop is a training step. Here is a single step from start to finish, with four attempts at “What is 17 × 24?”. The figures quoted from a real run are from DeepSeek’s R1-Zero, a model trained this way without being shown any worked examples of reasoning.

Figure 18. One training step in eight stages. Use Next, or jump to any stage from the list.

A few points that are easy to miss:

  • Where 1.41 comes from. The rewards are 2, 1, 1 and 0, so the average is 1. Their distances from it are 1, 0, 0 and 1. Square those, average them, and take the square root: the standard deviation is 0.71. Attempt A is 1 above average, and 1 ÷ 0.71 = 1.41.
  • What changes in a step. Only the weights. The attempts, rewards and caches are rebuilt from nothing every time.
  • When the weights change. Once per step, after every attempt in the batch has been scored. In R1-Zero that was one update for every 512 attempts.

Now the same loop, run over and over. The simulation below is a stand-in. Its model is a single number, the chance of a right answer, and the update is the rule from Figure 18 applied to that one number. It shows the shape of what happens.

Figure 19. The training loop under your control. Next moves one stage at a time, Back undoes the last stage, and “Run 25 steps” jumps ahead to show the long climb.

Click through one full step slowly and watch when things change. The tiles fill in at Attempts, get their marks at Verifier, and gain arrows at Compare. Nothing in the weights patch, the chance of a right answer or the chart moves until you reach Weights, which is the only stage where the tables are written. Step back from Weights and the update is undone.

Two more things show up if you run it for a while. When all eight attempts get the same reward, there is nothing to compare, and that step teaches the model nothing. And no single step does much. The climb comes from many small nudges in a row.

The loop improves whatever the verifier rewards, which is not always what was meant. If a set of tests can be passed by special-casing the tests, a model may learn to do that. Writing checks that cannot be gamed is much of the craft.

The same loop now trains models on far more than arithmetic: writing code that has to pass a test suite inside a sandbox, or working a browser or a terminal over many steps. Each of those is an environment in the sense of the table above, with tasks, a harness that lets the model act, and a verifier at the end.

The short version

  • A model is tables of numbers. Using it is multiply-and-add.
  • Each token carries a list that is only ever added to.
  • Attention is the one place where tokens exchange information. A query is matched against keys, the scores become shares, the values are blended by those shares, and the blend is written back.
  • The MLP works on each token alone.
  • The output table turns the last token’s list into odds, and one token is drawn.
  • Prefill reads the prompt in one pass and fills the KV cache. Decode adds one token per pass and reuses the cache.
  • Reasoning is decoding into a scratchpad. Reinforcement learning makes it a habit by rewarding answers that check out.

Appendix: the toy model in full

These are all the weights behind the figures. A list is a column of three numbers, and each table is used as in Figure 2.

Embedding table          Query table            Key table
dog    0.8   0.0   0.4    3.0  -2.0  -1.5        1.5   0.0   0.5
The    0.4   0.4  -0.6    0.0   0.5  -2.0        1.0  -2.0  -1.0
cat    1.0   0.2   0.4
mat    0.0  -0.2  -0.2   Value table            Write-back table
on     0.2   0.4  -0.8    1.0  -0.5   0.0        0.5   0.0   1.0
sat   -0.4  -0.8  -0.6    0.5  -1.0  -0.5       -1.0  -1.0   1.5
the    0.4   0.6  -0.8    0.5   0.0  -1.0       -0.5   0.5   1.0

First MLP table          Second MLP table
 1.5   1.0   2.0          0.5   0.5   0.5  -0.5   1.0
 0.5   1.0   0.0          1.0  -0.5   0.5   1.0   0.0
 0.0  -1.5  -2.0          0.5   0.0   0.0   0.5  -1.5
 0.0  -2.0   2.0
-1.5  -1.0   0.5         Output table
                         sofa     0.8  -0.4   0.2
                         floor   -0.2   0.9   0.1
                         mat      0.5   0.3  -0.7
                         banana   0.1  -0.6   0.4

Where the toy cuts corners:

  • Lists of three numbers, one layer and one head. The small real model has lists of 1,024 numbers and 24 layers.
  • Whole words as tokens, and a vocabulary of seven.
  • No position information in attention.
  • A hard squash in the MLP, where real models use a smooth one.
  • No rescaling of the lists between steps, which real models do to keep the numbers in a steady range.
  • Tables that were chosen by hand so the sums stay readable. A real model’s tables are learned.

Sources