Explainer

How transformers work: tokens, attention, and text generation

A separate visual guide to embeddings, Q/K/V, attention arithmetic, causal masks, transformer blocks, and the KV cache.

By Saud AlotaibiPublished 4 October 202612 min readSources checked 4 October 2026
The takeaway

Trace the representations, permitted context, weighted value mixtures, and token predictions to understand a transformer.

A language model receives a sentence, represents it as numbers, and repeatedly updates those representations before predicting what comes next. The central question is how one position can use information from other positions. In a transformer, attention provides a learned way to combine that information.

This article explains transformers as their own subject: tokens, embeddings, queries, keys, values, causal masking, and generation. It uses a small arithmetic example and three diagrams. All example weights and vectors are chosen for teaching; they are not extracted from a commercial model.

What makes a transformer a transformer?

The 2017 paper Attention Is All You Need introduced an encoder-decoder architecture built around attention rather than recurrent or convolutional sequence processing [1]. A transformer contains more than attention: embeddings, feed-forward transformations, residual connections, normalization, and an output mechanism all matter.

The practical idea is to update a token representation using relevant context. Consider “The bank approved the loan” and “We sat beside the river bank.” The token associated with bank can receive different contextual representations. That does not require a dictionary rule that explicitly selects a meaning; learned computations can make the representation depend on surrounding information.

Do not confuse this with a guarantee of understanding or factual accuracy. The architecture describes how information is processed. Whether a particular model performs a task well depends on its training, data, objective, and evaluation.

Tokens: the units the model processes

A tokenizer turns text into token identifiers. A token may represent a word, part of a word, punctuation, or another unit, depending on the tokenizer. Different tokenization algorithms make different choices [2].

For an easy diagram, imagine the sentence “The cat slept” as three tokens. A real tokenizer could split it differently. The token identifier is an index; its numerical size does not tell you how important the token is or what it means.

Tokenization affects sequence length. Two sentences with the same number of words can require different numbers of tokens. The same applies across languages and scripts, which matters when evaluating Arabic or bilingual material. Inspect the actual tokenizer rather than estimating all usage from English word counts.

Embeddings: identifiers become vectors

The token identifier selects a learned embedding vector. A vector is an ordered collection of numbers that the network can transform. The dimensions do not usually come with simple human-readable labels such as “animal” or “finance.”

An initial embedding differs from the contextual representation produced after several transformer blocks. The same initial token can acquire different representations in different sequences. TensorFlow's transformer tutorial shows embeddings and positional information as part of the input processing [3].

Imagine an input tensor with batch size 2, sequence length 5, and model dimension 8. Its shape is 2 × 5 × 8: two sequences, five token positions per sequence, eight features per token. These dimensions help you understand the computation without pretending that each feature has a fixed verbal meaning.

Position: order must enter the computation

The sentences “The dog chased the cat” and “The cat chased the dog” contain the same main words but describe different events. A model needs information about position to distinguish sequences.

Position can enter through positional encodings, learned position embeddings, or other designs. The original transformer used sinusoidal position encodings; subsequent designs can differ [4]. Do not assume every model uses the same position method because it belongs to the transformer family.

Position information and a causal mask solve different problems. Position helps represent order. A causal mask restricts which positions can supply information during a prediction. Neither should be described as replacing the other.

Queries, keys, and values

For each token representation, learned projections create a query, a key, and a value. In self-attention, all three come from the same sequence of representations. They have different roles [5].

Swipe or scroll sideways to see every column.

ObjectRoleHelpful question
Query, QUsed to score information sources for the position being updatedWhich sources match what this position needs?
Key, KCompared with queries to produce scoresHow does this source match a query?
Value, VSupplies the information mixed into the resultWhat information will be carried forward?

Those questions are a teaching analogy. Q, K, and V are numerical projections learned during training, not literal questions or manually assigned semantic tags.

A query from one position is compared with keys from permitted positions. The resulting scores determine mixing weights. The values are then combined using those weights. Attention therefore uses one computation to decide the mixture and another set of vectors to supply its contents.

Attention, worked through with numbers

Scaled dot-product attention can be written as:

Attention(Q, K, V) = softmax(QKᵀ / √d_k + mask) V

The dot products give scores, the square-root term scales them, the mask excludes prohibited positions, and softmax turns the permitted scores into weights. The matrix multiplication with V produces weighted mixtures [5].

Use this original toy example. Suppose one query has already produced scaled scores of 2, 1, and 0 for three permitted positions. Softmax exponentiates each score and divides by their sum:

exp(2) ≈ 7.389
exp(1) ≈ 2.718
exp(0) = 1
Sum ≈ 11.107

Weights ≈ [0.665, 0.245, 0.090]

Now choose simple value vectors:

V_A = [1, 0]
V_B = [0, 1]
V_C = [1, 1]

Mixture ≈ 0.665 V_A + 0.245 V_B + 0.090 V_C
        ≈ [0.755, 0.335]
An illustrative attention calculation mixes three value vectors with weights 0.665, 0.245, and 0.090, producing approximately 0.755 and 0.335.
An illustrative attention calculation mixes three value vectors with weights 0.665, 0.245, and 0.090, producing approximately 0.755 and 0.335. Open full size

The attention weights sum to one. The coordinates of the resulting value vector do not need to sum to one: the output is a representation, not necessarily a class-probability distribution. In a real model the vectors are larger, and the projections are learned rather than supplied by this example.

For a sequence length of 5 and one head dimension of 4, Q and K each have a 5 × 4 shape. QKᵀ produces a 5 × 5 score matrix: one row per query position and one column per key position. After masking and softmax, multiplying by a 5 × 4 value matrix returns five updated vectors with four features each.

Following those shapes is often more revealing than saying the model “looks at the sentence.” You can see exactly which quantities are being compared and what the operation returns.

Causal masking: preventing a peek at the answer

A decoder used for next-token prediction must not access future positions. A causal mask lets a position attend to itself and earlier positions while blocking later positions. The triangle in the picture shows this rule for a five-position sequence [6].

A five-by-five causal mask permits keys at or before each query position and blocks future keys. The lower triangle, including the diagonal, is allowed.
A five-by-five causal mask permits keys at or before each query position and blocks future keys. The lower triangle, including the diagonal, is allowed. Open full size

Imagine training on “The cat slept.” The representation at the position for cat can be used to predict slept. It may use the and cat, but cannot read slept as context for that prediction. Targets are shifted relative to the corresponding input positions.

Training can calculate many positions in parallel because the full training sequence is known and the mask enforces the restriction. During ordinary autoregressive generation, future generated tokens do not yet exist. The model must obtain a token, append it, and proceed to the next prediction. Parallel training and sequential generation are therefore compatible ideas.

Multi-head attention: several learned mixtures

Multi-head attention runs several attention operations using different learned projections, then combines their results [7]. Each head can use a different representation subspace. This gives the layer several ways to mix information rather than forcing everything through one set of attention weights.

It is tempting to label a head “grammar” or “facts.” Such labels require evidence and are not built-in responsibilities. A head may behave differently across inputs and layers; heads can also overlap in behavior.

For a conventional toy design with a model width of 8 and two heads, each head might use a width of 4. Both heads produce results that are combined and projected back into the model representation. Real designs can use different query and key/value head arrangements, so check the architecture rather than treating this arithmetic as universal.

The complete block: attention is one part

A simplified pre-normalization decoder block normalizes token vectors before causal attention, adds a residual, normalizes before a feed-forward network, and adds another residual.
A simplified pre-normalization decoder block normalizes token vectors before causal attention, adds a residual, normalizes before a feed-forward network, and adds another residual. Open full size

The diagram uses a simplified pre-normalization layout. Other layouts place normalization differently. A residual connection adds a sublayer's output to the representation that entered that sublayer. Normalization controls the scale of features under its defined rule. A feed-forward network transforms each position's features; in a standard dense block, the same feed-forward parameters are applied at every position [8].

Attention mixes information across positions. The feed-forward sublayer performs a nonlinear transformation within each position's representation. Repeating blocks allows successive transformations of the contextual information.

A residual is not permission to skip reading the rest of the block. It is an addition in the computation. Similarly, normalization is not the same as preprocessing raw training data. It operates on internal representations at its specified location.

Some architectures alter feed-forward layers, attention patterns, normalization, or position handling. The diagram provides a useful reference layout without claiming that every current model uses exactly these components in exactly this order.

Encoder, decoder, and encoder-decoder

These names describe architecture and information flow. They should not be confused with converting a token identifier to a vector or decoding a string for display.

Swipe or scroll sideways to see every column.

ArchitectureTypical information flowCommon application pattern
EncoderToken representations can use surrounding input contextClassification, retrieval representations, and token labeling.
DecoderCausal attention restricts future-token accessAutoregressive sequence generation.
Encoder-decoderA decoder uses its own permitted history and attends to encoder representationsTranslating or transforming an input sequence into an output sequence.

In an encoder-decoder model, cross-attention uses queries from the decoder and keys and values from encoder representations [8]. Self-attention uses its sequence's own representations. The distinction concerns the sources of Q, K, and V; it does not imply two unrelated kinds of mathematics.

Architecture identifies a processing design. The training objective and the application still determine what the model is being asked to do. A fluent interface can conceal those differences, which is why a technical comparison should identify the actual configuration.

From representations to the next token

For a decoder language model, the final representation is mapped to scores over the vocabulary. These scores are logits. A probability conversion and a decoding rule determine which token is selected. The selected token is appended, and generation continues until a stopping condition is met [3].

Suppose a toy vocabulary contains three candidate tokens with logits [2, 1, 0]. Their softmax probabilities are the same approximately [0.665, 0.245, 0.090] calculated earlier. Here the numbers describe token choices, while the earlier example used probabilities to mix value vectors. Same mathematical function, different role.

Choosing the highest-probability token gives one result; sampling can give another. Temperature changes the distribution used for selection. These controls affect output behavior but do not insert new factual evidence into the model.

During training, observed tokens supply targets for an objective such as next-token cross-entropy. During ordinary inference, the model computes outputs using its parameters. A conversation becoming longer does not by itself mean the model's weights are being updated. Keep temporary context and parameter learning conceptually separate.

The KV cache: saving repeated work

When generating successive tokens, earlier key and value representations can be reused. A KV cache stores these quantities so the model need not recompute all of them at every decoding step [9]. This can improve generation efficiency, but the cache consumes memory.

Think of appending another line to a running calculation: you retain reusable intermediate quantities and compute what is new. The cache is temporary working state for the sequence, not a permanent memory of every conversation and not a copy of the training dataset.

Exact cache size depends on sequence length, layers, head arrangement, batch size, and storage format. Long prompts and many simultaneous sequences can therefore create memory pressure even when the stored model weights fit comfortably.

Caching also does not erase the need to consider preceding information. A new query still uses the relevant cached keys and values. The saved work is more specific than saying generation becomes cost-free after the first token.

What attention does not prove

A bright cell in an attention map says something about one mixture under one layer and head. It is not automatically a complete explanation of the output. Many later computations, residual paths, and heads contribute to the final result.

A large context limit establishes a capacity constraint, not a guarantee of perfect recall or reasoning. To evaluate a long-document task, check whether the model retrieves the correct details, preserves exceptions, and gives answers supported by the supplied evidence.

For Arabic, use the actual tokenizer and task examples. Test names, dates, mixed-language technical text, and the register your readers expect. A smooth English response provides limited evidence about these requirements.

A learning path that makes the mechanism concrete

First, tokenize a short sentence and inspect its identifiers. Next, track an embedding tensor's dimensions. Then calculate the toy softmax and weighted mixture yourself. After that, draw the causal triangle and explain why each blocked cell must be blocked.

Finally, follow a small encoder-decoder tutorial and identify self-attention, cross-attention, feed-forward transformations, residuals, and the output layer. The TensorFlow tutorial [3] is an implementation route; the original paper [1] establishes the design; the mathematical chapters explain the individual operations; Hugging Face documents tokenization and caching in practical systems.

My view for Saud Alotaibi’s Opinions is that transformers become much easier to understand when you trace information rather than memorize labels. What is represented at this position? Which positions can contribute? How are their values mixed? What turns the resulting representation into a token prediction? Those questions give you a reliable way to read a diagram, inspect code, and challenge vague claims about how a model works.

Sources & further reading