Attention Is All You Need

Model architecture · 2017

How does a model find the relevant fact in changing context? Train a small attention head, change a delivery schedule, and watch its answer follow the record—not a memorized date.

Google Brain · Google Research · University of Toronto · Affiliations in the paper

TRY THIS

Change Mira’s delivery day without retraining.
Reorder the records: the answer should stay the same.
Compare learned attention, equal attention and untrained weights.

START WITH THE IDEA

The idea in brief.

The Transformer is a sequence-to-sequence architecture built around attention. At each layer, a token can gather information from other positions by comparing a query with their keys, then mixing their values. Multiple heads, position information and feed-forward layers make this more than a single lookup.

Recurrent path · schematic

Position 1 → state
Position 2 → state
Position 3 → state

Self-attention · one position’s view

Query ↔ all keys
Normalize relevance
Mix their values

A conceptual comparison, not a runtime benchmark. The original Transformer also needs position encodings, multiple heads, feed-forward layers and masking. RNNs and CNNs remain useful in other settings.

THE SIGNIFICANCE

Why it matters.

A different route through a sequence

Recurrent networks pass state through successive positions; convolutional models connect positions through local layers. This paper used attention throughout its encoder and decoder, allowing distant positions to interact directly within a layer.

Sections 1 and 4 ↗

More parallel work during training

Removing recurrence reduced the sequential work within a training sequence. The authors reported strong translation results with less training time than the systems they compared. Autoregressive generation still predicts successive output tokens; parallel training does not mean every part runs at once.

Sections 3 and 6 ↗

A mechanism you can inspect

Our small labs expose scores, weights and contributions. They isolate attention’s selection and mixing operation, so you can change inputs and follow each result. They do not reproduce the full Transformer or its translation results.

Hopper implementations ↗

A WORKING EXAMPLE

Find a changing fact

Runs locally · No API key

Train a small query/key attention head, then change who receives a delivery on which day. The answer should follow the record’s current value.

Compare the approaches. Compare learned attention, untrained weights and equal weighting. For exact structured records, an ordinary lookup is the simpler production choice; this lab makes learned selection visible.

Run this example yourself

From the implementation repository after cloning:

node packages/attention-is-all-you-need/cli.mjs

Source & instructions ↗

A small attention head with query/key embeddings trained by gradient descent in the browser. Structured name/day records expose context retrieval; not a language model.

CHOOSE THE RIGHT TOOL

When to use it.

Hopper’s application guidance. Compare on your own task before adopting a more complex approach.

Use it

Learn how a model retrieves and mixes context

Use these small labs to inspect scores, masks, weights and contributions before reading a full model implementation.

For a production language model, start from a maintained implementation and evaluate it on your actual task.

Start simpler

Look up an exact customer, name or delivery date

A map, SQL query or indexed search gives a deterministic answer without training.

Use the schedule lab to study learned matching, not as a replacement for a database.

Start simpler

Rank records or estimate from overlapping features

Compare with a category filter, dot-product ranking or a simple average. Attention adds normalized contributions and a weighted mixture.

Keep the extra mechanism only if held-out outcomes show that it helps. Our hand-authored project features are illustrative.

Avoid here

Treat high attention weight as truth or certainty

A weight shows contribution to this computation. It does not certify that a fact is correct or that an estimate is reliable.

Use source verification and task-specific evaluation; handle missing or contradictory records explicitly.

A small, usable interface.

import { trainLookup, lookupDelivery } from './packages/attention-is-all-you-need/workflow.mjs';

const model = trainLookup({ epochs: 800, seed: 42 });
const context = [
  { person: 'Mira', day: 'Thursday' },
  { person: 'Noah', day: 'Friday' }
];
console.log(lookupDelivery(model, context, 'Mira'));
// answer: Thursday; also returns scores, weights and values.

Free open-source code · GitHub ↗

Implementation scope

This isolates the retrieval operation inside attention: a query selects relevant information by matching keys, then combines values. In a Transformer, learned projections form these vectors from token representations and multiple heads operate together. Understanding this operation helps you read model code and reason about context, masking and information flow.

Use a database query for a real delivery schedule. The schedule here makes a neural operation legible; it is not a claim that a neural network is the best tool for this task. The model knows six name identities and five day values. It does not parse language, understand arbitrary names or implement the complete Transformer.

node packages/attention-is-all-you-need/cli.mjs