The Glance That Became a Mechanism
Bahdanau remembered doing translation exercises when he was learning English in school: to write each next word, he had to glance all around the source text to figure out what should come next. Source sentence on the left, his writing on the right, his head turning left-right, left-right. For decades, machine translation could not do this. It read a sentence once, squeezed everything into a fixed amount of memory, and then tried to write. The longer the sentence, the more meaning was lost in the squeeze.
In 2014, a 23-year-old intern named Dzmitry Bahdanau, working in a University of Montreal lab under the computer scientist Yoshua Bengio and the postdoc Kyunghyun Cho, was asked to get a computer to translate English into French. [1] He had coded competitively since childhood but had never done academic research. He quietly doubted that artificial neural networks — systems that learn by strengthening and weakening connections between nodes in response to training data — could reach anything like human-level intelligence. His adviser and the postdoc were believers.
The task was chosen well. Translation cannot be reduced to swapping one word for another, because languages carry different grammatical structures, idioms, and connotations. There are no clean rules to implement. Neural networks, which learn by example, seemed the right instrument: train them on huge databases of paired English and French sentences and let them absorb the quasi-rules and exceptions. But results were mediocre, and the reason was architectural. Researchers were cramming each sentence’s data into a fixed memory before decoding it in the target language — a bottleneck that capped how complex a translation could be.
Bahdanau’s response was to reproduce the child’s head-turn in mathematics. In a paper with Cho and Bengio, he introduced an operation the three called “attention,” which let the network identify the word or words anywhere in a source sentence that would most inform the next word of the translation. It worked on the very first attempt, he recalled. “It was just obviously the right thing to do,” he later recalled from Montreal, where he still works in AI — “one of these things that happens once in a while in science where it’s just very clear to someone who is not very experienced in the field.” [1] Their network translated long sentences as well as short ones, which meant the approach could scale.
Queries, Keys, and the Weight of Context
The problem attention solves is easiest to see in a pronoun. Take the sentence “The animal didn’t cross the street because it was too tired,” and then its twin, “The animal didn’t cross the street because it was too wide.” French forces a choice: “it” is masculine for the animal (il) and feminine for the street (elle). A machine must work out which noun the pronoun leans on. That is not a rule you can look up; it is a judgment about context.
In 2017, a team at Google devised a new architecture called a transformer that supercharged the attention method and stripped away everything else. Its paper, “Attention Is All You Need,” reported state-of-the-art translation at a fraction of the computation earlier models required, and it has since been cited more than 250,000 times. The design works like this: each word entering the network becomes a vector — a list of numbers that can be pictured as an arrow pointing in some direction — and trained numbers called weights transform that arrow into three new ones, a “query,” a “key,” and a “value.” An attention head compares each word’s query against the keys of all the other words. The better two vectors line up, the more strongly that other word’s value flows into an updated representation of the word at hand. Context literally becomes arithmetic.
In the tired-animal sentence, one head’s query for “it” can be read as a request: I need an earlier noun that could plausibly be tired. The key attached to “animal” advertises exactly that capacity, matches the query better than the key for “street,” and so the animal’s value pours into “it.” The enriched representation passes to the next layer, where the process repeats. Later layers build on the associations earlier ones established. No one programmed the network to know that animals tire and streets spread wide; it discovered such links during training, through trillions of small weight corrections sent backward from each translation error, until it could handle sentences it had never seen.
The second half of the transformer, the decoder, then writes the French one word at a time, consulting the encoder’s numerical picture of the English as it chooses each word — which is how “it” comes out as the masculine il. A whole grammar of relevance, assembled from nothing but examples.
From One Pronoun to the Whole Internet
The leap from translation to general capability rests on a plain observation. A network that can render English sentences in French can also finish sentences in either language, because predicting the next word of a translation is a subtask of predicting the next word, full stop. That is what large language models do. Trained on the contents of the internet, they try trillions of times to guess what comes next in whatever corner of the web they happen to be reading, and their attention patterns sharpen until they are useful for any context — or for answering a prompt from a human.
Llion Jones, one of the authors of “Attention Is All You Need” and now chief technology officer of Sakana AI, suspects that this next-word prediction, known as language modeling, is “AI-complete”: a task so hard that solving it amounts to solving artificial intelligence in general. [1] His reasoning is that the task contains nearly everything humans do in writing as a subtask. The internet holds translations, so completing a sentence that asks for the French of the tired-animal line requires translating. Completing “2 + 3 is [blank]” requires arithmetic. A run of mathematical symbols demands high-level mathematics. “That’s quite a leap, isn’t it?” he says — and it is, from one pronoun to a revolution.
What remains open is the question that determines how far the finding reaches. Attention is a way to understand how a single model can converse, diagnose diseases, generate realistic videos, and solve long-standing math problems, and it may sit near the heart of intelligence, artificial and otherwise. But the mechanism is only as good as what its weights absorbed, and those weights were shaped by text. Whether that is enough to call attention the essence of intelligence is the question the field is still working out.
