Transformer

A transformer is a neural network architecture introduced in the 2017 paper "Attention Is All You Need." It's the foundation of virtually all modern large language models, including GPT, Claude, Gemini, and Llama.

Key innovation: Transformers use "attention" to process all parts of an input simultaneously, rather than sequentially like previous architectures (RNNs, LSTMs).

Why it matters:

• Much faster to train (processes all tokens in parallel)
• Better at capturing long-range dependencies
• Scales efficiently to massive datasets
• Handles variable-length inputs naturally

The transformer architecture enabled the creation of models with billions of parameters, trained on trillions of words—the foundation of the current AI revolution.

Core components:

1. Tokenization Input text is split into tokens (words or subwords).

2. Embeddings Each token is converted to a vector representation.

3. Positional encoding Since attention processes all tokens at once, position information is added to preserve word order.

4. Attention layers Multiple "attention heads" learn which tokens to focus on when processing each position.

5. Feed-forward layers Process the attention outputs through additional neural network layers.

6. Output Final layer predicts the next token or produces the desired output.

Transformers typically stack many layers (GPT-3 has 96 layers), each refining the representation.

Attention lets the model focus on relevant parts of the input when processing each position.

Example: "The cat sat on the mat because it was tired."

When processing "it," attention helps the model focus on "cat" to understand what "it" refers to.

How attention works:

For each token, the model computes:

• Query: "What am I looking for?"
• Key: "What information do I have?"
• Value: "What do I return if selected?"

Attention scores determine how much each token influences others. High scores = strong influence.

Multi-head attention: Multiple attention "heads" run in parallel, each learning different relationships. One head might track subjects, another might track verbs, another might track sentiment.

Encoder-only (BERT-style) Process input and output a representation. Good for understanding tasks: classification, question answering, embeddings.

Decoder-only (GPT-style) Generate text token by token. Good for text generation, chatbots, code completion.

Encoder-decoder (T5, BART) Encode input, then generate output. Good for translation, summarization, and seq2seq tasks.

Modern LLMs: Most current models (GPT-4, Claude, Llama) use decoder-only architecture optimized for generation.

Variations:

• Sparse transformers: Efficient attention for very long sequences
• Flash Attention: Faster, memory-efficient attention computation
• Mixture of Experts: Only activate parts of the model for each input

Parallelization Unlike RNNs that process tokens sequentially, transformers process all tokens at once. This enables training on massive GPU clusters.

Scalability Transformers improve predictably with more parameters, more data, and more compute. This "scaling law" drove investment in larger models.

Flexibility The same architecture works for text, images, audio, and video with minor modifications.

Long-range understanding Attention can connect distant tokens, understanding relationships across thousands of words.

Pre-training effectiveness Transformers excel at self-supervised learning—predicting masked or next tokens—enabling training on unlimited text data.

Transfer learning Pre-trained transformers adapt well to specific tasks with minimal fine-tuning.

The transformer is to modern AI what the transistor was to computing—a foundational innovation that enabled everything built on top.