Pre-training

Pre-training is the initial, foundational phase of training an AI model on a large, diverse dataset. During pre-training, the model learns general patterns of language, knowledge, and reasoning that serve as the foundation for all downstream tasks.

For language models, pre-training involves: assembling a massive training corpus (trillions of tokens from books, websites, code, and more), training the model to predict the next token given preceding context, and running this process for weeks or months on thousands of GPUs.

Through this process, the model learns: grammar and language structure, factual knowledge about the world, reasoning and logic patterns, coding abilities, multilingual capabilities, and conversational patterns.

Pre-training is distinguished from: fine-tuning (additional training on a specific dataset after pre-training), RLHF (Reinforcement Learning from Human Feedback — aligning the model with human preferences after pre-training), and in-context learning (adapting behavior through examples in the prompt, no additional training).

Pre-training is extremely expensive. Training a frontier model like GPT-4 or Claude 3 Opus costs tens to hundreds of millions of dollars in compute. This is why only a handful of organizations train foundation models from scratch, while thousands of companies build applications on top of these pre-trained models.

For AI agent builders, pre-training has already been done by model providers (OpenAI, Anthropic, Google, Meta). Builders benefit from pre-training through the models' capabilities — language understanding, reasoning, and knowledge — which they customize through prompts, knowledge bases, and actions.