BERT: Pre-training of Deep Bidirectional Transformers

TL;DR

Pre-trains a Transformer encoder to read text in both directions at once by predicting masked-out words, then fine-tunes it for downstream tasks.

Why it matters

BERT popularized the pretrain-then-fine-tune recipe for language and showed that one bidirectional model could set records across many tasks, a template that shaped the encoder side of the field.

Key ideas

• Masked language modeling: hide random tokens and train the model to fill them in.
• Bidirectional context, each token sees words on both sides, unlike left-to-right models.
• One pre-trained model fine-tunes to many tasks with a small task head.
• Demonstrated the power of transfer learning for NLP at scale.

Related concepts