LoRA: Low-Rank Adaptation of Large Language Models

TL;DR

Fine-tunes a large model by freezing its weights and training tiny low-rank update matrices instead, cutting the tunable parameters by orders of magnitude.

Why it matters

LoRA made customizing large models affordable: you train a few million parameters instead of billions, and adapters are small and swappable. It underpins most practical LLM fine-tuning today.

Key ideas

• Represent each weight update as a product of two small low-rank matrices.
• Freeze the base model; train only the adapters.
• Adapters add no inference latency once merged back in.
• Small, portable adapters let one base model serve many tasks.

Related concepts