LLM Roadmap

Reading List

Curated papers, blogs, and posts. The foundation papers are worth reading in full; the rest are worth skimming for mental models.

·
The Illustrated Transformerblog

Best visual explanation of attention. Read this before the original paper.

transformersarchitecture
Attention Is All You Needpaper

The paper that started everything. Focus on the architecture section.

transformersarchitecturefoundational
BERT: Pre-training of Deep Bidirectional Transformerspaper
transformerspre-trainingfoundational
Language Models are Few-Shot Learners (GPT-3)paper

Introduced in-context learning as a paradigm. Focus on the scaling results.

llmfew-shotfoundational
Training Language Models to Follow Instructions with Human Feedback (InstructGPT)paper

Foundation of modern chat models. Understand RLHF from first principles here.

rlhfalignmentfine-tuning
Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelspaper
promptingreasoning
ReAct: Synergizing Reasoning and Acting in Language Modelspaper

The paper behind most production agent loops.

agentsreasoning
LoRA: Low-Rank Adaptation of Large Language Modelspaper

Makes fine-tuning accessible without massive GPU budgets.

fine-tuningefficiency
QLoRA: Efficient Finetuning of Quantized LLMspaper
fine-tuningquantization
RAGAS: Automated Evaluation of Retrieval Augmented Generationpaper

Go-to eval framework for RAG pipelines.

rageval
Retrieval-Augmented Generation for Large Language Models: A Surveypaper

Use as a reference, not a linear read.

ragsurvey
Constitutional AI: Harmlessness from AI Feedbackpaper

Anthropic's approach to training helpful and harmless models.

alignmentsafety
Toolformer: Language Models Can Teach Themselves to Use Toolspaper
agentstool-use
What We Learned from a Year of Building with LLMsblog

Practitioner wisdom from teams who have shipped. Read this early and re-read often.

productionapplied
Building LLM Applications for Production — Chip Huyenblog
productionengineering