Use this page as the short navigation map of the course. For the full day-by-day release order, use [[LLM/How to Create an LLM from scratch and deploy it|How to Create an LLM from scratch and deploy it]]. ```mermaid flowchart TD A["Core concepts"] --> B["Training and evaluation"] B --> C["Runtime and deployment"] C --> D["Final chatbot workflow"] D --> E["Optional advanced track"] ``` ## Main Path Follow the main path in this order: 1. learn the concept in the note 2. open the notebook or smallest code surface 3. connect it to the relevant `picollm/accelerated` file 4. move to the product and deployment notes once the model path is clear ## Main Path Milestones 1. Understand the model’s input and representation layers. Notes: [[LLM/concepts/01 - Tokenization|Tokenization]], [[LLM/concepts/02 - Embedding Layer|Embedding Layer]], [[LLM/concepts/03 - Positional Encoding|Positional Encoding]] 2. Understand the main Transformer operators and blocks. Notes: [[LLM/concepts/04 - Scaled Dot-Product Attention|Scaled Dot-Product Attention]], [[LLM/concepts/05 - Multi-head Attention|Multi-head Attention]], [[LLM/concepts/06 - Feed-Forward Network|Feed-Forward Network]], [[LLM/concepts/07 - Layer Normalization|Layer Normalization]], [[LLM/concepts/08 - Encoder Block|Encoder Block]], [[LLM/concepts/09 - Decoder Block|Decoder Block]], [[LLM/concepts/10 - Causal Language Modeling|Causal Language Modeling]] 3. Understand how the model is trained and measured. Notes: [[LLM/concepts/11 - Training Loop|Training Loop]], [[LLM/concepts/12 - Training Configuration and Hyperparameters|Training Configuration and Hyperparameters]], [[LLM/concepts/14 - Evaluation and Model Quality|Evaluation and Model Quality]], [[LLM/concepts/17 - Experiment Tracking and Run Analysis|Experiment Tracking and Run Analysis]], [[LLM/concepts/18 - Research Workflow and Ablations|Research Workflow and Ablations]] 4. Understand inference, runtime, and system behavior. Notes: [[LLM/concepts/13 - Inference and Sampling|Inference and Sampling]], [[LLM/concepts/22 - Failure Modes and Debugging|Failure Modes and Debugging]], [[LLM/product/23 - Inference Runtime and KV Cache|Inference Runtime and KV Cache]], [[LLM/concepts/15 - Compute, Time, and Cost of LLMs|Compute, Time, and Cost of LLMs]], [[LLM/concepts/16 - Distributed Training and Multi-GPU|Distributed Training and Multi-GPU]], [[LLM/concepts/24 - Quantization|Quantization]] 5. Understand data shaping and post-training. Notes: [[LLM/concepts/19 - Data Curation and Dataset Quality|Data Curation and Dataset Quality]], [[LLM/concepts/20 - Chat Format and SFT|Chat Format and SFT]], [[LLM/product/21 - SFT Flow|SFT Flow]] 6. Understand serving, deployment, and product clients. Notes: [[LLM/product/25 - Serving, Latency, and Observability|Serving, Latency, and Observability]], [[LLM/concepts/26 - FastAPI Chat App|FastAPI Chat App]], [[LLM/concepts/27 - Deployment|Deployment]], [[LLM/product/28 - OpenTUI Terminal Chat App|OpenTUI Terminal Chat App]], [[LLM/product/29 - Real Chatbot Workflow|Real Chatbot Workflow]], [[LLM/product/30 - picollm Code Map|picollm Code Map]], [[LLM/product/31 - Nanochat Architecture|Nanochat Architecture]], [[LLM/product/32 - Vercel AI SDK Chat App|Vercel AI SDK Chat App]] ## Optional Advanced Track Take the advanced track after the main path if you want the research and systems layer: - [[LLM/concepts/33 - Scaling Laws and Compute-Optimal Training|Scaling Laws and Compute-Optimal Training]] - [[LLM/concepts/34 - Optimizer Theory for Transformer Training|Optimizer Theory for Transformer Training]] - [[LLM/concepts/35 - Advanced Distributed Training Systems|Advanced Distributed Training Systems]] - [[LLM/concepts/36 - Advanced Inference Systems|Advanced Inference Systems]] - [[LLM/concepts/37 - Formal Evaluation and Benchmarking|Formal Evaluation and Benchmarking]] - [[LLM/concepts/38 - Reproducibility and Research Method|Reproducibility and Research Method]] - [[LLM/concepts/39 - Post-Training Beyond SFT|Post-Training Beyond SFT]] - [[LLM/concepts/40 - Safety and Alignment Evaluation|Safety and Alignment Evaluation]] - [[LLM/concepts/41 - Advanced Data Engineering for LLMs|Advanced Data Engineering for LLMs]] - [[LLM/concepts/42 - Interpretability and Mechanistic Analysis|Interpretability and Mechanistic Analysis]] ## Code Layers The runnable course material appears in three layers: - [notebooks/](https://github.com/Montekkundan/llm/tree/main/notebooks) for live walkthroughs - [course_tools/](https://github.com/Montekkundan/llm/tree/main/course_tools) for small concept-first runtimes - [picollm/accelerated/](https://github.com/Montekkundan/llm/tree/main/picollm/accelerated) for the serious [[LLM/Glossary#Tokenizer|tokenizer]], training, eval, and chat path Useful external comparisons: - [rasbt/LLMs-from-scratch](https://github.com/rasbt/LLMs-from-scratch) - [nanochat](https://github.com/karpathy/nanochat)