References

The sources behind this book, grouped by where they're most useful. Everything here is either a primary paper, a canonical textbook, or the official docs for a tool we used.

Foundational textbooks

1. Ian Goodfellow, Yoshua Bengio, Aaron Courville. Deep Learning. MIT Press, 2016. The standard reference for the math of neural networks, backprop, and regularization. Free at deeplearningbook.org.

2. Christopher Bishop. Pattern Recognition and Machine Learning. Springer, 2006. The classic on linear models, kernels, and the bias-variance tradeoff.

3. Trevor Hastie, Robert Tibshirani, Jerome Friedman. The Elements of Statistical Learning. Springer, 2009. Deep coverage of regularization (ridge, lasso), cross-validation, and classical ML. Free PDF online.

4. Kevin Murphy. Probabilistic Machine Learning: An Introduction. MIT Press, 2022. Modern, comprehensive, with excellent treatment of the numerical and probabilistic foundations.

Landmark papers

5. Vaswani et al. Attention Is All You Need. NeurIPS, 2017. arXiv:1706.03762. The transformer. The architecture behind every modern LLM (Ch 13, Ch 15).

6. Mikolov et al. Efficient Estimation of Word Representations in Vector Space. 2013. arXiv:1301.3781. word2vec; the king - man + woman ≈ queen embeddings (Ch 14).

7. Kingma & Ba. Adam: A Method for Stochastic Optimization. ICLR, 2015. arXiv:1412.6980. The default optimizer (Ch 8).

8. Rumelhart, Hinton & Williams. Learning representations by back-propagating errors. Nature, 1986. The paper that made backprop famous (Ch 8).

9. Srivastava et al. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. JMLR, 2014. (Ch 9).

10. He et al. Delving Deep into Rectifiers (He initialization). ICCV, 2015. arXiv:1502.01852. Why initialization scale matters (Ch 11).

11. Cortes & Vapnik. Support-Vector Networks. Machine Learning, 1995. The SVM and the kernel trick (Ch 5).

12. Lewis et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS, 2020. arXiv:2005.11401. RAG (Ch 15).

13. Breiman, Leo. Random Forests. Machine Learning, 2001. Bagging and the random forest (Ch 20).

14. Chen & Guestrin. XGBoost: A Scalable Tree Boosting System. KDD, 2016. arXiv:1603.02754. The gradient-boosting workhorse (Ch 20).

15. Hu et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR, 2022. arXiv:2106.09685. Efficient fine-tuning (Ch 27).

16. Ho, Jain & Abbeel. Denoising Diffusion Probabilistic Models. NeurIPS, 2020. arXiv:2006.11239. Diffusion generative models (Ch 27).

17. Kohavi, Tang & Xu. Trustworthy Online Controlled Experiments. Cambridge, 2020. The definitive A/B testing reference (Ch 24).

18. Andrej Karpathy. nanoGPT and Let's build GPT from scratch. The clearest reference for the GPT project (Ch 31).

19. Yao et al. ReAct: Synergizing Reasoning and Acting in Language Models. ICLR, 2023. arXiv:2210.03629. The agent loop (Ch 33).

Further reading by topic

• Classical ML & stats: Hastie/Tibshirani/Friedman, The Elements of Statistical Learning (free PDF); James et al., An Introduction to Statistical Learning (gentler) (Ch 20 to Ch 23).
• Linear algebra: Gilbert Strang, Introduction to Linear Algebra / MIT 18.06 lectures (Ch 23).
• MLOps: Huyen, Designing Machine Learning Systems; Google's MLOps white papers (Ch 26).
• ML system design interviews: Aminian/Xu, Machine Learning System Design Interview (Ch 28).

Tools & documentation

• NumPy: numpy.org/doc. The array library underpinning every chapter.
• PyTorch: pytorch.org/docs. Tensors, autograd, nn, DataLoader (Ch 12, Ch 16).
• scikit-learn: scikit-learn.org. Reference implementations of the classical models, metrics, and preprocessing we built by hand.
• Hugging Face: huggingface.co/docs. Pretrained transformers and tokenizers (Ch 15).

Sister books in this series

These build directly on the foundations here:

• HNSW from Scratch: fast approximate nearest-neighbor search; the engine behind embedding retrieval and RAG (Ch 14).
• IVF & Product Quantization from Scratch: the memory-efficient, compressed side of vector search.
• Kernel Temporal Segmentation from Scratch: uses the kernel matrix (Ch 5) to segment video.
• Recommendation Systems from Scratch: embeddings, matrix factorization, and a production capstone (MLflow, FastAPI, RAG) that exercises this whole book.

This book's code

• code/recipes.py, similarity, softmax, normalization, top-k, encodings.
• code/features.py, one-hot, bag-of-words, TF-IDF, scaling.
• code/nn_from_scratch.py, a 2-layer MLP with hand-derived backprop.
• code/pytorch_intro.py, the same network in PyTorch.

Capstone projects (code/projects/):

• projects/gpt.py, a transformer language model, trained and sampled.
• projects/finetune_lora.py, full fine-tuning vs. LoRA, compared.
• projects/agent.py, a ReAct tool-using agent (pure Python).
• projects/cnn.py, a CNN image classifier trained end to end.
• projects/diffusion.py, a diffusion model that generates from noise.

The Part I-VIII code depends only on NumPy; the capstone projects need PyTorch (the CPU build is fine), except agent.py, which is pure Python.