1AI

Blog

Notes on building 1AI.

Getting 300+ AI models to work together. What breaks, what works, what I learned. And a deep dive into each of the 30 papers Ilya Sutskever recommended.

Notes and guides

Sutskever 30

About the series
  1. Paper 01Why Coffee Mixes But Never UnmixesScott Aaronson's First Law of Complexodynamics explains why complexity rises, peaks, then falls. Paper #1 from Sutskever's 30.12 min
  2. Paper 02The Unreasonable Effectiveness of RNNsKarpathy's famous 2015 post showed RNNs could generate Shakespeare and Linux code by predicting one character at a time. Paper #2.10 min
  3. Paper 03Understanding LSTM NetworksChristopher Olah's 2015 post explained LSTM gates with clarity that textbooks lacked. Paper #3.8 min
  4. Paper 04Recurrent Neural Network RegularizationZaremba, Sutskever, and Vinyals figured out how to apply dropout to LSTMs without breaking them. Paper #4.7 min
  5. Paper 05Keeping Neural Networks SimpleHinton and van Camp showed that penalizing weight complexity leads to better generalization. Paper #5.8 min
  6. Paper 06Pointer NetworksVinyals, Fortunato, and Jaitly repurposed attention to point at input positions instead of blending hidden states. Paper #6.8 min
  7. Paper 07AlexNetKrizhevsky, Sutskever, and Hinton's 2012 ImageNet victory proved deep learning could outperform hand-engineered computer vision. Paper #7.9 min
  8. Paper 08Order MattersSets have no order, but neural networks need sequences. How to handle permutation invariance. Paper #8.8 min
  9. Paper 09GPipePipeline parallelism for training giant neural networks. Micro-batches keep accelerators busy. Paper #9.8 min
  10. Paper 10ResNetSkip connections let gradients flow through 152 layers. Learning residuals instead of direct mappings. Paper #10.7 min
  11. Paper 11Dilated ConvolutionsExponentially expanding receptive fields without losing resolution. WaveNet's secret for audio generation. Paper #11.6 min
  12. Paper 12Graph Neural NetworksMessage passing on graphs. Nodes aggregate information from neighbors to learn representations. Paper #12.7 min
  13. Paper 13Attention Is All You NeedThe Transformer architecture replaced recurrence with self-attention. Foundation of GPT, BERT, and modern AI. Paper #13.9 min
  14. Paper 14Neural Machine TranslationNeural machine translation by jointly learning to align and translate. The original attention mechanism. Paper #14.7 min
  15. Paper 15Identity Mappings in ResNetMoving activation before convolution enables training 1000-layer networks. Paper #15.6 min
  16. Paper 16Relation NetworksRelation Networks compare all object pairs to answer questions about relationships. Paper #16.6 min
  17. Paper 17Variational AutoencodersLearning to generate data by encoding it into structured latent spaces. ELBO and the reparameterization trick. Paper #17.7 min
  18. Paper 18Relational Recurrent Neural NetworksMemory slots that attend to each other enable multi-step reasoning. Paper #18.6 min
  19. Paper 19The Coffee AutomatonWhy does coffee mix but never unmix? Entropy, coarse-graining, and the arrow of time. Paper #19.8 min
  20. Paper 20Neural Turing MachinesDifferentiable computers with external memory banks. Content and location-based addressing. Paper #20.7 min
  21. Paper 21CTC LossTraining speech models without frame-level alignment. The blank token and alignment-free learning. Paper #21.7 min
  22. Paper 22Scaling LawsPower laws for neural language models. How performance scales with compute, data, and parameters. Paper #22.8 min
  23. Paper 23The MDL PrincipleMinimum Description Length balances model complexity against fit. Occam's razor made mathematical. Paper #23.7 min
  24. Paper 24Machine SuperintelligenceFormal definitions of machine intelligence. Recursive self-improvement and the path to superintelligence. Paper #24.8 min
  25. Paper 25Kolmogorov ComplexityThe shortest program that outputs a string. Incompressibility equals randomness. Paper #25.7 min
  26. Paper 26CNN FundamentalsConvolutional layers, pooling, ReLU, and backpropagation. Stanford's CS231n course distilled. Paper #26.8 min
  27. Paper 27Multi-Token PredictionPredicting multiple tokens at once improves sample efficiency and enables speculative decoding. Paper #27.6 min
  28. Paper 28Dense Passage RetrievalLearning embeddings for questions and passages. Dual encoders and contrastive learning beat BM25. Paper #28.7 min
  29. Paper 29Retrieval-Augmented GenerationCombining retrieval with generation. External knowledge grounds language models in facts. Paper #29.7 min
  30. Paper 30Lost in the MiddleLanguage models struggle with information in the middle of long contexts. The U-shaped attention pattern. Paper #30.6 min

Updates

I write when I ship something or figure something out. Maybe twice a month. No tracking, no growth hacks.