1AI

30 papers

Sutskever 30

Ilya Sutskever's recommended reading list. The 30 research papers he considers essential for understanding AI.

Back to all posts
  1. #01Why Coffee Mixes But Never UnmixesScott Aaronson's First Law of Complexodynamics explains why complexity rises, peaks, then falls.12 min: read more
  2. #02The Unreasonable Effectiveness of RNNsKarpathy's famous 2015 post showed RNNs could generate Shakespeare and Linux code by predicting one character at a time.10 min: read more
  3. #03Understanding LSTM NetworksChristopher Olah's 2015 post explained LSTM gates with clarity that textbooks lacked.8 min: read more
  4. #04Recurrent Neural Network RegularizationZaremba, Sutskever, and Vinyals figured out how to apply dropout to LSTMs without breaking them.7 min: read more
  5. #05Keeping Neural Networks SimpleHinton and van Camp showed that penalizing weight complexity leads to better generalization.8 min: read more
  6. #06Pointer NetworksVinyals, Fortunato, and Jaitly repurposed attention to point at input positions instead of blending hidden states.8 min: read more
  7. #07AlexNetKrizhevsky, Sutskever, and Hinton's 2012 ImageNet victory proved deep learning could outperform hand-engineered computer vision.9 min: read more
  8. #08Order Matters: Seq2Seq for SetsSets have no order, but neural networks need sequences. How to handle permutation invariance.8 min: read more
  9. #09GPipePipeline parallelism for training giant neural networks. Micro-batches keep accelerators busy.8 min: read more
  10. #10Deep Residual Learning (ResNet)Skip connections let gradients flow through 152 layers. Learning residuals instead of direct mappings.7 min: read more
  11. #11Dilated ConvolutionsExponentially expanding receptive fields without losing resolution. WaveNet's secret for audio generation.6 min: read more
  12. #12Graph Neural NetworksMessage passing on graphs. Nodes aggregate information from neighbors to learn representations.7 min: read more
  13. #13Attention Is All You NeedThe Transformer architecture replaced recurrence with self-attention. Foundation of GPT, BERT, and modern AI.9 min: read more
  14. #14Bahdanau AttentionNeural machine translation by jointly learning to align and translate. The original attention mechanism.7 min: read more
  15. #15Identity Mappings in Deep Residual NetworksMoving activation before convolution enables training 1000-layer networks.6 min: read more
  16. #16Relational ReasoningRelation Networks compare all object pairs to answer questions about relationships.6 min: read more
  17. #17Variational AutoencodersLearning to generate data by encoding it into structured latent spaces. ELBO and the reparameterization trick.7 min: read more
  18. #18Relational Recurrent Neural NetworksMemory slots that attend to each other enable multi-step reasoning.6 min: read more
  19. #19The Coffee AutomatonWhy does coffee mix but never unmix? Entropy, coarse-graining, and the arrow of time.8 min: read more
  20. #20Neural Turing MachinesDifferentiable computers with external memory banks. Content and location-based addressing.7 min: read more
  21. #21CTC LossTraining speech models without frame-level alignment. The blank token and alignment-free learning.7 min: read more
  22. #22Scaling LawsPower laws for neural language models. How performance scales with compute, data, and parameters.8 min: read more
  23. #23The MDL PrincipleMinimum Description Length balances model complexity against fit. Occam's razor made mathematical.7 min: read more
  24. #24Machine SuperintelligenceFormal definitions of machine intelligence. Recursive self-improvement and the path to superintelligence.8 min: read more
  25. #25Kolmogorov ComplexityThe shortest program that outputs a string. Incompressibility equals randomness.7 min: read more
  26. #26CNN Fundamentals (CS231n)Convolutional layers, pooling, ReLU, and backpropagation. Stanford's CS231n course distilled.8 min: read more
  27. #27Multi-Token PredictionPredicting multiple tokens at once improves sample efficiency and enables speculative decoding.6 min: read more
  28. #28Dense Passage RetrievalLearning embeddings for questions and passages. Dual encoders and contrastive learning beat BM25.7 min: read more
  29. #29Retrieval-Augmented GenerationCombining retrieval with generation. External knowledge grounds language models in facts.7 min: read more
  30. #30Lost in the MiddleLanguage models struggle with information in the middle of long contexts. The U-shaped attention pattern.6 min: read more