Abstract

1. Algorithms & Data Structures

This class, Algorithms & Data Structures, introduces the most important computer science topics for machine learning, enabling you to design and deploy computationally efficient data models.

Through the measured exposition of theory paired with interactive examples, you’ll develop a working understanding of all of the essential data structures across the list, dictionary, tree, and graph families. You’ll also learn the key algorithms for working with these structures, including those for searching, sorting, hashing, and traversing data.

The content covered in this class is itself foundational for the Optimization class of the Machine Learning Foundations series.

Over the course of studying this topic, you'll:

  • Use “Big O” notation to characterize the time efficiency and space efficiency of a given algorithm, enabling you to select or devise the most sensible approach for tackling a particular machine learning problem with the hardware resources available to you.
  • Get acquainted with the entire range of the most widely-used Python data structures, including list-, dictionary-, tree-, and graph-based structures.
  • Develop an understanding of all of the essential algorithms for working with data, including those for searching, sorting, hashing, and traversing.


2. Optimization

This class, Optimization, is the eighth of eight classes in the Machine Learning Foundations series. It builds upon the material from each of the other classes in the series -- on linear algebra, calculus, probability, statistics, and algorithms -- in order to provide a detailed introduction to training machine learning models.

Through the measured exposition of theory paired with interactive examples, you’ll develop a working understanding of all of the essential theory behind the ubiquitous gradient descent approach to optimization as well as how to apply it yourself — both at a granular, matrix operations level and a quick, abstract level — with TensorFlow and PyTorch. You’ll also learn about the latest optimizers, such as Adam and Nadam, that are widely-used for training deep neural networks.

Over the course of studying this topic, you'll:

  • Discover how the statistical and machine learning approaches to optimization differ, and why you would select one or the other for a given problem you’re solving.
  • Understand exactly how the extremely versatile (stochastic) gradient descent optimization algorithm works, including how to apply it
  • Get acquainted with the “fancy” optimizers that are available for advanced machine learning approaches (e.g., deep learning) and when you should consider using them.

DIFFICULTY LEVEL: BEGINNER

Instructor Bio:

Dr Jon Krohn

Chief Data Scientist, Author of Deep Learning Illustrated | untapt

Dr. Jon Krohn

Jon Krohn is Chief Data Scientist at the machine learning company untapt. He authored the 2019 book Deep Learning Illustrated, an instant #1 bestseller that was translated into six languages. Jon is renowned for his compelling lectures, which he offers in-person at Columbia University, New York University, and the NYC Data Science Academy. Jon holds a Ph.D. in neuroscience from Oxford and has been publishing on machine learning in leading academic journals since 2010; his papers have been cited over a thousand times.

Course Outline

1: Introduction to Data Structures and Algorithms

  • A Brief History of Data
  • A Brief History of Algorithms
  • “Big O” Notation for Time and Space Complexity


2: Lists and Dictionaries

  • List-Based Data Structures: Arrays, Linked Lists, Stacks, Queues, and Deques
  • Searching and Sorting: Binary, Bubble, Merge, and Quick
  • Set-Based Data Structures: Maps and Dictionaries
  • Hashing: Hash Tables, Load Factors, and Hash Maps


 3: Trees and Graphs

  • Trees: Decision Trees, Random Forests, and Gradient-Boosting (XGBoost)
  • Graphs: Terminology, Directed Acyclic Graphs (DAGs)
  • Resources for Further Study of Data Structures & Algorithms


4: The Machine Learning Approach to Optimization

  • The Statistical Approach to Regression: Ordinary Least Squares
  • When Statistical Approaches to Optimization Break Down
  • The Machine Learning Solution


5: Gradient Descent

  • Objective Functions
  • Cost / Loss / Error Functions
  • Minimizing Cost with Gradient Descent
  • Learning Rate
  • Critical Points, incl. Saddle Points
  • Gradient Descent from Scratch with PyTorch
  • The Global Minimum and Local Minima
  • Mini-Batches and Stochastic Gradient Descent (SGD)
  • Learning Rate Scheduling
  • Maximizing Reward with Gradient Ascent


6: Fancy Deep Learning Optimizers

  • A Layer of Artificial Neurons in PyTorch
  • Jacobian Matrices
  • Hessian Matrices and Second-Order Optimization
  • Momentum
  • Nesterov Momentum
  • AdaGrad
  • AdaDelta
  • RMSProp
  • Adam
  • Nadam
  • Training a Deep Neural Net
  • Resources for Further Study