Course Abstract

Training duration: 90 min (Hands-on)

Supervised Learning is a course series that walks through all steps of the classical supervised machine learning pipeline. We use python and packages like scikit-learn, pandas, numpy, and matplotlib. The course series focuses on topics like cross validation and splitting strategies, evaluation metrics, supervised machine learning algorithms (like linear and logistic regression, support vector machines, and tree-based methods like random forest, gradient boosting, and XGBoost), and interpretability. You can complete the courses in sequence or complete individual courses based on your interest. Part 3 of the course series focuses on the target variable and evaluation metrics that measure how well the machine learning model makes predictions. Classification problems have a larger variety of metrics and it might require quite some deliberation to choose the right metric for your problem. We will discuss what questions you should ask yourself while making these decisions and what are some recommended choices in certain scenarios. It is easier to choose an evaluation metric in regression. The choices you need to make are mostly subjective and usually do not have a strong impact on the overall predictive power of your model. We will also discuss how to calculate the baseline value of each evaluation metric. A baseline is a result of a very simple solution that uses only the known target variable for prediction. Baselines are useful anchors to compare the performance of your ML model against and we want our model to outperform the baseline.

DIFFICULTY LEVEL: BEGINNER | INTERMEDIATE

Learning Objectives

  • Describe the elements of the confusion matrix

  • Describe metrics derived from the confusion matrix such as accuracy, precision, recall, and the f_beta score

  • Summarize what the ROC and precision-recall curves and AUC are

  • Review the logloss metric and its properties

  • Outline metrics often used in regression (MSE, RMSE, MAE, R2 score)

  • Calculate the value of each metric given a set’s target variable and predictions from an ML model

  • Calculate the baseline of each metric given a set’s target variable

  • Choose an appropriate evaluation metric given your ML problem

Instructor Bio:

Andras Zsom, PhD

Lead Data Scientist and Adjunct Lecturer in Data Science | Brown University, Center for Computation and Visualization

Andras Zsom, PhD

Andras Zsom is a Lead Data Scientist in the Center for Computation and Visualization and an Adjunct Lecturer in Data Science at Brown University, Providence, RI, USA. He works with high-level academic administrators to tackle predictive modeling problems and to promote data-driven decision making, he collaborates with faculty members on data-intensive research projects, and he is the instructor of a mandatory course in the Data Science Master’s curriculum.

INTERESTED IN MORE HANDS-ON TRAINING SESSIONS?

VIEW PLANS

Course Outline

Module 1: Hard predictions in classification

  • Describe the difference between hard and soft predictions

  • Review the confusion matrix in binary and multiclass classification

  • Why cannot we use the confusion matrix directly to compare models?

  • Derive single number evaluation metrics based on the confusion matrix

  • Discuss the pros and cons of each metric and under what conditions they should be used

  • Calculate the baseline of each metric


Module 2: Working with predicted probabilities in classification

  • Predicted probabilities in sklearn

  • Review the logloss metric

  • The ROC and precision-recall curves and the AUC

  • Discuss the pros and cons of each metric and under what conditions they should be used

  • Calculate the baseline of each metric


Module 3: Regression metrics

  • MSE, RMSE and MAE

  • The R2 score and its properties

  • Discuss the pros and cons of each metric and under what conditions they should be used

  • Calculate/describe the baseline of each metric

Background knowledge

  • Python coding experience

  • Familiarity with pandas and numpy

  • Prior experience with scikit-learn and matplotlib are a plus but not required

Applicable Use-cases

  • The dataset can be expressed as a 2D feature matrix with the columns as features and the rows as data points

  • A continuous or categorical target variable exists

  • Some examples include but are not limited to fraud detection, predict if patients have a certain illness, predict the selling or rental price of properties, predict customer satisfaction.