Ai+ Training

At many companies ML models make high stakes decisions each day which makes it critical to monitor models to prevent poor decisions. However, there are lots of technical and organizational challenges in doing so effectively at scale. In this talk we’ll discuss a systematic framework to build and roll-out full-spectrum Model Monitoring for identifying and preventing problems with models. We’ll do a deep dive into Lyft’s model monitoring architecture (which includes real-time feature validation, performance drift detection, anomaly detection, and model score monitoring), how we leveraged open source, and the cultural change needed to get data scientists to effectively monitor their models. We’ll also discuss why we decided to build vs buy, our wins and learnings, and why monitoring in itself may not be sufficient for preventing model degradation.