Ai+ Training

As the world rapidly changing around us, business across industries are treading carefully with unprecedented challenges and, if lucky enough, new opportunities. Due to their statistical assumption of generalizable patterns from the past, machine learning models are facing more scepticism about their validity in the world we now live in. It is more crucial than ever for data scientists to keep close eye on our beloved models in production, understand the impact of business changes on them, and steer promptly from potential pitfalls.

In this session, I will share some experience of model monitoring and diagnosis from a leading UK fintech company. We will discuss how to detect the distributional change and analyze its impact on the model performance metrics. We will also look at how to decompose exogenous effect from vintage effect, which would help businesses understand the model validity in the long run. Finally, we will share some techniques to detect changes in the statistical relationships and discover new features.