Dynamic Model Validation: The New Wave

Key Findings

This paper addresses a case study of a consumer credit adjudication model, showcasing its performance under both traditional and ML/AI methods Followed by a discussion on examples of dynamic processes that can improve the process even more --- and how financial institutions might implement these models.

Abstract

Traditional model validation has become a staple of the finance industry, a well-understood and rigorously developed line of defense intended to help institutions avoid unnecessary financial losses caused by poorly performing models. Using tools such as logistic regression, time series modelling, and model benchmarking, the field has progressed steadily using traditional statistical validation techniques and tools. Many institutions by now have already improved their traditional model validation tools by incorporating machine learning (ML) and artificial intelligence (AI) tools to identify additional features used to better predict outcomes or disasters. The ML/AI algorithms use much larger datasets and uncover hidden relationships that often lead to big improvements in predictive power. However, the datasets used to develop ML/AI tools are also often static, even if they are updated at discrete intervals. Of course, one can also break any dataset into in-time and out-of-time samples, but even that can be considered a kind of static dataset. In real-life situations, one may need non-static models that go beyond static ML/AI. We are seeing a new wave in model validation in consumer credit that supplements the traditional and ML/AI algorithms by incorporating truly dynamic elements, intended to catch models as they might be failing, using real-time updated data and AI algorithms. These models can be sensitized to unexpected changes in data or predictions, triggering review or corrective action when necessary.