By: Bart Baesens, Seppe vanden Broucke
This QA first appeared in Data Science Briefings, the DataMiningApps newsletter as a “Free Tweet Consulting Experience” — where we answer a data science or analytics question of 140 characters maximum. Also want to submit your question? Just Tweet us @DataMiningApps. Want to remain anonymous? Then send us a direct message and we’ll keep all your details private. Subscribe now for free if you want to be the first to receive our articles and stay up to data on data science news, or follow us @DataMiningApps.
You asked: What are the limitations of credit scoring?
Our answer:
Although credit scoring systems are being implemented and used by most banks nowadays, they do face a number of limitations. A first limitation concerns the data that is used to estimate credit scoring models. Since data are the major, and in most cases the only, ingredient to build these models, its quality and predictive ability is key to their success. The quality of the data refers, e.g., to the number of missing values and outliers, and the recency and representativity of the data. Data quality issues can be difficult to detect without specific domain knowledge, but have an important impact on the scorecard development and resulting risk measures. The availability of high-quality data is a very important prerequisite for building good credit scoring models. However, the data need not only be of high quality, but it should be predictive as well, in the sense that the captured characteristics are related to the customer defaulting or not. Before constructing a scorecard, we need to thoroughly reflect why a customer defaults and which characteristics could potentially be related to this. Customers may default because of unknown reasons or information not available to the financial institution, thereby posing another limitation to the performance of credit scoring models. The statistical techniques used in developing credit scoring models typically assume a data set of sufficient size containing enough defaults. This may not always be the case for specific types of portfolios where only limited data is available, or only a low number of defaults is observed. For these types of portfolios, one may have to rely on alternative risk assessment methods using, e.g., expert judgment.
Financial institutions should also be aware that scorecards have only a limited lifetime. The populations on which they were estimated will typically vary throughout time because of changing economic conditions or new strategic actions (e.g., new customers segments targeted, new credit products introduced) undertaken by the bank. This is often referred to as population drift and will necessitate the financial institution to rebuild its scorecards if the default risk in the new population is totally different from the one present in the population that was used to build the old scorecard.
Many credit bureaus nowadays start disclosing how their bureau scores (e.g., FICO scores) are computed in order to encourage customers to improve their financial profile, and hence increase their success in getting credit. Since this gives customers the tools to polish up their scores and make them look “good” in future credit applications, this may trigger new types of default risk (and fraud), hereby invalidating the original scorecard and necessitating more frequent rebuilds.
Introducing credit scoring into an organization requires serious investments in information and communication technology (ICT, hardware and software), personnel training and support facilities. The total cost needs to be carefully considered beforehand and compared with future benefits, which may be hard to quantify.
Finally, a last criticism concerns the fact that most credit scoring systems only model default risk, i.e., the risk that a customer runs into payment arrears on one of his/her financial obligations. Default risk is, however, only one type of credit risk. Besides default risk, credit risk also entails recovery risk and exposure risk.
