the tolerance at most banks for complex/black box models and processes is gone. The business user expects a model that can be understood, justified, and where necessary, be tweaked based on business considerations, as well as an open and transparent process that can be controlled.
In this chapter, we will look at the various personas that should be involved in a scorecard development and implementation project. The level of involvement of staff members varies, and different staff members are required at various key stages of the process. By understanding the types of resources required for a successful scorecard development and implementation project, one will also start to appreciate the business and operational considerations that go into such projects.
Scorecard Development Roles
At a minimum, the following main participants are required.
Scorecard Developer
The scorecard developer is the person who performs the statistical analyses needed to develop scorecards. This person usually has:
● Some business knowledge of the products/tasks for which models are being developed. For example, if someone is responsible for building models for an auto loan product or a mobile phone account, they should be familiar with the car-selling business or the cell phone/telco business. Similarly, a person building scorecards for collections needs to understand the collections process. This is to make sure that they understand the data and can interpret it properly in the context of each subject. This would include knowing which types of variables are generally considered important for each product, how decisions and data collection at source impacts quality, and how the model will be used for decision making.
● An in-depth knowledge of the various databases in the company and the data sets being used. The single most important factor in determining the quality of the model is the quality of the data. When the users understand the quirks in the data, where and how the data was generated, deficiencies, biases, and interpretation of the data, they will be able to conduct intelligent analysis of that data. Otherwise, their analysis will be devoid of context. This task may also be covered by someone other than the scorecard developer – for example, a data scientist playing an advisory role.
● An in-depth understanding of statistical principles, in particular those related to predictive modeling. For example, knowledge of logistic regression, fit statistics, multicollinearity, decision trees, and so on.
● A good understanding of the legal and regulatory requirements of models and of the model development process. This includes documentation requirements, transparency, and any laws that control the usage of certain information. For example, in many countries the use of gender, marital status, race, ethnicity, nationality, and the like are prohibited. They would also need to know requirements expected by internal model validation teams so that minimum standards of model governance are met. Detailed knowledge of this subject is usually with model validation groups.
● Business experience in the implementation and usage of risk models. This is related to the business knowledge of the product. If analysts understand the end use of the model, it enables them to develop the one best suited for that task. The analyst will not develop a model that merely meets statistical acceptance tests.
This person ensures that data is collected according to specifications, that all data quirks are taken into account, and that the scorecard development process is statistically valid.
Data Scientist
The data scientist is the person who helps source and extract the required records and fields of information in order to populate the scorecard development database. This person usually has:
● An in-depth knowledge of the various databases in the company, and the data sets being used.
● Proficiency in the tools and systems to determine and document data lineage, to perform field-specific code mappings to common values and definitions from a variety of internal legacy transaction systems and external data reporters.
● Ability to merge/combine information from disparate sources and perform necessary preprocessing to deal with data issues, such as undefined codes, missing information, or extreme/suspect values.
● Familiarity with file formats and fields of information available from the different credit bureaus, rating agencies, and other third-party data providers.
A good example of the required knowledge for data sourcing and extraction is in mortgage lending, where there can be up to four co-applicants, and records for each must be found and joined into a single complete applicant record with individual and combined characteristics derived. These include characteristics such as combined loan-to-value ratio, combined income, payment to combined income, combined debt-to-income ratio, and payment shock to combined current housing payments. Even in a data warehouse, the co-applicant records may reside in a different table that the primary applicant record and matching logic must be used to associate related records. Typical scorecard developers do not possess this type of in-depth knowledge, especially in the larger, more complex financial institutions.
Product or Portfolio Risk Manager/Credit Scoring Manager
Risk managers are responsible for the management of the company’s portfolio and usage of scorecards. They are usually responsible for creating policies and strategies for approvals, credit limit setting, collections treatment, and pricing. In most companies, this person would be the business owner of the scorecard. This person usually has:
● Subject matter expertise in the development and implementation of risk strategies using scores.
● An in-depth understanding of corporate risk policies and procedures.
● An in-depth understanding of the risk profile of the company’s customers and applicants for products/services.
● A good understanding of the various implementation platforms for risk scoring and strategy implementation in the company.
● Knowledge of legal issues surrounding usage of particular characteristics/processes to adjudicate credit applications.
● Knowledge of credit application processing and customer management processes in the company.
● Knowledge of roll rate models; delinquency trends by product, region, and channel; and reports and the average time to charge-off.
When a modeler is asked to build a model (typically a process initiated by the business area), the first question they should ask the businessperson is “why?” That businessperson is typically the risk manager. The answer to that question determines everything else that is done from that point forward, including deciding the target, variable mix in the model, picking the best model, conditions, appropriate model fit measures, and, of course, the final cutoff for any decisions. This person ensures that business considerations are given sufficient thought in the design and implementation of scorecards. Early on in the process, the risk manager can tap their knowledge of the portfolio risk dynamics and performance experience to help with determining the definition of what constitutes “bad” performance for the population of interest. A good practice is to involve risk managers (or a representative) in each phase of the scorecard development process, and get their approval at the end of each one. Risk managers should be able to use some of their experience to point scorecard developers in a particular direction, or to give special consideration to certain data elements. For example, in cases where data is weak or biased, risk managers may use their experience to adjust weight of evidence (WOE) curves or to force certain variables (weak but logical) into the model. Experienced risk managers are also aware of historical changes in the market,
