Validating the Model

As the director of risk analysis for the Office of the Comptroller of the Currency, Jeffrey Brown spends most of his time reviewing how banks use credit scoring and other risk-management tools.

The regulator says he's generally satisfied with bank practices in this area, but warns that constant vigilance is required. The key to ensuring top-notch performance, he says, is to maintain the proper validation framework. When he visits banks, he often cites the following five features of a good model validation policy:

Independent Review. "Whether a bank builds its own model or buys it from a vendor, another competent and independent person should look at the model to ensure it makes sense." This can be an auditor with technical skills, a consultant, or somebody from another part of the bank who can think critically about the scorecard but has no direct interest in it.

Ongoing Validation. "Even if you're using really basic decisioning models, you have to monitor their performance on a regular, periodic basis." Brown recommends a routine schedule, with required monthly or quarterly reports, to gauge how accurately the models predict loan quality.

Define Responsibilities. "Everyone talks about validation, but if it's not somebody's job, it never gets done." The best policies are those placed in writing, and the responsibility for validation should be assigned to one person. This procedure typically includes sign-off sheets, which often must go to a department supervisor.

Documentation. "You would be surprised at how often models are built without good supporting documentation." Validation can't be done properly without a complete paper trail that explains everything the model-maker or department head had in mind when the scorecard was constructed. Weak documentation can be particularly damaging to the institution if the person in charge of scoring leaves and the replacement has to start from scratch. Good documentation also makes for easier regulatory exams.

Auditor Oversight. "When we tell banks that their policy should include a role for auditors, some people scoff: "Our auditors are accountants, not statisticians.' But auditors should be responsible for ensuring that the validation policies and procedures are being followed, just as they make sure that other parts of the bank are doing the things they're supposed to do."

— John R. Engen

Close this window to return to the article.