Blind Faith

By John R. Engen

Credit scoring has revolutionized lending, but critics worry institutions are pushing this technology beyond its limits.

Credit scoring has transformed lending in ways that were unimaginable 40 years ago. Back then, most loans were originated on the basis of the "three Cs" — character, collateral and capacity — all with a large dose of subjective human underwriting.

Today, statistical models often have the final say-so. That goes for most credit card, auto, home mortgage, home equity and small business loans. Lenders simply plug in the variables and let the computer do the work.

The benefits of this revolution are undeniable. Credit scoring helps lenders make decisions more quickly and cheaply, compared with old-style judgmental underwriting, and also more accurately and consistently. It gives a profound boost to the loan-backed securities market, moreover, equipping investors with objective measurements for analyzing loan pools.

But is this technology getting out of hand? That's the crux of the debate that now swirls around credit scoring. There's no denying the massive shift away from old-fashioned due diligence and expert personal judgment to what is, in essence, a system based on playing the odds. As with any quantitative model, credit scoring is only useful if its limitations are properly understood.

Some experts fear that bankers have become overly reliant on this "black box" approach to loan decisions and are pushing the system beyond its natural capabilities. There's a tendency to move "up market" to bigger loans, for example. But in areas such as large corporate credits, neither the seasoning of the data upon which scores are based nor the well of experience may justify such reliance. Another worry is that today's scoring models, grounded in a decade of strong economic growth, may be ill-prepared to predict borrower performance in recessionary conditions.

The advent of the Internet intensifies the gravity of this issue. The speed mandates of the Web, combined with the desire of lenders everywhere to grow portfolios and control costs, is accelerating banking's usage of scoring. "That's where the true danger lies," says David Zhai, a vice president and senior analyst in the structured finance group at Moody's Investors Service, New York. "There's an increasing tendency for lenders to sacrifice underwriting quality for speed, volume and cost savings."

While the effects on loan quality remain murky, the evidence of acceleration is crystal clear. In May, for example, Fair, Isaac and Co. Inc. began offering Internet delivery of credit scores based on models and data stored on the company's own server. The new setup allows the San Rafael, Calif.-based company — provider of an estimated 70% of all scores — to update its technology more quickly, and it eases pressures on banks to update software and expand individual server capacity.

Meanwhile, the three major credit bureaus — Equifax Inc., Experian Information Solutions, Inc. and Trans Union LLC — are all touting better, faster scoring models of their own. So too are many smaller vendors, such as Baker Hill Corp., Carmel, Indiana, which markets a scorecard for small-business loans aimed at community banks.

Amid this rush to automation, the challenge for bank officers is to strike a proper balance — to capture the immense benefits of scoring without overly exposing their institutions to the potential downside. Properly used, scoring can facilitate loan growth and minimize delinquencies. But a little knowledge can be a dangerous thing. The vast majority of scorecards used by bankers are generic, and built on information known only to the vendor.

Top executives are well advised to gain at least a basic grasp of how their institutions use scores. And they should avoid blind reliance on vendor tools, even if it means bringing in a third party to review that usage. "Senior bankers tend to say, "We buy these models from reputable vendors, so there's no problem because they pass the market test,'" says Jeffrey Brown, director of risk analysis for the Office of the Comptroller of the Currency. "But scoring models are power tools. They can get you into trouble quickly. Someone you trust should be watching how they are used."

To be truly effective, experts say, scoring models must work in concert with a full menu of internal controls as part of a cohesive business strategy. The models must be constantly validated and documented, and managed by people who understand their strengths and limitations.

The Scoring Lift

Credit scoring dates back to 1956, when William Fair and Earl Isaac founded their namesake firm to apply computer-aided statistical methodology to consumer credit. Initially met with skepticism, scoring soon changed the way lenders assessed risk, and it fueled the rise of credit cards and other forms of unsecured lending. Lenders used an estimated 10 billion scores last year, about half in the hyper-competitive credit card area.

Scores are now so fundamental to the consumer lending process that one is pulled as a matter of course with each application. Fair, Isaac estimates about one billion scores were applied directly to originations last year. The information primarily is used to assess the credit worthiness of potential borrowers. Many banks also consider it crucial in determining a rate offer that matches the risk — so-called risk-based pricing.

"I can't imagine what the industry would do without scoring," says Dan Ray, Columbus, Ohio-based chief credit officer for consumer lending at Bank One Corp. His secured lending unit originates about $24 billion in loans annually, pulling three million scores in the process. Having to do without that information, he says, "would be a dramatic shock."

In recent years, scoring has also become integral to small-business lending. The theory is that the small-business owner's personal score is a good predictor of how his or her firm will repay its loan. The small business services unit of Cleveland-based KeyCorp has been using Fair, Isaac scorecards to originate small-business loans for the last two years. The unit "auto-decisions" about 4,000 loans per month, or about 65% of its under-$100,000 credits, with no human intervention.

Many of those loans are approved in fewer than 10 minutes — an enticing carrot that the bank markets heavily — at only $19 a crack. That leaves KeyCorp's underwriters free to devote their costlier human scrutiny to more complex credits. Perhaps best of all, the models appear to be working to perfection in helping to predict delinquency rates on a growing portfolio. Vice chairman Michael Butler, who runs KeyCorp's small business unit, says the company soon will lift the ceiling on automated loans to $150,000. "Scoring has given our business a tremendous lift," he says.

When a lender gets a credit report from one or more of the big three reporting agencies, it typically uses the information to generate a Fair, Isaac score. The raw data from those bureau reports — including payment history, credit outstanding and the number of recent credit inquiries — is run through a Fair, Isaac algorithm to generate a score ranging between 300 and 800. The higher that score, the more credit-worthy a potential borrower is thought to be. About 85% of scores fall between 600 and 800.

Numerous other vendor-designed scoring algorithms are in use, and most large banks make liberal use of proprietary scorecards. It is now considered industry gospel that the probability of repayment can be reliably predicted by comparing a loan applicant's credit profile with those of millions of other people. Indeed, scorecard vendors claim that an almost blind faith in their tools is justified.

At a spring conference in San Francisco, for instance, Fair, Isaac representatives touted their generic small- business scoring model. They said this model, based on data from 25 lending institutions around the country, is able to give an instant yes or no on an astounding 80% of all applicants, leaving just 20% in the "gray area" that requires underwriter attention.

Executive vice president Robert Heller says a recent study by his firm found that default rates on credit cards issued purely on the basis of scoring were 30% lower than those where judgmental criteria were used to override scoring decisions. "The human mind can only hold so many variables, while a computer can hold a very large number of variables," Heller says.

Science and Art

Even as they enjoy the technology's benefits, however, bankers need to maintain their sense of perspective. Credit scoring is good at predicting the probability of default, but not the magnitude of potential losses. Furthermore, a scoring model that's good at gauging risks in an overall market segment might fail miserably with a specific bank's customer base.

Complexity explains some of this imprecision. A good credit score contains about 20 determining characteristics, built on analyses of "thousands of variables and permutations of variables," says Fair, Isaac's Heller. While lenders know the general ingredients in generic scores, they aren't privy to the details. Banks using scoring with an incomplete understanding of how the tools are made and should be used can unintentionally expose themselves to risks.

If scoring itself is a science, the way it is employed is more of an art. Each institution weights scores differently in its decision process, picks when and where in the lending process to inject them, and sets cutoffs that match its own risk appetite. The judgmental criteria can vary widely from bank to bank. "With scoring," Heller says, "the difference between a good bank and a bad one is in the management."

Bank One, for example, pulls custom and FICO scores for every consumer credit applicant, using this data to "frame" how the underwriter looks at the deal. But it also weighs myriad other factors, including the types of collateral offered, debt-to-income ratios and the lessons the bank itself has learned about borrowers in certain income or job history categories. "The score paints 60% of the picture," says Ray, who notes that credit scoring can't tell an underwriter what caused a bankruptcy or interpret what the level of credit utilization reflected in a score really says about a borrower.

Ultimately, lenders need a complete set of strong policies and procedures to fully leverage the technology's power. Tiers of borrowers with similar characteristics must be established to effectively use scoring as a pricing tool, for instance. And discipline must be maintained in implementing a pricing strategy. When rates don't accurately reflect risk, the institution stands to lose money.

Once the scoring plunge is taken, bankers agree, it's usually a mistake to try to override automated decisions. Like playing blackjack, scoring is about odds and temptations. "Once you approve a credit that fails to meet your scoring threshold, then you've broken your odds," Butler says. "You need the discipline to reject a score that's below the cutoff, even if it feels right."

Can bankers maintain the necessary discipline? There may be cause for concern. Zhai, the Moody's analyst, believes some lenders don't understand proper usage of the scores and aren't taking the basic steps required to minimize the risk of default. He says some lenders are inappropriately reducing the level of documentation required for loans and skipping basics such as collateral appraisals.

Discarding the tried-and-true fundamentals of the lending process in favor of scoring damages the integrity of underwriting systems, Zhai argues, and could open the door to fraud. "It's the user's obligation to understand scoring's proper place and integrate it with a comprehensive risk-management program."

Despite their appreciation of scoring's power, most bankers interviewed for this article concede that misuse is a problem in the industry. Bank One's Ray says the danger of fraud looms larger as banks rely more on scoring for large consumer credits. "If you're depending totally on a model and never even looking at the borrower, the risk of identity fraud, or of false statements about income, increases exponentially," he says.

Michael James is a group executive for small business and consumer lending at Wells Fargo & Co. He says his bank employs a staff of about 40 statisticians who are constantly retooling the inner workings of scores and how they're applied to the lending process. In contrast, many smaller banks "may either not understand or are not being as disciplined as they need to be" when using scoring, James says. "This is pretty sophisticated stuff, and I wonder if some companies have the expertise to manage it all."

Verify, Verify, Verify

Even when used properly, the effectiveness of scores can be constrained by various factors such as changing economic conditions and data quality. "Generic scores based on a broad population often aren't representative of a particular lender's strategy or customer base," says Yasmine Anavi, director of risk management for North America at Citigroup's credit card unit.

In the course of reviewing small bank-card portfolios that Citi considered acquiring during the last few years, Anavi says she witnessed first-hand "how over-reliance on generic models can make a portfolio stray by causing it to grow so quickly that it outstrips a bank's risk-management capabilities."

Some bankers fret that overconfidence may be leading some peers to move "up-market" with scores — using them, for instance, to auto-decision small-business loans as large as $250,000, or to gauge the risks of even larger corporate credits. Many major banks use software produced by San Francisco-based KMV Corp. and other firms to assess default probabilities for corporate borrowers.

While no one has apparently experienced serious difficulties doing this, scoring, by its very nature, works better with consumer businesses such as credit cards, where there is a wide and deep well of experience to draw upon. "The dependability of these models has a strong negative correlation to the size of the loan being considered," says the OCC's Brown. "The bigger the credit, the more traditional analysis is required."

For now, Brown says he's generally satisfied that no institution is "betting the bank" by relying too heavily on scoring. But since scoring is becoming ever more prevalent in the industry, bankers still need to guard against its potential perils.

Top executives should take the time to understand the basics — how the models are being used and how they interface with lending policies and procedures. And managers of specific lending units should be held accountable. Division heads should be able to explain their loss rates and pricing strategies, and show how scoring relates to the institution's overall business strategy. Line managers also need to monitor scoring's effect on portfolio performance.

This last recommendation falls under the broad banner of validation. Statisticians build scoring models, but someone else must examine the model-builder's work and check its accuracy in daily usage. Smart banks make this an ongoing process. They use an independent reviewer to monitor loan performance and ensure that the score is accurately differentiating different degrees of risk. This person can be either an impartial third-party or someone from a different part of the bank.

Ideally, such validation begins well before scoring tools are implemented. Prior to launching a formal program for auto-decisioning consumer loans in 1996, for example, KeyCorp spent several years as what Butler calls a "scoring shop in disguise." A traditional system based on standard judgmental criteria kept the underwriting wheels turning. Meanwhile, a shadow system based on credit scores was put in place and refined to the point that bank officers felt comfortable in letting it take over. Though time-consuming, this thorough process helped KeyCorp officials place the scores in the right context for daily usage.

Effective validation is not accidental. Rather, it's the result of carefully constructed policies and procedures meant to ensure that scoring models are being used correctly. It also includes ongoing monitoring efforts to assure that the scores are generating the intended kinds of credits.

In Wells Fargo's small-business unit, that means breaking down the portfolio into about 100 large cells and then tracking the performance of each cell in myriad ways. "You have to constantly evaluate your applicants, know whom you're targeting and who's applying, and assess how well your models are doing at generating the right business and predicting its performance," James says.

Banks should put their validation policies in writing, and hold one person accountable for ensuring they are followed. They should also require documentation for every step of a scoring model's construction. Should a designer leave, people not privy to the details would then still possess a roadmap to follow. Documentation also makes regulatory exams easier. "When I show up," says the OCC's Brown, "I like the bank's officers to hand over a big, thick binder that walks me through everything they did to build the model."

Boiled down to its essence, the lesson is simple: if you use credit scoring, first understand how it works. Then, supplement its usage with other, more traditional underwriting tools. And verify, verify, verify.

Mr. Engen is a freelance writer based in Minneapolis.

Copyright © 2003 by Banking Strategies, published by BAI.

back to top