Patch Management

By Chris Costanzo

Under assault from a proliferating number of computer viruses, banks are learning to provide software patches in a more timely and disciplined manner.

Patch management is, by definition, a patchwork affair. But as bankers face an increasing proliferation of threats from computer viruses, they are placing more emphasis on applying patches in a timely and disciplined manner to defend against these threats.

Managing patches sounds easy enough. When a piece of software becomes vulnerable to attack, the software maker sends out a patch for end-users to apply.

But the efficacy of this repair job depends on the end-user doing certain things right, like applying the patch in a timely fashion, testing it correctly, and watching to make sure the patch hasn't caused problems to other elements in the software system.

The pressures on end-users multiply with the proliferation of computer viruses. Banks now find themselves in the position of placing patch upon patch in an environment where their software systems often aren't well integrated to begin with. This patchwork approach increases the vulnerability of financial institutions to new threats and exposes them to unintended consequences from the patches they have applied.

Banks are so concerned about these vulnerabilities that they have rallied together under the auspices of industry associations such as BITS and the Financial Services Roundtable to prod their software providers into creating products that are not so open to attack. The vendors, meanwhile, are busily churning out new patch products, a market that Gartner Inc. predicted last year would experience a major growth spurt over the next few years, before plateauing in 2007 when patch management features become a standard part of larger security and systems configuration products.

All this attention to the problem, combined with a commensurate educational outreach, will likely help improve patch-management practices in the financial services industry. But information technology and security managers should not restrict their efforts to these technological fixes alone. They also need to layer in a "defense-in-depth" system to prevent viruses and hackers from attacking their systems in the first place.

"If you start finding a virus once it's in your organization, it's too late," says Rhonda MacLean, corporate security executive at Bank of America Corp.

Defense-in-depth should include such technological barriers as firewalls, intrusion-detection software and anti-virus software. Institutions also need to pool information about threats and vulnerabilities, a tactic aimed at stemming the spread of potential problems.

The essential dilemma of patch management is that so much time and effort goes into bringing software up to a level of safety that it probably should have had in the first place. It has been estimated that patching activities alone cost the banking industry $110 million a year. Hopefully, the industry's pressure on software vendors to build more secure products and accept greater responsibility for vulnerabilities will have some impact.

But in the meantime, banks have no choice but to step up to the challenges of patch management — immediately.

Speedy Response

At PNC Financial Services Group Inc. in Pittsburgh, some of the security budget has recently gone directly into patch management. "Patch management has a lot of complexity," says John Ericksen, PNC's chief technology risk officer. "Over the last year and a half, we've made some additional large investments to standardize it."

Because of the nature of computer virus attacks, the frequency with which PNC issues patches is not within its control, Ericksen says. The institution may perform three or four in a week, or none in several months, he says.

PNC has found that a crucial element of fixing vulnerable software is to do it quickly. The window of vulnerability, or time between when a threat is identified and when organizations are able to react, has shrunk immensely, Erickson says. Formerly, months could be allowed to pass before an institution needed to patch a known exposure in its system. Now, with more hackers active, attacks known as "zero-day exploits" are increasingly common. These attacks come out of the blue, seeking out unprotected systems, sometimes even before a patch has been developed to thwart them.

PNC has developed a process called "Computer Escalation Response and Forensic" that is designed to make the bank's response more timely and automatic. Under this process, PNC assesses the criticality of a patch and issues an immediate decision on what to do about it, with prescribed responses laid out in advance. "If you have to call a meeting, you are sunk," Ericksen says.

To speed response times, PNC has also standardized its approach to patching across platforms, which validates that patches have been applied and are operating effectively. PNC's patch management software is supplied by Marimba Inc. of Mountain View, Calif.

Industry experts say testing is the most crucial part of patching. It's not uncommon for banks to conduct two tests. The first is to assess whether a patch is correctly deployed and takes hold, the second to verify that the fix does not affect the functionality of other applications or devices.

One relatively common problem in patch management has to do with the fact that patches sometimes do not go into effect until after a machine has been rebooted. The patch management system will report that the machine has been fixed when it actually remains unprotected because it hadn't yet been rebooted.

These kinds of weaknesses in patch management software encouraged Washington-based BITS to begin an aggressive, public campaign to put pressure on vendors. The goal is not only to improve the patch management process, but also to encourage the development of software that is safer to begin with.

BITS began its campaign in February by hosting a cybersecurity summit in which it implored leaders in the financial services industry to be more communicative with vendors about the need for better security practices. It followed up in April by issuing with the FSR a joint policy statement reiterating the need for improved security. The statement does not mince words, calling upon software companies to "accept responsibility" for their role in supporting critical infrastructure, as well as be "more accountable" for the quality of their products.

"The statement reflects the broadly held frustration that the software industry is not doing enough to address risks," says John Carlson, a senior director at BITS. "It's not in tune with the fact that financial institutions are highly regulated and have a responsibility to safeguard customer information."

PNC's Erickson agrees that, "If vendors decreased the number of vulnerabilities at the get-go, we wouldn't have to patch as much." But he doesn't hold software vendors 100% accountable, noting that businesses are always "demanding new functionality, capability and speed to market," which relegates security to a secondary role.

Banking Clout

BITS and the FSR would like software companies to make security a fundamental component of software design and do a better job of informing financial institutions of new vulnerabilities and how to fix them. BITS is also encouraging vendors to make sure their products adhere to the organization's certification program, which bestows a "BITS Tested" mark on products that comply. So far, only two vendors have earned the mark, which BITS began aggressively promoting just over a year ago. The organization hopes that by talking up the program, more banks will require it of their vendors.

Finally, the associations would like vendors to improve the patch management process by issuing patch alerts as early as possible and continuing patch support for older software. They also want vendors to test patches before release so they do not cause a cascade of other problems once installed.

One issue is how much leverage the banks will have in getting their vendors to adopt these practices. BITS itself acknowledges that for anti-trust reasons, the financial services industry cannot band together and refuse to buy a particular vendor's products. Yet industry associations unquestionably have clout. "We're a critical infrastructure industry," Erickson says. "Because of our very nature, we can make things happen."

Banks also have important allies. BofA's MacLean, who chairs the Treasury Department's Financial Services Sector Coordinating Council, has been working with the heads of other critical infrastructure industries, such as telecommunications and energy, to make banking's concerns known to vendors. "There is enormous pressure coming from all the sectors," MacLean says. "This is critical to the assurance of homeland security, and it just makes good business sense for information technology providers."

One vendor that of necessity must receive a lot of attention from BITS is Microsoft Corp., the nation's dominant software company. Many of the recent computer viruses, such as Melissa and Blaster, were directed at Microsoft products.

Microsoft did not make an executive available for an interview with Banking Strategies. But a public relations spokesman stated, by e-mail, that the Redmond, Wash.-based company "has made a number of improvements to the security-update process, and we will continue working to improve." As part of its "trustworthy computing initiative" program, Microsoft has reconfigured more than 20 services in its Windows Server 2003 to reduce the risk of attacks. It also has created an engineering excellence program to establish reliability metrics, train developers to write more reliable code and increase individual accountability for product quality.

"Microsoft has really taken vulnerability management very much to heart," says Steve Katz, president and founder of Security Risk Solutions, LLC of Melville, N.Y., who previously held chief security and technology positions at J.P. Morgan Chase & Co., Citicorp Inc. and Merrill Lynch & Co. Unfortunately, Katz adds, the typical Microsoft operating system has millions of lines of code. So, "even with the best intentions, you're still going to have a significant number of vulnerabilities with that many lines of code."

Another problem is the large volume of already-installed Microsoft systems. "It's all about the legacy code," says Eric Hemmendinger, a research director at Aberdeen Group in Boston. "The problem exists until the legacy code retires, and some of this stuff has a very, very long life."

Beyond Patching

As computer viruses continue to proliferate, security executives are becoming more conscious of the limitations of patch management. Hackers are simply becoming quicker and more adept at exploiting vulnerabilities. "You really have to rely on a layered defense system," MacLean says.

The value of a layered defense, also known as "defense-in-depth," is its ability to prevent attacks in the first place. PNC, for example, relies on three to five controls intended to defend against 80% to 90% of threats, as opposed to one that supposedly deflects 100%, according to Ericksen. These controls include different types of technological barriers, such as firewalls, to block the entry of non-traditional traffic, intrusion detection software to identify suspicious traffic, and anti-virus software. Protection should be placed at different entry points to the system and should also include products from multiple vendors, Ericksen says.

MacLean also advocates that institutions keep a complete inventory of systems used on their networks. "Having that inventory is just absolutely critical to patch management," she says, noting that computer worms can sometimes make it look like new machines have been added to the network. A good management program for computer assets is necessary to help executives determine whether a new device showing up on the network is a friend or foe, she says.

Another key element in an effective defense is a rapid response team. Bank of America, for example, has a group of people dedicated to receiving information about vulnerabilities and determining how quickly they should get patched, she says. "You need an orchestrated way to ensure the right hands are on deck." This policy should apply as well to providers of outsourced systems, MacLean adds. "You need a good way of communicating and understanding what your suppliers are doing with vulnerability management."

Ultimately, patch management must be just one piece of an overall, effective security strategy. Given the threats lurking on networks today, and their potential for disrupting the financial system, a strategic focus is necessary. "We've been fortunate no crisis has happened yet," says Catherine A. Allen, the chief executive officer of BITS. "But it's coming and we should be prepared."

Ms. Costanzo is a freelance writer based in Brooklyn, N.Y.

Copyright © 2004 by Banking Strategies, published by BAI.

back to top