It looks like any other downtown Honolulu office building from the outside, but inside the gray walls are dozens of computers with support hardware that make up the nerve center of Bank of Hawaii's emergency operation facility. These Hawaii businesses
prepare before disaster strikesBy Dave Segal
dsegal@starbulletin.comThe center, in an undisclosed site near the bank's headquarters, has three disaster recovery rooms with rows of computers that can immediately access bank data, a war room in which top executives can make strategic decisions and a diesel generator that can be activated in case of a power outage.
There also is a wall-size map of the world, wall clocks set to different time zones, direct trunk lines to telephone provider Verizon Hawaii, Internet capability, telephones, printers, fax machines, copy machines, televisions, helmets that can be worn to enter damaged buildings, manual typewriters that can be sent to branches that have lost power, and tellular phones that are in essence oversized cellular phones capable of sending and receiving faxes.
And, in case bank officials are forced to hunker down for a few days, the center has ready-to-eat meals, bottled water, 30 military cots, two bathrooms, a shower and a kitchenette.
"The primary purpose of this particular facility is to maintain the day-to-day operation of the disaster preparedness program for the company and to provide the emergency operation center location for executive management to continue the operations of the company," said Ray Trombley, a Bank of Hawaii vice president and manager of the corporate disaster preparedness department.
Trombley said the center is backed up by a similar one in Kapolei, which also has three recovery rooms. In addition, the bank has a facility in Iwilei that is designed to recover the bank's information technology applications, data and customer records.
DENNIS ODA / DODA@STARBULLETIN.COM
Ray Trombley, Bank of Hawaii's vice president and manager of corporate disaster preparedness, is in charge of the company's disaster recovery rooms, which are designed to keep the business running in case of a catastrophic incident, such as a hurricane or tsunami.
The downtown center, built in 1997, was last put to the test in October 2001 when an anthrax scare forced the evacuation of several downtown offices. The center also was used on Sept. 11 when the bank monitored what was happening at its since-divested branch office four blocks from the World Trade Center. The facility, staffed by five employees, is routinely used for disaster recovery exercises when there's no real emergency.
"Our objective, essentially, is after a disaster occurs to be able to open up our doors to our clients the following day," said Trombley.
"The big issue we want to deal with is to ensure that our clients and our customers can have access to their accounts after a disaster," he said. "
Businesses received a wake-up call on Sept. 11. Many companies, including those in Hawaii, immediately began examining their storage capabilities and their ability to retrieve information. In many cases nationwide, though, companies stopped short of spending the money it would take to upgrade their systems.
"There are pockets of companies that are spending a lot of money to upgrade their recovery capabilities right now, and also companies even going as far as to reorganize where they put their people and their system," Gartner Inc. analyst Mark Nicolett said.
DENNIS ODA / DODA@STARBULLETIN.COM
Trombley, Emergency Procedures Coordinator Ron Mobley, Recovery Plan Coordinator Michael Shibata, Recovery Plan Coordinator Stella Allen and Database Analyst Lori Naito-Yrizarry look over disaster recovery manuals.
"But that's the exception, not the rule," added Nicolett, whose Stamford, Conn.-based company provides research and analysis on information technology. "The broad average is that companies are looking at their plans and trying to fill gaps, but they don't have a lot of money to spend because of the economy."
That also seems to be the trend with many Hawaii companies as well, according to John Agsalud, president of Honolulu-based ISDI Technologies Inc., a provider of information technology consulting services.
"My impression is that people who had disaster recovery plans prior to Sept. 11 went back and made sure they were updated and solid and tested," Agsalud said. "But for the people who did not have any kind of disaster recovery plan, very few actually put one in place.
"I think people looked at that stuff and made some preliminary plans, and I think a lot of them came to the conclusion that anything really robust would be kind of expensive. So they didn't jump right into it. Here we are a year and two months later, and I think people have kind of let it fall off their radar screen."
Larger Hawaii companies or agencies, however, appear to have intensified their efforts. The Honolulu Board of Water Supply is in the process of implementing redundant off-site systems for vital computer functions. It's also planning to build a backup data center at one of its facilities to ensure continuous operation in case of a disaster at its primary computer center. The agency currently backs up software and data on tape and stores it off-site.
Following Sept. 11, federal officials cited an attack on the nation's water supply as a possible terrorist target.
DENNIS ODA / DODA@STARBULLETIN.COM
Database Analyst Lori Naito-Yrizarry, left, Vice President Ray Trombley, Recovery Plan Coordinator Stella Allen and Recovery Plan Coordinator Michael Shibata displayed a cot that can be used in Bank of Hawaii's disaster recovery center should disaster strike.
But Brian McKee, acting chief information officer at the Board of Water Supply, said Sept. 11 wasn't the impetus for the decision to upgrade the systems. A recent disaster recovery and vulnerability study showed the agency needed a "mirror site" at another facility.
"We are much more confident in our capabilities now," McKee said. "Our organization has undergone a tremendous amount of change and progress in the last couple of years. Much of this is not a direct result of 9/11, but rather the byproduct of our efforts to improve our systems, guarantee customer service to the organization and streamline our process."
Hawaiian Airlines, which like other carriers was grounded for several days in the aftermath of Sept. 11, said it has done reviews and drills with its current systems and feels comfortable that no change is necessary. Its mainframe and server information is backed up on a daily basis and its reservations are stored in Southlake, Texas-based Sabre Holdings Corp.'s global reservation system, to which Hawaiian Airlines subscribes.
"There's a full spectrum of approaches that deliver speeds of recovery and different points of recovery, meaning very little data lost or a day or two or three days worth of data lost," said Nicolett, the Gartner analyst. "So there's a very wide range of approaches you can take. You can design for a one-week recovery time, or two days, or five minutes. Not surprisingly, the approaches that deliver the very fast recovery times and very current recovery points are much more expensive."
Verizon Hawaii, the state's biggest telephone company, said its current data storage and recovery process were in place prior to Sept. 11. All of the parent company's local data is backed up on mainframe computers within major data centers throughout the United States.
"These centers are highly secure with robust and redundant networks, which allow us to re-route traffic with minimal, if any, impact to customers," Verizon Hawaii spokeswoman Ann Nishida said. "All data is updated regularly and maintained constantly at these various locations."
Hawaiian Electric Co., which supplies the state with 95 percent of its electricity, said it has not implemented any special measures specifically due to Sept. 11, but is continuously reviewing its procedures.
"Customer data could be recovered very quickly, but even when it was not readily accessible, we would be able to maintain customer operations," said Hawaiian Electric spokes-woman Lynne Unemori. "Some functions such as customer billing inquiries might take a little longer,."
Matson Navigation Co. said Sept. 11 has prompted new security measures but it did not need to make dramatic changes in its recovery plans.
"In preparation for Y2K, the company thoroughly explored a number of 'worst case' scenarios -- which covered a variety of systems interruptions -- and developed plans on how processing systems and data could be reasonably protected," Matson spokesman Jeff Hull said.
Hull said Matson's data is distributed over several computers in different locations and there is sufficient redundancy in the systems' infrastructure to alleviate any concern regarding the loss of data.
