Before Hurricane Katrina struck the south coast of the US in August 2005, building materials wholesaler Interior Exterior Building Supply thought it was prepared for the worst that any weather system could throw at it.
True, the hub of the corporate network that served its seven US branches was situated in the company’s headquarters in New Orleans – notorious for its occasional ‘high water’ – but the company had installed its servers and other IT equipment on the second floor of its offices to ensure it was well out of reach of any flood water. Moreover, all company data was backed up every night, and the tapes transported and stored off site.
What the company had not prepared for, however, was a natural disaster so catastrophic that it would cut off all electricity, water and transport access to the entire area for days. But that was exactly what it had to deal with when the levees that protect New Orleans from the ocean were breached by the force of the hurricane and three-quarters of the city found itself impassable.
“We thought we were ready,” recalls Shannon Hill, the company’s controller, “but looking back, we were sitting ducks.”
The day after Katrina hit New Orleans, the host site of the company’s network was gone; Interior Exterior was, Hill says, “dead in the water”. At first, it was impossible even to access the building, and there was no way to find out whether the flooding had reached the second-floor server room.
When the office was eventually accessible, Hill discovered the extent of the damage. Inventory, customer files, eight months’ worth of paper receipts, HR and payroll files were all lost permanently.
What data would be recoverable from the water-damaged systems was at that point unknown.
“There was such chaos, it was impossible to think what we needed to do first.”
Shannon Hill, Interior Exterior Building Supply
In the first week after Katrina, Hill decided that to get core business applications back up and running, she had to set up a Unix box, with data and terminal servers. Hill ordered new hardware and, with help from enterprise resource planning (ERP) supplier Infor and IT services partner Greywolf, worked 14-hour days to get the company in a state where other branches could open – albeit using paper records.
The company moved its headquarters to a branch site in Birmingham, Alabama, transporting all salvageable equipment and documentation through what was still a flood zone.
As it was, the Birmingham site lacked the space, power, ventilation and bandwidth to host the network servers, forcing the company to install a completely separate power circuit for the building and a new air conditioning system.
Seventeen days after Katrina struck New Orleans, the ERP application was running again. Employees set about inputting data from the paper records that had survived the flooding and keying in the orders that had been taken since. Meanwhile, Interior Exterior’s account manager at Infor was searching for second-hand tape drives that would enable the company to restore its data from back-ups.
One by one, the surrounding branches were brought back online, but with every step, performance issues dogged progress. Network bandwidth was at a premium, and reconfiguring the network while still allowing business to continue proved difficult.
“It was a trying time for everyone,” says Hill. “The whole process took more inner strength than any of us really had.” Miraculously, throughout the period the company never missed a payroll.
In the time since Katrina, says Hill, Interior Exterior has completely rethought its disaster recovery plans. The Birmingham site is still the principle production site, but is replicated twice in other branches. All data is mirrored at every branch each night using Progress Software’s Fathom product, and the back-up disks are rotated every two weeks.
But even now, says Hill, the company knows that it should never fall into the trap of assuming that it has done enough to prepare for disaster. The unimaginable, its staff and management now well understand, can happen.
Further reading
Staying afloat The widespread flooding that hit the UK this summer proved a severe test for many businesses’ disaster recovery measures
Business continuity It’s expensive, unglamorous and easy to ignore, but business continuity planning is still an essential consideration for all companies.Find more stories in the Security & Continuity Briefing Room
