Had it not come so perilously close to tragedy, Northgate Information Services might have looked upon the Buncefield oil depot as the kind of marketing that money cannot buy.
The software services provider and business process outsourcing partner would be hard pressed to demonstrate the reliability of its services any better than having withstood the near total destruction of its headquarters with little – relatively speaking – disturbance to delivery.
In December 2005, a small explosion at the Buncefield Fuel Depot near Hemel Hempstead triggered a huge subsequent detonation, to eventually become the largest fire in Europe since World War Two.
Northgate’s headquarters, situated directly next door to the depot, took the brunt of the blast.
The impact of the explosion, which mercifully occurred around 6 o’clock on Sunday morning, when few people were nearby, measured 2.3 on the Richter scale and ignited several fires in the Northgate building. More than two hundred production systems were destroyed, along with a newly refurbished call centre and the company’s email hub.
Northgate’s disaster recovery partner SunGard fired up its customers’ back-up servers following a conference call early on Sunday morning. By Christmas Day, all of its 212 internal systems were restored. Of the 40 terabytes of data that had to be recovered, only one tape’s worth was lost.
Achieving this was the ultimate test of Northgate’s disaster recovery, business continuity and crisis management plans. The first task for its Emergency Management Team was to prioritise the services which had to get back on line immediately, explains Mark Farrington, then Northgate’s MD of public sector and corporate services and now its recovery director.
“In our HR services department, which manages the payroll for 30% of UK workers, there is a rolling payout window, because different companies pay their staff at different times,” says Farrington by way of example. A business continuity plan is useless if it fails to take account of the time of month or year that the hypothetical disaster occurs, he adds.
Once systems had been prioritised, Northgate’s technical staff of over 100 began to work on getting systems back on line, putting in twelve-hour shifts back to back.
“Northgate, being a technology company, was lucky to have these resources available,” says Farrington. “But had we lost any of the 30 core support staff that knew systems best, we would have been stuck.”
Also fortunate was Northgate’s ownership of a recently deserted building in nearby Dunstable, which became an operational site. This helped maintain a sense of purpose and even community for employees who, under many other companies’ business continuity plans, would have been working at home. “It was important to ensure that employees didn’t feel alienated from the organisation,” says Farrington. “In a disaster, interaction is key.”
Critical to managing the recovery effort was a flexible communications system, he says. “It was important to establish a staff communication hotline on day one, so that people knew where they stood. Both our extranet and the SMS messaging service we have were key to getting information out there.”
New phones had to be issued to key employees, after their mobiles became clogged with concerned calls from friends and relatives.
Although Northgate’s recovery of services was rapid, Farrington warns against thinking that “business as usual” returned too early. “You have to maintain focus on restoring the business to full capacity. Ours is a business that took 30 years to build. That’s not going to come back online in 10 days.”
