Flooding is now one of the most common natural disasters of our times. And that is raising a serious challenge for IT.
The probability of a business being caught in a flash flood or swamped by fluvial plain flooding or a tidal surge has grown sharply, according to the Environment Agency, with the onset of climate change and land shortage encouraging construction in what might be deemed flood zones. Meanwhile, research by the National Audit Office finds that 63% of England’s overall flood defence systems fail to meet necessary defence target levels.
That is showing up in business continuity numbers. Service providers such as BT estimate that flooding now accounts for nearly 10% of major business disruptions.
Yet when heavy rain began to fall across large swathes of the UK in June and July 2007, few organisations envisaged the widespread devastation that would immediately follow. Fewer still had prepared their business and IT infrastructure in readiness for such a disaster. Even Gloucestershire County Council (GCC) – situated in a region where flood levels rose to 10 feet – never anticipated what turned out to be the greatest disruption to local services since the end of the Second World War.
"Flooding creates an ever-moving scenario that’s very difficult to plan for."
Jonathan Smith, SunGard Availability Services
In fact, because flood waters had not reached the location of GCC’s chief site in Shire Hall since 1946, flood mitigation had not even been written into the council’s business continuity (BC) plan. “The plan was written for a fire in the building, or a pandemic that would prevent people from getting into the main office in Gloucester,” explains Glynis Morris, manager of GCC’s 96-seat call centre, which serves as the council’s initial point of public contact. “I don’t think anybody ever envisaged that a flood would effectively take out the whole county: it wasn’t written for that.”
Faced with rising water levels that threatened to submerge the Shire Hall site server room – unwisely, although not uniquely, housed in the basement – the council switched off the site’s entire IT infrastructure. Due to the widespread nature of the flooding, however, Morris was unable to relocate her team to the two intended back-up sites, both of which were situated within the county, at Gloucester City Council’s offices and at an office based in nearby Stroud. Forced to move to out-of-county sites, Morris and her contact centre staff had to attempt to serve callers without any of their normal IT infrastructure and its applications, principally vital information databases and a CRM system. “We had no information systems at all,” she recalls.
It was only when Morris remembered that one of GCC’s key IT services was remotely hosted by software-as-a-service provider RightNow Technologies that she realised all was not lost. The so-called Knowledge Base, used by the GCC contact centre, is hosted at four data centres globally. The system allowed call centre agents to access information about local services over the web, while speaking with customers. It proved to be the single most valuable source of information during the incident.
But as Morris freely admits, cost and convenience, not business continuity, proved to be the council’s primary consideration when originally procuring the system. It was only a matter of good fortune, rather than good planning, that not all the council’s applications were hosted in the county. “Nobody ever thought we’d have to go out of county,” says Morris. Now, she adds, the council’s new BC plan allows for relocation to any back-up site in the country. “It’s critical to have some kind of resource up to as far as 60 miles away,” she warns.
Complex contingency
But Gloucestershire County Council was not the only organisation forced to think on its feet following the summer’s unprecedented rainfall. In the months since, some 14,500 flood-related insurance claims have been filed, the average value of which comes in at around £90,000, according to the Business Continuity Institute (BCI). In total, the floods inflicted more than £1.3 billion worth of damage on the UK business community, and this figure could well grow.
Much of this damage resulted from the direct impact of flood waters on the integrity of business premises and physical assets. But as Morris’s account testifies, the compound effect of large-scale flooding can prove just as, if not more, disruptive and costly to commercial organisations in the long run. This includes denial of access through road closures, unusable facilities due to fresh water shortages, unsanitary working conditions and full-scale power outages. In Gloucestershire, for example, two substations and one switching station were at risk of inundation, while the E.ON power utility in Castlemead, which serves 285,224 public switch telephone network customers, as well as local businesses, lost service.
Due to such indirect but equally profound infrastructural problems, Morris was unable to move her team to the Stroud back-up site, which was situated in an area likely to lose power. Ironically, the Shire Hall site did not, it transpired, flood but the IT team had to shut down all its physical assets nonetheless. “As it turned out, the flood itself would not have been such a problem. It was the flooded electricity station and the water works going out that was the real issue,” says Morris. As such, it was the threat of disruption as the disaster unfolded, rather than its reality, that informed decisions regarding continuity of IT operations and people management. This is why flooding, argues Jonathan Smith, an account manager at SunGard Availability Services, creates an ever-moving scenario that can be especially difficult not only to plan for, but to manage in real time.
National Rail Enquiries: inbuilt resilience
Many organisations will feel the indirect effects of large-scale national disasters. For those services associated with the Critical National Infrastructure, however, it is especially important that IT systems are designed and developed with high levels of inbuilt resilience from the outset. As the UK’s chief source of real-time information on national rail routes, disruptions and transport services, National Rail Enquiries (NRE) is a case in point. After all, says Jason Webb, head of online services at NRE, “You can’t respond at the moment disaster strikes. You have to prepare that resilience from the start.”
During major national incidents such as flooding, NRE’s contact centre and website will experience up to ten times the usual demand. At such peak times, when the public is hankering for information, NRE’s website Journey Planner feature will be used by up to 1.3 million people a day. In addition, the site has to be able to publish real-time information, taken from the timetable and train running schedules. An IT failure during peak times would create a major information vacuum that could have a severe impact on the ground. “We are very keen to ensure that, during these times of peak demand, we don’t experience a degradation of service. So we had to develop a system that could cope with this level of demand,” says Webb.
NRE partnered with information security services provider Vistorm to develop a site that is always up and running, and that provides a fast, robust service at all times, regardless of demand volume. “We have a number of functional requirements and system requirements that we would like to deliver to the passengers. We share with our suppliers the kinds of patterns in customer behaviour that we observe when they are accessing the website. We then pass that onto Vistorm and ask them to translate that into a technical architecture,” Webb explains.
Vistorm went on to develop three models of differing traffic levels, from a day experiencing average traffic levels to that of a disaster. The company was then able to create traffic simulations, against which it tested the site’s application architecture. These models, adds Webb, also had to take into account the significant future growth in NRE’s online channels, which have experienced triple-digit growth in traffic volumes in recent years. “It’s quite a detailed exercise getting those projected volumetrics correct,” he adds. But once built in, Webb and his team are able to manage the indirect impact of natural disasters with considerable ease.
One organisation that has highly complex contingency planning embedded in its DNA, however, is logistics provider Parcelforce. Indeed, according to its operational support manager, Patrick Morgan, the company has planned for nearly every conceivable disruption, including those relating to disease outbreaks such as avian flu and foot and mouth. “Parcelforce is in a really competitive environment, meaning high availability is a key competitive differentiator,” Morgan explains. “The upshot is, if we fail in a commitment to a customer to collect or deliver on time, that could well be enough for a customer to decide they don’t want to trade with us. So we have built into our plan all the possible emergency scenarios you can imagine,” says Morgan.
In order to diversify the risk to its IT and telephony infrastructure nationwide, the company has avoided the creation of giant, centralised contact centres. Instead, Parcelforce operates a large number of fully networked ‘virtual’ contact centres situated in local depots, using a service provided by virtual contact centre specialist NewVoiceMedia. Hosted remotely, the service operates in the so-called ‘cloud’. Customers unknowingly call NewVoiceMedia’s central computer system, which then routes the call through to the appropriate Parcelforce depot.
Consequently, very few of Parcelforce’s depots host any major hardware, and the company enjoys considerable flexibility in its telephony infrastructure. When Parcelforce’s Rotherham depot was flooded and the power cut, Morgan was able to reroute calls to another depot just outside Sunderland, via NewVoiceMedia’s web facility. This was achieved in a matter of minutes, says Morgan. Furthermore, no individuals needed to be immediately relocated, minimising the company’s vulnerability to related people management and accessibility problems.
Diversification, not divestment
Like Gloucestershire County Council, Parcelforce was able to exploit hosted providers to reduce its risk exposure. Outsourcing in this way can often serve as a flexible and scaleable means to maintain the threads of continuity – with little additional internal hassle or expense. But as Rob Fyfe, IT business development director at Swindon Commercial Services (SCS), the spin-off service arm of the borough council, points out, spreading risk through the use of hosted providers does not effectively eliminate risk from the business.
He too regarded the implementation of a hosted mobile working solution as an important way to mitigate the risk inherent in a centralised in-house infrastructure. In the event of the floods, when many of his employees were knee-deep in water and the organisation’s cabling ducts had been inundated, SCS was able to maintain vital communication links and access critical data using the PDA mobile system. Fyfe simply issued staff with the necessary hardware, and told them to literally seek higher ground. But from the outset, “one precondition was that it would be hosted on an extremely secure, and very strong, BC framework”, says Fyfe. For this reason, Fyfe elected to work with IT solutions provider Consilium Technologies, whose hardware is provided by BT.
Through his contract with Consilium, Fyfe effectively partnered, albeit by proxy, with one of the world’s leading BC providers. This was also a strategy pursued with notable precision by Sheffield-headquartered Irwin Mitchell, the UK’s fourth-largest law firm, whose ground and first floors were completely swamped in late June. Like Morris, Gary Thomas, the firm’s head of IT operations, “hadn’t predicted anything on that scale”. Nonetheless, the company, which had partnered with SunGard Availability Services, was better prepared than most.
In conjunction with SunGard’s Jonathan Smith, Thomas and his team assessed the way in which information flows through the business in order to identify the data and services that were critical to the company’s ongoing operations. “We have a service with SunGard for high-availability hosting of servers that runs out of their London base using Network Appliance storage, from which all data is replicated on an hourly basis,” Thomas explains. “We also have the server infrastructure to connect that and bring it online.”
Irwin Mitchell’s provisions ensured the firm did not lose any vital information, while its IP telephony infrastructure allowed the firm to reroute calls. “We have a number of tiers in our recovery plans that allow us to fix problems in both the long and short term,” Thomas continues. But it would have been challenging for the firm to formulate or invoke a BC plan of that scale without an experienced partner. “Business continuity planning is something that we have always had in place, but the use of SunGard was an extension of those plans to ensure the recoverability of the environment,” says Thomas.
Not everyone can afford to be so rigorous, but as the potential impact of flooding on IT services grows, organisations need to assess their level of risk and plan accordingly. Floods discriminate in a way, for example, that other major disasters such as storms or fires do not. Businesses must identify high-risk areas, advises BT, so that they can chart the danger to the organisation’s immediate critical assets, as well as those relating to business suppliers. Once the risks are known, the business can prioritise. Planning for and managing the impact of flooding is the first, critical line of defence.
