European customers of Amazon’s EC2 cloud computing service and Microsoft’s BPOS online desktop application suite suffered unplanned downtime last night following stormy weather in Dublin, where the two services are hosted.
Amazon informed customers that a transformer at its Dublin data centre was struck by lightning, “sparking an explosion and fire”.
Not only did this cut the power to the data centre, but also prevented the back-up power system from working automatically. “The transient electric deviation caused by the explosion was large enough that it propagated to a portion of the phase control system that synchronizes the backup generator plant, disabling some of them,” the company said in a statement to customers. “Power sources must be phase-synchronized before they can be brought online to load. Bringing these generators online required manual synchronization.”
Having lost power, some of the Electric Block Storage (EBS) servers that support EC2 had to be recovered manually, Amazon wrote on its service health dashboard website at 11pm last night. “Restoring these volumes requires that we make an extra copy of all data, which has consumed most spare capacity and slowed our recovery process.”
“We anticipate that it will take 24-48 hours until the process is completed,” it said.
Affected customers include the Daily Telegraph newspaper, whose puzzles site is hosted on Amazon EC2, and social media analytics service PeerIndex.
It is the second high-profile outage at Amazon Web Services so far this year. In April, US-based customers were unable to access their hosted systems following a “network event” at its North Virginia data centre. That outage took three days to resolve, and some customers’ data was never recovered.
Meanwhile, Microsoft reported on one of its official Twitter feeds last night that “European data power issue affects access to BPOS”. Within a few hours, it said that “BPOS customers are back online for EMEA customers”. Microsoft also hosts its European cloud services in Dublin, in a 550,000 square foot facility that it opened in 2009.
An article published last week by the Daily Telegraph (which uses Google Apps) claimed that “if a major catastrophe befell the whole of Dublin … the whole operation would switch seamlessly to [Microsoft’s] Amsterdam [data centre]”.
