Forum Discussion
David_Larsen
Aug 29, 2022Employee
My favorite total data center outage was caused by a random rain drop.
We had just finished a major maintenance of our data center generators for power outages. A couple days later a major rainstorm comes through town and power gets cut off. Generator kicks on as it should and then a raindrop hit just right and ricocheted off the pavement and hit the water sensor on the bottom of the generator making the generator think it was flooding and powered off. Within 30 minutes all batteries are drained in the data center and the whole thing shut down. Took 36 hours straight to bring all the systems back online in a graceful manner and validate everything was working. Longest day ever!!!