Yes, I did.
As an added bonus, the shear number of fans (and giant dehumidifiers) in use are causing our UPSes (and a few circuit breakers) to complain. I’m speculating that we are currently the most non-green building in Hunt Valley.
I had to run out of the office at lunch today to hit Lowes for a few parts. About 2 flights down I realized I had left my Treo in its cradle. I quickly decided against going back and getting it since I was just taking a quick trip and would be back in no time.
What could possibly go wrong?
Thirty minutes later I return from my errand and walk back into the lobby of the building. The sound of rushing water is unmistakable. I’m greeted by someone from the property management company with the title ‘Chief Engineer’ stitched on his uniform. He proceeds to explain that ‘there’s been a major water line break on the third floor’ (my floor) and they need to ‘shut off the power to the building for the rest of the day because it’s running down the electrical closets on each floor’.
Turns out this happened 5 minutes after I left and work had been calling me frantically to come back and start shutting down my servers/racks.
(just one section of the flood that hit about 1/3 of our floor)
A 1.5″ line in the restroom burst while it was being worked on. The had… difficulty… in finding the water shutoff. A 1.5″ line under pretty high pressure will put out a lot of water in a very short time. A lot. The fun part is that my computer is about 4 foot away from these restrooms directly across a hallway. As luck would have it we had no water penetration into our server room. The water followed the path of least resistance (up and down the hallways and down the stairwell and elevator shaft (and the phone/electrical rooms of course)).
Most of the office was already vacant since they had (obviously) closed the office for the day. Turns out there was a bit of a mass panic as everyone reached under their desks and flipped off their surge strips because of the quickly advancing water. Hopefully I won’t get too many people asking me to recover ‘lost’ work as a result of the sudden powerdowns.
Since my equipment was at least 4″ off the floor (racks) and I have a minimum of 1 hour battery runtime on each rack, I elected to try and keep everything running for as long as possible. Turns out that was the right call since once the water was finally shut off, the danger to the electrical rooms passed and they ended up not having to cut the power after all.
We still have 72 hours of big-ass blower fans and other restoration work to look forward to.
Next time I’m going back for the phone.