Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

He is just having a temper tantrum. Everything can fail and everything does fail. There was an outage a few years ago in a well known colo/ip/managed services provider where the feeder line from the power company failed, the ATS flipped to the backup power which had a limited run time. And, due to one of those 1/1000 events that should never happen (because that ATS should flip maybe 5-6 times a year) it fused to the new position. And it happened in a place where the DC operator would cut off the service on the second line to ensure they can safely work on removing the affected ATS. So the redundant power lines + backup power did not work. If you happened to be in that specific area of the building and happened to know building engineers and data center engineers and power company engineers you would have had heard what actually happened. Otherwise you just got "Imminent power failure" notification. Hopefully you knew that it means you want to shutdown all your workloads remotely and send someone you have on call who can reach the data center in 10-15 minutes to physically disconnect your PDUs from the incoming lines just in case someone messed up when they play at fixing the power so you don't blow 10-30% of your PDUs.

That's the reality of the life in a data center. So yeah, either accept that stuff like this happens or build for stuff like this happening. Engineering around physical problems in the cloud environment is far easier than in the data center environment.



I’ve seen at least 3 variations of the problem you mention, where power failover caused protracted downtime requiring rush delivery of niche replacement hardware. (That last is big: I’ve seen 8-figure enterprise hardware spends down for a week because it requires flying someone in to fix it, whereas AWS/Google has 24x7 staffing along with redundancy).

Anyone thinking this doesn’t happen with private data centers is either very green or selectively excusing problems.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: