Report on last weekend’s Power Surge-Related Outage
November 19, 2012 1 Comment
This past weekend, several hundred Directcom residential customers were out of service after a massive power surge, which occurred at 4am Saturday morning, when City Power was turned back on after a maintenance outage to work on the city’s power grid. The surge damaged the circuitry in about 300 customer power adapters which power the fiber electronics box on each home. Our crews completed the immense task of replacing all the damaged transformer units on customer homes and bringing all affected customers back online by about 5:30pm on Sunday evening, after working around the clock through the weekend.
We are now back to regular trouble ticket procedures. If your connection is still down, please call tech support at 801 789 4999 for troubleshooting. We understand there are probably still a few fiber customers out there who are not online for various reasons, but it should not be because of the outdoor equipment not having power anymore. A simple reboot of your internal router will probably solve most problems at this point.
This power outage has obviously been far more disruptive than we expected. We actually fortunately had about 150 power units on hand as part of regular inventory and for emergencies, but when the power came back on, after the surge our system registered about 490 alarms from fiber ONTs (optical network terminals) on customer homes. This number just kept growing as the morning wore on, and the batteries on those ONTs ran out, so our techs had to sort through those alarms. Our general manager drove through the night to Oregon and back to pick up every power unit in stock at a supplier there, to make sure we had enough to replace all the damaged customer power units.
By Sunday night when the alarms were all taken care of, our techs in the field had replaced about 300 transformers that were burned out by the power surge.
Thank you to our amazing techs, many of whom have been going non-stop since 4am Saturday morning, and have just done a fantastic job, visiting literally hundreds of homes. The fact that they were able to bring everyone back online by Sunday evening after a disaster of this scale was a super-human effort. If you see them give them a big hug. They are pretty battle-fatigued at this stage. They gave up their Saturday and their Sunday with their families to get our customers back online, and probably deserve a bonus.
What actually happened? The fiber electronics, called an ONTs (optical network terminals), that convert the digital light pulses into an Ethernet signal are mounted on the outside of your home. These electronics are powered by a battery. (This battery is designed to last for about 8 hours to power your home phone during an emergency situation, even if the power is out.) This battery is usually continually being charged by a transformer which is directly wired onto the city power meter. When the city power came back on after the outage, this surge fried the transistors in about 300 customer transformers/inverters. Although the ONT electronics were still fine, the computer inside the box was not getting any power, so the customer could not get an internet connection. To restore the connection, we had to replace the damaged transformer.
The other question on the mind of those customers who were affected is probably—why this time? The city has had power outages before. The number of customers who were knocked out by the outage is obviously a concern to us. We expected some minor problems, as this is always possible with power surges, but this particular power outage caused more widespread damage than we expected with customers’ power supply equipment. We are not yet sure why this particular power outage was so destructive to electrical equipment, but we will be certainly be investigating, asking some questions of our suppliers and changing procedures to make sure this does not happen again.
We chose to power the units this way for two primary reasons.
1) We decided to pay for the power ourselves rather than having you the customer pay for the power to run the ONT, even though in truth they do not use much power.
2) We wanted to be able to access the ONT anytime for maintenance purposes, without having to intrude into customers homes.
Many of the transformers that were damaged were older units. The manufacturer assures us that this technology is getting better and more resistant to surges. We are now on the 3rd generation of this model, so hopefully newer customers will already be immune to surges.
This situation could obviously have been avoided if those ONTs had not been directly wired to the city power, so we will revisit how we do that. One possible solution would be to have an electrician install an outdoor electrical receptacle, wired back through the wall into the customers internal power, next to each ONT. Another solution, which we will already were planning to be launch soon, is to switch to a new ONT that also has a built-in wireless router feature, but would be installed inside a customer’s basement or garage rather than outside.
We really appreciate your patience as we worked through this situation–we know how frustrating it is to be without internet service. Our goal is to be up 100% of the time. Unfortunately this outage was caused by environmental factors outside of our control, namely a freakishly powerful power surge. We obviously have more work to do to surge-proof your equipment, and that work will start Monday morning. Thanks for supporting your local broadband company. We appreciate your loyalty.