States Press CenturyLink on Nationwide Outage Affecting 911

States are looking into the CenturyLink outage that disrupted 911 systems across the country, with more formal investigations possible, said utility commissions we surveyed Wednesday. State-level probes are important and complement the federal probe announced last week by FCC Chairman Ajit Pai (see 1812280033), said NARUC and the National Association of State 911 Administrators (NASNA).

CenturyLink said the outage began Dec. 27 and "impacted voice, IP and transport services" for some customers, plus its network troubleshooting ability, prolonging the event. "Customer impact varied across our operations," the telco said Wednesday. "Affected services began to restore as of December 28, and the network traffic had normalized as of December 29. Steps are being taken to help prevent the issue from reoccurring." It added, "Where CenturyLink is an emergency 911 provider, 911 calls completed during the incident. However, address locator information (ALI) did not accompany some of those calls."

"The outage was caused by a faulty network management card from a third-party equipment vendor that caused invalid traffic replication," CenturyLink said. "There is no indication that this outage was triggered by any maintenance activity on the card, software, or any neighboring equipment." The telco is "in contact with the FCC and policymakers and will cooperate fully with any investigation."

Pai last week called the outage "completely unacceptable." The commission didn't comment Wednesday.

State Reviews

The Washington Utilities and Transportation Commission has started the investigation it announced last week, a spokesperson said Wednesday. “These types of investigations tend to be more complex and technical,” taking six to 18 months, she emailed. When finished, “UTC staff will issue a report and, depending on staff’s findings, recommend the commission take formal action against the company with penalties and/or compliance reporting or requirements.”

The Idaho Public Utilities Commission asked CenturyLink about the outage's extent and hopes to finish gathering information by the weekend, said PUC Utility Analyst Carolee Hall. The agency, which doesn’t regulate telecom but maintains authority over service quality and cares about 911 issues, may investigate if it determines problems were severe, she said.

The Montana Public Service Commission’s “consumer assistance division is following this very closely, and the Commissioners are getting regular updates,” a spokesperson emailed. Commissioners would have to vote to open investigation, he said. Colorado PUC staff is gathering information, but it’s not a formal investigation, a spokesperson said.

The Wyoming PSC will make inquiries to determine whether it should get more involved but hasn’t decided whether to do a formal investigation, said Chief Counsel Chris Petrie. Cheyenne Mayor Marian Orr (I) supported a state-commission probe, tweeting that CenturyLink officials earlier told her the company had redundancies to prevent such issues. “The fact I see their local WY representatives and lobbyist on a regular basis, they know my cell number, and I was never contacted by anyone as to this outage in our community speaks loudly of their silence.”

The Minnesota PUC learned of a 911 outage in the Duluth area and is monitoring reports, said Consumer Affairs Office Supervisor Anne Thom. She noted the commission last August opened docket 18-152 about a prior 911 outage involving CenturyLink and West Safety Communications.

The Nebraska PSC is “still assessing the impacts of the outage” and “hasn’t made any decisions on what we will do formally, if anything,” said a spokesperson, saying its problems “don’t seem to be as severe here as what some of the other states have reported.” The North Dakota PSC hasn’t received any complaints about the outage or any notification that the state was affected, a spokesperson said. Utah commissioners haven’t notified staff of any investigation, said Utah PSC Telecommunications Manager Bill Duncan.

Accountability Sought

“A state needs to ensure accountability,” said NASNA Executive Director Evelyn Bailey. Some states have contracts with CenturyLink, she said. Also, “the issue manifested itself differently in different states,” so “the downstream effects from the primary issue weren't the same everywhere,” Bailey said. “The FCC can only do so much, and we view this as a partnership with the states and the Feds each having a role.”

A state with “the resources and ability to investigate outages and potentially fine carriers can only have one impact,” said NARUC General Counsel Brad Ramsay: “It necessarily increases the incentives for safe operation of the 911 emergency calling systems and decreases the odds for a repeat of preventable lapses.”

The faulty network card located in Denver propagated "invalid frame packets that were encapsulated and then sent over the network via secondary communication channels," CenturyLink said. Once there, the invalid packets "multiplied, forming loops and replicating high volumes of traffic across the network, which congested controller card CPUs (central processing unit) network-wide, causing functionality issues and rendering many nodes unreachable," the company emailed. "The source was an electronic network element within the transport layer of the CenturyLink network driven by [the] card ... CenturyLink teams worked around the clock until the issue was resolved."

The secondary communication channels "have been disabled network-wide" and CenturyLink "has established a network monitoring plan for key parameters that can cause this type of outage, based on advice from the third-party equipment vendor," the telco said. "Improvements to the existing monitoring and audits of memory and CPU utilization for this type of issue have been put into place. Enhanced visibility processes will quickly identify and terminate invalid packets from propagating the network."

“We appear to need a new process to review the readiness of the network to support 911, particularly as we move to a VoIP-centric world,” said National Regulatory Research Institute Telecom Principal Sherry Lichtenberg. States must be involved, she said. “Not only because these outages impact their constituents and their first responders, but because they are the best suited to understand and address these impacts.”