911 outage

verizon-screwup-caused-911-outage-in-6-states—carrier-agrees-to-$1m-fine

Verizon screwup caused 911 outage in 6 states—carrier agrees to $1M fine

911 outage, Policy, verizon wireless / /

That’ll teach ’em —

Verizon initially failed to remove a flawed update file that caused two outages.

–

A Verizon logo on top of a black background.

Verizon Wireless agreed to pay a $1,050,000 penalty to the US Treasury and implement a compliance plan because of a 911 outage in December 2022 that was caused by a botched update, the Federal Communications Commission announced today.

A consent decree explains that the outage was caused by “the reapplication of a known flawed security policy update file.” During the outage, lasting one hour and 44 minutes, Verizon failed to deliver hundreds of 911 calls in Alabama, Florida, Georgia, North Carolina, South Carolina, and Tennessee, the FCC said.

“The [FCC] Enforcement Bureau takes any potential violations of the Commission’s 911 rules extremely seriously. Sunny day outages, as occurred here, can be especially troubling because they occur when the public and 911 call centers least expect it,” Bureau Chief Loyaan Egal said.

The flawed update file was involved in another outage that happened two months earlier, in October 2022. After the October incident, Verizon “implemented a wide range of audits and technical system updates designed to protect against future recurrences of configuration and one-way audio issues,” the consent decree said.

Even before the December outage, Verizon knew that the problematic update file “was related to the root cause of the outage that occurred in October,” the FCC said. “Due to insufficient naming convention protocols and a failure to follow then-current implementation protocols, the flawed security policy update file was reintroduced into the Verizon Wireless network. This resulted in the [December] outage, however without the one-way audio issues.”

Verizon failed to remove flawed update file

The December outage happened when the flawed update file was re-applied by a Verizon Wireless employee. But the fault lies with more than one person, the FCC said:

Despite this prior outage and Verizon Wireless’s understanding that the flawed security policy update file resulted in that prior outage, Verizon Wireless did not remove that security policy update file from the inventory of available security policies, which enabled personnel to select and reapply the flawed security policy update file to the Verizon Wireless network. Additionally, Verizon Wireless admits its employees failed to comply with its “business-as-usual” operating and implementation procedures, which procedures required additional oversight prior to the implementation of the type of security policy update that caused the December Outage.

Verizon admitted in the consent decree that the FCC’s description is “a true and accurate description of the facts underlying the Investigation.” The agreed-upon compliance plan includes processes to prevent the reoccurrence of firewall and one-way audio problems, enhanced processes for implementing security policy updates, testing before significant network changes, risk assessments, a compliance training program for employees, and more.

Verizon must file four compliance reports over the next three years and “report any material noncompliance” with 911 rules and the consent decree terms to the FCC. In a statement provided to Ars, Verizon said the December 2022 outage “was a highly unusual occurrence. We understand the critical importance of maintaining a robust and reliable 911 network, and we’re committed to ensuring that our customers can always rely on our services in times of need.”

Verizon has 30 days to pay the $1.05 million fine. Verizon’s wireless service revenue was $19.5 billion in the first quarter of 2024. The entire company’s quarterly operating revenue was $33 billion, and net income was $4.7 billion.

Verizon isn’t the only major carrier to have a big outage caused by a faulty update. In February 2024, a major AT&T wireless outage caused by a botched network update led to warnings that 911 access could be disrupted. The FCC was investigating that outage.

There was also a statewide 911 outage for two hours in Massachusetts this month, but that one was caused by a faulty firewall used by the state’s 911 vendor.

Verizon screwup caused 911 outage in 6 states—carrier agrees to $1M fine Read More »

statewide-911-outage-was-caused-by-911-vendor’s-malfunctioning-firewall

Statewide 911 outage was caused by 911 vendor’s malfunctioning firewall

911 outage, Policy / /

911 outage —

911 vendor Comtech still investigating why firewall blocked emergency calls.

–

Emergency number 911 inputted on a cell phone dialing screen.

Getty Images | artas

A 911 vendor’s malfunctioning firewall caused a statewide outage in the emergency calling system in Massachusetts on Tuesday afternoon, the state government said. A Massachusetts government press release issued yesterday said the state’s 911 vendor, Comtech, “has advised State 911 that they have applied a technical solution to ensure that this does not happen again.”

“A preliminary investigation conducted by the State 911 Department and Comtech determined that the outage was the result of a firewall, a safety feature that provides protection against cyberattacks and hacking,” the announcement said. “The firewall prevented calls from getting to the 911 dispatch centers, also known as Public Safety Answer Points (PSAPs).”

Comtech’s initial review “confirmed that the interruption was not the result of a cyberattack or hack,” but “the exact reason the firewall stopped calls from reaching dispatch centers remains under review,” the state said. A full review is continuing.

The 911 outage lasted two hours. Shortly after it began, the State 911 Department alerted local law enforcement and issued a statewide emergency alert to residents advising them to call their local public safety business line directly if they had an emergency.

“Although some calls may not have gone through, the system allows dispatch centers to identify the phone number of callers and return those calls. The Department has not received any reports of emergencies impacted during the interruption,” the Massachusetts announcement said.

State 911 Department Executive Director Frank Pozniak promised that the department “will take all necessary steps to prevent a future occurrence.” Massachusetts has 204 Public Safety Answering Points that received an average of 8,800 calls, combined, per day in 2023.

Comtech announced a five-year contract extension with Massachusetts in May 2024. “Since 2014, Comtech has been developing, implementing and operating a secure, IP-based NG911 [Next Generation 911] system for the Commonwealth of Massachusetts,” the vendor announcement said. Comtech says it has provided public safety and security technology for over 25 years and that “service providers, states, and local jurisdictions nationwide rely on our portfolio of mission‑critical products and services.”

911 disruptions happen occasionally and are sometimes caused by broader outages in phone networks. A 37-hour CenturyLink outage in December 2018 that disrupted 911 service for millions of Americans was caused by “malformed packets.” In February 2024, a major AT&T wireless outage caused by a botched network update led to warnings that 911 access could be disrupted.

Statewide 911 outage was caused by 911 vendor’s malfunctioning firewall Read More »