That’ll teach ’em —
Verizon initially failed to remove a flawed update file that caused two outages.
Verizon Wireless agreed to pay a $1,050,000 penalty to the US Treasury and implement a compliance plan because of a 911 outage in December 2022 that was caused by a botched update, the Federal Communications Commission announced today.
A consent decree explains that the outage was caused by “the reapplication of a known flawed security policy update file.” During the outage, lasting one hour and 44 minutes, Verizon failed to deliver hundreds of 911 calls in Alabama, Florida, Georgia, North Carolina, South Carolina, and Tennessee, the FCC said.
“The [FCC] Enforcement Bureau takes any potential violations of the Commission’s 911 rules extremely seriously. Sunny day outages, as occurred here, can be especially troubling because they occur when the public and 911 call centers least expect it,” Bureau Chief Loyaan Egal said.
The flawed update file was involved in another outage that happened two months earlier, in October 2022. After the October incident, Verizon “implemented a wide range of audits and technical system updates designed to protect against future recurrences of configuration and one-way audio issues,” the consent decree said.
Even before the December outage, Verizon knew that the problematic update file “was related to the root cause of the outage that occurred in October,” the FCC said. “Due to insufficient naming convention protocols and a failure to follow then-current implementation protocols, the flawed security policy update file was reintroduced into the Verizon Wireless network. This resulted in the [December] outage, however without the one-way audio issues.”
Verizon failed to remove flawed update file
The December outage happened when the flawed update file was re-applied by a Verizon Wireless employee. But the fault lies with more than one person, the FCC said:
Despite this prior outage and Verizon Wireless’s understanding that the flawed security policy update file resulted in that prior outage, Verizon Wireless did not remove that security policy update file from the inventory of available security policies, which enabled personnel to select and reapply the flawed security policy update file to the Verizon Wireless network. Additionally, Verizon Wireless admits its employees failed to comply with its “business-as-usual” operating and implementation procedures, which procedures required additional oversight prior to the implementation of the type of security policy update that caused the December Outage.
Verizon admitted in the consent decree that the FCC’s description is “a true and accurate description of the facts underlying the Investigation.” The agreed-upon compliance plan includes processes to prevent the reoccurrence of firewall and one-way audio problems, enhanced processes for implementing security policy updates, testing before significant network changes, risk assessments, a compliance training program for employees, and more.
Verizon must file four compliance reports over the next three years and “report any material noncompliance” with 911 rules and the consent decree terms to the FCC. In a statement provided to Ars, Verizon said the December 2022 outage “was a highly unusual occurrence. We understand the critical importance of maintaining a robust and reliable 911 network, and we’re committed to ensuring that our customers can always rely on our services in times of need.”
Verizon has 30 days to pay the $1.05 million fine. Verizon’s wireless service revenue was $19.5 billion in the first quarter of 2024. The entire company’s quarterly operating revenue was $33 billion, and net income was $4.7 billion.
Verizon isn’t the only major carrier to have a big outage caused by a faulty update. In February 2024, a major AT&T wireless outage caused by a botched network update led to warnings that 911 access could be disrupted. The FCC was investigating that outage.
There was also a statewide 911 outage for two hours in Massachusetts this month, but that one was caused by a faulty firewall used by the state’s 911 vendor.