Security

One day last October, subscribers to an ISP known as Windstream began flooding message boards with reports their routers had suddenly stopped working and remained unresponsive to reboots and all other attempts to revive them.

“The routers now just sit there with a steady red light on the front,” one user wrote, referring to the ActionTec T3200 router models Windstream provided to both them and a next door neighbor. “They won’t even respond to a RESET.”

In the messages—which appeared over a few days beginning on October 25—many Windstream users blamed the ISP for the mass bricking. They said it was the result of the company pushing updates that poisoned the devices. Windstream’s Kinetic broadband service has about 1.6 million subscribers in 18 states, including Iowa, Alabama, Arkansas, Georgia, and Kentucky. For many customers, Kinetic provides an essential link to the outside world.

“We have 3 kids and both work from home,” another subscriber wrote in the same forum. “This has easily cost us $1,500+ in lost business, no tv, WiFi, hours on the phone, etc. So sad that a company can treat customers like this and not care.”

After eventually determining that the routers were permanently unusable, Windstream sent new routers to affected customers. Black Lotus has named the event Pumpkin Eclipse.

A deliberate act

A report published Thursday by security firm Lumen Technologies’ Black Lotus Labs may shed new light on the incident, which Windstream has yet to explain. Black Lotus Labs researchers said that over a 72-hour period beginning on October 25, malware took out more than 600,000 routers connected to a single autonomous system number, or ASN, belonging to an unnamed ISP.

While the researchers aren’t identifying the ISP, the particulars they report match almost perfectly with those detailed in the October messages from Windstream subscribers. Specifically, the date the mass bricking started, the router models affected, the description of the ISP, and the displaying of a static red light by the out-of-commission ActionTec routers. Windstream representatives declined to answer questions sent by email.

According to Black Lotus, the routers—conservatively estimated at a minimum of 600,000—were taken out by an unknown threat actor with equally unknown motivations. The actor took deliberate steps to cover their tracks by using commodity malware known as Chalubo, rather than a custom-developed toolkit. A feature built into Chalubo allowed the actor to execute custom Lua scripts on the infected devices. The researchers believe the malware downloaded and ran code that permanently overwrote the router firmware.

“We assess with high confidence that the malicious firmware update was a deliberate act intended to cause an outage, and though we expected to see a number of router make and models affected across the internet, this event was confined to the single ASN,” Thursday’s report stated before going on to note the troubling implications of a single piece of malware suddenly severing the connections of 600,000 routers.

The researchers wrote:

Destructive attacks of this nature are highly concerning, especially so in this case. A sizeable portion of this ISP’s service area covers rural or underserved communities; places where residents may have lost access to emergency services, farming concerns may have lost critical information from remote monitoring of crops during the harvest, and health care providers cut off from telehealth or patients’ records. Needless to say, recovery from any supply chain disruption takes longer in isolated or vulnerable communities.

After learning of the mass router outage, Black Lotus began querying the Censys search engine for the affected router models. A one-week snapshot soon revealed that one specific ASN experienced a 49 percent drop in those models just as the reports began. This amounted to the disconnection of at least 179,000 ActionTec routers and more than 480,000 routers sold by Sagemcom.

The constant connecting and disconnecting of routers to any ISP complicates the tracking process, because it’s impossible to know if a disappearance is the result of the normal churn or something more complicated. Black Lotus said that a conservative estimate is that at least 600,000 of the disconnections it tracked were the result of Chaluba infecting the devices and, from there, permanently wiping the firmware they ran on.

After identifying the ASN, Black Lotus discovered a complex multi-path infection mechanism for installing Chaluba on the routers. The following graphic provides a logical overview.

There aren’t many known precedents for malware that wipes routers en masse in the way witnessed by the researchers. Perhaps the closest was the discovery in 2022 of AcidRain, the name given to malware that knocked out 10,000 modems for satellite Internet provider Viasat. The outage, hitting Ukraine and other parts of Europe, was timed to Russia’s invasion of the smaller neighboring country.

A Black Lotus representative said in an interview that researchers can’t rule out that a nation-state is behind the router-wiping incident affecting the ISP. But so far, the researchers say they aren’t aware of any overlap between the attacks and any known nation-state groups they track.

The researchers have yet to determine the initial means of infecting the routers. It’s possible the threat actors exploited a vulnerability, although the researchers said they aren’t aware of any known vulnerabilities in the affected routers. Other possibilities are the threat actor abused weak credentials or accessed an exposed administrative panel.

An attack unlike any other

While the researchers have analyzed attacks on home and small office routers before, they said two things make this latest one stand out. They explained:

First, this campaign resulted in a hardware-based replacement of the affected devices, which likely indicates that the attacker corrupted the firmware on specific models. The event was unprecedented due to the number of units affected—no attack that we can recall has required the replacement of over 600,000 devices. In addition, this type of attack has only ever happened once before, with AcidRain used as a precursor to an active military invasion.

They continued:

The second unique aspect is that this campaign was confined to a particular ASN. Most previous campaigns we’ve seen target a specific router model or common vulnerability and have effects across multiple providers’ networks. In this instance, we observed that both Sagemcom and ActionTec devices were impacted at the same time, both within the same provider’s network.This led us to assess it was not the result of a faulty firmware update by a single manufacturer, which would normally be confined to one device model or models from a given company. Our analysis of the Censys data shows the impact was only for the two in question. This combination of factors led us to conclude the event was likely a deliberate action taken by an unattributed malicious cyber actor, even if we were not able to recover the destructive module.

With no clear idea how the routers came to be infected, the researchers can only offer the usual generic advice for keeping such devices free of malware. That includes installing security updates, replacing default passwords with strong ones, and regular rebooting. ISPs and other organizations that manage routers should follow additional advice for securing the management interfaces for administering the devices.

Thursday’s report includes IP addresses, domain names, and other indicators that people can use to determine if their devices have been targeted or compromised in the attacks.