What Colorado learned from treating a cyberattack like a disaster

The Colorado Department of Transportation joined the ranks of dozens of other U.S. government entities affected by the SamSam ransomware virus when it was infected with the malware in February 2018. While the incident was costly — nearly 2,000 computers, servers and network devices were encrypted, while the state spent about $1.5 million to undo the damage after refusing to pay the ransom — Colorado also created a new model for state and local governments dealing with cyberattacks in handling it like it would a natural disaster.

The decision by then-Gov. John Hickenlooper to declare a statewide emergency on March 1, ten days after the initial infection was detected, allowed officials to bring in resources from the National Guard and other states, create a unified command structure and perhaps most crucially, spare the state’s IT workers from having to work any more 20-hour shifts fueled by junk food, said Kevin Klein, Colorado’s director of homeland security and emergency management.

“We switched from Doritos and Mountain Dew to actual food,” Klein said Tuesday at the National Governors Association’s cybersecurity summit in Shreveport, Louisiana.

Klein also recounted for the audience of state IT and security officials how the SamSam malware infested CDOT’s network. In mid-February 2018, the department activated a new virtual server for testing, but the server’s security software was still on its default settings, making it an appealing target when it started broadcasting its IP address to the rest of the internet.

“It started broadcasting ‘I’m here, I’m here, come attack me,’ which of course happened within 48 hours,” Klein said.

Within a day, Klein said, the server was subjected to 40,000 brute-force attacks. A day after that, SamSam malware had found an entrance and used the server’s administrative privileges to penetrate the rest of the CDOT network.

In total, the ransomware infected 1,274 laptops, 427 desktops, 339 servers, 158 databases, 154 software applications and all voice-over-IP phones used by CDOT at 200 locations across the entire state, Klein said. While the state’s traffic operations were not impacted, the department’s internal business systems — including finance and payroll operations — had been knocked offline.

The first days after the attack were messy, as Colorado Chief Information Security Officer Deborah Blyth recounted to StateScoop last month, with teams from the state Office of Information Technology working around-the-clock and subsisting on pizza runs carried out by Blyth herself. Ten days in, with the malware starting to spread again, Hickenlooper signed his disaster declaration — the first time any state used one for a cyberattack.

The declaration reshuffled the response to the ransomware attack by bringing in Klein’s office to coordinate emergency operations — including better catering and shift scheduling — and allowing Colorado to call on other states for assistance, which is common practice following a hurricane or wildfire.

Klein said the first task after Hickenlooper’s order was to establish “recovery priorities,” starting with CDOT’s financial operations so the agency could make its next payday. Other priorities included protecting traffic operations by keeping those systems separated from the infected portions of CDOT’s network, and finally getting the department back to its regular operations. Now with several agencies responding to the incident — CDOT, OIT and the state emergency management office — they formed a unified command group and brought in more support from the National Guard, FBI and Department of Homeland Security. Workers who responded from other states helped re-image the large number of devices that had been taken out.

“Somebody’s got to be in charge, and that’s where the incident command structure comes into place,” Klein said. “Planning priorities were based on consensus.”

Still, there were missteps as the state took this new approach, he said. Organizing communications among the unified command group proved more difficult than expected because of the addition of vendors, federal help and spokespeople from multiple state agencies talking to the media. Klein also said IT workers struggled to get a complete picture of the affected systems after discovering the state did not maintain an offline version of its network map.

And one provision in CDOT’s continuity-of-operations plan could’ve inadvertently made the crisis worse, Klein said, as it instructed workers to take their laptops to the Department of Public Health’s headquarters, which could have exposed another agency’s network to an infected device. Klein said one CDOT official told him the agency’s continuity plan was more appropriate for a meteor strike than a cyberattack.

“We had two people who did that and fortunately we stopped them before they could get there,” he said.

Despite the hiccups, the disaster approach proved effective. About 80 percent of CDOT’s systems were recovered within a month of the initial SamSam attack. Other governments hit by ransomware, including Alaska’s Matanuska-Susitna Borough, have since issued their own disaster declarations, and many states are starting to incorporate simulated cyberattacks into their natural disaster drills.

“We put a structure around it, just like any other incident,” Klein said.