Snipped zip tie shuts down virtually all digital city services in Honolulu for a day

At 9:15 a.m. on Monday, a contractor replacing the fire suppression system in the basement of Honolulu’s Frank Fasi municipal building cut a zip tie that had been holding wires together — and within minutes, virtually all computer services the entire Honolulu city government had been shut down.

The movement of the wires, the contractor realized just minutes later, had imitated to the building — which houses the city’s data center — conditions consistent with a fire, though no fire was actually present. The building’s main power supply was automatically shut off as a result, and the secondary power systems only sustained the city’s larger systems — like its 1,000 terabyte primary enterprise storage system — for a few minutes, city officials said.

The resulting shutdown of computing affected nearly every government service the city offers, said city IT Director Mark Wong in a press conference on Monday. The daylong shutdown knocked out virtually every computer and telephone service outside of emergency services for the city and its 10 satellite city halls.

“The majority of the city network went down,” said city Department of Information Technology Director Mark Wong in a press conference on Monday. “That includes our networking, internet, websites, telephone systems, storage, mainframe and most of our application servers. For all practical purposes, the city was without computing except for emergency services, which are on a separate network.”

The city said some affected services, including the telephone, internet and mainframe, were back online by 10:00 a.m. The rest, including larger items like the storage system, stayed down until the end of the business day, the spokesperson said.

“These are some of the largest systems in the state. They’re complicated,” Wong said, referencing the length of time it took for the city to get its services back online.

In the wake of the outage, Wong said he is considering further segmenting volumes of the city’s data storage to more easily verify the security of the enterprise as a whole. The city also said it’s adding Saturday hours this week to its DMV and satellite city halls to accommodate the residents who were turned away because of Monday’s outage.

The city has had problems with the data center before. Almost exactly one year ago , the city saw its driver’s licensing and vehicle registration services nearly crippled for days as a result of glitches in its network attributed to incompatibility issues between new hardware and old software in the mainframe.