Seattle’s King County poised for full cloud archive switch

After years of using a magnetic-tape-based archive system, the IT staff in King County, Washington, faced a crossroads: Stay the course or move to the cloud.

Bob Micielli, the county’s IT enterprise manager, was tasked with charting how the county would handle the data of its more than 2 million residents. The stakes, he said, were high.

“The cost was continuing to increase and we were due for an infrastructure refresh this year,” Micielli told StateScoop. “We really wanted to eliminate tapes and come up with a more modern solution.”

But with the huge volume of data that comes with the massive county, which includes Seattle, and the security requirements associated with that information, Micielli also needed to proceed cautiously.

Yet during the yearlong process, Micielli and his team were able to decide on a new cloud solution and transfer all the county’s data over to the new system. By Labor Day on Sept. 7, Micielli believes his team will finish moving King County’s backups to the cloud, courtesy of Amazon Web Services’ Simple Storage Service.

“It has streamlined our process significantly,” Micielli said. “There’s a huge improvement not only from an operational point of view and a liability point of view, but from a financial point of view.”

Frustration breeds change

The team started planning for the move in July 2014. Frustration with the old system was growing among staffers, particularly with the off-site archive facility located 50 miles from the county’s offices.

“We were using 2 1/2 full-time employees to keep the system running and we were burning through 2,000 tapes a year, which is about $64,000 a year we spent on it,” Micielli said. “Back in the off-site storage, we had over 12,000 tapes and we were getting charged $50,000 a year to keep the tapes off-site.”

Those employees were frequently called on to do manual work on the systems, a tedious and time-consuming process.

“It was definitely on a weekly basis that there were issues with our tape infrastructure,” Micielli said. “We had multiple structure failures, like the robot arm would lose calibration, we would probably have to get that calibrated once every one or two months, and then we would have just plain old drive failures.”

Micielli noted that made the cloud seem more appealing, but it was no simple process to make the change. Once the staff settled on Amazon’s services, they tested the new technology from July to November last year.

“Once we got started, we started slow, and did a lot over time,” Micielli said. “We did backups in parallel, to tape and to cloud, to prove that we could restore from either one and that they were safe … That’s how folks really got the concept and we were able to move forward.”

But the cost savings associated with the move also went a long way toward persuading other decisionmakers in the county to support the change.

“We showed we could save over a million dollars in infrastructure refresh costs in this year alone and we could also save $200,000 in operating costs year over year,” Micielli said.

Security worries

Yet the savings would only go so far if Micielli’s team couldn’t provide the necessary security.

The county’s data included sensitive material, like court records and medical information, so Micielli ensured the new system met Criminal Justice Information Services security standards and the requirements laid out by the Health Insurance Portability and Accountability Act, known as HIPAA. But he also wanted to showcase four different capabilities of the system to assuage possible concerns.

“Encryption at rest, encryption at transit, manage the keys in-house and the data was stored 50 miles away from its origination,” Micielli said. “That’s the secret sauce, prove those four things and you’re OK.”

Despite his efforts to assure the county’s agencies, Micielli was surprised by the amount of resistance he still faced on the change, even from his staff.

“They’ve been doing this the same way for the past 10 years, we were changing something fundamental for them, and that was a big hurdle to overcome and we were surprised from the amount of pushback we got,” Micielli said. “So we had to kind of assure them and show them that, ‘Hey, by not having to do some of these menial things anymore, it can actually free you up and we can cross-train you to do other functions that are of more value to the county.’”

Transition time

But agencies came around, and in January, Micielli was finally ready to start transferring everything over. By March, the IT team had moved 500 of 790 total data sets from their agency clients using NetApp AltaVault data protection software, but they hit a snag.

“We started with virtual appliances, but they really just weren’t fast enough for some of the larger terabyte storage we were doing,” Micielli said.

Instead, Micielli’s group decided to switch to physical appliances to speed up the process. The team had wait until this July to get them in the door, but  Micielli feels the results — physical appliances are 10 times faster than the virtual ones, he said — made their patience pay off. That speed has positioned the team to complete the transition on Sept. 7, he said. And he believes they’re already seeing the benefits from the move.

“With the legacy system, we would see an average of 70 errors per day,” Micielli said. “Now we’re seeing less than 20 a day, so our error rate has significantly decreased.”

But Micielli is particularly energized about the new possibilities that the savings from the project will open up.

“In government, it’s not like in sales, you have a fixed budget every year,” Micielli said. “This helps us then look at other areas we can modernize as well, so we can take those savings and find other areas we can invest in and make improvements. From the manpower side, can we re-invest our manpower to work in other activities to bring more value as well?”