On Aug. 26, Francis Chan, the chief executive of the Hawaii Health Information Exchange, received troubling news.

The information exchange is a key institution on the data highway that moves Covid-19 information from testing labs to the Hawaii Department of Health, and eventually to the public.

Batches of Covid-19 test data, Chan learned, had gotten held up on the info path from one of the testing labs. As a result, the numbers that had been reported to the public over more than a week were wrong.

And when the health department did report the backlogged cases on Sunday, the numbers appeared to show an alarming spike in cases — some 1,678 total.

The case count was alarming enough that it prompted Hawaii’s Covid-19 liaison, Lt. Gov. Josh Green, to spring into action to figure out what was going on. He was soon explaining to a concerned public and the media that some 600 to 700 cases were part of the backlog and that the situation was not nearly as bad as it seemed.

Although the biggest data glitch, it wasn’t the first or last: in fact, another just happened on Wednesday and Thursday. The Department of Health reported 455 cases on Wednesday but said that didn’t include everything. And Thursday, the count indeed jumped more than 100% to 1,068.

Such fluctuations, Green says, pose problems for government leaders and others trying to make real time decisions during a crisis.

“There are some days when we feel we’re flying blind on a rocket cycle,” he said.

Are Data Glitches Unavoidable?

Such glitches underscore a sobering reality: the state’s system for collecting data on positive cases is being stressed from the volume of information staff is handling, and epidemiologists and information technology specialists are having to adapt on the fly.

It’s not just the large quantity of test data posing a challenge, it’s also that the data is coming from a staggering number of sources, not all of which are reporting the info in exactly the same way.

Before Covid-19, the health department received test results mainly from a handful of big labs, including Diagnostic Laboratory Services, Clinical Labs of Hawaii and some large medical centers, said Jonathan R. Johnson, Electronic Laboratory Capacity IT Specialist with the Hawaii Department of Health. And most of those flowed through the health information exchange.

Now there are upward of 100 labs reporting results, Johnson said. These include small labs and test sites in Hawaii as well as national players, like out-of-state labs and pharmacies that are part of Hawaii’s Safe Travels program.

The result is that every one of these labs has to be set up to send information to the health department, and if there’s a problem it can take significant time and effort to fix it – even if that particular lab reports a relatively small percentage of the state’s cases.

“It’s not that it’s less work,” Johnson said. “It’s the same amount of work” fixing a small lab’s problem as it is fixing one arising from a bigger lab.

In theory, sending Covid test data here and there should be easy and seamless. That’s because it’s all supposed to be sent using a standard computer language known as Health Level Seven, or HL7. But one challenge is not every system works immediately with HL7.

“You don’t just turn it on and it works,” said Dan Hall, chief information officer for Clinical Labs. Instead, he said, IT professionals need to integrate it into existing systems for the various labs.

What’s more, Chan said, HL7 has different rules for different data elements, and even a small data entry error can cause a glitch.

“Even the formatting of, say, a date — once in a while for whatever reason, the date becomes invalid, either because of a data entry error or whatever,” he said.

The result, he said, can be the information highway equivalent of a traffic accident during rush hour.

“It could hold up everything,” he said.

Chan wouldn’t say what caused the system backup leading to the Aug. 29 glitch. Neither would health department officials or Hall.

But Chan did provide a schematic diagram to show where in the information pipeline the Aug. 29 glitch happened.

The diagram indicates information from a lab got stuck on its way to another organization, a company called Datahouse, which the schematic calls an aggregator. Datahouse executives did not return a call for comment.

A schematic diagram created by the Hawaii Health Information Exchange shows how data flows from testing labs to the Department of Health. The item marked “B” shows where in the information pipeline a problem occurred that caused a major reporting problem on Aug. 29.

Even with automated systems, Chan said, mistakes can happen.

“Several hundred things can potentially go wrong with every message,” he said.

And as much information as the health exchange has to manage, Chan said, it is relatively easy because it still comes from a relatively small pool of labs.

“What we face is less stressful than what the Department of Health has to deal with,” he said.

Epidemiologists: Use Seven-Day Averages To Make Decisions

When the public goes to the health department’s Covid-19 data dashboard, it sees an array of stats showing latest case counts and seven-day averages, among other things. What the public doesn’t see is what goes into checking the integrity of the data before the agency posts it.

David Johnston is a veteran epidemiologist in charge of the health department’s data dashboard. As Johnston explains it, checking test data before reporting it is far more complex than it might seem.

For example, people who test positive often take multiple tests, and the department needs to make sure it isn’t counting the same person twice – something that can be especially difficult if, say, one test document misspells the person’s name.

“There’s still a lot of manually touching of the data that has to go on,” he said.

All of this might help explain the risks and challenges associated with what might seem like a simple task of reporting positive Covid-19 tests in close to real time. But it doesn’t explain what government officials and the public should do when looking at the numbers.

Joshua Quint, a health department epidemiologist, offered two pieces of advice. One, he said, is for lay people to avoid the temptation of using public health data to try to project or predict where the virus is going – a task that’s vexing even for people with advanced public health degrees.

And, he says, look at seven-day averages instead of data from just a day or two. The averages will smooth out the variations that at times swing wildly.

Green said he appreciates that perspective, but he urged the professionals collecting and reporting data to do better. Seemingly flawed or incomplete data from public health institutions, he said, can fuel conspiracy theorists trying to undermine the credibility of the institutions, he said.

In addition, elected officials often must quickly respond to the concerns of constituents, and it can be hard to get the public to ignore alarming daily numbers and focus on seven-day trends.

“I appreciate what they’re saying, but human nature doesn’t work that way,” he said.