“Basement level — very ‘X-Files.’”

That’s how Keith DeMello, senior communications manager at the state Office of Information Management and Technology, describes the offices in the basement of the Kalanimoku Building in downtown Honolulu. 

But the offices are more mold than Mulder, and the team is tasked with a feat far more pressing than solving cases of the paranormal. 

When I arrived in the Kalanimoku basement, I did something I almost always do: I got lost. The walls seemed damp. The white hallways lengthened and constricted. I looked in my pocket for a Xanax or a Prilosec, but nothing. 

Last February, Gov. Neil Abercrombie, center, announces promotions of Sonny Bhagowalia, second from right, who would soon depart for Washington, D.C., and Keone Kali, right.

State of Hawaii

To my surprise, very few doors were locked. I looked longingly at state employees with my “I’m lost” face. Nothing. I pressed what looked like a red DANGER button just to see what security protocols were in place. Surely, I’d have to solve a puzzle to pass. Or maybe a kind-hearted nymph (or state senator) would assist me to my desired location. 

An intercom erupted: “Come in. Door’s not locked.” 

Two Worlds

When I finally did arrive at the humble entrance to OIMT, I pressed another red DANGER button and watched as a small, eyeball-shaped camera scrolled slowly and loudly upwards, scanning my face. 

“I’m here to interview Keone Kali,” I said.

Another intercom: “Come in, and please sign the guest book on the front table.” 

A strange juxtaposition, I thought. Step 1: A little eyeball camera scans my face, records me, notates my human presence, saves my likeness in a cloud database somewhere, probably knows the time I arrived at the door and the time I lost my virginity. Then a quaint step 2: I use an ink pen to write the date, my name, and any of my feelings about my short stay at the local B&B.

Thus, before the interview even began, I had already unearthed what can only be described as the state’s central struggle to straddle two competing and very contradictory worlds. One is cutting-edge and digital, fast, ever-changing, adaptive. The other is entrenched, slow, chock-full of paper and pens, familiar, with music from “Little House on the Prairie” whistling in the background.

Kali’s Basement Purgatory

Keone Kali, the CIO of Hawaii, oversees all of the state’s IT operations. He’s tasked with renovating Hawaii’s crumbling 40-year-old IT infrastructure.

Some of those systems were built before Kali, age 43, took his first baby steps. And many of those systems, some crucial to the state’s functioning, sit in an underground room, in a flood zone, with no backups, running on ancient and irreplaceable machines, on the same floor and only feet away from Kali’s dungeon-like office.

It’s a tenuous situation that could go awry at any minute, and Kali literally sits in the middle of it all.

One might assume Kali received the misfortune of his appointment as a result of some kind of “Hunger Games” lottery selection. Or that he’s simply a glutton for binary punishment.

I caught my first glimpse of Kali while I was signing the guest book and clipping a Visitor badge to my shirt. He’s tall (almost too tall for the low basement ceilings), iridescent (possibly a side effect of the lighting), with a hefty ponytail and a calming smile. He looks right at home in his well-worn Aloha shirt — an accomplishment Hawaii’s first CIO, Sonny Bhagowalia, a D.C. transplant, never exactly mastered.

Kali’s Appointment, Sonny’s Retreat

Then-Gov. Neil Abercrombie promoted Kali to CIO in March after Sonny Bhagowalia retreated back to the comforts of Washington D.C. Kali had previously served under Bhagowalia as Deputy CIO of Operations.

Bhagowalia now serves as the CIO of the U.S. Department of the Treasury, leaving Kali with the daunting and somewhat terrifying task of achieving the 12-year transformation plan that Bhagowalia set out four years ago. It’s an aggressive 1,300-page plan that touches on all aspects of renovating the state’s tech infrastructure, and Kali is taking over just as the project shifts from research to the more difficult execution phase.

That transition — from research to execution — is aptly paralleled in the personalities of Bhagowalia and Kali.

Bhagowalia is a “big picture” thinker. A pie-in-the-sky kind of guy. He spent his first 18 months as CIO working almost exclusively on polishing the epic tome that is the 12-year transformation plan. He’s eloquent, forward-thinking, likeable, and garners heaps of positive PR wherever he turns. He makes Oprah-sized promises, like offering “1 Gigabit internet” to everyone in Hawaii — a promise that, according to Kali, is unfortunately unlikely to materialize any time soon.

Bhagowalia is more of a dreamer than a doer, and he conveniently left the show before any real elbow grease was needed.

Kali, in contrast, strikes me as more of a hands-on, no-bull workhorse. He lacks the suave political chops of Bhagowalia, but he’s more focused on prescribing explicit goals and carving out well-defined roles for the OIMT staff — something sorely lacking under the leadership of Bhagowalia, according to some.

After all, it’s Kali who faces the backlash if he doesn’t convert Bhagowalia’s grand vision into a slightly-less-grand reality.

But the task is a monstrous one. And Bhagowalia, though high-minded, didn’t exactly leave the state with a glaringly obvious path to success.

For starters, the data centers are still decentralized and poorly maintained. And different state organizations, particularly OIMT and the Information and Communication Services Division (ICSD), which was the state’s central IT organization prior to the creation of OIMT, have clashed over roles and responsibilities, stifling potential progress. And many OIMT employees hired by Bhagowalia touted vague, executive-ish titles, with no explicit goals attached to their tenure. That’s a little like toting a bench full of competing coaches and calling it a basketball team.

Couple that internal discord with a core group of state employees who have experienced little to no change in their work processes for the last 40 years. They’re masters of arcane systems, but masters nonetheless. And the longer the arcane systems remain intact, the more resistant to change the state culture has become.

It’s a nearly untenable problem, alas, and a Henry-James-length strategy document can only propel Kali so far.

Shaking Things Up

As Randy Baldemor, director of strategic initiatives, jumped on a plane to D.C. to congratulate his former boss Bhagowalia on his new position, Kali, according to one witness, “walked into the Enterprise Project Management Office and dismissed [Baldemor’s] entire staff, giving them one hour to pack their stuff and clear out.”

The Enterprise Project Management Office had been overseeing the state’s multi-million-dollar Business Transformation Initiative, but it seems that Kali has a slight distaste for the art of project management and project administration, especially when budgets are tight and not much tangible work is getting done. The brainstorming of methodologies and standards rarely moves mountains or replaces 40-year-old VAX machines.

That was one of Kali’s first acts as the CIO of Hawaii, and a controversial one at that, as Kali’s vision for the execution of the strategic plan required less managerial overhead (something Bhagowalia seemed overly fond of) and more hands-on support and resources.

“We need to make programmatic changes to ensure that we have the technical expertise to run technology projects,” Kali said. “We need the software and architecture to get installed. We need a lot more hands-on, high-tech skillsets for what we’re doing next.”

Merging of OIMT and ICSD

Another initial and potentially controversial step in Kali’s execution of the IT plan is the reorganization of OIMT and ICSD. Kali is proposing to merge the two governmental groups into a single entity. Acronyms, beware.

“We’re reorganizing our central function in terms of the people, including the functions of OIMT and ICSD,” he said. “We’re submitting our reorganization plan to get an entity created that will help us through the next couple of years.”

Implied in that reorganization is an existing organizational structure that parallels the state’s dilapidated and decentralized IT infrastructure: two governmental organizations competing for control and resources, possibly at odds with each other when they should ideally be working in unison, thus limiting the effectiveness of both groups and jeopardizing the longer term goals set forth in the IT plan.

ICSD, it seems, has had some trouble falling in line behind the initiatives of the younger OIMT, even if mandated to do so.

Kali’s negotiating of the merger with Hawaii Government Employees Association (HGEA) is likely a crucial first step toward unifying the state’s IT organizations and objectives, and will likely be the determining factor in either the success or failure of many of the aggressive goals set forth in Bhagowalia’s IT plan.

Two Years to System Meltdown?

What do Hawaii’s accounting system, inventory management system, tax system, and health care entitlement system all have in common?

They run on legacy machines right outside Keoni’s basement office. And if any of those machines were to get wet or break down for any reason, the data would be irretrievable. The state would be in disarray.

Even the major radio infrastructure for radio communications and public safety, systems which also reside on legacy machines in the same underground server room, are extremely fragile and susceptible to failure. If that equipment were to get wet during a hurricane or tsunami (the server room is in a flood zone, after all), the state’s emergency response would inevitably be stifled.

“The problem with the legacy environments is that the backups can only be backed up on the legacy equipment, in this building,” Kali said, “If you take the backup tapes offsite, and this facility becomes flooded, and you take the backup tapes back, you can’t restore the backups. So if this facility is disabled by water that ruins the legacy hardware, it’s basically done.”

With or without a natural disaster, most of these systems are likely to fail catastrophically sooner than later.

“In a lot of cases we have systems that are failing or that will completely fail within two years,” Kali said.

It’s a do-or-die situation, and the newly elected Gov. David Ige should take it very seriously.

“There’s kind of a failure window here,” Kali said in an interview with the Honolulu Star-Advertiser. “We’re so far beyond the end of life for some of these things that I give it two years before it’s irrecoverable data loss.”

Kali’s nose-to-the-grindstone attitude makes sense, then, especially given the urgency to execute before essential systems start to break down.

Interim Solution: State’s Private Cloud

On a limited budget and with limited resources, Kali’s team is working hard to move as much state data as possible to the state’s new private cloud.

“We have a cloud-first policy,” Kali said. “It’s secure. It’s a virtualized environment, and it’s available to all of the departments. Departments are lined up to use it. They’re no longer purchasing any server infrastructure.”

That’s a first step, and the impact is immediate and tangible. But legacy systems often can’t be moved to the cloud so easily, including most of the core systems residing in the basement of the Kalanimoku Building.

So, while new systems are built to replace the old — a process that could take many years — Kali is also working on opening a new data center, possibly in the Mililani area, where the older systems can reside and, hopefully one day, retire.

In the meantime, let’s all pray for calm waters. And, for now, let’s never forget to sign the guestbook on our way out.

About the Author