At Tulsi Gabbard‘s celebration Saturday night, the accolades came in a steady stream: “How ’bout that Civil Beat Poll?”

Five miles away, at Mazie Hirono‘s party, the same question was asked more derisively: “How ’bout that Civil Beat Poll?”

We were baffled by the turn of events. We nailed Gabbard’s big win over Mufi Hannemann right on the nose, but our otherwise sterling election survey record was tarnished with Hirono’s decisive victory over Ed Case in the Democratic primary for U.S. Senate.

Ten days before the vote, our poll showed Gabbard up 20 points and she ended up winning by 20.4 percent — not too shabby. In the same poll, Case led Hirono, 47-46, but Hirono went on to win by nearly 17 percent. We were way, way off on that one.

To be clear, the pre-election projections all came from the same voter survey, even though they were unrolled in separate articles last week. In other words, the people who said they’d vote for Gabbard and then did were in many cases the exact same people who said they’d vote for Case — and then didn’t.

Across town, the Honolulu Star-Advertiser- and Hawaii News Now-sponsored Hawaii Poll was right where we were wrong, and wrong where we were right. In their survey taken in mid-July, the Hawaii Poll showed Hirono leading 55-37 and Hannemann up 43-33.

So, what’s up with these polls?

Civil Beat’s Sample

In our pre-election poll articles, we outlined characteristics of our sample that might explain the glaring differences with the Hawaii Poll. Here’s a brief summary:

  • The Hawaii Poll had a 67-33 Oahu-to-neighbor-island ratio. We had a 70-30 split.
  • The Hawaii Poll had 51 percent of its sample aged 55 or older. We had 81 percent of our sample aged 50 or older.
  • The Hawaii Poll had 26 percent Caucasian, 35 percent Japanese, 18 percent Filipino and 21 percent Hawaiian. Our split was 37 percent Caucasian, 26 percent Japanese, 8 percent Filipino and 10 percent Hawaiian.
  • The Hawaii Poll had 72 percent Democrats, 9 percent Republicans and 18 percent independents. We had 59 percent Democrats, 8 percent Republicans and 27 percent independents.
  • The Hawaii Poll was conducted between July 12 and July 21. Ours was done between July 31 and Aug. 2, meaning we had a chance to catch any late momentum before Election Day.

Our split between the neighbor islands was spot on. Election results show it was 70.7 percent Oahu and 29.3 percent neighbor islands. But we don’t have breakdowns yet of voter demographics so we don’t know if, for instance, our sample was too old, too white or relied too much on non-Democrats voting in the Democratic primary.

We’re not trying to figure out what happened with the Hawaii Poll. The big question for us: How could we be so right with the CD2 results but so wrong with the Senate if it was the same people answering both questions? If our sample was wrong and the Hawaii Poll sample was right, then you’d think our results would have been wrong across the board and the Hawaii Poll would have been more accurate. But that’s not what happened.

On top of the mixed results on the federal races, the two polls’ projections in the Honolulu mayor’s race were a wash.

The Hawaii Poll accurately predicted that Ben Cayetano would fall short of the 50 percent threshold he needed to avoid a runoff, pegging his support at 44 percent. We had it at 51 percent. The final results had Cayetano receiving 44.7 percent of votes cast in the contest.

But we accurately predicted Kirk Caldwell would beat Mayor Peter Carlisle for second place. The Hawaii Poll had Carlisle with a narrow lead for second place.

Contest Hawaii Poll Civil Beat Poll Actual Result
U.S. Senate Hirono 55,
Case 37
Case 47,
Hirono 46
Hirono 58,
Case 41
U.S. Congress Hannemann 43,
Gabbard 33
Gabbard 49,
Hannemann 29
Gabbard 55,
Hannemann 34
Honolulu Mayor Cayetano 44,
Carlisle 27,
Caldwell 25
Cayetano 51,
Caldwell 24,
Carlisle 19
Cayetano 45,
Caldwell 29,
Carlisle 25

‘Social Desirability Bias’

The day after the primary, we began long conversations with our pollsters, Matt Fitch and Seth Rosenthal of the Merriman River Group.1 While we design the poll, write the questions and explain the results, they are longtime polling experts who have the technology and know-how to carry it out for us.

We also called Rebecca Ward from Ward Research, which did the polling for the Star-Advertiser and Hawaii News Now, as well as academics on the mainland to see what they might offer in the way of explaining what went wrong for us.

First, it’s important to know that we use interactive voice response (IVR) technology, also called touch-tone polling or, to critics, “robo-polling.” There are pros and cons to robo-polls, and one of the purported benefits is that using pre-recorded prompts is supposed to eliminate any bias issues created by the human being asking the questions.

In this case, that might not have worked perfectly.

That’s because the voice used in Civil Beat’s poll might have inadvertently created what’s known in the industry as a “social desirability bias” where poll respondents feel pressured to give answers they believe are socially acceptable.

“Are we not giving them what they need to feel comfortable?” said Matt Fitch, MRG’s executive director. “Because that’s what you need to have a good poll result: to have people in a place of comfort.”

Two academics who have studied biased responses say a pollster’s voice can make a difference.

“Absolutely, socially desirable biases do impact the way people respond to surveys,” said Sharon Shavitt, a research professor of marketing and psychology at the University of Illinois. “It doesn’t necessarily matter if it’s an actual person on the other end of the line or a recorded voice or typing something into a computer.”

Shavitt and two other researchers produced a paper in 2006 exploring how higher levels of “collectivism” — groupthink — lead to more socially desirable responses. One of those other researchers was Timothy Johnson, director of the Survey Research Laboratory at the University of Illinois at Chicago.

“People want to present themselves in a positive light as being open-minded,” Johnson said. “There’s no cost to them to endorsing someone who they don’t intend to vote for, and there’s an advantage there that they’re being open-minded.”

Culture plays a role. Shavitt said common ideals for those of European ancestry include independence and self-reliance, while those with Asian ancestry tend to strive to be “normative, appropriate, fitting into what’s expected.” They seek to “fit in, blend in, not to step over lines or do anything unusual relative to norms or frowned upon.”

“That impact could be seen more strongly among Asian respondents than among Caucasians,” Shavitt said. “It’s entirely plausible.”

Because Asian-Americans are such a large part of Hawaii’s population and culture, voters here might be particularly susceptible to the social desirability bias. That means Hawaii voters might tend to give the answer they think the pollster wants, instead of their real feeling.

But the bias only comes into play if the survey somehow creates the impression of a socially preferable response. In the case of The Civil Beat Poll, the voice used in all our surveys so far — a Caucasian woman in Ohio — could subconsciously cue respondents to opt for Caucasian candidates, like Case, over Japanese-American candidates like Hirono.

“It kind of depends on the pull of the situation,” Shavitt said. “One possibility is that they’re going to be more responsive to the supposed race or ethnicity of the communicator. Another possibility is that they’ll be more responsive to what they perceive to be the community’s norms.”

Ward said her firm uses 12 to 15 men and women at a time to make calls to voters.

“Their ability to pronounce names correctly is more important than the sound of their voice,” she said. “That they can pronounce Hanabusa correctly or Inouye correctly regardless of where they come form, I think that’s the most important consideration.”

Ward also uses some callers who are able to speak Tagalog or Ilocano so they can communicate with Filipino voters, but she said that’s pretty rare.

“We want them to feel comfortable, but the interviewers are discouraged from being chatty. They’re to follow the script,” Ward said. “I think professional is more important than the respondent feeling comfortable, but at the same time if the voice is pleasant … I’ve listened to our most productive callers, the voice is engaging, it’s a little energetic, it’s not just reading words on a screen. You can tell who connects well.”

Ward was skeptical that using one interviewer’s voice rather than another could cue respondents to favor one candidate over another. Instead, she said, it’s possible using a particular voice — one from the Deep South, for example — could discourage certain potential respondents from completing the poll at all, further skewing the demographics.

“It’s not what they answer, but whether or not they answer,” she said. “People make a decision in the first few seconds, ‘Am I going to invest the time in this or not?'”

Johnson said the discrepancy between Civil Beat’s Case-Hirono prediction and Hirono’s convincing victory less than two weeks later reminded him of the 1989 gubernatorial election in Virginia when an African-American candidate underperformed his poll numbers on Election Day.

The contest was studied in a paper published in The Public Opinion Quarterly and later gave rise to a phenomenon known as the “Wilder effect.”

The bias, Johnson said, always worked in the same direction: the pre-election polling would show too much support for the minority-race candidate. Because Hawaii is a majority-minority state where Caucasians make up less than half of the population, the bias would tend to overstate support for Caucasian candidates like Case.

“It certainly sounds like it’s a reverse of that, where the minority candidate was a Caucasian, and especially when the respondents are being cued by a Caucasian-sounding midwestern voice on the phone,” Johnson said. “My gut feeling about what I’ve been seeing around the world is that’s the minority culture within any particular area. People tend to behave a little differently when they’re in the minority than when they’re in the majority position.

“It really depends on where you are, where your culture is positioned within the society you’re living.”

Japanese-Americans and Election Polls

That’s where the Japanese-American factor comes in. Japanese-Americans hold a special place in Hawaii culture, from plantation days to Pearl Harbor and the 442nd and internment camps to the Democratic Revolution of 1954. The social desirability bias might be a particularly strong factor when it comes to races between Caucasians and Japanese-American candidates.

The Hirono-Case matchup was the first high-profile mistake in our polling history, but we looked back at previous polls and found two prior under-the-radar results that point to a potential pattern.

In the 2010 general election, Sen. Daniel K. Inouye won his ninth term by beating Republican Cam Cavasso, 75 percent to 22 percent. We had the numbers at 63-31.

That same year, Hirono won her third term in Congress by beating Republican John Willoughby, 72 percent to 25 percent. Less than two weeks earlier, we had polled Hirono’s contest at 59-35.

Cavasso and Willoughy, both Caucasians, significantly underperformed versus their pre-election polling numbers, just like Case in the 2012 primary.

There are two key differences between those 2010 contests and this one in 2012. First, Tea Party Republicans were enthusiastic and likely to respond to polls in 2010 across the country, but didn’t always follow through by voting. Second, Cavasso and Willoughby are not Ed Case, a former congressman. So those contests were blowouts, and blowouts are notoriously difficult to poll.

“From the person who’s getting crushed, all the money gets pulled. They do no get-out-the-vote, no advertising, nothing,” said Rosenthal, MRG’s director of polling. “In the final days in a blowout, the candidate who’s eventually going to win is out there doing everything they do, and the other candidate disappears. Blowouts usually become even more of a blowout in the end.”

Still, there’s a pattern of relative inaccuracy when it comes to contests between Caucasians and Japanese-Americans, and the issue appears to be confined to those contests.

Despite complicated Caucasian-Hawaiian, Caucasian-Chinese and Caucasian-Filipino relations, social desirability bias has yet to emerge in Civil Beat polls on contests involving those ethnic dynamics.

In particular, Civil Beat accurately predicted Colleen Hanabusa (a Japanese-American) would come in ahead of Case when Chinese- and Thai-American Charles Djou won a special election in May 2010. Civil Beat also predicted Caucasian Neil Abercrombie would beat Mufi Hannemann, who is of Samoan and German descent, in the 2010 Democratic primary for governor.

“That’s why the social desirability bias makes as much sense as anything else we can come up with,” Rosenthal said. “We have the same basic distribution of Democrats and independents and Republicans, and it’s always worked before.”

In the end, though, it’s unlikely that the Japanese-American issue alone is creating the error in Civil Beat’s polling results. Instead, that’s a factor that combines with other factors — like the sampling issues above — to lead to problems.

“It’s probably not one thing,” Fitch said. “It’s my instinct and your instinct to find the one thing that was wrong with it. But it’s not one thing.”

As an example, a higher percentage of independents in Civil Beat’s survey means more voters who are less committed to a particular candidate. Those “squishy” voters could be more susceptible to subconscious cues from the pollster’s voice. They’d also tend to be more in flux, and more likely to be swayed by Hirono’s superior last-minute get-out-the-vote efforts.

Hirono V. Lingle

The attempt to figure out what happened in Civil Beat’s pre-primary poll is more than an academic exercise, however, because the general election is just around the corner. And its key matchup, between Hirono and Republican Linda Lingle, again pits a Japanese-American candidate against a Caucasian.

For the next poll, we’re thinking of using two different voices — one mainland Caucasian and one local — to see if it makes a difference. Both Shavitt and Johnson said that experiment could shine light on the impacts of social desirability bias.

“I’m not sure how readily people can connect voices with ethnicities,” Shavitt said. “Especially people born and raised in Hawaii, it would be hard to discern their race or ethnicity from their voice.”

The voice we’ve used up until now was chosen because, Fitch said, “she’s kind of devoid of an accent.”

“You can’t tell she’s from the Midwest. She can sound midwestern, she can sound Southern, she can sound Eastern, she can sound Californian,” he said. “She could probably even sound like she’s from Hawaii.”

Take The Voice-Recognition Quiz

As an experiment, we recorded a half-dozen women, some from Hawaii and some not, reading the 15-second introductory paragraph to our most recent survey.

Can you identify which are “local” and which are not?

Follow Civil Beat on Facebook and Twitter. You can also sign up for Civil Beat’s free daily newsletter.