Pages Navigation Menu
Advertisement
advertisement

Stomping cognitive bias

How technology could help

Imagine you’re the intelligence analyst of the future and you think you’ve just solved a vexing international puzzle for your leaders, when suddenly a gentle voice rises from your computer: “Excuse me Dave, other explanations with more weight are available. Would you like to see them?”

This kind of ghost in the machine is called a software agent. Technologists are almost certain to avoid any similarities in their designs to H.A.L, the all powerful computer in “2001: A Space Odyssey.” But software agents are, in fact, a serious idea for mitigating cognitive biases – the brain’s natural tendency to find shortcuts through complex information.

  • Are video games the answer?

    The Intelligence Advanced Research Projects Activity thinks specialized video games could help student analysts and career professionals root out their cognitive biases.

    To find out, IARPA has launched a program called Sirius. The name is a play on the term serious games, which refers to games with goals beyond entertainment.

    IARPA says “cognitive biases are notoriously resistant to training,” and it wants to see if video games are the answer.

    Cognitive biases shouldn’t be confused with prejudices or stereotypes, brain experts say. They’re the shortcuts that our minds naturally seek when we’re presented with overwhelming data.

    In September, IARPA chose five teams to build competing games. By next June, analysts could be trying them out along with college students. Those tests will have to show a 50 percent reduction in bias for the developers to proceed any further.

    IARPA won’t say how much the games will cost, but last November, Raytheon BBN Technologies placed the potential value of its multiyear contract at $10.5 million. The company was not selected for the new phase that began last month.

    The remaining teams are led by:

    Applied Research Associates

    Boeing

    SET Corporation (Now part of SAIC)

    State University of New York at Albany

    University of Oklahoma


Intelligence managers took on the bias problem after the Sept. 11 terror attacks, and redoubled their efforts after the wildly wrong 2002 national intelligence estimate about Iraq’s weapons of mass destruction. For the most part, agencies addressed bias by training analysts to use new structured analytic techniques – software is now used to chart and visualize evidence, arguments and logic.

It’s hard to train away all cognitive biases, though, because the human mind innately applies mental shortcuts when faced with big data dumps. The mind falls back easily on previous experiences, for example, which can lead to confirmation bias. Cognitive biases should not be confused with stereotypes or prejudices. They’re much more subtle, cognitive experts say.

Scientists such as those at Raytheon think the time is right for science and technology to play a larger role in combating bias – but with limits. If software agents were to be deployed – one researcher says their use in the community is “incipient” at this point – analysts would have to retain the power to take or leave the computer’s advice.

“The analyst’s reaction to a tool like that would depend on how intrusive it is -- whether it’s Big Brother,” says former intelligence analyst James Steiner, who retired from the CIA in 2005 and teaches at Rockefeller College, an arm of the State University of New York at Albany.

This article is based on interviews with cognitive researchers, former analysts and industry experts. It shows that the community has options it has yet to exercise. Software agents are just one. Experts applaud the structured analytics trend but they decry the lack of quantitative evidence about their effectiveness in different scenarios. What if information technology could be applied to monitor analytical strategies and outcomes over time? Some of the structured techniques have yet to be institutionalized out of concern that they could overburden time-pressed analysts who find old-fashioned intuition to be a lot faster.

Raytheon is emerging as the most outspoken advocate of applying information technology and experimentation to the cognitive bias problem. The company wants to monitor the decisions of analysts in lab exercises, during training, and eventually in real time while they’re working. The monitoring could help the community assess which techniques work best, and give analysts instant feedback so they can avoid mistakes, rather than simply leaving an audit trail for investigators.

“Awareness alone isn’t enough to overcome (bias) – this is what research has shown. You have to have some way to intervene when the bias is emerging,” says Raytheon researcher and PhD student Don Kretz, who studies cognitive bias in complex problem solving scenarios.

Kretz’s words echo those in a prepared statement from IARPA, the Intelligence Advanced Research Projects Activity: “Research to date has found that simply knowing about the bias is not sufficient to help the individual avoid the bias.”

IARPA wants to see if specialized video games can improve training.

Raytheon wants to do even more. Installation of software agents is one idea.

Rand’s Gregory Treverton, who oversaw the drafting of national intelligence estimates from 1993 to 19995, says software agents might actually empower analysts if they are implemented in a non-heavy handed way.

“One of the things that people learn best from, the psychologists tell us, is feedback. And the closer it is to the event the better,” he says.

GAME TIME

Last June, 27 Raytheon employees, mostly from the engineering ranks, volunteered to participate in a one-day intelligence analysis exercise. The purpose was to show how elaborate games might be used to compare different structured analytic techniques for effectiveness. Raytheon plans to release the paper Nov. 15 at the IEEE Homeland Security Technologies conference in Waltham, Mass.

The company hopes the exercise and a similar one in August will prompt its mission partners – meaning intelligence agencies – to assign working analysts to participate in one or more exercises late this year or early next. Talks are underway, Kretz says.

In the June pilot, the volunteers were given fictional intelligence reports, cell call transcripts, and embassy reports describing three bombings near the Green Zone in Baghdad. The information was drawn from a library of synthetic counterinsurgency, or SYNCOIN, scenarios developed by Penn State University with funds from the Army Research Office. Participants had to figure out who planted the bombs and who the intended targets were.

The game designers made things difficult for the participants by including facts intended to draw out anchoring biases – the tendency of human minds to glom onto evidence supporting an initial hypothesis – and confirmation biases, the tendency to put more weight on evidence that confirms an underlying belief.

The participants were briefed on three  techniques for "debiasing" their analysis and given simple software tools. Raytheon calls these techniques analytic multipliers.

Some participants used link analysis software to chart the relationships among people, organizations, objects and events.

Others used an information extraction and weighting technique to assign numbers from 1 to 10 to specific pieces of evidence based on how critical they saw them – 10 being the most critical. The evidence and numbers were typed into spread sheets.

Others used software to draw up matrices documenting the competing explanations and the evidence supporting them. That technique is called Analysis of Competing Hypotheses or ACH.

EARLY RESULTS

How did the comparison turn out? The first thing Raytheon wants you to know is that picking a winner was not the main point. The purpose was to develop an experimental protocol to show how the community could, over time, assemble quantitative information to document the effectiveness of different techniques.

For CIA-funded researcher Richards Heuer, it’s about time someone gathered performance data about the structured techniques he advocates: “This is exactly what we need more of -- empirical testing and documentation of the effectiveness or the ineffectiveness of various analytic tools and a better understanding of which tools are best used for which type of problem assembled over time,” he said by email.

Raytheon has discussed its research with Heuer, but he is not affiliated with the Raytheon team.

To the degree we should care about the result, it was good news for advocates of ACH, a 30-year old structured technique developed by Heuer.

“The Analysis of Competing Hypotheses technique did in fact improve judgment the most,” says Kretz who is one of the paper’s authors and is scheduled to speak at the IEEE conference.

Heuer came up with ACH in the 1980s after retiring from the CIA and continuing his research as a contractor to the agency. His goal was to improve analysis of the Soviet Union:

“When analysts were asked if they thought the Soviets were deliberately trying to deceive us on a specific point, they would almost invariably say ‘No’, because they didn’t see any evidence of it,” Heuer recalls. “What they didn’t recognize was that if the deception was being done well, they shouldn’t expect to see evidence of it.”

ACH helped the CIA sort deception from reality. For about 15 years, that’s the only reason it was taught and used, says Heuer.

Then came the Sept. 11 attacks and the Iraq weapons fiasco. The agencies took a fresh look at ACH and other structured techniques. In 2010, Heuer and Randolph Pherson authored an unclassified book, “Structured Analytic Techniques for Intelligence Analysis,” which explained how the techniques could help analysts in intelligence, law enforcement and even the business world.

Heuer cautions that ACH is not a “cure-all” for all analysis problems even though it shined in the Raytheon exercise.

Some researchers suspect ACH has the most impact for novice or new analysts, which could explain why it came out on top among engineers pretending to be analysts.

If Raytheon convinces intelligence agencies to play, the June exercise could go down as one of the turning points in the fight against bias. The company insists its advocacy is not about winning the next big contract or selling the agencies more software.

“They’ve got shelves and shelves full of tools that they’ve bought and paid for that they don’t use,” Kretz says. “We’re not trying to develop more stuff like that. We want them to be partners in this.”

2 Comments

  1. This is an excellent article, but I need to point our one error. CIA did NOT fund the book by Randy Pherson and myself, “Structured Analytic Techniques for Intelligence Analysis.” As with all publications by former CIA employees, CIA screened the contents to be sure there were no security violations.

    • Editor’s note: We’ve fixed this error.