Notes on Analytic
This is the fourth in a series of Product Evaluation Staff notes to clarify
the standards used for evaluating DI assessments and to provide tradecraft
tips for putting the standards into practice.
This Note on Outlook continues coverage of DI standards for helping
policymakers, warfighters, and law enforcers manage uncertainty regarding
threats to and opportunities for US security interests. (Please see Note
3, Articulation of Assumptions.)
The outlook sections of DI assessments must address not only what analysts
believe to be the most likely course of development but also important alternative
outcomes that could confront US officials. This concern for the unlikely
or unexpected reflects an experience-based appreciation of (1) the fallibility
of analytic judgment, (2) the special analytic requirements of US officials
for addressing long-shot threats and opportunities, and (3) the adversarial
character of the policymaking process.
The perils of analytic judgment include:
- Limitations in the evidence and in the analysts' understanding of complex
issues that can lead to judgments structured around either the wrong drivers
(key variables) or incorrect linchpin assumptions.
- The impact of essentially unpredictable events on sound analytic judgments;
for example, changes in political stability caused by assassination of
a president or the sudden advent of a charismatic opposition leader.
Policymakers and other consumers who carry the burden of executing US
policy often have a different attitude than analysts toward unlikely events.
- Officials who are charged with advancing a long-shot objective may
probably see a 20 percent likelihood of success as promising opening odds
for arranging a negotiated settlement of a longstanding civil war or diverting
an entrenched nuclear weapons program.
- Officials also require an understanding of unlikely developments to
prepare contingency plans against low-probability, high-impact dangers
(such as protecting Persian Gulf oil suppliers from another Iraqi military
attack on Kuwait or an Iranian blockade of the Straits of Hommuz).
Finally, the outlook on important issues can be subject to deep-seated
disagreements - among and between analysts, Administration officials, and
congressmen - that generate suspicions about the objectivity of DI memoranda
- At such times, DI credibility can depend more on explanation of why
the outlook - for example, for Cuba after Castro or for Russia after the
next election cannot be predicted with high confidence than in an attempt
to trivialize uncertainty by "just making the call."
As indicated in previous Tradecraft Notes, each DI assessment represents
a distinctive challenge to analysts on how to assure credibility with the
consumers who count most in planning and executing US policy. The general
rules that follow should weigh heavily as analysts choose the most appropriate
1. In all cases, the analysts' judgment on the most likely outcomes must
follow clearly from the marshaling of evidence and articulation of assumptions.
- The more complex and controversial the issue, the more analytic effort
is required to ensure that drivers and linchpin assumptions are precisely
stated and well defended.
2. When the US stake is high (for example, political stability of an
important ally or adversary) and plausible alternative outcomes are judged
to be in the range of for instance, 20 percent or greater, the outlook section
should also structure these potentially important outcomes. Policymakers
and other consumers should get enough understanding of these alternatives
to decide whether to review contingency plans, ask for more evidence, or
levy additional tasks on analysts.
Analysts should identify:
- The factors at play that could lead to unexpected developments.
What alternative key variables could drive the outcome in a different direction
(perhaps the economy, or growing corruption)? Or what plausible reversal
of linchpin assumptions about correctly defined drivers could turn things
around (perhaps the Egyptian military will not remain loyal to the Mubarak
- Signposts or indicators of change. What intermediate developments
would indicate that events may not be working out as originally expected?
For example, such negative signposts for regime stability as the advent
of coup plotting or capital flight.
- Events that could trigger a major shift in direction. What essentially
unpredictable events could precipitate the unexpected outcome? For example,
a crop failure, or a mishandled natural disaster, or anti-government demonstrations
that get out of control.
3. On issues vital to US security (for example, military intentions of
adversaries), analysts must help policymakers and warfighters engage in
contingency planning by:
- Addressing all important factors at play that could influence subsequent
- Ranking their relative importance.
- Relating them to a range of plausible outcomes, including alternatives
analysts deem to be remote possibilities against which policy planners
may nonetheless decide to deploy resources.
1. Analysts should avoid unrealistic precision in citing odds. Weather
forecasters may have sufficient data to determine that the chances of rain
are slightly better than even. Analysts rarely have such luxury in
assessing the political or economic outlook.
2. Also avoid phrases that compound unavoidable uncertainty with unnecessary
confusion: for example, real possibility, good chance.
3. As a rule, use constructions that tie the outcome to the driver and
linchpin assumptions, rather than flat predictions.
- Castro is unlikely to be forced out of office over the next six
months unless his security forces mishandle disturbances and trigger prolonged
4. To minimize misinterpretation when making important judgments, combine
probabilistic phrases and rough numerical odds.
- On balance, Castro is unlikely to be forced from office in the next
six months (less than a one-in-five chance).
5. To understand the often intense concern of hands-on policy officials
about a 20 percent likelihood of a disastrous turn of events happening on
their watch, analysts should consider whether they would voluntarily get
on an airplane that had a one-in-five chance of crashing.
6. To deal rigorously with unlikely developments, analysts should switch
their focus from whether something will happen to how it could
happen. Some techniques for generating credible alternative scenarios by
structuring the available evidence differently:
- Who and What Counts. Generate lists of the key individual and
group players that can determine the outcome of an issue (for example,
whether Egyptian politics will continue to simmer or will explode), and
critically examine the factors most likely to determine their behavior
(such as institutional loyalties, personal rivalries, family interests).
- Thinking Backwards. Start with the assumption that the targeted
danger or opportunity has occurred. For example, revolutionary violence
undermines the Castro government! Then generate one or more plausible "how-it-could-happen"
scenarios, by postulating who would have to do what, when, how, and why.
- Judgments Discarded. List assumptions that fell by the wayside
in reaching the bottom line. Regarding the judgment that the military will
continue supporting the Egyptian government, for example, revisit any available
data or supportive reasoning indicating such support may be weakening,
for example, among junior officers.
Forward to "Note 5"
Return to "Compendium" Contents