Directorate of

Notes on Analytic

Product Evaluation

June 1995

Note 4


This is the fourth in a series of Product Evaluation Staff notes to clarify the standards used for evaluating DI assessments and to provide tradecraft tips for putting the standards into practice.

This Note on Outlook continues coverage of DI standards for helping policymakers, warfighters, and law enforcers manage uncertainty regarding threats to and opportunities for US security interests. (Please see Note 3, Articulation of Assumptions.)

The outlook sections of DI assessments must address not only what analysts believe to be the most likely course of development but also important alternative outcomes that could confront US officials. This concern for the unlikely or unexpected reflects an experience-based appreciation of (1) the fallibility of analytic judgment, (2) the special analytic requirements of US officials for addressing long-shot threats and opportunities, and (3) the adversarial character of the policymaking process.

The perils of analytic judgment include:

Policymakers and other consumers who carry the burden of executing US policy often have a different attitude than analysts toward unlikely events.

Finally, the outlook on important issues can be subject to deep-seated disagreements - among and between analysts, Administration officials, and congressmen - that generate suspicions about the objectivity of DI memoranda and briefings.

General Rules

As indicated in previous Tradecraft Notes, each DI assessment represents a distinctive challenge to analysts on how to assure credibility with the consumers who count most in planning and executing US policy. The general rules that follow should weigh heavily as analysts choose the most appropriate approach.

1. In all cases, the analysts' judgment on the most likely outcomes must follow clearly from the marshaling of evidence and articulation of assumptions.

2. When the US stake is high (for example, political stability of an important ally or adversary) and plausible alternative outcomes are judged to be in the range of for instance, 20 percent or greater, the outlook section should also structure these potentially important outcomes. Policymakers and other consumers should get enough understanding of these alternatives to decide whether to review contingency plans, ask for more evidence, or levy additional tasks on analysts.

Analysts should identify:

3. On issues vital to US security (for example, military intentions of adversaries), analysts must help policymakers and warfighters engage in contingency planning by:

Tradecraft Tips

1. Analysts should avoid unrealistic precision in citing odds. Weather forecasters may have sufficient data to determine that the chances of rain are slightly better than even. Analysts rarely have such luxury in assessing the political or economic outlook.

2. Also avoid phrases that compound unavoidable uncertainty with unnecessary confusion: for example, real possibility, good chance.

3. As a rule, use constructions that tie the outcome to the driver and linchpin assumptions, rather than flat predictions.

4. To minimize misinterpretation when making important judgments, combine probabilistic phrases and rough numerical odds.

5. To understand the often intense concern of hands-on policy officials about a 20 percent likelihood of a disastrous turn of events happening on their watch, analysts should consider whether they would voluntarily get on an airplane that had a one-in-five chance of crashing.

6. To deal rigorously with unlikely developments, analysts should switch their focus from whether something will happen to how it could happen. Some techniques for generating credible alternative scenarios by structuring the available evidence differently:

Forward to "Note 5"

Return to "Compendium" Contents