Directorate of
Intelligence

Notes on Analytic
Tradecraft

Product Evaluation
Staff


    July 1995
 


Note 5

 
 

Facts and Sourcing

 
 


This is the fifth in a series of Product Evaluation Staff notes to clarify the standards used for evaluating DI assessments and to provide tradecraft tips for putting the standards into practice.

The present note is longer than those previously published because it addresses two closely related evaluation standards.


The credibility of DI assessments with key consumers starts with the analyst's skill in organizing and evaluating the information presented in support of policymaking, warfighting, and law enforcement. Estimative judgments are also important, and at times essential for addressing complex and uncertain issues. But extensive feedback makes clear that US officials with hands-on responsibility for planning and executing policy take their first measure of DI analysts in terms of the judgment they exercise in providing actionable and reliable information:

This tradecraft note, on Facts and Sourcing, presents DI guidelines for depiction in memorandums and briefings of what the DI knows that is, the character and source of its information. The guidelines will be illustrated by hypothetical examples of "best practices" for handling recurring challenges in characterizing information for example, addressing the reported attitudes and intentions of foreign actors.


Definitions

DI analysts have a good track record in conveying what they know clearly and credibly to consumers of intelligence. What is undertaken here is codification of rules of thumb into general guidelines.

The following definitions are an attempt to promote standardized use of common terms and concepts relating to intelligence information.

Fact: Verified information; something known to exist or to have happened.

Information: The content of reports, research, and analytic reflection on an intelligence issue that helps analysts and their consumers evaluate the likelihood that something is factual and thereby reduces uncertainty.

Direct Information: Information relating to an intelligence issue under scrutiny the details of which can, as a rule, be considered factual, because of the nature of the source, the source's direct access to the information, and the concrete and readily verifiable character of the contents. For example:

Indirect Information: Information relating to an intelligence issue the details of which may or may not be factual, the doubt reflecting some combination of the source's questionable reliability, the source's lack of direct access, and the complex character of the contents. For example:

Sourcing: Depiction of the manner in which information was obtained, in order to assist in evaluating the likelihood that the content is factual. A single report from a source or collection platform can contain both direct and indirect information.

Data: Organized information that provides context for evaluating the likelihood that a matter under scrutiny is factual. The information can be either direct (a chronology of events based on observation by US Embassy officers) or indirect (a chronology based on reports provided by a liaison intelligence service.

These terms are illustrated by the following hypothetical example.

We believe country X has begun a major crackdown on the "Extremist Movement," which the government holds responsible for the campaign of terrorism over the past two years.

The Army has been ordered to support the police in cleaning out Extremist strongholds (direct information), according to special intelligence (sourcing). The President of X reportedly is using last week's attack on a shopping center in a working-class neighborhood to justify calling upon the Army to close down the terrorist campaign (indirect information). according to a reliable clandestine source (sourcing). The pro-government press reports (sourcing) the Extremists cannot match Army firepower and are taking high casualties (indirect information). A US Embassy observer reports (sourcing) seeing Army trucks deliver more than 100 prisoners, some badly wounded, to the Central Prison (direct information). According to country X police officials (sourcing), these were part of the 1,000 Extremists rounded up so far in the crackdown (indirect information). CIA's "Country X Terrorism Chronology" indicates this is the first time the Army has been used against the Extremists since the terrorism campaign began in 1993 (data).


Guidelines

Rule 1. Be precise about what is known.

With US interests, and at times lives, on the line, policymakers, warfighters, and law enforcement officials need to be informed precisely what the all-source analysts know and how they know it. In making decisions on whether and how to take action, it is important for them to know if the information is direct or indirect, and if and why the analysts have concluded it is factual.

Most key consumers of intelligence have learned to respect the complexity of the national security issues they are charged with managing and the frequent uncertainty about what is taking place and what lies ahead. DI analysts should write to their standard and not to that of the occasional consumer who wants answers no matter what the circumstances.

Thus, in the name of reliability, analysts should never exaggerate what is known. They should report any important gaps in information bearing on US decisionmaking and potential courses of action, as well as relevant information that seems to contradict the main flow of information.

Best Practices

Analysts should be precise as well in sourcing information. The phrase, According to the US Embassy, for example, does not inform the reader whether the information is direct or indirect.

Best Practices


Rule 2. Distinguish carefully between information and fact.

Analysts may have direct information on what a foreign leader said, for example, and responsibly conclude this as factual. But what that foreign leader believes, intends to do, and will do cannot be known to be true on the basis of a report on what he or she said.

Best Practices


Rule 3. Distinguish carefully between information and estimative judgment.

Analysts' estimative judgments, as indicated, are an important element in the process of supporting policymakers, warfighters, and law enforcement officials. As a rule, these judgments cannot rely solely on reported opinions of foreign players or clandestine sources.

They must be amply argued in terms of the entire body of available information and sound inferential reasoning.

Best Practices

Also, care should be taken to avoid confusion over whether the DI analyst is making an estimative judgment or a source is expressing an opinion. Thus, the following formulation should be avoided: "Country X has turned the corner toward recovery, as indicated by a reliable clandestine source with access to the Finance Minister."

Best Practices


Rule 4. Take account of substantive complexity.

The more complicated an issue (that is, the more inherently difficult it is to be certain on a matter), the greater the informational demands to establish that the matter under consideration is factual. The burden of proof for determining what foreign leaders or groups believe or intend to do, for example, is much greater than that required for determining what they have done or said. Again, analysts may properly make a conditional judgment about what a foreign leader intends to do. (We believe country X is preparing to invade country Y.) But only rarely can this be stated as verified or factual information. (Country X will invade country Y.)

Best Practices

The mindsets of foreign leaders - their attitudes, for example - are also extremely difficult to verify, even with direct or firsthand information, and should rarely be stated as factual. When appropriate, mindset can be a subject for the analyst's estimative judgment.

Best Practices


Rule 5. Take account of policy sensitivity.

As with substantively complex matters, the burden of proof is high on matters that are controversial among policymakers or politically sensitive between the administration and Congress. Directly stated, a solid informational base is needed to present as factual (rather than as the analysts' conditional judgment) something that will be seen by key consumers as bad news. As a rule, then, on controversial matters analysts should place emphasis on the relevant information, and not on estimative conclusions.

Best Practices

Similarly, when addressing the behavior of a given country regarding treaties and agreements with the United States, DI analysts should place emphasis on reporting rather than interpretation. As a rule in these matters, analysts monitor (report relevant information), and policymakers verify (decide whether a violation of a treaty or an agreement has taken place).

Best Practices

-- The President of country X told the US Ambassador that all official assistance had been terminated, although small-scale shipments of weapons by private groups that sympathize with the guerrillas' cause might still be taking place.

-- According to a reliable clandestine source, a senior assistant to the President of X has ordered the military to hide small shipments of weapons for the guerrillas in trucks engaged in normal cross-border trade.

-- Special intelligence indicates that a military mission from country X is to await further orders before purchasing abroad communications equipment requested by the guerrillas.


Rule 6. Take account of the possibility of deception.

Deception can be defined as the manipulation of information by a foreign government, group, or individual to get US intelligence analysts to reach an erroneous conclusion. Deception often works because it gives busy analysts what they are seeking - seemingly reliable information on which to base a conclusion.

Here is where the strength of the all-source analyst comes into play. One test for detecting and countering deception is to determine whether all the sources and collection platforms that should be reporting on a matter have indeed done so.

Databases, or organized information, also help. Is the reported or observed information consistent, in all important details, with past patterns? Do intelligence services with an interest in the matter have a record of perpetrating deceptions?

So does critical thinking. Is the information consistent with the analysts' best judgments on the subject's interests, capabilities, and methods of operation?

Best Practices


Rule 7. Use the term "evidence" sparingly.

This tradecraft note uses the term information as synonymous with the term evidence as it usually is employed in DI assessments. That is, both are used to refer to the content of reports and research that helps reduce the uncertainty surrounding a specific matter.

The US legal system, however, uses evidence in a more specialized manner to refer to matters introduced in a court case and subject to proof and refutation by contending parties. Because the DI is increasingly providing assessments to support law enforcement officials and because even other kinds of assessments can be subpoenaed for possible use in a court case, analysts should avoid using the term evidence when information serves their purposes just as well. At times, characterization of the information is sufficient to make the analysts' point for the benefit of consumers.

Best Practices


Special Circumstances

This note has concentrated on general guidelines for depicting the character and source of the DI's information. The DI serves a broad range of audiences with varied needs and entitlements for intelligence support on a multiplicity of substantive issues. Thus, the special character of the subject matter, the delivery vehicle, or the audience can require exceptions to the rules posited above.

For example, either the limited clearances of the recipients of an assessment or a customer's request for an unclassified memorandum can require the analyst to avoid precision in depicting information and sources.

Forward to "Note 6"

Return to "Compendium" Contents