Quantify your guestimates: Estimating hidden numbers in intelligence practice

Quantify your guestimates: Estimating hidden numbers in intelligence practice

Captain, M.Sc., Jens Flinch Joergensen

M.Sc. Melanie Hartvigsen

Have you ever wondered how intelligence reports come up with surprisingly accurate numbers? For example, the amount of money raised by al-Qaida through hostage-taking or the number of Russian ground forces in Ukraine. It is in most cases not because the intelligence professionals have actually counted the money or counted the number of Russians in Eastern Ukraine, but more likely, the intelligence professional has made a more or less valid estimation of the numbers based on judgement, previous experience and the best available data – i.e. an estimation of a hidden number.

This article will discuss the concept of hidden numbers and the different approaches to handling the concept; further, it will relate hidden numbers to the intelligence profession and thus identify methods to estimate hidden populations.

Intelligence in this connection is used as a broad term covering both intelligence in the police and law enforcement and intelligence in the military spanning from the tactical to the strategic level. The article will present methods to intelligence professionals to better reflect and choose tolls for estimating numbers and quantifiable elements of intelligence. The methods presented here can thus be seen as a catalogue for inspiration for both the intelligence analyst and for leaders in intelligence units.

Views on estimative intelligence

When discussing estimative intelligence, it is important to be aware of the various perceptions of what the purpose of intelligence estimates is. There are two views on how estimative intelligence is supposed to deal with uncertainty. They are respectively “eliminating uncertainty” and “assessing uncertainty”. The goal of eliminating uncertainty is.. well, to eliminate or reduce the uncertainty. As such, there are close to correct answers to estimative questions. The other view is that the goal of estimative intelligence should be to assess uncertainty and not eliminate it. In other words, intelligence analyses may reveal new uncertainties, and this should not be seen as a weakness in the intelligence work, but rather a strength as the intelligence product is more nuanced yet may be harder to operationalize.  

The first view is favored throughout the intelligence community as decision making is simplified by certainty, however, it should be noted that the strive for certainty may lead to flaws in how analysts compare alternate hypotheses, how they express likelihood and confidence and how they filter information. Vice versa, the attempt to assess uncertainty may not provide absolute answers, which does not make decision making any easier, but rather provides several realistic answers to estimative questions.

While this article does not argue that one view is more correct than another, this article does question the methods used to eliminate uncertainty through estimates, and encourages a combination of various estimative methods to enhance precision.

What is a hidden number?

The term “hidden numbers” used in this essay is closely linked with terms such as underreporting, known unknowns, shadow numbers etc. The phrase “hidden numbers“ has been chosen in this connection due to the focus on measurability (hence “numbers”) and the focus on intelligence (hence not merely a matter of underreporting, but often of an opponent trying deliberately to hide information, therefore “hidden”).

Hidden numbers have certain characteristics that define them:

• They represent data that would have been part of a certain investigation or research if the data had been known to the investigator or researcher.
• It is known or obvious, that the hidden numbers are there. If the investigator or researcher has no suspicion that they are there, the unknown data is merely another blind spot.
• It is – for various reasons - not possible to simply measure or count the hidden number

What characterizes hidden numbers is thus the awareness that there is a missing piece of the puzzle and that the size or measure of that missing piece is unknown.

A vital point here is to whom the numbers are hidden. The number of ballistic missile launching pads in North Korea may be a hidden number to US intelligence agencies, but a perfectly known number to the North Korean leadership. Similarly, a number hidden for the intelligence professional in 2014 may not be a hidden number to the researcher in 2016, as new data is brought to light. Finally, it is worth considering if information hidden to one branch of an intelligence service but known to another is actually a hidden number or merely an example of poor sharing of knowledge.

The phrase “hidden number” may not be the phrase of choice in the community. Other expressions like “ballpark estimate” or “ballpark figure” may be used or the more official “educated guess” or – indicating more certainty – the phrase “estimate”. Humor may be used to cover uncertainty, to distract from the lack of available data or simply to discreetly convey to the listener, that the number listed is at best a guess. The POMA analysis is a prime example hereof. POMA is an acronym for Pull Out My Ass. Another way of underlining uncertainty is to add the postfix “-ish” to the number or estimate in question. E.g., the intelligence professional may utter “the number of insurgent fighters in a given area is 250-ish”.

In the following, three methods of estimating hidden numbers will be discussed:                             

One: The use of proxies

Estimating the production and consumption of drugs can be difficult, as the illicit trade thrives in the shadows and seeks to evade detection. UNODC yearly publishes the “World Drug Report” with updated data on both production and consumption of drugs. The drug production is as such a hidden number, as there are no official registers for production or producers of say, cocaine in Colombia or say, heroin in Afghanistan. That there is a production is obvious, given the supply of drugs to the world market, and the size of the production is thus classical of estimating a “hidden number”. Instead of measuring the actual production, UNODC applies a variety of methods to estimate the production. In the case of drugs, that require a certain crop for production, for example cocaine or heroin, data from aerial photos can provide a guide for the size of the acreage under production and thus – indirectly – the production itself. In the case of drugs, that are manufactured synthetically, e.g. ecstasy or amphetamine, the amount of precursor materials imported to a given country, can give some insights on the size of the production.

The use of proxies is as old as the craft of intelligence itself. Instead of measuring the exact number of enemy soldiers, armies have resorted to counting the number of campfires at night. In modern times, allied intelligence used statistics to estimate the capacity of German tank production during WWII using a proxy – in this case the serial numbers of various spare parts.

Two: Survey

For a crime to be present in official statistics, the crime has to be reported either by a victim, a citizen or by law enforcement agencies. A typical challenge to crime statistics is that some crimes go unreported. For example, domestic violence is kept inside the homes, and rapes and sexual assaults may leave the victim ashamed of reporting. In order to estimate the hidden numbers of crime in Denmark, the Danish Crime prevention council conducts victim-surveys where a larger part of the general population is asked about their exposure to and perception of crime. From this, the general level of crime in Denmark can be assessed and compared to the level of reported crime, thus putting an (estimated) number on the hidden population.

While surveys may not entirely be applicable to many sub-branches of intelligence in the military, as it – for example - may not be possible to conduct a survey among insurgents in Afghanistan. However, the above example shows the applicability of the surveys within law enforcement, and can be adapted to military use: In relation to counter-intelligence anonymous surveys among friendly forces may help establish the number of underreported security breaches or within the cyber domain, surveys can be used to uncover infiltration attempts on social networks or via phishing emails. The survey method can also be applied to measure the reach and effect of targeted influence campaigns against a specific target audience within the domain of psychological operations.

Three: Extrapolation over time or space

While interviewing an experienced intelligence analyst serving in Iraq and Afghanistan, we’ve had the use of extrapolation over space explained: “If you know that a certain number of government officials in the area of operations next to yours are corrupt, then chances are, the same applies to your own area of operations.” This intuitively pragmatic approach may work well in some settings, but has certain limitations: First, for one to extrapolate from one area to another, the areas have to be comparable. In the above example, it could be argued that two regions in, say, Iraq would be comparable, but to the untrained eye, regional differences and peculiarities may go unnoticed. The second limitation is that for the extrapolation to work, the original estimate has to be valid. If the estimate of the level of corruption in your neighboring area is based on obsolete data or on a flawed analysis, then there is no value in applying the same results to your area.

Extrapolation over time bears the same inherent limitations as extrapolation over space; however, both methods are often used as a pragmatic approach. For example, the estimates of the level of underreported crime (see “survey”) conducted by the Danish crime prevention council, are quoted years after their publication based on the assumption that the levels remain comparatively constant over time and that data for example from the year 2009 may apply to the year 2016.

Basing one's situational awareness on experience from previous experiences from another time and space appears widespread among the intelligence community, and presents itself as a low-cost and fast way to estimate - among other things - the size of hidden numbers. This method is also an example of an attempt to eliminate uncertainty, hence risking ignoring new uncertainties that might have been revealed if attempting to assess uncertainty.

Methods in intelligence studies – are we heading in the right direction?

The above examples have provided the reader with an introduction to the concept of hidden numbers and the way to handle them in relation to intelligence. The applicability of each individual approach can be discussed and each has – as explained above - inherent limitations. What is called for is thus not necessarily for an intelligence professional to apply one specific method of analysis, but rather to raise the awareness of the choice of method. Oftentimes, a combination of several methods may yield good results. While the above-mentioned methods may not necessarily bring a specific number to light, the use of several methods may give a frame capturing the possible size of a hidden number rather than an exact hidden number, hence applying the assessing uncertainty perception. In this way, the intelligence professional increases the reliability of the estimate but at the same time reduces the precision. This – however – contradicts the “fetish for precision over relevance”. In other words, if the relevant parameter is not precisely measurable, the measurable parameter becomes relevant.

A rule of thumb in the intelligence profession is to rely on several independent sources. Say, a human intelligence source combined with signal intelligence and images on the same target gives more valid intelligence than merely a picture by itself. Similarly, the intelligence professional should employ a variety of analytical approaches to intelligence estimates. It various analysis reveal the same results, things look good. If results and estimates differ, caution is recommended just the same as when a human intelligence source contradicts what is learned from image intelligence.

In the 5^th century B.C., the Chinese military strategist, Sun Tzu, wrote: “All warfare is based on deception. Hence, when we are able to attack, we must seem unable; when using our forces, we must appear inactive; when we are near, we must make the enemy believe we are far away; when far away, we must make him believe we are near.” Using intelligence to deceive the enemy is hence not a new tactic. Hence, estimating hidden numbers is likewise related to counter-intelligence and deception efforts, as an opponent may try to make the estimating enemy numbers as troublesome or faulty as possible. A force may try to up- or downplay its numbers leading the opponent to believe that the actual force is weaker or stronger than it actually is. The use of dummy-tanks and artificial trenches at best make the opponent’s intelligence collection tools count more tank or trenches than are actually there or at the least, makes his intelligence collection more of a burden as additional resources have to be committed to evaluating pictures, satellite photos and human intelligence reports.

Concluding remarks

Summing up, while guestimating is and likely continues to be an integrated part of the intelligence profession, the article here has highlighted ways to raise awareness of how to estimate and what methods to use. Intelligence and intelligence studies can soak up a large amount of fruitful inspiration from various academic fields of study, as this essay has exemplified.  While uncovering hidden numbers may not lead directly to actionable intelligence, but can help provide or clarify situational awareness and establish a baseline understanding of a certain problem.  Further, awareness of the methods employed can highlight the precision of estimates.

In a world where more and more raw data and information is both provided and requested, and where the battlefield is increasingly dotted with sensors,  the ability to distinguish relevant from irrelevant and to apply sound judgements to intelligence estimates, may be what marks the difference between winners and losers in contemporary warfare.