Information Overload
Managing Intelligence Technologies
by Anthony G. Oettinger, Margaret S. MacDonald
From Intelligence, Vol. 24 (3) - Fall 2002
Print     Email article 1 2 3 Next

ANTHONY G. OETTINGER is Gordon McKay Professor of Applied Mathematics and Professor of Information Resources Policy at Harvard University.
MARGARET S. MACDONALD is Senior Editor at the MITRE Corporation in Bedford, Mass.

Advances in scientific knowledge, translated into new technology, have made previously unmanageable intelligence tasks feasible and greatly increased the speed at which intelligence professionals perform traditional activities. Improved sensors, transmission capabilities, and analytical tools deliver unprecedented volumes of information and processing capabilities to the intelligence community and its customers, military and political decision makers. Processes that used to take days or weeks now take only seconds. Activities carried out at dispersed locations throughout the world can be managed centrally, ensuring coherence in the information delivered and a rapid flow of intelligence between the field and administrative office.

And yet, problems that have always plagued intelligence seem impervious to the information revolution. A 1990 report titled Whence and Whither Intelligence, Command and Control? The Certainty of Uncertainty explains: “The continuing availability of ever smaller, faster, cheaper, better tools for information processing gives us the illusion that throwing these tools at perennial problems of intelligence, command and control can solve these problems once and for all. In reality, the new tools continuously trigger readjustments in numerous interlinked balancing acts … The endless frontier of complexity accounts for our simultaneous sensations of both progress and déjà vu.” For 22 years, the Seminar on Intelligence, Command, and Control has brought military and civilian leaders to Harvard University. Their opinions and anecdotes illuminate the persistent balances that the intelligence community and its customers must keep adjusting. While technology may tip some of these balances in one direction or another, it has not eliminated the conflicting forces that must be balanced—and never will.

Supply and Demand

The intelligence professionals of 20 or even 10 years ago might well have considered the technology available to the intelligence community today a dream come true. The information revolution has provided the intelligence community and its customers with superb tools, including supercomputers and software for intelligent information processing, high-resolution satellite imagery, and sensors that penetrate natural and manmade barriers to identify targets or link directly to precision-guided weapons. In-Q-Tel, the for-profit corporation spun off from the Central Intelligence Agency (CIA) to encourage commercial development of intelligence-related technology, has been designed to connect the intelligence community to some of the most innovative thinking in the private sector.

The result in many areas is that the intelligence community and its customers no longer suffer from information scarcity but from information overload. Analysis must cover enormous quantities of data, in which valuable information may at best be implicit. Even so, many decision makers who remember the information scarcity of the Cold War era still demand more from the intelligence community, thereby perpetuating approaches more appropriate to penetrating the Soviet Union than to dealing with terrorists.

Efficient search engines and vast databases certainly allow the intelligence community to use that mass of information to supply objective intelligence almost instantaneously (e.g. “Which airfields in Country X can accommodate C-130 transport planes?” or “What are the physical characteristics of the enemy’s new anti-aircraft radar?”). Geospatial information systems can combine classified satellite imagery, digital terrain elevation data, hydrographic information, aeronautical information, and foundation feature data to create maps that give deployed troops displays not only of the natural environment but also of manmade features and recent activity in an area. In many tactical situations, such intelligence products suffice to meet the operational user’s needs.

If intelligence professionals were geographers or historians, today’s technology might be a panacea. But, as James M. Simon, US assistant director of central intelligence for administration, points out, “Intelligence is not history; it is secret information of actionable use. Intelligence must also try to divine intent, or future developments, where facts are not available.” While electrons may move at the speed of light, human understanding does not. No matter how rapidly electronic information reaches an intelligence analyst, that analyst must still read, digest, and act upon it. That information must be winnowed, cross-checked for accuracy, analyzed for significance, and finally disseminated in a form appropriate for use by the intended customer.

By and large, therefore, high-speed computers and networks, drawing on the products of the latest generation of sensors, allow the intelligence community to perform its traditional tasks faster but have not changed these tasks fundamentally. Modern technology has not only increased the amount of information available to the intelligence community and the speed at which it can—and must—analyze, tailor, and disseminate it but also the speed at which customers demand intelligence products. The intelligence community must deliver intelligence as rapidly as forces move and faster than an enemy can deliver intelligence to its forces. Organizations threatened by an information attack need immediate warning. Thus, each improvement in technology brings about an equivalent increase in demand. The balancing act remains.

Knowledge is never complete, so the intelligence community must constantly balance the imperative to warn of an impending danger against the risk of overreacting on the basis of insufficient evidence. Information technology compounds this problem. Charles Allen, a former US national intelligence officer for warning, describes the dilemma facing intelligence analysts: “What policymakers and warning analysts must continually relearn is that there is a tradeoff relationship between the probability of false alarm and the probability of accurate warning. If the consequence of a false alarm is ‘missiles away!’ or the belittlement of an analyst who cried wolf, then the [analyst’s warning] threshold is inevitably going to go up.” Technology undoubtedly brings the analyst vastly more data to sift for relevant information, but the amount can be overwhelming and the tools to help in the sifting process have not yet matured. The same intelligence community that can deliver high-resolution images of the most remote parts of Afghanistan misidentified the Chinese embassy in Belgrade with disastrous results. Admiral Thomas R. Wilson, director of the US Defense Intelligence Agency, recalls, “I was the J-2 (director of intelligence) on the Joint Staff, so I was the one who showed the picture of the Chinese embassy to the president of the United States (among 900 other pictures I showed him) and said, ‘We’re going to bomb this because it’s the Yugoslav department of military procurement.’ We had good sources that said that’s what it was; another agency (not my own) got that information and gave us that identification, and our databases were not able to find that mistake in targeting. Even though we had people in the systems who knew that building was the Chinese embassy, it had not been entered into the database.” More recently, information that might have prevented some of the September 11 attacks apparently existed somewhere within the vast quantity of data collected by the intelligence community, but the systems for using such information have lagged far behind the ability to collect data.

1 2 3 Next