June 21, 2006: The current American efforts to "improve intelligence" continue to run into two seemingly intractable problems. First, agencies continue refusing to share data. The usual excuse, that such sharing would threaten sources, is wearing thin. The National Intelligence Director has asked for examples of such damage to sources, and has not received any. The fanatical dedication to the protection of sources is hard to overcome. A particularly valuable source is usually someone on the inside, who could be killed if the connection to American intelligence services was revealed. But as a practical matter, the vast majority of the "burned" (exposed, and often killed) sources were the result of a traitor within the intel or law enforcement agency. But logic has nothing to do with this. The institutional desire to protect sources is so strong that a major effort will be required to effect a change. The smart money is on things staying the way they are.

The other problem, the inability to analyze all the information collected, is more likely to be solved. This was a problem that arose during the Cold War, when spy satellites and electronic eavesdropping systems were developed, and quickly began to generate far more data than could be examined and analyzed. This is still a major problem. For the last two decades, there have been several major efforts to deal with the flood of data using computer systems. Huge databases and software that can scan large amounts of data quickly and extract the useful bits, were developed. Typically, only a few percent of the material collected (including a lot of written reports) are useful. It's really a needle in the haystack problem.

The current war on terror has, as wars tend to do, spurred development of more effective screening and analysis tools. This has put the spotlight back on the sharing problem. The more data you have, the better the screening software can work. This has also led to another problem. Some political groups have invoked fears of 1984 and "Big Brother" by calling the new techniques invasive and a threat to personal liberties. This makes politicians leery of getting behind any large scale automated data analysis systems. The 1984 angle is more hysteria than anything else. It comes up any time some pressure groups notice that the quantity of personal data has been growing at an enormous rate over the last few decades, and that the government, as well as commercial firms, can analyze it. This is much ado about nothing, but it makes for great headlines, and can easily be used to frighten people. The military gets around all this by calling the analysis systems something else, and keeping quiet about it.

But because of the institutional, technical and political problems, the intel agencies are still overwhelmed with data. The new software sifting systems are showing results, but only here and there. No big breakthroughs, and when these do occur, no one will be releasing press releases.