June 13, 2006: One of the most effective intelligence tools used in combat today, predictive analysis, was invented a century ago with the development of junk mail. Who knew? For decades, the statistical tools used to determine who to send junk mail to (so the sender would make a profit) were not much use to the military. Then came cheaper, and more powerful computers, and the development of data mining and analysis tools. This made a big difference, because the more data you have to work with, the easier it is to predict things. This has been known for centuries.

Now, with thousand dollar laptop computers equipped with 80 gigabyte hard drives, you can put large amounts of data in one place, do the calculations, and make accurate predictions. This wasn't possible thirty years ago, when a 75 megabyte hard drive cost $45,000, and the computer doing the calculations cost even more than that. You also didn't have digital photography (more data you can store for analysis), or a lot of data, in general, stored electronically. It's all different today. That 80 gigabyte hard drive (holding over a thousand times more data than the $45,000 one of yore) costs less than a hundred bucks.

In the last few years, intel analysts have realized how powerful their tools are. And for those who studied math, statistics or business in college, they know the power of data mining, because it has become a very popular business tool. In places like Iraq and Afghanistan, lots of data is being collected all the time. It was data mining that led to the capture of Saddam, and the death of Zarqawi. Actually, over a hundred senior (team leader and up) al Qaeda terrorists have been killed or captured in Iraq using these techniques.

Data mining is basically simple in concept. In any large body of data, you will find patterns. Even if the bad guys are trying to avoid establishing a pattern to their actions, they will anyway. It's human nature, and only the most attentive pros can avoid this trap. Some trends are more reliable than others, but any trend at all can be useful in combat. The predictive analysis carried out with data mining and other analytic tools has saved the lives of hundreds of U.S. troops, by giving them warning of where roadside bombs and ambushes are likely to be, or where the bad guys are hiding out. Similarly, when data was taken off the site of the Zarqawi bombing, it often consisted only of names, addresses and other tidbits. But with the vast databases of names, addresses and such already available, typing in each item began to generate additional information, within minutes. That's why, within hours, the trove of data generated dozens of raids, and even more leads.

Speed has always been an advantage in combat, but, until recently, rarely something intelligence analysis was noted for. No longer. Predictive analysis is something the troops depend on, not only to tips on what to avoid, but for names and places to go after.