April 13, 2011: The U.S. Army has upgraded its intelligence tools to more accurately predict where enemy leaders in Afghanistan are, as well as the location of enemy weapons storage sites, smuggling routes and bomb making workshops. The enemy finds this sort of thing very annoying. A sniper or smart bomb is something an Islamic terrorist can understand. Well, OK, the smart bombs smack of magic, but these intel tools are incomprehensible to most everyone. Yet, everyone in the United States is touched by these tools, everyday. Fortunately, snipers and smart bombs are not involved.

Thus it came to be that some of the most effective intelligence tools used in combat today (data mining and predictive analysis) were invented a century ago as part of the development of junk mail. Who knew? Now these tools predict what the enemy is going to do. For decades, the statistical tools used to determine who to send junk mail to (so the sender would make a profit) were not much use to the military. Then came cheaper, and more powerful computers, and the development of data mining and analysis tools. This made a big difference, because the more data you have to work with, the easier it is to predict things. This has been known for centuries.

By 2008, with thousand dollar laptop computers equipped with hundred gigabyte (or more) hard drives, you could put large amounts of data in one place, do the calculations, and make accurate predictions. This wasn't possible thirty years ago, when a 75 megabyte hard drive cost $45,000, and the computer doing the calculations cost even more than that. You also didn't have digital photography (more data you can store for analysis), or a lot of data, in general, stored electronically. It's all different today. That thousand gigabyte hard drive (holding over 10,000 times more data than the $45,000 one of yore) cost less than a hundred bucks. The laptop running the analysis software would have qualified as a supercomputer a decade ago. Back then, there were theories of how data analysis could predict things. Now all those theories are being put to the test, and many have worked.

In the last five years, intel analysts have realized how powerful their tools are. And for those who studied math, statistics or business in college, they know the power of data mining, because it has become a very popular business tool. In places like Iraq and Afghanistan, lots of data is being collected all the time. It was data mining that led to the capture of Saddam Hussein, and the death of al Qaeda-in-Iraq leader Zarqawi. Actually, over a hundred senior (team leader and up) al Qaeda terrorists have been killed or captured in Iraq using these techniques. The same thing is happening now in Afghanistan.

Data mining is basically simple in concept. In any large body of data, you will find patterns. Even if the bad guys are trying to avoid establishing patterns, they will anyway. It's human nature, and only the most attentive pros can avoid this trap. Some trends are more reliable than others, but any trend at all can be useful in combat. The predictive analysis carried out with data mining and other analytic tools has saved the lives of thousands of U.S. troops, by giving them warning of where roadside bombs and ambushes are likely to be, or where the bad guys are hiding out. Similarly, when data was taken off the site of a terrorist leader's death, it often consisted only of names, addresses and other tidbits. But with the vast databases of names, addresses and such already available, typing in each item began to generate additional information, within minutes. That's why, within hours, the trove of data can generate dozens of raids, and even more leads. The enemy tried to adapt to all this, and did to a certain extent. But the predictive analysis moves faster than the opposition can change and adapt. The only effective defense is a new enemy strategy, one that's a break with past practices. This sort of thing is very rare, and not easily done. Even so, the predictive analysis eventually sorts it all out.

Speed has always been an advantage in combat, but, until recently, rarely something intelligence analysis was noted for. No longer. Predictive analysis is something the troops depend on, not only to tips on what to avoid, but for names and places to go after.

Israeli intelligence taught the Americans how to look for terrorist organizations, and identify key leaders and technical specialists. These people became the key targets, and that tactic enabled Israel to defeat the Palestinian terror campaign five years ago. But this was done with old-fashioned police work, and a network of informers inside Palestinian communities. The new computerized systems move data collection and analysis into the 21st century, using technology and concepts that many police departments are using to good effect. But being able to speak to the system, and have it understand what you are looking for, raises the intel game to a whole new level.