March 9, 2010:  Israel is expanding its battlefield intelligence collection and analysis capabilities. The new Combat Collection Corps will take the existing Field Intelligence Corps, beef it up with more personnel, equipment and training, and collect more information on the battlefield, do it faster and analyze it more intensely. The impetus was a review of field intelligence work over the last few years, which revealed that technology, and the experience of the Americans in Iraq, indicated that more could, and should, be done.

The war in Iraq has changed the way American military intelligence operations were conducted. This came about because smaller combat units were now able to pass more information to the intelligence specialists, and these troops now had more powerful hardware and software to do something with the flood of new data. In the past, the brigade, division and higher level intelligence shops conducted a lot of their own intelligence gathering (and had their own aircraft and reconnaissance units for that purpose). In theory, they were supposed to be getting a lot of information from subordinate units. But that was a paperwork nightmare. The smaller units had lots of good intel, but they were out there getting shot at, and had a hard time filling out reports. Intelligence specialists were supposed to debrief patrols and troops involved in other combat operations. But, as always, there were shortages of manpower to collect the information and put it to use.

That began to change in Iraq, where a lot of the debriefing got automated, and even the smallest infantry patrol, could easily report their findings electronically. By capturing the data electronically at the lowest level, and building database tools to handle it, information did not expire (as battlefield information tends to quickly do).

With lots more fresh battlefield data available, it was now possible to do things that, only a few years earlier, were believed the province of much larger, and better funded, organizations. For example, there is data fusion. This is collecting intelligence from many sources, and sorting through it for useful patterns and items that, in seen in the right context, are very valuable. The first of these fusion centers were set up at the national level two decades ago. But during the course of the Iraq war, the concept moved down the food chain. Cheaper, and more powerful, computer hardware was able to use analysis software to speed the fusion process, even in the hands of a relatively inexperienced operator.

Some of the most effective intelligence tools used in combat today, data mining and predictive analysis, were invented a century ago along with the development of junk mail. Who knew? For decades, the statistical tools used to determine who to send junk mail to (so the sender would make a profit) were not much use to the military. Then came cheaper, and more powerful computers, and the development of data mining and analysis tools. This made a big difference, because the more data you have to work with, the easier it is to predict things. This has been known for centuries.

By 2008, with thousand dollar laptop computers equipped with hundred gigabyte (or more) hard drives, you could put large amounts of data in one place, do the calculations, and make accurate predictions. This wasn't possible thirty years ago, when a 75 megabyte hard drive cost $45,000, and the computer doing the calculations cost even more than that. You also didn't have digital photography (more data you can store for analysis), or a lot of data, in general, stored electronically. It's all different today. That hundred gigabyte hard drive (holding over a thousand times more data than the $45,000 one of yore) cost less than a hundred bucks.

In the last five years, intel analysts have realized how powerful their tools are. And for those who studied math, statistics or business in college, they know the power of data mining, because it has become a very popular business tool. In places like Iraq and Afghanistan, lots of data is being collected all the time. It was data mining that led to the capture of Saddam, and the death of Zarqawi. Actually, over a hundred senior (team leader and up) al Qaeda terrorists have been killed or captured in Iraq using these techniques.

Data mining is basically simple in concept. In any large body of data, you will find patterns. Even if the bad guys are trying to avoid establishing a pattern to their actions, they will anyway. It's human nature, and only the most attentive pros can avoid this trap. Some trends are more reliable than others, but any trend at all can be useful in combat. The predictive analysis carried out with data mining and other analytic tools has saved the lives of hundreds of U.S. troops, by giving them warning of where roadside bombs and ambushes are likely to be, or where the bad guys are hiding out. Similarly, when data was taken off the site of a terrorist leaders' death, it often consisted only of names, addresses and other tidbits. But with the vast databases of names, addresses and such already available, typing in each item began to generate additional information, within minutes. That's why, within hours, the trove of data can generate dozens of raids, and even more leads.

Speed has always been an advantage in combat, but, until recently, rarely something intelligence analysis was noted for. No longer. Predictive analysis is something the troops depend on, not only to tips on what to avoid, but for names and places to go after.

Israeli intelligence taught the Americans how to look for terrorist organizations, and identify key leaders and technical specialists. These people became the key targets, and that tactic enabled Israel to defeat the Palestinian terror campaign five years ago. But this was done with old-fashioned police work, and a network of informers inside Palestinian communities. The new Combat Collection Corps will move data collection and analysis into the 21st century, using technology and concepts that many police departments are using to good effect.