May 4, 2009: The Department of Defense and the CIA and other intelligence agencies are eagerly adopting the "mashup" techniques used to quickly and easily combine visual and text data from many sources into a single screen full of more useful information. It's another example of the whole being more useful than the sum of the parts.

Mashups were popularized with the appearance of web apps like Google Maps and Google Earth, and the ability to overlay other web data on the maps and terrain photos. Before this, mashups were restricted to hand crafted business applications. Some of these used maps, but others were simply clever (and highly useful) arrangements of financial data (in lists and charts) to support financial operations. The most famous example of this was the "Bloomberg Box", which first appeared in the 1980s, and made financial analysis and trading much more efficient. This was preceded by MIS (Management Information Systems), which sought to combine the growing amount of computerized business data into a format that managers could more easily comprehend.

The military and intelligence agencies are using the mashup to rapidly assemble data for analysis and planning. The military wants to give commanders the ability to quickly collect all available data (locally and worldwide) so that battle plans can be put together and executed before the enemy can realize, or react to the changing situation. Speed is a critical factor in planning, and lack of information was always a major impediment. As with the business world, the military now has enormous quantities of computerized data. The mashup techniques enable all this material to be quicky accessed and assembled in useful ways.

American troops have been improvising mashups for decades, and the current efforts are an extension of that. Many of those currently running the military, participated in those early, informal, mashup efforts, and appreciate the potential. Thus the seemingly sudden flood of funding and interest.

The mashup is also merging with other new uses the military has found for computers. For example, now you can take a thousand dollar laptop computer, equipped with 320 gigabyte hard drive (which puts large amounts of data in one place), employ powerful statistical calculations (on a 64 bit, multi core CPU), and make accurate predictions. This wasn't possible thirty years ago, when a 75 megabyte hard drive cost $45,000, and the computer doing the calculations cost even more than that while using an 8 bit CPU. You also didn't have digital photography (more data you can store for analysis), or a lot of data, in general, stored electronically. It's all different today. That 320 gigabyte hard drive (holding several thousand times more data than the $45,000 one of yore) costs less than a hundred bucks.

Since September 11, 2001, intel analysts have realized how powerful their tools are. And for those who studied math, statistics or business in college, they know the power of data mining, because it has become a very popular business tool. In places like Iraq and Afghanistan, lots of data is being collected all the time. It was data mining that led to the capture of Saddam, and the death of al Qaeda leader Zarqawi, and most of his successors and associates.

Data mining is basically simple in concept. In any large body of data, you will find patterns. Even if the bad guys are trying to avoid establishing a pattern to their actions, they will anyway. It's human nature, and only the most attentive pros can avoid this trap. Some trends are more reliable than others, but any trend at all can be useful in combat. The predictive analysis carried out with data mining and other analytic tools has saved the lives of hundreds of U.S. troops, by giving them warning of where roadside bombs and ambushes are likely to be, or where the bad guys are hiding out. Similarly, when data was taken off the laptops of dead or captured terrorist leaders, it often consisted only of names, addresses and other tidbits. But with the vast databases of names, addresses and such already available, typing in each item began to generate additional information, within minutes. That's why, within hours, the trove of data generated dozens of raids, and even more leads.

Speed has always been an advantage in combat, but, until recently, rarely something intelligence analysis was noted for. No longer. Predictive analysis is something the troops depend on, not only to tips on what to avoid, but for names and places to go after. Adding this form of analysis to a mashup, it makes your next move more obvious, and further speeds up your decision making. The enemy has discovered that this speed is the most dangerous weapon the Americans possess. For intelligence agencies, it enables them to take many small, seemingly useless, bits of information, and rapidly tease out leads, and add them to the mashup.