April 6, 2015:   For over a decade now the U.S. Army intelligence bureaucracy has been trying to cope with a procurement disaster that will not go away. This all began when the U.S. Air Force developed a data mining and analysis system that, when adapted for army use, turned out to be more expensive and less effective than commercial products. A 2012 government investigation reported the problems in great detail. But senior army commanders and Department of Defense procurement bureaucrats continued to block the use of commercial products the troops preferred. Now SOCOM (Special Operations Command) troops are complaining that a superior system (Palantir) they have been using since 2009 is becoming more difficult to obtain because of more aggressive interference from the procurement bureaucracy and contractor lobbyists.

The problem was that the army system (DCGS or Distributed Common Ground System) was cobbled together on the fly, in the midst of a war and has not aged well. Several investigations, in response to growing complaints from the troops, found that the army refused to recognize the problems with DCGS or get them fixed, or allow cheaper and more capable commercial software (like Palantir) to be used instead of DCGS.

After 2010 complaints from users and maintainers of DCGS got louder (as in more politicians receiving emails about it). Some of the troops asked for specific commercial systems that were more robust, powerful, and easier to use commercial data mining and predictive analysis software. The army complained that these commercial systems were expensive and required a lot of effort and money to integrate into DCGS. The troops insisted that this was not so and that commercial products like Palantir would save lives. Army bean counters insisted that it was probably only a few dozen lives at most and the additional money needed has to be taken from somewhere else, which might also cost lives in combat. But SOCOM and other organizations point out that they have been able to sneak Palantir into service in some areas and have lots of proof that Palantir outperformed DCGS in combat conditions. But now even SOCOM is being blocked from getting Palantir even though Palantir is officially approved for army use and SOCOM is supposed to be able to buy whatever they need, even if it is not on the “approved” list..

The DCGS controversy also involves professional pride, as the army techs and managers have spent years building DCGS and are confident they can match any commercial products and do it cheaper. But that is rarely the case, as the army simply can't hire the best software engineers and project managers. When it comes to complex software systems, things go better if you keep an eye on the commercial side. If there is something there that does what you need done and does it faster, better, and more reliably it's worth paying the commercial price.

So far, when DCGS works, which is some of the time, the enemy finds this sort of thing very annoying. A sniper or smart bomb is something an Islamic terrorist can understand. Well, OK, the smart bombs smack of magic but these intel tools are incomprehensible to most everyone. Yet everyone in the United States is touched by these tools, every day. Fortunately snipers and smart bombs are not involved.

Thus it came to be that some of the most effective intelligence tools used in combat today (data mining and predictive analysis) were invented a century ago as part of the development of junk mail. Who knew? Now these tools predict what the enemy is going to do. For decades the statistical tools used to determine who to send junk mail to (so the sender would make a profit) were not much use to the military. Then came cheaper and more powerful computers and the development of data mining and analysis tools. This made a big difference because the more data you have to work with, the easier it is to predict things. This has been known for over a century.

By 2008, with thousand dollar laptop computers equipped with hundred gigabyte (or more) hard drives, you could put large amounts of data in one place, do the calculations, and make accurate predictions. This wasn't possible in the 1970s, when a 75 megabyte hard drive cost $45,000 and the computer doing the calculations cost even more than that. You also didn't have digital photography (more data you can store for analysis) or a lot of data, in general, stored electronically. It's all different today. That thousand gigabyte hard drive (holding over 10,000 times more data than the $45,000 drive of yore) cost less than a hundred bucks. The laptop running the analysis software would have qualified as a supercomputer a decade ago. Back then there were theories of how data analysis could predict things. Now all those theories are being put to the test and many have worked.

Since September 11, 2001 intel analysts have come to realize how powerful their tools are. And for those who studied math, statistics, or business in college they know the power of data mining because it has become a very popular business tool. In places like Iraq and Afghanistan, lots of data was being collected all the time. It was some local data mining that led to the capture of Saddam Hussein, the death of al Qaeda-in-Iraq leader Zarqawi, and al Qaeda head Osama bin Laden. Over a hundred senior (team leader and up) al Qaeda terrorists have been killed or captured in Iraq using these techniques. The same thing happened in Afghanistan, although the introduction of DCGS has created a lot of complaints because, although DCGS had access to a lot more data, it often broke down or was too slow. Many units already had database and analysis systems they had put together (usually from commercial products) and found themselves using these "inferior" systems to get the job done when DCGS stumbled. This encouraged the DCGS developers to fix bugs more quickly but DCGS was still inferior, in many respects, to similar commercial software. Having spent nearly $3 billion on DCGS, the military was reluctant to admit defeat and spend a lot more on commercial software to replace parts of DCGS.

Meanwhile, the basic tasks DCGS must perform are not that complicated. Data mining is basically simple in concept. In any large body of data you will find patterns. Even if the bad guys are trying to avoid establishing patterns, they will create them anyway. It is human nature and only the most attentive pros can avoid this trap. Some trends are more reliable than others but any trend at all can be useful in combat. The predictive analysis carried out with data mining and other analytic tools has saved the lives of thousands of U.S. troops by giving them warning of where roadside bombs and ambushes are likely to be or where the bad guys are hiding out. Similarly, when data was taken off the site of a terrorist leader's death, it often consisted only of names, addresses, and other tidbits. But with the vast databases of names, addresses, and such already available, typing in each item began to generate additional information, within minutes. That's why, within hours, the trove of data can generate dozens of raids and even more leads. The enemy tried to adapt to all this and did to a certain extent. But the predictive analysis moves faster than the opposition can change and adapt. The only effective defense is a new enemy strategy, one that's a break with past practices. This sort of thing is very rare and not easily done. Even so, the predictive analysis eventually sorts it all out.

Speed has always been an advantage in combat but, until recently, rarely something intelligence analysis was noted for. This is no longer the case, at least as long as DCGS does not crash. Predictive analysis is something the troops depend on, not only for tips on what to avoid but for names and places to go after.

Israeli intelligence taught the Americans how to look for terrorist organizations and identify key leaders and technical specialists. These people became the primary targets and that tactic enabled Israel to defeat the Palestinian terror campaign in 2005. But this was done with old-fashioned police work and a network of informers inside Palestinian communities. The new computerized systems move data collection and analysis into the 21st century, using technology and concepts that many police departments are using to good effect. But being able to speak to the system, and have it understand what you are looking for, raises the intel game to a whole new level.