June 25, 2015: The U.S. Army intelligence bureaucracy is again in trouble with SOCOM (Special Operations Command) over a long-term dispute about computer software. Troops in combat zones and especially SOCOM prefer to use an intelligence database management system called Palantir. But many senior people in the army intelligence and computer tech insist on using another system (DCGS). To further complicate matters it wasn’t even the army who initially created the mess. It all began when the U.S. Air Force developed a data mining and analysis system that, when adapted for army use (as DCGS), turned out to be more expensive and less effective than commercial products (like Palantir). A 2012 government investigation reported the problems in great detail. But senior army commanders and Department of Defense procurement bureaucrats continued to block the use of commercial products the troops preferred. For nearly a year now SOCOM (Special Operations Command) troops have been complaining that a superior system (Palantir) they have been using since 2009 is becoming more difficult to obtain because of more aggressive interference from the procurement bureaucracy and contractor lobbyists. SOCOM was ordered back into Iraq during 2014 and one of their assignments was to collect intelligence on what was going on there. SOCOM preferred Palantir but many procurement officials interfered with doing that.

The basic problem was that the army system (DCGS or Distributed Common Ground System) was cobbled together on the fly, in the midst of a war and has not aged well. Several investigations, in response to growing complaints from the troops, found that the army refused to recognize the problems with DCGS or get them fixed, or allow cheaper and more capable commercial software (like Palantir) to be used instead of DCGS. After 2010 complaints from users and maintainers of DCGS got louder (as in more politicians receiving emails about it). Some of the troops asked for specific commercial systems that were more robust, powerful, and easier to use commercial data mining and predictive analysis software. The army complained that these commercial systems were expensive and required a lot of effort and money to integrate into DCGS. The troops insisted that this was not so and that commercial products like Palantir would save lives. Army bean counters insisted that it was probably only a few dozen lives at most and the additional money needed has to be taken from somewhere else, which might also cost lives in combat. But SOCOM and other organizations point out that they have been able to sneak Palantir into service in some areas and have lots of proof that Palantir outperformed DCGS in combat conditions. But now even SOCOM is being blocked from getting Palantir even though Palantir is officially approved for army use and SOCOM is supposed to be able to buy whatever they need, even if it is not on the “approved” list.

The DCGS controversy also involves professional pride, as the army techs and managers have spent years building DCGS and are confident they can match any commercial products and do it cheaper. But that is rarely the case, as the army simply can't hire the best software engineers and project managers. When it comes to complex software systems, things go better if you keep an eye on the commercial side. If there is something there that does what you need done and does it faster, better, and more reliably it's worth paying the commercial price.