October 25, 2009:
The American flying death squad of Predator and Reaper UAVs over Pakistan and Afghanistan have a target list of about 420 Taliban and drug gang leaders. The CIA handles most of the data collection, and actual hits on the targets. Finding these targets is a much larger operation than the effort to keep twenty or so armed UAVs in the air.
The CIA and Department of Defense have enormous databases of people, including basic information like names, physical descriptions, family connections and so. But there is much more, like video and still images of the subjects, their vehicles, hideouts and work places.
The proliferation of video cameras on the battlefield (in UAVs, ground robots, for base security and in the hands of the troops) has provided a huge library of images that show bad guys doing what bad guys do and what they look like while doing it. This can range from moving around carrying weapons, to using those weapons, to the particular driving patterns of people up to no good. This is a unique resource, and the U.S. is putting together a library of these images. This is similar to older still pictures libraries, which were eventually used by pattern recognition software to let machines examine the millions of images digital photo satellites began producing decades ago. The basic problem was that there were quickly too many pictures for human analysts to examine. Computers had to do much of the work, or else most of the images would go unexamined. This technology was quickly adapted to the kind of combat encountered in Iraq and Afghanistan, and terrorist operations in general.
Research has shown that people staring at live video feeds start losing their ability to concentrate on the images after about twenty minutes. This problem has been known for some time, and the military (not to mention civilian security firms) have been seeking a technological solution. It's actually not as bad with UAVs, because the picture constantly changes, but cameras that are fixed can wear operators out real quick.
The basic tech solution is pattern analysis. Since the most common video is digital, it's possible to translate the video into numbers, and then analyze those numbers. Government security organizations have been doing this for some time, but after the fact. It's one thing to have a bunch of computers analyze satellite photos for a week, to see if there was anything useful there. It's quite another matter to do it in real time. But computers have gotten faster, cheaper and smaller in the last few years, and programmers have kept coming up with more efficient routines for analyzing the digital images. Commercial firms already have software on the market that will analyze, in real time, video, and alert a human operator if someone, or something (you are looking for) appears to be there.
While some military analysis does not have to be real time (like the system used in Iraq and Afghanistan to compare today's and yesterdays photos of a road to see if a bomb may have been planted), the most common need is for real time analysis. Several times a year now, a new software package shows up that does that, or tries to. These systems are getting better. Many can definitely beat your average human observer over time (several hours of viewing video). The real time analysis software is rapidly evolving. You don't hear much about it, because if the enemy knows the details of how it works, they can develop moves that will deceive it (or, to be more accurate, make the pattern analysis less accurate.) Already, this software is being used as an adjunct to human observers, and gradually taking over. There will always be a human in the loop, to confirm what the software believes it has found.
But the big breakthrough, which may already have been achieved, is a predictive analysis system that can quickly examine thousands of hours of video from a specific area, and calculate the probability that certain vehicles, or individuals, down there, are up to no good, or will simply be travelling down a certain road. This works if you have lots of examples of people you know, and are trying to find. The predictive analysis looks for enough indicators to make it likely that something specific is going to happen. When done in real time, the analysis software can instantly alert that something specific is about to happen at a specific location. If nothing does happen, that is saved and added to the library of experience the analysis software uses to make predictions. In effect, the predictive analysis software gets smarter the more often it is used. And the library of combat zone video images grows larger as well, making it possible for the analysis software to sniff more behavior patterns that predict bad actions.
The CIA also uses informants on the ground, and interrogations of prisoners, or of people who have been to a certain place. Every little bit of information can be valuable, when viewed as part of a mosaic of activity. Tracking people, who do not want to be tracked, requires all this data in order to find them, and do so in a timely enough fashion to fire a missile at them.