March 4, 2018: South Korea is installing “intelligent surveillance cameras” along the DMZ (DeMilitarized Zone) and other areas that are vulnerable to attack by North Korean commandos or stealthy infantry. In addition to deliberate attacks the cameras along the 243 kilometer long DMZ will make it easier to spot, especially at night, people trying sneak across the four kilometers wide DMZ. In the last decade there have been incidents of North Korean soldiers just walking across the DMZ at night and defecting. What was embarrassing was that these casual defectors were not spotted and in one case had to knock on the door at South Korean border station to find someone to accept his surrender. Earlier attempts to avoid this sort of thing using motion detectors failed because the DMZ has become something of a wildlife refuge with all sorts of four legged critters are out and about and don’t care if they set off a motion detector. The “intelligent surveillance cameras” were designed and tested to avoid this problem.

What makes “intelligent surveillance cameras” work is real-time analysis of video by software that can detect suspicious (as designated by operators) movement day or night. Software that can do this has been evolving rapidly since the late 1990s and the South Korean military has been monitoring progress in this area for over a decade confident that soon a combination of inexpensive but powerful hardware and video analysis software would reach the point where it would be accurate and reliable enough for use in critical situations where the system had to match the accuracy of a trained human operator. Israel, the United States, South Korea and China have all made major strides in developing such software for a largely non-military market. But in a combat zone the stakes are higher and the video image analysis software had to be very good. The South Koreans believe they now have what they need.

While most of the Chinese development of these systems has been in support of police operations (especially facial recognition) the Chinese also aim to dominate civilian markets. Israel, South Korea and the Americans have, since 2000, been prompted by military necessity to make this stuff work. South Korea has paid close attention to U.S. and Israeli progress in this area. For example both the Americans and Israelis have developed several generations of video analysis software that would spot in real time suspicious activity captured by airborne vidcams. This was all in an effort to deal with Islamic terrorism.

An example of this was revealed by the Israelis in 2017 when they described a new aerial surveillance system called SkEye. This is a multi-camera system built into a UAV as small as the Hermes 450 (max payload 150 kg) or even small manned aircraft like the single engine Cessna 208B. SkEye uses ten high-resolution video cameras to simultaneously watch, track and record activity in an areas as large as 80 square kilometers at a time. SkEye uses digital image scanning software to seek out specific items and patterns of movement below. Since SkEye records all that it sees it can also detect patterns over time. This approach to aerial surveillance has existed in theory for decades but the technology to make it work didn’t exist until after the 1990s when the U.S. Army needed an effective way to find newly placed roadside bombs and landmines in Iraq and Afghanistan.

The Americans spent billions on this bomb detection program and in 2005 the first workable, although somewhat crude, system (“Constant Hawk”) showed up and worked so well that the army named Constant Hawk one of the top ten inventions for 2006. The army does this to give some of the more obscure, yet very valuable, developments some well-deserved recognition. Constant Hawk, like most new technology did not get a lot of media attention otherwise. Mainly it's the math and complex tech, and TV audiences that get uneasy watching a geek trying to explain this stuff in something resembling English. But Constant Hawk worked, and the troops wanted more of it. Tools like this were popular mainly because the system retains photos of areas troops have patrolled, and allows them to retrieve photos of a particular place on a particular day. Often, the troops returning from, or going out on a patrol, can use the pattern analysis skills we all have, to spot something suspicious, or potentially so.

Pattern analysis is one of the fundamental tools Operations Research (OR) practitioners have been using since World War II (when the newly developed field of OR got its first big workout). Pattern analysis is widely used on Wall Street, by engineers, law enforcement, marketing specialists, medical researchers and now, the military. Constant Hawk uses a special video camera system to observe a locality and find useful patterns of changing behavior. While many of the Constant Hawk systems were mounted on light aircraft, others are mounted on towers or other ground structures. Special software compares photos from different times. When changes are noted, they are checked more closely, which has resulted in the early detection of thousands of roadside bombs and terrorist ambushes. This largely eliminated roadside bomb attacks on some supply convoys, which travel the same routes all the time. Those routes were regularly watched by Constant Hawk. No matter what the enemy did, the Hawk would notice. Typically the enemy would shift their bombing efforts elsewhere.

Even as Constant Hawk entered service it was realized that such a system would be even more effective if there was a multiple camera system in a UAV or light aircraft that could detect useful patterns in real time. By 2010 there were two systems that attempted to do that; "Gorgon Stare" and “Angel Fire.” Both were similar to the Israeli SkEye in many ways but not as compact, capable and reliable. By 2010 there was not as much urgency for the Americans as their troops were being withdrawn from Iraq and were mostly gone from Afghanistan by 2014.

But the Israelis still had Islamic terrorists coming at them from several directions and needed better tech to secure its borders and seek out rocket launching sites (all along their Lebanese, Syrian and Egyptian borders) plus roadside bombs and anti-vehicle mines in wartime. Since Israeli firms were responsible for major advances in automated digital image analysis (for automatically checking lots of vidcam images for something specific) and UAV based sensors in general this led to ground based GroundEye in 2016 and the airborne SkEye in 2017.

Despite setbacks with early versions of Gorgon Stare and Angel fire there was an urgent need to deal with the shortage of human operators (of real time systems) and analysts (for non-real time). The human operators are expensive, subject to fatigue (and missing vital information at key moments) and of varying degrees of skill. For generations soldiers and police have been trained to observe and analyze what they see quickly and accurately for crucial information. Scientific studies of what is crucial and how well human observers can detect it was used by software developers to enable software to achieve a high level of accuracy constantly. The proliferation of large UAVs using vidcams to watch combat zones on a regular basis, not to mention the growing number of video surveillance space satellites created a growing backlog of unanalyzed digital video. Capable video analysis software was the only solution. This particularly true for American forces in Iraq and Afghanistan where searching for enemy activity was possible with all the video but crippled by the inability to analyze it all in a timely manner.

The manpower shortage problem became the prime motivation to develop automated monitoring and analysis. Digital image analysis software had been the next big thing since the 1990s but by 2014 there were some very effective systems in use, especially by the Israelis, for whom getting this right was a matter of life or death. South Korea monitored, and learned from Israeli experience in this area because, like the Israelis, they had long borders that had to be monitored. For South Korea it was mainly the DMZ as well as maritime borders that North Koreans had long used to insert commandos and spies. Similar situation with Israel, especially the Gaza border and to a lesser extent along the borders with Lebanon and Syria. South Korea was particularly interested in the Israeli use of border vidcams and remotely controlled machine-guns in fortified towers along the Gaza border. The use of video analysis software enabled the Israelis to greatly reduce the number of military personnel monitoring all the vidcams.

Now it is the Israelis observing the South Korean experience with their new real time video analysis system along the DMZ. Meanwhile the usefulness of the American airborne surveillance and video analysis systems can be seen in Iraq and Afghanistan, where locals don’t want a lot of American combat troops but do want the video analysis systems and intelligence analysis systems in general to stick around, to help keep an eye on things that go boom in the night and kill lots of locals.