August 13, 2021: U.S. military leaders are urging the current laws be modified so that the OSINT (Open Source Intelligence) is recognized as a valid, often crucial part of what is really going on in foreign countries. Currently it is information collected by espionage and “technical means” (satellite and aerial surveillance and other forms of electronic eavesdropping) that is given the most weight when it comes to analyzing what the enemy is up to and what they plan to do.

One major example of the effectiveness of OSINT is how effective it is as used by commercial firms that provide BI (Business Intelligence) to customers. BI has been around since the mid-19th century when the mass media became available as a result of new technology (steam powered printing presses) that made mass market newspapers possible. In Western countries, especially the United States, where most of the population was literate and able to regularly purchase and read newspapers, it was quickly discovered that newspapers from enemy territory were an excellent source of military information in war or peace. This was particularly the case during the American Civil War where espionage efforts on both sides depended on getting newspapers from enemy territory to accurately assess what the enemy military was up to and what the state of civilian morale and economy was.

“News from foreign lands” was actually an ancient practice where “commercial letters” often became a business as skilled observers in foreign countries regularly sent letters to customers or partners that detailed the latest state of markets and local conditions. When the printing press became widely available in the 1600s, the first newspapers appeared. These were expensive but affordable to major businesses and the wealthy in general. Astute diplomatic and intelligence advisors learned that a regular supply of foreign newspapers or free material, often in the form of one-page documents distributed to the public, provided insights on political, military and economic trends.

The term “Business Intelligence” or “BI” first appeared in 1989 and was widely adopted by the latest incarnation of firms that provided collected, analyzed and reported on OSINT to their customers. While some BI firms used illegal means to steal information, this criminal activity, not true BI and avoided. In many countries new laws often criminalized some existing data collection practices. In some countries passing on local newspapers, especially translations of them, was often considered a crime. There are always gray areas in BI that rarely exist in government approved espionage.

Analyzing the massive amounts of OSINT available from newspapers, radio, television and the Internet, BI firms were often providing superior information to what the government intelligence agencies obtained using traditional espionage methods. In the 1970s, as the rapidly growing Japanese economy began shipping competitive products to Western markets it was discovered that the larger and more successful Japanese firms relied on BI and OSINT from foreign countries to detect customer preferences and market trends in the West faster than Western firms could. At first dismissed as another odd Japanese custom, the more adept Western firms realized this use of BI gave Japanese firms a crucial advantage in foreign markets where Japan was able to sell their products at premium prices, not competing on price as they had initially done. Other East Asian firms, and governments, adopted these techniques. China was a particularly avid practitioner.

The more effective Western intel agencies quietly adopted BI techniques, finding newspapers from the Middle East a particularly useful source of valuable political and military developments as well as a better sense of what foreign government and populations were thinking. This kind of OSINT required skilled translators and experts in foreign cultures to get the most out of this OSINT and by the 1980s it was a growing, if largely unrecognized, source of superior intel on friends and enemies overseas.

Another development was that since the 1990s ancient espionage techniques have become obsolete and 21st century spies have had to adapt to keep up. The old ways have largely been replaced with new methods that take advantage of new tech like the Internet, cellphones and more powerful and numerous computers along with new software that can do pattern analysis and automatic analysis of photos or video. This made it possible to regularly monitor “the street” or what the public was really thinking.

For spies, the most immediate impact of improved OSINT and BI was that it suddenly became much more difficult for them to hide their identities and activities. These new tools were most disruptive in police states where it had long been easy to control mass media, communications and free movement. It has taken several decades but some police states developed and implemented ways to deal with the new tech. China is the best example of this and that was no accident. China had the money, the tech and the trained (and loyal) personnel to tame these new technologies and bend them to serve the state rather than enable people to live more freely. Cellphones and the Internet along with the widespread use of security cameras proved capable of creating a surveillance and monitoring system that made it much more difficult to use traditional spies.

On the plus side, the World Wide Web has made OSINT more valuable and difficult to manipulate or suppress. OSINT has always meant using information available to the public. Even during the Cold War, everyone found OSINT useful, if at times tedious to use. With the Internet available, much better OSINT could be collected much more quickly and eventually analyzed quickly using software. Again, it was civilian BI and marketing firms that led the way in turning all this OSINT into commercially useful products.

China led the way by spending billions of dollars to wall off most of its citizens from those many aspects of the World Wide Web that enabled Chinese to find out what was actually happening elsewhere, including in their own country. China now sells this technology to other nations or provides it at a big discount for allies who want some modern police state tech to control their own populations. Dictators have found that they cannot just cut their country off from the Internet, as Cuba and North Korea did for a while, because of commercial and government need for Internet access. Moreover, the Internet has more ways to leak into a police state than can be blocked.

Perhaps the most notable loss to conventional espionage efforts has been the ability to use "Illegals." These are spies who do not have diplomatic immunity, like the "legals" or spies posing as embassy personnel, and can be imprisoned, or even executed, if caught. For over a century, the worldwide acceptance of “diplomatic immunity” for embassy personnel was a bonanza for espionage. With this diplomatic immunity, you could have some spies in the most restrictive police states. But all that new tech has rendered the “legals” much less effective because they are easier to detect and monitor. It was much easier and efficient to steal secrets via the Internet or by tapping into enemy communications wirelessly. Illegals are costly and more vulnerable because of the surveillance state and better search tools that make life difficult for "legals" and "illegals".

The United States military was able to use OSINT more aggressively after 2001 when there was a wartime atmosphere and new techniques were welcome if they worked. SOCOM (Special Operations Command) was particularly active in developing and using new OSINT techniques. These were still disdained by the professionals in the intel agencies. Out in the battle zone SOCOM and the American military in general used what worked and now that generation of young officers with first-hand knowledge of these techniques is asking that these techniques be recognized and used more by the traditional intel agencies.