August 11, 2006: British intelligence services admitted that they had a program that collected and analyzed data from thousands of web sites and databases. Britain pioneered the use of such open-source analysis in the 1930s, although political activists in the United States have attacked American efforts in this area on the grounds that it threatens individual privacy. As a result, in both Europe and the United States, such efforts are now carried out with as little publicity as possible.
Companies, especially marketing research firms, have long performed the same kind of work that the activists attack intelligence agencies for doing. Activists have much less success trying to halt private research efforts, for these can be easily protected by lawyers, publicists and lobbyists. Attacks on similar government programs are political poison, and elected officials will back away. Invoking the specter of government misuse of databases is usually all it takes to get intelligence efforts cancelled. This has turned into a bonanza for private research firms, who get paid big bucks to set up and run the classified research efforts that examine open source data for links that will lead to terrorists or other criminal activity.
The British revelations included admissions that agents and analysts trolled the Internet, talking to people in chat rooms and discussion boards in their search for useful information. But the most useful work was done with software that was able to uncover patterns. In this way, links between terrorists and supportive individuals and organizations, could be found. It's very difficult for terrorists to avoid discovery via this kind of analysis, although if terrorists know details of the methods used, they could develop techniques that would make the searches less effective. For this reason, the intel people like to keep their work quiet.