The Internet has become a major source of useful intelligence information, even though the people leaving messages on the web dont believe they are talking out of class. The U.S. Department of Defense, NSA, NRO and CIA have long worked at creating tools to pull useful information from the masses of data they accumulate. In the past there have been mainframe and workstation based systems, that tended to get overwhelmed by the sheer mass of data to be searched. As a result, these earlier attempts at data mining, or simply data query, were not terribly successful. Although these systems were classified, eventually they word got out on some of them. 

In the last decade, computers have gotten much more powerful, and cheaper. Then the Internet came along, and before long there were a billion people on it. This had a largely unforeseen impact on the intelligence business. In the last few years, the U.S. government has made reference to Internet chatter as proof that al Qaeda was up to something. What few people realize is that there is probably a new generation of search tools the intel people are using, that work much more effectively than the old ones from the 1980s (and earlier.) How do these tools work? That can be seen from an examination of commercial versions of this software that are beginning to appear. For more than a decade, financial firms developed data mining tools for the news, the better to get advance warning of changing trends. These search systems were generally kept secret, as they were very lucrative tools for those who worked the markets in stocks, currency and related items. Some of these systems now mine the Internet as well. 

Now an Israeli firm, Trendum, comes along and offers to sell anyone the ability to mine the Internet. Openly using the same tools (neural networks, rule induction, decision trees, and pattern analysis, text and linguistic analysis) everyone else has been using in secret , and propelled by lots of powerful, but cheap, computer hardware. Trendum mines the Internet for developing trends. Major corporations, news organizations, and the UN, are lining up to use Trendums services. What Trendum will reveal is what the chatter of thousands, or millions, of people (depending on the subject) indicates. Most questions are of commercial significance, like how popular a new car, beer or movie is. But such systems can also reveal if terrorists are really about to make some attacks, or are just chattering.