OSINT and Momentum
I came across the following essay by John Robb (Brave New War, Small Wars Foundation Advisory Board) posted on his Global Guerrillas page. I found it interesting, thought provoking and worth sharing. John has granted Small Wars Journal permission to repost here:
Here’s a short essay on a new software tool my team built called Momentum. It does something pretty unusual and worthy of note. It uses crowdsourcing (by leveraging the work of 500 million users on global social networks) to make open source intelligence gathering easy.*
Here are some real life events: A YouTube video sparks massive ethnic riots in China. 3 Colombians start a Facebook group that puts 5 million protesters on the streets. The online manifesto of a disturbed man holding hostages at a US corporation. The list of keywords that China’s great firewall blocks. A YouTube video showing Iranian militia actually shooting protesters. The list goes on and on.
These are the kinds of events that should be detected by OSINT analysts. However, they aren’t detecting them. Why not? The tools they are using don’t make finding this information easy.
Here’s why. A major problem with the tools that automate open source intelligence gathering is that relevance, serendipity (the propensity of making fortunate discoveries while looking for something else), and timeliness (the ability to find content that matters quickly) are considered tradeoffs. You can’t focus on relevance without losing serendipity and you can’t focus on serendipity without losing timeliness.
For example:
I am gathering open source intelligence on Iran using the Web. But I run into the following problems with the tools I normally use:
If I search all available content, both traditional Web pages and social networks, using the keyword: Iran, I’ll be buried by the amount of content that is retrieved. Everything that includes the term ‘Iran’ will be returned to me as a ‘result.’ The time needed to sort the relevant content from irrelevant content and/or find surprising results will take inordinate amounts of time (even with the help of algorithms).
If I narrow that search to specific topics related to Iran (or keywords that are in proximity to the keyword: Iran) to reduce the time needed to analyze it, I lose any chance of serendipity — the chance of finding important trends going on re: Iran that I don’t know about.
These problems are fixed by a new software tool called Momentum. Momentum leverages global social network crowdsourding to automate OS intelligence gathering in a way that conserves relevance, serendipity, and timeliness. Here’s how:
Relevance is conserved. Momentum monitors social networks for the use of metatags — the keywords people use to describe a bit of content (video, Web page, etc.) that they link to. In other words, Momentum leverages the work of hundreds of millions of people (the people already using social networks globally) to categorize the Internet’s content for you. This categorization ensures that you only get results that are relevant to your topic.
Serendipity is conserved. Even a one-time use of a targeted keyword returns content. This means that even if only one person, out of hundreds of millions, sees a connection between a bit of content and the targeted topic, you get the results.
Timeliness is conserved. People on social networks gravitate to new and surprising content. When they find content that meets this criterion, they share it with their network. Momentum uses an index (calculated from indicators of use on Facebook, Twitter, Reddit, Digg, and YouTube over a rolling 48 hour period) to find the content that people think is surprising or new right now.
For more information go to our site. We’re still working on the site, so suggestions are welcome.
The tool is currently being sold as a service. Perhaps with a little success, we’ll be able to put together a version of the interface anybody can use.
If I had to build a tag line, this would probably be it. Momentum: The tool that puts 500 million social networking researchers on your team.
*NOTE: Another way of looking at it (for those of you familiar with the term OSINT), Momentum turns open source intelligence into a real and tangible process rather than just a description of the source’s availability.
