Here's a short essay on a new software tool my team built called Momentum. It does something pretty unusual and worthy of note. It uses crowdsourcing (by leveraging the work of 500 million users on global social networks) to make open source intelligence gathering easy.*
Here are some real life events: A YouTube video sparks massive ethnic riots in China. 3 Colombians start a Facebook group that puts 5 million protesters on the streets. The online manifesto of a disturbed man holding hostages at a US corporation. The list of keywords that China's great firewall blocks. A YouTube video showing Iranian militia actually shooting protesters. The list goes on and on.
These are the kinds of events that should be detected by OSINT analysts. However, they aren't detecting them. Why not? The tools they are using don't make finding this information easy.
Here's why. A major problem with the tools that automate open source intelligence gathering is that relevance, serendipity (the propensity of making fortunate discoveries while looking for something else), and timeliness (the ability to find content that matters quickly) are considered tradeoffs. You can't focus on relevance without losing serendipity and you can't focus on serendipity without losing timeliness.
For example:
I am gathering open source intelligence on Iran using the Web. But I run into the following problems with the tools I normally use:
If I search all available content, both traditional Web pages and social networks, using the keyword: Iran, I'll be buried by the amount of content that is retrieved. Everything that includes the term 'Iran' will be returned to me as a 'result.' The time needed to sort the relevant content from irrelevant content and/or find surprising results will take inordinate amounts of time (even with the help of algorithms).
If I narrow that search to specific topics related to Iran (or keywords that are in proximity to the keyword: Iran) to reduce the time needed to analyze it, I lose any chance of serendipity -- the chance of finding important trends going on re: Iran that I don't know about.
These problems are fixed by a new software tool called Momentum. Momentum leverages global social network crowdsourding to automate OS intelligence gathering in a way that conserves relevance, serendipity, and timeliness. Here's how:
Relevance is conserved. Momentum monitors social networks for the use of metatags -- the keywords people use to describe a bit of content (video, Web page, etc.) that they link to. In other words, Momentum leverages the work of hundreds of millions of people (the people already using social networks globally) to categorize the Internet's content for you. This categorization ensures that you only get results that are relevant to your topic.
Serendipity is conserved. Even a one-time use of a targeted keyword returns content. This means that even if only one person, out of hundreds of millions, sees a connection between a bit of content and the targeted topic, you get the results.
Timeliness is conserved. People on social networks gravitate to new and surprising content. When they find content that meets this criterion, they share it with their network. Momentum uses an index (calculated from indicators of use on Facebook, Twitter, Reddit, Digg, and YouTube over a rolling 48 hour period) to find the content that people think is surprising or new right now.
For more information go to our site. We're still working on the site, so suggestions are welcome.
The tool is currently being sold as a service. Perhaps with a little success, we'll be able to put together a version of the interface anybody can use.
If I had to build a tag line, this would probably be it. Momentum: The tool that puts 500 million social networking researchers on your team.
*NOTE: Another way of looking at it (for those of you familiar with the term OSINT), Momentum turns open source intelligence into a real and tangible process rather than just a description of the source's availability.
Comments
What might be interesting here is to sit down a group of targeting CWOs who have just returned from either Irag or Afghanistan and then ask them of their opinion of the various link analysis tools and none have even heard of Palentir.
The powers to be that handle the implementation of tools should do that more often as they might just get a taste for what the field users really think of their tools vs the hype produced to sell them.
By the way any software produced by John Robb both walks and talks the walk and talk maybe because he understands what he is designing it for---not like the big boys---produce something and then sell it as a be all end all product that later fails to do even that.
Hello! I'm the re-seller of Momentum here in Oceania and I was directed to this thread to get some more context before I walked into the NZDF tomorrow morning to present it to them.
Now, I'm no programmer, and certainly no Sun Tzu, but I am an entrepreneur who's seen good times and bad, and a lot of talkers who don't necessarily walk.
The program does the job, and in fact I see this being used more lucratively by a clever stock trader than any public or military organization. That, and a media corp looking for content they can tag along on.
We have a product called Media Monitors here in Australasia, which, well, 'monitors the media' with real humans on behalf of organizations who care enough to pay for that service. Like a gentleman commented earlier - this doesn't make that service obsolete, but it sure as heck is a powerful tool for that company. (Whom I've started conversations with.)
For what it's worth, I'll keep everyone updated on what the NZ Defense force thought of it - unless it's classified or something. Apparently I'm not allowed any wireless devices in the meeting...makes demos difficult...sigh.
If you want me to propose a solution, Anon, you'll have to specify a problem. We've quite a few, each requiring its own solutions.
Certainly we need to monitor social networks and manage the data that such monitoring provides. Better ways of doing this are useful and welcome. They are not game-changers, they are not "the solution", and anyone who thinks they are sexy needs to get out more.
Better acquisition and management of social networking data will not compensate for our woeful lack of quality targeted human intelligence. It will not compensate for our woeful lack of area specialists who really understand the places and issues we're involved with. It will not compensate for our lamentably woeful reluctance to listen to the area specialists we have, especially when what they tell us is not compatible with pet theories and preconceived ideas. It will not correct or prevent boneheaded decision-making at the policy level.
You comment on my perceived <i>totally (sic) disregard for the fact that currently we have three link analysis tools in use, two social network analysis tools in use and five different targeting processes in use from Iraq to Afghanistan and still we cannot get the "who and what" accurately.</i>. You might consider the possibility that if barking up that tree isn't getting you what you need, you may not just need a better way of barking up that tree. You may want to have a look at some of the other trees in the forest. All of these techniques examine one piece of the puzzle, and while it's an important piece, if you focus on that piece alone you won't get the full picture. If link analysis and network analysis aren't getting you what you need, you may need better tools, and you may be coming up against the limitations of those techniques. We need to get all that IT can get us, but we also need to understand that won't be enough, and that there's a whole lot that we need that IT won't get us.
Useful tools, yes. "The solution", no. Keeping things in perspective and refraining from breathless hype actually leads to faster adoption and understanding of new processes. We've all seen "the next big thing" come and go, many times.
Consider also that buzzwords and taglines, jargon and hype, do not and are not meant to advance understanding. They're the intellectual equivalent of a dog pissing on a tree; a way for individuals to establish status and declare membership in an exclusive subset. Understanding and acceptance will be better advanced if all participants in dialogue make their points in plain language and without talking like salespeople. There's lots of people out there telling us that they know far more than we peons do. It's all terribly impressive, but show goes farther than tell. A calm, plain-language description of the problem you propose to solve and the way you propose to solve it is always a good beginning.
How many posts have we all seen on blogs like this that press a dozen or so fashionable buttons without actually saying anything?
Anon,
I think your comments were a little harsh. I think you need to consider that many folks legitimately concerned about being taken advantage of (or allowing our gov to be taken advantage of) by a modern day snake oil salesman. There have been a lot of false prophets out there in the IT land that didn't deliver on their promises. I'm not accusing John Robb of that, but I'm sure he understands that seeing is believing.
As you stated earlier, Palentir is a proven technology that is valued added, as are a few other tools that still haven't gone mainstream yet due to resistance to change, or worse the desire to wait for the next level of technology to be developed until they invest. We can play that game forever and remain paralyzed because we're waiting on XXX 2.0 or XXX 3.0
I think your comments on crowd sourcing and global movements enabled by networking are right on the money, and I'm not convinced we understand beyond a superfiscial level what that really means.
Dayuhan;
With your comments you tend to reflect a distinct lack of understanding of what Robb was pointing too.
Namely the coming movement globally is social networking and crowdsourcing and AQ has proven this beyond a doubt---not to sure what planet you are on.
Before your throw stones have you researched the IT background, technologies he has developed and the companies he has formed plus his AFSOF experience? If you have ever used RSS then you are riding his technology---he is thorough in what he approaches for solutions.
Then you can understand that he was not writing for you to understand -some of us do fully understand what he has done and some of us know fully how to integrate the tools.
Your comment-
I confess that when terms like "sexy", "quantum analysis", and "open-source" get thrown together in a single response, I'm torn between watching the red flags pop up and the need to stifle a yawn. A good tool doesn't need hype or hyperinflated buzzwords to support it. If it works, wonderful.
REFLECTS a totally disregard for the fact that currently we have three link analysis tools in use, two social network analysis tools in use and five different targeting processes in use from Iraq to Afghanistan and still we cannot get the "who and what" accurately.
If you thoroughly know the defense contracting world you would know that the best and brightest tools never make it past the incumbent defense contractor who is pushing usually his own tools---but again I am assuming you know that.
So Dayuhan just what is your solution?--throwing around comments distracts from solutions and solutions that were really needed six years ago.
I agree that some of the eccentric youth might make very useful recruits, and that their skill sets are in many ways remarkable... but I doubt that the verbiage in question would get very far with them. I've a marginally eccentric 20 year old physics major son who can do things with a computer that I'd never have dreamed possible, despite an absence of colored hair and nose rings. In his (global) circle hype is death, deadpan understatement is the order of the day, and anyone who described a process as "sexy" would be thought quite ridiculous.
Agreed, not quite sure what target audience they're trying to address with those trigger words, the 19 to 22 y/o hacker or the intelligence professional? I think it was GEN McCrystal who said we start to start recruiting some these kids with nose rings, brightly colored hair, etc. so we can tap into their unique skill sets and view of the world. This may be one of those cases.
True enough, to an extent. Tools are useful things, and it's silly not to use them where they are useful. If we fall for the hype and rely too much on the tool and not enough on the operator, though, we do ourselves no favors.
I confess that when terms like "sexy", "quantum analysis", and "open-source" get thrown together in a single response, I'm torn between watching the red flags pop up and the need to stifle a yawn. A good tool doesn't need hype or hyperinflated buzzwords to support it. If it works, wonderful.
Dayuhan, I find your response is typical to what I find in the military to a new idea or technology. Yet, despite the large bureaucracy of naysayers the military has still managed to evolve (that is the mystery to me). The fact is computers already filter through GBs of data daily in support of intelligence operations, and sort through the GBs of data to support business intelligence (they find it useful). The sad part in a way is this type of technology isn't new, it has been proven to work, it is simply being applied in a new way (useful to those of us interested in emergent social movement). You and I, even if we collaborating can't effectively monitor the volume of traffic on a particular topic on the various social networking sites. We can look up stuff after the fact, but it isn't possible for a human to sort through this volume of data and make the connections. Technology doesn't replace humans, it augments their capabilities. Assuming Robb's program works it would provide useful insights to the intelligence community that they could further explore. It would show potential trouble or opportunities brewing that otherwise wouldn't be visible until it was a history lesson.
The cynic in me says this sounds like a bit of advertising for a service that's being sold. A demonstration would be better than a description.
I do a great deal of research on the internet, though I'm nowhere near pompous enough to call it "open source intelligence gathering". The ability do develop and refine effective search strings, along with a certain amount of experience and a bit of craft, finds you what you want and need. The person behind the computer is more important than the software in it. Field experience and contextual knowledge in the specific area of interest is critical... that's what builds up the integrated BS filter.
Sexy intelligence processes have a lot in common with sexy people. They look great on the surface, but when you spend some time with them you generally find that the substance doesn't live up to the surface.
It's a lot like financial analysis. Step 1 is to strain out and discard all the sexy surfaces, the buzzwords, taglines, and self-justifying theories, toss all jargon, all hype, all gloss in the bin where it belongs. Then you can take what's left and begin the process of contextually informed analysis... assuming, that is, that you have the contextual knowledge that you need. If you don't, better go do something else, because you won't generate any useful product in any event. Salable, maybe, but not useful!
Tie John's tool to Palentir and tie both to a recent quantum analysis research model and you can drive Dr. Kilcullen's "conflict ecosystem" model literally around the world as you now have the basis for "open source warfare analysis" thus a sexy 21st intelligence process that is both easy to learn and will remain current regardless of global events as you are literally "ahead of curve". And proactive is where one has to be---this reactive thing is simply killing us.
Think about it.
Welcome to the brave new world of "open source warfare". There are some who would say that in our intelligence analysis of anything-by not actively including everything on the OSINT side one tends to miss 30% of all chatter going on at anyone time.
NOW tie the Robb tool to a link analysis tool called Palentir and everything else on the link analysis /social network analysis side pales in comparison.
Bill --you are right-not so sure the powers to be are ready for this level of understanding as it opens up the world of "wicked problems".
And understanding is where everything has to go---even the standard targeting model of D3E is changing to UD3E with the U meaning "understanding"---this type of tool will get us there faster.
This type of technology is impressive and very useful if put in the right hands (adaptive analysts who understand the current operational environment). The challenge with any of these technologies is that is that it actually transforms our way of doing business, so there is an automatic cultural backlash by the establishment bureaucrats, and then a reluctance by many analysts to adopt and apply the new technologies, because "they" think what they're doing now is just fine, although failure after failure indicates there is ample room for improvement. Of course this reinforces the arguments of the bureaucrats who are always happy with the status quo. In short I'm simply pointing out the obvious, and that is change is hard. I only hope this technology is adopted quicker than many other good innovative ideas and technologies I have seen linger out there in indecision land for year after year.