Clive Thompson has an excellent piece in today's NYT Magazine that asks whether blogs, wikis and other social networking technology could do a better job of intelligence gathering than the traditional methods of the spy agencies. The simple answer is that they could hardly be worse--the article is sobering reminder of how ancient and feeble existing intelligence technology, such as it is, really is. The databases don't talk to each other, the instant messaging doesn't interoperate and classified information gets accidentally released because the filters, appallingly, weren't programmed to catch "Secret" in all-caps or with spaces between the letters. The average spam filter is apparently light years ahead of the state-of-the-art in military data analysis.
But the really sobering thing is Clive's reminder that this is not just due to incompetence and bureaucracy. US intelligence is also intentionally hobbled by laws designed to protect privacy and civil liberties. Total Information Awareness probably would have failed on technology grounds, but it never even got a chance to prove anything one way or another because it ran afoul of laws and politics first. Since we think those laws are Good Things, Clive asks whether a more open alternative--the Web 2.0 tactics of peer-production and the wisdom of crowds--within the intelligence services are a better approach. Rather than trying to work around the laws or building increasingly complex and fragile systems, why not tap the collective intelligence of the people already in the system operating under the existing rules by simply letting them communicate better with the same tools that made the web what it is today?
His thesis is that radical transparency within the intelligence services could compensate the failures of command-and-control information sharing. Perhaps it could. But I was surprised that Clive didn't take the question to the obvious next level. What if, rather than just starting blogs and wikis behind military firewalls where the rules are most strict, the intelligence agencies encouraged them out in the open, catalyzing conversations between people who aren't constrained by the same laws? Between the intelligence analysts, military bloggers, and intelligence technology providers, there are already loads of spook-types blogging in public. They're not writing about classified stuff, of course, but there is plenty of information in the unclassified world that could lead to useful intelligence if only it were easier to connect the dots. Given enough eyeballs, all bugs are shallow.
In other words, what if spotting early-warning terrorism signals became an open-source project?
That's not such a crazy idea. It is, after all, the core thesis of David Brin's increasingly-prescient "The Transparent Society". It's also quite brilliantly imagined in the beginning of Rainbows End, the new science fiction novel by Vernor Vinge. Here are the first four sentences of Vinge's book:
"The first big of dumb luck came disguised as a public embarrassment for the European Center for Defense against Disease. On July 23, schoolchildren in Algiers claimed that a respiratory epidemic was spreading across the Mediterranean. The claim was based on clever analysis of antibody data from the mass-transit systems of Algiers and Naples.
CCD had no immediate comment, but in less than three hours, public-health hobbyists reported similar results in other cites, complete with contagion maps."
Silly sci-fi stuff? Compare it with the real-world example cited in Clive's story:
"In July, [the Director of National Intelligence] staff decided to create a test blog...that would focus on spotting and predicting possible avian-flu outbreaks and function as part of a larger portal on the subject to collect information from hundreds of sources around the world, inside and outside of the intelligence agencies. An agent in Southeast Asia might be the first to hear news of dangerous farming practices; a medical expert in Chicago could write a crucial paper on transmission that was never noticed by analysts.
In the months that it has been operational, the portal has amassed 38,000 “active” participants, though not everyone posts information. By September, the site had become so loaded with information and discussion that Rear Adm. Arthur Lawrence, a top official in the health department [said] it had become the government’s most crucial resource on avian flu."
UPDATE: Clive reminds me that one of the best examples of this is the community and services formed around Oregon and California's Emergency Digital Information Service project. And the best story on that is Gary Wolf's seminal piece in, er, Wired a year ago (I knew that!). "Reinventing 911: How a swarm of networked citizens is building a better emergency broadcast system." Read it.
UPDATE2: Blaise Zerega (ex Wired, now Portfolio) reminds me of another prescient Wired story. "We Need Spy Blogs: An Army officer calls for better information gathering"