Radical Transparency and National Security

Clive Thompson has an excellent piece in today's NYT Magazine that asks whether blogs, wikis and other social networking technology could do a better job of intelligence gathering than the traditional methods of the spy agencies. The simple answer is that they could hardly be worse--the article is sobering reminder of how ancient and feeble existing intelligence technology, such as it is, really is. The databases don't talk to each other, the instant messaging doesn't interoperate and classified information gets accidentally released because the filters, appallingly, weren't programmed to catch "Secret" in all-caps or with spaces between the letters. The average spam filter is apparently light years ahead of the state-of-the-art in military data analysis.

But the really sobering thing is Clive's reminder that this is not just due to incompetence and bureaucracy. US intelligence is also intentionally hobbled by laws designed to protect privacy and civil liberties. Total Information Awareness probably would have failed on technology grounds, but it never even got a chance to prove anything one way or another because it ran afoul of laws and politics first. Since we think those laws are Good Things, Clive asks whether a more open alternative--the Web 2.0 tactics of peer-production and the wisdom of crowds--within the intelligence services are a better approach. Rather than trying to work around the laws or building increasingly complex and fragile systems, why not tap the collective intelligence of the people already in the system operating under the existing rules by simply letting them communicate better with the same tools that made the web what it is today?

His thesis is that radical transparency within the intelligence services could compensate the failures of command-and-control information sharing. Perhaps it could. But I was surprised that Clive didn't take the question to the obvious next level. What if, rather than just starting blogs and wikis behind military firewalls where the rules are most strict, the intelligence agencies encouraged them out in the open, catalyzing conversations between people who aren't constrained by the same laws? Between the intelligence analysts, military bloggers, and intelligence technology providers, there are already loads of spook-types blogging in public. They're not writing about classified stuff, of course, but there is plenty of information in the unclassified world that could lead to useful intelligence if only it were easier to connect the dots. Given enough eyeballs, all bugs are shallow.

In other words, what if spotting early-warning terrorism signals became an open-source project?

That's not such a crazy idea. It is, after all, the core thesis of David Brin's increasingly-prescient "The Transparent Society". It's also quite brilliantly imagined in the beginning of Rainbows End, the new science fiction novel by Vernor Vinge. Here are the first four sentences of Vinge's book:

"The first big of dumb luck came disguised as a public embarrassment for the European Center for Defense against Disease. On July 23, schoolchildren in Algiers claimed that a respiratory epidemic was spreading across the Mediterranean. The claim was based on clever analysis of antibody data from the mass-transit systems of Algiers and Naples.

CCD had no immediate comment, but in less than three hours, public-health hobbyists reported similar results in other cites, complete with contagion maps." 

Silly sci-fi stuff? Compare it with the real-world example cited in Clive's story:

"In July, [the Director of National Intelligence] staff decided to create a test blog...that would focus on spotting and predicting possible avian-flu outbreaks and function as part of a larger portal on the subject to collect information from hundreds of sources around the world, inside and outside of the intelligence agencies. An agent in Southeast Asia might be the first to hear news of dangerous farming practices; a medical expert in Chicago could write a crucial paper on transmission that was never noticed by analysts.

In the months that it has been operational, the portal has amassed 38,000 “active” participants, though not everyone posts information. By September, the site had become so loaded with information and discussion that Rear Adm. Arthur Lawrence, a top official in the health department [said] it had become the government’s most crucial resource on avian flu."

UPDATE: Clive reminds me that one of the best examples of this is the community and services formed around Oregon and California's Emergency Digital Information Service project. And the best story on that is Gary Wolf's seminal piece in, er, Wired a year ago (I knew that!). "Reinventing 911: How a swarm of networked citizens is building a better emergency broadcast system." Read it.

UPDATE2: Blaise Zerega (ex Wired, now Portfolio) reminds me of another prescient Wired story. "We Need Spy Blogs: An Army officer calls for better information gathering"

Comments

[Mike Abundo]

While social intelligence can be a tool to save lives, I would caution against creating a paranoid blogosphere.

[jkd]

"While social intelligence can be a tool to save lives, I would caution against creating a paranoid blogosphere."

I'd say the blogosphere is already pretty durn paranoid - that's one of the things makes it so interesting and effective, the propensity to openly question dominant and accepted narratives.

As to the larger point, I agree entirely about the open-sourcing of intelligence. One of the best things that DoD has ever done in this context was opening up GPS to the public, a measure which has had positive consequences far beyond what the system was originally envisioned for. There are still many things that it's better to have restricted access for - e.g., Iraq's pre-1991 nuclear schematics - but there's also a lot of information that can be of great use to citizens, and the presumption ought to be towards openness rather than secrecy.

[Paul DiPerna]

Intellipedia is one step forward to address the larger issue of reforming the U.S. Intelligence Community culture. By what I've been learning the past few months, the wiki technology should improve information-sharing and transparency. Early reviews have been cautiously optimistic to enthusiastic.

By crazy coincidence of timing, this past Friday I had a brief article published in the Washington Examiner on this topic. Not out for self-promotion, but thought maybe something of interest to folks here:

http://www.examiner.com/articlePDF.cfm?articleID=430145

[Gunnar]

I have a series of posts on this topic here

http://1raindrop.typepad.com/1_raindrop/2006/10/decentralizatio.html

which ties in some of the work Ross Mayfield and JASON and others are doing in this space, more from a business perspective than a national security perspective. In both cases, decision making is decentralized and the amount of information is huge, so the concepts have a lot in common.

[David Hume]

Note also the Global Public Health Intelligence Network, located in the Public Health Agency of Canada. From its website at
http://www.phac-aspc.gc.ca/media/nr-rp/2004/2004_gphin-rmispbk_e.html:

"GPHIN is a secure, Internet-based "early warning" system that gathers preliminary reports of public health significance in seven languages on a real-time, 24/7 basis.

This unique, multilingual system gathers and disseminates relevant information on disease outbreaks and other public health events by monitoring global media sources such as news wires and web sites. The information is filtered for relevancy by an automated process, and then analyzed by Public Health Agency of Canada GPHIN officials. The output is categorized and made accessible to users. Notifications about public health events that may have serious public health consequences are immediately forwarded to users."

I believe Larry Brilliant said in a TED talk of his that GPHIN found SARS 6 weeks before the WHO did.

Amazing stuff, and makes me proud to be Canadian.

[Business for sale]

Yes wiki technology should improve information-sharing and transparency!!!!!

We are from http://www.worldbusinessforsale.com

[Karl Schroeder]

I've done some work around this; at last year's Prospective Protective Security Futures workshop in Ottawa, I proposed something I called "wiki-security". I expanded on the idea in the fiction piece I wrote for the conference proceedings, which you can read here on WorldChanging Canada.

We had found during the conference that current national security systems have a human bottleneck--not enough eyes. So the security experts themselves are seriously discussing this stuff.

[Chris Crosby]

This track is dead on. I've wondered how effective an open source multilingual version of "google alerts" would be at spotting keywords on websites and blogs around the world. With the vast array of bloggers arising globally we could create a more effective intelligence system than our own government's.

[Greg Banville]

Frankly, this apprears to be the best possible solution to the problem of modern national security needs.
Now if we just had more effective open networks for tracking, preventing and redressing violations of civil liberties by the government against individuals.