The Long Tail

Update

It's been a busy past two weeks. Most of it was in Chicago enjoying the fruits of a year's work at the Wired NextFest, our World's Fair-like festival  of technology, which was a huge success. We had 35,000 visitors, up from 25,000 last year in San Francisco. Twice the space as last year, half again as many exhibits (more than 120 in all), and three times as well organized. Next year: NYC, which will be bigger yet.

There are loads of pictures here if you want to get a feel for it. My favorite exhibits included David Hanson's robotically animated head of Philip K. Dick, complete with voice recognition and pre-programmed AI of all things Dick.  Spooky. Another mesmerizing display was a pair of Juke Bot DJ robots, which were beautifully choreographed, even if you couldn't (yet) dance to their mixes. And the leaping dolphin submarine was the talk of the show.

Next week I begin a two-month sabbatical to actually write the book. In the meantime, I've got a half-dozen posts to add here on some of the new ideas and research we've been working on for the past two weeks. I hope to get to them over the weekend, so stay tuned.

Why is this blog so, well, wonky?

When the supersmart members of the Stanford MBA team that's helping with the book research don't quite follow my posts, I know I've gone down the rabbit hole of obscurity.  (I think it was the signal/noise debate that did it).  But their polite bafflement was a useful reminder not to get too abstract here. It also raises the question of what level to pitch the discussion in the Long Tail book.

Should the book be series of gripping narrative stories around a phenomena, like The Tipping Point? (If only!) A collection of historical examples and academic research to illustrate an overarching theory, like The Wisdom of Crowds?  Or a relatively straightforward, data and anecdote-fueled articulation of a big thesis, like Out of Control? I'm leaning towards the last of those, mostly because that's what comes most naturally and fits with where my research is going.

I wish I were as good a storyteller as Malcolm Gladwell and that my tales were as narratively rich. But I'm not, and they're not. Instead, what I hope to offer is clarity, via data and a decent grounding in economics, lightened with loads of fun case studies and telling anecdotes.

In the meantime, a slight explanation for why I've been indulging in so much theory here. I originally trained to be a physicist, in part because my hero growing up was Richard Feynman. One of the virtues of physics is that it's based on the concept of understanding the world via first principles, the underlying rules that explain all the complexity around us.

Before we came up with the chemical table of the elements and understood the underlying rules of valence theory and atomic structure, the properties of matter were impossible to understand. Likewise for the movement of the planets and stars before Newton's Laws. Today biology remains brain-meltingly complicated largely because we don't have a good underlying theory for it.

What I'm trying to do here is to establish the first-principle rules of the Long Tail. I realize that the search for a grand unified theory is usually a recipe for ending your days muttering at a blackboard covered in scribbles. But I do think that the economics of abundance are poorly understood, and the Long Tail is as good an opportunity as any to lay out some pointers to how they might work. With your help, we'll work through some of that here and I'll find a way to make it easier to digest in the book.

To fork or not to fork

Since I wrote admiringly of the LA Times experiment with a blogish revamp of their editorial page, I am compelled to note that one of the most interesting parts, the "wikitorials", has failed, at least for now.  Jeff Jarvis describes some of the reasons, but at the root it appears that they simply got one of the architecture of participation calculations wrong. (Note: I just created that Wikipedia entry myself. How ironic is it that Wikipedia didn't have an entry for the architecture of participation, when it itself is the best example?)

The question in all open participation projects is how to deal with dissent. Should you seek rough consensus or should you "fork", letting each group go their own way? Wikipedia deals with this pretty gracefully by letting a million entries bloom. If you don't agree with an entry and the normal process of collective editing is diverging into acrimony rather than converging on a compromise, then the policy is this: at the point of disagreement note that there is a controversy and take each argument to its own sub-entry, curated by those who care.

This works great where multiple views can coexist without diminishing the overall goals of the project. If you don't like this post, you can write your own and the collective wisdom of the blogosphere will probably be better for it. But in other projects, such as Linux and similar open source software, there is often real benefit to having one commonly accepted version or standard around which others can build.  Which is why Linux has chosen the rough consensus model to minimize the problems of forking (at least in the kernel) and has thus largely avoided the balkanization that plagued Unix.

As best as I can tell (and it's hard to reconstruct now, because they've disappeared the wikitorial entirely) the mistake the LA Times made was to try to apply a rough consensus model to the wikitorials, when a forking model would have been more appropriate. Which is odd, because one of its other innovations was to encourage dissenting views and open debates between opposing sides of its editorial board and other official contributors. Surely what's good for the board is even better for the public?

Anyway, I'm glad to hear that Jimmy Wales, the Wikipedia founder, is advising them on how to revamp this.  It may be that wikitorials are simply a dumb idea, but we won't know until they at least try to architect them better.

UPDATE: Ross Mayfield has a great roundup on how, indeed, the wikitorials did eventually fork; Jimmy Wales set up a "counterpoint" version on the LA Times site. Sadly, it's gone now, too. Based on these unfortunate samples found in the Slashdot discussion of this experiment, they probably made the right call shutting it all down until they could think it through a bit more.

Six-month anniversary stats

Today is the six-month anniversary of this blog's public life. It's been hugely fun so far, and very encouraging as an experiment in sharing book research, thinking and some writing (or at least phrasing) in public. Thanks to the extraordinary quality of the comments, emails and other feedback I've received on my posts, I think the book is going to be far better thought-through than I could have made it myself. I won't go as far as to call it open-source, but there is definitely a hint of the architecture of participation at work here.

This month has also seen a few other gratifying blog milestones:

• 100 posts (about 4 a week, on average; I'm no Instapundit)
• 1,000 comments
• 1,000 Bloglines subscribers to the RSS feed. (I have no idea how many I have through other readers)
• 1,000 daily visitors on average to the site over the entire six months (more recently, the daily average has been more than 3,000).
• 10,000 visitors on a peak day

My average post is about 700 words long, which means I've written about 70,000 words for this blog over the past six months. Which, coincidentally, is the target length of the book. So either I will be using a reasonable fraction (20%?) of the words here in the book, or over the short nine months of the project (from the end of Dec, when the deal was done, to my due date in September) I will have written the equivalent of not one book but two. In my spare time. Which seems, well, nuts. Needless to say, I hope the synergies between blog and book extend beyond the quest for intellectual rigor and to actual, you know, writing. Or I'll be hosed come September.

My thanks to all of you, who make this so worth doing. Even today I am still amazed and thrilled at how rewarding it is to give away my time, ideas and research here, because I get back so much more in return. The power of the gift economy is truly remarkable, and we've just begun to see all the places it can work.

Massively parallel culture

One of my chapters is focused on how the rise of niche culture is reshaping the American social landscape. We're leaving the water-cooler era, when most of us listened, watched and read from the same, relatively small pool of mostly hit content, and entering the microculture era where we're all into different things.

I was struck by the recent example of Anil Dash, who hacked the NYT by wearing a GOATSE t-shirt in a photo shoot for an otherwise innocuous article about how hard it is to change what Google says about you. 

Most of my geek friends know what GOATSE refers to, because it's a retina-scarring shock-pic that Slashdot trolls try to get noobs to click on by claiming it's a link to something irresistible, like a picture of Natalie Portman or a hot Linux distro (for an explanation of this online hazing ritual, see this exhaustive, brilliant and very nearly worksafe Wikipedia entry, which, by the way, you will not find in Britannica. Ever.)

But I was amazed to find out that almost none of my staff (and obviously no NYT editors) knew about it. So I tried a few other cultural references that have become clichés in my little world: "All Your Base Are Belong To Us"; "More Cowbell!"; "I for one welcome our new [fill in the blank] overlords", and so on.

Turns out that these snippets of culture that I thought were ubiquitous are actually pretty obscure even in my own office. And when I took an informal poll at a PR conference I was speaking at I found that only about 10% of the audience had heard of any of them, and for each phrase it was a different 10%. My tribe is not always your tribe, even if we work together, play together and otherwise live in the same world; same bed, different dreams.

If you check out the Wikipedia entry for Internet phenomena you'll find hundreds of these viral memes. Here are ten of the most famous. Have you heard of all of them?

• Ellen Feiss
• The Star Wars Kid
• Dancing baby
• Bert is Evil
• Bonzai Kitten
• Tourist Guy
• MC Hawking
• Leet speak
• Subservient Chicken
• First post

Here's my take on what the Long Tail is doing to pop culture. Rather than the scary fragmentation of our society into a nation of disconnected people doing their own thing, I think we're reforming into thousands of cultural tribes, connected less by geographic proximity and workplace chatter than by shared interests. Whether we think of it this way or not, each of us belongs to many different tribes simultaneously, often overlapping (geek culture and Lego), often not (tennis and punk-funk).

What's interesting is that the same Long Tail forces and technologies that are leading to an explosion of variety and abundant choice in the content we consume are also helping to connect us to other consumers, whether through Amazon and Netflix reviews, blogs, p2p networks or playlist sharing. I've described this in the past, somewhat obscurely, as the rise of orthogonal trust networks, which are the new recommendation and word-of-mouth effects that will drive demand down the tail from hits to niches.

As a result, we can now treat culture not as one big blanket but as the superposition of many interwoven threads, each of which is individually addressable and and connects different groups of people simultaneously.

In short, we're seeing a shift from mass culture to massively parallel culture. This is a big deal, and I'll be writing more on it in the posts to come.

Earcandy

For those who are into such things, here are some recent Long Tail podcasts:

PodTech.net:

Chris is a [redacted] who is doing some important research on the effects and economics of the Long Tail trend. Towards the end he speaks volumes about the role of open source and open source media.

Impact on Marketing and PR: Traditionally marketing communications has been a function of pr and marketing but now it’s becoming increasingly a function of blogs, podcasting, word of mouth and other network effects. Now marketing and PR executives have to figure out how to use their skills that have traditionally been mainstream mass market and mainstream media skill set - contacting a handful of journalists. The goal of professional marketers is to translate those traditional skills to an era where the number of people you want to influence is in the thousand and millions of users and it’s a peer to peer influence marketing rather than a journalist to reader marketing.

The Gadget Show:

I’d talked about The Long Tail so much over the last few months that it made sense to get Chris Anderson, the man behind the original article, on the show to chat about the concept. So this week's interview isn't just about gadgets, it can be applied across a number of areas. Well worth a listen to experience what is driving our culture today.

[Most amusing part:

"23:45   Skype drops the call."

Ten days pass...

"24:00   Chris returns to the show."]

Marketplace:

Tom Standage, author of this Economist piece on the Long Tail, gave an excellent overview of the theory and its practice May 10th on Marketplace, which is broadcast daily on most NPR affiliates.

Listen to the 10:50 cast; the Long Tail piece comes on at the 5:25 mark.

Michael Kinsley, reanimator

Yesterday the New York Times asked me to comment on Michael Kinsley's reimagination of the LA Times editorial page for this article. But being on West Coast time (and spending the afternoon frolicking with the kids at the pool) I responded too late and missed the deadline. Since the first rule of the blogosphere is that no thought should go unpublished, I am duly required to post it here instead.

Kinsley (full disclosure: I wrote for him a few times at Slate and the New Republic, and we're both Economist alumni) has one of the tougher problems in media transformation. The newspaper op-ed pages may have once been the main forum for national debate, but the marketplace for opinion has changed over the past decade. Between talk radio and blogging, there has been an absolute explosion of punditry, rendering the op-ed pages largely irrelevant. What to do?

His solution is a mix of public debate and public participation, including something he calls a "wikitorial" that will allow readers to revise editorials to their liking online. The editor's note isn't long, so you might want to read it to get a flavor of the plan. My take is that it seems like a healthily radical experiment to try, and about the only thing I could imagine that would get me to read a newspaper op-ed again.

If you'll forgive the blog triumphalism, Kinsley's plan seems to draw heavily from blog-culture conventions. Examples include: 

• A human (ie, non-institutional, non-bossy) voice, ranging from the editor's note itself to A SoCal Life, which is meant to be personal reflections on LA living.
• Publicly acknowledging internal disagreement. 
• Encouraging audience participation (wikitorials). 
• Links to comment elsewhere, in the form of critiques of other newspaper op-eds. 
• A willingness to offer incomplete thinking, acknowledged as such, on the hope that the public exercise will result in a better exploration of the issues.

It remains to be seen if this will actually work, bringing life to a dull page in a declining medium. But if anyone can pull it off, it's Kinsley, the founding editor of Slate and a refreshingly independent thinker. I suspect I'll never see that page in its paper form again, but it would be great to see more of the paid pundits of this world duking it out online.

Bring tha noize!

Following my original post on the subject, I've been having some interesting discussions with smart folk about signal-to-noise ratios in the Long Tail. Those ratios are important because they dictate consumer behavior. Too much noise, and people don't buy. And without good filters (from search to recommendations), the Long Tail is just noise.

This may seem a bit arcane, but as I'm writing a whole chapter on filters, recommendations and other tools and techniques to drive demand down the Tail, it's worth drilling down a bit more here.

John Hagel thought I'd gone off the rails a bit with my analysis. He's as bright as they come, so I've gone to more than the usual lengths to reconsider this issue. Here's our debate, which I'll share with all of you because I think it eventually led to an interesting insight.

I originally posted this conceptual graph:

In a post, John took issue with it:

[Chris] asserts that signal to noise ratios decrease as one move down the tail.  Really?  Isn't that subjective?  I may be a real outlier (aren't we all?) but, at least for me in the realm of music, the signal to noise ratio decreases as I move up the tail. The real point, I think, is that the sheer quantity (rather than the quality) of items increases as we move down the tail and the ready availability of information about these items diminishes - that's what increases the difficulty of connecting with relevant resources as we move down the tail.

We batted this back and forth in several emails, but I didn't make much headway. John's taste in music is quite niche, he says. He doesn't like anything in the top 100, or even much in the top 1,000. And he suspects that there are lots of people like him (not necessarily in his particular niche, but in ones like it). As a result, he argues that his s/n ratio actually goes the other way, which is to say that it's zero at the head (no signal), peaks somewhere in the middle or even further down, and only then eventually falls under the weight of a zillion garage bands at the end of the  Tail.

I tried to persuade John with yet more conceptual charts to explain that shape I drew was the aggregation of everyone's s/n ratios, which are indeed all different but together end up looking like the one I originally posted. Like this:

But he was still unconvinced. This was surprising, since I think it's totally straightforward: there's more noise in the tail because there's more everything there. Most stuff doesn't sell very well, so the volume of the material available--and by extension the volume of stuff you don't want--rises as the Long Tail falls. Like this:

Whatever you're looking for, there's more stuff you aren't looking for the further you go down the tail. Which is why the signal-to-noise ratio gets worse, even if you're more likely (with good search and filters) to find what you want as you go down the tail.

It sounds like a paradox, but it isn't. Much of what you want is in the tail. Most of what you don't want is also in the tail. That's why you need increasingly powerful filters to extract the good from the greater bad.

But conceptual graphs weren't doing it; John was still skeptical. So I turned to actual data. I analyzed the music collection of five people: Me, Anne (my wife), my assistant Peter Arcuni, John himself (turns out that he's into rockabilly, surf music and Algerian "rai music", which seems to be some sort of ethno-techno thing), and Koranteng Ofusu-Amaah, for no other reason than that he showed up amongst my trackbacks and appears to have quite fringy music taste (African pop, mostly) that he was kind enough to provide Amazon links to. I've got Koranteng twice: once, for a random list of things he listening to, and the second for his best of the year list.

Overall, the number of albums that I included from each subjects' collections ranged from a few dozen to more than 200 titles. (This is too small to be anything more than suggestive. But I'm working with some companies to extend this analysis to proper large-n data sets, which I hope to be able to include in the book.)

This is what I found (using logarithmic curve-fitting to smooth the underlying numbers):

What's important to note here is that everyone, no matter how niche, shows a falling s/n ratio as you go farther down the tail. Why is this, when John, Koraneng and I  (indeed, all of us except for Anne) have no albums in the top 100, and only a few in the top 1,000? Why did it not show the rising-then-falling shape I predicted in my conceptual graph above?

The answer  is that there is so much music out there that even what we consider niche is usually still top-decile.

In the above chart I binned the Amazon ranks of record collections by 5,000s, which is the smallest unit that gives any decent overview. I cut off the chart at 100,000 for visual impact, although the full analysis and the collections included several albums in the 600,000 range.

All that rising-then-falling shape that I illustrated with the conceptual s/n graph actually takes place entirely in the top 5,000-10,000 for most people. By the time you're past that the density of almost everyone's music collection (which is to say, their s/n ratio) falls as you go down the tail. When you're binning by 5,000s, as I have, all that the fine structure in the head is obscured by the larger perspective.

Top 100 is irrelevant in an abundant market. Even the library of my most mainstream subject (Anne) had an average rank of 3,000. In the big picture of the Long Tail, there are so many items that even today's niche looks relatively popular. For instance, the average sales rank in my own collection was 25,000. That may sound super-fringe, but it still puts my average in the top 5% of Amazon's offerings. You've got to pull back and see the whole market. And at that resolution, the falling s/n ratio curve I originally described emerges for almost all of us.

Long Tails are long, and it's illuminating to stand back and see the whole curve. The microstructure of the current hits business, the blockbuster charts our culture has so long fixated on, is quickly lost in the macrostructure of the entire music universe. It's a big world out there, and the top 40 is just the beginning of it, not the end.

What the Long Tail isn't

After witnessing much misuse of the Long Tail phrase, this silly post has finally pushed me over the edge.  It's time to draw the line. Long Tails are found everywhere, but not, you know, actually everywhere.

There are many distortions of the term, but the most common one is to use it as a newly-positive synonym for "fringe". Invoking the Long Tail is not a magic wand to explain away the apparent lack of demand for what you've got.  The Long Tail is not a get-out-of-jail-free card for poor-selling product. Or weak sectors. Or bad ideas.

The fact that something isn't popular doesn't mean that it's just a matter of time before it will benefit from all sorts of  powerful demand-creation Long Tail effects. More likely, it's just not good enough to be commercially interesting, and probably never will be.

Most of the "niche" products in the tail are simply crap. That's okay, because some significant fraction of them aren't and with a functioning way to separate the good from the bad, they can be found by those who will appreciate them. But note the essential element: a functioning way to drive demand.

As I've mentioned in the original article, for Long Tail effects to work, you need both a head of relatively few hits and a tail of many niches, so that recommendations and other filters can lead consumers from one to the other.

A tail without a head is too noisy and apparently random to get consumer traction; people need to start with the familiar and then move, via trusted recommendations, to the unfamiliar. Likewise a head without a tail is too limited in choice; the odds of finding a niche you want are too low to bother exploring much beyond what you already know.

Thus the two big Long Tail opportunities are:

1. Aggregating hits and niches into a one big curve, from head to tail.
2. Creating content and products that can plug into someone else's aggregated curve.

If what you're selling is fringe, it may well enjoy Long Tail benefits, but only if it can fit nicely into an existing market that has the capacity to drive demand. If that market doesn't exist, it's unlikely that throwing some niche products out there is going to create it. Even if it does exist, those products will only reach their audience if the filters and recommendations are good enough to find them.

While I'm at it, here are some other things the Long Tail is not (to protect the guilty I will refrain from linking to the offending examples):

• Commodification
  • The LT is about nicheification, which is different.
• Simple variety
  • Offering a few different choices or a bit of customization (like the sandwich filling options in the risible example above) is not enough. Long Tail effects kick in when you're expanding variety and choice by orders of magnitude--from 10x to infinity.
• The case for an all amateur, self-published future
  
  • The LT will probably have as much commercial content as ever. It will just be joined by far more amateur fare, forming a relatively seamless continuum from pros to ams.
• The actual end of hits
  • The LT ends the tyranny of hits, shifting the market equally to niches. But it certainly acknowledges that some things will continue to be a lot more popular than others. Powerlaw distributions are as natural as diversity itself.
• A focus on small markets at the exclusion of large ones
  • Again, you need both hits and niches to allow the filters and recommendation engines to work, driving demand down the curve from the known to the unknown.
• Just any powerlaw
  • Powerlaws are ubiquitous. Long Tails are not. The first shows up anywhere you have variety, inequality, and network effects (word of mouth). The second requires massive variety and a wide range between the hits and niches. After all, many short tails are simply truncated powerlaw distributions. They just aren't, er, long.

Venture capitalists who are tired of bogus Long Tail pitches are invited to point the hapless entrepreneurs to this post.

Trolling

Six reasons why I prefer good blogs to most traditional journalism in the niche domains where my interests are greatest:

1. They respect their readers enough to open their comments.
2. When they make mistakes, they tend to correct them.
3. They understand that every factual statement that can be linked to its source, should be.
4. Because they have little default institutional authority, they go overboard backing up what they say with evidence. Unsourced assertions are frowned on. In this way, paradoxically enough, blogs are often more rigorous than traditional journalism, because they have to earn their readers' trust, not just assume it.
5. They're often written by practitioners, not just observers, and as a result they tend to get the details right.
6. If their information source is some random, unverified bar conversation or even just their own opinion, they're usually big enough to admit it.

(Quickly heading off the obvious question--why don't we do all this at Wired?--the answer is that we do some, we should certainly do more, and someday soon I hope we will. But it's tricky, because the print version exists independently from the website [see this Wikipedia entry for the complicated details] and we don't want the two versions of each article to diverge too much, for fear of confusion over which one is the "right" one. Meanwhile, nobody else in print media has really solved this problem yet either. And yes, I am aware of the irony that this is another distortion caused by inefficient distribution, exactly what I spotlight in other industries. Let's just say I know of what I speak.)

Three reasons why podcasts aren't a big deal (yet):

1. They don't have internal permalinks to section and subjects, so they don't get much link-love. UPDATE: Commentor #1 says it's technically doable, but clearly not many take advantage of that technology yet.
2. They aren't searchable. How hard would it be for some service to run podcasts through a quick-n-dirty voice recognition program to autogenerate transcripts? They don't need to be exactly right; 80% accurate search is better than the 0% we've got now. UPDATE: Michael Glenn in the comments points me to Podscope, which claims to do exactly what I asked for. Although it does seem to return relevent results, I can't get the podcasts to play in Firefox, so I can't comment on it yet.
3. They're meant to be consumed linearly, and pretty much at the (agonizingly slow and amateurish) pace they were created. Who, aside from trapped commuters, has time for that?

Three reasons why I'm not totally anti-DRM:

1. I'm not willing to install Myth TV just to avoid the broadcast flag.
2. I want more music and video. I'm happy to pay for it. Just don't jerk me around.
3. Even if Microsoft or whomever screw up their DRM, markets self correct. Millions of users tend to eventually get what they want.

Three reasons why I don't use a Mac:

1. Somebody at Wired should know what 97% of the country has to put up with.
2. Rhapsody.
3. Windows Media Center Edition 2005.

Three reasons why Linux is teh sux!

1. Okay, I'm not that much of a troll. I can't really go through with this.
2. I totally run it on all my machines, even the Roomba and the universal remotes.
3. Honestly, what's not to like about command lines?

(BTW, that wall painting at the top is a St. Augustine rebuffing the heretics. It's less grindingly literal than a picture of a troll.)