The Long Tail

Generally Excellent

I really will resist indulging in too much Wired autohornblowing here, but this is worth making an exception for. Yesterday we won the prestigious National Magazine Awards' top prize, General Excellence, for the 500,000-1m circulation category. Our sister titles at Conde Nast, The New Yorker and Glamour, won in 1m-2m and 2m+, respectively, and completing the clean sweep of eligible categories, Style.com, the website of CN's Vogue and W, won the top online award.

The three 2004 issues we submitted were February:

October (which included  the original Long Tail article):

and November (Creative Commons CD):

Congratulations to the whole team!

FAQ: Does the rise of the LT = the fall of mass culture?

The Long Tail is about the shift from hits to niches. Several readers have asked what this means for the future of mass (hit-driven, mainstream) culture in America.  The short answer is that it will not only get less mass, but that this is a trend that's already well underway.

Let's start by defining mass culture. The usual test is the "watercooler effect", the buzz in the office around a shared cultural event, be it the finale of The Apprentice or the opening of the last Star Wars. The number of such events has been shrinking for years, driven mostly by the fragmentation of the television audience.

Cable TV started this with its explosive increase in the number of shows broadcast at any one time, which soon resulted in half of American viewership moving to cable, where they are today scattered among nearly 300 networks.  The arrival of TiVo and other DVRs amplified this by taking the time component out, too. Today even if people are watching the same shows, they may not be watching them on the same night. Who wants to listen to the morning-after recaps of real-timers, who risk ruining the surprise of shows you're yet to replay?

The result is that the day when most of America watched the same things on the same night is long gone. All but one of the top rated TV shows were in the late 70s and early 80s; the one newer one was  the 1994 Winter Olympics, still more than a decade ago. Today's top shows have Nielsen scores that wouldn't have put then in the top twenty two decades ago.

Likewise for music. By my count only ten of the top 100 best-selling albums were released in the last decade, and only four of those were in the last five years. Occam's razor says this is more likely due to the diminishing power of radio as a hit-making machine than it is to some Darwinian decline in music talent. In 1993, American spent on the average 23 hours and 15 minutes a week listening to radio. As of spring 2004, that figure had declined to 19 hours and 45 minutes per week. America's Top 40 just doesn't matter as much anymore.

So instead of the office water cooler, which crosses cultural boundaries as only the random assortment of personalities found in the workplace can, we increasingly have our own tribes.  My tribe may have all picked up the "all your base" meme at the same time, but your tribe was into the steroid scandal. These days our water coolers are increasingly virtual, there are many different ones, and the people who gather around them are self-selected. We are turning from a mass market into a niche nation.

Measuring Television (short form)

Today's very, very long NYT Magazine cover story on Neilsen and the new people meter, compressed and read with a Long Tail filter:

"...One day this January I sat in a Greenwich Village workroom with Bob Luff, the chief technology officer at Nielsen, as he pulled out gadget after gadget to show me what he's up against. Luff seemed to view the modern American home as a digital zoo where the lion is about to lie down with the lamb: radio is going on the Web, TV is going on cellphones, the Web is going on TV and everything, it seems, is moving to video-on-demand (V.O.D.) and (quite possibly) the iPod and the PlayStation Portable. ''Television and media,'' Luff said over the noise of five sets tuned to five different channels, ''will change more in the next 3 or 5 years than it's changed in the past 50.''...

...The average American household now sees 8 hours 1 minute of TV every day and has access to more than a hundred channels and several different sets -- often tuned to different channels in different rooms. Industry types call this phenomenon audience fragmentation. The days of a family gathering together on the couch are dying out for good. We're in pieces. Or as Steve Morris, the Arbitron C.E.O., put it more gently, ''People are dividing.'' Every age group, every cultural group and every demographic group, Morris added, is in the process of getting media packaged expressly for its members. In the next few years, this ''personalization'' will become only more and more pronounced....

...Whatever this transition means for TV viewers, it has different implications for advertisers. In recent months, in fact, a host of executives from big corporations, most notably Jim Stengel of Procter & Gamble, have begun publicly demanding that measurement companies like Nielsen and Arbitron provide better information about audiences. These advertisers don't mind talking to smaller groups of Americans. In fact, companies like fragments. The more specific an audience, the more confident they can be of reaching out to and persuading its constituents....

...Turow said he now sees little difference between television and the Internet. Nor do his students at Penn. They watch ''The O.C.'' wherever and whenever -- on their laptops, at home on TiVo and by swapping the show (perhaps illegally) through a Web-based file-sharing program called BitTorrent. The coming generation is accustomed to the idea of watching or listening to anything on any device that's nearby, Turow said. In the meantime, his generation (he's in his 50's) ''still thinks of media in these compartmentalized ways.''..."

Mainstream Media Meltdown

In the spirit of endism, here's a list of all the forms of major media and how they're trending. Make of it what you will.

Flat to Down to Way Down:

• Music: sales last year were down 21% from their peak in 1999
• Television: network TV's audience share has fallen by a third since 1985
• Radio: listenership is at a 27-year low
• Newspapers: circulation peaked in 1987, and the decline is accelerating
• Magazines: total circulation peaked in 2000 and is now back to 1994 levels (but a few premier titles are bucking the trend!)
• Books: sales growth is lagging the economy as whole

Up:

• Movies: 2004 was another record year, both for theaters and DVDs
• Videogames: even in the last year of this generation of consoles, sales hit a new record
• Web: online ads will grow 30% this year, breaking $10 billion (5.4% of all advertising)
  

The Economics of Variety

This is a drill-down on some issues glossed over in the short preceding FAQ on whether the Long Tail grows demand or just shifts it. One of the aims of my quantitative work with the large data sets now available from online retailers is to answer the question statistically; in the meantime, here's the academic background.

More variety does not, by itself, always encourage more demand. Indeed, as some studies have shown, people can be overwhelmed by too much choice and actually buy less, as in the 24 kinds of jam example cited by Barry Schwartz in his "Tyranny of Choice" arguments.  Amazon has found that one solution to this is to add information about the products, from something as simply as "rank by bestselling" to more detailed customer reviews and ratings, which can cut through that confusion and encourage consumers to buy with confidence. (See my previous post on why variety is not enough.)

But anecdotally, we all know of instances where increased variety and better ways to find things encourages people to consume more. I know, for instance, that Napster reawakened my interest in exploring new music, a passion that continues more easily (and legally) with Rhapsody, which has probably doubled my music spending. I also know I buy more books (and practically everything else) because of Amazon's selection. And my family certainly watches more DVD thanks to Netflix.

The army of white-earbud-wearing New Yorkers are surely listening to more music than they were before the iPod, extending the effect created by the Walkman a generation before. But are they buying more music, too? The hard numbers are, unfortunately, inconclusive. As of February, Apple had sold 10m iPods and 250m tracks on iTunes, for an average of 25 tracks sold per iPod, or less than the price of two CDs over the nearly two years  the iTunes music store had been in business.  That is not impressive.

Perhaps they're just buying CDs and ripping them. Or just downloading more, but not paying for it. Studies of the industry at large are sadly inconclusive. The RIAA says its members are shipping fewer CDs, but Soundscan says they're selling more of those they do ship (see Kensei for more on this mess).

It's not just music that's in a muddle. In books, overall book sale growth has slowed over the last decade, even as Amazon (which accounts for about 10% of the market) rose. Meanwhile, total entertainment spending has remained pretty constant between 5-6% of the average Americans' spending for decades, despite an explosive rise in entertainment options.

Indeed, although there is a general presumption that more availability leads to more sales, there is precious little statistical work that proves this is the case, especially for large numbers of products. The small-number consumer psychology work, however,  does suggest that when the choice is meaningful, more is better--you're simply more likely to find what you want, or at least something that strikes your fancy.

There are a few studies that look at examples such as the effect of increasing the number of yogurt flavors on offer by one or two, which does help sales.  One of the better-known bits of "more is better" research is Brown, Read and Summers' 2002 paper "The Lure of Choice", which conducted bank, nightclub and casino experiments to show that consumers picked options more often when they were accompanied by other choices.

Likewise, Malcolm Gladwell has highlighted the case study of the would-be spaghetti sauce competitors who learned not to out-Prego Prego in making the platonic ideal sauce, but to instead celebrate diversity--chunky, homemade, spicy--and expand the market through multiple niche sauces. In part because of this introduction of more variety, which pushed buttons consumers didn't even know they had, spaghetti sauce is now one of the top six growth categories in the dressing and sauce market.

But all this says is that some choice is better than none. What we've got in the Amazons, Netflixes, and iTunes of the world is another phenomena entirely: lots of choice turning into near-infinite choice. It is not just three hits going to 30 hits, but 300 hits going to 300,000 niches. This is a new effect, just a few years old, and I'm working with those companies to quantify it. I'm betting that the effect of massive increases in available variety coupled with good ways to find stuff is to not only shift demand more toward niches but also to grow it, following the greater-satisfaction principle. But the data will have to speak for itself.

FAQ: Does the LT increase demand or just shift it?

Does the Long Tail grow the pie or simply slice it differently? In other words, as the number of available products grows many-fold with the infinite shelf space of virtual retailers, does it encourage people to buy more stuff or just less popular stuff? Several readers have written to ask about this, some arguing that at most the Long Tail will shift demand because consumers don’t have extra time to consume more content and they don’t have extra money to pay for it.

I've touched on this some in the economics of abundance discussion, in particular noting that scarcity of human attention is indeed a limiting factor in demand. But in general, the answer depends on the sector: some do seem to have huge opportunities for growth as their niches become widely available and some do not.

Although human attention and spending power are finite, you can get more for your time and money. Some forms of entertainment, such as music, are "non-rivalrous" for attention, which is to say you can consume them while you're doing something else. For instance, some explanations of the rise of average hours of TV watching in the 70s and 80s were because a generation had grown up used to television on in the background of their lives; as the novelty wore off it went from a rivalrous to a non-rivalrous medium, and thus we consumed more of it. 

Other media, such as text, can be consumed more efficiently and with higher quality through better pre-selection. Indeed, it’s quite extraordinary how much we’ve been able to increase our consumption bandwidth of information, speed-reading pages of Google search results and custom RSS feeds. I may not read any more words than I once did, but they're more likely to be meaningful to me, thanks to much better filters (better at suiting my own interests than, say, the editors of the New York Times) pre-selecting what I do read. So because the words are more relevant, my meaningful bandwidth has increased; I have, in a sense, compressed human attention.

But once you combine the scarcity of disposable income with the scarcity of time, some non-rivalrous media may become rivalrous. The reason people have the television on in the background is because it doesn't cost them anything to do so. But if that were pay-per-view video, you can bet it would suddenly have become the center of their attention. This then favors all-you-can eat subscription services in the Long Tail stakes. You're likely to consume more if it doesn't cost you more to do so.

So, bottom line: human attention is more expandable than money. The primary effect of the Long Tail is to shift our taste towards niches, but to the extent we're more satisfied by what we're finding, we may well consume more of it. We just won't necessarily pay a lot more for the privilege. I hope that doesn't screw up anyone's business model.

Robert X. Cringely, Longtailer

Mark Stephens, AKA Robert X. Cringely of Triumph of the Nerds fame, works for PBS so he's got a good insider perspective on just how ripe TV is for reinvention. His column last week was a cracker, laying out an immaculate case for Long Tail TV. Some excerpts:

Ted Turner became a billionaire by stealing intellectual property rights. He'd buy re-run rights for Gilligan's Island in the Atlanta TV market, then throw his WTBS signal up on a satellite where it could be viewed over cable systems all over America.  The contract didn't specifically say he couldn't do this, but that was just because the program sellers hadn't thought of it.  They didn't see the value, just as CBS saw no value in re-run rights for "I Love Lucy," leaving them in the hands of Desi Arnaz. Turner also realized the value of re-runs, not just of TV shows but also movies.  He bought the MGM movie studio primarily for its library of old films, realizing long before most of the rest of us that there was gold in that dusty celluloid.

These are oft-told stories of Turner and Arnaz, but now let's place the ideas in a more modern and relevant context -- pbs.org.  Some of the most popular pages on this web site are for shows that PBS no longer even has the right to air.  My own "Triumph of the Nerds" is in that category.  Though PBS no longer shows the documentary, thousands of people from all over the world still visit its mangy old web site. They find the site useful, even if it isn't the show, itself.  And I find it fascinating that there is this audience for material that -- like Lucy re-runs -- was never thought to have enduring value.

There is an audience, however small, for just about every show ever made.  What we need to do is to find a way to make the cost of keeping those shows available less than the benefit derived from people seeing them.  And that brings us to the economics of Blockbuster Video.

Blockbuster, or any video rental store, works on the same business model.  The cost of keeping a movie on the rental shelf is $10 per week for the first 3 to 4 weeks, then $0.50 per week thereafter.  Most customers rent new releases, but those are no longer new after the first month.  At that point, extra copies are sold off and a few archival copies remain to serve the few renters per week who are looking for them.  The new release and archival rental businesses are very different, but they complement each other.  In television, though, we aren't nearly as efficient.  PBS typically buys the right to air shows for five years, which ought to support both the new release and archival models, but most shows are played a lot at first and then hardly at all for the rest of the time.  That's because the broadcast distribution network isn't efficient for small audiences.

He then goes on to describe exactly how a TV broadcaster like a PBS affiliate might actually act on this. Well worth reading (even though he somehow manages to avoid namechecking the Long Tail!).

My God, it's full of powerlaws!

Yes, everywhere you look you will find powerlaw distributions. They're an inevitable consequence of variety and inequality, both of which are forces of nature--human and otherwise. I'm delighted that my work on the Long Tail has encouraged people to turn their gaze from the steep left ("hit") side of these curves and notice that the shallow right goes on and on and on. But I will resist turning this into the Powerlaw Blog, which is really more Clay Shirky's domain anyway.

That said, I can't resist posting this one from Alex Barnett, who asks "is this the Long Tail of conversations and relationships?"

The data comes from a paper by the HCI lab of the University of Maryland, which analyzed one individual’s fifteen-year email archive, consisting of about 45,000 messages and over 4,000 relationships.

The Long Tail of design

Domic Muren, who's behind IDFuel, a good industrial design blog that I follow, applies the LT to design for niches. He distinguishes between designing and selling products for what he calls the "body"of the marketplace (an unfortunate word to describe the mass market, given the confusion likely in his world, where people really do design for bodies) and for specialty retailers in the tail.  

"Designing for the body is about designing for fashion, licensing, heavy TV/print promotion. It's designing for mass market, Target-Wal Mart-Big Box Store. It booms huge in sales, and then it fizzles out, eclipsed by the next boom. It's also finicky as hell. ... The Tail is about creating pockets of absolutely devoted users in a sea of others who don't get it, rather than fitting everyone out with the same taupe cargo chinos.

Before the Long Tail of specialty retail existed, you were only safe designing for the body of the market. You were only safe designing to hit as many people as you could. Now that there are so many more places to sell targeted innovative designs, you can afford to be a little more exciting. Like Uglydoll -- here's a concept which could only have been started because of long tail distribution online and in not-so-common sales locations, like CB2 in Chicago.

That's what the Long Tail is really about: Freedom. Freedom from re-using tried and tired designs re-branded with new blockbuster movies. Freedom from lowering your ideal to the lowest bidder's values. Freedom to design the things that you believe really matter. Because if they really matter to someone, the Tail connects them with it.

This dovetails nicely with Virginia Postrel's work on variety (other examples) and aesthetic design, the subjects of her next and previous books, respectively.