The Long Tail

80/20...cracked!

Thanks to all your smart and insightful comments, I've discovered the source of my 80/20 conceptual problem: I'm an idiot.  Although many of you were gently pushing me in the right direction, it was Jakob Neilsen (comment 15) who put his finger on it. I'd somehow forgotten that the 80 and 20 are supposed to be percentages of different things.

The conventional definition is that 20% of products make up 80% of revenues. But when I suggested that this would evolve to a 50-50 rule--that the money in markets of the future will be roughly balanced between hits and niches--I'd changed the definition. In the new formulation, both term refer to revenue. Doh!

One of the best bits of advice many of you gave was not to take 80/20 too seriously. Its power is in its universal resonance and familiarity, not the absolute accuracy of its numbers. I'll keep that in mind as I continue work on the chapter and I thank everyone for helping me clear this up.

Etech wrapup

I thought Etech this year was the best one yet. The highlights ranged from Danny Hills' virtual tour of his fantasy workshop at Applied Minds, to George Dyson's history of proto-geeks in John von Neuman's world, to Clay Shirky's devastating takedown of taxonomies to Larry Lessig's inspirational call to arms to spur remix culture.

This morning I spoke on the economics of the Long Tail, a subject I've been noodling around a bit here. Your comments were incredibly helpful in steering the presentation, which you can download here. (Note: it's a largish file--6.9MB--and is best viewed in presentation mode, so the animations show up in the right order.)

I followed the talk with an onstage conversation with Joe Kraus, who talked about how he missed the Long Tail opportunity at Excite and is aiming to capture it with JotSpot, his new company. You can find coverage here, here, here and, well, I guess here too.

In other news, I'm delighted to report that Wired has been nominated in the prestigious General Excellence category of the National Magazine Awards, which are the Oscars of our industry. That makes three times in four years (we had a sophomore slump), and I'm really proud of the whole team.

The real meaning of 80/20

I am as guilty as, well, nearly everyone else of sloppily defining the "80/20 Rule" to mean whatever I want. Pareto's principle really is an all-purpose widget, broadly applicable to almost anything humans do (and, of course, the behavior of atoms in a Bose-Einstein condensate). But it's time to crisp things up, at least as far as the Long Tail goes.

The general articulation of the principle is: "for many phenomena 80% of consequences stem from 20% of the causes." (Vilfredo Pareto [shown] first articulated this in 1906 when he noticed that 80% of the property in Italy was owned by 20% of its population).

In my original piece I explained it as follows:

The 80/20 rule is all around us. Only 20 percent of major studio films will be hits. Same for TV shows, games, and mass-market books - 20 percent all.

We're stuck in a hit-driven mindset - we think that if something isn't a hit, it won't make money and so won't return the cost of its production.  We assume therefore that there is little demand for the stuff that isn't carried by Wal-Mart and other major retailers; if people wanted it, surely it would be sold. The rest, the bottom 80 percent, must be subcommercial at best. We assume, in other words, that only hits deserve to exist.

So in this case I was using 80/20 to refer to the notion that 80% of the revenues come from 20% of the products. Furthermore, I was echoing the Hollywood mantra of hit-driven economics: the top 20% of films will make virtually all the profit, subsidizing the loss-making bottom 80%. Like this:

In other words, if you could only know ahead of time which films would be hits, the economically rational decision would be to make just 20% as many of them. And, indeed, this is largely what DVD rental and retail outlets do: since they do know which films were hits in the box office, they mainly stock and push just those. As a result, theatrical hits from the past year make up 90% of the transactions in even video superstores (other titles are stocked, but in smaller numbers and are not promoted).

The stores do this because shelf space is expensive, attention is scarce and there are few good ways to offer consumers information about products to help them with their choices.  Ironically, the real world, despite its ability to flood our five senses, is still a pretty narrowband  information channel when it comes to things like "people like you bought...", "IMDB says..." and "rank by...".

My thesis in the continuing research on the Long Tail is that as distribution and information bottlenecks diminish, many 80/20 markets will become closer to 50/50 markets.  What I mean by that is that the demand curve will flatten somewhat, so that the few "hits" that now generate 80% of the revenues will instead make up about half the market, and the many niches that now represent 20% of the revenues will, in aggregate, make up the other half. The reason is that more goods will be available and they will become easier to find as information about products gets richer and recommendations drive demand down the tail.

That's revenue. The profit implications could be even more dramatic. We're seeing that in many markets (especially those for digital goods that can be delivered online) the retail margin on small sellers can be as high as for the blockbusters. The marginal cost of storing and delivering a track on iTunes is the same tiny amount whether it's top ten or bottom half-million. Because storage space is essentially free, the market can become non-discriminatory:  10 sales each of 1,000 niche tracks is, to first approximation, economically identical to 10,000 sales of one hit track. So rather than all the profit coming from the top 20%, it too can evolve to be split evenly between hits and niches. Thus the Long Tail can, in aggregate, be as profitable as the head.

However, here's where it gets tricky. In the above definitions, I've been referring to 80% and 20% of the existing markets. But the Long Tail is about expanding those markets to include products and customers that are currently "sub-economic"--traditionally not worth carrying or otherwise including at all. How does that fit into the traditional 80/20 construction?

Good question. I'm afraid I may have confused many people on this point, because in the charts in my original piece I imply that the Long Tail refers exclusively to products that are not available in standard retail, which is to say outside the existing 80/20 world entirely. That's one perfectly reasonable definition of the Long Tail (see below), at least for products that have a retail presence, but it doesn't help us in our quest for 80/20 clarity. The fact that in the Netflix example this beyond-retail category happens to represent 20% of rentals only muddies the waters further. No wonder interpretations of this point have been all over the map.

(In all the Long Tail markets that I've looked at so far the revenues from products not available in traditional retail range from 10%-30% of the total. Those figures have all risen since my article last year, but I'll save the latest numbers for the book. Also note that I've revised my Amazon estimates after more consultation with them. )

Making matters worse, virtually none of the media markets I've looked at actually show an 80/20 distribution today. In most instances, they're much more hit-driven: typically a single digit percentage of products represent 80% of the revenues. In other words, the 80/20 Rule not only won't hold in the future, but it doesn't even hold today. Argh!

Needless to say, this is a mess and I'm going to have to be a lot clearer in the book. As I see it there are two possible solutions:

1. Keep my definition of the head category constant even as the tail expands, so I'm at least comparing apples to apples. Thus my thesis would be: As more niche products become available, the hits (whatever percentage of the total number of products they may be) that now make up 80% of the revenues will decline to 50%. And the niches will grow to make up the rest.
   
2. Find another way to draw a line between head and tail and simply observe how the expansion of one affects the other and the market as a whole. Inevitably, that's going to be subjective and will be different in each category. Also, in most cases it won't fit neatly into either the 80/20 or 50/50 rules. As in the Netflix example above, I could use typical bricks-and-mortar inventory to represent head, and the rest tail. In TV, I could use Nielsen-ranked shows (the top 100) to define the head. In music, perhaps the best metric is the Soundscan top 1,000 (the Billboard 100 is probably too few to be a fair representation, given how many albums are released each year). For advertising, companies that spend more than $10,000 a year on ads. And so on...

Number 1 is conceptually neat but somewhat arbitrary in its line-drawing. Number 2, while reflecting real-world dividing lines, doesn't fit neatly into a Pareto context.  On balance, I'm leaning towards Number 1--neat is good--but we'll see how the next batch of data comes in to see if it's doable. Rigor is great, but plenty of people smarter than me have ended their careers still trying to turn their ideas into the grand unified theory of everything. Me, I'll happily settle for almost everything.

Help me write a subtitle

Last night I had a nightmare that actually woke me up. I dreamed I was standing in line at a store with my parents, who are now in their 70s. They're telling a stranger that their son is writing a book, and he says "How interesting. What's it about?" My father goes blank for a second and then turns to me and says "I'm sorry, Chris, I've forgotten. What's the Long Tail about again?" So I get furious and chastise him for not caring enough to remember.  But in truth I'm just stalling because I can't figure out how to explain it to this stranger either.

What I need is a subtitle that can do double duty as a one-sentence summary. I started with this:

1) The Long Tail: How the mass market is turning into a mass of niches

That touches on some of the high points--the decline of the mainstream and the rise of niches--and it benefits from a little lyricism in the form of the repeated "mass". But I'm not sure that really is the best collective noun for niches. So I considered another approach at lyricism, our old friend Mr Alliteration:

2) The Long Tail: How the mass market is turning into a million niches

I quite like that one (I can justify the random precision of a million as poetic license and numbers, especially big ones, are always good), but I'm slightly concerned that it needs better context. Which mass market? Perhaps a "future of..." construction would help. Like this:

3) The Long Tail: How the future of business is turning from mass markets to millions of niches

The main problem with that is the sweeping nature of "the future of business". I could focus it on one thing--"the future of entertainment", "the future of shopping", etc--but each of those only represents a portion of  the scope of the book.  Another problem is that "business" is a supply-side word, telling readers that the book is mostly of interest to businesspeople. But I'm writing the book to be of equal interest to consumers--indeed, I see the Long Tail future as being something of a consumer paradise. Perhaps the way to do that is to make the point of view that of a shopper:

4) The Long Tail: How the rise of infinite aisles is turning mass markets into millions of niches

But does anyone know what I mean by "infinite aisles"? Of course I'm referring to unlimited shelf space and other ends of distribution scarcity, but there's always a risk of confusion when you're coining one neologism to explain another.

So all four of these have their plusses and minues. I'd love your thoughts on them (and suggestions of better alternatives), either by comment or email. Who knows: you might just see your words on the cover of a book someday.

The Long Tail of software

Joe Kraus, one of the founders of Excite, has a new company that's based on what he calls the Long Tail of software. Called JotSpot, it's building a wiki-like platform that makes it easy for firms to create custom applications for their specific business needs, much as they do with Excel spreadsheets today. He briefed me about this some time ago, but I wasn't able to talk about it until now. Fortunately, he's done all the talking for me with a really interesting post on his own blog that nicely shows how broadly the Long Tail theory can be applied.

The long tail doesn’t just apply to music and movies. There’s a long tail for software as well. Here’s why.

First, every business has multiple processes. Things like hiring, firing, selling, ordering, etc. Second, while some of these are pretty common in name from business to business (recruiting, for example), in practice, they are usually highly customized. Finally, there are simply a large number of processes that are either unique or that are common to millions of very small markets and therefore not traditionally worth the effort to buy software for (for example, the process by which an architecture firm communicates between it’s clients and the city planning office).

These three facts

• every business has multiple processes
• processes that are similar in name between businesses are actually often highly customized
• there exist a large number of processes unique to millions of small clusters of industries.

means that there is a combinatorial explosion of process problems to solve and, it turns out, little software to actually support them.

Said another way, there is a long tail of very custom process problems that software is supposed to help businesses solve.

There are also loads of great charts illustrating the point in a ppt file you can download from his site here.

The Tragically Neglected Economics of Abundance

I'm preparing for my talk on Long Tail economics at O'Reilly's Emerging Technology conference in ten days, and I've run into a slight problem. The Long Tail is all about abundance: the economic effects of infinite shelf space. Unfortunately, neoclassical economics has virtually nothing to say about abundance. Indeed, the economics of abundance is almost exclusively the domain of extropians, a few other transhumanists, and science fiction writers. How can this be?

Well, for starters the classic definition of economics is "the science of choice under scarcity". That's a warning sign right there. From Adam Smith on, economics has focused almost exclusively on behavior within constraints. My college textbook, Gregory Mankiw's otherwise excellent Principles of Economics, doesn't mention the word abundance.  And for good reason: if  you let the scarcity term in most economic equations go to nothing, you get all sorts of divide-by-zero problems. They basically blow up.

But clearly abundance (AKA "plentitude") is all around us, especially in technology. Moore's Law is a classic example. What Carver Mead recognized in 1970 when he encouraged his students to "waste transistors" was that transistors were becoming abundant, which is to say effectively free. The shift in thinking from making the most of scarce computing resources to "wasting" cycles by, say, drawing windows and icons on the screen led to the Mac and the personal computing revolution. To say nothing of the scandalous profligacy--a supercomputer used for fun!--of a Playstation 2.

We also have similar abundance laws working in storage and bandwidth and virtually everything else digital. Outside of technology, the green revolution brought abundance to much of agriculture (so now, to prop up prices, we pay farmers not to plant their crops).  And what is the motive force behind China and India's rise if it is not abundant labor, allowing them to, in a sense, waste people?

Even ideas can on some level be considered abundant, because they can propagate without limit due to their "non-rivalrous" nature (a feature they share with fire and digital media). As Thomas Jefferson, the father of the US patent system, put it, "He who receives an idea from me, receives instruction himself without lessening mine; as he who lights his taper at mine, receives light without darkening me."

More than a decade ago George Gilder, the apostle of abundance, offered a good way to think about all this. In an interview (in Wired, as it happens) he said:

"In every industrial revolution, some key factor of production is drastically reduced in cost. Relative to the previous cost to achieve that function, the new factor is virtually free. Physical force in the industrial revolution became virtually free compared to its expense when it derived from animal muscle power and human muscle power. Suddenly you could do things you could not afford to do before. You could make a factory work 24 hours a day churning out products in a way that was just incomprehensible before the industrial era. It really did mean that physical force became virtually free in a sense. The whole economy had to reorganize itself to exploit this physical force. You had to "waste" the power of the steam engine and its derivatives in order to prevail, whether in war or in peace."

That suggests a way to put this in an economic context. If the abundant resources are just one factor in a system otherwise constrained by scarcity, they may not challenge the economic orthodoxy. They are then like learning curves and minimized transaction costs--drivers of production effeciency that serve to lower prices and increase productivity but do not upend the economy as a whole.

And, indeed, the abundance of the Long Tail, for all its power, is surrounded by such constraints. Although there may be near infinite selection of all media, there is still a scarcity of human attention and hours in the day. Our disposable income is limited. On some level, it's still a fixed-pie game. Offer a couch potato a million TV shows and they may end up watching no more television than they did before; just different television, better suited to them.

By analogy, Moore's Law just shifted the scarcity bottlenecks from the CPU to the user. Now the limits of productivity are not instructions per second but ease of use, which is why that's where those transistors are being so wastefully deployed on the desktop today. The consequence is that our productivity increases, a phenomenon entirely compatible with economic theory. But that happens at a pace of 2-4% a year, not the annual 60-70% increase of Moore's Law. In other words, there are still limits to growth.

There's still a lot to be said about the economics of the Long Tail, from how the shape of the head changes as the tail opens up to the question of whether tails have a nested heirarchy of mini-tails. I hope to have enough data to answer those questions by Etech, or at least phrase them well enough so that I have the answers in the book. I'll certainly be further along than I am today; I've got some interviews with good economists next week, and they're sure to be of help.

In the meantime, it's worth noting that the fact that economics doesn't have a good theory for something doesn't mean that it isn't real. For instance, economics is still struggling to explain growth itself. Seriously.

Exploding Radio

All media is getting Long Tailed at breathtaking speed these days. Music, TV, film, news and even advertising are being transformed by an explosion of variety and availability as the traditional bottlenecks of distribution fall. This month in Wired we focused on radio, where the combination of satellite, streaming, digital broadcast, radio TiVo, and podcasting is doing the same.

The features in the package--Wonkette on Howard Stern in space, Annalee Newitz on ipodder Adam Curry, and Charles Mann on How Sex Pistol Steve Jones is reinventing indie radio--inside Clear Channel are online, but the cool sidebars on sat radio, HD, radio TiVo, and how to podcast are not, so pick up a copy and check em out.

To whet your appetite, here's a list of ten reasons why traditional radio is hosed. Please add more in the comments.

1. The iPod. The ultimate personal radio.
2. The cellphone. Commuters stuck in traffic were the salvation of radio in the 80s. Now we're still stuck in traffic, but we're on the phone instead.
3. The 1996 Telecommunications Act, which liberalized the airwaves, adding a thousand FM stations to the dial and, due to the increased competition, depressing the economics of the incumbents. The act also relaxed the ownership limits in each market, leading to....
4. Clear Channel. This giant is often blamed for radio's woes, but it's as much a symptom of the brutal economics in the industry as it is the cause. As the aforementioned competition ruined the business of local radio in the late 1990s, Clear Channel was able to do a roll-up of distressed stations and now owns more than 1,200 of them. Its plan was to dramatically lower the costs of radio with centralized programming and robot DJs.  The result, however, was bland homogenization (with one exception). And still radio was broken.
5. To whit: Clear Channel just wrote off 4.7$ billion in radio losses. A few days earlier, its main competitor, Viacom's Infinity, wrote off $18 billion. So much for the power of monopoly media. 
6. Radio listenership is at a 27-year low.
7. Music is polarizing. This is why MTV doesn't play music anymore--there is no lowest-common-denominator playlist that enough people agree on to slow the tide of channel switching. The same is true for radio: Top 100, Top 40, Top 10--there is no magic list that can strike the right balance between broad popularity and grinding repetition to keep enough listeners. MTV went to reality TV and other "sticky" programming; radio went to cheap talk radio on the AM dial, where callers provide most of the content for free.
8. The chilling effect of the FCC's obscenity crackdown, as evidenced by Howard Stern's departure to satellite.
9. Howard Stern's rise in the first place.
10. Washington DC's WHFS, my hometown alt.radio station and the soundtrack to my misspent youth, has just switched to spanish.