The Long Tail

Reality Check

Hugh Macleod (gapingvoid) offers a cautionary perspective:

Mark Cuban, Longtailer

For a billionaire owner of a high-profile basketball team, Mark Cuban is often underestimated. Perhaps it's because he made his fortune selling a dot.com to another dot.com in the bubble days. Or perhaps it's his fratboy persona, which is mostly due to his boyish haircut and sports nut antics at Mavs games.  But in person, Mark is smart, geeky in a good way, and on the right side of most issues. That's why we profiled him in this month's Wired, in a great piece by Xeni Jardin that went live this morning.

Here are ten reasons why Mark Cuban is today's Long Tail hero:

1. He's retrofitting all the Landmark cinemas, which he owns, for digital projection. This not only dramatically lowers the costs of getting films into theaters and ups their projection quality, but will also potentially open that distribution channel to independents who can't afford to print hundreds of huge rolls of acetate and truck them around. 
2. He's promoting HDTV the right way, by commissioning and distributing content. 
3. He's funding the Grokster defense. 
4. He's a real blogger, warts and all. 
5. He says that he won't use the broadcast flag. 
6. He got the idea of reinventing TV a decade ago, and his work lives on in Yahoo!'s excellent video search and music video services. 
7. He doesn't believe the RIAA. 
8. He sent a team to cover the Iraq elections, live, in high-def. 
9. He has a successful reality TV show. 
10. He really gets the Long Tail: "Popular items are just that, popular. My guess however is that in absolute numbers, the long tail of the download distribution curves, both in terms of number of songs and in number of songs downloaded, overwhelm the number of copyrighted songs illegally being downloaded."

The Grokster Case's Silent Majority

The LA Times asked me to write an op-ed on the Grokster case from a Long Tail perspective. I thought the one missing element from the debate so far was the recognition that it's not just companies vs. consumers this time. The new amateur creative class, the "pro-ams", who populate much of the deep tail and depend on the open distribution technologies that are under attack, could be the losers  if the Supreme Court sides with MGM and forces p2p networks to police content.

Two decades ago, when the famous Betamax case set a precedent that protected the VCR, it was consumers versus the studios and record labels. But now there's an equally important third party: the creative amateur — people like you and me who not only consume but also produce content. And they're on the side of Grokster and the extraordinary power of the new distribution networks.

Read it all here.

Microstructure in the Long Tail

In this post I'm going to attempt the death-defying stunt of connecting social network theory, wealth distribution, the behavior of atoms in a gas and gangsta rap.  Coherence would be a plus.

In my Etech speech I listed several interesting economic questions I was hoping to answer about the Long Tail in the book. One of the was whether the LT had a "fractal dimension", by which I meant whether the shape of the popularity curve in the many niches is the same, at a smaller scale, as the overall shape of the total demand curve. Functions that have this characteristic are said to exhibit self-similarity at multiple scales. I speculated that this was indeed the case in many industries, and that the Long Tail is in fact made up of many "minitails" (below), all adding up to the powerlaw ("Pareto") shape we know so well.

This is important because it explains why a very effective network-effect (viral word-of-mouth) recommendation system, which is essential in driving demand down the tail,  does not actually do the opposite: drive content up the tail to further amplify hit/niche inequality. The explanation, I argue, lies in the only semi-permeable membrane between niches and mass-markets. Popularity exists at multiple scales, and ruling a clique doesn't necessarily make you the homecoming queen.

It's possible, for example, to be the most popular drum-and-bass artist at the very head of the drum-and-bass popularity curve, but that doesn't mean that you're about to knock 50 Cent off his top-ten perch. Music is made up of thousands of niche micromarkets, miniature ecosystems that, when smooshed together into an overall ranking, look like one Long Tail. But look closer and each has its own head and tail.

My question was whether these minitails resemble the Long Tail under a microscope, or whether they have a different shape. (Clearly this depends on the size of the niche, but I was focusing on relatively substantial ones where the n is large enough to be statistically significant). 

This is a reasonable question. Ducan Watts and Albert-Laszlo Barabasi's work on "small worlds" suggests that the microstructure of small social networks is distinctly different from large ones, going from a near-broadcast relationship to narrowcast to multilateral as the group size shrinks. That would tend to lead to very different popularity curves, assuming that social network theory does indeed apply to the shape of the Long Tail (further research for my to-do list).

I'm still gathering data to answer this empirically, but in the course of my digging I came across (thanks to Julian Bond) a fascinating New Scientist article about new "econophysics" thinking on the best-known of the Pareto distributions--wealth. It explains why we appear to have two economic classes, one in which the rich grow richer and the other in which the poor stay poor:

In 1897, a Paris-born engineer named Vilfredo Pareto showed that the distribution of wealth in Europe followed a simple power-law pattern, which essentially meant that the extremely rich hogged most of a nation's wealth (New Scientist, 19 August 2000, p 22). Economists later realized that this law applied to just the very rich, and not necessarily to how wealth was distributed among the rest.                                            

Now it seems that while the rich have Pareto's law to thank, the vast majority of people are governed by a completely different law. Physicist Victor Yakovenko of the University of Maryland in College Park and his colleagues analyzed income data from the US Internal Revenue Service from 1983 to 2001. They found that while the income distribution among the super-wealthy - about 3 per cent of the population - does follow Pareto's law, incomes for the remaining 97 per cent fitted a different curve - one that also describes the spread of energies of atoms in a gas
(see graph at right)

In the gas model, people exchange money in random interactions, much as atoms exchange energy when they collide. While economists' models traditionally regard humans as rational beings who always make intelligent decisions, econophysicists argue that in large systems the behavior of each individual is influenced by so many factors that the net result is random, so it makes sense to treat people like atoms in a gas....The atoms assume an exponential distribution of energy when they are in thermal equilibrium, and pushing the gas away from this state takes a lot of energy. It could prove similarly difficult to shift an economy to a different state.

Basically, it's so hard being poor that there's no time to work on getting rich. Is that true for struggling artists at the butt end of the tail, too? I've assumed that demand can shift down the tail and quality can rise up it, almost without limit. But there may indeed be a threshold at which this egalitarian mobility no longer works.

Sidebar Shoutout

Just a quick note to mention that I've added a few dozen summaries and links to interesting Long Tail comment elsewhere to my sidebar, adding to the 120 or so that were already there. The SixApart folks have been promising a Typelist (which is what the sidebar is) RSS feed for a while, but until they deliver you'll just have to actually visit the site to see how smart folks from all over the web are adding to the LT conversation. It's practically open source!

Only you can save television

It would be easy to make this blog a running chronicle of all media and entertainment in the throes of radical change. If nothing else it would be a breeze to update daily with a selection from the overwhelming number of examples out there. And as it happens, the radical change theme also plays strongly to the Long Tail. As the business models of traditional media and entertainment crumble, so will the hit-machine distribution bottlenecks that stand between us and a million niches.

I'm going to resist being such a chronicle, however, mostly on the grounds that you can get that elsewhere and I've got a book to write. But every now and then the tide of evidence will compel me to make an exception to note the obvious: that we're ending one era and entering another where the rules are sure to be different.

The best example of that is television, which is in the sweet spot of all Long Tail forces. So today, let me make one of those exceptions by explaining why TV is the first place to look for Long Tail opportunities. Here's why: 

• TV produces more content than any other media and entertainment industry. There are an estimated 31 million hours of original television content produced each year. Although that isn't as much as radio, most radio is either chat or recorded music that is available elsewhere, so it's not in the same league. In addition, 115m digital video tapes are sold each year for personal camcorders.
 
• Only a tiny fraction of it is available to you. First, the average American household now gets 100 channels of TV. While that sounds like a lot--it's 876,000 hours of video broadcast to the average home each year--that's still less than 3% of the commercial video that's produced for broadcast.
 
• Making matters much worse, unless that home has a DVR (and only 4-5% of US households do) and someone is spending a good chunk of their free time scouring listings to program it, they're going to miss virtually all of that TV.  Once TV is missed, it's usually gone. Only a tiny fraction of shows are syndicated, and an even smaller fraction makes it to DVD.
 
• Thus the ratio of produced content to available content is the highest of any industry I've looked at. Other industries may produce more content--print, for instance--but it's far more available (see Google). Only television treats its premium content as disposable. True, a lot of it actually is. But not all, and not as much as is effectively thrown away after a brief moment in the sun.
 
• Other industries with great Long Tail potential either know it, are already feeling the pressure, and are well on their way to doing something about it (music, print media, books, radio) or they are doing so well that they have little incentive to shake things up (film, games). TV, on the other hand,  is just about to hit its crisis, whether its executives are willing to admit it or not.  That crisis includes: ad-skipping, the disappearing 18-34 male, rising ad rates despite chronically falling ratings, and the death of the 30-second spot (see also Sunday's NYT piece). My friends in the biz suggest that this year's upfront will be the first in memory to go negative. If so, that would be a real wake-up call.

There is no shortage of smart people thinking about how TV can find its way out of its corner. But it's not easy. For starters, most of the networks are content renters, not content owners. (NBC, which bought Universal ten months ago, is now an exception). This means that the archives are often not theirs to monetize.

Rights also continue to be a total hairball, made even more complicated by exclusive geographic distribution deals (which conflict with the Internet's global nature) and syndication options. And then there's music, which is a nightmare. Want to know why you can't watch old WKRP in Cincinnati episodes? It's too hard to license the music that was used in the show.

Bottom line: TV is begging to be reinvented. Fortunately, there are a lot of new companies that have emerged to try to do just that. One of them is Brightcove (formerly Vidmark), whose CEO,  Jeremy Allaire, gave me a briefing last week at PC Forum. They've got an open-access video publishing platform that could make it as easy to be a video publisher tomorrow as it is to be a bookseller today. Impressive. 

Finally, I'll finish this post with additional links for those who still have an appetite for more:

* Berlesmann is launching a P2P platform for video and games downloading.
* The director of the Blair Witch Project is releasing his new video episodes on BitTorrent.
* Fox is releasing some of its shows as $10 DVD "starter sets".
* PSFK provides a good roundup of other IP TV news and stats.

Very Creative Commons

Great news on the Creative Commons front:

Yahoo!, whose metabolic rate of innovation has been goosed into overdrive by Google, last night release a Creative Commons search engine.  Larry Lessig, CC supremo (and Wired columnist), describes his meetings with them and why they turned out to be so productive:

They had, imho, precisely the right vision of a future net. Not a platform for delivering whatever, but instead a platform for communities to develop. With the acquisition of Flickr, the step into blogging and now this tool to locate the welcome mats spread across the net, that vision begins to turn real.

To which all I can add is: Thanks, Yahoo! Way to be very not evil.

By the way, I've recently changed my own CC license. I had assumed the right one for this blog was the non-commercial license, but after discussion with David Hornik, I realized that I really don't want to stop people from using material on this site commercially, I just want them to credit me as the source. So I've switched to a "BY: 2.0" license. It's on my About page.

Tail Whipped

Normally I put interesting Long Tail comment that I've found elsewhere in my sidebar, but this one was too good to thumbnail:

"My wife is boycotting Burger King and has been for the last 8 years. Of course, that means that I boycott Burger King too. We haven't been in one voluntarily since the day my wife got a badly burned chicken sandwich and an employee refused to take it back or make her another one....

[But] our boycott of Burger King is meaningless in the company's eyes. As Kelly Mooney notes in The Ten Demandments, our boycott is "invisible."

Back in the day, there wasn't much you could do. You could leave a nasty comment card that would likely wind up in the trash. You could call or write the company, but that takes effort and the results, based on experience, are likely to be less than satisfying.

But several articles have emerged recently that have reminded me how things have changed. The aforementioned Kelly Mooney's own blog references a new Forbes report on the top corporate hate sites. A very entertaining read. Warning: You may waste a lot of time reading other people's gripes about companies that you don't even frequent. They are just that interesting sometimes.

I also ran across an article on the Lawler's Rule blog that talks about the "Long Tail" and using search engines to reach a niche with regard to your complaints about a specific company....

I guess this is sort of like the difference between being a little fish in a big pond versus being a big fish in a little pond. If a "YourCompanySucks.com" web site gets a better Google ranking than your own corporate web site, heaven help you. The little pond has a way of getting bigger....quickly.

The take-home message? You can't afford to allow bad experiences to flourish in your company. Customers have a bigger arsenal to fight back with than they ever had before. If you allow bad experiences to thrive, the long tail will whip around and beat you senseless."

Long Tail vs. Bottom of Pyramid

In January I spent a week in India in part to wrap my head around C.K Prahalad's "Bottom of the Pyramid" theory, which in many ways resembles the Long Tail. Are they in fact the same?

As I mentioned in an earlier post, The Fortune at the Bottom of the Pyramid: Eradicating Poverty Through Profits, like the Long Tail, is about finding ways to efficiently address sub-economic markets. In this case, Prahalad is talking about how to sell goods and services to the world's 4 billion poor, for mutual benefit. Diagrammatically, the market he's talking about addressing looks like this:

Is the bottom of the pyramid the Long Tail?
Is it, in other words, just another way to draw this:

The similarities are notable. Both theories are based on the notion that if you break the economic and physical bottlenecks of distribution you can reach a huge, previously neglected market. They both recognize that millions of small sales can, in aggregate, add up to big profits. And they're both focused on ways to lower the cost of providing goods and services so that you can offer them at lower price point while still maintaining margins.

But despite the fact that it took me a trip to India to clear my head on this, I think there is a key difference between them that makes them fundamentally incompatible.

The Bottom of the Pyramid (BOP) argument is essentially based on commodification. Take existing goods and services and make them an order of magnitude or two cheaper, either to buy or to make but ideally both. Typically, this means reducing goods to their bare essentials and delivering them on a massive scale.

This requires: 1) low price points; 2) minimal marginal costs (reduce consumables and packaging to the bare minimum); 3) "de-skilling" services so non-experts can deliver them; 4) the use of local entrepreneurs.

The canonical example is the 1 cent single-serve shampoo sachet, but others include the Grameen phone (loan an entrepreneur the price of a phone and she can resell it in minute increments to an entire village), tiny tubes of toothpaste, and "Ruf and Tuf", a  $6 size-and-stitch-locally jeans kit that has become the top seller in India. (You can read the article that inspired the book here. Free registration required)

I visited a hospital that was offering advanced health care to the poor by increasing the utilization rates of its expensive CAT scan and X-ray machines by a factor of 100 by running them 24/7 and using digital imaging rather than expensive film. They also brought top-quality consultation to remote villages by having low-skill technicians place sensors on a patient's body while a high-skill doctor evaluates the results in real time via a communications link from a hospital in the city.

In all these cases, the BOP model is focused on taking a single product or service and finding ways to make it cheap enough to offer to a larger, poorer, market. This is why I think it's essentially about commodification.

The Long Tail, on the other hand, is about nicheification. Rather than finding ways to create an even lower lowest common denominator, the Long Tail is about finding economically efficient ways to capitalize on the infinite diversity of taste and demand that has heretofore been overshadowed by mass markets. The millions who find themselves in the tail in some aspect of their life (and that includes all of us) are no poorer than those in the head. Indeed, they are often drawn down the tail by their refined taste, in pursuit of qualities that are not afforded by one-size-fits-all. And they are often willing to pay a premium for those goods and services that suit them better. The Long Tail is, indeed, the very opposite of commodification.

So the Long Tail is made up of millions of niches. The Bottom of the Pyramid is made up of mass markets made even more mass.  Both lower costs to reach more people, but they do so in different ways for different reasons. They're complimentary forces, but fundamentally different in their approach and aims. The Mystery of the Apparently Similar Theories: Solved.

Exploding Radio II

This week has brought more evidence that radio is an industry in the midst of disruptive transformation.  The big news was that Viacom is planning to split off its broadcasting side, including the troubled Infinity radio division. But there were many smaller signals as well that radio as we know it is about to change. For instance:

* Arbitron and comScore Media Metrix released the online radio ratings for January, which for the first time include Live365. Led by Yahoo! Music, AOL and MSN, the four top streaming radio sites now have nearly 5m listeners a week. That's not only a lot, but it means (if I'm reading the stats right on the "persons using radio data" here) that the top Internet radio sites have more listeners than any US radio station.

* The WSJ had a great page one piece (inspired by our own cover package, I suspect) that discusses how the trend to ever-tighter playlists is now reversing as stations play more music. With listenership at a multi-decade low, they recognize that the quest to keep listeners by using market research to pick the perfect top 40 has now proven a failure. Instead, more stations are playing more eclectic, even iPod-like, mixes. They're also starting to cut down on ads, which today make up an oppressive 22 minutes per hour for many stations.

Meanwhile, there's been a lot of interesting Exploding Radio commentary and analysis around the web this week:

* Barry Ritholtz, a good financial analyst, adds an interesting perspective to the WSJ piece:

One thing I note as missing is a discussion of the long term generational effect, and the threat to a possible radio recovery: Since 1996, radio's decay has led to an entire generation of listeners who have essentially written off radio (at least, when it comes to music).

The other key issue: Radio as a source of new music, and its relationship to the labels. It used to be part of the draw -- a relationship with a trusted DJ who plays music you like, combined with introducing you to new songs (trust is the key component in granting someone taste-maker status).

* Technology Liberation Front, a blog worth subscribing to, has an excellent analysis of why concern over Clear Channel's radio "monopoly" is so misplaced. It also notes an amazing (and very encouraging) new white paper from the FCC:

"[T]he Scarcity Rationale for regulating traditional broadcasting is no longer valid.” So begins a stunning new white paper from the Federal Communications Commission. In the paper, “The Scarcity Rationale for Regulating Traditional Broadcasting: An Idea Whose Time Has Passed,” author John Beresford, an attorney with the FCC’s Media Bureau, lays out a devastating case against the Scarcity Rationale, which has governed spectrum & broadcast regulation in the United States for over seven decades.

* The Pew Internet Project released a report that finds that the Internet surpassed radio as a source of political news for the first time last year.

* NPR's "Here and Now" show had a segment on exploding radio, based on our issue (I was a guest). You can listen to it here.

* Fred Wilson, a NYC VC who has  written some of the best analysis of  radio's future, disagrees with my flip assessment that radio is "hosed".  Two responses: 1) I meant "radio as we know it" 2) That was "endism"--exaggerating decline for rhetorical effect--and it's okay.