The Long Tail

Meanwhile, at my day job....

I'm in Puerto Rico for the American Magazine Conference, where last night AdAge announced their annual A-List winners. I was named Editor of the Year. I'm both delighted and humbled by this, which is a every editor's dream. Most importantly, I owe a huge debt of gratitude to my team at Wired, led by Bob Cohn, Thomas Goetz and Blaise Zerega, who have not only worked with me over the past four years to make Wired what it is today, but also carried much of the weight this summer, when I was working on the book.

Scott Donatan, the editor of AdAge, wrote "Madison and Vine" a hugely influential book about the convergence of the advertising and entertainment industries, which has taken on a life its own as a conference series and more. He knows the power of a big idea, and it's safe to say that the Long Tail, both book and this blog, figured into his choice of winner this year. To Scott, my team, and the executives at Conde Nast who have supported and encouraged us every step of the way, a huge thanks.

A Long Tail Game Manifesto

Game industry verteran Greg Costikyan has launched a new independent games publisher called Manifesto Games that is built on a classic Long Tail strategy. Here's an excerpt from the announcement:

The site will offer independently-developed games for sale via direct download--a single place where fans of offbeat and niche games can find "the best of the rest," the games that the retail channel doesn't think worth carrying. Three types of games will be offered: truly independent, original content from creators without publisher funding; the best PC games from smaller PC game publishers, including games in existing genres like wargames, flight sims, and graphic adventures; and niche MMOs.

While games were once the domain of hobbyists, today, the game industry considers any title that sells fewer than 1 million copies to be a failure; "The typical game store only has 200 facings," notes Costikyan, Manifesto’s CEO. "They can only carry best-sellers. On the Internet, there is no shelf space and you are limited only by how well you can market yourself, your site. This is where niche product can rule."

Manifesto believes that an independent game market is analogous to film or music, where less commercial offerings aimed at identifiable markets and produced at lower budgets than the "blockbusters" can achieve profitability and critical success. "The game industry has become moribund,” notes Costikyan. "Because of ballooning budgets and the narrowness of the retail channel, it is now essentially impossible for anything other than a franchise title or licensed product to obtain distribution. Yet historically, the major hits, the titles that have expanded the industry to new markets and created new audiences have been highly innovative. It is time for us to find a way to foster innovation, because it's not going to happen if we leave it to the large publishers."

You can read more about this in a long and lively interview Greg did with Joystiq, and follow all the progress of the company on his blog.

How could cats kill us all?

At Google, the geeks (which is to say, nearly everyone) entertain themselves over lunch in the famous cafeteria by asking each other thought-experiment questions that don't necessarily have a right answer. Think of those notorious puzzlers ("how many manhole covers are there in the US?") that Microsoft used to ask in job interviews.

Anyway, a friend who works there relates that one recent question was this: "How could cats kill everyone in the world?" Apparently even Larry and Sergey tossed out amusing answers.

Sadly, I immediately thought of a less amusing answer: Cats could be a vector for avian flu.

Let's hope I'm wrong.

Richard Florida, Shameless Hitist

After a century of hit-driven thinking, it's hard to shake the habit. Seeing the world though the lens of the most popular things is second nature, and we all do it. But that doesn't make it any less of a mistake. I call this tendency "hitism" (or, to go with my Tail metaphor, headism) and the latest example of it I spotted was an article in this month's Atlantic by Richard Florida (The Rise of the Creative Class), entitled "The World is Spiky".

Florida's piece, which is nicely illustrated with fascinating maps, is meant to be a rebuttal to Tom Friedman's "The World is Flat." But it fails, but for a non-obvious reason.

John Hagel, who has posted his own interesting critique of the article, summarizes Florida's argument as follows:

Richard focuses on one particular quote from Tom’s book: “In a flat world you can innovate without having to emigrate.”  Richard responds that location still matters and that, by a variety of measures, the world is extremely spiky – meaning that activity is very concentrated in a relatively few locations.  Richard looks at:

• population concentration in urban areas
• light emissions (as an interesting proxy for economic activity)
• patent filings
• citations to scientists in leading fields to demonstrate this spikiness.

Focusing on the peaks definitely highlights the spikiness of the world.  For example,

When it comes to actual economic output, the ten largest US metropolitan areas combined are behind only the United States as a whole and Japan. New York’s economy alone is about the size of Russia’s or Brazil’s . . .  Together New York, Los Angeles, Chicago, and Boston have a bigger economy than all of China.  If US metropolitan areas were countries, they’d make up forty-seven of the biggest 100 economies in the world.

That seems indisputable. So what's the problem? The problem is Forida's insistence on measuring the world in terms of concentrations, which is akin to measuring culture in terms of hits. He defends this as follows:

Concentrations of creative and talented people are particularly important for innovation, according to the Nobel Prize–winning economist Robert Lucas. Ideas flow more freely are honed more sharply, and can be put into practice more quickly when large numbers of innovators, implementers, and financial backers are in constant contact with one another, both in and out of the office.

Again, it's hard to argue with that. But Friedman's point is that ubiquitous connectivity--broadband, cellphones, FedEx and so on--can achieve much the same effect as proximity. Sure it's easier to be a software programmer in Silicon Valley. But if you happen to be Indian and poor that may not be an option. Not long ago, being a poor Indian would have meant that you couldn't help make the next great operating system. But now you can.

In between Florida's geographic spikes are a million smaller concentrations, ranging from one person to a few thousand. Many of them choose not to live in big cities; many other don't even have that option. They don't show up as bright line on a map where the scale is set by cities, but they're there all the same in a faint tint spread across the land, just as the niches in the Tail of entertainment demand curves are overshadowed by the Head. Collectively--and that's what networks do: collect people by connecting them--all these distributed talents can add up to a real force, even though they'll never earn a pointy peak on Florida's radar.

In a previous life, I briefly built a videogame software company based around a microcluster of talented programmers based in Chiang Mai, Thailand. They lived there because it offered a concentration, to use Florida's term, of the quality-of-life factors that they cared about, from family to culture. Once, that would have come at the cost of their careers. But thanks to the Internet, they're now able to live in Friedman's Flat World, collaborating and competing with other programmers around the world. And there are hundreds of thousands of other talents like them, scattered to the four corners of the globe. They don't figure on Florida's map, but any one of them can have an impact. 

If you set your viewing scale to hits, that's all you'll see. But the real surprises are just as likely to come from below, in the noise below the spikes. I can't find Tallinn, Estonia amongst Florida's beautifully mapped peaks, but that didn't stop some smart people there from changing the world not once, but twice.

Friday Fanboy: Lego's Long Tail

I'm often asked for Long Tail examples outside of entertainment, especially those that apply to physical goods, not just digital bits sent down a broadband connection. There are quite a few (aside from eBay, the obvious one), but perhaps my favorite is Lego.

If you just know Lego from kids' birthday parties and the display shelves of a toy store, you've only seen half of the company. The other half is the Lego that caters to enthusiasts, ranging from kids who want more than the stock kits to adults who have turned to bricks as the ultimate prototyper's toolkit.

It all starts with Lego's mail-order business, which began as a traditional shop-at-home catalog and is now increasing organized around its website. In a typical toy store, Lego may have a few dozen products. On its online store, it has nearly 1,000, ranging from bags of roof tiles to a $300 Deathstar (shown). If you want to see how different the online market is from the traditional retail market for Lego, check out their topsellers list. Only a few of those products are even available in stores, and most of those are inexpensive items added to other purchases to bring them over $50 and thus qualify for free shipping.

It's worth pausing here and considering the Long Tail implications of this. At least 90% of Lego's products are not available in traditional retail. They're only available in the catalogs and online, where the economics of inventory and distribution are far friendlier to niche products. Overall, those non-retail parts of the business represent 10-15% of Lego's annual $1.1 billion in sales. But the margins on these products are higher than the kits sold through Toys R Us, thanks to not having to share the revenues with the retailer. And because the virtual store can carry products for all Lego fans, from kids to adult enthusiasts, and not just the sweet spot of nine-year-old boys, the range of prices can be a lot greater online, from $1 bricks to the aforementioned $300 Star Wars kit.

The next level of Lego obsession is joining one its two clubs. One gets you the monthly Lego magazine and catalogs. The other is the online club, where all the games and other cool stuff is. Basic membership for both is free, but if your kids are really into Lego you might want to consider upgrading to the Brickmaster level, which brings a bigger magazine with a lot of DIY projects (like MAKE for bricks), five exclusive kits that show up at your door, and a ticket to Legoland.

After that, it's time to start getting serious about your own creations. Lego has a long history of offering tools online to encourage model trading and other collaborative peer production. In 2000, its "My Own Creation" project led to a contest for the best user-created model. The winner was  blacksmith shop that Lego licensed from its creator and offered for a while as a commercial kit. After that, it offered Lego Mosaic, which allowed users to upload images that were converted into 2D Lego brick patterns, downloadable by all.

Earlier this year, Lego launched its most ambitious peer-production effort of all, Lego Factory. The idea is that you download software that allows you to design and build virtual creations, then upload them to the company. A week or so later, you get a kit with the necessary parts delivered in a box with an image of your creation on the front. What's especially cool is that others can buy your kit, too, and there's a nice selection of user-created models, such as this truck, available for purchase. More than 77,000 models have been designed this way, and some of the best of them are also being released as official Lego products (Lego pays the creators a royalty).

However, all is not what it could be in Factory land. Mass customization is cool, but when you have 7,000 possible parts in 75 possible colors (that's more than a half-million possibilities), the fulfillment challenge of offering users full freedom quickly becomes overwhelming. So Lego limits choice in two ways. First, each model can only be built from a single set brick palette, such as car parts. Second, those parts come in pre-packaged bags of a fixed number of bricks, so you'll likely get more than you needed. If you're not careful, a simple vehicle that might cost less than $10 in retail can turn out to cost nearly $100 in Lego Factory simply because it uses those bags of parts inefficiently.

Fortunately, there's a hack-around. Lego enthusiasts compiled a database of what bags were in which palettes and created software that helped builders use those bags efficiently, avoiding having to buy an expensive bag of parts for a single brick. And to its credit, Lego encouraged this. But that's still too hard and limiting for most people (including me), so Lego is now considering how to improve the experience, starting with easier-to-use design software.

I asked Michael McNally, Lego's senior brand-relations manager, whether Lego saw parallels in any other company's of approach to catering to niche market segments and encouraging peer production. Interestingly, he gave Apple's iTunes as an analogy. iTunes lets you download individual songs, not just albums. Although you can't upload your own music to iTunes, there are plenty of companies that can do that for you. And can make your own playlists and share them with other users, which is a bit like a custom Lego creation from standard parts. "What iTunes does for music, Lego Factory is doing for people who like to build," McNally says.

The Long Tail of social software

Last week I wrote about Salesforce.com's new AppExchange and the Long Tail of software. Now comes Ning, aiming to help create the Long Tail of social software, which only sounds the same.

What's social software? Well, at risk of getting into another Web 2.0 definitional mess, it's web-based software that builds on communities and peer-produced content. The old examples were social-networking sites like Friendster and LinkedIn, both of which I found annoying and ultimately of little use. The newer examples are far better: Flickr, Craigslist, Bloglines and Wikipedia, for example. They're not just community for the sake of connection; there's actually something to do there.

Ning, which is the fruit of Marc Andreessen's 24 Hour Laundry stealth project, aims to make it easy to create such sites for niche applications. It's designed for projects that are below the level of a business plan and dreams of an IPO. It's meant to bring big-community online tools and features to hypernarrow interests and needs.

Examples include a sermon exchange, a babysitting co-op or a college semester abroad community site. You might do it because you needed it yourself, and--who knows--it might snowball into something more. That's the great thing about community and user-contributed content. Simply providing a catalyst can often be enough.

I haven't had a chance to spend much time with it yet, but right now it reminds me of Joe Kraus's JotSpot, which uses the wiki approach to creating niche hosted applications. That is to say that it looks very capable, but it's too hard to figure out yet. Ning just needs a few more really cool examples to highlight on its front page, and then show people how easy it is copy them and make their own. And some sort of directory of the apps already created.

Right now I'm looking for a Ning site for Bay Area biking routes, and there's really no good way to do it. The search function, which is tag-based, returns nothing I understand. Give them time--I met with Marc and his team while they were in stealth mode, and they're smart and practical. They'll figure it out.

Business blogging != executive blogging

Another example of creeping "headism": Stephen Baker at Business Week warns that "The business blog backlash is nigh".

How long will it be before a parade of CEOs and other top execs turn their backs on blogging with a dismissive 'Been There, Done That?' It's the rare CEO who has the time and energy and openness to blog.

    I fully agree that it's hard to imagine many CEOs keeping up a decent blog for long. Not only do they not have the time, but the natural voice of the boss is fundamentally incompatible with the voice of the blogger, at least as regards their own company affairs (which is why I don't write much about Wired here).

    But it's a huge mistake to equate executive blogs with business blogging, just as it's a huge mistake to see the world only through the economic and culture lens of stars and hits (what I call "headism"). The best business blogs come from the employees, not the bosses. They have more time, and are less prone to marketing gobbledygook and gnomic platitudes. And those kind of blogs are on the rise, not the decline.

[Definitional update: I'm using "business blogs" here the same way Baker did--blogs from within a company that are largely about that company and/or its products. I'm not referring to blogs about business generally. Other terms for this are "corporate blogs" or "company blogs".]

    The most successful business blogs are peer-to-peer: engineers, designers and managers within a company blogging about their own projects for the engineers, designers and other customers outside the company who use those products or care about that project.

    Traditionally corporate messaging is directed up through the management structure within a company until it is released via an executive speech or press release, at which point it is supposed to be picked up by the press, filtered again, and trickled down to the public and ultimately the customers. But now that sort of top-down messaging is losing its effectiveness as consumers vote with their browser to go directly to the unfiltered voice of people like them.

    Simply put, we're starting to trust what executives say less and what employees say more. And if given a choice, as is the case with companies that let their employees blog, we'll take the word of an articulate engineer in the belly of the beast over the double-speak of a press release any day. As institutional credibility declines (from Enron to the White House), individual credibility is taking its place.

    Markets are conversations, not speeches. People want to hear from real people, not remote authority figures. Abstract institutions are turning transparent to reveal communities of regular people. And as that trend continues, consumer attention will shift from the institutional voice of the management to the human voice of their peers in the rank and file.

    In a speech last week at the m-squared marketing conference, I called this the power of "ants with megaphones". What's great about company blogging below the executive level is that it's impedance matched to blogging about the company from outside. Robert Scoble and Dave Weiner debating in public is a conversation of peers. Reading a New York Times account of a Bill Gates speech is not.

    Scoble himself expands on this in a great post that ends on the right note:

So, CEOs, if you don't get blogging, that's OK. It gives guys like me who are seven levels down from the CEO something to do.

Web 2.0 and the Long Tail, Part 2

After posting his Web 2.0 "meme map", which he apparently hadn't expected people to find until he was finished with an accompanying essay, Tim O'Reilly has now posted the full essay: What is Web 2.0. He'll be talking about it at his Web 2.0 conference next week in San Francisco.

I think Web 2.0 is a real and powerful concept, but the precise definition is, despite Tim's work, still a bit of a muddle. His essay, for instance, is approximately 6,000 words long and largely defines by example. You might quibble with his examples or think different ones make the case better. The Wikipedia entry is no clearer.

I don't think that means that Web 2.0 is any less important. In an era of bottoms-up creation and user-driven networks, the messy heterogeneity of the real world makes simple, monolithic definitions difficult.

What, for instance, defines a "blog"? Although you could try to give some common characteristics (individual voice? chronological order? permalinks?) my friend Jeff Jarvis is wise enough to refuse to answer that question:

A blog is merely a tool that lets you do anything from change the world to share your shopping list. People will use it however they wish. And it is way too soon in the invention of uses for this tool to limit it with a set definition.

This is no surprise. As I've said before, "The first rule of the blogosphere is not to generalize about the blogosphere."

Blogs are a Long Tail, and it is always a mistake to generalize about the quality or nature of content in the Long Tail--it is, by definition, variable and diverse.

The same goes for Web 2.0. Paradoxically, the lack of a crisp definition is a feature, not a bug. And as the world shifts from the limited variety of bottlenecked distribution to the infinite variety of open distribution, there will be more examples of phenomena that are hard to define but are nevertheless real and true. The future is increasingly heterogeneous, not homogeneous.One size doesn't fit all.

Regardless, I think Tim's off to a good start (and not just because he begins with the LT) in laying out a framework for Web 2.0 with these general principles:

Web 2.0 Design Patterns

1) The Long Tail
Small sites make up the bulk of the internet's content; narrow niches make up the bulk of internet's the possible applications. Therefore: Leverage customer-self service and algorithmic data management to reach out to the entire web, to the edges and not just the center, to the long tail and not just the head.

2) Data is the Next Intel Inside
Applications are increasingly data-driven. Therefore: For competitive advantage, seek to own a unique, hard-to-recreate source of data.

3) Users Add Value
The key to competitive advantage in internet applications is the extent to which users add their own data to that which you provide. Therefore: Don't restrict your "architecture of participation" to software development. Involve your users both implicitly and explicitly in adding value to your application.

4) Network Effects by Default
Only a small percentage of users will go to the trouble of adding value to your application. Therefore: Set inclusive defaults for aggregating user data as a side-effect of their use of the application.

5) Some Rights Reserved.
Intellectual property protection limits re-use and prevents experimentation. Therefore: When benefits come from collective adoption, not private restriction, make sure that barriers to adoption are low. Follow existing standards, and use licenses with as few restrictions as possible. Design for "hackability" and "remixability."

6) The Perpetual Beta
When devices and programs are connected to the internet, applications are no longer software artifacts, they are ongoing services. Therefore: Don't package up new features into monolithic releases, but instead add them on a regular basis as part of the normal user experience. Engage your users as real-time testers, and instrument the service so that you know how people use the new features.

7) Cooperate, Don't Control
Web 2.0 applications are built of a network of cooperating data services. Therefore: Offer web services interfaces and content syndication, and re-use the data services of others. Support lightweight programming models that allow for loosely-coupled systems.

8) Software Above the Level of a Single Device
The PC is no longer the only access device for internet applications, and applications that are limited to a single device are less valuable than those that are connected. Therefore: Design your application from the get-go to integrate services across handheld devices, PCs, and internet servers.

By the way, if you'd like to learn more about Tim, I can highly recommend Steven Levy's profile of him in this month's Wired. I'm biased, of course, but I think it's the best thing ever written about Tim, who really is emerging as the guru of the participation age.