The Long Tail

Annual “cutting the Long Tail down to size” roundup

Around the Christmas season each year there are a flurry of stories about hit toys, books and other retail blockbuster phenomena that seem to defy the overall trend towards more niche demand. This year is no exception, so I’ll round up a couple of the recent pieces here:

1) Quite a good article in New Scientist surveys recent research on consumer behavior in the face of massive variety, and concludes that we still like to buy stuff that other people buy. It mentions the Elberse research as well as comments from Duncan Watts about how socially connected groups tend to clump in their taste. (Image from the article shown above.)

My take: I’ve already discussed the problems with Elberse’s research (it’s based on percentages, not absolute numbers, which become meaningless when inventory grows by orders of magnitude. She also defines head and tail in a way that doesn’t make sense and doesn’t correspond to my own definitions).

Watts’ work is more interesting, and touches on the same point Eric Schmidt from Google made about network effects creating winner-take-all consequences. My answer to that is that fortunately social media creates an infinite number of networks, many of them focused on niche subjects, so that many winners can take “all” of their micromarket, while still having the collective effect of redistributing demand in the entire market over more variety. This is the “fractal dimension” of the Long Tail that I’ve written about in the blog and the book. 

2) The Times UK did a piece on Will Page’s work on music sales, which I wrote about here. The piece is a bit confused, and seems to think this refers to all music online, rather than some UK online retailer(s) that has not been disclosed, as is actually the case.

My take: Nothing new, and my problem with the work remains. Because Will hasn’t published the data or said where it came from, we can’t really know what it means. Is it iTunes UK? Some mobile provider? Ringtones? Some streaming provider? Some combination of the above?  Ugh.

His observation that the data is fit better by a lognormal than a powerlaw is interesting, but until we know what filters were available in that marketplace (or marketplaces), we can’t say whether that’s surprising or not. In general, marketplaces with good filters (recommendations and other discovery tools) tend to follow a powerlaw, where marketplaces with poor filters follow the steeper lognormal of the older bricks-and-mortar markets. Again, there’s a whole chapter on this in the book.

Interestingly, the article says that Page and his co-author plan to write a book on this. No doubt they’ll come up with a better title than “Some Online Markets are Better Fit by a Lognormal Than a Powerlaw”, but it will be fun to see how far they can take this.

The Conservation Law of Transparency (you can't be open in all things all the time)

I wrote the Long Tail in public here on this blog. Not every word, of course, but most of the main ideas emerged here first. I was thinking out loud, beta testing my theory with my readers. In exchange for sharing my research in progress, people helped make it better. It worked great.

Now I'm pretty much finished with Free (it will published on July 6th; the wheels of the book industry move slowly) and I'm reflecting on why this one took a different path. Although I blogged a lot about Free (and will continue to do so), very little of the book proper can be found on these pages.

Part of that is that it's a much more narrative-driven book, with lots of history (as befits an economic phenomenon that predates money), and that kind of writing is best done in isolation. It also reflects the fact that I've been thinking about this as a book from the start, so by the time I started writing it was all pretty well worked out in my head.

But I'm no less wedded to transparency and sharing work in process than I was. It's just that I've shifted that method to one of my other projects, the open source hardware company that I'm running on the side. At DIY Drones, we've been blogging daily on the progress of our airplane autopilot and autonomous blimp projects. Just as with the Long Tail, the community there is helping making the better and otherwise collectively beta testing them. We give away our ideas and work and people return the gift with their own ideas, feedback and encouragement.

I wonder if there is some sort of conservation law at work here. Perhaps most people can only be truly transparent about one thing at a time. After all, it takes a lot of energy to loop the whole world into every twist and turn of your progress. Transparency is hard work. Constantly updating the world on your status can become a job all by itself.

For example, our experiments with transparency at Wired have mostly been one-offs. The behind-the-scenes politics of a Microsoft story. The creation and editing process of a Charlie Kaufman profile. Collective editing of Wired.com story.

Why don't we do this with every story? Because it's a huge amount of work, easily doubling the time required for any project. We can only do a few a year, and that's why it's been relegated to proof of concept rather than standard practice. Nobody's figured out how to introduce true transparency into company practice without making it somebody's full-time job.

That's why we see so little true transparency in practice (Fred Wilson's superhuman efforts aside) and even those like me who are drawn to it have to relegate it to just a single aspect of their life. For all that Twitter and other microblogging makes lifecasting and other status updating easier, for most people it still feels like another obligation, taking time to do well and causing guilt when neglected.

You can't be transparent about everything all the time. Indeed, you're lucky if you can be transparent about just one thing well. Maybe someday we'll figure out parallel transparency, but for now serial will have to do.

The rise of "retail blogs"

We've been through lots of experiments in giving companies a human voice, few of them successful. Paid pitchmen and celebrity endorsements. Awkward commercials featuring the aw-shucks CEO. Quirky corporate styles and creation myths. In the end they all feel as manufactured as they are.

But now I'm seeing a new approach that works far better. Some of the smaller companies that I follow are redesigning their websites to put their blog front and center. Not a link to the blog at the bottom of the page, but a front page that is almost all blog, with the products relegated to a sidebar. For lack of a better term, I call them "retail blogs".

One of the best examples of this is Sparkfun. This is a company that caters to electronics hobbyists and DIY hardware hackers, of which I am one thanks to my UAV/autopilot side project. Speaking of which, this is what Sparkfun's site looks like today:

See? It's all voice and community. This is a corporate home page that you'll want to subscribe to in your RSS reader--the mix of insidery chat about new products with fanboy admiration of customer projects is infectious.

Other examples include Woot's hilarious blog-like new product listings each day:

Others are just the blog side of a physical store, and so they're not as well integrated with online sales, such as the trendy Reed Space store in NYC:

 

Then there are blogs that are pared with a retail side, like the fantastic Make blog + Maker Shed store and Slashdot + ThinkGeek.

As DIY Drones starts releasing products, this is the direction we'll go. Blog and community first, selling things second. Voice builds trust and trust sells stuff.

Do something new every three years

When I was at The Economist, there was a policy to rotate everyone every three years. The idea was that fresh eyes were more important than experience. "Foreign everywhere" was the mantra, and around your second year in Cairo, you could expect to get a call from the editor asking you to consider Mumbai or Sao Paolo--ideally two places you'd never been to and knew nothing about.

The first year after arriving to your new assignment was terrifying and exhilarating. It was a vertiginous learning curve, but you could ask dumb questions without fear and note that the emperor has no clothes.

In the second year, after the emperor had invited you in a few times to explain the subtle political dynamics that require him to go garbless for the ultimate good of the nation (but surely there were more important things to write about, such as his new elevated rail project), you would find yourself writing sophisticated analyses, traveling easily through the region, admiring your bulging rolodex and otherwise feeling very productive.

In the third year, you'd find yourself returning to stories with a certain cynicism and worldweary accounting of endless process. The elevated rail project has been delayed once again because of infighting within the opposition party. The emperor has no fiscal discipline. You understand everything all too well. It's time to move on.

For the first ten years of my career, I changed jobs every three years. Then, for the seven years I was at The Economist, I changed countries every three years (London, Hong Kong, and New York, although sadly not long enough at the last). Here at Wired, I seem to have achieved the same rhythm by publishing a book after my fifth year and, next summer, my eighth. Each time it changes my life and puts me back on a steep learning curve with a new subject to immerse in and a new pace of travel and speaking. I've got a new foreign land to explore.

I was thinking about the three-year rule while reading about Malcolm Gladwell's observation that it takes 10,000 hours to become truly expert at something. If you really throw yourself into a job, you'll spend 60 hours a week working. That's 3,000 hours a year (allowing for vacation), which means you'll hit the 10,000 hour mark a few months after your third year.

So maybe that's where the three-year rule comes from. You're now expert at what you set out to master. Great. Now go do something else.

Some interesting pricing stats in iPhone apps

Ben Lorica at O'Reilly has updated his analysis of pricing trends in iPhone apps. Although it doesn't include the actual sales figures for the apps, it does look at the number of apps in each price category and the top 100 most popular.

Some highlights:

During the last week of November, there were close to 9,800 unique apps, 22% of which were free:

The average price of a Top 100 paid app continued to decline, falling to a little over $2.60 in the last week of November:

 

In both the Social Networking and News categories, the Free outnumber the Paid apps. (Social Networking apps tend to be apps designed to help users access social web sites from their iPhone, while News apps do the same for news/media sites.):

Living life without money

A WSJ video about an economics graduate in the UK who's going to go a year without using money. It sounds like a step up from dumpster diving for food to me, but perhaps he's got a more subtle plan than this video reveals.

[Thanks to Barry Ritholtz for the tip]