Microsoft and the Long Tail of search

A former senior Microsoft manager (he didn't want his name used) emailed me with an interesting perspective on the Long Tail of search:

As you know, search engine query logs have a Zipfian distribution (Rank * Frequency = Constant).  When I presented this concept to senior execs at Microsoft, including BillG, they never quite got it.  I would draw the graph just like the logo on your site, but it just wouldn't sink in.  I realized that the size of the X axis was the problem.  People see a graph, but they don't comprehend the scale of the X axis. 

Looking at your logo, for example, it looks as if the (yellow) long tail is what, 4x, 5x the size of the red portion?  And it is, if you are comparing the integral of the yellow portion (i.e. query volume) compared to the red.  But the X axis, on a linear scale, extends almost infinitely to the right, and no visual can communicate that.   In essence, in order to represent the long tail in graphical/visual form, you implicitly have to represent the X axis logarithmically, as you know, and people (generally) don't comprehend logarithmic / exponential scale.

He eventually found a way to explain the curve using stacks of pennies, which ends up with a few high stacks followed by a trail of single pennies extending out the door, down the hall and, if you have enough pennies, as far as the eye can see.

He also found a handy rule of thumb to estimate the consequences of this distribution:

Loosely speaking, if you divide the number of queries by 4, you'll get the frequency of the most popular query, and if you divide by two you'll get the number of queries that occurred only once over whatever time period you are measuring.

Finally, it turned out that Zipf distribution turned up in all sort of unexpected places at Microsoft:

A second example refers to what the external world knows as Windows Crash Analysis but what inside Microsoft is known as Watson or Dr. Watson (of "come here, Watson, I need you" fame) because that's what it was first called.  Anyway this is the dialog that appears when a Windows application crashes on Windows XP -- an alert appears and offers to send the information to Microsoft.  Back at Microsoft, they compile that information in a SQL Server database, indexed by application name, module name, module version, and the internal address where the crash occurred.  Back in 2001 or 2002, they started showing off a graph of all the application crashes, with Rank on the X axis and Frequency on the Y axis.  Internally they called this the "Watson Curve," even in front of Gates. 

When I saw the curve, I smiled, because it looked familiar.  I asked one of the guys on that team to do me a favor and plot the results log-log.  He got back to me a few days later and said "wow, it's a straight line!"  I wasn't surprised.  I don't think they call it the Watson curve anymore because it's just yet another example of a Zipfian distribution at work.   

For more, check out this amazing site, which is all about the many Zipfian distributions found in the real world.

Comments

[Barry Ritholtz]

That's Fascinating stuff to a numbers geek (like me) -- I would hope you include it in the book.

Of course, the more math, the less copies you will sell.

Whaich raises a question: Which part of the fat head/long tail do you hope the book occupies?

[ecolson]

Very interesting stuff. However, regarding this stat:

"Loosely speaking, if you divide the number of queries by 4, you'll get the frequency of the most popular query..."

My research shows that 4 is much too small of a constant to characterize the Search Head. That is to say, the head is not nearly that large - rather, you would need to divide the total queries by a number like 100 to 400 to get the frequency of the most popular query.

I would be curious to see where the stat came from.

[Mathew]

For the best on the long tail, Zipf, etc,
Try reading Mandelbrot and Taleb:
http://misbehaviorofmarkets.com
http://fooledbyrandomness.com

[chris anderson]

Barry,

Well, I'd like to put a good bit of math into the book and accept my inevitable place down the tail. But my publisher, whose money is paying the rent, would rather that I not. At the moment, there is exactly one equation in the book (unsurprisingly, y=ax^k), but we'll see if that survives the editing.

[Adam Marsh]

Funny, I just wrote a post on this exact topic:

"Loosely speaking, if you divide the number of queries by 4, you'll get the frequency of the most popular query."

This says that in a Zipfian distribution, the top bin gets 25% of the total. As I detail in my post, the percentage of the total in the top bin depends upon the number of bins, assuming that the distribution follows an exact inverse power law. According to this reasoning, if the top query gets 25% of the total queries, there were less than 100 queries!

Of course, in fitting real data to a power law all kinds of offsets, etc. are swept under the rug, so this isn't really a fair thing to say -- but it is an interesting topic in general. You can find more on similar long tail / power law stuff here, here, and here.

[Michiel]

Doesn't have a lot to do with the Tail, but dr. Watson: I turn it off. On all workstations. People don't get it, or get annoyed. They are much happier when the application just crashes and they can restart it.

[MelbournePhilosopher]

I came here from out-of-context. Frequency of *what*, measured in units of what, according to whom?

Cheers,
-MP

[Mendy]

This says that in a Zipfian distribution, the top bin gets 25% of the total. As I detail in my post, the percentage of the total in the top bin depends upon the number of bins

[Arik]

That is to say, the head is not nearly that large - rather, you would need to divide the total queries by a number like 100 to 400 to get the frequency of the most popular query.

[levitra]

Among all the erectile dysfunction pills available in the market, it is levitra that has carved out a niche name for itself and levitra prescription has become much sought after for the impotency afflicted patients. The chief factor responsible for the tremendous popularity of levitra is that this medication to treat erectile dysfunction has shown successful results in treating impotency in men with health risks like hypertension, diabetes and prostate surgery and as such you should opt for levitra prescription online instantly. Bear in mind that levitra without prescription would not be able to show off the necessary results as levitra is a prescription-based drug only and should be taken only according to the instructions of the doctor.

[gtgg]

fggffgfg

[gtgg]

fggffgfg

[gtgg]

fggffgfg

[debt consolidation]

In the fast-paced uncertainty of the modern hoodia it's all too easy to fall behind with payments until one day you realize you have serious hoodia chaser problems. Other than the blow to your self-esteem this causes, the stress that arises from being in debt can also have devastating effects on your hoodia gordonii and mental health. So if you're living from paycheck to paycheck and suffer the constant gnawing worry that your property will be repossessed, you really need to sort out your how much debt problems as quickly as possible. If you're a homeowner, a very attractive revitol known as sell and rent back is open to you. So if debt is making your life unbearable, consider the many advantages of this scheme.

First you contact a reputable menozac specializing in sale and rent back schemes. After surveying your property, the company will make you an offer on your home. Then in addition to an immediate cash payment, they will issue a contract for you to stay on in your home as a tenant paying rent at the market rate. Very often, debt loan they will offer a rent holiday. penis enlargement You have a lot of say in the contract you agree to; you can specify for example whether you want a short-term rental or long-term arrangement. You can also request a buy back option which gives you the option to later buy back your house when your situation improves.

Why It's The Best Way

The major bowel advantage of course is that you can stop smoking all your debts and continue living in your home as before. You don't have to suffer any of the upheaval that accompanies moving house especially if the move is unwanted. What's more, you get a quick, guaranteed sale with none of the debt consolidationwaiting around and having potential buyers traipsing through your home day after day. Plus, you don't have to pay and estate agent's commission on the sale.

There's also the matter of discretion and privacy. Reputable prostate health can arrange a sale that's private and confidential. There's no need to put up a 'for sale' board in front of your house or put an advert in the local press. So friends, neighbours and colleagues are none the wiser to your circumstances.

A number of other methods exist to release the equity tied up in yourunderstanding debt home. These include lifetime mortgages and home reversion plans.

However the Which Consumer Group reported that these other schemes tended to be expensive, inflexible and risky.

Studies have shown that being in diabetes debt is major cause of stress with serious consequences for both your mind and body. So if debt is wreaking havoc in your life, look into the sale and rent back scheme.

Be sure to choose a specialist firm with bad debt loan many years of experience in this field. Only a specialist firm can give you the efficiency and discretion you deserve.

[power]

wow power leveling
wow powerleveling
cheap wow power leveling
wow power leveling
wow powerleveling
wow power lvl

[Debri]

Great articles. added your blog in favorites viegra

[asdad]

货架 月嫂 大屏幕 婚庆公司 吸声喷涂 进口轴承 保洁 保洁公司 性病 加固 滑雪 高血压 北京滑雪 滑雪场 北京滑雪场 梅毒 疱疹 前列腺炎 尖锐湿疣 非淋菌性尿道炎 高血压治疗 治疗高血压 北京真人cs 司法考试 国际机票 保温喷涂 机柜 北京保洁 北京保洁公司 国际机票 婚庆 标签纸 网站流量分析 条码打印机 星辰科技 司法考试 拓展训练

[acne treatment]

Acne treatment best rated product. Clear pores natural acne treatment products for skin care. Acne cure NOW!

[sky]

大人のおもちゃを即日発送します。
ドレス 通販を中心に幅広く扱うセレクトショップです。
高級レギンス専門店。お買い上げ金額5000円以上で送料無料！レギンスとカラータイツの専門店です。

[Magnificent]

大人のおもちゃを秘密厳守で全国に即日発送します。
あなたのクレジットカードのショッピング枠 現金化を最短５分で。公安委員会許可。安心と信頼の１８年の優良店。
不動産担保融資においてもお客様のあらゆるニーズにお応え致します。