The Long Tail

Nobody goes there anymore--it's too crowded

In the continuing battle against crimes of "hitism", let me take a gentle swipe at an otherwise excellent piece on the cultural implications of the peer production age by music critic John Pareles in today's NYT. He writes:

"The open question is whether those new, quirky, homemade filters will find better art than the old, crassly commercial ones. The most-played songs from unsigned bands on MySpace — some played two million or three million times — tend to be as sappy as anything on the radio; the most-viewed videos on YouTube are novelty bits, and proudly dorky. Mouse-clicking individuals can be as tasteless, in the aggregate, as entertainment professionals.

The key word there is "aggregate". Popularity is simply a place where many roads--each one a single consumer's path through culture--intersect. Each road is different, but for a brief moment many crossed that point. Hits are products that reflect the coincidence of our collective tastes, and the reality is that most of the things that we agree on are relatively banal (that's why they call it the lowest common denominator).

Individually we may have excellent taste, but collectively we're as low-brow as they come. This is simply an artifact of the statistics of the Long Tail--when demand is spread over a huge number of products, most things won't be popular. And the things that are popular won't necessarily define their consumers.

T'was ever thus: Yogi Berra's quote in the title reflects the reality of minority taste. We're as likely to avoid doing what everyone else is doing as were are to join them. For the discriminating, popularity is often a curse, even if it was their early embrace that kick-started that popularity in the first place.

Once the most popular fare defined our culture. Now a million niches define our culture and the few blockbusters are the exceptions that define none of us, even through many of us brush by them. 

David Foster Wallace, writing about television, said it best:

"TV is not vulgar and prurient and dumb because the people who compose the audience are vulgar and dumb. Television is the way it is simply because people tend to be extremely similar in their vulgar and prurient and dumb interests and wildly different in their refined and aesthetic and noble interests."

(Image taken from our photo shoot this month for our cover story on LonelyGirl15, an example used in the NYT.)

I, for one, welcome our new Christian homeschooled Lego robotics overlords

This is a bit off topic, but my fondness for Lego robotics cannot be suppressed. Check out this video of a team of Christian home-schooled 9-14 year olds winning the New Hampshire FIRST Lego League Nano-Quest Challenge in a single autonomous outing. There are so many impressive things going one here that it's hard to highlight just one, but the fact that the robot changes its own tools is unbelievable.

In our house, we're still learning the basics such as line-following. This turns out to be a real challenge because we only have the one light sensor that comes with the basic Mindstorms NXT kit. In the FIRST Lego League rules, you can have two light sensors which makes it relatively easy to stay on the black line (Left dark, right light--turn left. Right dark, left light--turn right). But with just one sensor the fact that you're not seeing dark anymore doesn't tell you which side of the line you've gone off. It's like having a single-pixel camera--there's no sense of position context.

The solution, we've found, is to add noise to the system. By programming the robot to follow an s-curve motion at a higher frequency than the curves of the line on the table, you can predict which side of the line you've just crossed when the sensor records a transition from dark to light. Adding the known noise of a sine curve allows you to make sense of the unknown noise of the boundary transitions as the robot leaves the line, on one side or another. I'm sure there's some fancy math to optimize this for maximum speed and minimum thrashing about, but we just tweak it until it works.

Sorry for the random geek-out, but I am unreasonably delighted that we got this to work.

(Update: I know that the usual method is to simply follow the edge of the line, which has a similar effect--construct the robot with a slight left bias to its forward motion and keep correcting to the right when you hit the line. But we wanted to actually travel on a quite thick line, which is like driving down a road. The reason is that in many of the official Lego line-following contests, you need to go straight through a four-way intersection, and edge-followers tend to fail there.)

I'm not responsible for this

Reason Online reminds me of a New York Magazine story from July that attributed to the Long Tail the strange success of Kevin Smith's DVD lecture series (featuring "thousands of young Canadians who came to a Toronto auditorium to hear Kevin Smith joke and whine about his life in the movies. The audience members range in age and gender, but the vast majority are young, awkward and male - late teens and early twenties.")

Reason asks::

How did these people become Kevin Smith fans? If you assume that people become fans of a director or movie star by seeing his/her movies, there hasn't been a reason to worship Kevin Smith in nearly a decade, since the well-constructed sex comedy Chasing Amy. But Smith hasn't made an influential movie or created any lasting characters since 1994's Clerks. If you're one of the 18-year old college freshmen who's buying a ticket to see Smith right now, you were six when Clerks was released.

New York Mag answers::

The long-tail economy makes a passionate fan base more important to entertainers than ever: In the same way that the real money for rock stars has been in merchandise and concerts, a filmmaker’s fortune isn’t just dependent on ticket sales but on video-on-demand, online downloads, DVDs, and then special-edition DVDs; in short, on the ardor of his devotees. So, naturally, every with-it director is on MySpace—but Kevin Smith has them all beat by a mile. “He was so ahead of his time, because he was always communicating with his fans,” says Harvey Weinstein.

Reason is not convinced:

Anderson produced a more compelling theory in his interview with Nick Gillespie.

The fact that you and I both watched American Idol last night probably doesn't define us, whereas our niche interests really do. We go deep and find people who share our affinities, which represent much tighter connections between us. So my suspicion is that we're going to have fewer loose connections with lots of people but tighter connections with fewer people.

That's why Kevin Smith's popularity is indestructible, while his friend Ben Affleck's career exploded then spent a few years tumbling back to earth. Smith doesn't produce movies; he produces new offshoots of the Kevin Smith brand. Just like they did 10 years ago, lonely/nerdy/smart teenage boys see in Smith a humor they identify with and a personality they want to emulate. The movies are incidental: Something like Clerks II has the relationship to the Smith brand that a communion wafer has to Catholicism.

Radical Transparency and National Security

Clive Thompson has an excellent piece in today's NYT Magazine that asks whether blogs, wikis and other social networking technology could do a better job of intelligence gathering than the traditional methods of the spy agencies. The simple answer is that they could hardly be worse--the article is sobering reminder of how ancient and feeble existing intelligence technology, such as it is, really is. The databases don't talk to each other, the instant messaging doesn't interoperate and classified information gets accidentally released because the filters, appallingly, weren't programmed to catch "Secret" in all-caps or with spaces between the letters. The average spam filter is apparently light years ahead of the state-of-the-art in military data analysis.

But the really sobering thing is Clive's reminder that this is not just due to incompetence and bureaucracy. US intelligence is also intentionally hobbled by laws designed to protect privacy and civil liberties. Total Information Awareness probably would have failed on technology grounds, but it never even got a chance to prove anything one way or another because it ran afoul of laws and politics first. Since we think those laws are Good Things, Clive asks whether a more open alternative--the Web 2.0 tactics of peer-production and the wisdom of crowds--within the intelligence services are a better approach. Rather than trying to work around the laws or building increasingly complex and fragile systems, why not tap the collective intelligence of the people already in the system operating under the existing rules by simply letting them communicate better with the same tools that made the web what it is today?

His thesis is that radical transparency within the intelligence services could compensate the failures of command-and-control information sharing. Perhaps it could. But I was surprised that Clive didn't take the question to the obvious next level. What if, rather than just starting blogs and wikis behind military firewalls where the rules are most strict, the intelligence agencies encouraged them out in the open, catalyzing conversations between people who aren't constrained by the same laws? Between the intelligence analysts, military bloggers, and intelligence technology providers, there are already loads of spook-types blogging in public. They're not writing about classified stuff, of course, but there is plenty of information in the unclassified world that could lead to useful intelligence if only it were easier to connect the dots. Given enough eyeballs, all bugs are shallow.

In other words, what if spotting early-warning terrorism signals became an open-source project?

That's not such a crazy idea. It is, after all, the core thesis of David Brin's increasingly-prescient "The Transparent Society". It's also quite brilliantly imagined in the beginning of Rainbows End, the new science fiction novel by Vernor Vinge. Here are the first four sentences of Vinge's book:

"The first big of dumb luck came disguised as a public embarrassment for the European Center for Defense against Disease. On July 23, schoolchildren in Algiers claimed that a respiratory epidemic was spreading across the Mediterranean. The claim was based on clever analysis of antibody data from the mass-transit systems of Algiers and Naples.

CCD had no immediate comment, but in less than three hours, public-health hobbyists reported similar results in other cites, complete with contagion maps." 

Silly sci-fi stuff? Compare it with the real-world example cited in Clive's story:

"In July, [the Director of National Intelligence] staff decided to create a test blog...that would focus on spotting and predicting possible avian-flu outbreaks and function as part of a larger portal on the subject to collect information from hundreds of sources around the world, inside and outside of the intelligence agencies. An agent in Southeast Asia might be the first to hear news of dangerous farming practices; a medical expert in Chicago could write a crucial paper on transmission that was never noticed by analysts.

In the months that it has been operational, the portal has amassed 38,000 “active” participants, though not everyone posts information. By September, the site had become so loaded with information and discussion that Rear Adm. Arthur Lawrence, a top official in the health department [said] it had become the government’s most crucial resource on avian flu."

UPDATE: Clive reminds me that one of the best examples of this is the community and services formed around Oregon and California's Emergency Digital Information Service project. And the best story on that is Gary Wolf's seminal piece in, er, Wired a year ago (I knew that!). "Reinventing 911: How a swarm of networked citizens is building a better emergency broadcast system." Read it.

UPDATE2: Blaise Zerega (ex Wired, now Portfolio) reminds me of another prescient Wired story. "We Need Spy Blogs: An Army officer calls for better information gathering"

Living with "No Visible Technology"

Many of the greatest innovations have their roots in constraints. Having to work around limits, be they cost, size, weight, power or simply rules, shrinks the possibility space but also focuses the mind. And the workarounds can often open paths to something new and extraordinary. Just think of the T9 predictive text entry system that makes texting on phones with only 12 keys nearly as fast as on a full-size keyboard. Or the iPod's scroll wheel, designed to make navigating a huge library of music possible with the simplest interface imaginable.

Even Google sees the virtues of limitations, be they screen real estate or download times. "Some of the most inspiring art forms--haikus, sonatas, religious paintings--are fraught with constraints," wrote Google's Marissa Mayer in Business Week. "They're beautiful because creativity triumphed over the rules. Constraints shape and focus problems, and provide clear challenges to overcome as well as inspiration. Creativity, in fact, thrives best when constrained."

This is the flipside of abundance. Yes, when things become nearly free, it can liberate all sorts of creative thinking about how to exploit them. But when things become really expensive (or scarce), that too can stimulate creative thinking about how to use them most efficiently, or not at all. Necessity, after all, is the mother of invention. For more than two decades India blocked US computer technology in a misguided effort to stimulate a domestic computer hardware industry. It failed, but it did train a generation of Indian engineers to become incredibly good at writing code that used underpowered computers most efficiently, a legacy that helped birth a software superpower once those import restrictions were lifted.

At our house the import restrictions are set by my wife. To bring some sense of balance to our lives (being married to the editor of Wired is a constant battle against geek overload), she has a general rule that, outside of the study and the kids' rooms, there should be no visible technology. Aside from the usual kitchen appliances and the occasional Romba intrusion, we pretty much stick to this--most of the house, aesthetics aside, would have looked little different in any decade from the past half-century.

This presented an interesting challenge when it came time to set up a home theater in the living room. We already had a regular-sized TV hidden in a Chinese wedding cabinet (above) and a stereo behind the doors of some built-in shelving, with speakers behind curtains. But a big screen and 6.1 surround sound would be harder to disguise.

The main solution was to go with a high-def projector. We found a clear rectangular expanse of wall on the other side of the room and put a Panasonic PT-AE900U above the TV in that same Chinese cabinet, along with a proper home theater amplifier and an Xbox 360 as a Windows Media Center extender and HD DVD player (as well as a high-def game console, of course, although most gaming happens on the kids' 360 upstairs). The components are all connected to the Media Center in the study via ethernet (for DVR video streaming and gaming) and optical digital cable (for music, which is all streamed from Rhapsody running on the Media Center), which required drilling a hole in the floor and snaking the cables under the house to the study. 

Surround sound was even easier, thanks to the fact that it doesn't matter much where subwoofers go (ours is hidden under that same cabinet). The satellite speakers are small enough to tuck away behind curtains and in corners behind pictures, although the wiring required more drilling in the floor. The larger front channel speaker is usually hidden behind the fireplace screen (except when we light a fire).

The result is a room that pretty much adheres to my wife's rule--it looks like a living room, not a screening room. If you look closely enough you can find some of the speakers, but you'd have no idea that it can turn into a 15-foot-screen high-def movie theater by just opening a cabinet door. The cost of this sleight of hand is that it's certainly not up to videophile standards--we're projecting on a tan wall, not a proper screen, and the bottom of the screening area has a part where the fireplace overhang slopes out a little, which slightly distorts the bottom of the picture. Also projectors are only really satisfying at night, when ambient light isn't a problem (we use the smaller TV if anyone needs to watch something during the day). But the system is still plenty good enough to push my geek buttons without triggering a technology ban. Constraints conquered, problem solved.

Pixar Quiz

I recently had coffee with a friend at Pixar and he mentioned a surprising stat, which I'll phrase here in the form of a quiz:

Q: On 1995 computer hardware, the average frame of Toy Story took two hours to render. A decade later on 2005 hardware, how long did it take the average frame of Cars to render?

A: 30 minutes

B: 1 hour

C: 2 hours

D: 15 hours

Highlight the invisible text below the pictures for the answer:

 

Answer: D. The average Cars frame took 15 hours, despite a 300x overall increase in compute power. The artists have an essentially infinite appetite for detail and realism, and Pixar's resources have grown over the decade so it can afford to allocate more computers to the task, allowing each to run longer to achieve the artist's and animator's ambitions for the scenes.