Anyone who’s used Twitter or follows Web 2.0 knows about Twitter’s problems. They became a victim of their own success when they became the new “it” thing, gaining hundreds of thousands of new users, but were unable to scale adequately to keep up the level of service. Since then, the “Fail Whale” graphic they use to show users that they’re having problems has become a famous inside joke amongst users and pundits about the problems Twitter and other Web 2.0 platforms are running into, including my own Hunting the Fail Whale post (buy a shirt). Since the problems started, numerous suggestions have been put forth for how Twitter can scale their service, both plausible and implausible, well thought out and not. One particular idea that’s caught hold recently has been the idea of de-centralizing and distributing the Twitter service, much like Email servers are run by all ISP’s. The technology being put forward to support this is a new extension to the Jabber XMPP instant messaging spec (the standard used by Google’s GTalk) , Microblogging over XMPP. You may think this is a great idea, and that you want a distributed Twitter-like service, but you don’t, and I’ll tell you why, but first let’s look at the advantages of a distributed Twitter…

Advantages of a Distributed Twitter

Twitter goes down. It’s almost Twitter’s defining characteristic at this point. For people addicted to Twitter for communicating and keeping in touch with friends using Twitter, it’s a frustrating but all too frequent occurrence to see the Fail Whale mocking them from their screen. Twitter has become the most famous single point of failure in software history. The problem Twitter has is that it can’t scale up by simply throwing more hardware at it. The reasons why are the topic of another post, but the fact is that as the number of messages going through Twitter goes up, it reaches a breaking point where it can no longer serve those messages. They can move that point up or down by disabling and re-enabling some resource-intensive services, but the breaking point remains.

The obvious solution to this problem is to spread out the load and let other sites handle part of the load. In a decentralized system like email, any one email server going down does not effect the rest of the servers, and email is still delivered for everyone else. Each mail server can scale its own box or boxes to match their load, and no one mail host has to handle the entire load.

A distributed microblogging infrastructure like this could work, eventually, but is it the right direction?

The problem with obvious solutions is that they copy old models, whether they’re a good fit or not, and they ignore the non-obvious, innovative ones…

Disadvantages of a Distributed Twitter

Why aren’t we just using email and mailing lists if this model is a good fit? The problems with a distributed microblogging system are the same as the problems with email. While there’s no single point of failure, there are many points of partial failure. The lag time of a distributed system make email less realtime and give a different feel to the communication. But these are minor points compared to the real issues.

Innovation

What was the last innovation in email? I’m not talking about email clients or contact management, I’m talking about the email infrastructure. What was it? IMAP? How long ago was that? Email is built on a set of standards, and is implemented by a number of different servers, clients, etc. All this makes innovation in the email space move at a glacial pace. Any change has to go through a standards body, which immediately makes it a long-term process, and then it has to go through an adoption and implementation cycle through servers and clients which may take years as not all servers will feel the need to upgrade immediately.

Microblogging’s potential for innovation has not been mined out yet. It’s barely scratched the surface, and been stalled for over a year by Twitter’s technical issues. Standardizing and distributing microblogging would kill innovation in the space and we’d be left with exactly what we have now and nothing more. That would be a shame, as there are several new ideas I want to implement in microblogging.

Rogue Nodes and Users

In fact, the most innovation in email has been around identifying and correcting misbehaving users and servers, i.e. spam filtering. Distributed systems are fundamentally susceptible to rogue elements using the infrastructure for their own purposes. Think of how much spam you get in your email inbox. Think of how many spams you don’t get on Twitter. An open, distributed system will be open for spammers to set up their own servers and ignore normal subscription rules to send messages directly to users. It will be the spam wars all over again, with the spammers working to stay one step ahead of the filters. With a centralized system, there is much more control over subscriptions, and they cannot be gamed from the outside.

No Single View of the System

This isn’t so much a problem with email, since it never existed there, but with a distributed Twitter system you’d lose the ability to look at all of the activity happening on the platform. That means not only is there no single user registry, but there’s no room for 3rd party add-on apps. There’s no public timeline, no Summize, no Tweetscan, no WhoShouldIFollow, no Twellow, etc. Those things are only possible because there’s a single place to look for the information. What’s more, these services have only begun to scratch the surface of mining the information in the social graph. With a distributed system, all that value is thrown away.

So What do people really want?

Users like Twitter. That is, they like Twitter when it’s working. Turning off the features that made Twitter easy and convenient to use, like Jabber IM support, have really hurt Twitter’s reputation. Users want Twitter’s ease of use AND the large number of users, because the network effect of more users is what really makes Twitter valuable. Unfortunately, Twitter doesn’t seem capable of giving us both reliably. Users also want more features. I’m not going to go into all of my plans for adding features, but if you’ve been following the discussions about Twitter over the last year you’ve heard some of them. Twitter is busy trying to get their current features all running again, though, so don’t expect new features any time soon.

So what’s needed? What’s needed is a scalable microblogging platform. It needs a sound system architecture that scales linearly by adding more hardware. It also needs a scalable business model that actually includes a revenue model that involves making more money as the number of users goes up, instead of losing more money (but making it up in volume!). It needs some innovation in technology, features, and business. In short, it needs Twitter done right.

It’s called Babelnote, and I’m building it.

So what do we do? We’re hunting the Fail Whale, that frightening creature which has been plaguing Web 2.0 services around the web, crashing sites and services. If you’re out there in Web 2.0, you’re in danger of a Fail Whale attack… Want to find a place safe from the Fail Whale? Well, we’re working on it…

In the meantime, if you want to join the hunt, you can get some cool Schwag over at CafePress.