thoughts on computing

In the past week+ the whole business about Twitter scalability & reliability came to a head.

Yet, despite infrastructure that is visibly “hitting the wall”, now it appears that the company is gaining interest in a funding round at a decent valuation (maybe even signed one, but more on that later).

How is this possible?

I think the answer to this is that

1. Building a scalable micro-blogging site is not that hard
2. There’s hoped-for value in all that traffic

Building a Scalable Micro-Blogging Site
If you’re starting from a clean sheet, the answer is that it’s not very hard to make a scalable micro-blogging platform. Unlike some recent comments, the solution is not rocket science.

The key is simple:

1. take the database out of the flow of messages. Of course, you still write to the db as it’s able to keep up (for archival purposes), but that’s about it.
2. create objects that stand-in for each subscriber, whether follower, followee, or both
3. have them interact over a simple pub / sub model, reliable in-memory space, or both
4. wrap all this in our application fabric to handle organization, reliability, and operations as you scale
5. deploy on commodity (in-house or cloud)

Of course this becomes pretty hard if you’re committed to Ruby on Rails, which is very tied to a database. Terrific for some stuff, not so good for high-volume messaging apps.

Are there other approaches? Sure, but nothing this conceptually simple, easy to implement, cheap to deploy, and brain-dead simple to operate.

What’s a Twitter to do?
Arrington posted some speculated usage numbers today that are useful in validating this approach. Remember that all of this is greatly aided by two simple facts:

1. twitter messages are limited to 140 characters
2. delivery expectations for SMS etc. are modest, at best

So easy to deliver, forgiving with regards to when they get delivered … this is really fairly straightforward.

Of course, I’m sure the first order of business for the new technical folks is to stabilize the existing platform … then get to work on something that can be counted on for 10x 100x 1000x this amount of traffic.

Final Thoughts
The very fact that Twitter is able to raise a round (at a decent valuation) despite the obvious problems is a vote from the venture community that the business will be worth building (traffic is decent, it is hoped to grow significantly, will be of some TBD value to someone), and that building a real infrastructure is eminently doable.

Give us a call!

Yesterday we talked about whether Twitter really ever need to be reliable or not … some said yes, others contend that it’s not necessary.

It’s been bugging me for awhile that something this popular … and Twitter is so … just keels over as often as it does.

Anyhow, the whole argument turned into a bona-fide debacle this morning when GroupTweet (a relatively new feature that seems to have been confusing) was at the heart of disclosing private messages (DMs in tweet-speak) to tons of folks.

Oops.

So now it looks like Blaine Cook is out as chief architect, and Michael Arrington is calling it the end of amateur hour. That’s probably a bit harsh, because my (limited) interactions with Cook have been pretty decent.

Btw the comment thread on that last post is going crazy. My favorite so far is a short video comment from a Loren Feldman (warning … his language is a bit over the top, but you do know where he stands!) Btw, check here if the first link to the video doesn’t work.

Moving On
Having said that, we just have to build apps that act like real, grown up (and you can call that boring if you want) apps … taking care of the data entrusted to them, working as expected, and working when we need them to work.

So I’m thinking that the answer to yesterday’s question is … YES. Twitter does need to figure out how to be reliable … and secure, scalable, and all the rest.

This is exactly the point that I’ve been making for awhile … why build to POC quality when it’s now possible to ensure reliability, scalability, and so forth from the beginning?

People are Still People
I don’t care if this is Web 2.0, Enterprise 2.0, or Web 10,000,000,000.0 … consumer or enterprise … people are still people. They still care about their privacy, the reliability of stuff that they come to rely on, basic stuff like that. No free pass.

Even consumer oriented web 2.0 apps need to ensure this, from the beginning.

Pretending that innovation in communication, biz, or technology somehow exempts us from the basics of social interaction is just … well, it’s just wrong.

This lesson is for our whole industry. Those who learn it will prosper, those who don’t …

A few outages ago I wondered aloud whether Twitter was taking the whole business of failure somewhat casually (triggered by some comments Blaine Cook made at SXSW).

Blaine replied with some great points, including

For the record, saying that the press surrounding the downtimes was a plus was a joke. Downtime is never good, and you should do everything you can to avoid it. However, it’s a misrepresentation to say that you can build something successful without any downtime.

…

Our foremost concern has been and will continue to be ensuring a stable platform; we’ve been working hard on numerous fronts, and that work is paying off. Bad press is horrible, and I’ll be the first to take pleasure in never again seeing a “can Twitter scale?” story.

I believe him, and have quite a bit of empathy for the position he’s in. (I have a friend who always used to talk about “high class problems” and “low class problems” … Blaine and the other Tweetsters have a high class problem, but that’s a post for another day)

There was one point that he made that I fundamentally disagree with, however:

Scaling is a commitment, and one you should only make once you’re sure about an idea.

Yes it is most definitely a commitment, but it is my contention is that we’re fast entering the time when it can be built-in from the beginning with little to no additional effort.

The Latest
This past weekend there’s was a bunch more instability. Looks like they were putting in some more caching to take the heat off of the data tier, and things went wacky.

Now Robert Scoble is making the case that Twitter is leaving the door wide-open for Friendfeed … (on Twitter of course, though my friend read it on Friendfeed!). Check out this short burst:

Michael Arrington thinks that the mass of the Twitter community makes this concern moot … basically, he contends that Twitter no longer needs to be reliable.

The Days of the Free Pass on Reliability are Over
Basically, I think that 1) the days of web2 services getting a free pass on reliability are rapidly passing, and are probably already over, and 2) it’s a shame to see stuff go whump when it’s sooo unnecessary.

As for the days of the free pass being over, check out Dennis Howlett’s (zdnet) comments on the most recent outage … he’s generally making the case that Twitter itself is really a POC for some better service yet to come, something more suitable to much larger markets.

Could that V1 service be Friendfeed? Maybe. Of course, it’s too early to write-off Twitter entirely … they’ve also hired their own scaling calvary (including the every-helpful Google expat!), so maybe they’ll catch a second wind before the whole sector passes them by.

Build to Scale … From the Beginning
Back to Blaine’s comments. I can completely understand the notion of building a proof of concept … besides, in the web 2.0 world it’s long been accepted practice to throw something out there, and only build to scale when you figure out whether anyone cares.

That makes a lot of sense when building apps to scale is so freakin’ hard. BUT … easing that pain is precisely the point of stuff like our app fabric.

That is why it is my core contention that the ability to scale and be reliable, even for the most trivial services, is going to become the price of entry very soon (if it has not already become so).

Alistair Croll has an interesting post where he argues that, for all intents and purposes, source code is irrelevant – the new valuable commodity is app tone.

… In the software-as-a-service world … source code becomes irrelevant. If someone offered us the schematics to a telephone, we wouldn’t care. We don’t want to know how to make a phone. We want a dial tone. When it comes to IT, we want app tone.

As another way of saying people want apps to work, this could make sense. But irrelevant? That’s just silly.

… If 37 Signals gave me the Basecamp source code for free, I’d still use their service. If Freshbooks burned me a copy of their app, I’d still subscribe to them. Even if Salesforce.com handed me their software, I’d use their hosted portal.

Ok that makes sense to most folks using SaaS offerings … after all, who wants to go through all the trouble to install and run something when a perfectly acceptable alternative is already available?

Of course, if a competing service came out (made much easier to do with that “irrelevant source code”!), perhaps with some improvements in quality of service, or more generous free capabilities or some other such advantage, wouldn’t folks simply switch? Of course they would.

Anyhow, Croll continues on this theme

In the license world, it’s all about the ability to make copies of the software. By contrast, in the world of app tone, it’s about the ability to run instances of the code. It’s about operating an application reliably … and the ecosystem the SaaS provider can build around it through APIs, partners and extensions

Sure that’s all true, but how do you think all that reliability gets there to begin with? Is it purely a consequence of skilled operations? Of course not … the application source code itself can do a lot to improve it’s reliability.

On a Roll …
So in the interest of making an interesting point Alistair gets carried away, leading him to miss the mark entirely here:

Even the open-source movement is feeling the change: Recent modifications to the third revision of the GNU Public License recognize that it’s the service, not the source code, that has value — and that any user of the service has the rights to its source code.

Not exactly. What these changes recognize is that most SaaS offerings are not posting their key source code much at all … even when they incorporate open source libraries that would trigger that posting for software that is delivered conventionally. They don’t have to by the terms of most open-source licenses, so why should they?

Of course, methinks that this might be because there’s significant value in said source code!

The Service DOES Have Value
Of course, the service itself has beaucoup value … for all of the reasons that Croll cites in his post. It’s just that the question of which has value, the service or the source, in not a simplistic case of either / or, but more of a both-and. That is, both the service and the source code itself have significant value, and will continue to do so for some time.

After all, SaaS is not an alternative to meaningful source … it’s another way to deliver that meaningful source.

That both the music and software businesses are going through profound changes is, maybe … the understatement of the year. Pretty much of a Capt. Obvious statement.

Like any changes of that magnitude there is a fair amount of uncertainty, along with normally sane people working up histrionic firestorms.

For example, I think Arrington has been making some compelling points about the flattening of the music distribution model … his reasoning just goes haywire when he says that

music itself is nothing more than marketing material for the artist.

That’s absurd at the face of it … but let’s move on to some interesting parallels.

Parallels Between Music and Software
In commenting on my last post, Alex noted the parallels between open source and the music business. He makes some great points, he got me to thinking. Like Alex, I find Fred Wilson’s reasoning on this (particularly with regards to music and what the internet rewards) compelling … though not absolute.

Will free music eventually win out? Maybe, maybe not. Lots of artists (both known and indie) are experimenting with all sorts of combinations of free, pay per play, deprecated free combined with premium upgrades, and all sorts of discovery / distribution models are being built, discarded, built again, morphed, twisted, and rising like so many phoenixes from the ashes … some only to crumble and fade away yet again.

Yet with these obvious changes (from which there’s definitely / thankfully no going back), this is probably the best time in history to be a new artist, and furthermore folks are actually making plenty of money selling music. In fact, today iTunes became the top music retailer in the US … all that in less than five years. So there’s clearly room for more than one model in the music biz, probably even for an extended period of time.

Open Source and Open Distribution
A month or so back we announced our open distribution initiative, which is proving to be a great way to get our technology into the hands of developers and architects. Easy to use and free … a great combination.

Yet clearly not open source … a decision we made for some basic intellectual property and building-the-business reasons.

Think back to what’s happening in the music biz right now … how closely is music really following the open source model? Are people really creating desirable music via open collaboration?

Not that I’ve seen.

What’s really going on is that the reality of frictionless, no cost distribution is being used as the underpinnings of a bunch of different distribution models, with the precise natures of the winning distribution models TBD.

Creation vs. Distribution
In any case, while the players in the music biz fight their transition out, I think it’s clear that the creation of software component of open source need not necessarily be combined with the distribution component of open source.

It’s certainly fine to combine both into one model … it’s just not the only choice. We’ve chosen otherwise for our community edition, and I think you’ll see more and more of that going on, even from traditional open source suppliers.

In essence, that is what we’re doing with open distribution – going down the path of creating viable, low friction, high value ubiquitous communities (free is good!), while having more to say about the process of creating the software itself.

Different circumstances, different needs, different choices … sometimes even within the same company, perhaps even within the same product family.

I LOVE having choices!