Facebook is pretty amazing in that respect. They managed to engineer a scaleable infrastructure without making it too obvious that everything is being cached. There are times when it takes a minute or two for my profile picture to change everywhere after I change it in my profile, but aside from minute-or-two scenarios, Facebook's interface (and the real-time notification system) is just amazing in comparison.
I've heard (from a coworker that knows some people that work at FB; don't take this as gospel) they use some clever trickery to get this to work. Basically, there's a localized cache of information just for you, storing information entered by you that you might see. Since this is a limited set and is easily sharded by user, you can put everything in memcached, have the frontends hit that record and merge it with the rest of the data, and pretty much guarantee immediate consistency - as long as it's just you looking at the profile.
When other people look at your profile, they have to go through the normal eventual-consistency mechanisms, deal with the normal replication lag and message-passing delays, etc. But they have no idea that the information is out of date, because they're not the ones who inputted it. As long as you don't go ask someone through backchannels "did you see my FB update?", they'll never be the wiser.
FWIW, I've seen data-consistency problems on my FB wall before where the same entry will appear multiple times, usually due to pagination bugs.
Facebook runs largest memcached cluster in the world and largest sharded MySQL installation in the world (which they use as a key-value store since it's hard to evolve schema otherwise). So, essentially, quick updates are tremendous amount of duct tape and hand-crafted code to make it work more or less correctly.
I've always wondered why alternative datastores, like MongoDB, get so much good press when a RDBMS like MySQL or PostgreSQL can be used for storing non-relational data in a way that scales tremendously.
There's a video (I don't have the link handy) where a Facebook engineer says they actually had to make code modifications to MySQL to make it do what they want.
They're using the tool in a way that it wasn't designed to be used. Therefore it's probable that that there's an opportunity for specialized software to fill the role better.
Except for the iPhone app, with multi-hour delays required for a refresh to actually load new notifications, comments, etc. It'd be fine if push notifications could simply purge the cache since those arrive instantly even if it takes the better part of a day for new comments to actually be displayed on the post.
In my experience (and many friends' experiences), Facebook is and always has been unbelievably buggy on that front. I don't know why, and I personally think Facebook is better than G+ for general social networking (I see G+ as more of a fusion of LinkedIn and Tumblr with some FB-like features), but it's still a pain to use.
Considering the unprecedented scale they are operating on, their track record is not too shabby and because of that I would take any opinions to the contrary with a big grain of salt. Also, that article is filled with hyperbole and meaningless words, with sentences that contradict each other. In one place he's are saying ...
MySQL ... wasn’t built for webscale applications
But in another he's are saying ...
Facebook has split its MySQL database into
4,000 shards
Well, if the author had any clue whatsoever, he would know that splitting a database in multiple shards effectively goes against the relational model (Facebook is effectively using MySQL as a storage engine, not as a RDBMS) and so inherent limitations that make people say MySQL wasn't built for webscale applications ... no longer apply.
SQL databases ... consume too many resources for
overhead tasks (e.g., maintaining ACID compliance
and handling multithreading)
Again, ACID compliance and multithreading becomes a local issue, relevant for a single shard. ACID compliance does not apply to the whole cluster of shards they have and it is still freakishly useful, even if applied locally, because you want the guarantee that the user's submission has been saved somewhere in a consistent state, from which you can safely replicate.
At their scale maybe they could have chosen something better, more suitable to their needs, however the question is ... how can you make a choice when nobody else has dealt before with a social network handling submissions from hundreds of millions of users and that's expected to reach 1 billion?
You can't. On the other hand it is easy to blabber about webscale (WTF does that mean anyway?)
PS: I'm not going against the opinion of the great Michael Stonebraker, but he's not the author of the article and his views must be taken in context ;) He's not wrong to say that MySQL will be problematic to Facebook. What I'm saying here is that MySQL was as good as any other available alternatives, especially considering how Facebook uses it.
