I’ve you’ve been following tech news this week you would have heard about the Twitter #FAIL that has users up in arms. I’m not going to bother typing up a full recap as this has been done many times already this week. However, I would like to reflect on the blog post that Twitter co-founder Biz Stone (@Biz) posted a couple hours ago. Biz admits that he did not “do his homework” when posting the initial blog post about the change. He then goes into detail about the Product and Technical Design flaws that this setting caused. Here’s an excerpt from the technical flaws:

Even though only 3% of all Twitter accounts ever changed this setting away from the default, it was causing a strain and impacting other parts of the system. Every time someone wrote a reply Twitter had to check and see what each of their followers’ reply setting was and then manifest that tweet accordingly in their timeline—this was the most expensive work the database was doing and it was causing other features to degrade which lead to SMS delays, inconsistencies in following, fluctuations in direct message counts, and more.

I’ve been waiting for this. I was very interested to learn what sort of issues this was causing. Everyone knew that it was somehow related to scaling issues with the database and they were right. As you can figure out from the paragraph above, this is a MAJOR scalability issue!! Here’s a scenario:

Ashton Kutcher (@aplusk) tweets a lot. He also has nearly 1.8 Million followers at the time of this post. Let’s say Ashton makes a reply to a random dude (Jim) in Kansas. Twitter has to check EACH of his 1.8M followers and figure out whether or not they should see Ashton’s reply to Jim in their feed. Think about that. We’re talking nearly 2 million database queries (correct me if i’m wrong…) for each reply. 

Note: He posts LOTS of replies.

This can now be known as “The Great Scalability Problem of 2009″.