7

How do sites like Facebook and Twitter optimize their sites for massive traffic. Aside from spending big bucks on getting the best servers, what can be optimized in your code to accommodate massive traffic?

I've read about caching your pages to static HTML, but that's impractical for social networking sites where the pages are constantly updated.

Improve this question

3 Answers 3

Reset to default
10

Massive sites like Google, Facebook and Twitter don't necessarily get the 'best' servers in that they don't run a small number of high-powered servers, they run a massive number of smaller and cheaper servers. They expect hardware to die and be replaced and the code allows for that.

Some things that are typical in massive scale sites:

  • They don't use SQL databases like mySQL. Instead they key-value stores like HBase or Cassandra. mySQL and other SQL DBs are too slow when the numbers of requests are huge.
  • They cache as much as possible. HTML caching as you say. User's data is stored in memory using things like memcached.
  • Some sites, like Reddit, pre-cache pages before a user has even requested it.
  • Pre-calculate as much as possible, sites tend to work out stuff like your number of friends (or whatever) and cache that too - a little as possible is done dynamically.

http://highscalability.com/ is a great site to learn more about this.

Improve this answer
3
  • I voted you up since all of your statements are mostly true for the largest applications, like Google, Amazon, and other similar-scale sites. However, it should be noted that there are exceptions to most rules. For example, YouTube still uses MySQL, as do Digg, Flickr and supposedly AdWords (purportedly, they started with MySQL, then dropped it for a "commercial" solution, and then went back when that didn't work).
    – Lèse majesté
    Commented Dec 12, 2010 at 10:21
  • Also, even if these types of top-tier sites were all using NoSQL DBs, these are the extreme edge cases. There are different definitions/levels of scalability. And for most large, enterprise-scale applications out there, it's debatable whether NoSQL is better than a traditional relational database. Your other points are spot on for all applications requiring scalability though.
    – Lèse majesté
    Commented Dec 12, 2010 at 10:24
  • 1
    @Lese - totally agree but the OP did mention Facebook and Twitter
    – Steve Claridge
    Commented Dec 12, 2010 at 10:44
3

Offloading database traffic via sharding is probably one of the best ways to scale a high-traffic site.

Reducing page load time by compressing resource files using gzip and combining multiple resource files into one to reduce the amount of requests the browser needs to make is also a worthy optimization.

Improve this answer
1
3

Thought I'd mention HipHop for PHP, which turns PHP into C++ and was developed by Facebook. Facebook did some work on APC as well I believe.

Improve this answer
2
  • Just in case it HipHop doesn't turn PHP into C++, it connects the two languages in a way that Facebook logic behind the site is done in C++ (which is much faster and CPU usage friendly) but the "inputs to that logic" is done by PHP.
    – Trufa
    Commented Dec 12, 2010 at 19:55
  • 1
    Really? To quote the HipHop site: HipHop transforms your PHP source code into highly optimized C++ and then compiles it with g++ to build binary files. I understand it can only handle a subset of PHP as the more dynamic bits (think eval) don't easily translate...
    – Andy
    Commented Dec 13, 2010 at 9:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.