How Facebook Keeps Its 500 Million Users Running Smoothly: The Architecture Behind the World’s Largest Social Network

Facebook’s Scalability Challenges

Before we dive into the details, here are some numbers that illustrate the sheer scale Facebook operates at:

• Facebook serves 570,000,000,000 monthly page views (according to Google Ad Planner).
• Facebook hosts more photos than all other photo-sharing sites combined (including Flickr and others).
• Over 3 billion photos are uploaded each month.
• Facebook’s systems serve 1.2 million photos per second, not counting those served via CDN.
• Over 2.5 billion pieces of content (status updates, comments, etc.) are shared every month.
• Facebook operates over 30,000 servers (a figure from last year).

The Software Stack Powering Facebook’s Scale

To a certain extent, Facebook is still a LAMP site, but it has evolved far beyond a typical LAMP stack, incorporating many additional elements and services and modifying established practices.

For example:

• Facebook still uses PHP, but it has built a compiler for it that transforms PHP into native code running on the web server, thereby boosting performance.
• Facebook uses Linux, but has optimized it specifically for network throughput.
• Facebook uses MySQL, but primarily as a key-value persistent store; joins and server-side logic are handled at the web server layer, where it is easier to execute.

Then there are custom-built systems like Haystack, a highly scalable object store for Facebook’s photos, and Scribe, a logging system capable of running at Facebook’s enormous scale.

Now, let’s explore the software that powers the world’s largest social network.

Memcached

Memcached is one of the most famous pieces of software on the internet today. It is a distributed memory caching system used as a caching layer between web servers and MySQL servers (because database access is relatively slow). Over the years, Facebook has developed several optimizations for Memcached and its surrounding software, such as compressing the network stack.

At any given moment, tens of terabytes of data are cached across thousands of Memcached servers at Facebook. It is likely the largest Memcached cluster in the world.

HipHop for PHP

PHP, as a scripting language, runs slowly compared to native programs. HipHop converts PHP into C++ code, which is then compiled for much better performance. Since Facebook relies heavily on PHP, this enables web servers to run much more efficiently.

A small team of engineers at Facebook (initially just three people) spent 18 months developing HipHop, and it is now production-ready.

Haystack

Haystack is Facebook’s high-performance photo storage/retrieval system (strictly speaking, it is an object store, so it doesn’t necessarily have to store only photos). It has a lot of work to do; there are over 20 billion uploaded photos, and each one is saved in four different resolutions, resulting in over 80 billion photo files.

It’s not just about handling billions of photos; operational performance is critical too. As mentioned earlier, Facebook serves about 1.2 million photos per second, a figure that excludes CDN traffic. That’s an astonishing number. For more on Facebook’s photo storage, see 51CTO’s previous report: “Facebook Photo Storage Architecture Technical Deep Dive.”

BigPipe

BigPipe is a dynamic web page serving system developed by Facebook. Facebook uses it to process each web page in sections (called “pagelets”) to achieve optimal performance.

For example, the chat window is separate, the News Feed is separate, and so on. These pagelets can be processed concurrently while a page is being rendered, fetched as the page loads. Even if parts of the pipeline are shut down or fail, users can still get portions of the page.

Cassandra

Cassandra is a distributed storage system with no single point of failure. It is a significant component of the NoSQL movement and has been open-sourced (it even became an Apache project). Facebook uses it for its search functionality.

Besides Facebook, others use it too, such as Digg. However, Twitter recently moved away from Cassandra. For more on Cassandra, check out 51CTO’s special feature: “Breaking Free? A Guide to the Cassandra Database.”

Scribe

Scribe is a flexible logging system that Facebook uses extensively internally. It is designed to handle logging at Facebook’s massive scale and automatically deals with new logging categories—Facebook has hundreds of log categories.

Hadoop and Hive

Hadoop is an open-source map-reduce implementation that enables computation on large datasets. Facebook uses this for data analysis (and as we all know, Facebook has massive amounts of data). Hive originated at Facebook and makes SQL-like queries on Hadoop possible, making it easier for non-programmers to use.

Hadoop and Hive are open-source (Apache projects) and have a large following, including users like Yahoo and Twitter.

Thrift

Facebook uses several different languages for different services. PHP is ultimately used for the front-end, Erlang is used for chat, Java and C++ are used in various places, and there may be other languages too. Thrift is an internally developed cross-language framework that binds languages together so they can interoperate and communicate with each other. This makes it easier for Facebook to continue its cross-language development approach.

Facebook has open-sourced Thrift, and more language support has been added to it since.

Varnish

Varnish is an HTTP accelerator that can act as a load balancer and cache content, which it can then serve at lightning speed.

Facebook uses Varnish to handle photos and profile pictures, serving billions of requests daily. Like many other things here, Varnish is open-source.

Other Things That Keep Facebook Running Smoothly

The software we’ve mentioned forms the core of Facebook’s systems and helps it run at scale. But managing such a massive system is a complex task, so we’ll list some other things that keep Facebook running smoothly.

Gradual Rollouts and Dark Launches

Facebook has a system they call Gatekeeper, which allows them to run different code paths for different sets of users. This enables Facebook to gradually roll out new features, run A/B tests, and activate certain features only for Facebook employees.

Gatekeeper also enables Facebook to do “dark launches,” where a feature is activated behind the scenes before users get access to it (it’s called “dark” because users are unaware of it). This acts as a real-world stress test and helps uncover bottlenecks and other issues before the official launch. Dark launches typically happen about two weeks before the official launch.

Profiling the Live System

Facebook carefully monitors its systems, and interestingly, it also profiles the performance of every single PHP function in the production environment. It monitors the configuration and runtime of various PHP environments using an open-source tool called XHProf.

Gracefully Degrading Features for Performance

If Facebook encounters performance issues, one approach is to gradually disable less critical features to boost the performance of Facebook’s core functions.

Things We Haven’t Mentioned

We haven’t touched on hardware-related aspects, which are also a crucial part of improving scalability. For example, like other large sites, Facebook leverages CDNs to handle static content. Facebook also has a huge data center that helps it scale more services.

Facebook’s Affinity for Open Source

Not only does Facebook use (and help improve) open-source software like Linux, Memcached, MySQL, and Hadoop, but it has also contributed many of its internally developed tools back to the community.

Facebook has also open-sourced Tornado, a high-performance web server framework developed by the FriendFeed team. For a list of open-source software, you can check out Facebook’s Open Source page.