How .NET Technology and 25 Servers Power the World’s 54th Largest Website

【Editor’s Note】StackOverflow is a Q&A website for IT professionals where users can submit and answer questions. Today, StackOverflow boasts 4 million users, 40 million answers, 560 million monthly pageviews, and ranks 54th globally. What’s remarkable is that their entire site runs on just 25 servers, all maintaining very low resource utilization. This is a triumph of high availability, load balancing, caching, databases, search, and highly efficient code. Recently, High Scalability founder Todd Hoff summarized the reasons behind StackOverflow’s success, drawing from Marco Cecconi’s talk video “The architecture of StackOverflow” and Nick Craver’s blog post “What it takes to run Stack Overflow”.

Translation Below

Both expectedly and unexpectedly, Stack Overflow still heavily uses Microsoft products. Their reasoning is simple: since Microsoft’s infrastructure meets their needs and is affordable enough, there’s no compelling reason to make fundamental changes. Where needed, they also use Linux. Ultimately, it all comes down to performance.

Another noteworthy point is that Stack Overflow still employs a scale-up strategy and does not use the cloud. They use servers with 384GB of RAM and 2TB SSDs to power their SQL Servers. The cost of using AWS for equivalent resources would be staggering. Another reason for avoiding the cloud is Stack Overflow’s belief that it can degrade performance to some degree and also make it harder to optimize and troubleshoot system issues. Moreover, their architecture doesn’t require horizontal scaling. Traffic spikes are the killer use case for horizontal scaling, but they have extensive system tuning experience to handle them. The company still adheres to Jeff Atwood’s famous saying —— Hardware is always cheaper than programmers.

Marco Cecconi once mentioned that when discussing systems, one thing must be clarified first —— the type of problem you need to solve. Let’s start with the simple aspect: what exactly does StackExchange do —— first there are topics, then communities are built around those topics, and finally, it becomes this admirable Q&A website.

The next aspect is scale. StackExchange is growing rapidly and needs to handle massive data transfers. So how is all this accomplished, especially with just 25 servers? Let’s dig into the details:

Status

• StackExchange has 110 sites, growing at a rate of 3 to 4 per month.
• 4 million users
• 8 million questions
• 40 million answers
• Ranked 54th globally
• 100% annual growth
• 560 million monthly pageviews
• Peak traffic on most weekdays ranges from 2600 to 3000 requests per second. As a programming-related site, weekday traffic is generally higher than on weekends.
• 25 servers
• 2TB of SQL data stored on SSDs
• Each web server is configured with two 320GB SSDs in RAID 1
• Each ElasticSearch host is equipped with a 300GB mechanical hard drive and also uses SSDs
• Stack Overflow’s read-to-write ratio is 40:60
• Average CPU utilization on DB Servers is 10%
• 11 web servers running IIS
• 2 load balancers, 1 active, using HAProxy
• 4 active database nodes running MS SQL
• 3 application servers implementing a tag engine, with all searches going through tags
• 3 servers handling search via ElasticSearch
• 2 servers using Redis to support distributed caching and messaging
• 2 Networks (Nexus 5596 + Fabric Extenders)
• 2 Cisco 5525-X ASAs 
• 2 Cisco 3945 Routers
• 2 read-only SQL Servers primarily serving the Stack Exchange API
• VMs used for deployment, domain controllers, monitoring, operations databases, and similar purposes

Platform

• ElasticSearch
• Redis
• HAProxy
• MS SQL
• Opserver
• TeamCity
• Jil——Fast .NET JSON Serializer, built on top of Sigil
• Dapper——a micro-ORM

UI

• The UI features an inbox for receiving notifications about new badges earned, messages from users, and major events, implemented using WebSockets and backed by Redis.
• The search box is implemented via ElasticSearch, using a REST interface.
• Because users ask questions at a very high frequency, displaying the newest questions is challenging—new questions appear every second. This required developing an algorithm that follows user behavior patterns to only show questions they are interested in. It uses complex tag-based queries, which is why a standalone Tag Engine was developed.
• Server-side templates are used to generate pages.

Servers

• The 25 servers are not fully loaded; CPU usage is quite low. Stack Overflow (SO) alone requires only 5 servers for computation.
• Database server resource utilization hovers around 10%, except when performing backups.
• Why is it so low? Because the database servers have a full 384GB of RAM, while the web server CPU utilization is also only 10%-15%.
• Vertical scaling has not yet hit a bottleneck. Typically, handling this level of traffic with horizontal scaling would require roughly 100 to 300 servers.
• A simple system. Built on .Net, it uses only 9 projects, whereas other systems might need 100. Such a minimal number of projects is used to pursue extreme compilation speed, which needs to be planned from the system’s inception. The compilation time per server is about 10 seconds.
• 110,000 lines of code, which is very small relative to the traffic.
• This minimalist approach is based on several reasons. First, extensive testing isn’t necessary because Meta.stackoverflow is inherently a community for discussing issues and bugs. Second, Meta.stackoverflow also serves as a software testing site; if users discover problems, they often report them and propose solutions.
• The New York data center uses Windows 2012 and has been upgrading to 2012 R2 (Oregon has already completed the upgrade). Linux systems run Centos 6.4.

SSD

• Default drives are Intel 330 (Web tier, etc.)
• Intel 520 for middle-tier writes, such as Elastic Search
• Data tier uses Intel 710 and S3700
• The system uses both RAID 1 and RAID 10 (any set of 4+ disks uses RAID 10). Not afraid of failures—even with over a thousand 2.5-inch SSDs in production, they have yet to see a single drive failure. More than one spare is kept per model, so multiple simultaneous disk failures are not a concern.
• ElasticSearch performs exceptionally well on SSDs because SO writes/re-indexes are very frequent operations.
• SSDs have changed how search is used. Due to locking issues, Lucene.net could not handle SO’s concurrency load, so they moved to ElasticSearch. In an all-SSD environment, there is no need to build locks around Binary Reader.

High Availability

• Off-site backup——Primary data center is in New York, backup data center is in Oregon.
• Redis has two slaves, SQL has two backups, Tag Engine has three nodes, Elasticsearch has three nodes. Redundancy everywhere, and it exists simultaneously in both data centers.
• Nginx is used for SSL; HAProxy is used for switching after SSL termination.
• Not everything is master-slave; some temporary data is only placed in the cache.
• All HTTP traffic sent is only 77% of the total volume; there are also backups in the Oregon data center and some other VPN traffic. This traffic is mainly generated by SQL and Redis backups.

Databases

• MS SQL Server
• Stack Exchange has a database per site, so Stack Overflow has one, Server Fault has one, and so on.
• In the primary New York data center, each cluster typically uses a 1 master and 1 read-only replica configuration, with an additional replica in the Oregon data center. If running on the Oregon cluster, both replicas in the New York data center are read-only and synchronized.
• Databases for other content. There is also a “network-wide” database for storing login credentials and aggregated data (mostly stackexchange.com user files or APIs).
• Careers Stack Overflow, stackexchange.com, and Area 51 all have their own independent database schemas.
• Schema changes need to be deployed to all site databases simultaneously and must be backward compatible. For example, if you need to rename a column, it is very troublesome and requires multiple steps: add a new column, add code that works on both columns, write data to the new column, change code to make the new column effective, and remove the old column.
• Sharding is not needed; everything is solved through indexing, and the data volume isn’t that large. If filtered indexes are needed, why not do it more efficiently? A common pattern is only indexing on DeletionDate = Null, otherwise specifying a type for the enumeration. Each type of vote has its own table, for instance, one table for post votes, one table for comment votes. Most pages are rendered in real time and only cached for anonymous users, so there are no cache updates, only heavy queries.
• Scores are denormalized, so they require frequent queries. They contain only IDs and dates. The post votes table currently has about 56,454,478 rows; using indexes, most queries complete in milliseconds.
• The Tag Engine is completely independent, meaning core functionality does not depend on any external applications. It is a massive in-memory array structure, optimized specifically for the SO use case and pre-computes heavy load combinations. The Tag Engine is a simple Windows service running redundantly across multiple hosts. CPU usage basically stays at 2-5%; 3 hosts are dedicated purely for redundancy and handle no load. If all hosts fail simultaneously, the web servers will load the Tag Engine into memory and keep it running.
• Regarding Dapper’s compiler-free checked queries versus traditional ORMs. Using a compiler has many benefits, but there is still a fundamental disconnect at runtime. More importantly, due to generating nasty SQL, you often need to hunt down the raw code, and the lack of control over Query Hints and parameterization makes query optimization even more complex.

Coding

• Process
• Most programmers work remotely, choosing their own coding location

• Compilation is very fast
• Then run a small number of tests
• Once compiled successfully, the code is moved to a dev/delivery prep server
• Hide new features behind feature flags
• Tested on the same hardware as other sites
• Then moved to Meta.stackoverflow for testing, used by thousands of programmers daily — a great testing environment
• If it passes, it goes live and is tested on the wider community

• Heavy use of static classes and methods for simplicity and better performance
• The coding process is very simple because complex parts are packaged into libraries, which are open-sourced and maintained. The number of .Net projects is very low because shared community code is used.
• Developers use 2 to 3 monitors simultaneously; multiple screens significantly boost productivity.

Caching

• Cache everything
• 5 levels of caching
• Level 1 is network-level caching, cached in the browser, CDN, and proxy servers.
• Level 2 is handled by the .Net framework’s HttpRuntime.Cache, in the memory of each server.
• Level 3 is Redis, a distributed in-memory key-value store, sharing cache items across multiple servers supporting the same site.
• Level 4 is SQL Server Cache, the entire database, with all data placed in memory.
• Level 5 is SSD. Usually only takes effect after SQL Server is warmed up.
• For example, every help page is cached, and the code to access a page is very simple:

• Uses static methods and classes. It’s terrible from an OOP perspective, but it’s very fast and conducive to clean coding.
• Caching is powered by Redis and Dapper, a micro ORM

• To solve garbage collection issues, only one copy of a class is used in templates, created and stored in the cache. Monitor everything, including GC operations. Statistics show that when the abstraction layer increases GC pressure to a certain degree, performance significantly degrades.
• CDN Hits. Since query strings are hashed based on file content, they are only fetched again when there is a new build. 30 to 50 million hits per day, with bandwidth around 300GB to 600GB.
• The CDN is not used to handle CPU or I/O load, but to help users get answers faster

Deployment

• Deploy 5 times a day, avoiding building overly large applications. Mainly because

• Performance can be monitored directly
• Keep builds as minimal as possible — just enough to work is the key

• After the product is built, it’s copied across web tiers via powerful scripts. The steps for each server are:

• Notify HAProxy via POST to take a server offline
• Delay IIS to finish existing requests (about 5 seconds)
• Stop the website (terminate all downstream via the same PSSession)
• Robocopy files
• Start the website
• Re-enable HAProxy via another POST

• Nearly all deployments are done via Puppet or DSC. Upgrades are typically just a massive RAID array adjustment and a PXE boot install — it’s very fast.

Collaboration

• Teams

• SRE (System Reliability Engineering): 5 people
• Core Dev (Q&A site): 6-7 people
• Core Dev Mobile: 6 people
• Careers team dedicated to SO Careers product development: 7 people

• DevOps and developers work very closely together
• Variation between teams is significant
• Most employees work remotely
• Offices are mainly used for sales, except Denver and London
• Everything is equal, with a slight bias toward New York-based workers because face-to-face helps with communication, though working online doesn’t impact much either
• Over being able to work in the same office, they prefer passionate and talented engineers who love the product — they can weigh the pros and cons well
• Many choose remote work because of family. New York is nice, but life isn’t easy there
• The office is in Manhattan, a birthplace of talent. The data center can’t be too remote since upgrades happen frequently
• Build a strong team — prefer geeks. Early Microsoft gathered tons of geeks, and that’s how they conquered the world
• The Stack Overflow community is also a hiring ground; they look for people who love coding, are eager to help, and enjoy communication.

Budgeting

• Budget is the foundation of the project. Money is only spent on building infrastructure for new projects — the web servers with such low utilization were purchased when the data center was set up 3 years ago.

Testing

• Rapid iteration and deprecation
• Many tests are performed by the release team. Developers share an identical SQL server and run on the same web tier, so performance testing is not a bottleneck.
• Very few tests. Stack Overflow does not do much unit testing because they use a lot of static code and have a very active community.
• Infrastructure changes. Since everything is duplicated, every old configuration has a backup and uses a fast failover mechanism. For example, keepalived enables quick rollbacks in the load balancer.
• They prefer relying on redundant systems over scheduled maintenance. SQL backups are tested on a dedicated server just for restorability. They plan to do a full data center failover every two months, or use a completely read-only secondary data center.
• Unit tests, integration tests, and UI tests are done for every new feature release, meaning product features with predictable inputs are tested and then pushed to the incubation site, meta.stackexchange (formerly meta.stackoverflow).

Monitoring/Logging

• Currently considering http://logstash.net/ for log management. They currently use a dedicated service to transmit syslog UDP into an SQL database. Timers are added to headers on web pages so they can be captured by HAProxy and merged into the syslog stream.
• Opserver and Realog are used to display measurement results. Realog is a log display system built by Kyle Brandt and Matt Jibson using Go.
• Logs are handled via syslog through the HAProxy load balancer rather than IIS, because its functionality is richer than IIS.

On the Cloud

• The age-old adage: hardware is always cheaper than developers and efficient code. Based on the bucket effect, speed is always constrained by a weakest link, and existing cloud services essentially all have capacity and performance limits.
• If they had built Stack Overflow on the cloud from the start, it might have reached the same level. However, there is no doubt that achieving the same performance on the cloud would cost far more than self-hosting a data center.

Performance First

• Stack Overflow is a hardcore performance enthusiast. The homepage load time is always kept under 50 milliseconds, and the current response time is 28 milliseconds.
• Programmers are passionate about reducing page load times and improving user experience.
• Every single web request is timed and recorded. This instrumentation helps identify exactly where changes need to be made to improve performance.
• The primary reason for such low resource utilization is efficient code. Web server CPU average utilization is between 5% and 15%, memory usage is 15.5 GB, and network throughput is 20 Mb/s to 40 Mb/s. SQL server CPU usage is between 5% and 10%, memory usage is 365 GB, and network throughput is 100 Mb/s to 200 Mb/s. This yields three benefits: plenty of headroom for upgrades; keeping the service available when serious errors occur; and the ability to roll back quickly when needed.

Lessons Learned

1. Why use Redis alongside Microsoft products? Use whatever works best; avoid pointless system wars. For example, C# runs best on Windows machines, so we use IIS; Redis performs best on *nix machines, so we use *nix.

2. Overkill is the strategy. Normal utilization rates don’t tell the whole story; when specific things happen, like backups or rebuilds, resource usage can be maxed out completely.

3. Robust SSDs. All databases are built on SSDs, which allows for zero latency.

4. Understand your read/write workload.

5. Efficient code means fewer servers. Hardware is only added for new projects with special requirements, typically more memory, but beyond that, efficient code equals zero hardware additions. So discussions often boil down to just two things: adding new SSDs for storage; and adding hardware for new projects.

6. Don’t be afraid of customization. SO uses complex queries on Tags, so a dedicated Tag Engine was built for that purpose.

7. Only do what must be done. The reason testing isn’t necessary is because there’s an active community backing it. For instance, developers don’t worry about the “Square Wheel” effect—if a developer can create a lighter-weight component, just replace it.

8. Value hardware knowledge, like IL. Some code uses IL instead of C#. Focus on SQL query plans. Explore what a web server’s memory dump actually does. Investigate, for example, why a split generates 2GB of garbage.

9. Never be bureaucratic. There are always new tools you need, like an editor, a new version of Visual Studio—remove every obstacle in the improvement process.

10. Garbage Collection-Driven Programming. SO has put tremendous effort into reducing garbage collection costs, skipping practices like TDD, avoiding abstraction layers, and using static methods. It’s extreme, but it truly produces highly efficient code.

11. The value of efficient code far exceeds your imagination. It makes hardware run faster, reduces resource usage, and always remember—make code easier for programmers to understand.

Original link: StackOverflow Update: 560M Pageviews A Month, 25 Servers, And It's All About Performance (Compiled by Zhong Hao / Reviewed by Wei Wei)