Lessons from 12306.cn: Large-Scale Website Architecture and Performance Optimization

<<>>
PS：Regarding the 12306.cn website, there was a lot of criticism earlier. But the pressure and architecture of this site are far more complex than what non-professionals might imagine. The following is a series of analyses on the architecture and performance of the 12306.cn ticketing website by senior architect Chen Hao. I find it highly insightful and am reposting it here:

The 12306.cn website crashed and was met with nationwide criticism. I have been thinking about this matter for the past couple of days and would like to use it as a case to discuss website performance issues broadly. As this is hastily written and based entirely on my own limited experience and knowledge, I welcome any discussion and corrections if issues are found. (This is another long article, focusing solely on performance issues, not on UI, user experience, or functional aspects like whether the payment and order placement should be separated).

Business Requirements

No technology can be divorced from business requirements. Therefore, to explain performance issues, I think we need to first discuss the business requirements.

• First, some people might compare this to QQ or online games. However, I believe these are different. Online games and QQ primarily access user-specific data when online or logging in, whereas a ticketing system accesses centralized ticket inventory data — these are not the same. Do not assume that just because online games or QQ can handle their loads, this is equivalent. The backend load for online games and QQ is relatively simpler compared to e-commerce systems.

• Second, some say that booking train tickets during the Spring Festival travel rush resembles a flash sale event on a website. It does look very similar, but if you think beyond the surface, you will find differences. With train tickets, there are many query operations — checking times, checking seat types, checking berth types. If one train doesn’t work, another is queried. This is accompanied by massive query operations, and when placing an order, database operations are required. Whereas for a flash sale, you just go straight for the kill. Additionally, flash sales can easily be designed to only accept requests from the first N users (without any backend data operations, merely logging the user’s order action). This kind of business only requires keeping the flash sale quantity in memory cache, and the data can even be distributed — for 100 items across 10 servers, each server holds 10. No real-time database operations are needed. Once the order quantity is reached, the flash sale stops, and orders can be written to the database in batch. Moreover, flash sale items are limited. Train ticket booking is not as simple as a flash sale. During the Spring Festival travel season, almost all tickets are in high demand, and nearly the entire nation is trying to buy them. (Taobao’s Double Eleven event only has about 3 million users, whereas train ticket booking can reach tens of millions or even hundreds of millions of users instantly).

• Third, some compare this system to the Olympic ticketing system. I also think that is different. Although the Olympic ticketing system also crashed immediately upon launch that year, the Olympics used a lottery method, meaning there was no first-come, first-served rush. Furthermore, it was a post-event draw; beforehand, only information needed to be collected. There was no need to guarantee data consistency beforehand, no locking, making horizontal scaling very easy.

• Fourth, the ticketing system should be quite similar to an e-commerce order system, both requiring inventory operations: 1) Reserve inventory, 2) Pay (optional), 3) Deduct inventory. This requires consistency checks, meaning data locking is necessary during concurrency. Most B2C e-commerce sites handle this asynchronously. That is, your placed order is not processed immediately but with a delay. Only when successfully processed will the system send you a confirmation email stating the order is successful. I believe many of you have received emails saying an order was unsuccessful. This means data consistency under concurrency is a bottleneck.

• Fifth, the railway ticketing business is quite extreme. It uses sudden ticket releases, and some tickets are far from sufficient for everyone, which has led to the uniquely Chinese practice of ticket snatching. So, when tickets are released, millions or even tens of millions of people rush in to query and place orders. Within tens of minutes, a website receiving tens of millions of visits is a terrifying scenario. It is said that 12306’s peak traffic reaches 1 billion PV, concentrated between 8 a.m. and 10 a.m., with peak PV reaching tens of millions per second.

A few more points:

• Inventory is the nightmare of B2C; inventory management is extremely complex. If you doubt this, ask any traditional or e-commerce retail company how difficult managing inventory is. Otherwise, there wouldn’t be so many people questioning Vancle’s inventory issues. (You can also read the biography of Steve Jobs to understand why Tim Cook succeeded as Apple‘s CEO — the main reason was that he solved Apple’s inventory cycle problem).

• >>
  40（12306），570。，。

• Taobao is much simpler than B2C websites because it has no warehouses, so there are no operations like B2C sites where N warehouses need to update and query inventory for the same product. When placing an order, a B2C website has to find a warehouse that is both close to the user and has stock, which requires a lot of computation. Imagine you buy a book in Beijing, but the Beijing warehouse is out of stock. It then has to check if the Shenyang or Xi’an warehouse has it, and if not, check the Jiangsu warehouse, and so on. Taobao doesn’t have these complications. Each merchant has their own inventory, which is just a number, and since inventory is distributed per merchant, it actually helps with performance scalability.

• Data consistency is the real performance bottleneck. Some say nginx can handle 100,000 static requests per second, and I don’t doubt that. But those are just static requests, a theoretical value. As long as bandwidth, I/O are strong enough, server computing power is sufficient, and the number of concurrent connections can withstand the establishment of 100,000 TCP connections, there is no problem. But in the face of data consistency, that 100,000 figure becomes a completely unattainable theoretical value.

I’ve said all this to explain from a business perspective that we need to truly understand the extreme nature of a system like the Spring Festival railway ticketing.

Front-End Performance Optimization Techniques

To solve performance problems, there are many common methods. I’ll list some below, and I believe using the following techniques would give the 12306 website a qualitative leap in performance.

I. Front-End Load Balancing

Using DNS load balancers (typically redirecting based on routing load on the router) can evenly distribute user access across multiple Web servers. This reduces the request load on individual Web servers. Since HTTP requests are short jobs, this can be accomplished with a very simple load balancer. Ideally, a CDN network should be used so users connect to the nearest server (CDNs usually come with distributed storage). (For a more detailed explanation of load balancing, see “Back-End Load Balancing”)

II. Reduce Front-End Connections

I took a look at 12306.cn. Opening the homepage requires establishing over 60 HTTP connections, and the ticket booking page has over 70 HTTP requests. Modern browsers make concurrent requests (of course, a single browser page has a limited number of concurrent connections, but you can’t stop users from opening multiple pages, and backend server TCP links aren’t immediately released or reused when the front end disconnects). So, with just 1 million users, there could be 60 million connections (this number drops after the first visit due to browser cache, but even if only 20% remain, it’s still millions of connections). That’s way too many. A single login/query page would be better. Bundle JS into one file, CSS into one file, and icons into one file, using CSS sprites for display. Minimize the number of connections.

III. Reduce Page Size, Increase Bandwidth

Not every company dares to run an image service, because images consume too much bandwidth. In the broadband era, it’s hard to remember the dial-up days when you hardly dared use images on a webpage (the same applies to mobile browsing today). I checked the total download size needed for 12306’s homepage, which is about 900KB. If you’ve visited before, the browser caches a lot, and you only need to download about 10K. But we can imagine an extreme case: 1 million users accessing simultaneously for the first time, each downloading 1MB. If this needs to be returned within 120 seconds, you’d need 1M * 1M / 120 * 8 = 66Gbps of bandwidth. Astounding, right? So, I estimate that on the day, 12306’s blockage was primarily due to network bandwidth, which is why you might have seen no response. Later, as browser caches helped reduce much of 12306’s bandwidth usage, the load shifted directly to the backend, and the backend data processing bottleneck immediately surfaced. That’s when you’d see many HTTP 500 errors, indicating the backend servers had collapsed.

IV. Static Front-End Pages

Make infrequently changing pages and data static, and gzip them. An extreme method is to place these static pages in /dev/shm, which is a memory directory, reading files directly from memory to return, thus reducing expensive disk I/O. Using nginx‘s sendfile feature allows these static files to be exchanged directly in kernel mode, dramatically increasing performance.

V. Optimize Queries

Many people are querying the same thing. A reverse proxy can be used to merge these concurrent identical queries. This technique is mainly implemented using query result caching: the first query goes to the database to fetch data and places it in the cache, and all subsequent queries directly access the high-speed cache. Hashing each query and using NoSQL technology can accomplish this optimization. (This technique can also be used for static pages)

For querying train ticket availability, I personally think they shouldn’t display a number, just “Available” or “Sold Out”. This would greatly simplify system complexity and improve performance, offloading query load from the database so it can better serve those placing orders.

VI. Caching Issues

Caching can be used to cache dynamic pages or query data. Caching typically has a few issues:

1) Cache<<>>
: georgia; font-size: 15px;”>2) Cache Eviction. Memory may be insufficient, so it is necessary to swap out some inactive data from memory. This is very similar to memory paging and swapping in operating systems. FIFO, LRU, and LFU are all classic page replacement algorithms. For related content, refer to Wikipedia’s Cache Algorithms.

3) Cache Rebuilding and Persistence. Caches reside in memory, and systems always require maintenance, so cache data will be lost. If the cache is gone, it needs to be rebuilt. If the data volume is large, the cache rebuilding process will be very slow, which can impact the production environment. Therefore, cache persistence is also a consideration.

Many powerful NoSQL solutions provide excellent support for the three major caching issues mentioned above.

Backend Performance Optimization Techniques

Having discussed frontend performance optimization techniques, the frontend may no longer be the bottleneck. Performance issues will then shift to the backend data. Here are several common backend performance optimization techniques.

I. Data Redundancy

Regarding data redundancy, it means processing our database data with redundancy, i.e., reducing expensive operations like table joins, but this sacrifices data consistency. The risk is relatively high. Many people use NoSQL as a data store; it is fast because of data redundancy, but this poses a significant risk to data consistency. This needs to be analyzed and handled based on different business requirements. (Note: It is easy to migrate from a relational database to NoSQL, but the reverse, from NoSQL to relational, is difficult.)

II. Data Mirroring

Almost all mainstream databases support mirroring, i.e., replication. The benefit of database mirroring is that it enables load balancing. Distributing the load of one database server across multiple servers while ensuring data consistency (Oracle‘s SCN). Most importantly, this also provides high availability — if one server fails, another is still serving.

Data consistency in data mirroring can be a complex issue, so we need to perform data partitioning on individual data records. That is, evenly distributing the inventory of a hot-selling product across different servers. For example, if a hot-selling product has 10,000 units in stock, we can set up 10 servers, each holding 1000 units of stock, much like a B2C warehouse.

III. Data Partitioning

One problem that data mirroring cannot solve is having too many records in a data table, causing database operations to become too slow. Hence, data partitioning. There are many approaches to data partitioning, generally including the following:

1) Classifying data based on some logic. For example, a train ticket booking system could be partitioned by railway bureau, by train type, by departure station, by destination…… In short, it means splitting one table into multiple tables with the same fields but different categories, so these tables can reside on different machines to share the load.

2) Partitioning data by field, i.e., vertical table splitting. For instance, placing data that is not frequently modified in one table, and frequently modified data in several other tables. Turning one table into a 1-to-1 relationship reduces the number of fields in a table, which can also improve some performance. Additionally, having too many fields can cause a single record’s storage to be spread across different page tables, which is problematic for both read and write performance. However, this introduces a lot of complex control logic.

3) Even table splitting. Since the first method does not guarantee an even distribution—certain categories of data might still be very large—an even distribution approach is also used, splitting tables based on the range of the primary key ID.

4) Same-data partitioning. This was mentioned above under data mirroring. It involves distributing the inventory value of the same product across different servers. For example, if there are 10,000 units in stock, they can be distributed across 10 servers, each having 1,000 units. Then load balancing is applied.

Each of these three partitioning methods has its pros and cons. The most commonly used is still the first one. Once data is partitioned, you need one or more schedulers to let your frontend application know where to find the data. Partitioning train ticket data and placing it in various provinces and cities would provide a very meaningful and qualitative performance improvement for the 12306 system.

IV. Backend System Load Balancing

We discussed data partitioning earlier. Data partitioning can alleviate load to some extent, but it cannot reduce the load from hot-selling products. For train tickets, this can be considered as tickets for certain mainline routes in major cities. This requires using data mirroring to reduce the load. Using data mirroring necessitates load balancing. On the backend, using load balancers like those on routers might be difficult because those balance traffic, and traffic does not represent server busyness. Therefore, we need a task distribution system that can also monitor the load status of each server.

Task distribution servers have some challenges:

• Load conditions are quite complex. What defines “busy”? Is it high CPU? High disk I/O? High memory usage? High concurrency? Or a high memory paging rate? You may need to consider all of these. This information needs to be sent to the task distributor, which then selects the least loaded server to handle the request.

• What if the task assignment server dies? Here we need high-availability techniques like Live-Standby or failover. We also need to address how to transfer the persistent task queue to another server.

I’ve seen many systems use static distribution methods—some use hashing, others simply round-robin. None of these are good enough. For one, they can’t achieve perfect load balancing. More critically, the fatal flaw of static methods is that if one computing server goes down or we need to add a new server, the distributor must be aware of it. Additionally, you have to recalculate the hash (consistent hashing can partially solve this problem).

Another approach is preemptive load balancing, where downstream computing servers go to the task server to pull tasks themselves. These computing servers decide on their own whether to take a task. The advantage is that it simplifies system complexity and allows you to add or remove computing servers in real time at will. The only downside is that if certain tasks can only be processed on specific types of servers, it may introduce some complexity. But overall, this method is probably a better load balancing strategy.

5. Asynchronous Processing, Throttling, and Batch Processing

Asynchronous processing, throttling, and batch processing all require queuing concurrent requests.

• Asynchronous processing, from a business perspective, generally means collecting requests and handling them with a delay. Technically, it allows individual processing routines to run in parallel, enabling horizontal scaling. However, the technical challenges of asynchronous processing include: a) Returning results from the callee involves inter-process or inter-thread communication issues. b) If rollback is needed, it becomes somewhat complex. c) Asynchronous processing usually involves multithreading and multiprocessing, making concurrency control relatively tricky. d) Many asynchronous systems use message mechanisms, and message loss or out-of-order delivery are also complex problems.

• Throttling doesn’t actually improve performance; it is primarily a protective mechanism to prevent the system from being overwhelmed by traffic it cannot handle. Throttling is generally used for systems you cannot control, such as a banking system that interfaces with your website.

• Batch processing is the technique of handling a group of nearly identical requests all at once. For example, if many people are buying the same item simultaneously, there’s no need to write to the database for each purchase; you can collect a certain number of requests and perform a single operation. This technique can be applied in many areas, such as saving network bandwidth. We all know about the MTU (Maximum Transmission Unit) on networks: for Ethernet it’s 1500 bytes, and fiber optics can reach over 4000 bytes. If your network packet doesn’t fill this MTU, you’re wasting bandwidth because the NIC driver is only efficient when reading block by block. Therefore, when sending network packets, we need to collect enough information before performing network I/O—this is also a form of batch processing. The enemy of batch processing is low traffic volume, so batch processing systems typically set two thresholds: one for job quantity and another for timeout. Processing starts as soon as either condition is met.

Therefore, any asynchronous system will generally have a throttling mechanism, and will generally have a queue for lining up requests. With a queue comes persistence, and the system will generally use batch processing methods.

The “queuing system” designed by Yunfeng is precisely this technique. It’s very similar to an e-commerce order system. Meaning, my system receives your ticket purchase request, but I haven’t actually processed it yet. My system will throttle these massive requests based on its own processing capacity and handle them bit by bit. Once processing is complete, I can send an email or SMS telling the user they can now proceed to make the actual purchase.

Here, I want to discuss Yunfeng’s queuing system from business and user requirement perspectives. While it technically appears to solve the problem, there may be some aspects worth deeper consideration from the standpoint of business and user needs:

1) DoS attacks on the queue. First, let’s consider: is this just a simple queue? This is not good enough because it cannot eliminate scalpers, and a simple ticket_id is highly vulnerable to DoS attacks. For instance, I could generate N ticket_ids, enter the purchase flow, but not buy, just tying up the system for half an hour. Very easily, I could prevent legitimate buyers from getting tickets for days. Some might say users should queue using their ID number, and then be required to buy with that same ID. But this still can’t stop scalpers or ticket resellers because they can register N accounts to queue up, yet never buy. The scalpers’ only job at that point is to make the website inaccessible to normal people, forcing users to buy through them.

2) Queue consistency? Does operating this queue require locks? Where there are locks, performance inevitably suffers. Imagine 1 million people simultaneously requesting a position number assignment—this queue will become a performance bottleneck. It certainly won’t perform as well as a database, so it might be even worse than the current situation. Racing for database access and racing for queue access are essentially the same thing.

3) Queue wait times. Is a 30-minute purchase window enough? Is it too long? What if the user is offline exactly then? If the window is too short, users who don’t have enough time to complete the operation will complain; if it’s too long, those waiting behind in the queue will complain too. This method likely has many practical operational issues. Moreover, 30 minutes is far too long and completely unrealistic. Let’s use 15 minutes as an example: Suppose there are 10 million users, and only 10,000 can be let in at any moment. These 10,000 users need 15 minutes to complete all operations. Then, processing all 10 million users would require 1000 * 15m = 250 hours, or ten and a half days. The train would have long since departed. (I’m not making this up. According to Ministry of Railways experts: In recent days, the average daily order volume is 1 million. So, processing 10 million users would take ten days. This calculation might be a bit simplistic, but my point is, even under such a low-load system, queuing may not solve the business problem.)

I think we can absolutely learn from online shopping. When queuing (placing an order), collect user information and the tickets they want, and allow users to set ticket purchasing priorities. For example, if a sleeper berth on Train A is unavailable, buy a sleeper berth on Train B; if that is also unavailable, buy a hard seat, and so on. Then users top up the required funds in advance, and the system processes the orders completely automatically and asynchronously. Whether successful or not, notify the user via SMS or email. This way, the system not only saves half an hour of user interaction time and speeds up processing through automation, but can also merge identical ticket purchase requests for batch processing (reducing database operations). The most brilliant aspect of this method is knowing the demands of these queued users. It can not only optimize user queues and distribute users across different queues, but also, much like an Amazon Wishlist, through some calculations, enable the Ministry of Railways to coordinate and adjust train schedules (Finally, the queuing system (order system) must still be stored in a database or persisted; it cannot exist only in memory, otherwise, if the machine goes down, just wait for the public backlash).

Summary

Having written so much, let me summarize:

0）No matter how you design it, your system must be able to scale out horizontally with ease. That is to say, every link in your entire data flow must be capable of horizontal scaling. This way, when your system encounters performance issues, the suggestion to “add 30 times more servers” won’t become a laughingstock.

1）The technologies mentioned above cannot be mastered overnight; without long-term accumulation, it’s basically hopeless. We can see that whichever approach you use will introduce some complexity; design is always about making trade-offs.

2）Centralized ticket selling is very difficult to handle. Using the aforementioned technologies can give the ticketing system a performance boost of several hundred times. However, setting up sub-stations in various provinces and cities to sell tickets separately is the best way to achieve a qualitative leap in the existing system’s performance.

3）The business model of ticket snatching during the Spring Festival travel rush, where supply is far less than demand, is quite perverse. The business model of making tens of millions or even hundreds of millions of people log in simultaneously at 8 o’clock on a certain morning to snatch tickets is the most perverse of all. The perversity of this business model dictates that no matter what they do, they will definitely be criticized.

4）Building such a massive system just for one or two weeks, while it sits idle the rest of the time, is a bit of a waste. This is something probably only the railway system would do.

Update September 27, 2012 

 Alexa Statistics on 12306’s PV (Note: Alexa’s PV definition is: multiple clicks by a single user on a single page within a day are only counted once)

（Please credit the author and source when reposting this article; do not use for commercial purposes）

Reposted from: http://coolshell.cn/articles/6470.html