When a website reaches a certain scale, performance issues easily arise across the board: the database, overloaded front-end servers, I/O bottlenecks, insufficient cache as data grows, or excessive network overhead for data synchronization. Below, I’ll discuss problems encountered in my own project and possible optimization approaches鈥攆eedback is welcome. (Our project currently sees around 20 million daily page views.)
Initially, our project had two front-end servers, using DNS-based distribution for load balancing. As traffic grew, those two servers became extremely sluggish鈥攑age load latency increased, and the servers clearly showed high CPU load with idle hovering near zero. Logging into the servers to perform operations was painfully slow. It was obvious we were short on server capacity. Because the front-end servers responded so slowly, many requests piled up, and since the front-end and database used persistent connections, this indirectly affected the database as well, frequently causing it to exceed the connection limit (we had set the maximum to 800). Naturally, we needed more machines. We added two more servers, similarly using DNS-based distribution, so we ended up with two servers on China Telecom and two on China Netcom to handle the traffic. So, the first rule is: increase hardware resources鈥攕imple and effective.
Next, as traffic continued to climb, we noticed the database connection count stayed persistently high and easily maxed out. The database also gradually showed signs of heavy I/O, with reads around 6鈥?0 MB/s. Since we hadn’t yet planned for database sharding in the code, we decided to use the backup slave database to serve front-end read requests. Fortunately, our code already separated reads and writes (meaning each CGI connected to the database with two connections鈥攐ne for writing, one for reading), so it was easy to switch front-end read requests over to the slave database. This instantly lightened the load on the master database, which now mainly handled write operations. The master was freed up and has been idling happily ever since, while the slave database took on the heavy burden of front-end read requests from that point on.
Later, we discovered that the slave database, under heavy read load, was experiencing replication lag with the master. At the time, I actually considered modifying the MySQL source code to raise the priority of the replication thread to solve this. We didn’t end up doing that, because we soon added another slave database. Now with two slave databases sharing the majority of front-end read requests, the situation improved.
Later still, as traffic kept climbing, both slave databases began to suffer from replication lag. Of course, our first thought was: add more slave databases? That might be a solution, but we also had to consider other issues, like: if they all replicate from the master, would that increase pressure on the master? We tried the following approaches:
1. Static Page Generation
Our project is similar to a news system but with some distinctions. We considered statically generating list pages, regenerating them every 10 minutes. The setup was roughly this: every 10 minutes, Machine A would query the database, regenerate the list pages, then sync those pages to front-end Machines B, C, and so on. The front-end machines were also configured to set HTTP Expires headers to reduce browser requests and Last-Modified headers to cut down on network transfers. This was effective鈥攊t reduced pressure on the front-end and also decreased database requests. But we discovered another problem: Machine A would periodically create a spike of load against the database, and this proactive generation method was very indiscriminate. Moreover, generating all list pages鈥攗ncompressed, totaling several gigabytes鈥攁nd syncing them to all front-end machines within minutes was a huge challenge for both Machine A and the front-end servers. In practice, we found that just generating a portion of the list pages already made Machine A quite busy.
2. mod_cache
Similar to static generation, using mod_cache switches the approach from proactive to reactive, making page generation no longer indiscriminate. The setup was roughly this: all four front-end servers were configured with mod_cache, caching for 5 minutes. Whenever a page was missing locally or had expired, the server would request the new page from Machine A and cache it locally. This required minimal code changes, and as an Apache module, it avoided the overhead of spawning new FastCGI or WSGI processes to serve requests (on the front-end machines). Considering that all four servers fetched pages from Machine A, we could actually add another layer of mod_cache on Machine A itself: the front-end would request Machine A, Machine A would request the page to generate a static copy, then return that static page to the front-end. Subsequent front-end requests to Machine A could then directly fetch the already-generated static page. Unfortunately, we never got to properly test this method in practice, but I believe it’s viable鈥攅specially if the generated static pages could be Gzip-compressed, reducing network transfer overhead between Machine A and the front-end, making it even more efficient.
3. Memcache
Memcache and mod_cache are actually quite similar鈥攐ne uses memory, the other uses disk. To our surprise, compared to the several gigabytes of disk consumption for static files, caching in memory with Memcache only took up a few hundred megabytes, all thanks to Memcache’s compression.
Using Memcache, the project structure is also very similar to mod_cache. The front-end receives a request, tries to fetch data from memory; if it’s not there, it retrieves it from the database and then stores it in memory. This Memcache machine is analogous to Machine A in the second approach, but using Memcache itself offers several advantages: First, data transfer is already compressed, so memory usage is small and network transfer overhead is minimal. Second, from real-world observation, the Memcache server is extremely efficient, using very little CPU. Compared to mod_cache, it essentially distributes the pressure from Machine A across the front-end machines (with Memcache, the front-end servers are still the ones querying the database; with mod_cache, only Machine A queries the database). This spreads the load and improves efficiency.
In actual testing, none of the three approaches above ended up being used in our project鈥攏ot even Memcache.