June 9, 2010 was a day of holy war. It kicked off right at midnight, and the main event unfolded as expected at 7 p.m. The battleground on Renren was the SJ public page: http://page.renren.com/sj
For different people, this day meant different things, with different flavors. As the Renren tech team, our job was to ensure that our service capacity and user experience could handle the challenge.
A single server’s capacity is always limited. To cope with the sudden surge in reads and writes, the web service architecture, internal service architecture, database architecture, and so on, all had to be able to scale easily by provisioning more servers. For the web servers alone, we doubled the number of machines. Looking back at the monitoring data now, everything looks perfect. Throughout this period, the service experienced zero interruptions in availability. Beyond that, during this holy war, a piece of technology unique to us played a critical role: rose portal . Here is an introduction to it:
This is SJ’s main page:
The page is divided into three columns:
- On the left are “Recommend to Friends”, “Basic Info”, “Albums”;
- In the middle are “Leave SJ a Message”, “Friends’ Messages”;
- On the right are “Friends”, “Pages Renren Users Also Follow”, etc.
In the backend, these are broken down into different modules, which we call ”windows”. Each of these windows might connect to an independent service cluster鈥攕uch as the basic info service, message service, friends service, album service, and so forth. In this way, a public page is composed of multiple independent, configurable window modules, as illustrated below:
As the great holy war intensified, the page turned into this (the right column disappeared):
When the product folks saw this situation, they were still quite happy: “As long as the message window is there, it’s okay if the others aren’t.”
But soon, it deteriorated further:
And even:
Huang Jing looked at the public page in this state and laughingly described such an image as “missing an arm and a leg”: “Why haven’t we added more machines yet?” When the public page tech team gradually doubled the number of machines, this situation became rarer, or even disappeared entirely.
Although these pages looked “like they were missing an arm and a leg,” it’s important to know that before, in this kind of scenario, one blocked module on our page could cause the user’s browser to stay blank for a long time, until it finally indicated that the web page could not be displayed. This created a terrible user experience, and because the connection was never released, a vicious cycle would set in, eventually causing the web servers to go completely silent.
Fortunately, we had developed Rose Portal six months earlier.