Static Page Design Principles and Steps

<<>>
Static content generation helps reduce server load, but it also introduces a series of issues, including increased development complexity and maintenance difficulty. If not applied properly, it can backfire. Many alternatives, such as page caching, can also achieve excellent results when used correctly. Therefore, before starting, you must conduct a detailed evaluation to determine whether static generation is appropriate and devise a suitable approach. Let’s first cover some fundamentals.
l         Evaluate the Read/Write Ratio:
The read/write ratio, more precisely the read/write load ratio, is the ultimate consideration for whether static generation is worthwhile. Since write pressure is generally significantly greater than read pressure, if writes are too frequent or each write consumes too many resources, the desired effect cannot be achieved. I believe a read/write ratio of 10:1 should be an upper limit. The specifics need to be judged based on your own business logic.
 
l         Determine if the Page Content is Suitable for Static Generation:
When designing the solution, you must carefully consider each prototype page, identifying the information displayed on the page, along with its update method, timing, and frequency. Pay close attention to inconspicuous details; they can make or break your design.
For example, let’s analyze any forum post on CSDN:

The content presented in the post above mainly consists of these parts: the post content, reply content, and user information for the original poster and those replying.
n         Post content and reply content are updated when posted. After posting, users can modify the content. Update frequency is high.
n         User information may change when a user modifies their profile or when their user level increases, such as gaining a star. Update frequency is low.
n         The reply count must be updated after every reply. Update frequency is high.
n         Pay attention to details during design, such as the parts circled in the image above. How are these parts modified, and how frequently? You cannot afford to overlook any of them.
l         Determine the Generation Method:
In the forum post example above, regenerating the entire page upon every change is not advisable. For a post with many replies, the amount of data required (user info for each floor, reply content) is massive. Any modification requiring fetching all that data again for regeneration is unacceptable. Generally, a full-page regeneration approach is only suitable if your page rarely needs updating or if the update cost is extremely low (like an embedded ad script). Otherwise, we need to find appropriate methods for updating local regions of a page:
There are generally two methods:
1)      Regex Modification Method:

        For instance, if the HTML code for the reply count in a post looks like this:

        <label>Reply Count<var id="replyCount">34</var></label>

        We can find and replace the count using the following regex:

        (?<=id="replyCount">)/d{1,}

2)      Page Region Chunking:
Divide the page into many small chunks that are assembled when displayed. DotText, for example, uses this method.

This is a typical DotText blog page. The area marked in red is an independent file, while the content within the yellow box is dynamically loaded by a script. These parts are combined during final display to constitute a blog post. Specific combination methods vary; you can use Include or implement your own. DotText implemented its own loading mechanism.
 
The two methods above are not mutually exclusive and can be used together as needed.
 
l         Determine Information Requiring Dynamic Loading:
There is always some content on a page that seems unsuitable for static generation. The most typical examples are statistical results. For instance, if you are creating a book introduction page, you might need to display the book’s current daily composite rating or its ranking. This content needs to be dynamically loaded via scripts.
Since the goal of static generation is to reduce server load, dynamically loaded data is always a last resort. Sometimes, when requirements permit, we make compromises between data real-timeness and performance. For example, regarding the user star level and nickname in the forum post above: from a data real-timeness perspective, when a user’s star level increases, all their past posts should reflect the change, hence dynamic loading should be used. However, in reality, if this information doesn’t change, it’s harmless; users might actually see the level and nickname they had when they posted years ago.

Real-World Project

Website X is a large-scale movie information and community site, providing movie-related information services and a user community. The information service section mostly consists of information presentation pages with relatively high read traffic (millions of PVs). Information is primarily published by editors in the backend, with infrequent updates. However, its pages contain a significant amount of interactive content, such as comments and favorite lists. Simultaneously, much content is user-generated, like image uploads and adding annotations. The volume and frequency of interactive content exceed that of a standard information page. This adjustment aims to statically generate the most heavily visited sections: movie info pages and filmmaker info pages. If successful, it will be expanded to more channels, eventually achieving site-wide static generation.
 
Based on an analysis of the page design and the previous version, here are the challenging aspects. These characteristics are common to most web 2.0 sites and are quite typical.
 
l         Complex Triggers for Page Generation
Typically, forum posts or blogs have a relatively simple update method: primarily triggered by replies, with a few modification actions. However, for this website, a single page has over 20 different triggers based on various conditions. For example, just for a secondary menu: a user uploading an image, deleting an image, publishing or deleting movie info, publishing or modifying a video, or a backend edit to movie info could all be triggers.
 
l         A Single Action Can Trigger Generation of Many, Potentially Overlapping Pages
Every action triggers a series of generations, and different actions might involve the generation of the same page or region.
For example: When a user rates a movie, it requires generating the rating page, the rating statistics page, the “Who else is interested in this movie” sidebar on the homepage, etc. If a user favorites a movie, it also requires updating the “Who else is interested in this movie” sidebar on the homepage.
 
l         High Frequency Triggers:
Although not on the scale of some larger sites, due to the extensive user-participation content—comments, favorites, etc.—there are many trigger points, and the frequency of occurrence is quite high.
 
l         Many Pages, Complex Structure, Large Disk Footprint:
Typically, the scale of pages needing generation can be roughly estimated as Rn * P, where Rn is the number of resources and P is the number of pages per resource. A resource can be viewed as a generation unit. The number of pages can simply be seen as the count of all related pages generated when publishing one resource. For example, publishing a blog requires generating a blog page and also generating or updating the blog list on the personal homepage. Including the small chunk for category article counts on the right side of the personal homepage, that’s at most about ten pages or regions. But publishing a movie involves at least 50 related pages, some with pagination. A movie rich in information can have over a thousand pages, consuming 10-20MB of space. The total number of resources is also significant: around 80,000 movies. Although the P-value for filmmakers is lower, their total count is in the hundreds of thousands. The estimated static page disk usage is several hundred GB.
 
l         Backward Compatibility
This is an existing system. The constraints of the old system need to be overcome, but there is no time, or it’s impossible, to fully break through them. For instance, URLs already indexed by search engines cannot be arbitrarily changed. Some areas were not originally designed with static generation in mind, while others need to remain compatible with the old design.
 
l         Multiple Front-End Web Servers
This structure requires that generated files may need to be distributed to multiple servers (another solution is placing them on a few dedicated machines for the front-ends to fetch).
 
l         Tight Schedule
The architecture discussion concluded in early June, leaving only two months until the Olympic Games launch deadline. This meant all underlying framework implementation, page template development, debugging, testing, and action collation had to be completed by the end of July. Based on my original estimate, just implementing the hundreds of page templates and population methods for these sections would take that long.
 
Considering the above factors comprehensively, the architecture must have the following key characteristics:
l         Actions must be flexibly extensible and configurable. Which generations correspond to which actions should be configurable and groupable.
l         A file distribution mechanism is essential.
l         Distribution and generation must be decoupled and support distributed processing.
l         All actions must be converted into messages and sent to the generation and distribution servers for processing.
l         For frequent actions on the same resource, the ability to merge them when variables are identical is needed.
l         Actions must be logged.
l         Leverage existing mature technologies as much as possible to save development time.

Below is the first architecture designed

User actions are transmitted via MSMQ>>
According to tests, there is a huge performance difference between piping content and returning HTML directly. Using SSI offers a performance compromise and can work with high-performance web servers like Light HTTPd.
 
The template generation method uses XSLT and another custom template (a mechanism developed by my colleague, which is very interesting — in theory, it can completely eliminate the performance overhead of traditional template replacement). The final output is shtml, chosen specifically to leverage its SSI (Server Side Include) features. This ensures a certain level of flexibility and enables the separation of hotspot data: some parts of a page may be updated and regenerated frequently while others remain unchanged, or a certain part is common across all pages (such as the header and footer). Compared to commonly using Smarty in PHP to generate PHP files, although the flexibility is slightly less than PHP, the performance is comparable, even slightly higher.
 
However, the problem with this design is that dynamic and static content are not separated. The generated HTML pages and dynamic pages are both placed on the front-end servers, accessed via load balancing. This also means the distribution server needs to distribute files to multiple servers, resulting in lower network I/O efficiency. Additionally, static content requires significant disk space and involves a huge number of small files. Mixing them with dynamic pages makes optimization difficult. Therefore, the second approach uses different servers for generated static content and dynamic content.
 

In summary, static generation impacts not only the architecture but also the development and testing processes.
Higher Demands on Testing:
Because once a site goes live, if an issue is found on a page, even a text modification requires regenerating many pages. Therefore, testers must be extremely meticulous, and the testing cycle needs to be extended.
 
Developers Need to Master a Template Language:
They need to master a template language, whether it is XSLT or a custom-developed one, which requires a certain amount of time to learn.
 
Sufficient Time Must Be Allocated for the Initial Generation:
If it is not a brand-new system, the data migration and generation process can be painful. Due to the large number of pages, the initial generation process might take days to calculate. This aspect must be considered when formulating the go-live plan.
 
Nginx acts as the front-end redirector. Based on the experiences of other websites, it should be able to handle 20,000 to 30,000 concurrent connections. However, after deployment, we often experienced stalling issues. The specific symptoms were connection timeouts or persistent unresponsiveness when accessing pages in the browser, while Nginx’s connection count was not high at the time. Fortunately, we had the first solution as a backup, giving us time to resolve this issue. If anyone has insights into this problem, I welcome an exchange of ideas.
My Contact Information
MSN:[email protected]
Gtalk:[email protected]
 

Postscript:

In large-scale web development, I feel that the Microsoft product structure (including achievements from the Microsoft open-source community) still has some shortcomings in certain aspects:
 
Too Few High-Performance Server Choices
Under Linux, you can use many servers like Light HTTPd and Nginx. The performance of these servers in many aspects overshadows the only choice under Windows — IIS.
 
Distributed File Systems
Microsoft and its community lack notably famous products in this area, whereas Linux has MogileFS.
 
Under the Microsoft Architecture, File System Choices Are Too Limited:
Under Linux, we can choose from systems like Ext3 and ReiserFS, while in the Windows environment, NTFS is the only choice. To its credit, however, the efficiency and stability of NTFS are quite decent.
 
Open-Source Technology Support for Windows Versions Lacks Enthusiasm
Many open-source products renowned under Linux are reluctant to provide corresponding Windows versions, or the provided Windows versions have barely satisfactory results. This leaves companies using Microsoft servers with many fewer options.
 
Modern web development has entered an era of mixing and integrating various technologies. No single vendor can cover all aspects. In terms of back-end architecture and logic, the strengths of .Net and Java lie in their rigorous, clean programming style, clear design ideas, high operational efficiency, and stable supporting services. For web engineers and architects primarily skilled in Microsoft technologies, they should enhance their understanding of Linux and the open-source community to design reasonable architectures based on requirements.