<<
This afternoon, at the Storage and System Architecture sub-forum of the IT168 System Architect Conference, Zhang Wensong, Chairman of the Taobao Technology Committee and core engineer at Taobao, gave a detailed introduction to the architecture of Taobao’s image processing and storage system. Dr. Zhang Wensong’s presentation covered Taobao’s overall system architecture, the Taobao image storage system architecture, the independently developed TFS clustered file system, the front-end CDN system, and Taobao’s applications and explorations in energy-efficient servers.
This article focuses on the back-end image storage system architecture of Taobao, including the TFS clustered file system and the front-end processing server architecture.
Solving the System Nightmare of Massive Concurrent Small Files
For a high-traffic e-commerce website like Taobao, the requirements for an image system are on a completely different level from everyday photo sharing. Daily photo sharing usually involves a limited circle of friends and family, with relatively low traffic. However, product images in Taobao shops, especially those of popular items, generate immense access traffic. Moreover, for sellers, images are far superior to text descriptions, so they place great importance on image display quality, upload time, and access speed. According to Taobao’s traffic analysis, image access traffic accounts for over 90% of the entire Taobao traffic, while the main site’s web pages account for less than 10%.
At the same time, the storage and retrieval of these images present some troublesome requirements: for instance, these images need thumbnails generated in various sizes and specifications based on different application placements. Considering multiple different application scenarios and the possibility of redesigns, a single original image might need over 20 thumbnails of different dimensions generated.
The total capacity of Taobao’s image storage system is 1800TB (1.8PB), with 990TB (approximately 1PB) currently occupied. The number of saved image files has reached over 28.6 billion, including thumbnails generated from original images. The average image size is 17.45K; images under 8K constitute 61% of the total image count and 11% of the storage capacity.
This poses a huge challenge for Taobao’s system. As is well known, the most troublesome issue for most systems is the large-scale storage and reading of small files, because the disk head requires frequent seeking and track switching, which can easily lead to longer read delays. Under massive high concurrency, this is simply a system nightmare.
Analyzing the Economic Benefits of Self-Development vs. Commercial Systems
Taobao was founded in 2003 and has undergone considerable experimentation and exploration in system construction and planning.
The diagram below shows Taobao’s image storage system before 2007. Taobao previously used commercial storage systems, employing NetApp’s file storage systems. As the number of image files on Taobao grew at a rate of doubling each year (i.e., 3 times the original volume), the backend NetApp storage systems were continuously upgraded from low-end to high-end. By 2006, even NetApp’s most advanced products could no longer meet Taobao’s storage requirements.
Dr. Zhang Wensong summarized several limitations and shortcomings of commercial storage systems here:
First, commercial storage systems are not specifically optimized for the environment of storing and reading small files; second, the huge number of files overwhelms network storage devices; additionally, the number of servers connected to the system was increasing, and the number of network connections had reached the limit of the network storage equipment. Furthermore, the expansion cost of commercial storage systems is high (10TB of storage capacity costs several million RMB), and they suffer from single points of failure, making it difficult to guarantee disaster recovery and security.
Discussing the economic efficiency comparison between commercial systems and self-developed ones, Dr. Zhang Wensong listed the following points of experience:
1. Commercial software often struggles to meet the application demands of large-scale systems, whether for storage, CDN, or load balancing, because it’s very difficult for vendors to simulate such large-scale data tests in their labs.
2. During the development process, combining open-source and self-development leads to better controllability. If a system issue arises, it can be solved from the bottom layer, and system scalability is also higher.
Economic Efficiency Comparison: Self-Development vs. Purchasing Commercial Systems
3. Based on the effect of a certain scale, R&D investment is worthwhile. The chart above shows the input-output ratio comparison between self-development and purchasing commercial systems. In fact, to the left of the intersection point on the chart, purchasing commercial systems is the more practical and economically efficient choice. Only when the scale exceeds the intersection point can self-development achieve better economic results. Actually, not many companies reach such a scale, but Taobao has far surpassed that point.
4. Self-developed systems can be continuously optimized at both the software and hardware levels.
TFS Version 1.0 Clustered File System
Starting in 2006, Taobao decided to develop a file system specifically addressing the challenge of storing massive small files to solve its image storage problem. By June 2007, TFS (Taobao File System) officially went live. The cluster scale in the production environment reached 200 PC Servers (146G*6 SAS 15K Raid5), with file counts reaching hundreds of millions; system deployed storage capacity: 140 TB; actual used storage capacity: 50 TB; single server supported 200+ random IOPS, with traffic of 3MBps.
The diagram shows the logical architecture of TFS 1.0, the first version: The cluster consists of a pair of Name Servers and multiple Data Servers. The two Name Server machines operate as a high-availability pair, representing the management node concept within the clustered file system.
• Each Data Server runs on a standard Linux host
• Data files are stored in the form of block files (typically 64M per block)
• Multiple copies of blocks are stored to ensure data safety
• The ext3 file system is used to store data files
• Disk RAID5 provides data redundancy
• Metadata information is embedded in the filename; users maintain the mapping relationship between TFS filenames and actual files themselves – making the metadata volume very small.
The most ingenious core design of the Taobao TFS file system lies in the fact that in traditional cluster systems, there is only one copy of metadata, usually managed by a management node, which easily becomes a bottleneck. For Taobao users, they don’t actually care about the specific names used to save image files. Therefore, TFS’s design considered hiding some metadata information within the saved filenames of images, such as image size, time, access frequency, and the logical block number. Consequently, the metadata stored is very minimal, making the metadata structure extremely simple. It only needs a fileID to accurately locate the file.
Because a large amount of file information is hidden in the filename, the entire system completely abandons the traditional directory tree structure, which incurs the highest overhead. Removing it greatly improves the high scalability of the entire cluster. In fact, this design concept is similar to the industry’s current “object storage.” The Taobao TFS file system has been updated to version 1.3, and its performance has been verified and continuously improved and optimized in the production system. Taobao is currently at the forefront of research in the object storage field.
TFS Version 1.3 Clustered File System
By June 2009, TFS version 1.3 went online. The cluster scale expanded significantly and was deployed in Taobao’s image production system. The entire system grew from the original 200 PC servers to 440 PC Servers (300G*12 SAS 15K RPM) + 30 PC Servers (600G*12 SAS 15K RPM). The supported file count also expanded to tens of billions; system deployed storage capacity: 1800TB (1.8PB); current actual storage capacity: 995TB; a single Data Server supports 900+ random IOPS and 15MB+ traffic; the current physical memory usage of the Name Server is 217MB (the server uses a Gigabit Ethernet card).
TFS 1.3 Version Logical Architecture Diagram
The diagram shows the logical architecture of TFS version 1.3. In TFS 1.3, Taobao’s software working group focused on improving heartbeat and synchronization performance. In the latest version, heartbeat and synchronization can be switched within seconds. Some new optimizations were also implemented, including storing metadata in memory, cleaning disk space, and performance optimizations, which include:
- A completely flat data organization structure, abandoning the directory structure of traditional file systems.
- Establishing a proprietary file system on top of block devices, reducing performance loss caused by data fragmentation in file systems like EXT3.
- Managing a single disk with a single process, discarding the RAID5 mechanism.
- A central control node with an HA mechanism, balancing safety, stability, and performance complexity.
- Minimizing metadata size and loading all metadata into memory to improve access speed.
- Load balancing and redundancy safety strategies across racks and IDCs.
- Completely seamless capacity expansion.
The topology diagram of the entire Taobao image processing system is detailed later in the “Image Server Deployment and Caching” section. We can see that in Taobao’s deployment environment, TFS has two layers of cache in front. Requests reaching the TFS system are very discrete, so there is no data memory cache within TFS, including the memory cache typical of traditional file systems.
The main performance parameter for TFS is not IO throughput, but the random read/write IOPS provided by a single PC Server. Due to different hardware models and some technical confidentiality reasons, it’s difficult for Taobao to give a reference value for performance. However, it can basically achieve around 60% of the theoretical maximum random IOPS of a single disk, and the overall output increases linearly with the number of disks.
TFS 2.0 Under Development and Open-Source TFS
TFS 2.0 is already under development, mainly addressing the challenges of large file storage. TFS was initially developed to tackle the problem of frequent concurrent reading of small files, designed with a block size of 64MB, meaning each file is smaller than 64MB. This is perfectly sufficient for general image storage, but there are bottlenecks for large file storage.
TFS 2.0 will focus on optimizing storage for large files spanning blocks. Additionally, it includes optimizing for the characteristics of different hard drives like SSDs and SAS drives. According to Taobao’s data, the storage cost of SSDs is about 20 RMB per GB, SAS drives cost about 5-6 RMB per GB, and SATA drives cost less than 1 RMB per GB. As application performance demands increase, adopting SSDs is a future trend, and optimizing for the access characteristics of different hard drives is essential.
Furthermore, Zhang Wensong announced that TFS will be fully open-sourced in September. Full open-source means Taobao will provide all source code, and the open-source version of TFS will be completely identical to the system running in Taobao’s production environment.
Image Server Deployment and Caching
