Currently, domestically developed file systems in China are few and far between. Taobao has conducted effective exploration and practice in this field. The Taobao File System (TFS), a distributed file system used internally by Taobao, has been specially optimized for random read/write access performance of massive small files, carrying the data storage for all images, product descriptions, and more on the Taobao main site.
Recently, Chu Cai (Li Zhen), an engineer from Taobao’s Core Systems Team, published an article on the official blog (titled “Introduction to TFS,” hereinafter referred to as the article) briefly introducing the basics of the TFS system, which attracted community attention.
The article first summarized the features of TFS:
A completely flattened data organization structure, abandoning the traditional directory structure of file systems.
Establishes its own file system on top of block devices, reducing performance loss caused by data fragmentation in file systems like EXT3.
Manages a single disk per process, eliminating the RAID5 mechanism.
A central control node with an HA mechanism, striking a balance between security, stability, and performance complexity.
Minimizes metadata size, loading all metadata into memory to boost access speed.
Cross-rack and cross-IDC load balancing and redundancy security strategies.
Fully smooth scaling.
Currently, the scale of TFS deployment at Taobao reaches “hundreds of PC servers, petabyte-level data volume, tens of billions of data items.” Regarding its performance parameters, Chu Cai disclosed:
In Taobao’s deployment environment, TFS has two layers of caching at the front end, making requests arriving at the TFS system very discrete. Therefore, TFS does not have any internal data memory cache, including the memory cache of traditional file systems… Basically, we can achieve about 60% of the theoretical maximum random IOPS (I/O per second) for a single disk, and the overall machine output increases linearly with the number of disks.
Chu Cai provided further explanation with an architecture diagram:
- TFS has not yet provided traditional file system APIs to end users; access requires using TFSClient, with existing clients for JAVA, JNI, C, and PHP.
- TFS’s NameServer acts as the central control node, monitoring the operational status of all data nodes, responsible for load balancing of read/write scheduling, and managing primary metadata to help clients locate the data nodes they need to access.
- TFS’s DataServer acts as the data node, responsible for the load balancing and data redundancy of actual data storage, and managing secondary metadata to help clients retrieve real business data.
After the article was published, reader response was enthusiastic, with various questions raised in the comments for technical exchange with the author Chu Cai, demonstrating the domestic community’s interest in independently developed file systems.
To allow readers a deeper understanding of TFS’s secrets, InfoQ China conducted an exclusive interview with Chu Cai (Li Zhen) focusing on TFS’s origin, operating environment, scalability, architecture, development roadmap, and open-source matters.
InfoQ: Why did Taobao choose to develop its own file system?
TFS (Taobao File System) is a distributed file system independently developed by Taobao for storing data on the Taobao main site, such as product images, product descriptions, transaction snapshots, community images, and more.
A prominent characteristic of this data is that individual file sizes are small, typically not exceeding 1MB, but the quantity is enormous. Traditional file systems or network storage devices struggle to solve similar problems.
Taobao initially used network storage devices provided by NetApp. However, during 2006-2007, due to the explosive growth in data volume, we faced pressure to upgrade the system. Because of limitations on application connection counts and file numbers, simple upgrades had actually become very expensive and not cost-effective.
Data is the core of Taobao’s applications, and in this area, we needed to provide a safer, more efficient, and cheaper solution. We needed to develop our own technology, better suited to Taobao’s unique application characteristics. After evaluation, none of the existing open-source distributed file systems could fully meet our needs. Taobao might have been the first domestic internet company at the time to encounter similar problems, although it’s possible other companies encountered them but didn’t publicize it. In any case, at the time, there weren’t some relatively reliable open-source systems to choose from like there are now.
InfoQ: What is the operating environment of the TFS system?
When selecting the platform for TFS, we used the mainstream operating system employed by Taobao at the time, REHL, and built it on top of the LINUX EXT3 file system. The reason for choosing REHL was that it’s an open-source operating system, giving us more freedom and more cost savings.
Regarding the hardware platform, it underwent a relatively long transition period alongside hardware development and changes in our understanding. Distributed file systems do not have high CPU requirements. Meanwhile, because we adopted some special methods that reduced system metadata, the requirements for memory size were also not high. All performance and cost bottlenecks were concentrated on disk IO. All evolutions of our hardware architecture were reflected in the way disks were used.
Initially, we used SAS 146G × 6 disks for storage, later upgrading to SAS 300G × 6 disks with a RAID5 configuration, which was the mainstream hardware platform at the time. After gaining experience and gradually building confidence in system stability, this approach became outdated. The redundancy provided by RAID5 duplicated the data redundancy we had already implemented at the application layer, wasting significant disk space. Furthermore, the working principles of the RAID card could not optimize for TFS’s greater emphasis on random IO. Eventually, we removed the RAID card and adopted a multi-process approach, where each process manages a single disk, fully leveraging the random read/write performance of the disks.
We used the SAS 300G × 6 disk configuration for a period, and it proved relatively stable. This gave us the confidence to use higher-capacity nodes without being overly concerned about the impact of data migration on system performance during server failures. Currently, TFS’s primary machine configuration uses SAS 300G × 12 disks.
As the CDN in front of Taobao becomes increasingly efficient and stable, the request traffic reaching TFS is smoother. This allows us to plan the service capacity of individual nodes more reliably and enables the use of cheaper storage devices. In the future, our main storage nodes will use 1TB SATA × 12 or even larger disk configurations, further reducing costs.
InfoQ: TFS is deployed on hundreds of PC servers with petabytes of data. How is its scalability? How is this guaranteed architecturally?
In TFS, we merge a large number of small files into a single large file, similar to the concept of a Chunk in GFS. The location information for these Chunks is called the primary index, while the specific file location information within a chunk is called the secondary index. This index information is also embedded in the TFS filename. Before a user writes a file, they must request a filename from the TFS system. While this approach can be less flexible than traditional file systems in some scenarios, it grants our system greater scalability. We ensure that the memory of the central control node can support a petabyte-level primary index, while the secondary index only needs to handle data for a single node. This avoids the scaling difficulties caused by data volume expansion.
When storage capacity becomes insufficient and we need to scale out, we can plan based on data growth and add any number of servers providing corresponding storage. These new storage servers will report to the central control node. When there is data to be written, the central control node dynamically assigns the write destination server based on parameters such as the percentage of used capacity and current system load. During system idle periods, the central control node also formulates data migration plans based on the current data distribution and gradually completes data rebalancing. Similarly, when a server crashes, the central control node performs data migration to ensure sufficient backups, along with data rebalancing operations. These operations are all performed automatically without manual intervention.
This approach works well for clusters up to around 1,000 nodes. If the cluster scale is larger, the central control node might experience some performance bottlenecks. At that point, we could use a distributed cluster to solve this, and there are already many relatively mature technical solutions available for this.
InfoQ: When introducing TFS’s features, you often mention the keyword “performance.” What factors should generally be considered regarding file system performance? What are the common methods for improving file system performance currently?
Now we can discuss the performance dimensions considered for TFS. In fact, every popular distributed file system today has its own focus, tailored to different application scenarios.
For offline data analysis needs, due to the massive total data volume, the underlying distributed file system prioritizes overall system throughput. To adapt to the current popular map_reduce model, these types of file systems split files into multiple parts, divide and conquer. They cooperate with upper-layer logic, using large-block sequential write and read methods to boost performance. A typical representative is HDFS used by Hadoop.
In distributed file systems providing actual online services, differentiated storage methods are also adopted based on service types. For instance, for relatively large files like audio and video, an approach similar to offline applications might be used, slicing files for concurrent read/write access to achieve higher transfer speeds. Since the number of files is relatively small for the same capacity, it might even be possible to fully implement features similar to traditional file systems, like directories and permissions, without being limited by inodes. Facing the same massive volume of small files that Taobao deals with, this method completely fails to provide performance support. The explosion in the number of inodes quickly consumes large amounts of expensive memory. If we compromise on cost by placing some inodes on disk, we cannot guarantee addressing efficiency when faced with essentially completely random, non-hotspot access. We can only solve this problem by reducing metadata.
Why do we say TFS faces completely random data access with virtually no hotspots? In Taobao’s data deployment, there are two caching layers closer to the user in front of TFS to ensure the speed of user display pages. The requests that eventually reach TFS account for only about 2% of the total. As the efficiency of the front-end cache continues to improve, this proportion will further decline. Can we imagine if hotspot data still exists? Meanwhile, the business characteristics of Taobao dictate that the volume of write requests is orders of magnitude lower than read requests, and modification operations are even fewer. This determines that we use random modification and random writing methods to avoid the cost that sequential writing without modification would impose on random reads.
Different application scenarios, different storage architectures, different performance considerations — is there any universal performance optimization method? From our practical experience, it seems not. It’s difficult to have one method solve all problems. Or rather, after solving these problems, people find that compared to specialized file systems, a general-purpose system never achieves the expected performance and simultaneously introduces significant complexity in development and tuning. In this case, perhaps establishing different clusters to solve different problems is more efficient. It all depends on the business requirements you need to support. Architectures at the scale of Google clusters and a simply built NFS both have their value.
Of course, there are still some general rules. If you need to support an online service, you should minimize the number of IOs triggered by a single request. IO includes network and physical disk operations; the performance overhead caused by these operations is determined by physical structure and cannot be optimized by technical means. That is to say, the number of these operations essentially forms the performance ceiling of your system. When you design a file system architecture, you can estimate the system’s performance under different circumstances based on this ceiling.
Secondly, you need to strike a balance between concurrency and synchronization overhead. Choose a solution better suited to your business characteristics. For example, would you choose to read 1MB of data continuously from one disk or concurrently read 100KB of data from 10 disks over the network and return it to the user? What about reading 100MB of data continuously from one disk versus 100 concurrent reads of 1MB each? Does the system have the capacity to support such concurrency? Most of the time, we are making similar choices.
Thirdly, you need to balance cost and performance. We can conveniently use memory-based caching to dramatically improve system performance, provided your system’s hotspot data is concentrated. As SSD technology matures, the random read performance it offers is exciting compared to traditional mechanical disks, and random write performance has also greatly improved, although not as dramatically as reads. If your application has a high read/writex; line-height: 21px; color: rgb(0, 0, 0); clear: none; width: 610px; font-family: Arial, sans-serif;”>InfoQ: Looking at TFS’s logical architecture diagram, with NameServer as the central control node and DataServer as the data node, what factors or requirements drove this architectural decision?
TFS distinguishes between a central control node and data nodes, managing a two-level index to address data access routing and transfer. Compared to the decentralization trend seen in some current open-source distributed systems, this might not seem particularly elegant. But as with the discussion on performance, every system architecture has its own merits. A central node can guarantee transactional integrity; in certain scenarios we must ensure strong consistency for written data. At the same time, within our current application scale, it solves problems in a very safe and efficient manner. If higher availability and scalability are required, we will partially incorporate decentralized concepts into certain parts of TFS.
InfoQ: What are the future development plans for TFS?
TFS’s future development will primarily continue to be rooted in Taobao’s own business needs, while also catering to the demands of the developer community. We will gradually support large file storage, as well as directories and user permissions. We also plan to implement tiered storage distributed based on access characteristics, striving to strike a balance between performance and cost. There will also be more refined management features, such as data volume quota management. Based on the discussions above, we are unlikely to build a general-purpose distributed file system; we will consistently focus on solving problems that currently lack good solutions.
InfoQ: You mentioned on your blog that the TFS open-source community will be established in September, which is a fantastic learning opportunity for the domestic community. Could you describe a bit more about the open-sourcing details?
TFS is scheduled to be open-sourced at the end of September. By the end of June this year, Taobao will launch its own open-source community——code.taobao.org, and TFS will be fully open-sourced based on this community. Everyone will be able to see it soon. It has also been essentially decided to use the GPL V2 open-source license.
InfoQ: What advice do you have for developers interested in contributing to TFS or those currently working on similar system-level core applications?
For peers who aspire to join TFS development or are already engaged in foundational platform and core system R&D, I believe we share a common sentiment: within today’s computer architecture, revolutionary technological breakthroughs are rare. Currently known large-scale distributed file systems are all built on Unix-like operating systems, or rather, the vast majority are built on Linux, which is also a consideration from a cost perspective. The architectures of various distributed file systems are largely similar; what determines success lies in the details. These details include OS-level specialization, file system-level specialization, and whether the implementation is sufficiently excellent and stable. Therefore, you need to understand the system kernel and have knowledge of file systems. For example, knowing how EXT3 is organized helps you avoid scenarios where reading a piece of data causes multiple disk head movements, allowing you to maximize the utilization of system resources. Implementation specialization might manifest in writing an excellent algorithm or an efficient communication mechanism, which requires solid coding skills and profound expertise in algorithms and data structures. As everyone knows, the devil is in the details!
InfoQ China will continue to follow the development of TFS and other domestically developed file systems, and we welcome everyone to share their views.
About the Expert
Li Zhen, nicknamed Chucai, works in the Core System R&D department of Taobao’s Technology R&D division, where he leads the storage team. His focus areas include R&D of large-cluster distributed systems, massive data processing, and massive data retrieval.
To submit articles or participate in content translation for InfoQ China, please email [email protected]. You are also welcome to join the InfoQ China User Discussion Group to interact with our editors and fellow readers.
http://www.infoq.com/cn/articles/tao-tfs/