<<
NoSQL gained tremendous momentum in 2010. Web sites large and small, in their pursuit of high performance and high reliability, instinctively turned to NoSQL technologies as a top priority. Early this year, InfoQ Chinese was honored to invite Mr. Sun Li from Phoenix New Media to share his experience and insights on NoSQL.
I am truly honored to be invited by InfoQ to start this NoSQL column. InfoQ is a tech media outlet I respect immensely, and I hope to leverage InfoQ to help drive NoSQL adoption in China, welcoming all like-minded friends to join in. This NoSQL column series will first provide an overall introduction to NoSQL, then explain how to apply NoSQL to suitable scenarios in your own projects, and also appropriately analyze some success stories. I hope friends with successful NoSQL experience can provide me with leads and information.
NoSQL Concepts
With the rapid development of web 2.0, non-relational, distributed data storage has advanced swiftly; they do not guarantee the ACID properties of relational data. The term NoSQL was introduced in 2009. Its most common interpretation is “non-relational,” while “Not Only SQL” is also widely accepted. (The name “NoSQL” was first used in 1998 for a lightweight relational database.) The most widely used NoSQL model is key-value storage, but there are also document stores, column-family stores, graph databases, XML databases, and others. Before the term NoSQL was coined, these databases were already used in various systems, though rarely in web-based Internet applications—for example, cdb, qdbm, and bdb databases. The Bottlenecks of Traditional Relational Databases Traditional relational databases offer decent performance, high stability, a time-tested track record, ease of use, powerful features, and a large body of successful case studies.
In the Internet domain, MySQL has become the absolute king. It is no exaggeration to say that MySQL has made outstanding contributions to the development of the Internet. In the 1990s, a website’s traffic was generally low, easily handled by a single database. At that time, most sites were static HTML, and dynamically interactive websites were few. Over the last decade, websites began developing rapidly. Hot trends like forums, blogs, SNS, and microblogs gradually led the way in the web world. In the early days, forum traffic was actually not that high either; if you were online early enough, you might remember text-file-based forum software, which gives you an idea of the traffic volume typical forums handled. Memcached+MySQL Later, as traffic increased, almost all websites built on a MySQL architecture began experiencing database performance issues. Web programming no longer focused solely on functionality but also pursued performance. Developers began extensively using caching techniques to relieve database pressure and optimize database structure and indexes. File-based caching was initially popular for reducing database load, but as traffic continued growing, multiple web servers could not share file caches, and a large number of small file caches also introduced high I/O pressure. At this point, Memcached naturally became a highly fashionable technology product. As an independent, distributed caching server, Memcached provides a shared, high-performance caching service for multiple web servers. Scaling across multiple Memcached servers was achieved through hash-based algorithms, and later consistent hashing emerged to solve the problem of massive cache invalidation caused by rehashing when adding or removing cache servers. Back then, if you mentioned Memcached experience during an interview, it was definitely a plus. MySQL Master-Slave Read/Write Splitting As database write pressure increased, Memcached could only relieve read pressure. Having reads and writes concentrated on a single database overwhelmed it, so most websites began using master-slave replication to achieve read/write splitting, improving read/write performance and read-database scalability. MySQL’s master-slave model became the standard setup for websites at this stage.
Sharding (Table and Database Partitioning)
As web 2.0 continued its high-speed growth, even with Memcached caching, MySQL master-slave replication, and read/write splitting, the write pressure on the MySQL master began hitting bottlenecks, while data volumes continued surging. Because MyISAM uses table-level locking, serious locking problems arose under high concurrency, and many high-concurrency MySQL applications began using the InnoDB engine to replace MyISAM. At the same time, sharding (splitting tables and databases) became popular to relieve write pressure and data growth scaling issues. Sharding became a hot topic—both in interviews and in industry technical discussions. It was also around this time that MySQL introduced its still somewhat unstable table partitioning feature, which brought hope to companies with average technical capabilities. Although MySQL also launched MySQL Cluster, it had almost no successful Internet use cases and its performance could not meet Internet demands, though it did provide very strong high-reliability guarantees.
MySQL’s Scalability Bottleneck
On the Internet, most MySQL instances should be I/O-bound; in fact, if your MySQL is CPU-bound, it likely indicates a performance design problem that needs optimization. MySQL application development under large data volumes and high concurrency environments became increasingly complex and technically challenging. Mastering sharding rules requires experience. Even though technically strong companies like Taobao developed transparent middleware layers to shield developers from complexity, they could not avoid the complexity of the overall architecture. Sharded sub-databases eventually face expansion challenges at later stages. Requirement changes might also demand a new sharding approach. MySQL databases also often store large text fields, causing tables to become enormous and making database recovery very slow and difficult to perform quickly. For example, 10 million text fields of 4KB each approaches 40GB in size; if this data could be offloaded from MySQL, MySQL itself would become much smaller. Relational databases are powerful, but they cannot handle all application scenarios perfectly. MySQL’s poor scalability (requiring complex techniques to achieve), high I/O pressure under large data loads, and difficulty in changing table structures are precisely the problems developers currently face when using MySQL. Advantages of NoSQL Easy to Scale There are many types of NoSQL databases, but a common characteristic is the removal of the relational features found in relational databases. With no relationships between data, scaling becomes extremely easy, inherently bringing scalability to the architecture level. Large Data Volumes, High Performance NoSQL databases all exhibit very high read/write performance, performing equally well especially under large data volumes. This is due to their non-relational nature and simple database structure. Generally, MySQL uses Query Cache, where every table update invalidates the cache—this is a coarse-grained Cache, and for web 2.0’s frequently interactive applications, Cache performance is low. NoSQL’s Cache is record-level, a fine-grained Cache, which makes NoSQL much higher-performing in this regard. Flexible Data Models NoSQL does not require predefining fields for stored data; custom data formats can be stored at any time.
In relational databases, adding or deleting fields is a very troublesome affair. For tables with extremely large data volumes, adding a field is nothing short of a nightmare. This is especially evident in the web 2.0 era of massive data. High Availability NoSQL can easily achieve high-availability architectures without significantly affecting performance. For example, Cassandra and HBase models can achieve high availability through replication models. Summary The emergence of NoSQL databases compensates for certain shortcomings of relational databases (like MySQL) and can greatly save development and maintenance costs in some aspects. MySQL and NoSQL each have their own characteristics and suitable application scenarios; the tight integration of the two will bring new thinking to database development for web 2.0. Let relational databases focus on relationships, and NoSQL focus on storage.
References NoSQL: http://nosql-database.org/ NoSQL
Introduction on Wikipedia: http://en.wikipedia.org/wiki/No
As web 2.0 continued its high-speed growth, even with Memcached caching, MySQL master-slave replication, and read/write splitting, the write pressure on the MySQL master began hitting bottlenecks, while data volumes continued surging. Because MyISAM uses table-level locking, serious locking problems arose under high concurrency, and many high-concurrency MySQL applications began using the InnoDB engine to replace MyISAM. At the same time, sharding (splitting tables and databases) became popular to relieve write pressure and data growth scaling issues. Sharding became a hot topic—both in interviews and in industry technical discussions. It was also around this time that MySQL introduced its still somewhat unstable table partitioning feature, which brought hope to companies with average technical capabilities. Although MySQL also launched MySQL Cluster, it had almost no successful Internet use cases and its performance could not meet Internet demands, though it did provide very strong high-reliability guarantees.
On the Internet, most MySQL instances should be I/O-bound; in fact, if your MySQL is CPU-bound, it likely indicates a performance design problem that needs optimization. MySQL application development under large data volumes and high concurrency environments became increasingly complex and technically challenging. Mastering sharding rules requires experience. Even though technically strong companies like Taobao developed transparent middleware layers to shield developers from complexity, they could not avoid the complexity of the overall architecture. Sharded sub-databases eventually face expansion challenges at later stages. Requirement changes might also demand a new sharding approach. MySQL databases also often store large text fields, causing tables to become enormous and making database recovery very slow and difficult to perform quickly. For example, 10 million text fields of 4KB each approaches 40GB in size; if this data could be offloaded from MySQL, MySQL itself would become much smaller. Relational databases are powerful, but they cannot handle all application scenarios perfectly. MySQL’s poor scalability (requiring complex techniques to achieve), high I/O pressure under large data loads, and difficulty in changing table structures are precisely the problems developers currently face when using MySQL. Advantages of NoSQL Easy to Scale There are many types of NoSQL databases, but a common characteristic is the removal of the relational features found in relational databases. With no relationships between data, scaling becomes extremely easy, inherently bringing scalability to the architecture level. Large Data Volumes, High Performance NoSQL databases all exhibit very high read/write performance, performing equally well especially under large data volumes. This is due to their non-relational nature and simple database structure. Generally, MySQL uses Query Cache, where every table update invalidates the cache—this is a coarse-grained Cache, and for web 2.0’s frequently interactive applications, Cache performance is low. NoSQL’s Cache is record-level, a fine-grained Cache, which makes NoSQL much higher-performing in this regard. Flexible Data Models NoSQL does not require predefining fields for stored data; custom data formats can be stored at any time.
In relational databases, adding or deleting fields is a very troublesome affair. For tables with extremely large data volumes, adding a field is nothing short of a nightmare. This is especially evident in the web 2.0 era of massive data. High Availability NoSQL can easily achieve high-availability architectures without significantly affecting performance. For example, Cassandra and HBase models can achieve high availability through replication models. Summary The emergence of NoSQL databases compensates for certain shortcomings of relational databases (like MySQL) and can greatly save development and maintenance costs in some aspects. MySQL and NoSQL each have their own characteristics and suitable application scenarios; the tight integration of the two will bring new thinking to database development for web 2.0. Let relational databases focus on relationships, and NoSQL focus on storage.
Introduction on Wikipedia: http://en.wikipedia.org/wiki/No