Mass small-file storage has always been a classic use case. Below is a reposted article that analyzes the pros and cons of three solutions for massive small-file storage: Nginx+NFS, Redis, and Distributed File SystemMoosefs.
There are currently 30 million data records, each one very small. Viewed as key-value pairs, the key is a 32-character MD5 string and the average value size is around 100 bytes.
We need to cache this data so that it can still respond quickly under high concurrency.
Since this data has virtually no distinction between hot and cold data, all of it needs to be placed in the cache.
1. Generate static files directly and leverage Nginx’s efficiency with static files for static caching.
- Server hardware resources were limited at the time, so this approach was adopted and has been in use ever since.
- Servers share files via NFS.
- Too many small files make management inconvenient.
- NFS is not easy to operate, maintain, or scale.
- File content is very small (around 100 bytes). 30 million records amount to roughly 2.5GB.
- However, file storage is related to the cluster size of the disk partition. Here the disk partition cluster size is 8K, so even though the file content is only 100 bytes, the actual disk space occupied by a single file is 8K.
- Therefore, the actual disk space occupied by 30 million files is approximately 200GB (a serious waste of space).
2. Redis (V2.2.11) [KV Database]
- Colleagues said that enabling VM causes severe performance degradation, so VM is basically ignored and all data is kept in memory.
- Key is a 32-character MD5 string.
- Test data: 100K records occupy about 20MB of memory.
- Test data: 5 million records occupy about 1GB of memory, and the persistent RDB data file is about 350MB.
- Estimated for 30 million records: 6GB of memory, persistent RDB data file about 2GB (compressed?).
- Because Redis memory usage doubles during persistence, and accounting for data growth, one machine with 20G of RAM should be sufficient (disaster recovery scenarios aside).
- Redis is extremely fast. If your hardware conditions are fine, this is basically the best choice.
3. Moosefs (MFS) [Distributed File Storage System]
- MFS supports mounting locally via NFS for direct operation (if using MFS, the current architecture would barely need changes)
- The minimum setup requires one Master Server and one Chunk Server
- All file and directory indexes must be loaded into the Master Server’s memory, so there are specific memory requirements for the Master Server
- Writing 300K files, the write speed started dropping significantly around the 200K mark (on my 5400 RPM laptop hard drive)
- For 300K files and 40K directories, the Master Server used approximately 120MB of memory
- The file storage method appears to store files individually like a normal filesystem (not certain), and 300K files took up roughly 2.4G of disk space (also at 8K per file).
- Still produces a massive number of small files, making maintenance and migration inconvenient (maybe I misconfigured it?).
- Both memory and disk usage are relatively high, and the performance is not outstanding compared to others.
- So, this option is basically ruled out.
Copyright Notice:
Author: QLeelulu
Source: http://QLeelulu.cnblogs.com/
This article’s copyright is jointly owned by the author and Blog Park. Reprinting is welcome, but this notice must be retained without the author’s consent, and a clear link to the original source must be provided in a prominent position on the article page. Otherwise, the right to pursue legal liability is reserved.