Ceph, as one of the leading representatives of software-defined storage, has seen strong growth momentum in recent years. There are numerous cases of companies testing and deploying Ceph in production environments, although complaints about it have persisted alongside its adoption. This article attempts to compile some use cases the author has come across.
1. Ctrip
Current state of various storage systems used by Ctrip:
- Commercial Storage:
- SAN (HP/HPS), 1+ PB, Databases
- NAS (HW), 800+ TB, File Sharing
- Open-Source Storage
- GlusterFS, 1+ PB, Database Backups
- FastDFS, 1+ PB, Massive Photo Storage
- HDFS, 10+ PB, Big Data
In the near future, as the company’s business grows, Ctrip will need to expand its storage capacity by more than tenfold.
Ctrip’s reasons for choosing Ceph: Low Cost + SDS + Scale-out + Unified Storage + Enterprise Features
Configuration of Ctrip’s current Ceph cluster:
- Ceph Version: 0.94.2, Hammer release
- Object Storage: RGW + Swift API
- SDK: Python/ Java/ C#/ Ruby
- OS: CentOS 6.4
- Hardware: CPU (2 channels & 32 Core), Mem 128GB, Disk (12 * 3TB SATA disk + 2 * 256GB RAID1 SSD), NIC (4 * Gigabit LAN, bonded in 2 pairs)
RGW Usage Architecture:
Ctrip had a requirement for synchronizing data between data centers. After studying CRUSHmap, Radosgw-agent, and Federate gateway (which was unstable, inflexible — only supporting sync between Zones, and hard to scale), they developed their own COS solution. It features stability, flexibility, and scalability:
Next steps:
- Database on Ceph (Dev & QA Farm)
- OpenStack/ Docker Integrate with Ceph
- IT “Dropbox”
Source: Ctrip’s presentation at the 2015/10/18 Shanghai Ceph Day.
Author’s comments:
- Aligns with typical practices of internet companies: careful selection, thorough testing, phased deployment (often starting in dev/test environments), and in-house development when open-source falls short.
- Hopes Ctrip will share more and give back to the community.
2. China Unicom Research Institute
China Unicom Research Institute is using Ceph object and file storage:
The cluster is still relatively small and is mostly in the preparatory stage before large-scale deployment. Their test environment:
Test results:
They found that SSDs provided only average performance improvement:
Source: China Unicom Research Institute’s presentation at the 2015/10/18 Shanghai Ceph Day.
Author’s comments:
- Still in small-scale testing and trial phase
- The testing methods or tuning techniques used may not have been fully optimized, otherwise the performance gains wouldn’t be so minimal
3. PLCloud
Why PLCloud uses Ceph:
- Pure Software
- Open Source, Commercial Support
- Unified Storage: RBD, RGW, CephFS
- Scale Out
- Self Healing
- Replication and Erasure Coding
- Integrate well with OpenStack
PLCloud’s approach: OpenStack + Ceph (RDB, CephFS) + Docker
- All OpenStack storage placed on Ceph
- 18*(5 OSD+1SSD) / CephRBD / CephFS
- 785VM / 4vCPU32GB per VM
- Ubuntu14.04 / Docker1.6.1 / 150+ Containers per VM
- All VM Mount CephFS
- Mount VM Directory as Container’s Data Volume
- Boot 1 VM < 5s
- Boot 1 Container < 1s
- Boot 150+Containers < 120s
-
Ceph Rados Gateway driver for Docker Registry
-
Map RBD device inside DockerContainer
-
CephFS as Data Volume
-
CephFS as NAS Storage
-
Run Ceph in Containers
Use Case: AIMU Streaming Media Service on PLCloud
- Run media web/app/dbvmover OpenStackand CephRBD
- Use CephRGW as media resource storage
- Put video TransportStream/jpg file via c-language programme
- Manage resource via python-swiftclient
- 400+KB per video tsfile
- Reserved video ts/jpg file 7 days or 30 days
- Allow media server temporary access to objects
- Provide media service for Internet and Intranet User
Source: PLCloud’s presentation at the 2015/10/18 SH Ceph Day.
Author’s Comments:
- Bold enough (considering CephFS is still unstable as of now), and cutting-edge (using whatever’s new)
- Did not clearly explain how they support AIMU’s massive streaming data
4. CERN (European Organization for Nuclear Research) Lab (Source)
4.1 Test Environment
Some interns at CERN set up an environment using the NetApp Cinder driver and Ceph to compare performance.
NetApp Environment (adapted for iSCSI driver): Ceph Cluster:
FAS2040 Storage Systems
- Data ONTAP 8
- 52 Disks
Benchmark Environment:
Method: Created volumes of 100G, 200G, and 400G on both storage systems, attached them to three virtual machines respectively, and used hdparm, Flexible I/O Tester, and the dd command as benchmarking tools.
4.2 Test Results
（FIO uses the writeback cache mechanism）
Conclusion:
（1）For reads, Ceph is faster than NetApp; for writes, both are roughly comparable.
（2）Using caching significantly impacts Ceph’s I/O performance. Writeback can greatly improve performance, while writethrough only provides a minor boost.
（3）Applying different striping parameters to individual volumes can enhance their performance. This feature will be implemented in Cinder.
5. LeCloud（Source）
LeEco adopted Ceph RBD as its unified storage. For OpenStack, the Cinder service is backed by Ceph, and Glance also shares the Ceph storage. They also provide S3-compatible object storage, used as the origin for their CDN, storing LeEco’s videos and resources that customers need to distribute. The S3 service is deployed in a geographically distributed manner nationwide, allowing users to upload data from the nearest location, which is then pushed to Beijing. Currently, LeCloud’s OpenStack deployment has reached 900 physical nodes, with object storage data amounting to several petabytes. LeEco believes that “Ceph’s data distribution and performance are quite excellent, and the CRUSH algorithm is its highlight“.
6. UnitedStack （Source）
China-based UnitedStack has arguably made significant contributions to Ceph, including numerous bug fixes and new features, and has also built a very large-scale cloud using Ceph. Their Ceph team’s blog is well worth following. Below is their test report:
IOPS: Throughput
Write Penalty:
Conclusion:
http://www.cnblogs.com/sammyliu/p/4908668.html