Want to host public or private repositories anywhere for free? GitLab.com is a great choice. We’ve already hosted a single-instance GitLab on it, with nearly 20,000 people actively using GitLab to host repositories, and a single server has already hosted over 100,000 repositories.
A Single Server
Previously, GitLab.com was hosted on Amazon, using the highest-configuration instances available on AWS. But as the user base grew, and we could only scale vertically and were CPU-bound, we had to look for an AWS alternative.
100K repositories require several TBs of space, so storage capacity became quite important. Because we use git, we can only choose a single file system, not object storage (like S3). We wanted to easily scale storage, and additionally, thousands of users pushing and pulling their code puts a certain burden on the CPU. Therefore, we needed more CPU cores to alleviate the pressure from high loads.
As it turns out, using our own servers has been the most cost-effective choice by far.
Currently, we have two standalone servers running GitLab.com, one active primary server and one standby. The server specifications are as follows:
-
Server Model: HP DL180 G6 (introduced in 2009)
-
Processor: 2x X5690 (24 cores total)
-
32GB RAM
-
12x 2TB HDDs (two used for the root volume with RAID 1, the other 10 disks use RAID 10 with an ext4 file system)
Actually, we started with only 16 cores but added another 8 to replace the CPU, thereby reducing CPU-bound load.
Failures and Failover
Abandoning AWS means we can no longer use any AWS features. Therefore, to prevent downtime and similar issues, we need to perform failover.
We use DRBD to create a primary server and a secondary server. One server acts as the application server and should be active. If a problem occurs, we instruct DRBD to bring up the other server as a standby.
Our DRBD tooling is already built and made available to our users.
Future Scaling
GitLab.com currently runs well on existing hardware, but it is growing at an unprecedented rate. Scaling the current hardware will be very expensive and won’t be easy to accomplish.
In the future, GitLab.com will once again be hosted on AWS, making horizontal scaling very easy. Furthermore, Amazon just announced EBS volumes over 10TB, which will make our migration easy.
Original: http://www.csdn.net/article/2015-01-06/2823447-the-hardware-that-powers-100k-git-repos
Translation: https://about.gitlab.com/2015/01/03/the-hardware-that-powers-100k-git-repos/