<p>Popular version control servers (like github) are likely having an immense amount of traffic and need a scalable &amp; durable data storage. I was wondering how is this implemented in the background. </p>

<p>I have few guesses/assumptions on how it works but I'm not sure if they are 100% accurate:</p>

<ul>
<li>Repositories are probably stored on disk instead of some database solution (because git server is already self sufficient AFAIK)</li>
<li>A single host to serve the entire traffic is probably not enough, so some load balancing is needed</li>
<li>Since multiple servers are needed, each having their own storage, there is no point in keeping all repositories in all servers. (So I would expect each repository to be mapped to a host)</li>
<li>For reliability, probably servers are not running on single hosts but rather on a cluster of replicates that are actually synced (maybe using kubernetes etc) and these are probably backed up periodically along with database backups.</li>
<li>There probably is a main load balancer application that redirects the request to appropriate cluster (so it knows which repository is mapped to which cluster)</li>
</ul>

<p>One other possibility is just storing the entire <code>.git</code> in a database as blob and have a scalable stateless application fetch that <code>.git</code> for each request, do operations, store the result again and the send response however this is probably a really inefficient solution so I thought it is unlikely to be the underlying mechanism. </p>

<p>So my main questions are: </p>

<ul>
<li>Do assumptions above make sense / are they accurate? </li>
<li>How would one implement a load balancer application that all git requests are directed to the appropriate cluster? (eg. would mapping repositories with cluster id&amp;ips, storing this in a database, and putting up a nodejs application that redirects the incoming requests to matching cluster ip work?)</li>
<li>How would one go about implementing a git server that scales in general if above is inaccurate? (in case there are any better approaches)</li>
</ul>

<p>No need to rely on guesses.</p>

<p>For GitHub specifically, the <a href="https://githubengineering.com/" rel="noreferrer">githubengineering blog</a> details what they had to use in order to scale to their current usage level.</p>

<p>Beside upgrading Rails or removing JQuery, on the frontend side, they have:</p>

<ul>
<li><a href="https://github.blog/2018-08-08-glb-director-open-source-load-balancer/" rel="noreferrer">GLB: GitHub’s open source load balancer</a>: At GitHub, we serve tens of thousands of requests every second out of our network edge, operating on <a href="http://githubengineering.com/githubs-metal-cloud/" rel="noreferrer">GitHub’s metal cloud</a>. </li>
<li><a href="https://github.blog/2018-06-20-mysql-high-availability-at-github/" rel="noreferrer">MySQL High Availability at GitHub</a>: GitHub uses MySQL as its main datastore for all things non-git, and its availability is critical to GitHub’s operation. </li>
<li><a href="https://github.blog/2017-10-13-stretching-spokes/" rel="noreferrer">Stretching Spokes</a>: GitHub’s Spokes system stores multiple distributed copies of Git repositories. This article discusses how we got Spokes replication to span widely separated datacenters.</li>
</ul>

<p>Regarding Kubernetes:</p>

<ul>
<li>"<a href="https://github.blog/2017-08-16-kubernetes-at-github/" rel="noreferrer">Kubernetes at GitHub</a>" (2017)</li>
<li>"<a href="https://github.blog/2019-11-21-debugging-network-stalls-on-kubernetes/" rel="noreferrer">Debugging network stalls on Kubernetes </a>" (2019)</li>
</ul>
