title: Getting listing of all repositories on Github with tags listing for a specific 'topic'
body: <p>I am working on a simple package manager GUI which uses <a href="http://npmjs.com" rel="nofollow noreferrer">npm</a> to install packages from <a href="http://github.com" rel="nofollow noreferrer">Github</a>.</p>
  
  <p>In order for this to work I need the following information for <strong>all</strong> repositories that have a specific "topic":</p>
  
  <ul>
  <li>Full name of the repository (account/repository)</li>
  <li>Description of the repository.</li>
  <li>The list of "topics"</li>
  <li>The list of tag names on the repository (all tags beginning with a 'v').</li>
  </ul>
  
  <p>My GUI would cache this data per session but potentially a lot of people would be using the GUI on separate computers.</p>
  
  <p>Looking at the <a href="https://developer.github.com/v3/" rel="nofollow noreferrer">Github v3 API documentation</a> I have been able to find the following two commands:</p>
  
  <ul>
  <li><p><a href="https://developer.github.com/v3/search/" rel="nofollow noreferrer">GET /search/repositories?q=topic:my-package-marker</a> (with <code>Accept: application/vnd.github.mercy-preview+json</code> header to include the topics data).</p></li>
  <li><p>For each of the repositories retrieved by the previous command, <a href="https://developer.github.com/v3/git/refs/#get-all-references" rel="nofollow noreferrer">GET /repos/{repositoryName}/git/refs/tags</a></p></li>
  </ul>
  
  <p>I have a few concerns with this approach:</p>
  
  <ol>
  <li><p>The Github documentation says:</p>
  
  <blockquote>
    <p>Find repositories via various criteria. This method returns up to 100 results per page.</p>
  </blockquote>
  
  <p>Which means that the client would need to loop through and download all of the pages each time.</p></li>
  <li><p>The Github documentation also says:</p>
  
  <blockquote>
    <p><strong>Rate limit</strong></p>
    
    <p>The Search API has a custom rate limit. For requests using Basic Authentication, OAuth, or client ID and secret, you can make up to 30 requests per minute. For unauthenticated requests, the rate limit allows you to make up to 10 requests per minute.</p>
  </blockquote>
  
  <p>If there are more than 30 pages of packages then this limit would be exceeded quite quickly. But perhaps even with fewer pages since for each page I would also need to get the list of tags for each of the 100 per-page repositories.</p></li>
  </ol>
  
  <p>Here is pseudo-code for what it seems that I must do to get this data:</p>
  
  <pre><code>dataPages = { };
  
  for each page in GET /search/repositories?q=topic:my-package-marker
                   with "Accept: application/vnd.github.mercy-preview+json"
  {
    dataPage = new DataPage();
  
    for each entry in page
    {
      entry = new Entry();
      entry.FullName = entry.full_name
      entry.Description = entry.description
      entry.Keywords = entry.topics
      entry.Versions = { };
  
      for each tag in GET /repos/{entry.full_name}/git/refs/tags
      {
        if (tag.ref starts with "refs/tags/v")
        {
          entry.Versions.Add(tag.ref substring from character index 11);
        }
      }
  
      dataPage.Add(entry);
    }
  
    dataPages.Add(dataPage);
  }
  </code></pre>
  
  <p><strong>What do I want to do this?</strong></p>
  
  <p>As of npm v5 you can now install packages directly from Git repositories. This is really useful because it means that npm can be used for packages that don't really fit into the node.js/npm ecosystem. npm is a perfect fit for these packages but the npm registry is not a suitable fit for hosting these packages.</p>
  
  <p><strong>What is the best way to achieve this?</strong></p>