app/jobs/github_repo_scraper.rb
# Retrieves the commits for each Repository
#
# NOTE: you can use all options together, but whichever one ends first
#       will be the one that stops the scraper
#
# Options
# repositories: repos to be scraped for data
# page_limit: maximum number of pages to iterate
# user_limit: max number of users to add
Retrieves the commits for each Repository
NOTE: you can use all options together, but whichever one ends first
will be the one that stops the scraper

Options
repositories: repos to be scraped for data
page_limit: maximum number of pages to iterate
user_limit: max number of users to add
Retrieves the commits for each Repository NOTE : you can use all options together but whichever one ends first will be the one that stops the scraper Options repositories : repos to be scraped for data page_limit : maximum number of pages to iterate user_limit : max number of users to add
def commits(scrape_limit_opts={}, get_repo_meta=false)
      handle_scrape_limits(scrape_limit_opts)
      catch :scrape_limit_reached do
        @repositories.each do |repo|
          break unless get_repo_doc(repo, "/commits")
          update_repo_meta if get_repo_meta
          puts "Scraping #{repo.name} commits"
          catch :recent_commits_finished do
            traverse_commit_pagination
          end
        end
      end
    end
def commits ( scrape_limit_opts = { } , get_repo_meta = false ) handle_scrape_limits ( scrape_limit_opts ) catch :scrape_limit_reached do @repositories . each do | repo | break unless get_repo_doc ( repo , "/commits" ) update_repo_meta if get_repo_meta puts "Scraping #{repo.name} commits" catch :recent_commits_finished do traverse_commit_pagination end end end end
Retrieves the commits for each Repository
NOTE: you can use all options together, but whichever one ends first
will be the one that stops the scraper
Retrieves the commits for each Repository NOTE : you can use all options together but whichever one ends first will be the one that stops the scraper
scrape_limit_opts
<not_specific>
get_repo_meta
<not_specific>