{"action":"closed","issue":{"url":"https://api.github.com/repos/rubygems/rubygems/issues/1415","repository_url":"https://api.github.com/repos/rubygems/rubygems","labels_url":"https://api.github.com/repos/rubygems/rubygems/issues/1415/labels{/name}","comments_url":"https://api.github.com/repos/rubygems/rubygems/issues/1415/comments","events_url":"https://api.github.com/repos/rubygems/rubygems/issues/1415/events","html_url":"https://github.com/rubygems/rubygems/issues/1415","id":122359779,"number":1415,"title":"Performance issues in DependencyInstaller?","user":{"login":"lamont-granquist","id":454857,"avatar_url":"https://avatars.githubusercontent.com/u/454857?v=3","gravatar_id":"","url":"https://api.github.com/users/lamont-granquist","html_url":"https://github.com/lamont-granquist","followers_url":"https://api.github.com/users/lamont-granquist/followers","following_url":"https://api.github.com/users/lamont-granquist/following{/other_user}","gists_url":"https://api.github.com/users/lamont-granquist/gists{/gist_id}","starred_url":"https://api.github.com/users/lamont-granquist/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lamont-granquist/subscriptions","organizations_url":"https://api.github.com/users/lamont-granquist/orgs","repos_url":"https://api.github.com/users/lamont-granquist/repos","events_url":"https://api.github.com/users/lamont-granquist/events{/privacy}","received_events_url":"https://api.github.com/users/lamont-granquist/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rubygems/rubygems/labels/status:%20triage","name":"status: triage","color":"fef2c0"}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2015-12-15T20:30:25Z","updated_at":"2016-08-30T17:20:00Z","closed_at":"2016-08-30T17:20:00Z","body":"To first explain what I'm trying to do (since I may simply be barking up entirely the wrong tree).  I'm trying to improve how Chef uses the rubygem API in order to install gems.  In order to do this we must be able to fill in what we call a \"candidate_version\" which is the available version on the remote source of the gem the user wants to install so that Chef can do its own idempotency checks.\r\n\r\nWhat I've determined is that the existing Chef code winds up doing a `GET /latest_specs.4.8.gz` and then inflates a ruby cache of every single one of those specs which is terrible for RAM performance.\r\n\r\n![Chef Memory Usage](https://cloud.githubusercontent.com/assets/454857/9555300/f6b3a578-4d80-11e5-99bd-54325192fa2f.png)\r\n\r\nThat graph is of the live slots (each point taken after instrumenting the code to force a GC).  The big red spike is an unrelated performance issue that we had in our remote_directory provider which was fixed in chef/chef#3837, so please ignore that.  The sharp rise to 400,000 live slots is caused by the first time we try to install a gem and we grab latest_specs.4.8.gz and inflate the in-memory cache.  That's the problem that I'm trying to address.\r\n\r\nI'd like to instead hit the newer APIs like 'gem install' uses that starts with API requests like `GET /api/v1/dependencies?gems=rspec` which does not slurp down the latest specs of all the gems as the first call.  There is a tradeoff here, obviously, where we'd add more round trips to the rubygems API and slow things down a bit, but should be able to significantly reduce our memory footprint (and not have our memory footprint grown without bounds as rubygems publishes ever more gems).\r\n\r\nThe 'slurpy' code that we're using does the following query to the DependencyInstaller:\r\n\r\n```\r\n                available_gems = dependency_installer.find_gems_with_sources(gem_dependency)\r\n                available_gems.pick_best!\r\n                best_gem = available_gems.set.first\r\n```\r\n\r\nThe code that I've been playing with from looking how 'gem install' works looks like this instead:\r\n\r\n```\r\n                begin\r\n                  rs = dependency_installer.resolve_dependencies gem_dependency.name, gem_dependency.requirement\r\n                  rs.specs.first\r\n                rescue Gem::UnsatisfiableDependencyError\r\n                  nil\r\n                end\r\n```\r\n\r\nOn the wire, this appears to do what I want.\r\n\r\nHowever, the performance is much worse than I expected because:\r\n\r\n1.  there seems to be no caching at all behind resolve_dependencies\r\n2.  the performance when pointing at 'http://production.cf.rubygems.org' is both extremely bad and not cached at all.\r\n\r\nThe 2nd point is important to me, because we may have users that have created internal rubygems mirrors naively by duplicating the s3 buckets and which do not offer the newer API endpoints.\r\n\r\nSome benchmarks of the \"slurpy\" API that pulls down latest.specs.4.8.gz:\r\n\r\nfor a gem that exists on rubygems.org:\r\n\r\n```\r\n       user     system      total        real\r\nfirst:  0.420000   0.070000   0.490000 (  1.556174)\r\nsecond:  0.000000   0.000000   0.000000 (  0.000373)\r\nthird:  0.000000   0.000000   0.000000 (  0.000308)\r\n```\r\n\r\nfor a gem that does not exist on rubygems.org:\r\n\r\n```\r\n       user     system      total        real\r\nfirst:  0.270000   0.010000   0.280000 (  0.958228)\r\nsecond:  0.000000   0.000000   0.000000 (  0.000174)\r\nthird:  0.000000   0.000000   0.000000 (  0.000115)\r\n```\r\n\r\nfor a gem that exists accessed through http://production.cf.rubygems.org:\r\n\r\n```\r\n       user     system      total        real\r\nfirst:  0.240000   0.020000   0.260000 (  0.422814)\r\nsecond:  0.000000   0.000000   0.000000 (  0.000510)\r\nthird:  0.010000   0.000000   0.010000 (  0.000395)\r\n```\r\n\r\nThe aggressive caching here is pretty obvious.  Now looking at `#resolve_dependencies` hitting the newer API endpoints.  In the case of a gem that exists on rubygems.org:\r\n\r\n```\r\n       user     system      total        real\r\nfirst:  0.040000   0.010000   0.050000 (  0.944599)\r\nsecond:  0.010000   0.000000   0.010000 (  0.795025)\r\nthird:  0.020000   0.000000   0.020000 (  0.821814)\r\n```\r\n\r\nFaster at first, but no caching.\r\n\r\nFor a gem that does not exist:\r\n\r\n```\r\n       user     system      total        real\r\nfirst:  0.030000   0.010000   0.040000 (  0.470843)\r\nsecond:  0.000000   0.000000   0.000000 (  0.410874)\r\nthird:  0.000000   0.000000   0.000000 (  0.399161)\r\n```\r\n\r\nAgain, no caching.\r\n\r\nFor a gem accessed through production.cf.rubygems.org:\r\n\r\n```\r\n       user     system      total        real\r\nfirst:  8.510000   0.110000   8.620000 (  9.356115)\r\nsecond:  8.490000   0.040000   8.530000 (  8.653654)\r\nthird:  8.390000   0.030000   8.420000 (  8.525400)\r\n```\r\n\r\nThe performance here is 10x to 20x slower, and there's no caching.  This is really the blocker for me, since a perf hit of 10 seconds times every gem resource in a chef-client run for users who have internal rubygems mirrors is very poor.\r\n\r\nSo to start off with, am I going about this roughly correctly, or is there just some different Gem API that I should be using?\r\n\r\nThen, the second question would be if there are legitimate performance bugs in rubygems here or not?\r\n\r\nAnd finally the work that I'm doing inside chef is at https://github.com/chef/chef/pull/3856 -- although the rubygems provider in Chef is definitely top 10 of the worst example of code inside of the Chef codebase that I can think of so you might not want to look at that."}}
{"actor":{"display_login":"lamont-granquist"}}