{"action":"created","issue":{"url":"https://api.github.com/repos/conda-forge/conda-forge-webservices/issues/63","repository_url":"https://api.github.com/repos/conda-forge/conda-forge-webservices","labels_url":"https://api.github.com/repos/conda-forge/conda-forge-webservices/issues/63/labels{/name}","comments_url":"https://api.github.com/repos/conda-forge/conda-forge-webservices/issues/63/comments","events_url":"https://api.github.com/repos/conda-forge/conda-forge-webservices/issues/63/events","html_url":"https://github.com/conda-forge/conda-forge-webservices/issues/63","id":176558235,"number":63,"title":"Team update/feedstocks update webservice","user":{"login":"jakirkham","id":3019665,"avatar_url":"https://avatars.githubusercontent.com/u/3019665?v=3","gravatar_id":"","url":"https://api.github.com/users/jakirkham","html_url":"https://github.com/jakirkham","followers_url":"https://api.github.com/users/jakirkham/followers","following_url":"https://api.github.com/users/jakirkham/following{/other_user}","gists_url":"https://api.github.com/users/jakirkham/gists{/gist_id}","starred_url":"https://api.github.com/users/jakirkham/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jakirkham/subscriptions","organizations_url":"https://api.github.com/users/jakirkham/orgs","repos_url":"https://api.github.com/users/jakirkham/repos","events_url":"https://api.github.com/users/jakirkham/events{/privacy}","received_events_url":"https://api.github.com/users/jakirkham/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-09-13T06:14:04Z","updated_at":"2016-09-26T00:01:10Z","closed_at":null,"body":"While we have done a good job at limited the effect of these scripts, there are still problems. If they haven't surfaced or resurfaced yet, they will soon. So we should start planning to handle them. Here are some issues that come to mind.\r\n\r\n1. Scales poorly\r\n  1. GitHub Rate limit issues.\r\n    1. https://github.com/conda-forge/conda-forge.github.io/issues/88\r\n    2. https://github.com/conda-forge/conda-forge.github.io/issues/172\r\n2. Fragile\r\n  1. Feedstock level issues break it.\r\n    1. https://github.com/conda-forge/conda-forge.github.io/issues/86\r\n    2. https://github.com/conda-forge/conda-forge.github.io/issues/139\r\n    3. https://github.com/conda-forge/conda-forge.github.io/issues/228\r\n3. Inefficient\r\n  1. Disproportional time spent on unchanged feedstocks. ( https://github.com/conda-forge/conda-forge.github.io/issues/88#issuecomment-217548992 )\r\n  2. Run time increased from 2min to 12min in last 6mos.\r\n4. Lagged response\r\n  1. User are frequently surprised they aren't added right away.\r\n    1.1. Recent example ( https://github.com/conda-forge/traitlets-feedstock/pull/5#issuecomment-245930774 ), but there are many.\r\n5. Duplicated functionality.\r\n  1. staged-recipes - https://github.com/conda-forge/staged-recipes/pull/733\r\n  2. webpage repo - https://github.com/conda-forge/conda-forge.github.io/pull/15\r\n6. Poor separation of concerns\r\n  1. Run in CI on webpage repo.\r\n  2. Run on PR merges that consist of...\r\n    1. Minutes\r\n    2. Pinnings\r\n    3. Assorted helper scripts\r\n7. Lacking test coverage\r\n\r\nLooking at these issues they are all directly or indirectly because we have been using a global update script. As a result, we are spending lots of time doing work that is not particularly useful and paying a high price for it. However, when going back through the feedstocks repo (which is updated roughly once daily), it looks like ~20 feedstocks are added/changed daily. So there is no reason it should be so difficult to keep up with this especially as often as people want us to for cheap. What can we do?\r\n\r\n---\r\n\r\nWe need to switch from global update to asynchronous local updates. This model has been highly successful for us with things like the linter. To solve this particular problem, we would need a webservice that listens to [`PushEvents`]( https://developer.github.com/v3/activity/events/types/#pushevent ) on each feedstock. When there is a push event, we do the following.\r\n\r\n1. New repos: `conda-smithy` adds webhook for this service (just like the linter).\r\n2. Update the sha of the feedstock in the feedstocks repo.\r\n3. Get the list of GitHub teams.\r\n4. Add (if not present)/Get the `all-members` and `<feedstock>` teams.\r\n5. Get the member lists for both teams.\r\n6. Compare our maintainers list to both team list.\r\n7. Add any members that are not present to both teams.\r\n\r\nAll of this is actually pretty fast to run through.\r\n\r\nBy working only with feedstocks that see updates, we no longer are scaling by number of feedstocks, but are scaling by number of active feedstocks, which is much better behaved. Even if there is a lot of activity our GitHub API usage for each feedstock is quite low (well below the rate limit). As far as the feedstocks repo, we are only updating one feedstock at a time. So this significantly cuts down time spent cloning or trying to clone feedstocks.\r\n\r\nAs pushes are localized to each feedstock, issues with one feedstock and the service don't effect another feedstock's ability to use the service. This makes it more resilient and easier to pinpoint where issues are. It also avoids having team updates blocked for the org by issues with one feedstock. Further if there are issues with a feedstock, they are easy to localize and start working on immediately. GitHub API limits will not play a significant role in blocking progress testing fixes.\r\n\r\nFurther no time is spent on feedstocks that are not updated. Only feedstocks that are being getting pushes (directly or via merges) are analyzed for changes. This already makes things massively more efficient. We haven't even begun to discuss optimizations. For instance, we could further cutdown our GitHub API rate limit by looking at the maintainer's list before and after each push. Still I think it will be awhile before we need to give that serious thought.\r\n\r\nAs this works on each push, there is no lag in the process. Teams are created/updated immediately. This improves the user experience by giving them immediate feedback. Also it helps more quickly find issues with the service. If someone is not added immediately, there is a service problem that needs to be investigated. It also cuts down on notifications of false positives.\r\n\r\nCode snippets from the webpage repo/staged-recipes can be used to fill this out. Also `conda-smithy` would learn how to add the webhook for this service. Once all of this is done, neither place needs to know about how to handle team creation or updates. Further there is no special case for creating a new feedstock vs. updating an existing one as all pushes trigger it just the same.\r\n\r\nAdditionally by moving the code to manage these updates here, not only will they not need to live at the webpage repo, but they won't need to be triggered by the webpage repo. This will avoid many spurious failures at the webpage repo due to GitHub API rate limits or other issues with the update scripts that are largely orthogonal to the PRs being added there. The freed up CI resource can then be used for testing other scripts or another purpose. \r\n\r\nFinally as doing this will give us a simple service that runs here, we will be in much better shape for testing. The test infrastructure here is well suited for working with the GitHub API. There may be some tricks required in order to get pushes or mock pushes. Still we already have a good system for testing different branches and reporting what was found. It may take a bit of work to get good coverage, but we should be able to get some coverage to start.\r\n\r\nIn short, setting up a webservice to manage team and feedstock updates is a clear winner because it moves from a global update model to a local update model."},"comment":{"url":"https://api.github.com/repos/conda-forge/conda-forge-webservices/issues/comments/249455216","html_url":"https://github.com/conda-forge/conda-forge-webservices/issues/63#issuecomment-249455216","issue_url":"https://api.github.com/repos/conda-forge/conda-forge-webservices/issues/63","id":249455216,"user":{"login":"jakirkham","id":3019665,"avatar_url":"https://avatars.githubusercontent.com/u/3019665?v=3","gravatar_id":"","url":"https://api.github.com/users/jakirkham","html_url":"https://github.com/jakirkham","followers_url":"https://api.github.com/users/jakirkham/followers","following_url":"https://api.github.com/users/jakirkham/following{/other_user}","gists_url":"https://api.github.com/users/jakirkham/gists{/gist_id}","starred_url":"https://api.github.com/users/jakirkham/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jakirkham/subscriptions","organizations_url":"https://api.github.com/users/jakirkham/orgs","repos_url":"https://api.github.com/users/jakirkham/repos","events_url":"https://api.github.com/users/jakirkham/events{/privacy}","received_events_url":"https://api.github.com/users/jakirkham/received_events","type":"User","site_admin":false},"created_at":"2016-09-26T00:01:10Z","updated_at":"2016-09-26T00:01:10Z","body":"Also cc-ing @ericdill given our bonding over the problems the current strategy will run into."}}
{"actor":{"display_login":"jakirkham"}}