{"action":"created","issue":{"url":"https://api.github.com/repos/pachyderm/pachyderm/issues/2882","repository_url":"https://api.github.com/repos/pachyderm/pachyderm","labels_url":"https://api.github.com/repos/pachyderm/pachyderm/issues/2882/labels{/name}","comments_url":"https://api.github.com/repos/pachyderm/pachyderm/issues/2882/comments","events_url":"https://api.github.com/repos/pachyderm/pachyderm/issues/2882/events","html_url":"https://github.com/pachyderm/pachyderm/issues/2882","id":319320984,"node_id":"MDU6SXNzdWUzMTkzMjA5ODQ=","number":2882,"title":"Allow input specs' branch/repo names to contain glob patterns","user":{"login":"gabrielgrant","id":493968,"node_id":"MDQ6VXNlcjQ5Mzk2OA==","avatar_url":"https://avatars2.githubusercontent.com/u/493968?v=4","gravatar_id":"","url":"https://api.github.com/users/gabrielgrant","html_url":"https://github.com/gabrielgrant","followers_url":"https://api.github.com/users/gabrielgrant/followers","following_url":"https://api.github.com/users/gabrielgrant/following{/other_user}","gists_url":"https://api.github.com/users/gabrielgrant/gists{/gist_id}","starred_url":"https://api.github.com/users/gabrielgrant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gabrielgrant/subscriptions","organizations_url":"https://api.github.com/users/gabrielgrant/orgs","repos_url":"https://api.github.com/users/gabrielgrant/repos","events_url":"https://api.github.com/users/gabrielgrant/events{/privacy}","received_events_url":"https://api.github.com/users/gabrielgrant/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-05-01T20:41:44Z","updated_at":"2019-06-11T21:30:09Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"There have been a few cases recently where it would make sense to have a dynamically changing set of inputs into a pipeline\r\n\r\nExamples:\r\n- DIUX are currently spinning up a new instance of their scoring pipeline for each user submission. This could probably be simplified as a single pipeline that takes in a union of all `*_inference` repos' master branches\r\n- Ingestion from multiple spout processes: each process puts commits into it's own branch on a repo, and one pipeline unions those branches into a single merged commit for actual processing\r\n- Doing post-processing on all stats commits\r\n- Reacting to creation/changes of pipelines by watching all branches on spec repo\r\n\r\nA couple possible solutions:\r\n\r\n1. ~A `branches` branch, that is automatically updated with, say, a file per branch containing a pointer to the branch's current HEAD commit. A pipeline using this as a trigger could update a pipeline to add inputs. This is probably pretty straightforward, and potentially useful in other places, but still pretty hacky for this use-case (details: #2923)~ Edit: as per @jdoliner 's comment below, the second solution is preferred.\r\n2. Probably a more straightforward solution is to allow pattern-based matching of branch and/or repo names in atom inputs. I am imagining that specifying an atom input in a union or cross with a pattern would effectively expand to be the list of all inputs matching that pattern. It's not 100% clear, however, what should happen if a pattern is specified for a bare atom input -- this should probably be disallowed.  I understand the internals of implementing this might be a bit annoying: probably entails iterating through all inputs to build a list* of patterns to match against any time a new branch or repo is created, and updating the corresponding pipeline infos (my inclination is that a pipeline info should contain the currently-matched set of inputs, rather than the raw pattern -- that can be obtained via [`ExtractPipeline`](https://github.com/pachyderm/pachyderm/blob/master/src/client/admin/admin.proto#L52) )\r\n\r\n*: If raw pipeline specs are only stored in the spec repo (ie object store), it may make sense to cache this list in etcd"},"comment":{"url":"https://api.github.com/repos/pachyderm/pachyderm/issues/comments/501030563","html_url":"https://github.com/pachyderm/pachyderm/issues/2882#issuecomment-501030563","issue_url":"https://api.github.com/repos/pachyderm/pachyderm/issues/2882","id":501030563,"node_id":"MDEyOklzc3VlQ29tbWVudDUwMTAzMDU2Mw==","user":{"login":"gabrielgrant","id":493968,"node_id":"MDQ6VXNlcjQ5Mzk2OA==","avatar_url":"https://avatars2.githubusercontent.com/u/493968?v=4","gravatar_id":"","url":"https://api.github.com/users/gabrielgrant","html_url":"https://github.com/gabrielgrant","followers_url":"https://api.github.com/users/gabrielgrant/followers","following_url":"https://api.github.com/users/gabrielgrant/following{/other_user}","gists_url":"https://api.github.com/users/gabrielgrant/gists{/gist_id}","starred_url":"https://api.github.com/users/gabrielgrant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gabrielgrant/subscriptions","organizations_url":"https://api.github.com/users/gabrielgrant/orgs","repos_url":"https://api.github.com/users/gabrielgrant/repos","events_url":"https://api.github.com/users/gabrielgrant/events{/privacy}","received_events_url":"https://api.github.com/users/gabrielgrant/received_events","type":"User","site_admin":false},"created_at":"2019-06-11T21:30:09Z","updated_at":"2019-06-11T21:30:09Z","author_association":"CONTRIBUTOR","body":"Expanding on how this would be shown via inspect-pipeline/the dash: my thinking is that we'd have a pipeline spec that the user submits with the wildcard, but the info that comes back from inspect-pipeline is a snapshot of the current DAG structure as matched at the time of the request. So, in the dash, users would see the current DAG structure from the overview, but then delving into any single pipeline would show the list of currently matched inputs, but also the fact that it's a wildcard"}}
{"actor":{"display_login":"gabrielgrant"}}