{"action":"created","issue":{"url":"https://api.github.com/repos/attic-labs/noms/issues/2627","repository_url":"https://api.github.com/repos/attic-labs/noms","labels_url":"https://api.github.com/repos/attic-labs/noms/issues/2627/labels{/name}","comments_url":"https://api.github.com/repos/attic-labs/noms/issues/2627/comments","events_url":"https://api.github.com/repos/attic-labs/noms/issues/2627/events","html_url":"https://github.com/attic-labs/noms/issues/2627","id":179621305,"number":2627,"title":"Proposal: `noms pump`","user":{"login":"aboodman","id":80388,"avatar_url":"https://avatars.githubusercontent.com/u/80388?v=3","gravatar_id":"","url":"https://api.github.com/users/aboodman","html_url":"https://github.com/aboodman","followers_url":"https://api.github.com/users/aboodman/followers","following_url":"https://api.github.com/users/aboodman/following{/other_user}","gists_url":"https://api.github.com/users/aboodman/gists{/gist_id}","starred_url":"https://api.github.com/users/aboodman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aboodman/subscriptions","organizations_url":"https://api.github.com/users/aboodman/orgs","repos_url":"https://api.github.com/users/aboodman/repos","events_url":"https://api.github.com/users/aboodman/events{/privacy}","received_events_url":"https://api.github.com/users/aboodman/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-09-27T21:38:35Z","updated_at":"2016-09-30T19:41:37Z","closed_at":null,"body":"[wip - this isn't ready for serious feedback yet]\r\n\r\n# Problem 1\r\n\r\nIt's really common when working with Noms to want to setup \"pipelines\" of programs that feed one into the next.\r\n\r\nFor example we currently have running on demo.noms.io many instances of an adhoc 2-stage pipeline in which the first stage sucks down a particular csv blob, and the second stage parses that csv. In a few cases, we also have a third stage where we index that csv using `nomdex`.\r\n\r\nRight now these pipelines are totally adhoc. We arrange them in `crontab` such that the first stage runs on the 0th minute every hour, the second stage on the 5th minute, and so-on.\r\n\r\nIt would be nice to have a little bit more structure - to be able to declare \"continuously replicate dataset x to dataset y via job z\", and have some infrastructure that makes sure that happens.\r\n\r\n# Problem 2\r\n\r\nRelatedly, it would be nice if these transforms didn't have to redo all the work everytime -- if they could work incrementally.\r\n\r\n# Proposal\r\n\r\nIntroduce a new subcommand to noms:\r\n\r\n```\r\nnoms pump [--incremental] <db> <source> <dest> <via>\r\n\r\nexample:\r\nnoms pump http://localhost:8000 csv/raw csv/parsed $HOME/bin/csv/read\r\n```\r\n\r\nThe job of `pump` is to ensure that for each commit in a *source dataset* there is a corresponding commit in a *destination dataset* transformed by a particular *job*.\r\n\r\n**Pump is stateless** in that each time it runs it looks at its inputs, the source dataset, and the destination dataset, and does whatever work it needs to in order to achieve its invariants, then exits. Thus it can easily be run as part of cron.\r\n\r\n## Incremental Mode\r\n\r\nTODO\r\n\r\n# Implementation Handwavery\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/attic-labs/noms/issues/comments/250834605","html_url":"https://github.com/attic-labs/noms/issues/2627#issuecomment-250834605","issue_url":"https://api.github.com/repos/attic-labs/noms/issues/2627","id":250834605,"user":{"login":"aboodman","id":80388,"avatar_url":"https://avatars.githubusercontent.com/u/80388?v=3","gravatar_id":"","url":"https://api.github.com/users/aboodman","html_url":"https://github.com/aboodman","followers_url":"https://api.github.com/users/aboodman/followers","following_url":"https://api.github.com/users/aboodman/following{/other_user}","gists_url":"https://api.github.com/users/aboodman/gists{/gist_id}","starred_url":"https://api.github.com/users/aboodman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aboodman/subscriptions","organizations_url":"https://api.github.com/users/aboodman/orgs","repos_url":"https://api.github.com/users/aboodman/repos","events_url":"https://api.github.com/users/aboodman/events{/privacy}","received_events_url":"https://api.github.com/users/aboodman/received_events","type":"User","site_admin":false},"created_at":"2016-09-30T19:41:37Z","updated_at":"2016-09-30T19:41:37Z","body":"I'm not sure, we've gone around on that a few times. STDIN/STDOUT is always a common instinct people have because unix. But at least the way we end up doing it in practice is that we tend to use datasets as temporary storage between pipeline stages.\r\n\r\nSo for example, what we are currently doing for the data on demo.noms.io/cli-tour is something like:\r\n\r\n```\r\n0  *   *   *   *   url_fetch http://data.com/blah.csv http://localhost:8000::blah/raw\r\n5  *   *   *   *   csv-import --dest-type='map:0' http://localhost:8000::blah/raw http://localhost:8000::blah/parsed\r\n10  *   *   *   *   nomdex up --in-path=http://localhost:8000::blah/parsed --out-ds=blah/by-foo --by=.foo\r\n```\r\n\r\nBut this is all very squishy. It is possible for there to be races or for transformations that should have run to get dropped. But I think the basic idea of using datasets as the links in the chain makes more sense than stdin/out for a variety of reasons (mainly the data can be huge and it's also not really line-oriented)."}}
{"actor":{"display_login":"aboodman"}}