{"action":"created","issue":{"url":"https://api.github.com/repos/iterative/dvc/issues/792","repository_url":"https://api.github.com/repos/iterative/dvc","labels_url":"https://api.github.com/repos/iterative/dvc/issues/792/labels{/name}","comments_url":"https://api.github.com/repos/iterative/dvc/issues/792/comments","events_url":"https://api.github.com/repos/iterative/dvc/issues/792/events","html_url":"https://github.com/iterative/dvc/issues/792","id":333444004,"node_id":"MDU6SXNzdWUzMzM0NDQwMDQ=","number":792,"title":"Recommended dvc practices for evolving input data","user":{"login":"villasv","id":6609289,"node_id":"MDQ6VXNlcjY2MDkyODk=","avatar_url":"https://avatars1.githubusercontent.com/u/6609289?v=4","gravatar_id":"","url":"https://api.github.com/users/villasv","html_url":"https://github.com/villasv","followers_url":"https://api.github.com/users/villasv/followers","following_url":"https://api.github.com/users/villasv/following{/other_user}","gists_url":"https://api.github.com/users/villasv/gists{/gist_id}","starred_url":"https://api.github.com/users/villasv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/villasv/subscriptions","organizations_url":"https://api.github.com/users/villasv/orgs","repos_url":"https://api.github.com/users/villasv/repos","events_url":"https://api.github.com/users/villasv/events{/privacy}","received_events_url":"https://api.github.com/users/villasv/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2018-06-18T21:36:45Z","updated_at":"2018-10-17T22:00:04Z","closed_at":"2018-10-17T22:00:04Z","author_association":"CONTRIBUTOR","body":"As part of one of my projects, I have to scrape some data to augment the base dataset. The scraped data changes regularly (weekly) thanks to manual data curation on the original source.\r\n\r\nWhat is the best way to track that data using DVC? I have the following possibilities in mind:\r\n\r\n- Simply `dvc add` the data, but following #754, by using copy caches I can simply overwrite the file with the new scraping output. `dvc add` again to update the hashes and caches.\r\n- Run the scraping script with `dvc run`, and when updating is desired I simply reproduce ignoring cache\r\n- Create a dummy dependency file containing the scraping timestamp for example, and put that as a dependency of the `dvc run` scraping command. When updating, I just change the timestamp and reproduce the pipeline\r\n- Lastly, embed the latest run timestamp in the scraping script itself. Less hacky in the DVC side, more hacky in the git side. But I'd have to commit the new `.dvc` files anyway.\r\n\r\nIs there another way? Opinions on which is best? Is this a particular use case that could benefit for newer dvc features?\r\n\r\nThis is not a truly reproducible step, so I believe the first option is the most aligned with the philosophy here... so I won't be disappointed if that's the prevailing opinion. "},"comment":{"url":"https://api.github.com/repos/iterative/dvc/issues/comments/430804893","html_url":"https://github.com/iterative/dvc/issues/792#issuecomment-430804893","issue_url":"https://api.github.com/repos/iterative/dvc/issues/792","id":430804893,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMDgwNDg5Mw==","user":{"login":"villasv","id":6609289,"node_id":"MDQ6VXNlcjY2MDkyODk=","avatar_url":"https://avatars1.githubusercontent.com/u/6609289?v=4","gravatar_id":"","url":"https://api.github.com/users/villasv","html_url":"https://github.com/villasv","followers_url":"https://api.github.com/users/villasv/followers","following_url":"https://api.github.com/users/villasv/following{/other_user}","gists_url":"https://api.github.com/users/villasv/gists{/gist_id}","starred_url":"https://api.github.com/users/villasv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/villasv/subscriptions","organizations_url":"https://api.github.com/users/villasv/orgs","repos_url":"https://api.github.com/users/villasv/repos","events_url":"https://api.github.com/users/villasv/events{/privacy}","received_events_url":"https://api.github.com/users/villasv/received_events","type":"User","site_admin":false},"created_at":"2018-10-17T22:00:04Z","updated_at":"2018-10-17T22:00:04Z","author_association":"CONTRIBUTOR","body":"Pinging back on this issue. What I developed as the routine that would accommodate well with the dvc way of doing things:\r\n\r\n1. There's an in-place version of the scraped data, hardlinked to the cache\r\n2. The scraping script will copy the old version to a temporary file and append to it\r\n3. Copying the temp file back, it will overwrite the old version hardlink, being correctly treated as a new file and won't corrupt the cache.\r\n\r\nAs for hand curation in-place, it's just forbidden. We settled that all curation should be done in the original source, or if very necessary to be an inside venture, versioned in a \"patch-like\" file that is treated as code instead of data (this versioned with git).\r\n\r\nI think I can close the issue. This may not work for everyone, but it has been working for me :)"}}
{"actor":{"display_login":"villasv"}}