{"action":"created","issue":{"url":"https://api.github.com/repos/pyviz/pyct/issues/63","repository_url":"https://api.github.com/repos/pyviz/pyct","labels_url":"https://api.github.com/repos/pyviz/pyct/issues/63/labels{/name}","comments_url":"https://api.github.com/repos/pyviz/pyct/issues/63/comments","events_url":"https://api.github.com/repos/pyviz/pyct/issues/63/events","html_url":"https://github.com/pyviz/pyct/pull/63","id":372254481,"node_id":"MDExOlB1bGxSZXF1ZXN0MjI0NTAyMzg4","number":63,"title":"Support 'test data'.","user":{"login":"ceball","id":1929,"node_id":"MDQ6VXNlcjE5Mjk=","avatar_url":"https://avatars2.githubusercontent.com/u/1929?v=4","gravatar_id":"","url":"https://api.github.com/users/ceball","html_url":"https://github.com/ceball","followers_url":"https://api.github.com/users/ceball/followers","following_url":"https://api.github.com/users/ceball/following{/other_user}","gists_url":"https://api.github.com/users/ceball/gists{/gist_id}","starred_url":"https://api.github.com/users/ceball/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ceball/subscriptions","organizations_url":"https://api.github.com/users/ceball/orgs","repos_url":"https://api.github.com/users/ceball/repos","events_url":"https://api.github.com/users/ceball/events{/privacy}","received_events_url":"https://api.github.com/users/ceball/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2018-10-20T21:27:39Z","updated_at":"2018-10-31T19:33:36Z","closed_at":"2018-10-31T16:39:14Z","author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/pyviz/pyct/pulls/63","html_url":"https://github.com/pyviz/pyct/pull/63","diff_url":"https://github.com/pyviz/pyct/pull/63.diff","patch_url":"https://github.com/pyviz/pyct/pull/63.patch"},"body":"For testing that examples work at all (e.g. on travis), we can't always use full datasets (e.g. takes too long to download data, or requires too many resources to run example with the full dataset).\r\n\r\nThis PR supports 'test data' in pyct's fetch-data command, allowing to specify a small version of the full dataset. The idea is that example authors should provide a small test version of the data if the full data is big, and the example should work (i.e. run ok) with the small data.\r\n\r\nIf pyct.cmd is called with `--use-test-data`, then if a dataset contains `test_files`, these will be copied in place of `files`. The test files should exist already in the data directory (i.e. be committed to the examples/data directory in git).\r\n\r\nExample:\r\n\r\n```yaml\r\ndata:\r\n  - url: https://s3.amazonaws.com/datashader-data/nyc_taxi_wide.parq\r\n    title: 'NYC Taxi Data'\r\n    files:\r\n      - nyc_taxi_wide.parq\r\n    test_files:\r\n      - nyc_taxi_50k.parq\r\n```\r\n\r\n```sh\r\n$ pyviz fetch-data --path=examples --use-test-data\r\n[...]\r\nCopying test data file '/home/ceball/code/pyviz/pyviz3/examples/data/nyc_taxi_50k.parq' to '/home/ceball/code/pyviz/pyviz3/examples/data/nyc_taxi_wide.parq'\r\n[...]\r\n```\r\n\r\nBehavior should be same as for downloading, e.g. skips existing files by default:\r\n\r\n```sh\r\n$ pyviz fetch-data --path=examples --use-test-data\r\n[...]\r\nSkipping NYC Taxi Data\r\n[...]\r\n```\r\n\r\nHave also added `force` for data, which can be used with or without `--use-test-data` (e.g. to replace test data with real data):\r\n\r\n```sh\r\n$ pyviz fetch-data --path=examples --force\r\n[...]\r\nDownloading NYC Taxi Data 1 of 1\r\n[##                              ] 31000/375976 - 00:01:50\r\n[...]\r\n```\r\n\r\nI'm not familiar with the various datasets or the download/extract mechanism. So, I'm not sure my changes are correct - would be great if someone familiar with the download/extract mechanism could review it, although if we go ahead with this, I will end up testing and reviewing it in practice later anyway (when using for pyviz).\r\n\r\nI'm not sure what I've done will make sense for all datasets. In particular, the way I've done it here, there should be a corresponding entry in `test_files` for each entry in `files`, which might not make sense.\r\n\r\nIt's also possibly quite confusing that \"the files on disk are a lie\" after using `use_test_data`, but I can't think of another way that doesn't involve altering how data is loaded in examples.\r\n\r\nNote that if there are no `test_files` for a given dataset, `files` are used instead (i.e. assumes that if no test files are supplied, it's because the original dataset is not large/problematic).\r\n\r\nIf this change seems ok, then the `examples` command should probably support the same options (passing through to `fetch-data`) before merging.\r\n\r\nAnd then separately, nbsite will need some modifications (part of already planned modifications, e.g. https://github.com/pyviz/nbsite/issues/36).\r\n\r\nI would be delighted to hear about an entirely better way of doing things (if it doesn't involve changing how examples load data)."},"comment":{"url":"https://api.github.com/repos/pyviz/pyct/issues/comments/434819001","html_url":"https://github.com/pyviz/pyct/pull/63#issuecomment-434819001","issue_url":"https://api.github.com/repos/pyviz/pyct/issues/63","id":434819001,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNDgxOTAwMQ==","user":{"login":"jsignell","id":4806877,"node_id":"MDQ6VXNlcjQ4MDY4Nzc=","avatar_url":"https://avatars1.githubusercontent.com/u/4806877?v=4","gravatar_id":"","url":"https://api.github.com/users/jsignell","html_url":"https://github.com/jsignell","followers_url":"https://api.github.com/users/jsignell/followers","following_url":"https://api.github.com/users/jsignell/following{/other_user}","gists_url":"https://api.github.com/users/jsignell/gists{/gist_id}","starred_url":"https://api.github.com/users/jsignell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jsignell/subscriptions","organizations_url":"https://api.github.com/users/jsignell/orgs","repos_url":"https://api.github.com/users/jsignell/repos","events_url":"https://api.github.com/users/jsignell/events{/privacy}","received_events_url":"https://api.github.com/users/jsignell/received_events","type":"User","site_admin":false},"created_at":"2018-10-31T19:33:36Z","updated_at":"2018-10-31T19:33:36Z","author_association":"COLLABORATOR","body":"Or possible changes to `pyctdev`"}}
{"actor":{"display_login":"jsignell"}}