{"action":"created","comment":{"url":"https://api.github.com/repos/spectre-team/spectre-data/pulls/comments/183515952","pull_request_review_id":114509205,"id":183515952,"diff_hunk":"@@ -0,0 +1,24 @@\n+\"\"\"Constants and types common for all modules in package\n+\n+Copyright 2018 Spectre Team\n+\n+Licensed under the Apache License, Version 2.0 (the \"License\");\n+you may not use this file except in compliance with the License.\n+You may obtain a copy of the License at\n+\n+    http://www.apache.org/licenses/LICENSE-2.0\n+    \n+Unless required by applicable law or agreed to in writing, software\n+distributed under the License is distributed on an \"AS IS\" BASIS,\n+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n+See the License for the specific language governing permissions and\n+limitations under the License.\n+\"\"\"\n+\n+import os\n+\n+_FILESYSTEM_ROOT = os.path.abspath(os.sep)\n+DATA_ROOT = os.path.join(_FILESYSTEM_ROOT, 'data')\n+\n+Name = str\n+Path = str","path":"spdata/common.py","position":24,"original_position":24,"commit_id":"580384da8413916c2913106e40c432fab60e15ce","original_commit_id":"580384da8413916c2913106e40c432fab60e15ce","user":{"login":"mg6","id":7475587,"avatar_url":"https://avatars1.githubusercontent.com/u/7475587?v=4","gravatar_id":"","url":"https://api.github.com/users/mg6","html_url":"https://github.com/mg6","followers_url":"https://api.github.com/users/mg6/followers","following_url":"https://api.github.com/users/mg6/following{/other_user}","gists_url":"https://api.github.com/users/mg6/gists{/gist_id}","starred_url":"https://api.github.com/users/mg6/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mg6/subscriptions","organizations_url":"https://api.github.com/users/mg6/orgs","repos_url":"https://api.github.com/users/mg6/repos","events_url":"https://api.github.com/users/mg6/events{/privacy}","received_events_url":"https://api.github.com/users/mg6/received_events","type":"User","site_admin":false},"body":"I know this is huge to refactor, but shouldn’t we utilize [pathlib](https://docs.python.org/3/library/pathlib.html) for working with all paths? Aside from standardized way of checking for file existence, directory tests, we get a first-class citizen type `Path` we could pass to dataset loaders.","created_at":"2018-04-23T19:47:38Z","updated_at":"2018-04-23T19:47:53Z","html_url":"https://github.com/spectre-team/spectre-data/pull/3#discussion_r183515952","pull_request_url":"https://api.github.com/repos/spectre-team/spectre-data/pulls/3","author_association":"MEMBER","_links":{"self":{"href":"https://api.github.com/repos/spectre-team/spectre-data/pulls/comments/183515952"},"html":{"href":"https://github.com/spectre-team/spectre-data/pull/3#discussion_r183515952"},"pull_request":{"href":"https://api.github.com/repos/spectre-team/spectre-data/pulls/3"}}},"pull_request":{"url":"https://api.github.com/repos/spectre-team/spectre-data/pulls/3","id":183222837,"html_url":"https://github.com/spectre-team/spectre-data/pull/3","diff_url":"https://github.com/spectre-team/spectre-data/pull/3.diff","patch_url":"https://github.com/spectre-team/spectre-data/pull/3.patch","issue_url":"https://api.github.com/repos/spectre-team/spectre-data/issues/3","number":3,"state":"open","locked":false,"title":"Normalized dataset discovery and access","user":{"login":"Daroslaw","id":7458086,"avatar_url":"https://avatars1.githubusercontent.com/u/7458086?v=4","gravatar_id":"","url":"https://api.github.com/users/Daroslaw","html_url":"https://github.com/Daroslaw","followers_url":"https://api.github.com/users/Daroslaw/followers","following_url":"https://api.github.com/users/Daroslaw/following{/other_user}","gists_url":"https://api.github.com/users/Daroslaw/gists{/gist_id}","starred_url":"https://api.github.com/users/Daroslaw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Daroslaw/subscriptions","organizations_url":"https://api.github.com/users/Daroslaw/orgs","repos_url":"https://api.github.com/users/Daroslaw/repos","events_url":"https://api.github.com/users/Daroslaw/events{/privacy}","received_events_url":"https://api.github.com/users/Daroslaw/received_events","type":"User","site_admin":false},"body":"This is an attempt of writing code for dataset discovery and loading that could be universally used across the entire project. \r\n\r\nIn terms of discovery following functionalities are enclosed:\r\n- conversions between dataset name and its id\r\n- discovering path for single dataset (~~_and/or multiple paths if such dataset has been uploaded in several formats - this is a matter for discussion_~~)\r\n- returning all datasets in form of list of entries in form `name: dataset name, value: dataset_name`\r\n- check if dataset under given readable name (_without '\\_'s_) exists\r\n\r\nIn terms of loading the datasets:\r\n- made a decorator that can help with efficient addition of supports for future dataset formats (_all in KISS and DRY manner_)\r\n- made a single method for loading arbitrary dataset of arbitrary format, taking the readable name of desired dataset (~~_added possibility to launch multiple loaders on single dataset, if applicable - a matter for discussion_~~)\r\n- every specific loader takes now path to the file as argument\r\n\r\nIn terms of testing following problems were encountered:\r\n1. I couldn't find an efficient way to mock filesystem when `glob` comes into consideration. The `glob.glob` directly searches the real filesystem looking for entries that match input regex, so I don't know how to change the filesystem underneath it. Therefore `dataset_path` remains untested.\r\n2. For similar causes the `load_dataset` is not fully tested. In addition I think that mocking absolutely everything would be needed here in order to perform a successful load test, ~~let alone multiple load~~.","created_at":"2018-04-21T13:14:05Z","updated_at":"2018-04-23T19:47:53Z","closed_at":null,"merged_at":null,"merge_commit_sha":"23d9f61756feea8d2255d91549dfdee88a9347a3","assignee":{"login":"Daroslaw","id":7458086,"avatar_url":"https://avatars1.githubusercontent.com/u/7458086?v=4","gravatar_id":"","url":"https://api.github.com/users/Daroslaw","html_url":"https://github.com/Daroslaw","followers_url":"https://api.github.com/users/Daroslaw/followers","following_url":"https://api.github.com/users/Daroslaw/following{/other_user}","gists_url":"https://api.github.com/users/Daroslaw/gists{/gist_id}","starred_url":"https://api.github.com/users/Daroslaw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Daroslaw/subscriptions","organizations_url":"https://api.github.com/users/Daroslaw/orgs","repos_url":"https://api.github.com/users/Daroslaw/repos","events_url":"https://api.github.com/users/Daroslaw/events{/privacy}","received_events_url":"https://api.github.com/users/Daroslaw/received_events","type":"User","site_admin":false},"assignees":[{"login":"Daroslaw","id":7458086,"avatar_url":"https://avatars1.githubusercontent.com/u/7458086?v=4","gravatar_id":"","url":"https://api.github.com/users/Daroslaw","html_url":"https://github.com/Daroslaw","followers_url":"https://api.github.com/users/Daroslaw/followers","following_url":"https://api.github.com/users/Daroslaw/following{/other_user}","gists_url":"https://api.github.com/users/Daroslaw/gists{/gist_id}","starred_url":"https://api.github.com/users/Daroslaw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Daroslaw/subscriptions","organizations_url":"https://api.github.com/users/Daroslaw/orgs","repos_url":"https://api.github.com/users/Daroslaw/repos","events_url":"https://api.github.com/users/Daroslaw/events{/privacy}","received_events_url":"https://api.github.com/users/Daroslaw/received_events","type":"User","site_admin":false}],"requested_reviewers":[{"login":"gmrukwa","id":1897842,"avatar_url":"https://avatars0.githubusercontent.com/u/1897842?v=4","gravatar_id":"","url":"https://api.github.com/users/gmrukwa","html_url":"https://github.com/gmrukwa","followers_url":"https://api.github.com/users/gmrukwa/followers","following_url":"https://api.github.com/users/gmrukwa/following{/other_user}","gists_url":"https://api.github.com/users/gmrukwa/gists{/gist_id}","starred_url":"https://api.github.com/users/gmrukwa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gmrukwa/subscriptions","organizations_url":"https://api.github.com/users/gmrukwa/orgs","repos_url":"https://api.github.com/users/gmrukwa/repos","events_url":"https://api.github.com/users/gmrukwa/events{/privacy}","received_events_url":"https://api.github.com/users/gmrukwa/received_events","type":"User","site_admin":false}],"requested_teams":[],"labels":[{"id":795814857,"url":"https://api.github.com/repos/spectre-team/spectre-data/labels/enhancement","name":"enhancement","color":"84b6eb","default":true}],"milestone":null,"commits_url":"https://api.github.com/repos/spectre-team/spectre-data/pulls/3/commits","review_comments_url":"https://api.github.com/repos/spectre-team/spectre-data/pulls/3/comments","review_comment_url":"https://api.github.com/repos/spectre-team/spectre-data/pulls/comments{/number}","comments_url":"https://api.github.com/repos/spectre-team/spectre-data/issues/3/comments","statuses_url":"https://api.github.com/repos/spectre-team/spectre-data/statuses/580384da8413916c2913106e40c432fab60e15ce","head":{"label":"spectre-team:feature/normalized-discovery-and-access","ref":"feature/normalized-discovery-and-access","sha":"580384da8413916c2913106e40c432fab60e15ce","user":{"login":"spectre-team","id":25571382,"avatar_url":"https://avatars2.githubusercontent.com/u/25571382?v=4","gravatar_id":"","url":"https://api.github.com/users/spectre-team","html_url":"https://github.com/spectre-team","followers_url":"https://api.github.com/users/spectre-team/followers","following_url":"https://api.github.com/users/spectre-team/following{/other_user}","gists_url":"https://api.github.com/users/spectre-team/gists{/gist_id}","starred_url":"https://api.github.com/users/spectre-team/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spectre-team/subscriptions","organizations_url":"https://api.github.com/users/spectre-team/orgs","repos_url":"https://api.github.com/users/spectre-team/repos","events_url":"https://api.github.com/users/spectre-team/events{/privacy}","received_events_url":"https://api.github.com/users/spectre-team/received_events","type":"Organization","site_admin":false},"repo":{"id":116371476,"name":"spectre-data","full_name":"spectre-team/spectre-data","owner":{"login":"spectre-team","id":25571382,"avatar_url":"https://avatars2.githubusercontent.com/u/25571382?v=4","gravatar_id":"","url":"https://api.github.com/users/spectre-team","html_url":"https://github.com/spectre-team","followers_url":"https://api.github.com/users/spectre-team/followers","following_url":"https://api.github.com/users/spectre-team/following{/other_user}","gists_url":"https://api.github.com/users/spectre-team/gists{/gist_id}","starred_url":"https://api.github.com/users/spectre-team/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spectre-team/subscriptions","organizations_url":"https://api.github.com/users/spectre-team/orgs","repos_url":"https://api.github.com/users/spectre-team/repos","events_url":"https://api.github.com/users/spectre-team/events{/privacy}","received_events_url":"https://api.github.com/users/spectre-team/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/spectre-team/spectre-data","description":"Common data-access layer for all workers","fork":false,"url":"https://api.github.com/repos/spectre-team/spectre-data","forks_url":"https://api.github.com/repos/spectre-team/spectre-data/forks","keys_url":"https://api.github.com/repos/spectre-team/spectre-data/keys{/key_id}","collaborators_url":"https://api.github.com/repos/spectre-team/spectre-data/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/spectre-team/spectre-data/teams","hooks_url":"https://api.github.com/repos/spectre-team/spectre-data/hooks","issue_events_url":"https://api.github.com/repos/spectre-team/spectre-data/issues/events{/number}","events_url":"https://api.github.com/repos/spectre-team/spectre-data/events","assignees_url":"https://api.github.com/repos/spectre-team/spectre-data/assignees{/user}","branches_url":"https://api.github.com/repos/spectre-team/spectre-data/branches{/branch}","tags_url":"https://api.github.com/repos/spectre-team/spectre-data/tags","blobs_url":"https://api.github.com/repos/spectre-team/spectre-data/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/spectre-team/spectre-data/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/spectre-team/spectre-data/git/refs{/sha}","trees_url":"https://api.github.com/repos/spectre-team/spectre-data/git/trees{/sha}","statuses_url":"https://api.github.com/repos/spectre-team/spectre-data/statuses/{sha}","languages_url":"https://api.github.com/repos/spectre-team/spectre-data/languages","stargazers_url":"https://api.github.com/repos/spectre-team/spectre-data/stargazers","contributors_url":"https://api.github.com/repos/spectre-team/spectre-data/contributors","subscribers_url":"https://api.github.com/repos/spectre-team/spectre-data/subscribers","subscription_url":"https://api.github.com/repos/spectre-team/spectre-data/subscription","commits_url":"https://api.github.com/repos/spectre-team/spectre-data/commits{/sha}","git_commits_url":"https://api.github.com/repos/spectre-team/spectre-data/git/commits{/sha}","comments_url":"https://api.github.com/repos/spectre-team/spectre-data/comments{/number}","issue_comment_url":"https://api.github.com/repos/spectre-team/spectre-data/issues/comments{/number}","contents_url":"https://api.github.com/repos/spectre-team/spectre-data/contents/{+path}","compare_url":"https://api.github.com/repos/spectre-team/spectre-data/compare/{base}...{head}","merges_url":"https://api.github.com/repos/spectre-team/spectre-data/merges","archive_url":"https://api.github.com/repos/spectre-team/spectre-data/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/spectre-team/spectre-data/downloads","issues_url":"https://api.github.com/repos/spectre-team/spectre-data/issues{/number}","pulls_url":"https://api.github.com/repos/spectre-team/spectre-data/pulls{/number}","milestones_url":"https://api.github.com/repos/spectre-team/spectre-data/milestones{/number}","notifications_url":"https://api.github.com/repos/spectre-team/spectre-data/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/spectre-team/spectre-data/labels{/name}","releases_url":"https://api.github.com/repos/spectre-team/spectre-data/releases{/id}","deployments_url":"https://api.github.com/repos/spectre-team/spectre-data/deployments","created_at":"2018-01-05T10:09:38Z","updated_at":"2018-01-05T14:11:13Z","pushed_at":"2018-04-23T19:27:21Z","git_url":"git://github.com/spectre-team/spectre-data.git","ssh_url":"git@github.com:spectre-team/spectre-data.git","clone_url":"https://github.com/spectre-team/spectre-data.git","svn_url":"https://github.com/spectre-team/spectre-data","homepage":null,"size":49,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":1,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"base":{"label":"spectre-team:master","ref":"master","sha":"861e2c62bc1d0eb20ff03cbbe1f38bd64f735130","user":{"login":"spectre-team","id":25571382,"avatar_url":"https://avatars2.githubusercontent.com/u/25571382?v=4","gravatar_id":"","url":"https://api.github.com/users/spectre-team","html_url":"https://github.com/spectre-team","followers_url":"https://api.github.com/users/spectre-team/followers","following_url":"https://api.github.com/users/spectre-team/following{/other_user}","gists_url":"https://api.github.com/users/spectre-team/gists{/gist_id}","starred_url":"https://api.github.com/users/spectre-team/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spectre-team/subscriptions","organizations_url":"https://api.github.com/users/spectre-team/orgs","repos_url":"https://api.github.com/users/spectre-team/repos","events_url":"https://api.github.com/users/spectre-team/events{/privacy}","received_events_url":"https://api.github.com/users/spectre-team/received_events","type":"Organization","site_admin":false},"repo":{"id":116371476,"name":"spectre-data","full_name":"spectre-team/spectre-data","owner":{"login":"spectre-team","id":25571382,"avatar_url":"https://avatars2.githubusercontent.com/u/25571382?v=4","gravatar_id":"","url":"https://api.github.com/users/spectre-team","html_url":"https://github.com/spectre-team","followers_url":"https://api.github.com/users/spectre-team/followers","following_url":"https://api.github.com/users/spectre-team/following{/other_user}","gists_url":"https://api.github.com/users/spectre-team/gists{/gist_id}","starred_url":"https://api.github.com/users/spectre-team/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spectre-team/subscriptions","organizations_url":"https://api.github.com/users/spectre-team/orgs","repos_url":"https://api.github.com/users/spectre-team/repos","events_url":"https://api.github.com/users/spectre-team/events{/privacy}","received_events_url":"https://api.github.com/users/spectre-team/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/spectre-team/spectre-data","description":"Common data-access layer for all workers","fork":false,"url":"https://api.github.com/repos/spectre-team/spectre-data","forks_url":"https://api.github.com/repos/spectre-team/spectre-data/forks","keys_url":"https://api.github.com/repos/spectre-team/spectre-data/keys{/key_id}","collaborators_url":"https://api.github.com/repos/spectre-team/spectre-data/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/spectre-team/spectre-data/teams","hooks_url":"https://api.github.com/repos/spectre-team/spectre-data/hooks","issue_events_url":"https://api.github.com/repos/spectre-team/spectre-data/issues/events{/number}","events_url":"https://api.github.com/repos/spectre-team/spectre-data/events","assignees_url":"https://api.github.com/repos/spectre-team/spectre-data/assignees{/user}","branches_url":"https://api.github.com/repos/spectre-team/spectre-data/branches{/branch}","tags_url":"https://api.github.com/repos/spectre-team/spectre-data/tags","blobs_url":"https://api.github.com/repos/spectre-team/spectre-data/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/spectre-team/spectre-data/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/spectre-team/spectre-data/git/refs{/sha}","trees_url":"https://api.github.com/repos/spectre-team/spectre-data/git/trees{/sha}","statuses_url":"https://api.github.com/repos/spectre-team/spectre-data/statuses/{sha}","languages_url":"https://api.github.com/repos/spectre-team/spectre-data/languages","stargazers_url":"https://api.github.com/repos/spectre-team/spectre-data/stargazers","contributors_url":"https://api.github.com/repos/spectre-team/spectre-data/contributors","subscribers_url":"https://api.github.com/repos/spectre-team/spectre-data/subscribers","subscription_url":"https://api.github.com/repos/spectre-team/spectre-data/subscription","commits_url":"https://api.github.com/repos/spectre-team/spectre-data/commits{/sha}","git_commits_url":"https://api.github.com/repos/spectre-team/spectre-data/git/commits{/sha}","comments_url":"https://api.github.com/repos/spectre-team/spectre-data/comments{/number}","issue_comment_url":"https://api.github.com/repos/spectre-team/spectre-data/issues/comments{/number}","contents_url":"https://api.github.com/repos/spectre-team/spectre-data/contents/{+path}","compare_url":"https://api.github.com/repos/spectre-team/spectre-data/compare/{base}...{head}","merges_url":"https://api.github.com/repos/spectre-team/spectre-data/merges","archive_url":"https://api.github.com/repos/spectre-team/spectre-data/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/spectre-team/spectre-data/downloads","issues_url":"https://api.github.com/repos/spectre-team/spectre-data/issues{/number}","pulls_url":"https://api.github.com/repos/spectre-team/spectre-data/pulls{/number}","milestones_url":"https://api.github.com/repos/spectre-team/spectre-data/milestones{/number}","notifications_url":"https://api.github.com/repos/spectre-team/spectre-data/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/spectre-team/spectre-data/labels{/name}","releases_url":"https://api.github.com/repos/spectre-team/spectre-data/releases{/id}","deployments_url":"https://api.github.com/repos/spectre-team/spectre-data/deployments","created_at":"2018-01-05T10:09:38Z","updated_at":"2018-01-05T14:11:13Z","pushed_at":"2018-04-23T19:27:21Z","git_url":"git://github.com/spectre-team/spectre-data.git","ssh_url":"git@github.com:spectre-team/spectre-data.git","clone_url":"https://github.com/spectre-team/spectre-data.git","svn_url":"https://github.com/spectre-team/spectre-data","homepage":null,"size":49,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":1,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/spectre-team/spectre-data/pulls/3"},"html":{"href":"https://github.com/spectre-team/spectre-data/pull/3"},"issue":{"href":"https://api.github.com/repos/spectre-team/spectre-data/issues/3"},"comments":{"href":"https://api.github.com/repos/spectre-team/spectre-data/issues/3/comments"},"review_comments":{"href":"https://api.github.com/repos/spectre-team/spectre-data/pulls/3/comments"},"review_comment":{"href":"https://api.github.com/repos/spectre-team/spectre-data/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/spectre-team/spectre-data/pulls/3/commits"},"statuses":{"href":"https://api.github.com/repos/spectre-team/spectre-data/statuses/580384da8413916c2913106e40c432fab60e15ce"}},"author_association":"MEMBER"}}
{"actor":{"display_login":"mg6"}}