{"action":"closed","number":26,"pull_request":{"url":"https://api.github.com/repos/stcorp/harp/pulls/26","id":78160417,"html_url":"https://github.com/stcorp/harp/pull/26","diff_url":"https://github.com/stcorp/harp/pull/26.diff","patch_url":"https://github.com/stcorp/harp/pull/26.patch","issue_url":"https://api.github.com/repos/stcorp/harp/issues/26","number":26,"state":"closed","locked":false,"title":"Implementation of harp-dataset","user":{"login":"ElessarWebb","id":475738,"avatar_url":"https://avatars.githubusercontent.com/u/475738?v=3","gravatar_id":"","url":"https://api.github.com/users/ElessarWebb","html_url":"https://github.com/ElessarWebb","followers_url":"https://api.github.com/users/ElessarWebb/followers","following_url":"https://api.github.com/users/ElessarWebb/following{/other_user}","gists_url":"https://api.github.com/users/ElessarWebb/gists{/gist_id}","starred_url":"https://api.github.com/users/ElessarWebb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ElessarWebb/subscriptions","organizations_url":"https://api.github.com/users/ElessarWebb/orgs","repos_url":"https://api.github.com/users/ElessarWebb/repos","events_url":"https://api.github.com/users/ElessarWebb/events{/privacy}","received_events_url":"https://api.github.com/users/ElessarWebb/received_events","type":"User","site_admin":false},"body":"- [x] Refactor global_attributes into harp_product_metadata\r\n- [x] Add harp_dataset_add_file\r\n- [x] Add harp_dataset_add_directory\r\n- [x] Refactor harp_collocation_result to use harp_dataset and harp_product_metadata properly.\r\n- [x] Test that things still work\r\n\r\nThe last task is not as trivial as it sounds, mostly because we want to build harp_collocation_result instances without requiring the reading of a whole directory of products.\r\nWe therefor split the task in two:\r\n\r\n- [x] Refactor harp_collocation_result to a less string-centered, more efficient representation\r\n- [ ] ~~Add a function fx that takes a harp_collocation_result and a directory and builds a dataset that stores the metadata for all the files associated with products that appear in the result set.~~ (next PR)\r\n\r\nImportant operations  on datasets and collocation results to optimize for are:\r\n\r\n- `harp_dataset_add_file` (for incremental builds of datasets using e.g. `harp_dataset_add_directory`),\r\n- `harp_collocation_result_read`: Read a result csv file¸\r\n- `harp_collocation_result_read_metadata`: Building a dataset from a directory associated with a harp_collocation result\r\n- finding the product metadata associated with an entry in a collocation_result, assuming that we have the associated dataset.\r\n\r\nTherefor we propose the following representations:\r\n\r\n### Harp product metadata\r\n\r\n```\r\nstruct harp_product_metadata_struct {\r\n    char *filename; /* can be NULL if no file is associated with the product */\r\n    double datetime_start;\r\n    double datetime_stop;\r\n    long dimension[HARP_NUM_DIM_TYPES];\r\n    char *source_product;\r\n};\r\n```\r\n\r\nA downside of this representation is that for the following sequence of operations, we end up with a metadata instance `m` without a filename, even though it exists:\r\n\r\n`harp_import_product(.., p) -> harp_product_get_metadata(p, m)`\r\n\r\n### Harp dataset\r\n\r\n```\r\n/** HARP Dataset struct */\r\nstruct harp_dataset_struct {\r\n    hashtable_struct *source_to_index; /* maps product_source names to indices */\r\n    char **product_source; /* set of product sources - every source appears only once */\r\n    int num_metadata;\r\n    harp_product_metadata **metadata; /* stores metadata for source in product_source */\r\n};\r\n\r\n/** HARP Dataset typedef */\r\ntypedef struct harp_dataset_struct harp_dataset;\r\n```\r\n\r\n### Harp collocation results\r\n\r\n```\r\nstruct harp_collocation_pair_struct\r\n{\r\n    long collocation_index;\r\n    int product_index_a;  /* index into dataset_a of associated collocation result */\r\n    long sample_index_a;\r\n    int product_index_b;  /* index into dataset_b of associated collocation result */\r\n    long sample_index_b;\r\n    double difference[HARP_COLLOCATION_RESULT_MAX_NUM_DIFFERENCES];\r\n};\r\n\r\n/** HARP Collocation result */\r\nstruct harp_collocation_result {\r\n    harp_dataset dataset_a;\r\n    harp_dataset dataset_b;\r\n    int difference_available[HARP_COLLOCATION_RESULT_MAX_NUM_DIFFERENCES];\r\n    char *difference_unit[HARP_COLLOCATION_RESULT_MAX_NUM_DIFFERENCES];\r\n    int num_pairs;\r\n    harp_collocation_pair **pair;\r\n};\r\n```\r\n\r\nharp_dataset can have it's attribute `metadata == NULL` to indicate that no metadata is known for it's product's sources.\r\n\r\n`harp_collocation_result->pair` can be sorted as before.","created_at":"2016-07-20T14:47:33Z","updated_at":"2016-07-27T10:13:14Z","closed_at":"2016-07-27T10:13:13Z","merged_at":"2016-07-27T10:13:13Z","merge_commit_sha":"b4321db4ce924c1b6a10b7450640c52f677f8859","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/stcorp/harp/pulls/26/commits","review_comments_url":"https://api.github.com/repos/stcorp/harp/pulls/26/comments","review_comment_url":"https://api.github.com/repos/stcorp/harp/pulls/comments{/number}","comments_url":"https://api.github.com/repos/stcorp/harp/issues/26/comments","statuses_url":"https://api.github.com/repos/stcorp/harp/statuses/2ef349f099b4d76e7835f1899b8426ea5ba7ee9f","head":{"label":"stcorp:harp-dataset","ref":"harp-dataset","sha":"2ef349f099b4d76e7835f1899b8426ea5ba7ee9f","user":{"login":"stcorp","id":13534981,"avatar_url":"https://avatars.githubusercontent.com/u/13534981?v=3","gravatar_id":"","url":"https://api.github.com/users/stcorp","html_url":"https://github.com/stcorp","followers_url":"https://api.github.com/users/stcorp/followers","following_url":"https://api.github.com/users/stcorp/following{/other_user}","gists_url":"https://api.github.com/users/stcorp/gists{/gist_id}","starred_url":"https://api.github.com/users/stcorp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stcorp/subscriptions","organizations_url":"https://api.github.com/users/stcorp/orgs","repos_url":"https://api.github.com/users/stcorp/repos","events_url":"https://api.github.com/users/stcorp/events{/privacy}","received_events_url":"https://api.github.com/users/stcorp/received_events","type":"Organization","site_admin":false},"repo":{"id":45833593,"name":"harp","full_name":"stcorp/harp","owner":{"login":"stcorp","id":13534981,"avatar_url":"https://avatars.githubusercontent.com/u/13534981?v=3","gravatar_id":"","url":"https://api.github.com/users/stcorp","html_url":"https://github.com/stcorp","followers_url":"https://api.github.com/users/stcorp/followers","following_url":"https://api.github.com/users/stcorp/following{/other_user}","gists_url":"https://api.github.com/users/stcorp/gists{/gist_id}","starred_url":"https://api.github.com/users/stcorp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stcorp/subscriptions","organizations_url":"https://api.github.com/users/stcorp/orgs","repos_url":"https://api.github.com/users/stcorp/repos","events_url":"https://api.github.com/users/stcorp/events{/privacy}","received_events_url":"https://api.github.com/users/stcorp/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/stcorp/harp","description":"Data harmonization toolset for scientific earth observation data ","fork":false,"url":"https://api.github.com/repos/stcorp/harp","forks_url":"https://api.github.com/repos/stcorp/harp/forks","keys_url":"https://api.github.com/repos/stcorp/harp/keys{/key_id}","collaborators_url":"https://api.github.com/repos/stcorp/harp/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/stcorp/harp/teams","hooks_url":"https://api.github.com/repos/stcorp/harp/hooks","issue_events_url":"https://api.github.com/repos/stcorp/harp/issues/events{/number}","events_url":"https://api.github.com/repos/stcorp/harp/events","assignees_url":"https://api.github.com/repos/stcorp/harp/assignees{/user}","branches_url":"https://api.github.com/repos/stcorp/harp/branches{/branch}","tags_url":"https://api.github.com/repos/stcorp/harp/tags","blobs_url":"https://api.github.com/repos/stcorp/harp/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/stcorp/harp/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/stcorp/harp/git/refs{/sha}","trees_url":"https://api.github.com/repos/stcorp/harp/git/trees{/sha}","statuses_url":"https://api.github.com/repos/stcorp/harp/statuses/{sha}","languages_url":"https://api.github.com/repos/stcorp/harp/languages","stargazers_url":"https://api.github.com/repos/stcorp/harp/stargazers","contributors_url":"https://api.github.com/repos/stcorp/harp/contributors","subscribers_url":"https://api.github.com/repos/stcorp/harp/subscribers","subscription_url":"https://api.github.com/repos/stcorp/harp/subscription","commits_url":"https://api.github.com/repos/stcorp/harp/commits{/sha}","git_commits_url":"https://api.github.com/repos/stcorp/harp/git/commits{/sha}","comments_url":"https://api.github.com/repos/stcorp/harp/comments{/number}","issue_comment_url":"https://api.github.com/repos/stcorp/harp/issues/comments{/number}","contents_url":"https://api.github.com/repos/stcorp/harp/contents/{+path}","compare_url":"https://api.github.com/repos/stcorp/harp/compare/{base}...{head}","merges_url":"https://api.github.com/repos/stcorp/harp/merges","archive_url":"https://api.github.com/repos/stcorp/harp/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/stcorp/harp/downloads","issues_url":"https://api.github.com/repos/stcorp/harp/issues{/number}","pulls_url":"https://api.github.com/repos/stcorp/harp/pulls{/number}","milestones_url":"https://api.github.com/repos/stcorp/harp/milestones{/number}","notifications_url":"https://api.github.com/repos/stcorp/harp/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/stcorp/harp/labels{/name}","releases_url":"https://api.github.com/repos/stcorp/harp/releases{/id}","deployments_url":"https://api.github.com/repos/stcorp/harp/deployments","created_at":"2015-11-09T11:20:33Z","updated_at":"2016-01-19T10:46:52Z","pushed_at":"2016-07-27T10:13:13Z","git_url":"git://github.com/stcorp/harp.git","ssh_url":"git@github.com:stcorp/harp.git","clone_url":"https://github.com/stcorp/harp.git","svn_url":"https://github.com/stcorp/harp","homepage":null,"size":4945,"stargazers_count":0,"watchers_count":0,"language":"C","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":17,"forks":0,"open_issues":17,"watchers":0,"default_branch":"master"}},"base":{"label":"stcorp:master","ref":"master","sha":"c7df8d7d8f2c8fd03f871e8f0398ae88d9543132","user":{"login":"stcorp","id":13534981,"avatar_url":"https://avatars.githubusercontent.com/u/13534981?v=3","gravatar_id":"","url":"https://api.github.com/users/stcorp","html_url":"https://github.com/stcorp","followers_url":"https://api.github.com/users/stcorp/followers","following_url":"https://api.github.com/users/stcorp/following{/other_user}","gists_url":"https://api.github.com/users/stcorp/gists{/gist_id}","starred_url":"https://api.github.com/users/stcorp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stcorp/subscriptions","organizations_url":"https://api.github.com/users/stcorp/orgs","repos_url":"https://api.github.com/users/stcorp/repos","events_url":"https://api.github.com/users/stcorp/events{/privacy}","received_events_url":"https://api.github.com/users/stcorp/received_events","type":"Organization","site_admin":false},"repo":{"id":45833593,"name":"harp","full_name":"stcorp/harp","owner":{"login":"stcorp","id":13534981,"avatar_url":"https://avatars.githubusercontent.com/u/13534981?v=3","gravatar_id":"","url":"https://api.github.com/users/stcorp","html_url":"https://github.com/stcorp","followers_url":"https://api.github.com/users/stcorp/followers","following_url":"https://api.github.com/users/stcorp/following{/other_user}","gists_url":"https://api.github.com/users/stcorp/gists{/gist_id}","starred_url":"https://api.github.com/users/stcorp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stcorp/subscriptions","organizations_url":"https://api.github.com/users/stcorp/orgs","repos_url":"https://api.github.com/users/stcorp/repos","events_url":"https://api.github.com/users/stcorp/events{/privacy}","received_events_url":"https://api.github.com/users/stcorp/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/stcorp/harp","description":"Data harmonization toolset for scientific earth observation data ","fork":false,"url":"https://api.github.com/repos/stcorp/harp","forks_url":"https://api.github.com/repos/stcorp/harp/forks","keys_url":"https://api.github.com/repos/stcorp/harp/keys{/key_id}","collaborators_url":"https://api.github.com/repos/stcorp/harp/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/stcorp/harp/teams","hooks_url":"https://api.github.com/repos/stcorp/harp/hooks","issue_events_url":"https://api.github.com/repos/stcorp/harp/issues/events{/number}","events_url":"https://api.github.com/repos/stcorp/harp/events","assignees_url":"https://api.github.com/repos/stcorp/harp/assignees{/user}","branches_url":"https://api.github.com/repos/stcorp/harp/branches{/branch}","tags_url":"https://api.github.com/repos/stcorp/harp/tags","blobs_url":"https://api.github.com/repos/stcorp/harp/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/stcorp/harp/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/stcorp/harp/git/refs{/sha}","trees_url":"https://api.github.com/repos/stcorp/harp/git/trees{/sha}","statuses_url":"https://api.github.com/repos/stcorp/harp/statuses/{sha}","languages_url":"https://api.github.com/repos/stcorp/harp/languages","stargazers_url":"https://api.github.com/repos/stcorp/harp/stargazers","contributors_url":"https://api.github.com/repos/stcorp/harp/contributors","subscribers_url":"https://api.github.com/repos/stcorp/harp/subscribers","subscription_url":"https://api.github.com/repos/stcorp/harp/subscription","commits_url":"https://api.github.com/repos/stcorp/harp/commits{/sha}","git_commits_url":"https://api.github.com/repos/stcorp/harp/git/commits{/sha}","comments_url":"https://api.github.com/repos/stcorp/harp/comments{/number}","issue_comment_url":"https://api.github.com/repos/stcorp/harp/issues/comments{/number}","contents_url":"https://api.github.com/repos/stcorp/harp/contents/{+path}","compare_url":"https://api.github.com/repos/stcorp/harp/compare/{base}...{head}","merges_url":"https://api.github.com/repos/stcorp/harp/merges","archive_url":"https://api.github.com/repos/stcorp/harp/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/stcorp/harp/downloads","issues_url":"https://api.github.com/repos/stcorp/harp/issues{/number}","pulls_url":"https://api.github.com/repos/stcorp/harp/pulls{/number}","milestones_url":"https://api.github.com/repos/stcorp/harp/milestones{/number}","notifications_url":"https://api.github.com/repos/stcorp/harp/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/stcorp/harp/labels{/name}","releases_url":"https://api.github.com/repos/stcorp/harp/releases{/id}","deployments_url":"https://api.github.com/repos/stcorp/harp/deployments","created_at":"2015-11-09T11:20:33Z","updated_at":"2016-01-19T10:46:52Z","pushed_at":"2016-07-27T10:13:13Z","git_url":"git://github.com/stcorp/harp.git","ssh_url":"git@github.com:stcorp/harp.git","clone_url":"https://github.com/stcorp/harp.git","svn_url":"https://github.com/stcorp/harp","homepage":null,"size":4945,"stargazers_count":0,"watchers_count":0,"language":"C","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":17,"forks":0,"open_issues":17,"watchers":0,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/stcorp/harp/pulls/26"},"html":{"href":"https://github.com/stcorp/harp/pull/26"},"issue":{"href":"https://api.github.com/repos/stcorp/harp/issues/26"},"comments":{"href":"https://api.github.com/repos/stcorp/harp/issues/26/comments"},"review_comments":{"href":"https://api.github.com/repos/stcorp/harp/pulls/26/comments"},"review_comment":{"href":"https://api.github.com/repos/stcorp/harp/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/stcorp/harp/pulls/26/commits"},"statuses":{"href":"https://api.github.com/repos/stcorp/harp/statuses/2ef349f099b4d76e7835f1899b8426ea5ba7ee9f"}},"merged":true,"mergeable":null,"mergeable_state":"unknown","merged_by":{"login":"svniemeijer","id":4107291,"avatar_url":"https://avatars.githubusercontent.com/u/4107291?v=3","gravatar_id":"","url":"https://api.github.com/users/svniemeijer","html_url":"https://github.com/svniemeijer","followers_url":"https://api.github.com/users/svniemeijer/followers","following_url":"https://api.github.com/users/svniemeijer/following{/other_user}","gists_url":"https://api.github.com/users/svniemeijer/gists{/gist_id}","starred_url":"https://api.github.com/users/svniemeijer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/svniemeijer/subscriptions","organizations_url":"https://api.github.com/users/svniemeijer/orgs","repos_url":"https://api.github.com/users/svniemeijer/repos","events_url":"https://api.github.com/users/svniemeijer/events{/privacy}","received_events_url":"https://api.github.com/users/svniemeijer/received_events","type":"User","site_admin":false},"comments":7,"review_comments":0,"commits":27,"additions":2047,"deletions":496,"changed_files":36}}
{"actor":{"display_login":"svniemeijer"}}