{"action":"created","comment":{"url":"https://api.github.com/repos/codice/ddf/pulls/comments/73034155","id":73034155,"diff_hunk":"@@ -465,6 +474,23 @@ private void skipBytes(InputStream is, String bytesToSkip) throws IOException {\n         }\n     }\n \n+    private void alignStream(InputStream in, long requestedBytesToSkip, long responseBytesSkipped)\n+            throws IOException {\n+        try {\n+            if (requestedBytesToSkip > responseBytesSkipped) {\n+                LOGGER.error(\n+                        \"Source incorrect byte range, skipping ahead [{}] bytes in the input stream\",\n+                        requestedBytesToSkip - responseBytesSkipped);\n+                long skipped = in.skip(requestedBytesToSkip - responseBytesSkipped);\n+            } else if (requestedBytesToSkip < responseBytesSkipped) {\n+                throw new IOException(\n+                        \"Source server skipped more bytes than requested in the range header.\");\n+            }\n+        } catch (IOException e) {","path":"catalog/core/catalog-core-urlresourcereader/src/main/java/ddf/catalog/resource/impl/URLResourceReader.java","position":48,"original_position":48,"commit_id":"e7149817c7616e9605985e798ec5de34af53d365","original_commit_id":"e7149817c7616e9605985e798ec5de34af53d365","user":{"login":"figliold","id":4838043,"avatar_url":"https://avatars.githubusercontent.com/u/4838043?v=3","gravatar_id":"","url":"https://api.github.com/users/figliold","html_url":"https://github.com/figliold","followers_url":"https://api.github.com/users/figliold/followers","following_url":"https://api.github.com/users/figliold/following{/other_user}","gists_url":"https://api.github.com/users/figliold/gists{/gist_id}","starred_url":"https://api.github.com/users/figliold/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/figliold/subscriptions","organizations_url":"https://api.github.com/users/figliold/orgs","repos_url":"https://api.github.com/users/figliold/repos","events_url":"https://api.github.com/users/figliold/events{/privacy}","received_events_url":"https://api.github.com/users/figliold/received_events","type":"User","site_admin":false},"body":"Should we log the exception message here?","created_at":"2016-08-01T19:04:28Z","updated_at":"2016-08-01T19:04:28Z","html_url":"https://github.com/codice/ddf/pull/1064#discussion_r73034155","pull_request_url":"https://api.github.com/repos/codice/ddf/pulls/1064","_links":{"self":{"href":"https://api.github.com/repos/codice/ddf/pulls/comments/73034155"},"html":{"href":"https://github.com/codice/ddf/pull/1064#discussion_r73034155"},"pull_request":{"href":"https://api.github.com/repos/codice/ddf/pulls/1064"}}},"pull_request":{"url":"https://api.github.com/repos/codice/ddf/pulls/1064","id":79234374,"html_url":"https://github.com/codice/ddf/pull/1064","diff_url":"https://github.com/codice/ddf/pull/1064.diff","patch_url":"https://github.com/codice/ddf/pull/1064.patch","issue_url":"https://api.github.com/repos/codice/ddf/issues/1064","number":1064,"state":"open","locked":false,"title":"DDF-2300: Range Headers are now supported for Product Retrieval","user":{"login":"garrettfreibott","id":12499654,"avatar_url":"https://avatars.githubusercontent.com/u/12499654?v=3","gravatar_id":"","url":"https://api.github.com/users/garrettfreibott","html_url":"https://github.com/garrettfreibott","followers_url":"https://api.github.com/users/garrettfreibott/followers","following_url":"https://api.github.com/users/garrettfreibott/following{/other_user}","gists_url":"https://api.github.com/users/garrettfreibott/gists{/gist_id}","starred_url":"https://api.github.com/users/garrettfreibott/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/garrettfreibott/subscriptions","organizations_url":"https://api.github.com/users/garrettfreibott/orgs","repos_url":"https://api.github.com/users/garrettfreibott/repos","events_url":"https://api.github.com/users/garrettfreibott/events{/privacy}","received_events_url":"https://api.github.com/users/garrettfreibott/received_events","type":"User","site_admin":false},"body":"#### What does this PR do?\r\nAdds range header support for product retrieval retries. When the ReliableResourceDownloader attempts to retry retrieval of a product, it will no longer start over from the beginning of the download every time. There was also a major bug where DDF would properly use range headers to retrieve a product during a retry, but it would skip bytes in the input stream regardless of whether or not the server responding supported it or not, meaning unread data would be skipped and the file would become corrupted.\r\n\r\n#### Who is reviewing it?\r\n@pklinef \r\n@oconnormi \r\n@jrnorth \r\n\r\n#### Choose 2 committers to review/merge the PR:\r\n@coyotesqrl\r\n@figliold \r\n\r\n#### How should this be tested?\r\nitests were added to check that the CSW retrieval works properly and to test retries via simulated network disconnects. There were also a few itests for simple opensearch downloads. There is a ticket out that we will be picking up soon to add an opensearch stub server similar to the csw stub server, but for now, the best way to test is to spin up an instance and test that product retrieval still works for all the various sources WITHOUT the data becoming corrupted. Ingest some sort of plain text file so that it can be easily double checked for missing characters. It would also probably be best to re-download the file immediately to check that the returned cache copy is not corrupted either.\r\n\r\n#### Any background context you want to provide?\r\nSince I needed to change ReliableResourceDownloader, it was necessary to change the URLResourceReader to also support Partial Content responses, so that had the side-effect of (hopefully) making all the sources support range headers. Hooray!\r\n\r\n#### Commit Messages:\r\nProduct Retrievals will now properly request range headers and handle\r\nPartial Content responses so that subsequent retries will not have\r\nto redownload the entire product. Previously the endpoints would send\r\nrequests with range headers properly, but would not handle the response\r\ncorrectly, potentially resulting in corrupted data.\r\n\r\nAfter modifying the ReliableResourceDownloader[RRD], changes were needed\r\nin the URLResourceReader to adjust for the fact that the RRD no longer\r\nskipped ahead bytes when re-retrieving a resource. These changes also cause\r\nall of the other endpoints that use it to support range header retrieval.\r\nHooray!","created_at":"2016-07-28T17:17:59Z","updated_at":"2016-08-01T19:04:28Z","closed_at":null,"merged_at":null,"merge_commit_sha":"759f49939fe86a7dbfa97b594259ccd5ef5a3f06","assignee":{"login":"coyotesqrl","id":1469860,"avatar_url":"https://avatars.githubusercontent.com/u/1469860?v=3","gravatar_id":"","url":"https://api.github.com/users/coyotesqrl","html_url":"https://github.com/coyotesqrl","followers_url":"https://api.github.com/users/coyotesqrl/followers","following_url":"https://api.github.com/users/coyotesqrl/following{/other_user}","gists_url":"https://api.github.com/users/coyotesqrl/gists{/gist_id}","starred_url":"https://api.github.com/users/coyotesqrl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/coyotesqrl/subscriptions","organizations_url":"https://api.github.com/users/coyotesqrl/orgs","repos_url":"https://api.github.com/users/coyotesqrl/repos","events_url":"https://api.github.com/users/coyotesqrl/events{/privacy}","received_events_url":"https://api.github.com/users/coyotesqrl/received_events","type":"User","site_admin":false},"assignees":[{"login":"coyotesqrl","id":1469860,"avatar_url":"https://avatars.githubusercontent.com/u/1469860?v=3","gravatar_id":"","url":"https://api.github.com/users/coyotesqrl","html_url":"https://github.com/coyotesqrl","followers_url":"https://api.github.com/users/coyotesqrl/followers","following_url":"https://api.github.com/users/coyotesqrl/following{/other_user}","gists_url":"https://api.github.com/users/coyotesqrl/gists{/gist_id}","starred_url":"https://api.github.com/users/coyotesqrl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/coyotesqrl/subscriptions","organizations_url":"https://api.github.com/users/coyotesqrl/orgs","repos_url":"https://api.github.com/users/coyotesqrl/repos","events_url":"https://api.github.com/users/coyotesqrl/events{/privacy}","received_events_url":"https://api.github.com/users/coyotesqrl/received_events","type":"User","site_admin":false},{"login":"figliold","id":4838043,"avatar_url":"https://avatars.githubusercontent.com/u/4838043?v=3","gravatar_id":"","url":"https://api.github.com/users/figliold","html_url":"https://github.com/figliold","followers_url":"https://api.github.com/users/figliold/followers","following_url":"https://api.github.com/users/figliold/following{/other_user}","gists_url":"https://api.github.com/users/figliold/gists{/gist_id}","starred_url":"https://api.github.com/users/figliold/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/figliold/subscriptions","organizations_url":"https://api.github.com/users/figliold/orgs","repos_url":"https://api.github.com/users/figliold/repos","events_url":"https://api.github.com/users/figliold/events{/privacy}","received_events_url":"https://api.github.com/users/figliold/received_events","type":"User","site_admin":false}],"milestone":null,"commits_url":"https://api.github.com/repos/codice/ddf/pulls/1064/commits","review_comments_url":"https://api.github.com/repos/codice/ddf/pulls/1064/comments","review_comment_url":"https://api.github.com/repos/codice/ddf/pulls/comments{/number}","comments_url":"https://api.github.com/repos/codice/ddf/issues/1064/comments","statuses_url":"https://api.github.com/repos/codice/ddf/statuses/e7149817c7616e9605985e798ec5de34af53d365","head":{"label":"garrettfreibott:DDF-2300-v2","ref":"DDF-2300-v2","sha":"e7149817c7616e9605985e798ec5de34af53d365","user":{"login":"garrettfreibott","id":12499654,"avatar_url":"https://avatars.githubusercontent.com/u/12499654?v=3","gravatar_id":"","url":"https://api.github.com/users/garrettfreibott","html_url":"https://github.com/garrettfreibott","followers_url":"https://api.github.com/users/garrettfreibott/followers","following_url":"https://api.github.com/users/garrettfreibott/following{/other_user}","gists_url":"https://api.github.com/users/garrettfreibott/gists{/gist_id}","starred_url":"https://api.github.com/users/garrettfreibott/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/garrettfreibott/subscriptions","organizations_url":"https://api.github.com/users/garrettfreibott/orgs","repos_url":"https://api.github.com/users/garrettfreibott/repos","events_url":"https://api.github.com/users/garrettfreibott/events{/privacy}","received_events_url":"https://api.github.com/users/garrettfreibott/received_events","type":"User","site_admin":false},"repo":{"id":40502755,"name":"ddf","full_name":"garrettfreibott/ddf","owner":{"login":"garrettfreibott","id":12499654,"avatar_url":"https://avatars.githubusercontent.com/u/12499654?v=3","gravatar_id":"","url":"https://api.github.com/users/garrettfreibott","html_url":"https://github.com/garrettfreibott","followers_url":"https://api.github.com/users/garrettfreibott/followers","following_url":"https://api.github.com/users/garrettfreibott/following{/other_user}","gists_url":"https://api.github.com/users/garrettfreibott/gists{/gist_id}","starred_url":"https://api.github.com/users/garrettfreibott/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/garrettfreibott/subscriptions","organizations_url":"https://api.github.com/users/garrettfreibott/orgs","repos_url":"https://api.github.com/users/garrettfreibott/repos","events_url":"https://api.github.com/users/garrettfreibott/events{/privacy}","received_events_url":"https://api.github.com/users/garrettfreibott/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/garrettfreibott/ddf","description":"The Distributed Data Framework - Distribution, SDK, and Test","fork":true,"url":"https://api.github.com/repos/garrettfreibott/ddf","forks_url":"https://api.github.com/repos/garrettfreibott/ddf/forks","keys_url":"https://api.github.com/repos/garrettfreibott/ddf/keys{/key_id}","collaborators_url":"https://api.github.com/repos/garrettfreibott/ddf/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/garrettfreibott/ddf/teams","hooks_url":"https://api.github.com/repos/garrettfreibott/ddf/hooks","issue_events_url":"https://api.github.com/repos/garrettfreibott/ddf/issues/events{/number}","events_url":"https://api.github.com/repos/garrettfreibott/ddf/events","assignees_url":"https://api.github.com/repos/garrettfreibott/ddf/assignees{/user}","branches_url":"https://api.github.com/repos/garrettfreibott/ddf/branches{/branch}","tags_url":"https://api.github.com/repos/garrettfreibott/ddf/tags","blobs_url":"https://api.github.com/repos/garrettfreibott/ddf/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/garrettfreibott/ddf/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/garrettfreibott/ddf/git/refs{/sha}","trees_url":"https://api.github.com/repos/garrettfreibott/ddf/git/trees{/sha}","statuses_url":"https://api.github.com/repos/garrettfreibott/ddf/statuses/{sha}","languages_url":"https://api.github.com/repos/garrettfreibott/ddf/languages","stargazers_url":"https://api.github.com/repos/garrettfreibott/ddf/stargazers","contributors_url":"https://api.github.com/repos/garrettfreibott/ddf/contributors","subscribers_url":"https://api.github.com/repos/garrettfreibott/ddf/subscribers","subscription_url":"https://api.github.com/repos/garrettfreibott/ddf/subscription","commits_url":"https://api.github.com/repos/garrettfreibott/ddf/commits{/sha}","git_commits_url":"https://api.github.com/repos/garrettfreibott/ddf/git/commits{/sha}","comments_url":"https://api.github.com/repos/garrettfreibott/ddf/comments{/number}","issue_comment_url":"https://api.github.com/repos/garrettfreibott/ddf/issues/comments{/number}","contents_url":"https://api.github.com/repos/garrettfreibott/ddf/contents/{+path}","compare_url":"https://api.github.com/repos/garrettfreibott/ddf/compare/{base}...{head}","merges_url":"https://api.github.com/repos/garrettfreibott/ddf/merges","archive_url":"https://api.github.com/repos/garrettfreibott/ddf/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/garrettfreibott/ddf/downloads","issues_url":"https://api.github.com/repos/garrettfreibott/ddf/issues{/number}","pulls_url":"https://api.github.com/repos/garrettfreibott/ddf/pulls{/number}","milestones_url":"https://api.github.com/repos/garrettfreibott/ddf/milestones{/number}","notifications_url":"https://api.github.com/repos/garrettfreibott/ddf/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/garrettfreibott/ddf/labels{/name}","releases_url":"https://api.github.com/repos/garrettfreibott/ddf/releases{/id}","deployments_url":"https://api.github.com/repos/garrettfreibott/ddf/deployments","created_at":"2015-08-10T19:49:40Z","updated_at":"2016-01-11T19:32:20Z","pushed_at":"2016-08-01T15:07:10Z","git_url":"git://github.com/garrettfreibott/ddf.git","ssh_url":"git@github.com:garrettfreibott/ddf.git","clone_url":"https://github.com/garrettfreibott/ddf.git","svn_url":"https://github.com/garrettfreibott/ddf","homepage":"","size":146004,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"codice:master","ref":"master","sha":"786a19812494008d7c2e707feea084536ea28f90","user":{"login":"codice","id":2386734,"avatar_url":"https://avatars.githubusercontent.com/u/2386734?v=3","gravatar_id":"","url":"https://api.github.com/users/codice","html_url":"https://github.com/codice","followers_url":"https://api.github.com/users/codice/followers","following_url":"https://api.github.com/users/codice/following{/other_user}","gists_url":"https://api.github.com/users/codice/gists{/gist_id}","starred_url":"https://api.github.com/users/codice/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/codice/subscriptions","organizations_url":"https://api.github.com/users/codice/orgs","repos_url":"https://api.github.com/users/codice/repos","events_url":"https://api.github.com/users/codice/events{/privacy}","received_events_url":"https://api.github.com/users/codice/received_events","type":"Organization","site_admin":false},"repo":{"id":15173563,"name":"ddf","full_name":"codice/ddf","owner":{"login":"codice","id":2386734,"avatar_url":"https://avatars.githubusercontent.com/u/2386734?v=3","gravatar_id":"","url":"https://api.github.com/users/codice","html_url":"https://github.com/codice","followers_url":"https://api.github.com/users/codice/followers","following_url":"https://api.github.com/users/codice/following{/other_user}","gists_url":"https://api.github.com/users/codice/gists{/gist_id}","starred_url":"https://api.github.com/users/codice/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/codice/subscriptions","organizations_url":"https://api.github.com/users/codice/orgs","repos_url":"https://api.github.com/users/codice/repos","events_url":"https://api.github.com/users/codice/events{/privacy}","received_events_url":"https://api.github.com/users/codice/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/codice/ddf","description":"Distributed Data Framework - an open source, modular integration framework.","fork":false,"url":"https://api.github.com/repos/codice/ddf","forks_url":"https://api.github.com/repos/codice/ddf/forks","keys_url":"https://api.github.com/repos/codice/ddf/keys{/key_id}","collaborators_url":"https://api.github.com/repos/codice/ddf/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/codice/ddf/teams","hooks_url":"https://api.github.com/repos/codice/ddf/hooks","issue_events_url":"https://api.github.com/repos/codice/ddf/issues/events{/number}","events_url":"https://api.github.com/repos/codice/ddf/events","assignees_url":"https://api.github.com/repos/codice/ddf/assignees{/user}","branches_url":"https://api.github.com/repos/codice/ddf/branches{/branch}","tags_url":"https://api.github.com/repos/codice/ddf/tags","blobs_url":"https://api.github.com/repos/codice/ddf/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/codice/ddf/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/codice/ddf/git/refs{/sha}","trees_url":"https://api.github.com/repos/codice/ddf/git/trees{/sha}","statuses_url":"https://api.github.com/repos/codice/ddf/statuses/{sha}","languages_url":"https://api.github.com/repos/codice/ddf/languages","stargazers_url":"https://api.github.com/repos/codice/ddf/stargazers","contributors_url":"https://api.github.com/repos/codice/ddf/contributors","subscribers_url":"https://api.github.com/repos/codice/ddf/subscribers","subscription_url":"https://api.github.com/repos/codice/ddf/subscription","commits_url":"https://api.github.com/repos/codice/ddf/commits{/sha}","git_commits_url":"https://api.github.com/repos/codice/ddf/git/commits{/sha}","comments_url":"https://api.github.com/repos/codice/ddf/comments{/number}","issue_comment_url":"https://api.github.com/repos/codice/ddf/issues/comments{/number}","contents_url":"https://api.github.com/repos/codice/ddf/contents/{+path}","compare_url":"https://api.github.com/repos/codice/ddf/compare/{base}...{head}","merges_url":"https://api.github.com/repos/codice/ddf/merges","archive_url":"https://api.github.com/repos/codice/ddf/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/codice/ddf/downloads","issues_url":"https://api.github.com/repos/codice/ddf/issues{/number}","pulls_url":"https://api.github.com/repos/codice/ddf/pulls{/number}","milestones_url":"https://api.github.com/repos/codice/ddf/milestones{/number}","notifications_url":"https://api.github.com/repos/codice/ddf/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/codice/ddf/labels{/name}","releases_url":"https://api.github.com/repos/codice/ddf/releases{/id}","deployments_url":"https://api.github.com/repos/codice/ddf/deployments","created_at":"2013-12-13T20:38:28Z","updated_at":"2016-07-28T15:05:45Z","pushed_at":"2016-08-01T19:02:53Z","git_url":"git://github.com/codice/ddf.git","ssh_url":"git@github.com:codice/ddf.git","clone_url":"https://github.com/codice/ddf.git","svn_url":"https://github.com/codice/ddf","homepage":"http://ddf.codice.org","size":280902,"stargazers_count":40,"watchers_count":40,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":84,"mirror_url":null,"open_issues_count":24,"forks":84,"open_issues":24,"watchers":40,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/codice/ddf/pulls/1064"},"html":{"href":"https://github.com/codice/ddf/pull/1064"},"issue":{"href":"https://api.github.com/repos/codice/ddf/issues/1064"},"comments":{"href":"https://api.github.com/repos/codice/ddf/issues/1064/comments"},"review_comments":{"href":"https://api.github.com/repos/codice/ddf/pulls/1064/comments"},"review_comment":{"href":"https://api.github.com/repos/codice/ddf/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/codice/ddf/pulls/1064/commits"},"statuses":{"href":"https://api.github.com/repos/codice/ddf/statuses/e7149817c7616e9605985e798ec5de34af53d365"}}}}
{"actor":{"display_login":"figliold"}}