{"action":"opened","number":2337,"pull_request":{"url":"https://api.github.com/repos/KSP-CKAN/CKAN/pulls/2337","id":172121853,"html_url":"https://github.com/KSP-CKAN/CKAN/pull/2337","diff_url":"https://github.com/KSP-CKAN/CKAN/pull/2337.diff","patch_url":"https://github.com/KSP-CKAN/CKAN/pull/2337.patch","issue_url":"https://api.github.com/repos/KSP-CKAN/CKAN/issues/2337","number":2337,"state":"open","locked":false,"title":"Invalidate stale cached files from GitHub in Netkan","user":{"login":"HebaruSan","id":1559108,"avatar_url":"https://avatars2.githubusercontent.com/u/1559108?v=4","gravatar_id":"","url":"https://api.github.com/users/HebaruSan","html_url":"https://github.com/HebaruSan","followers_url":"https://api.github.com/users/HebaruSan/followers","following_url":"https://api.github.com/users/HebaruSan/following{/other_user}","gists_url":"https://api.github.com/users/HebaruSan/gists{/gist_id}","starred_url":"https://api.github.com/users/HebaruSan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HebaruSan/subscriptions","organizations_url":"https://api.github.com/users/HebaruSan/orgs","repos_url":"https://api.github.com/users/HebaruSan/repos","events_url":"https://api.github.com/users/HebaruSan/events{/privacy}","received_events_url":"https://api.github.com/users/HebaruSan/received_events","type":"User","site_admin":false},"body":"## Background\r\n\r\nBoth CKAN and Netkan cache module downloads to save network bandwidth and time. Downloaded files are placed in a cache folder with names like `A1B2C3D4-Modname-v1.0.zip`, with the first eight characters being the beginning of the download URL's SHA1 in hexadecimal.\r\n\r\nIn #2243, we added validation to make sure that invalid or corrupted files do not end up in the CKAN cache. This uses the `download_hash` property to check that the file we downloaded matches the file that Netkan indexed. Thanks to this change, `download_hash` properties must be correct and up to date for CKAN to work properly.\r\n\r\n## Problem\r\n\r\n1. GPP released a new version\r\n2. The netkan bot indexed it\r\n3. GPP replaced the previous download in-place with an updated version (fix to the KSP-AVC file)\r\n4. Users started getting mismatched hash errors that blocked installation of the new GPP\r\n5. The bot did not update `download_hash` property of the registry for the new version of GPP (it still matched the original upload)\r\n6. We manually set the new correct hash values (KSP-CKAN/CKAN-meta#1326)\r\n7. The bot reverted the hashes, and we had to fix them again\r\n8. GPP graciously released a _new_ new version to help us escape this cycle\r\n\r\n## Cause\r\n\r\nThe module cache currently assumes that a given URL will always correspond to the exact same download file. Netkan will download each URL once, and then every time it tries to access the same URL after that, it will re-use the same downloaded file. If the upstream file has changed, then the older version is _still_ used, and the `download_hash` values in CKAN's registry do not match the current download, so users can't install it.\r\n\r\nAfter thirty days, the file is removed from the cache, so it would be redownloaded at that time if needed.\r\n\r\n## Changes\r\n\r\nNow for GitHub downloads specifically, we check when the download file was last updated via the API. This feeds into new properties `GithubRelease.AssetUpdated` and `Model.Metadata.RemoteTimestamp`, which are passed to a new parameter of `Core.Net.NetFileCache.GetCachedFileName`, which is responsible for looking up cached files. If this function finds a file that looks like a match, it checks the local timestamp, and if it's older than the timestamp from GitHub, it deletes the file instead of using it, forcing a re-download. This will ensure that stale files do not remain in the cache if they are replaced on GitHub.\r\n\r\nWe could probably do this for other download hosts as well, but that is not included in this pull request. You can consider GitHub downloads to be a sort of pilot program for this functionality.","created_at":"2018-03-01T01:38:02Z","updated_at":"2018-03-01T01:38:02Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[{"id":122535234,"url":"https://api.github.com/repos/KSP-CKAN/CKAN/labels/Bug","name":"Bug","color":"e11d21","default":false},{"id":141308719,"url":"https://api.github.com/repos/KSP-CKAN/CKAN/labels/Netkan","name":"Netkan","color":"006b75","default":false},{"id":780580939,"url":"https://api.github.com/repos/KSP-CKAN/CKAN/labels/Network","name":"Network","color":"cdba96","default":false},{"id":139897064,"url":"https://api.github.com/repos/KSP-CKAN/CKAN/labels/Pull%20request","name":"Pull request","color":"1d76db","default":false}],"milestone":null,"commits_url":"https://api.github.com/repos/KSP-CKAN/CKAN/pulls/2337/commits","review_comments_url":"https://api.github.com/repos/KSP-CKAN/CKAN/pulls/2337/comments","review_comment_url":"https://api.github.com/repos/KSP-CKAN/CKAN/pulls/comments{/number}","comments_url":"https://api.github.com/repos/KSP-CKAN/CKAN/issues/2337/comments","statuses_url":"https://api.github.com/repos/KSP-CKAN/CKAN/statuses/d12ea9d95fdba6c21c269a7363aa889036dd7fbf","head":{"label":"HebaruSan:fix/netkan-cache-invalidation","ref":"fix/netkan-cache-invalidation","sha":"d12ea9d95fdba6c21c269a7363aa889036dd7fbf","user":{"login":"HebaruSan","id":1559108,"avatar_url":"https://avatars2.githubusercontent.com/u/1559108?v=4","gravatar_id":"","url":"https://api.github.com/users/HebaruSan","html_url":"https://github.com/HebaruSan","followers_url":"https://api.github.com/users/HebaruSan/followers","following_url":"https://api.github.com/users/HebaruSan/following{/other_user}","gists_url":"https://api.github.com/users/HebaruSan/gists{/gist_id}","starred_url":"https://api.github.com/users/HebaruSan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HebaruSan/subscriptions","organizations_url":"https://api.github.com/users/HebaruSan/orgs","repos_url":"https://api.github.com/users/HebaruSan/repos","events_url":"https://api.github.com/users/HebaruSan/events{/privacy}","received_events_url":"https://api.github.com/users/HebaruSan/received_events","type":"User","site_admin":false},"repo":{"id":101696387,"name":"CKAN","full_name":"HebaruSan/CKAN","owner":{"login":"HebaruSan","id":1559108,"avatar_url":"https://avatars2.githubusercontent.com/u/1559108?v=4","gravatar_id":"","url":"https://api.github.com/users/HebaruSan","html_url":"https://github.com/HebaruSan","followers_url":"https://api.github.com/users/HebaruSan/followers","following_url":"https://api.github.com/users/HebaruSan/following{/other_user}","gists_url":"https://api.github.com/users/HebaruSan/gists{/gist_id}","starred_url":"https://api.github.com/users/HebaruSan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HebaruSan/subscriptions","organizations_url":"https://api.github.com/users/HebaruSan/orgs","repos_url":"https://api.github.com/users/HebaruSan/repos","events_url":"https://api.github.com/users/HebaruSan/events{/privacy}","received_events_url":"https://api.github.com/users/HebaruSan/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/HebaruSan/CKAN","description":"The Comprehensive Kerbal Archive Network","fork":true,"url":"https://api.github.com/repos/HebaruSan/CKAN","forks_url":"https://api.github.com/repos/HebaruSan/CKAN/forks","keys_url":"https://api.github.com/repos/HebaruSan/CKAN/keys{/key_id}","collaborators_url":"https://api.github.com/repos/HebaruSan/CKAN/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/HebaruSan/CKAN/teams","hooks_url":"https://api.github.com/repos/HebaruSan/CKAN/hooks","issue_events_url":"https://api.github.com/repos/HebaruSan/CKAN/issues/events{/number}","events_url":"https://api.github.com/repos/HebaruSan/CKAN/events","assignees_url":"https://api.github.com/repos/HebaruSan/CKAN/assignees{/user}","branches_url":"https://api.github.com/repos/HebaruSan/CKAN/branches{/branch}","tags_url":"https://api.github.com/repos/HebaruSan/CKAN/tags","blobs_url":"https://api.github.com/repos/HebaruSan/CKAN/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/HebaruSan/CKAN/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/HebaruSan/CKAN/git/refs{/sha}","trees_url":"https://api.github.com/repos/HebaruSan/CKAN/git/trees{/sha}","statuses_url":"https://api.github.com/repos/HebaruSan/CKAN/statuses/{sha}","languages_url":"https://api.github.com/repos/HebaruSan/CKAN/languages","stargazers_url":"https://api.github.com/repos/HebaruSan/CKAN/stargazers","contributors_url":"https://api.github.com/repos/HebaruSan/CKAN/contributors","subscribers_url":"https://api.github.com/repos/HebaruSan/CKAN/subscribers","subscription_url":"https://api.github.com/repos/HebaruSan/CKAN/subscription","commits_url":"https://api.github.com/repos/HebaruSan/CKAN/commits{/sha}","git_commits_url":"https://api.github.com/repos/HebaruSan/CKAN/git/commits{/sha}","comments_url":"https://api.github.com/repos/HebaruSan/CKAN/comments{/number}","issue_comment_url":"https://api.github.com/repos/HebaruSan/CKAN/issues/comments{/number}","contents_url":"https://api.github.com/repos/HebaruSan/CKAN/contents/{+path}","compare_url":"https://api.github.com/repos/HebaruSan/CKAN/compare/{base}...{head}","merges_url":"https://api.github.com/repos/HebaruSan/CKAN/merges","archive_url":"https://api.github.com/repos/HebaruSan/CKAN/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/HebaruSan/CKAN/downloads","issues_url":"https://api.github.com/repos/HebaruSan/CKAN/issues{/number}","pulls_url":"https://api.github.com/repos/HebaruSan/CKAN/pulls{/number}","milestones_url":"https://api.github.com/repos/HebaruSan/CKAN/milestones{/number}","notifications_url":"https://api.github.com/repos/HebaruSan/CKAN/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/HebaruSan/CKAN/labels{/name}","releases_url":"https://api.github.com/repos/HebaruSan/CKAN/releases{/id}","deployments_url":"https://api.github.com/repos/HebaruSan/CKAN/deployments","created_at":"2017-08-28T23:16:51Z","updated_at":"2017-11-18T21:47:09Z","pushed_at":"2018-03-01T00:58:10Z","git_url":"git://github.com/HebaruSan/CKAN.git","ssh_url":"git@github.com:HebaruSan/CKAN.git","clone_url":"https://github.com/HebaruSan/CKAN.git","svn_url":"https://github.com/HebaruSan/CKAN","homepage":"","size":25284,"stargazers_count":0,"watchers_count":0,"language":"C#","has_issues":false,"has_projects":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":null,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"KSP-CKAN:master","ref":"master","sha":"b75a4502c1e59edb0fdc063a1326967beccb102c","user":{"login":"KSP-CKAN","id":9023970,"avatar_url":"https://avatars0.githubusercontent.com/u/9023970?v=4","gravatar_id":"","url":"https://api.github.com/users/KSP-CKAN","html_url":"https://github.com/KSP-CKAN","followers_url":"https://api.github.com/users/KSP-CKAN/followers","following_url":"https://api.github.com/users/KSP-CKAN/following{/other_user}","gists_url":"https://api.github.com/users/KSP-CKAN/gists{/gist_id}","starred_url":"https://api.github.com/users/KSP-CKAN/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/KSP-CKAN/subscriptions","organizations_url":"https://api.github.com/users/KSP-CKAN/orgs","repos_url":"https://api.github.com/users/KSP-CKAN/repos","events_url":"https://api.github.com/users/KSP-CKAN/events{/privacy}","received_events_url":"https://api.github.com/users/KSP-CKAN/received_events","type":"Organization","site_admin":false},"repo":{"id":23050594,"name":"CKAN","full_name":"KSP-CKAN/CKAN","owner":{"login":"KSP-CKAN","id":9023970,"avatar_url":"https://avatars0.githubusercontent.com/u/9023970?v=4","gravatar_id":"","url":"https://api.github.com/users/KSP-CKAN","html_url":"https://github.com/KSP-CKAN","followers_url":"https://api.github.com/users/KSP-CKAN/followers","following_url":"https://api.github.com/users/KSP-CKAN/following{/other_user}","gists_url":"https://api.github.com/users/KSP-CKAN/gists{/gist_id}","starred_url":"https://api.github.com/users/KSP-CKAN/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/KSP-CKAN/subscriptions","organizations_url":"https://api.github.com/users/KSP-CKAN/orgs","repos_url":"https://api.github.com/users/KSP-CKAN/repos","events_url":"https://api.github.com/users/KSP-CKAN/events{/privacy}","received_events_url":"https://api.github.com/users/KSP-CKAN/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/KSP-CKAN/CKAN","description":"The Comprehensive Kerbal Archive Network","fork":false,"url":"https://api.github.com/repos/KSP-CKAN/CKAN","forks_url":"https://api.github.com/repos/KSP-CKAN/CKAN/forks","keys_url":"https://api.github.com/repos/KSP-CKAN/CKAN/keys{/key_id}","collaborators_url":"https://api.github.com/repos/KSP-CKAN/CKAN/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/KSP-CKAN/CKAN/teams","hooks_url":"https://api.github.com/repos/KSP-CKAN/CKAN/hooks","issue_events_url":"https://api.github.com/repos/KSP-CKAN/CKAN/issues/events{/number}","events_url":"https://api.github.com/repos/KSP-CKAN/CKAN/events","assignees_url":"https://api.github.com/repos/KSP-CKAN/CKAN/assignees{/user}","branches_url":"https://api.github.com/repos/KSP-CKAN/CKAN/branches{/branch}","tags_url":"https://api.github.com/repos/KSP-CKAN/CKAN/tags","blobs_url":"https://api.github.com/repos/KSP-CKAN/CKAN/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/KSP-CKAN/CKAN/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/KSP-CKAN/CKAN/git/refs{/sha}","trees_url":"https://api.github.com/repos/KSP-CKAN/CKAN/git/trees{/sha}","statuses_url":"https://api.github.com/repos/KSP-CKAN/CKAN/statuses/{sha}","languages_url":"https://api.github.com/repos/KSP-CKAN/CKAN/languages","stargazers_url":"https://api.github.com/repos/KSP-CKAN/CKAN/stargazers","contributors_url":"https://api.github.com/repos/KSP-CKAN/CKAN/contributors","subscribers_url":"https://api.github.com/repos/KSP-CKAN/CKAN/subscribers","subscription_url":"https://api.github.com/repos/KSP-CKAN/CKAN/subscription","commits_url":"https://api.github.com/repos/KSP-CKAN/CKAN/commits{/sha}","git_commits_url":"https://api.github.com/repos/KSP-CKAN/CKAN/git/commits{/sha}","comments_url":"https://api.github.com/repos/KSP-CKAN/CKAN/comments{/number}","issue_comment_url":"https://api.github.com/repos/KSP-CKAN/CKAN/issues/comments{/number}","contents_url":"https://api.github.com/repos/KSP-CKAN/CKAN/contents/{+path}","compare_url":"https://api.github.com/repos/KSP-CKAN/CKAN/compare/{base}...{head}","merges_url":"https://api.github.com/repos/KSP-CKAN/CKAN/merges","archive_url":"https://api.github.com/repos/KSP-CKAN/CKAN/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/KSP-CKAN/CKAN/downloads","issues_url":"https://api.github.com/repos/KSP-CKAN/CKAN/issues{/number}","pulls_url":"https://api.github.com/repos/KSP-CKAN/CKAN/pulls{/number}","milestones_url":"https://api.github.com/repos/KSP-CKAN/CKAN/milestones{/number}","notifications_url":"https://api.github.com/repos/KSP-CKAN/CKAN/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/KSP-CKAN/CKAN/labels{/name}","releases_url":"https://api.github.com/repos/KSP-CKAN/CKAN/releases{/id}","deployments_url":"https://api.github.com/repos/KSP-CKAN/CKAN/deployments","created_at":"2014-08-17T21:11:54Z","updated_at":"2018-02-28T18:07:33Z","pushed_at":"2018-02-28T00:59:21Z","git_url":"git://github.com/KSP-CKAN/CKAN.git","ssh_url":"git@github.com:KSP-CKAN/CKAN.git","clone_url":"https://github.com/KSP-CKAN/CKAN.git","svn_url":"https://github.com/KSP-CKAN/CKAN","homepage":"http://forum.kerbalspaceprogram.com/index.php?/topic/154922-ckan","size":25397,"stargazers_count":665,"watchers_count":665,"language":"C#","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":188,"mirror_url":null,"archived":false,"open_issues_count":256,"license":{"key":"other","name":"Other","spdx_id":null,"url":null},"forks":188,"open_issues":256,"watchers":665,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/KSP-CKAN/CKAN/pulls/2337"},"html":{"href":"https://github.com/KSP-CKAN/CKAN/pull/2337"},"issue":{"href":"https://api.github.com/repos/KSP-CKAN/CKAN/issues/2337"},"comments":{"href":"https://api.github.com/repos/KSP-CKAN/CKAN/issues/2337/comments"},"review_comments":{"href":"https://api.github.com/repos/KSP-CKAN/CKAN/pulls/2337/comments"},"review_comment":{"href":"https://api.github.com/repos/KSP-CKAN/CKAN/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/KSP-CKAN/CKAN/pulls/2337/commits"},"statuses":{"href":"https://api.github.com/repos/KSP-CKAN/CKAN/statuses/d12ea9d95fdba6c21c269a7363aa889036dd7fbf"}},"author_association":"MEMBER","merged":false,"mergeable":null,"rebaseable":null,"mergeable_state":"unknown","merged_by":null,"comments":0,"review_comments":0,"maintainer_can_modify":true,"commits":1,"additions":75,"deletions":37,"changed_files":14}}
{"actor":{"display_login":"HebaruSan"}}