{"action":"closed","number":2127,"pull_request":{"url":"https://api.github.com/repos/dCache/dcache/pulls/2127","id":56668119,"html_url":"https://github.com/dCache/dcache/pull/2127","diff_url":"https://github.com/dCache/dcache/pull/2127.diff","patch_url":"https://github.com/dCache/dcache/pull/2127.patch","issue_url":"https://api.github.com/repos/dCache/dcache/issues/2127","number":2127,"state":"closed","locked":false,"title":"(2.12) replica manager (old): fix countable logic when rescanning poo…","user":{"login":"alrossi","id":3985564,"avatar_url":"https://avatars.githubusercontent.com/u/3985564?v=3","gravatar_id":"","url":"https://api.github.com/users/alrossi","html_url":"https://github.com/alrossi","followers_url":"https://api.github.com/users/alrossi/followers","following_url":"https://api.github.com/users/alrossi/following{/other_user}","gists_url":"https://api.github.com/users/alrossi/gists{/gist_id}","starred_url":"https://api.github.com/users/alrossi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alrossi/subscriptions","organizations_url":"https://api.github.com/users/alrossi/orgs","repos_url":"https://api.github.com/users/alrossi/repos","events_url":"https://api.github.com/users/alrossi/events{/privacy}","received_events_url":"https://api.github.com/users/alrossi/received_events","type":"User","site_admin":false},"body":"…l repository\r\n\r\nMotivation:\r\n\r\nUsers of the old replica manager have reported inconsistent behavior regarding source files\r\nwhich, depending on the dCache configuration, can cause thrashing or unbounded copy-retries.\r\n\r\nThe problem has its origin in the fact that the replica manager was designed to handle only\r\nprecious files, but since then there has been a change such that the RetentionPolicy\r\nand AccessLatency tags are mapped to the attributes precious, cached and sticky.  The\r\nmanager was apparently not adjusted to take into consideration this change.  Inconsistency\r\ncan thus arise if the resilient pools are linked to storage units which are in turn linked to\r\ndirectories where the RetentionPolicy is not CUSTODIAL [=> precious], because the\r\noriginal/source copy will land on the pool as \"cached+sticky\", regardless of whether the\r\nvalue of the pool.lfs property is 'precious' (i.e., no-flush) for that pool.\r\n\r\nThe initial replication proceeds as normal in this case, because the entry method simply\r\ntakes all files from a resilient pool and marks them as 'countable', the prerequisite for\r\nbeing included in the hard replica count.  However, if for any reason (such as a pool state change,\r\nreboot of the replica manager, etc.) a rescan of the pool occurs, and the pool contains such\r\noriginals, the replicas database entry for this file now becomes marked countable=false,\r\nbecause it is cached, not precious (the scan uses a different method from the\r\ninitial one).  Even though the required number of copies may still exist on readable pools,\r\nthe replica manager nevertheless thinks one is missing and proceeds to attempt a p2p.  This may\r\nlead, as in the reported ticket, to repeated failures.  In any case, it leads to more work than is\r\nneeded, since there is no danger that a cached+sticky (i.e., system sticky) file will be removed and thus\r\nshould count as a hard replica.\r\n\r\nModification:\r\n\r\nThe method used to process pool repository entries into the replicas table now considers\r\nboth precious and cached+sticky copies as 'countable'.\r\n\r\nResult:\r\n\r\nInconsistent counting of hard replicas is eliminated via a minimal code intervention.\r\n\r\nNote 1:  Soft replicas (cached but not sticky) are still countable='f', since they are subject to removal by the sweeper.\r\nNote 2:  The other solution to this issue is simply to inform our users that RetentionPolicy should always be CUSTODIAL\r\n         for the directories linked to resilient pools.   This, however, still would require the site to (a) propagate\r\n         the tag change throughout the directory tree where needed; (b) change the tags for the files in these directories\r\n         in the namespace; (b) modify those files to countable='t' in the replicas table.   This could be a complicated and\r\n         lengthy process for a big installation, especially one like BNL with 6 separate replication managers.\r\n\r\nTarget: 2.12\r\nRT: 8871  (replication problem: bitmask=258 and countable=false for a file copy in Replication Manager DB)\r\nRequire-notes: yes\r\nRequire-book: yes\r\nAcked-by: Dmitry\r\n\r\nRELEASE NOTES:  Fixes a bug where source files which are written to directories with REPLICA ONLINE tags are\r\n                at first marked countable but then on rescan of the pool are marked not countable.  The fix\r\n                makes 'cached' + (system-owned)'sticky' files the equivalent of precious files on pools\r\n                with pool.lfs='precious'.\r\n\r\nConflicts:\r\n\r\n\tmodules/dcache/src/main/java/diskCacheV111/replicaManager/ReplicaDbV1.java","created_at":"2016-01-20T21:14:22Z","updated_at":"2016-01-22T14:10:53Z","closed_at":"2016-01-22T14:10:53Z","merged_at":null,"merge_commit_sha":"e37c9f293dbdbb1c3ff3db52e12cb65a6a2df472","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/dCache/dcache/pulls/2127/commits","review_comments_url":"https://api.github.com/repos/dCache/dcache/pulls/2127/comments","review_comment_url":"https://api.github.com/repos/dCache/dcache/pulls/comments{/number}","comments_url":"https://api.github.com/repos/dCache/dcache/issues/2127/comments","statuses_url":"https://api.github.com/repos/dCache/dcache/statuses/4a416005bbde8262cbdf3be73bbaf7921d21a103","head":{"label":"alrossi:fix/2.12/rm-countable-files","ref":"fix/2.12/rm-countable-files","sha":"4a416005bbde8262cbdf3be73bbaf7921d21a103","user":{"login":"alrossi","id":3985564,"avatar_url":"https://avatars.githubusercontent.com/u/3985564?v=3","gravatar_id":"","url":"https://api.github.com/users/alrossi","html_url":"https://github.com/alrossi","followers_url":"https://api.github.com/users/alrossi/followers","following_url":"https://api.github.com/users/alrossi/following{/other_user}","gists_url":"https://api.github.com/users/alrossi/gists{/gist_id}","starred_url":"https://api.github.com/users/alrossi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alrossi/subscriptions","organizations_url":"https://api.github.com/users/alrossi/orgs","repos_url":"https://api.github.com/users/alrossi/repos","events_url":"https://api.github.com/users/alrossi/events{/privacy}","received_events_url":"https://api.github.com/users/alrossi/received_events","type":"User","site_admin":false},"repo":{"id":9675694,"name":"dcache-server","full_name":"alrossi/dcache-server","owner":{"login":"alrossi","id":3985564,"avatar_url":"https://avatars.githubusercontent.com/u/3985564?v=3","gravatar_id":"","url":"https://api.github.com/users/alrossi","html_url":"https://github.com/alrossi","followers_url":"https://api.github.com/users/alrossi/followers","following_url":"https://api.github.com/users/alrossi/following{/other_user}","gists_url":"https://api.github.com/users/alrossi/gists{/gist_id}","starred_url":"https://api.github.com/users/alrossi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alrossi/subscriptions","organizations_url":"https://api.github.com/users/alrossi/orgs","repos_url":"https://api.github.com/users/alrossi/repos","events_url":"https://api.github.com/users/alrossi/events{/privacy}","received_events_url":"https://api.github.com/users/alrossi/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/alrossi/dcache-server","description":"dCache source code","fork":true,"url":"https://api.github.com/repos/alrossi/dcache-server","forks_url":"https://api.github.com/repos/alrossi/dcache-server/forks","keys_url":"https://api.github.com/repos/alrossi/dcache-server/keys{/key_id}","collaborators_url":"https://api.github.com/repos/alrossi/dcache-server/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/alrossi/dcache-server/teams","hooks_url":"https://api.github.com/repos/alrossi/dcache-server/hooks","issue_events_url":"https://api.github.com/repos/alrossi/dcache-server/issues/events{/number}","events_url":"https://api.github.com/repos/alrossi/dcache-server/events","assignees_url":"https://api.github.com/repos/alrossi/dcache-server/assignees{/user}","branches_url":"https://api.github.com/repos/alrossi/dcache-server/branches{/branch}","tags_url":"https://api.github.com/repos/alrossi/dcache-server/tags","blobs_url":"https://api.github.com/repos/alrossi/dcache-server/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/alrossi/dcache-server/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/alrossi/dcache-server/git/refs{/sha}","trees_url":"https://api.github.com/repos/alrossi/dcache-server/git/trees{/sha}","statuses_url":"https://api.github.com/repos/alrossi/dcache-server/statuses/{sha}","languages_url":"https://api.github.com/repos/alrossi/dcache-server/languages","stargazers_url":"https://api.github.com/repos/alrossi/dcache-server/stargazers","contributors_url":"https://api.github.com/repos/alrossi/dcache-server/contributors","subscribers_url":"https://api.github.com/repos/alrossi/dcache-server/subscribers","subscription_url":"https://api.github.com/repos/alrossi/dcache-server/subscription","commits_url":"https://api.github.com/repos/alrossi/dcache-server/commits{/sha}","git_commits_url":"https://api.github.com/repos/alrossi/dcache-server/git/commits{/sha}","comments_url":"https://api.github.com/repos/alrossi/dcache-server/comments{/number}","issue_comment_url":"https://api.github.com/repos/alrossi/dcache-server/issues/comments{/number}","contents_url":"https://api.github.com/repos/alrossi/dcache-server/contents/{+path}","compare_url":"https://api.github.com/repos/alrossi/dcache-server/compare/{base}...{head}","merges_url":"https://api.github.com/repos/alrossi/dcache-server/merges","archive_url":"https://api.github.com/repos/alrossi/dcache-server/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/alrossi/dcache-server/downloads","issues_url":"https://api.github.com/repos/alrossi/dcache-server/issues{/number}","pulls_url":"https://api.github.com/repos/alrossi/dcache-server/pulls{/number}","milestones_url":"https://api.github.com/repos/alrossi/dcache-server/milestones{/number}","notifications_url":"https://api.github.com/repos/alrossi/dcache-server/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/alrossi/dcache-server/labels{/name}","releases_url":"https://api.github.com/repos/alrossi/dcache-server/releases{/id}","deployments_url":"https://api.github.com/repos/alrossi/dcache-server/deployments","created_at":"2013-04-25T16:02:34Z","updated_at":"2015-06-15T14:14:36Z","pushed_at":"2016-01-22T14:10:53Z","git_url":"git://github.com/alrossi/dcache-server.git","ssh_url":"git@github.com:alrossi/dcache-server.git","clone_url":"https://github.com/alrossi/dcache-server.git","svn_url":"https://github.com/alrossi/dcache-server","homepage":null,"size":342845,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"dCache:2.12","ref":"2.12","sha":"995bb3a4c7461376cb13a980f341cb063b583944","user":{"login":"dCache","id":1384620,"avatar_url":"https://avatars.githubusercontent.com/u/1384620?v=3","gravatar_id":"","url":"https://api.github.com/users/dCache","html_url":"https://github.com/dCache","followers_url":"https://api.github.com/users/dCache/followers","following_url":"https://api.github.com/users/dCache/following{/other_user}","gists_url":"https://api.github.com/users/dCache/gists{/gist_id}","starred_url":"https://api.github.com/users/dCache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dCache/subscriptions","organizations_url":"https://api.github.com/users/dCache/orgs","repos_url":"https://api.github.com/users/dCache/repos","events_url":"https://api.github.com/users/dCache/events{/privacy}","received_events_url":"https://api.github.com/users/dCache/received_events","type":"Organization","site_admin":false},"repo":{"id":9113580,"name":"dcache","full_name":"dCache/dcache","owner":{"login":"dCache","id":1384620,"avatar_url":"https://avatars.githubusercontent.com/u/1384620?v=3","gravatar_id":"","url":"https://api.github.com/users/dCache","html_url":"https://github.com/dCache","followers_url":"https://api.github.com/users/dCache/followers","following_url":"https://api.github.com/users/dCache/following{/other_user}","gists_url":"https://api.github.com/users/dCache/gists{/gist_id}","starred_url":"https://api.github.com/users/dCache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dCache/subscriptions","organizations_url":"https://api.github.com/users/dCache/orgs","repos_url":"https://api.github.com/users/dCache/repos","events_url":"https://api.github.com/users/dCache/events{/privacy}","received_events_url":"https://api.github.com/users/dCache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/dCache/dcache","description":"dCache - a system for storing and retrieving huge amounts of data, distributed among a large number of heterogenous server nodes, under a single virtual filesystem tree with a variety of standard access methods","fork":false,"url":"https://api.github.com/repos/dCache/dcache","forks_url":"https://api.github.com/repos/dCache/dcache/forks","keys_url":"https://api.github.com/repos/dCache/dcache/keys{/key_id}","collaborators_url":"https://api.github.com/repos/dCache/dcache/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/dCache/dcache/teams","hooks_url":"https://api.github.com/repos/dCache/dcache/hooks","issue_events_url":"https://api.github.com/repos/dCache/dcache/issues/events{/number}","events_url":"https://api.github.com/repos/dCache/dcache/events","assignees_url":"https://api.github.com/repos/dCache/dcache/assignees{/user}","branches_url":"https://api.github.com/repos/dCache/dcache/branches{/branch}","tags_url":"https://api.github.com/repos/dCache/dcache/tags","blobs_url":"https://api.github.com/repos/dCache/dcache/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/dCache/dcache/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/dCache/dcache/git/refs{/sha}","trees_url":"https://api.github.com/repos/dCache/dcache/git/trees{/sha}","statuses_url":"https://api.github.com/repos/dCache/dcache/statuses/{sha}","languages_url":"https://api.github.com/repos/dCache/dcache/languages","stargazers_url":"https://api.github.com/repos/dCache/dcache/stargazers","contributors_url":"https://api.github.com/repos/dCache/dcache/contributors","subscribers_url":"https://api.github.com/repos/dCache/dcache/subscribers","subscription_url":"https://api.github.com/repos/dCache/dcache/subscription","commits_url":"https://api.github.com/repos/dCache/dcache/commits{/sha}","git_commits_url":"https://api.github.com/repos/dCache/dcache/git/commits{/sha}","comments_url":"https://api.github.com/repos/dCache/dcache/comments{/number}","issue_comment_url":"https://api.github.com/repos/dCache/dcache/issues/comments{/number}","contents_url":"https://api.github.com/repos/dCache/dcache/contents/{+path}","compare_url":"https://api.github.com/repos/dCache/dcache/compare/{base}...{head}","merges_url":"https://api.github.com/repos/dCache/dcache/merges","archive_url":"https://api.github.com/repos/dCache/dcache/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/dCache/dcache/downloads","issues_url":"https://api.github.com/repos/dCache/dcache/issues{/number}","pulls_url":"https://api.github.com/repos/dCache/dcache/pulls{/number}","milestones_url":"https://api.github.com/repos/dCache/dcache/milestones{/number}","notifications_url":"https://api.github.com/repos/dCache/dcache/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/dCache/dcache/labels{/name}","releases_url":"https://api.github.com/repos/dCache/dcache/releases{/id}","deployments_url":"https://api.github.com/repos/dCache/dcache/deployments","created_at":"2013-03-30T08:58:49Z","updated_at":"2016-01-12T11:41:44Z","pushed_at":"2016-01-22T11:34:18Z","git_url":"git://github.com/dCache/dcache.git","ssh_url":"git@github.com:dCache/dcache.git","clone_url":"https://github.com/dCache/dcache.git","svn_url":"https://github.com/dCache/dcache","homepage":"http://www.dcache.org","size":345278,"stargazers_count":29,"watchers_count":29,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":29,"mirror_url":null,"open_issues_count":47,"forks":29,"open_issues":47,"watchers":29,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/dCache/dcache/pulls/2127"},"html":{"href":"https://github.com/dCache/dcache/pull/2127"},"issue":{"href":"https://api.github.com/repos/dCache/dcache/issues/2127"},"comments":{"href":"https://api.github.com/repos/dCache/dcache/issues/2127/comments"},"review_comments":{"href":"https://api.github.com/repos/dCache/dcache/pulls/2127/comments"},"review_comment":{"href":"https://api.github.com/repos/dCache/dcache/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/dCache/dcache/pulls/2127/commits"},"statuses":{"href":"https://api.github.com/repos/dCache/dcache/statuses/4a416005bbde8262cbdf3be73bbaf7921d21a103"}},"merged":false,"mergeable":true,"mergeable_state":"clean","merged_by":null,"comments":0,"review_comments":0,"commits":1,"additions":4,"deletions":5,"changed_files":1}}