{"action":"created","issue":{"url":"https://api.github.com/repos/druid-io/druid/issues/3697","repository_url":"https://api.github.com/repos/druid-io/druid","labels_url":"https://api.github.com/repos/druid-io/druid/issues/3697/labels{/name}","comments_url":"https://api.github.com/repos/druid-io/druid/issues/3697/comments","events_url":"https://api.github.com/repos/druid-io/druid/issues/3697/events","html_url":"https://github.com/druid-io/druid/pull/3697","id":189894038,"number":3697,"title":"Lookup cache refactoring (the main part of #3667)","user":{"login":"leventov","id":609240,"avatar_url":"https://avatars.githubusercontent.com/u/609240?v=3","gravatar_id":"","url":"https://api.github.com/users/leventov","html_url":"https://github.com/leventov","followers_url":"https://api.github.com/users/leventov/followers","following_url":"https://api.github.com/users/leventov/following{/other_user}","gists_url":"https://api.github.com/users/leventov/gists{/gist_id}","starred_url":"https://api.github.com/users/leventov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/leventov/subscriptions","organizations_url":"https://api.github.com/users/leventov/orgs","repos_url":"https://api.github.com/users/leventov/repos","events_url":"https://api.github.com/users/leventov/events{/privacy}","received_events_url":"https://api.github.com/users/leventov/received_events","type":"User","site_admin":false},"labels":[{"id":321808195,"url":"https://api.github.com/repos/druid-io/druid/labels/Improvement","name":"Improvement","color":"0052cc","default":false}],"state":"open","locked":false,"assignee":{"login":"b-slim","id":7376396,"avatar_url":"https://avatars.githubusercontent.com/u/7376396?v=3","gravatar_id":"","url":"https://api.github.com/users/b-slim","html_url":"https://github.com/b-slim","followers_url":"https://api.github.com/users/b-slim/followers","following_url":"https://api.github.com/users/b-slim/following{/other_user}","gists_url":"https://api.github.com/users/b-slim/gists{/gist_id}","starred_url":"https://api.github.com/users/b-slim/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/b-slim/subscriptions","organizations_url":"https://api.github.com/users/b-slim/orgs","repos_url":"https://api.github.com/users/b-slim/repos","events_url":"https://api.github.com/users/b-slim/events{/privacy}","received_events_url":"https://api.github.com/users/b-slim/received_events","type":"User","site_admin":false},"assignees":[{"login":"b-slim","id":7376396,"avatar_url":"https://avatars.githubusercontent.com/u/7376396?v=3","gravatar_id":"","url":"https://api.github.com/users/b-slim","html_url":"https://github.com/b-slim","followers_url":"https://api.github.com/users/b-slim/followers","following_url":"https://api.github.com/users/b-slim/following{/other_user}","gists_url":"https://api.github.com/users/b-slim/gists{/gist_id}","starred_url":"https://api.github.com/users/b-slim/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/b-slim/subscriptions","organizations_url":"https://api.github.com/users/b-slim/orgs","repos_url":"https://api.github.com/users/b-slim/repos","events_url":"https://api.github.com/users/b-slim/events{/privacy}","received_events_url":"https://api.github.com/users/b-slim/received_events","type":"User","site_admin":false},{"login":"drcrallen","id":8213081,"avatar_url":"https://avatars.githubusercontent.com/u/8213081?v=3","gravatar_id":"","url":"https://api.github.com/users/drcrallen","html_url":"https://github.com/drcrallen","followers_url":"https://api.github.com/users/drcrallen/followers","following_url":"https://api.github.com/users/drcrallen/following{/other_user}","gists_url":"https://api.github.com/users/drcrallen/gists{/gist_id}","starred_url":"https://api.github.com/users/drcrallen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drcrallen/subscriptions","organizations_url":"https://api.github.com/users/drcrallen/orgs","repos_url":"https://api.github.com/users/drcrallen/repos","events_url":"https://api.github.com/users/drcrallen/events{/privacy}","received_events_url":"https://api.github.com/users/drcrallen/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/druid-io/druid/milestones/21","html_url":"https://github.com/druid-io/druid/milestone/21","labels_url":"https://api.github.com/repos/druid-io/druid/milestones/21/labels","id":1831000,"number":21,"title":"0.9.3","description":"","creator":{"login":"fjy","id":428325,"avatar_url":"https://avatars.githubusercontent.com/u/428325?v=3","gravatar_id":"","url":"https://api.github.com/users/fjy","html_url":"https://github.com/fjy","followers_url":"https://api.github.com/users/fjy/followers","following_url":"https://api.github.com/users/fjy/following{/other_user}","gists_url":"https://api.github.com/users/fjy/gists{/gist_id}","starred_url":"https://api.github.com/users/fjy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fjy/subscriptions","organizations_url":"https://api.github.com/users/fjy/orgs","repos_url":"https://api.github.com/users/fjy/repos","events_url":"https://api.github.com/users/fjy/events{/privacy}","received_events_url":"https://api.github.com/users/fjy/received_events","type":"User","site_admin":false},"open_issues":36,"closed_issues":124,"state":"open","created_at":"2016-06-15T22:35:58Z","updated_at":"2016-12-13T18:20:28Z","due_on":null,"closed_at":null},"comments":0,"created_at":"2016-11-17T00:20:34Z","updated_at":"2016-12-13T21:51:02Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/druid-io/druid/pulls/3697","html_url":"https://github.com/druid-io/druid/pull/3697","diff_url":"https://github.com/druid-io/druid/pull/3697.diff","patch_url":"https://github.com/druid-io/druid/pull/3697.patch"},"body":"The main part of druid-io/druid#3667\r\n\r\n**Why**\r\n1. Reduce locking. The prev. version of this subsystem has a lot of locks, concurrentHashMaps, etc. There were two deadlocks found in this subsystem in the last couple of months: #3459 and #3593\r\n2. Try to prevent memory leaks\r\n\r\n**Refactoring outline**\r\n1. Cache management is decoupled from scheduling and management of `Future`s, which update caches periodically. The latter is moved to `CacheScheduler` class from `NamespaceExtractionCacheManager`. `KafkaLookupExtractorFactory` now works with `NamespaceExtractionCacheManager` directly, because it doesn't need the \"scheduling\" functionality. `NamespaceLookupExtractorFactory` works with `CacheScheduler`.\r\n2. Previously lookup caches were identified by \"namespaceID\" strings, which were UUIDs, generated in runtime. Now caches (and futures) are identified by object identity. It eliminates the need for synchronization and even storing them in `NamespaceExtractionCacheManager` or `CacheScheduler`. So instead of (pseudocode):\r\n```java\r\n manager.schedule(namespaceID, namespace);\r\n Map cache = manager.getCacheMap(namespaceID);\r\n ...\r\n manager.delete(namespaceID);\r\n```\r\nthe new approach is:\r\n```java\r\nCacheHandler handler = manager.createCache();\r\nMap cache = handler.getCache();\r\n...\r\nhandler.close();\r\n```\r\nor, for `CacheScheduler`:\r\n```java\r\nCacheScheduler.Entry entry = scheduler.schedule(namespace);\r\nMap cache = entry.getCache();\r\n...\r\nentry.close();\r\n```\r\n3. `ExtractionNamespaceCacheFactory` and subclasses are changed so they now create a new cache and populate it only after they identify that they are not up-to-date. If they are up-to-date, they create nothing. Before, a cache was first created, and then passed to `populateCache()`, which could do nothing, because the lookup is up-to-date.\r\n\r\n**Result**\r\n 1.\r\n - Don't use intrinsic locks in this subsystem at all (before they were used in several places on several kinds of objects)\r\n - Just one `ConcurrentMap` left (not counting caches themselves), in `OnHeapNamespaceExtractionCacheManager` (before there were ~4 ConcurrentMaps in `NamespaceExtractionCacheManager` alone, also in OnHeap and OffHeap subclasses.\r\n 2. `CacheHandler`s and `CacheScheduler`'s Entries are provided with `Cleaner`s, i. e. if some `CacheHandler` or `CacheScheduler.Entry` is not closed, it will be identified, logged as error, and resources disposed.\r\n\r\n**Notes**\r\n - Some tests use PowerMock now, to be able to mock final classes.\r\n - Some classes (e. g. `NamespaceExtractionCacheManagerExecutorsTest`) were not renamed in order not to break the git history and minimise diff of this PR. But their names are quite misleading now. I'm going to rename the classes in a different PR (along with renames promised here: https://github.com/druid-io/druid/pull/3651#issue-186923217)\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/druid-io/druid/issues/comments/266873209","html_url":"https://github.com/druid-io/druid/pull/3697#issuecomment-266873209","issue_url":"https://api.github.com/repos/druid-io/druid/issues/3697","id":266873209,"user":{"login":"drcrallen","id":8213081,"avatar_url":"https://avatars.githubusercontent.com/u/8213081?v=3","gravatar_id":"","url":"https://api.github.com/users/drcrallen","html_url":"https://github.com/drcrallen","followers_url":"https://api.github.com/users/drcrallen/followers","following_url":"https://api.github.com/users/drcrallen/following{/other_user}","gists_url":"https://api.github.com/users/drcrallen/gists{/gist_id}","starred_url":"https://api.github.com/users/drcrallen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drcrallen/subscriptions","organizations_url":"https://api.github.com/users/drcrallen/orgs","repos_url":"https://api.github.com/users/drcrallen/repos","events_url":"https://api.github.com/users/drcrallen/events{/privacy}","received_events_url":"https://api.github.com/users/drcrallen/received_events","type":"User","site_admin":false},"created_at":"2016-12-13T21:51:02Z","updated_at":"2016-12-13T21:51:02Z","body":"Cool, 👍  after Travis CI"}}
{"actor":{"display_login":"drcrallen"}}