{"action":"created","comment":{"url":"https://api.github.com/repos/druid-io/druid/pulls/comments/72848144","id":72848144,"diff_hunk":"@@ -0,0 +1,122 @@\n+# Cached Lookup Module\n+\n+## Description\n+This module provides a one to one caching mechanism for a specific lookup table source like RDBMS or distributed file systems.\n+The main goal of this cache is to speed up the access to a high latency lookup sources and to provide a caching isolation for every lookup source.\n+Thus user can define various caching strategies or and implementation per lookup, even if the source is the same.\n+\n+## Architecture\n+Generally speaking this module can be divided into two main component, namely, the data fetcher layer and caching layer.\n+\n+### Data Fetcher layer:\n+\n+First part is the data fetcher layer API `DataFetcher`, that exposes a set of fetch methods to fetch data from the actual Lookup dimension source.\n+For instance `JdbcDataFetcher` provides an implementation of `DataFetcher` that can be used to fetch key/value from a RDBMS via JDBC driver.\n+If you need new type of data fetcher, all you need to do, is to implement the interface `DataFetcher` and load it via another druid module.\n+### Caching layer\n+\n+This extension comes with two different caching strategies. First strategy is a poll based and the second is a load based.\n+#### Poll lookup cache:\n+\n+The poll strategy cache strategy will fetch and swap all the pair of key/values periodically from the lookup source.\n+Hence, user should make sure that the cache can fit all the data. \n+The current implementation provides 2 type of poll cache, the first is onheap (uses immutable map), while the second uses MapBD based offheap map.\n+User can also implement a different lookup polling cache by implementing `PollingCacheFactory` and `PollingCache` interfaces. \n+\n+#### Loading lookup:\n+Loading cache strategy will load the key\\value pair upon request on the key it self, the general algorithm is load key if absent.\n+Once the key/value  pair is loaded eviction will occur according to the cache eviction policy.\n+This module comes with two loading lookup implementation, the first is onheap and based on Guava cache implementation, the second is MapDB offheap implementation.\n+Both implementations offer various eviction strategies.\n+Same for Loading cache, user can implement a new type of loading cache by implementing `LookupLoadingCache` interface.","path":"docs/content/development/extensions-core/druid-lookups.md","position":31,"original_position":31,"commit_id":"0785b2f275d5b83053b759430e1ff63f9c5c763c","original_commit_id":"0785b2f275d5b83053b759430e1ff63f9c5c763c","user":{"login":"b-slim","id":7376396,"avatar_url":"https://avatars.githubusercontent.com/u/7376396?v=3","gravatar_id":"","url":"https://api.github.com/users/b-slim","html_url":"https://github.com/b-slim","followers_url":"https://api.github.com/users/b-slim/followers","following_url":"https://api.github.com/users/b-slim/following{/other_user}","gists_url":"https://api.github.com/users/b-slim/gists{/gist_id}","starred_url":"https://api.github.com/users/b-slim/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/b-slim/subscriptions","organizations_url":"https://api.github.com/users/b-slim/orgs","repos_url":"https://api.github.com/users/b-slim/repos","events_url":"https://api.github.com/users/b-slim/events{/privacy}","received_events_url":"https://api.github.com/users/b-slim/received_events","type":"User","site_admin":false},"body":"done","created_at":"2016-07-29T19:30:39Z","updated_at":"2016-07-29T19:30:39Z","html_url":"https://github.com/druid-io/druid/pull/2819#discussion_r72848144","pull_request_url":"https://api.github.com/repos/druid-io/druid/pulls/2819","_links":{"self":{"href":"https://api.github.com/repos/druid-io/druid/pulls/comments/72848144"},"html":{"href":"https://github.com/druid-io/druid/pull/2819#discussion_r72848144"},"pull_request":{"href":"https://api.github.com/repos/druid-io/druid/pulls/2819"}}},"pull_request":{"url":"https://api.github.com/repos/druid-io/druid/pulls/2819","id":66146528,"html_url":"https://github.com/druid-io/druid/pull/2819","diff_url":"https://github.com/druid-io/druid/pull/2819.diff","patch_url":"https://github.com/druid-io/druid/pull/2819.patch","issue_url":"https://api.github.com/repos/druid-io/druid/issues/2819","number":2819,"state":"open","locked":false,"title":"[QTL]Cached lookup module for JDBC connectors.","user":{"login":"b-slim","id":7376396,"avatar_url":"https://avatars.githubusercontent.com/u/7376396?v=3","gravatar_id":"","url":"https://api.github.com/users/b-slim","html_url":"https://github.com/b-slim","followers_url":"https://api.github.com/users/b-slim/followers","following_url":"https://api.github.com/users/b-slim/following{/other_user}","gists_url":"https://api.github.com/users/b-slim/gists{/gist_id}","starred_url":"https://api.github.com/users/b-slim/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/b-slim/subscriptions","organizations_url":"https://api.github.com/users/b-slim/orgs","repos_url":"https://api.github.com/users/b-slim/repos","events_url":"https://api.github.com/users/b-slim/events{/privacy}","received_events_url":"https://api.github.com/users/b-slim/received_events","type":"User","site_admin":false},"body":"In This PR introduces a cached Lookup leveraging eviction and various caching strategies.\r\nThis Lookup implementation is meant to be user with large Lookup tables that need evictions.\r\n\r\n \r\n","created_at":"2016-04-12T12:11:27Z","updated_at":"2016-07-29T19:30:39Z","closed_at":null,"merged_at":null,"merge_commit_sha":"df0e0d71b41b6749f6145b52b803f55b06d02cb0","assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/druid-io/druid/milestones/20","html_url":"https://github.com/druid-io/druid/milestones/0.9.2","labels_url":"https://api.github.com/repos/druid-io/druid/milestones/20/labels","id":1571409,"number":20,"title":"0.9.2","description":"","creator":{"login":"fjy","id":428325,"avatar_url":"https://avatars.githubusercontent.com/u/428325?v=3","gravatar_id":"","url":"https://api.github.com/users/fjy","html_url":"https://github.com/fjy","followers_url":"https://api.github.com/users/fjy/followers","following_url":"https://api.github.com/users/fjy/following{/other_user}","gists_url":"https://api.github.com/users/fjy/gists{/gist_id}","starred_url":"https://api.github.com/users/fjy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fjy/subscriptions","organizations_url":"https://api.github.com/users/fjy/orgs","repos_url":"https://api.github.com/users/fjy/repos","events_url":"https://api.github.com/users/fjy/events{/privacy}","received_events_url":"https://api.github.com/users/fjy/received_events","type":"User","site_admin":false},"open_issues":25,"closed_issues":93,"state":"open","created_at":"2016-02-08T22:01:21Z","updated_at":"2016-07-29T17:19:55Z","due_on":null,"closed_at":null},"commits_url":"https://api.github.com/repos/druid-io/druid/pulls/2819/commits","review_comments_url":"https://api.github.com/repos/druid-io/druid/pulls/2819/comments","review_comment_url":"https://api.github.com/repos/druid-io/druid/pulls/comments{/number}","comments_url":"https://api.github.com/repos/druid-io/druid/issues/2819/comments","statuses_url":"https://api.github.com/repos/druid-io/druid/statuses/0785b2f275d5b83053b759430e1ff63f9c5c763c","head":{"label":"b-slim:new_lookup_module","ref":"new_lookup_module","sha":"0785b2f275d5b83053b759430e1ff63f9c5c763c","user":{"login":"b-slim","id":7376396,"avatar_url":"https://avatars.githubusercontent.com/u/7376396?v=3","gravatar_id":"","url":"https://api.github.com/users/b-slim","html_url":"https://github.com/b-slim","followers_url":"https://api.github.com/users/b-slim/followers","following_url":"https://api.github.com/users/b-slim/following{/other_user}","gists_url":"https://api.github.com/users/b-slim/gists{/gist_id}","starred_url":"https://api.github.com/users/b-slim/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/b-slim/subscriptions","organizations_url":"https://api.github.com/users/b-slim/orgs","repos_url":"https://api.github.com/users/b-slim/repos","events_url":"https://api.github.com/users/b-slim/events{/privacy}","received_events_url":"https://api.github.com/users/b-slim/received_events","type":"User","site_admin":false},"repo":{"id":26687338,"name":"druid","full_name":"b-slim/druid","owner":{"login":"b-slim","id":7376396,"avatar_url":"https://avatars.githubusercontent.com/u/7376396?v=3","gravatar_id":"","url":"https://api.github.com/users/b-slim","html_url":"https://github.com/b-slim","followers_url":"https://api.github.com/users/b-slim/followers","following_url":"https://api.github.com/users/b-slim/following{/other_user}","gists_url":"https://api.github.com/users/b-slim/gists{/gist_id}","starred_url":"https://api.github.com/users/b-slim/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/b-slim/subscriptions","organizations_url":"https://api.github.com/users/b-slim/orgs","repos_url":"https://api.github.com/users/b-slim/repos","events_url":"https://api.github.com/users/b-slim/events{/privacy}","received_events_url":"https://api.github.com/users/b-slim/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/b-slim/druid","description":"Real²time Exploratory Analytics on Large Datasets","fork":true,"url":"https://api.github.com/repos/b-slim/druid","forks_url":"https://api.github.com/repos/b-slim/druid/forks","keys_url":"https://api.github.com/repos/b-slim/druid/keys{/key_id}","collaborators_url":"https://api.github.com/repos/b-slim/druid/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/b-slim/druid/teams","hooks_url":"https://api.github.com/repos/b-slim/druid/hooks","issue_events_url":"https://api.github.com/repos/b-slim/druid/issues/events{/number}","events_url":"https://api.github.com/repos/b-slim/druid/events","assignees_url":"https://api.github.com/repos/b-slim/druid/assignees{/user}","branches_url":"https://api.github.com/repos/b-slim/druid/branches{/branch}","tags_url":"https://api.github.com/repos/b-slim/druid/tags","blobs_url":"https://api.github.com/repos/b-slim/druid/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/b-slim/druid/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/b-slim/druid/git/refs{/sha}","trees_url":"https://api.github.com/repos/b-slim/druid/git/trees{/sha}","statuses_url":"https://api.github.com/repos/b-slim/druid/statuses/{sha}","languages_url":"https://api.github.com/repos/b-slim/druid/languages","stargazers_url":"https://api.github.com/repos/b-slim/druid/stargazers","contributors_url":"https://api.github.com/repos/b-slim/druid/contributors","subscribers_url":"https://api.github.com/repos/b-slim/druid/subscribers","subscription_url":"https://api.github.com/repos/b-slim/druid/subscription","commits_url":"https://api.github.com/repos/b-slim/druid/commits{/sha}","git_commits_url":"https://api.github.com/repos/b-slim/druid/git/commits{/sha}","comments_url":"https://api.github.com/repos/b-slim/druid/comments{/number}","issue_comment_url":"https://api.github.com/repos/b-slim/druid/issues/comments{/number}","contents_url":"https://api.github.com/repos/b-slim/druid/contents/{+path}","compare_url":"https://api.github.com/repos/b-slim/druid/compare/{base}...{head}","merges_url":"https://api.github.com/repos/b-slim/druid/merges","archive_url":"https://api.github.com/repos/b-slim/druid/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/b-slim/druid/downloads","issues_url":"https://api.github.com/repos/b-slim/druid/issues{/number}","pulls_url":"https://api.github.com/repos/b-slim/druid/pulls{/number}","milestones_url":"https://api.github.com/repos/b-slim/druid/milestones{/number}","notifications_url":"https://api.github.com/repos/b-slim/druid/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/b-slim/druid/labels{/name}","releases_url":"https://api.github.com/repos/b-slim/druid/releases{/id}","deployments_url":"https://api.github.com/repos/b-slim/druid/deployments","created_at":"2014-11-15T17:20:52Z","updated_at":"2016-02-09T13:51:50Z","pushed_at":"2016-06-05T19:33:16Z","git_url":"git://github.com/b-slim/druid.git","ssh_url":"git@github.com:b-slim/druid.git","clone_url":"https://github.com/b-slim/druid.git","svn_url":"https://github.com/b-slim/druid","homepage":"http://druid.io","size":106571,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":0,"forks":1,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"druid-io:master","ref":"master","sha":"c738c0e1cd402885511e2b658b7b40770fd18a36","user":{"login":"druid-io","id":4203474,"avatar_url":"https://avatars.githubusercontent.com/u/4203474?v=3","gravatar_id":"","url":"https://api.github.com/users/druid-io","html_url":"https://github.com/druid-io","followers_url":"https://api.github.com/users/druid-io/followers","following_url":"https://api.github.com/users/druid-io/following{/other_user}","gists_url":"https://api.github.com/users/druid-io/gists{/gist_id}","starred_url":"https://api.github.com/users/druid-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/druid-io/subscriptions","organizations_url":"https://api.github.com/users/druid-io/orgs","repos_url":"https://api.github.com/users/druid-io/repos","events_url":"https://api.github.com/users/druid-io/events{/privacy}","received_events_url":"https://api.github.com/users/druid-io/received_events","type":"Organization","site_admin":false},"repo":{"id":6358188,"name":"druid","full_name":"druid-io/druid","owner":{"login":"druid-io","id":4203474,"avatar_url":"https://avatars.githubusercontent.com/u/4203474?v=3","gravatar_id":"","url":"https://api.github.com/users/druid-io","html_url":"https://github.com/druid-io","followers_url":"https://api.github.com/users/druid-io/followers","following_url":"https://api.github.com/users/druid-io/following{/other_user}","gists_url":"https://api.github.com/users/druid-io/gists{/gist_id}","starred_url":"https://api.github.com/users/druid-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/druid-io/subscriptions","organizations_url":"https://api.github.com/users/druid-io/orgs","repos_url":"https://api.github.com/users/druid-io/repos","events_url":"https://api.github.com/users/druid-io/events{/privacy}","received_events_url":"https://api.github.com/users/druid-io/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/druid-io/druid","description":"Column oriented distributed data store ideal for powering interactive applications","fork":false,"url":"https://api.github.com/repos/druid-io/druid","forks_url":"https://api.github.com/repos/druid-io/druid/forks","keys_url":"https://api.github.com/repos/druid-io/druid/keys{/key_id}","collaborators_url":"https://api.github.com/repos/druid-io/druid/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/druid-io/druid/teams","hooks_url":"https://api.github.com/repos/druid-io/druid/hooks","issue_events_url":"https://api.github.com/repos/druid-io/druid/issues/events{/number}","events_url":"https://api.github.com/repos/druid-io/druid/events","assignees_url":"https://api.github.com/repos/druid-io/druid/assignees{/user}","branches_url":"https://api.github.com/repos/druid-io/druid/branches{/branch}","tags_url":"https://api.github.com/repos/druid-io/druid/tags","blobs_url":"https://api.github.com/repos/druid-io/druid/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/druid-io/druid/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/druid-io/druid/git/refs{/sha}","trees_url":"https://api.github.com/repos/druid-io/druid/git/trees{/sha}","statuses_url":"https://api.github.com/repos/druid-io/druid/statuses/{sha}","languages_url":"https://api.github.com/repos/druid-io/druid/languages","stargazers_url":"https://api.github.com/repos/druid-io/druid/stargazers","contributors_url":"https://api.github.com/repos/druid-io/druid/contributors","subscribers_url":"https://api.github.com/repos/druid-io/druid/subscribers","subscription_url":"https://api.github.com/repos/druid-io/druid/subscription","commits_url":"https://api.github.com/repos/druid-io/druid/commits{/sha}","git_commits_url":"https://api.github.com/repos/druid-io/druid/git/commits{/sha}","comments_url":"https://api.github.com/repos/druid-io/druid/comments{/number}","issue_comment_url":"https://api.github.com/repos/druid-io/druid/issues/comments{/number}","contents_url":"https://api.github.com/repos/druid-io/druid/contents/{+path}","compare_url":"https://api.github.com/repos/druid-io/druid/compare/{base}...{head}","merges_url":"https://api.github.com/repos/druid-io/druid/merges","archive_url":"https://api.github.com/repos/druid-io/druid/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/druid-io/druid/downloads","issues_url":"https://api.github.com/repos/druid-io/druid/issues{/number}","pulls_url":"https://api.github.com/repos/druid-io/druid/pulls{/number}","milestones_url":"https://api.github.com/repos/druid-io/druid/milestones{/number}","notifications_url":"https://api.github.com/repos/druid-io/druid/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/druid-io/druid/labels{/name}","releases_url":"https://api.github.com/repos/druid-io/druid/releases{/id}","deployments_url":"https://api.github.com/repos/druid-io/druid/deployments","created_at":"2012-10-23T19:08:07Z","updated_at":"2016-07-29T08:33:30Z","pushed_at":"2016-07-29T18:59:50Z","git_url":"git://github.com/druid-io/druid.git","ssh_url":"git@github.com:druid-io/druid.git","clone_url":"https://github.com/druid-io/druid.git","svn_url":"https://github.com/druid-io/druid","homepage":"http://druid.io","size":153083,"stargazers_count":3516,"watchers_count":3516,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":809,"mirror_url":null,"open_issues_count":477,"forks":809,"open_issues":477,"watchers":3516,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/druid-io/druid/pulls/2819"},"html":{"href":"https://github.com/druid-io/druid/pull/2819"},"issue":{"href":"https://api.github.com/repos/druid-io/druid/issues/2819"},"comments":{"href":"https://api.github.com/repos/druid-io/druid/issues/2819/comments"},"review_comments":{"href":"https://api.github.com/repos/druid-io/druid/pulls/2819/comments"},"review_comment":{"href":"https://api.github.com/repos/druid-io/druid/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/druid-io/druid/pulls/2819/commits"},"statuses":{"href":"https://api.github.com/repos/druid-io/druid/statuses/0785b2f275d5b83053b759430e1ff63f9c5c763c"}}}}
{"actor":{"display_login":"b-slim"}}