{"action":"closed","number":83,"pull_request":{"url":"https://api.github.com/repos/yahoo/pulsar/pulls/83","id":90838803,"html_url":"https://github.com/yahoo/pulsar/pull/83","diff_url":"https://github.com/yahoo/pulsar/pull/83.diff","patch_url":"https://github.com/yahoo/pulsar/pull/83.patch","issue_url":"https://api.github.com/repos/yahoo/pulsar/issues/83","number":83,"state":"closed","locked":false,"title":"Non blocking partitioned Consumers","user":{"login":"sschepens","id":3072509,"avatar_url":"https://avatars.githubusercontent.com/u/3072509?v=3","gravatar_id":"","url":"https://api.github.com/users/sschepens","html_url":"https://github.com/sschepens","followers_url":"https://api.github.com/users/sschepens/followers","following_url":"https://api.github.com/users/sschepens/following{/other_user}","gists_url":"https://api.github.com/users/sschepens/gists{/gist_id}","starred_url":"https://api.github.com/users/sschepens/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sschepens/subscriptions","organizations_url":"https://api.github.com/users/sschepens/orgs","repos_url":"https://api.github.com/users/sschepens/repos","events_url":"https://api.github.com/users/sschepens/events{/privacy}","received_events_url":"https://api.github.com/users/sschepens/received_events","type":"User","site_admin":false},"body":"### Motivation\n\nAs discussed in [#80](https://github.com/yahoo/pulsar/issues/80), current `PartitionedConsumerImpl` behavior es to block `listenerExecutor` when no space is left on the queue.\n\nI've made a first approach at solving this issue, I wanted you to review it and see if this is a good approach or if you can figure out a better way of solving this.\n### Modifications\n\nI've modified `PartitionedConsumerImpl` to behave more like `ConsumerImpl`, it keeps track of the amount of space in the queue and tries to fill the queue when a certain threshold has been surpassed, or when `ConsumerImpl` notifies that messages have been received.\nI've added a new `ReceiveListener` to `ConsumerImpl` that, unlike `MessageListener`, does not remove messages from the queue, but only notifies that it has new messages.\nThis Listener is used by the new `PartitionedConsumerImpl` to try to fill it's queue by pulling messages in a round-robin fashion from `ConsumerImpl`s starting from the notifying consumer.\n`PartitionedConsumerImpl` also keeps track of when a receive is already running as to not swarm `listenerExecutor` with tasks that will probably do nothing.\n### Result\n\n`PartitionedConsumerImpl` should no longer block `listenerExecutor`.\n\n_Comment: I'm not particularly sure about the `AtomicBoolean` flag to keep track if a receive is running, I guess, there could be a case when receive is finishing, not looping anymore and we cannot schedule another run because it has not yet disabled the flag._\n\nCheers!\n","created_at":"2016-10-25T16:01:37Z","updated_at":"2016-11-14T18:07:03Z","closed_at":"2016-11-14T18:07:02Z","merged_at":null,"merge_commit_sha":"7943440408d4ab051fa62ccbc3ed8579e5f3c3fd","assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/yahoo/pulsar/milestones/2","html_url":"https://github.com/yahoo/pulsar/milestones/1.16","labels_url":"https://api.github.com/repos/yahoo/pulsar/milestones/2/labels","id":2056012,"number":2,"title":"1.16","description":"","creator":{"login":"merlimat","id":62500,"avatar_url":"https://avatars.githubusercontent.com/u/62500?v=3","gravatar_id":"","url":"https://api.github.com/users/merlimat","html_url":"https://github.com/merlimat","followers_url":"https://api.github.com/users/merlimat/followers","following_url":"https://api.github.com/users/merlimat/following{/other_user}","gists_url":"https://api.github.com/users/merlimat/gists{/gist_id}","starred_url":"https://api.github.com/users/merlimat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/merlimat/subscriptions","organizations_url":"https://api.github.com/users/merlimat/orgs","repos_url":"https://api.github.com/users/merlimat/repos","events_url":"https://api.github.com/users/merlimat/events{/privacy}","received_events_url":"https://api.github.com/users/merlimat/received_events","type":"User","site_admin":false},"open_issues":7,"closed_issues":22,"state":"open","created_at":"2016-10-07T23:59:26Z","updated_at":"2016-11-14T18:07:02Z","due_on":null,"closed_at":null},"commits_url":"https://api.github.com/repos/yahoo/pulsar/pulls/83/commits","review_comments_url":"https://api.github.com/repos/yahoo/pulsar/pulls/83/comments","review_comment_url":"https://api.github.com/repos/yahoo/pulsar/pulls/comments{/number}","comments_url":"https://api.github.com/repos/yahoo/pulsar/issues/83/comments","statuses_url":"https://api.github.com/repos/yahoo/pulsar/statuses/65f2a5b4f5c151febbd4a3880c4562209ce8add0","head":{"label":"sschepens:non_blocking_partitioned_topic_take_2","ref":"non_blocking_partitioned_topic_take_2","sha":"65f2a5b4f5c151febbd4a3880c4562209ce8add0","user":{"login":"sschepens","id":3072509,"avatar_url":"https://avatars.githubusercontent.com/u/3072509?v=3","gravatar_id":"","url":"https://api.github.com/users/sschepens","html_url":"https://github.com/sschepens","followers_url":"https://api.github.com/users/sschepens/followers","following_url":"https://api.github.com/users/sschepens/following{/other_user}","gists_url":"https://api.github.com/users/sschepens/gists{/gist_id}","starred_url":"https://api.github.com/users/sschepens/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sschepens/subscriptions","organizations_url":"https://api.github.com/users/sschepens/orgs","repos_url":"https://api.github.com/users/sschepens/repos","events_url":"https://api.github.com/users/sschepens/events{/privacy}","received_events_url":"https://api.github.com/users/sschepens/received_events","type":"User","site_admin":false},"repo":{"id":70098520,"name":"pulsar","full_name":"sschepens/pulsar","owner":{"login":"sschepens","id":3072509,"avatar_url":"https://avatars.githubusercontent.com/u/3072509?v=3","gravatar_id":"","url":"https://api.github.com/users/sschepens","html_url":"https://github.com/sschepens","followers_url":"https://api.github.com/users/sschepens/followers","following_url":"https://api.github.com/users/sschepens/following{/other_user}","gists_url":"https://api.github.com/users/sschepens/gists{/gist_id}","starred_url":"https://api.github.com/users/sschepens/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sschepens/subscriptions","organizations_url":"https://api.github.com/users/sschepens/orgs","repos_url":"https://api.github.com/users/sschepens/repos","events_url":"https://api.github.com/users/sschepens/events{/privacy}","received_events_url":"https://api.github.com/users/sschepens/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/sschepens/pulsar","description":"Pulsar - distributed pub-sub messaging system","fork":true,"url":"https://api.github.com/repos/sschepens/pulsar","forks_url":"https://api.github.com/repos/sschepens/pulsar/forks","keys_url":"https://api.github.com/repos/sschepens/pulsar/keys{/key_id}","collaborators_url":"https://api.github.com/repos/sschepens/pulsar/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/sschepens/pulsar/teams","hooks_url":"https://api.github.com/repos/sschepens/pulsar/hooks","issue_events_url":"https://api.github.com/repos/sschepens/pulsar/issues/events{/number}","events_url":"https://api.github.com/repos/sschepens/pulsar/events","assignees_url":"https://api.github.com/repos/sschepens/pulsar/assignees{/user}","branches_url":"https://api.github.com/repos/sschepens/pulsar/branches{/branch}","tags_url":"https://api.github.com/repos/sschepens/pulsar/tags","blobs_url":"https://api.github.com/repos/sschepens/pulsar/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/sschepens/pulsar/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/sschepens/pulsar/git/refs{/sha}","trees_url":"https://api.github.com/repos/sschepens/pulsar/git/trees{/sha}","statuses_url":"https://api.github.com/repos/sschepens/pulsar/statuses/{sha}","languages_url":"https://api.github.com/repos/sschepens/pulsar/languages","stargazers_url":"https://api.github.com/repos/sschepens/pulsar/stargazers","contributors_url":"https://api.github.com/repos/sschepens/pulsar/contributors","subscribers_url":"https://api.github.com/repos/sschepens/pulsar/subscribers","subscription_url":"https://api.github.com/repos/sschepens/pulsar/subscription","commits_url":"https://api.github.com/repos/sschepens/pulsar/commits{/sha}","git_commits_url":"https://api.github.com/repos/sschepens/pulsar/git/commits{/sha}","comments_url":"https://api.github.com/repos/sschepens/pulsar/comments{/number}","issue_comment_url":"https://api.github.com/repos/sschepens/pulsar/issues/comments{/number}","contents_url":"https://api.github.com/repos/sschepens/pulsar/contents/{+path}","compare_url":"https://api.github.com/repos/sschepens/pulsar/compare/{base}...{head}","merges_url":"https://api.github.com/repos/sschepens/pulsar/merges","archive_url":"https://api.github.com/repos/sschepens/pulsar/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/sschepens/pulsar/downloads","issues_url":"https://api.github.com/repos/sschepens/pulsar/issues{/number}","pulls_url":"https://api.github.com/repos/sschepens/pulsar/pulls{/number}","milestones_url":"https://api.github.com/repos/sschepens/pulsar/milestones{/number}","notifications_url":"https://api.github.com/repos/sschepens/pulsar/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/sschepens/pulsar/labels{/name}","releases_url":"https://api.github.com/repos/sschepens/pulsar/releases{/id}","deployments_url":"https://api.github.com/repos/sschepens/pulsar/deployments","created_at":"2016-10-05T20:52:50Z","updated_at":"2016-10-05T20:52:52Z","pushed_at":"2016-11-14T15:51:19Z","git_url":"git://github.com/sschepens/pulsar.git","ssh_url":"git@github.com:sschepens/pulsar.git","clone_url":"https://github.com/sschepens/pulsar.git","svn_url":"https://github.com/sschepens/pulsar","homepage":"","size":1830,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"yahoo:master","ref":"master","sha":"4d685b515a429c1eaba93a55939b826e7aa3f9e2","user":{"login":"yahoo","id":16574,"avatar_url":"https://avatars.githubusercontent.com/u/16574?v=3","gravatar_id":"","url":"https://api.github.com/users/yahoo","html_url":"https://github.com/yahoo","followers_url":"https://api.github.com/users/yahoo/followers","following_url":"https://api.github.com/users/yahoo/following{/other_user}","gists_url":"https://api.github.com/users/yahoo/gists{/gist_id}","starred_url":"https://api.github.com/users/yahoo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yahoo/subscriptions","organizations_url":"https://api.github.com/users/yahoo/orgs","repos_url":"https://api.github.com/users/yahoo/repos","events_url":"https://api.github.com/users/yahoo/events{/privacy}","received_events_url":"https://api.github.com/users/yahoo/received_events","type":"Organization","site_admin":false},"repo":{"id":62117812,"name":"pulsar","full_name":"yahoo/pulsar","owner":{"login":"yahoo","id":16574,"avatar_url":"https://avatars.githubusercontent.com/u/16574?v=3","gravatar_id":"","url":"https://api.github.com/users/yahoo","html_url":"https://github.com/yahoo","followers_url":"https://api.github.com/users/yahoo/followers","following_url":"https://api.github.com/users/yahoo/following{/other_user}","gists_url":"https://api.github.com/users/yahoo/gists{/gist_id}","starred_url":"https://api.github.com/users/yahoo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yahoo/subscriptions","organizations_url":"https://api.github.com/users/yahoo/orgs","repos_url":"https://api.github.com/users/yahoo/repos","events_url":"https://api.github.com/users/yahoo/events{/privacy}","received_events_url":"https://api.github.com/users/yahoo/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/yahoo/pulsar","description":"Pulsar - distributed pub-sub messaging system","fork":false,"url":"https://api.github.com/repos/yahoo/pulsar","forks_url":"https://api.github.com/repos/yahoo/pulsar/forks","keys_url":"https://api.github.com/repos/yahoo/pulsar/keys{/key_id}","collaborators_url":"https://api.github.com/repos/yahoo/pulsar/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/yahoo/pulsar/teams","hooks_url":"https://api.github.com/repos/yahoo/pulsar/hooks","issue_events_url":"https://api.github.com/repos/yahoo/pulsar/issues/events{/number}","events_url":"https://api.github.com/repos/yahoo/pulsar/events","assignees_url":"https://api.github.com/repos/yahoo/pulsar/assignees{/user}","branches_url":"https://api.github.com/repos/yahoo/pulsar/branches{/branch}","tags_url":"https://api.github.com/repos/yahoo/pulsar/tags","blobs_url":"https://api.github.com/repos/yahoo/pulsar/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/yahoo/pulsar/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/yahoo/pulsar/git/refs{/sha}","trees_url":"https://api.github.com/repos/yahoo/pulsar/git/trees{/sha}","statuses_url":"https://api.github.com/repos/yahoo/pulsar/statuses/{sha}","languages_url":"https://api.github.com/repos/yahoo/pulsar/languages","stargazers_url":"https://api.github.com/repos/yahoo/pulsar/stargazers","contributors_url":"https://api.github.com/repos/yahoo/pulsar/contributors","subscribers_url":"https://api.github.com/repos/yahoo/pulsar/subscribers","subscription_url":"https://api.github.com/repos/yahoo/pulsar/subscription","commits_url":"https://api.github.com/repos/yahoo/pulsar/commits{/sha}","git_commits_url":"https://api.github.com/repos/yahoo/pulsar/git/commits{/sha}","comments_url":"https://api.github.com/repos/yahoo/pulsar/comments{/number}","issue_comment_url":"https://api.github.com/repos/yahoo/pulsar/issues/comments{/number}","contents_url":"https://api.github.com/repos/yahoo/pulsar/contents/{+path}","compare_url":"https://api.github.com/repos/yahoo/pulsar/compare/{base}...{head}","merges_url":"https://api.github.com/repos/yahoo/pulsar/merges","archive_url":"https://api.github.com/repos/yahoo/pulsar/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/yahoo/pulsar/downloads","issues_url":"https://api.github.com/repos/yahoo/pulsar/issues{/number}","pulls_url":"https://api.github.com/repos/yahoo/pulsar/pulls{/number}","milestones_url":"https://api.github.com/repos/yahoo/pulsar/milestones{/number}","notifications_url":"https://api.github.com/repos/yahoo/pulsar/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/yahoo/pulsar/labels{/name}","releases_url":"https://api.github.com/repos/yahoo/pulsar/releases{/id}","deployments_url":"https://api.github.com/repos/yahoo/pulsar/deployments","created_at":"2016-06-28T07:00:03Z","updated_at":"2016-11-14T16:17:28Z","pushed_at":"2016-11-14T18:05:32Z","git_url":"git://github.com/yahoo/pulsar.git","ssh_url":"git@github.com:yahoo/pulsar.git","clone_url":"https://github.com/yahoo/pulsar.git","svn_url":"https://github.com/yahoo/pulsar","homepage":"","size":2023,"stargazers_count":1057,"watchers_count":1057,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":207,"mirror_url":null,"open_issues_count":15,"forks":207,"open_issues":15,"watchers":1057,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/yahoo/pulsar/pulls/83"},"html":{"href":"https://github.com/yahoo/pulsar/pull/83"},"issue":{"href":"https://api.github.com/repos/yahoo/pulsar/issues/83"},"comments":{"href":"https://api.github.com/repos/yahoo/pulsar/issues/83/comments"},"review_comments":{"href":"https://api.github.com/repos/yahoo/pulsar/pulls/83/comments"},"review_comment":{"href":"https://api.github.com/repos/yahoo/pulsar/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/yahoo/pulsar/pulls/83/commits"},"statuses":{"href":"https://api.github.com/repos/yahoo/pulsar/statuses/65f2a5b4f5c151febbd4a3880c4562209ce8add0"}},"merged":false,"mergeable":false,"mergeable_state":"dirty","merged_by":null,"comments":9,"review_comments":10,"commits":1,"additions":138,"deletions":32,"changed_files":4}}
{"actor":{"display_login":"merlimat"}}