{"action":"created","issue":{"url":"https://api.github.com/repos/alibaba/jstorm/issues/252","repository_url":"https://api.github.com/repos/alibaba/jstorm","labels_url":"https://api.github.com/repos/alibaba/jstorm/issues/252/labels{/name}","comments_url":"https://api.github.com/repos/alibaba/jstorm/issues/252/comments","events_url":"https://api.github.com/repos/alibaba/jstorm/issues/252/events","html_url":"https://github.com/alibaba/jstorm/issues/252","id":152038434,"number":252,"title":"多spout的拓扑负载不均衡的问题","user":{"login":"youngcle","id":15942918,"avatar_url":"https://avatars.githubusercontent.com/u/15942918?v=3","gravatar_id":"","url":"https://api.github.com/users/youngcle","html_url":"https://github.com/youngcle","followers_url":"https://api.github.com/users/youngcle/followers","following_url":"https://api.github.com/users/youngcle/following{/other_user}","gists_url":"https://api.github.com/users/youngcle/gists{/gist_id}","starred_url":"https://api.github.com/users/youngcle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/youngcle/subscriptions","organizations_url":"https://api.github.com/users/youngcle/orgs","repos_url":"https://api.github.com/users/youngcle/repos","events_url":"https://api.github.com/users/youngcle/events{/privacy}","received_events_url":"https://api.github.com/users/youngcle/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-04-30T16:10:07Z","updated_at":"2016-06-24T12:44:30Z","closed_at":null,"body":"我刚设计了个拓扑，\r\n多个spout，从不同地方取数据，发送给一个bolt处理。每个spout并行度为1，处理bolt并行度设为200.\r\ngroup模式为localorshuffle。\r\n多个spout来源的数据量有多有少。\r\n刚刚启动时，数据可以把bolt 的所有task填满。随着若干spout结束，最后只有一个spout有数据，该spout数据发送的tuple似乎并没有填满我的200个bolt的task，与启动之初情形类似，只发送到了少数几个task中。造成整体资源利用率不高，性能低下。\r\n\r\n我认为应该无论何时，spout都应该随机发送填满所有bolt task，但实际情况不是这样。\r\n\r\n做了很多尝试，不知该如何优化。\r\n\r\n是jstorm特性如此，还是我设计使用不当？请指教改如何优化。谢谢了"},"comment":{"url":"https://api.github.com/repos/alibaba/jstorm/issues/comments/228336108","html_url":"https://github.com/alibaba/jstorm/issues/252#issuecomment-228336108","issue_url":"https://api.github.com/repos/alibaba/jstorm/issues/252","id":228336108,"user":{"login":"youngcle","id":15942918,"avatar_url":"https://avatars.githubusercontent.com/u/15942918?v=3","gravatar_id":"","url":"https://api.github.com/users/youngcle","html_url":"https://github.com/youngcle","followers_url":"https://api.github.com/users/youngcle/followers","following_url":"https://api.github.com/users/youngcle/following{/other_user}","gists_url":"https://api.github.com/users/youngcle/gists{/gist_id}","starred_url":"https://api.github.com/users/youngcle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/youngcle/subscriptions","organizations_url":"https://api.github.com/users/youngcle/orgs","repos_url":"https://api.github.com/users/youngcle/repos","events_url":"https://api.github.com/users/youngcle/events{/privacy}","received_events_url":"https://api.github.com/users/youngcle/received_events","type":"User","site_admin":false},"created_at":"2016-06-24T12:44:30Z","updated_at":"2016-06-24T12:44:30Z","body":"@longdafeng 的建议给我很大帮助，经过一段时间的测试和观察，我发现造成这个问题的还存在，可能cache造成了我的问题，在spout都发送完成后，还有bolt在工作，处于比较慢的机器上的bolt的task（or executer？）的cache中还有许多tuple等着处理，造成了严重的木桶效应。\r\n\r\n降低cache 应该可以使得计算更加均匀。\r\n\r\n那么问题来了，修改哪些参数可以取得较好的效果内。我专门学习了longdafeng的调优讲座视频，还是有些概念模糊。下面这些参数建议能够进一步说明，以便用户群可以正确的理解和使用jstorm\r\n\r\ntopology.ctrl.buffer.size: 256 #control queue size\r\ntopology.executor.receive.buffer.size: 1024 #batched\r\ntopology.executor.send.buffer.size: 1024 #individual messages\r\ntopology.receiver.buffer.size: 8 # setting it too high causes a lot of problems (heartbeat thread gets starved, throughput plummets)\r\ntopology.transfer.buffer.size: 2048 # batched"}}
{"actor":{"display_login":"youngcle"}}