{"action":"created","issue":{"url":"https://api.github.com/repos/apache/flink/issues/1685","repository_url":"https://api.github.com/repos/apache/flink","labels_url":"https://api.github.com/repos/apache/flink/issues/1685/labels{/name}","comments_url":"https://api.github.com/repos/apache/flink/issues/1685/comments","events_url":"https://api.github.com/repos/apache/flink/issues/1685/events","html_url":"https://github.com/apache/flink/pull/1685","id":135268465,"number":1685,"title":"[WIP][FLINK-3422][streaming][api-breaking] Scramble HashPartitioner hashes.","user":{"login":"Xazax-hun","id":922897,"avatar_url":"https://avatars.githubusercontent.com/u/922897?v=3","gravatar_id":"","url":"https://api.github.com/users/Xazax-hun","html_url":"https://github.com/Xazax-hun","followers_url":"https://api.github.com/users/Xazax-hun/followers","following_url":"https://api.github.com/users/Xazax-hun/following{/other_user}","gists_url":"https://api.github.com/users/Xazax-hun/gists{/gist_id}","starred_url":"https://api.github.com/users/Xazax-hun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Xazax-hun/subscriptions","organizations_url":"https://api.github.com/users/Xazax-hun/orgs","repos_url":"https://api.github.com/users/Xazax-hun/repos","events_url":"https://api.github.com/users/Xazax-hun/events{/privacy}","received_events_url":"https://api.github.com/users/Xazax-hun/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-02-21T22:23:33Z","updated_at":"2016-02-22T00:08:45Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/flink/pulls/1685","html_url":"https://github.com/apache/flink/pull/1685","diff_url":"https://github.com/apache/flink/pull/1685.diff","patch_url":"https://github.com/apache/flink/pull/1685.patch"},"body":"This pull request contains a fix for FLINK-3422. Some of the tests are failing at the moment, because they utilized prior knowledge about the user hash function. Fixing those tests require knowledge about the internals of Flink that I do not possess yet, so Marton Balassi helps me.\r\n\r\nThe Jira ticket mentions both Murmur and Jenkins hash.\r\nMurmur hash is already used in the batch implementation: https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/operators/shipping/OutputEmitter.java#L187\r\n\r\nMy approach was to move Jenkins hash from CompactingHashTable to MathUtils and use that in HashPartitioner. In case you think it is better to use murmur hash here, or it has some value to be consistent in this regard with the batch implementation, please let me know. "},"comment":{"url":"https://api.github.com/repos/apache/flink/issues/comments/186948795","html_url":"https://github.com/apache/flink/pull/1685#issuecomment-186948795","issue_url":"https://api.github.com/repos/apache/flink/issues/1685","id":186948795,"user":{"login":"StephanEwen","id":1727146,"avatar_url":"https://avatars.githubusercontent.com/u/1727146?v=3","gravatar_id":"","url":"https://api.github.com/users/StephanEwen","html_url":"https://github.com/StephanEwen","followers_url":"https://api.github.com/users/StephanEwen/followers","following_url":"https://api.github.com/users/StephanEwen/following{/other_user}","gists_url":"https://api.github.com/users/StephanEwen/gists{/gist_id}","starred_url":"https://api.github.com/users/StephanEwen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/StephanEwen/subscriptions","organizations_url":"https://api.github.com/users/StephanEwen/orgs","repos_url":"https://api.github.com/users/StephanEwen/repos","events_url":"https://api.github.com/users/StephanEwen/events{/privacy}","received_events_url":"https://api.github.com/users/StephanEwen/received_events","type":"User","site_admin":false},"created_at":"2016-02-22T00:08:45Z","updated_at":"2016-02-22T00:08:45Z","body":"It is pretty crucial that different hash functions are used for the partitioning across machines, and the internal partitioning of data structures. If the same hash function is used for both, many internal data structure partitions will be empty.\r\n\r\nSo far we divided it the following way (admittedly not documented)\r\n  - murmur hash across machines\r\n  - Jenkins hash internally in data structures\r\n\r\nHow about we stick with that division and use Murmur Hash in the streaming partitioner as well?\r\n\r\n"}}