{"action":"created","issue":{"url":"https://api.github.com/repos/pravega/pravega/issues/3173","repository_url":"https://api.github.com/repos/pravega/pravega","labels_url":"https://api.github.com/repos/pravega/pravega/issues/3173/labels{/name}","comments_url":"https://api.github.com/repos/pravega/pravega/issues/3173/comments","events_url":"https://api.github.com/repos/pravega/pravega/issues/3173/events","html_url":"https://github.com/pravega/pravega/issues/3173","id":387904386,"node_id":"MDU6SXNzdWUzODc5MDQzODY=","number":3173,"title":"Non-unique projection of client key to segmentstore key","user":{"login":"jkhalack","id":25368596,"node_id":"MDQ6VXNlcjI1MzY4NTk2","avatar_url":"https://avatars2.githubusercontent.com/u/25368596?v=4","gravatar_id":"","url":"https://api.github.com/users/jkhalack","html_url":"https://github.com/jkhalack","followers_url":"https://api.github.com/users/jkhalack/followers","following_url":"https://api.github.com/users/jkhalack/following{/other_user}","gists_url":"https://api.github.com/users/jkhalack/gists{/gist_id}","starred_url":"https://api.github.com/users/jkhalack/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jkhalack/subscriptions","organizations_url":"https://api.github.com/users/jkhalack/orgs","repos_url":"https://api.github.com/users/jkhalack/repos","events_url":"https://api.github.com/users/jkhalack/events{/privacy}","received_events_url":"https://api.github.com/users/jkhalack/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-12-05T19:15:07Z","updated_at":"2018-12-07T00:29:50Z","closed_at":null,"author_association":"NONE","body":"**Problem description**\r\nWe are running a longevity test where the client writes to a stream at a constant event rate to 20 randomly pre-chosen keys. The stream has an event-rate triggered scaling policy. There are three writers in one writer group.\r\n\r\nThe event rate for an individual client key is **lower** than the scaling trigger, and we expect that for each value of the 20 client keys it would be possible to have a segment of a **finite** width that contains only one key, so we don't expect any segments to be splitting infinitely.\r\n\r\nHowever, in reality we observe that after one day of running the test the stream arrives at a state when the number of  stream segments is less than 20 (the number of client keys) and almost half of the segments are empty (not being written into).\r\nAs to the active segments, some of them are infinitely narrow within the numerical precision (so that further splitting is impossible) and have the write rates that are double of the client write rate. \r\nThis is the indication that more than one String-typed client key is being written to the same hash value of the segment key.\r\n\r\nExample of the stream that stopped scaling at 19 segments per stream:\r\n![heat-chart](https://user-images.githubusercontent.com/25368596/49536044-ffcc5280-f89b-11e8-87af-7626a0c12b0f.png)\r\n(most of the active segments here are very narrow to be seen and are located between the light-blue inactive segments).\r\n\r\nThe client is writing to the following 20 keys:\r\n```\r\n0f5c8e82-4a90-4dab-b866-745cac949d72\r\n1203a27d-d5aa-405c-87ba-cc4811b80e82\r\n22cd4044-32a3-4f59-b49b-c467bdc09aaf\r\n37138de7-c701-4042-bc9d-91a1fc52d4be\r\n41f79861-39fe-4231-a980-d84e8a753c5d\r\n452e4ea9-ca70-48a8-9f4b-59a0c7d3030f\r\n4a83d604-83ca-4e27-9119-48d764ef4f2b\r\n7eb437db-631d-49e8-8fd6-46c5bda879d1\r\n8133ce55-0206-4e62-9991-c4c89e4d5822\r\n883e76c3-6556-4751-93d8-08e9a734ab9d\r\n93c9f3ad-a76e-4046-9fe6-63909c303988\r\naab63967-6ca0-4b15-89f3-e0625a558e54\r\nad88ceb4-703a-4fc0-9102-4ab6f4d6fab4\r\nb476f659-e8f3-42b1-bea4-47c691c3a6fe\r\ncf45b9fc-3244-4ec8-a26e-298a52866a1c\r\nd6a890a4-051d-4f63-8b22-dade01cc0a6d\r\nf617e1c4-eb5a-4e85-b576-1f0e5c9466bc\r\nf62a2f91-7fe2-4e49-aa54-0b0c1cee4cdf\r\ne7851825-f9b6-4601-bd42-0ce4fad2fb26\r\ne94ce3e6-26d4-48ab-a48a-448aa9bd5203\r\n```\r\n(See the [client log file](https://github.com/pravega/pravega/files/2650036/logs-from-small-scale-young-worker-in-small-scale-young-worker-6fcd78db45-r99d7.txt.zip) for detail)\r\n\r\nHowever, only 11 segments are being written into:\r\n![write_rates](https://user-images.githubusercontent.com/25368596/49535324-491ba280-f89a-11e8-80ba-11a212d085a7.png) \r\n\r\nLooking at the response from the `scaling_events` API, 9 of the 11 active segments  (namely 904, 991, 1259, 1346, 1363, 1411, 1584, 1638, and 1656) have zero width (to the numerical precision) and have the write rates of 3 kB/s which is twice the 1.5 kB/s write rate per single client key. Check the [attached file](https://github.com/pravega/pravega/files/2650043/scaling.txt) for the exact key values.\r\nSo each of this 9 segments has its numerical segment key value being mapped from two distinct string values of client key.  \r\n\r\nThe two remaining active segments (1391 and 1647) have finite width and a write rate of 1.5 kB/s, suggesting that they are getting writes from one client key each.\r\n\r\nThe above behavior is observed for commit `26fb466`. \r\nIt did not happen when we were running `0.3` pravega.\r\n\r\n**Problem location**\r\nSegmentstore\r\n\r\n**Suggestions for an improvement**\r\nClient keys should be uniquely mapped to segment keys"},"comment":{"url":"https://api.github.com/repos/pravega/pravega/issues/comments/445080312","html_url":"https://github.com/pravega/pravega/issues/3173#issuecomment-445080312","issue_url":"https://api.github.com/repos/pravega/pravega/issues/3173","id":445080312,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NTA4MDMxMg==","user":{"login":"tkaitchuck","id":4756752,"node_id":"MDQ6VXNlcjQ3NTY3NTI=","avatar_url":"https://avatars1.githubusercontent.com/u/4756752?v=4","gravatar_id":"","url":"https://api.github.com/users/tkaitchuck","html_url":"https://github.com/tkaitchuck","followers_url":"https://api.github.com/users/tkaitchuck/followers","following_url":"https://api.github.com/users/tkaitchuck/following{/other_user}","gists_url":"https://api.github.com/users/tkaitchuck/gists{/gist_id}","starred_url":"https://api.github.com/users/tkaitchuck/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tkaitchuck/subscriptions","organizations_url":"https://api.github.com/users/tkaitchuck/orgs","repos_url":"https://api.github.com/users/tkaitchuck/repos","events_url":"https://api.github.com/users/tkaitchuck/events{/privacy}","received_events_url":"https://api.github.com/users/tkaitchuck/received_events","type":"User","site_admin":false},"created_at":"2018-12-07T00:29:50Z","updated_at":"2018-12-07T00:29:50Z","author_association":"MEMBER","body":"I have a unit test that might be reproducing this. It looks like a bug on line 91 of `StreamSegments`. "}}
{"actor":{"display_login":"tkaitchuck"}}