{"action":"created","issue":{"url":"https://api.github.com/repos/WallarooLabs/wallaroo/issues/2264","repository_url":"https://api.github.com/repos/WallarooLabs/wallaroo","labels_url":"https://api.github.com/repos/WallarooLabs/wallaroo/issues/2264/labels{/name}","comments_url":"https://api.github.com/repos/WallarooLabs/wallaroo/issues/2264/comments","events_url":"https://api.github.com/repos/WallarooLabs/wallaroo/issues/2264/events","html_url":"https://github.com/WallarooLabs/wallaroo/pull/2264","id":341663297,"node_id":"MDExOlB1bGxSZXF1ZXN0MjAxNzQ4NDQ5","number":2264,"title":"Ensure that messages of len() > _max_size don't overflow buffer/cause memory explosion","user":{"login":"pzel","id":344774,"node_id":"MDQ6VXNlcjM0NDc3NA==","avatar_url":"https://avatars0.githubusercontent.com/u/344774?v=4","gravatar_id":"","url":"https://api.github.com/users/pzel","html_url":"https://github.com/pzel","followers_url":"https://api.github.com/users/pzel/followers","following_url":"https://api.github.com/users/pzel/following{/other_user}","gists_url":"https://api.github.com/users/pzel/gists{/gist_id}","starred_url":"https://api.github.com/users/pzel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pzel/subscriptions","organizations_url":"https://api.github.com/users/pzel/orgs","repos_url":"https://api.github.com/users/pzel/repos","events_url":"https://api.github.com/users/pzel/events{/privacy}","received_events_url":"https://api.github.com/users/pzel/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-07-16T20:17:43Z","updated_at":"2018-07-16T22:36:23Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/WallarooLabs/wallaroo/pulls/2264","html_url":"https://github.com/WallarooLabs/wallaroo/pull/2264","diff_url":"https://github.com/WallarooLabs/wallaroo/pull/2264.diff","patch_url":"https://github.com/WallarooLabs/wallaroo/pull/2264.patch"},"body":"This PR solves the problem of [the cluster locking up when individual messages are bigger than the max_size](https://github.com/WallarooLabs/wallaroo/issues/2258) parameter that DataChannel was started with.\r\n\r\nThere were 2 unwanted behaviors causing that bug, and removing one of them only exposed the second.\r\n\r\n### Problem 1:\r\n\r\n`_read_buf_size()` kept thinking it doesn't have a big enough buffer, because `_next_size` was smaller than `_expect`. Adding a special case for `_expect > _next_size` in this method allowed the DataChannel to handle messages bigger than `_max_size`, but eventually triggered a `Fail()` because of Problem 2. The fix involved not having a special case for when `_expect > _next_size`, but always maintaining the invariant that the buffer will be the next power of 2 above `_next_size`.\r\n\r\n### Problem 2:\r\n\r\n`_read_buf.space()` was being used instead of `_read_buf.size()` when deciding to consume data via `@pony_os_recv`. Space is allocated in power-of-2 blocks, and by reading `.space()` bytes off of the stream, we were effectively reading more that we could fit in our buffer, because usually `_read_buf.size() < _read_buf.space())`. So the DataChannel would ended up losing data and then, at some point, passing random junk to the notifier to interpret as a header. The fix reallocates the buffer in powers-of-2 increments, (which means that .size() will equal .space()) but also calls `.size()` explicitly in `_pending_reads()`,  to remove ambiguity.\r\n\r\nThis fix includes a unit test in data_channel/_test.pony to make sure any regressions are caught, although I feel generative testing of our TCPNotify-alikes would be beneficial in flushing out other cases like this one.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/WallarooLabs/wallaroo/issues/comments/405402555","html_url":"https://github.com/WallarooLabs/wallaroo/pull/2264#issuecomment-405402555","issue_url":"https://api.github.com/repos/WallarooLabs/wallaroo/issues/2264","id":405402555,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNTQwMjU1NQ==","user":{"login":"pzel","id":344774,"node_id":"MDQ6VXNlcjM0NDc3NA==","avatar_url":"https://avatars0.githubusercontent.com/u/344774?v=4","gravatar_id":"","url":"https://api.github.com/users/pzel","html_url":"https://github.com/pzel","followers_url":"https://api.github.com/users/pzel/followers","following_url":"https://api.github.com/users/pzel/following{/other_user}","gists_url":"https://api.github.com/users/pzel/gists{/gist_id}","starred_url":"https://api.github.com/users/pzel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pzel/subscriptions","organizations_url":"https://api.github.com/users/pzel/orgs","repos_url":"https://api.github.com/users/pzel/repos","events_url":"https://api.github.com/users/pzel/events{/privacy}","received_events_url":"https://api.github.com/users/pzel/received_events","type":"User","site_admin":false},"created_at":"2018-07-16T22:36:23Z","updated_at":"2018-07-16T22:36:23Z","author_association":"MEMBER","body":"@SeanTAllen the analogous fix for TCPSource is [included in this PR](https://github.com/WallarooLabs/wallaroo/pull/2264/files#diff-fa266cf4eca7c3ef8c26082f86fc72b2)"}}
{"actor":{"display_login":"pzel"}}