{"action":"created","issue":{"url":"https://api.github.com/repos/zalando/nakadi/issues/515","repository_url":"https://api.github.com/repos/zalando/nakadi","labels_url":"https://api.github.com/repos/zalando/nakadi/issues/515/labels{/name}","comments_url":"https://api.github.com/repos/zalando/nakadi/issues/515/comments","events_url":"https://api.github.com/repos/zalando/nakadi/issues/515/events","html_url":"https://github.com/zalando/nakadi/pull/515","id":196020663,"number":515,"title":"Aruha 473 check size of events","user":{"login":"lmontrieux","id":203099,"avatar_url":"https://avatars.githubusercontent.com/u/203099?v=3","gravatar_id":"","url":"https://api.github.com/users/lmontrieux","html_url":"https://github.com/lmontrieux","followers_url":"https://api.github.com/users/lmontrieux/followers","following_url":"https://api.github.com/users/lmontrieux/following{/other_user}","gists_url":"https://api.github.com/users/lmontrieux/gists{/gist_id}","starred_url":"https://api.github.com/users/lmontrieux/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lmontrieux/subscriptions","organizations_url":"https://api.github.com/users/lmontrieux/orgs","repos_url":"https://api.github.com/users/lmontrieux/repos","events_url":"https://api.github.com/users/lmontrieux/events{/privacy}","received_events_url":"https://api.github.com/users/lmontrieux/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-12-16T09:55:51Z","updated_at":"2016-12-23T09:26:14Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/zalando/nakadi/pulls/515","html_url":"https://github.com/zalando/nakadi/pull/515","diff_url":"https://github.com/zalando/nakadi/pull/515.diff","patch_url":"https://github.com/zalando/nakadi/pull/515.patch"},"body":"This PR checks the size of all events in a batch. If there is at least one event that is over the maximum size *after enrichment*, then the entire batch is rejected with a 422."},"comment":{"url":"https://api.github.com/repos/zalando/nakadi/issues/comments/268962326","html_url":"https://github.com/zalando/nakadi/pull/515#issuecomment-268962326","issue_url":"https://api.github.com/repos/zalando/nakadi/issues/515","id":268962326,"user":{"login":"lmontrieux","id":203099,"avatar_url":"https://avatars.githubusercontent.com/u/203099?v=3","gravatar_id":"","url":"https://api.github.com/users/lmontrieux","html_url":"https://github.com/lmontrieux","followers_url":"https://api.github.com/users/lmontrieux/followers","following_url":"https://api.github.com/users/lmontrieux/following{/other_user}","gists_url":"https://api.github.com/users/lmontrieux/gists{/gist_id}","starred_url":"https://api.github.com/users/lmontrieux/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lmontrieux/subscriptions","organizations_url":"https://api.github.com/users/lmontrieux/orgs","repos_url":"https://api.github.com/users/lmontrieux/repos","events_url":"https://api.github.com/users/lmontrieux/events{/privacy}","received_events_url":"https://api.github.com/users/lmontrieux/received_events","type":"User","site_admin":false},"created_at":"2016-12-23T09:26:14Z","updated_at":"2016-12-23T09:26:14Z","body":"@valgog @v-stepanov I've had a short discussion offline with @valgog about this, but with Hack week I hadn't updated this PR yet. The way things work is:\r\n* producer sends an event of size x;\r\n* nakadi enriches the event, whose size is now x + e;\r\n* the kafka client serialises the event somehow, and add a bit of metadata. Size is now x + e + k.\r\n* kafka has 4 different configuration options for the max size of a message: the max size that a broker will accept (default 1MB); the max size of a log segment (must be higher than the max size of a message, default 1GB); the max size of data that a broker can replicate (default 1MB); the max size a consumer can read (default 1MB). Currently we use the defaults.\r\n\r\nThe idea behind this fix was to limit the size of an event to 1MB *before* enrichment (limit x only), but bump up all max values in Kafka to 2MB, to be safe (e+k are just a few bytes long). This way we will accept slightly larger messages than we do now. This only works assuming that increasing max values in Kafka will not increase its memory usage. I haven't looked into it but I'd be surprised in Kafka were to always allocate the max message size for each message.\r\n\r\nAs I recall, the outcome of this discussion was:\r\n1/ limit size in validation, as originally planned;\r\n2/ next week, as a different ticket, work on a more accurate way to limit the size of events, and keep statistics about the size of enrichments."}}
{"actor":{"display_login":"lmontrieux"}}