{"action":"created","issue":{"url":"https://api.github.com/repos/apache/incubator-beam/issues/1674","repository_url":"https://api.github.com/repos/apache/incubator-beam","labels_url":"https://api.github.com/repos/apache/incubator-beam/issues/1674/labels{/name}","comments_url":"https://api.github.com/repos/apache/incubator-beam/issues/1674/comments","events_url":"https://api.github.com/repos/apache/incubator-beam/issues/1674/events","html_url":"https://github.com/apache/incubator-beam/pull/1674","id":196966389,"number":1674,"title":"[BEAM-1146] Decrease spark runner startup overhead","user":{"login":"aviemzur","id":5708714,"avatar_url":"https://avatars.githubusercontent.com/u/5708714?v=3","gravatar_id":"","url":"https://api.github.com/users/aviemzur","html_url":"https://github.com/aviemzur","followers_url":"https://api.github.com/users/aviemzur/followers","following_url":"https://api.github.com/users/aviemzur/following{/other_user}","gists_url":"https://api.github.com/users/aviemzur/gists{/gist_id}","starred_url":"https://api.github.com/users/aviemzur/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aviemzur/subscriptions","organizations_url":"https://api.github.com/users/aviemzur/orgs","repos_url":"https://api.github.com/users/aviemzur/repos","events_url":"https://api.github.com/users/aviemzur/events{/privacy}","received_events_url":"https://api.github.com/users/aviemzur/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-12-21T15:52:41Z","updated_at":"2016-12-25T14:36:24Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/incubator-beam/pulls/1674","html_url":"https://github.com/apache/incubator-beam/pull/1674","diff_url":"https://github.com/apache/incubator-beam/pull/1674.diff","patch_url":"https://github.com/apache/incubator-beam/pull/1674.patch"},"body":"Be sure to do all of the following to help us incorporate your contribution\r\nquickly and easily:\r\n\r\n - [ ] Make sure the PR title is formatted like:\r\n   `[BEAM-<Jira issue #>] Description of pull request`\r\n - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable\r\n       Travis-CI on your fork and ensure the whole test matrix passes).\r\n - [ ] Replace `<Jira issue #>` in the title with the actual Jira issue\r\n       number, if there is one.\r\n - [ ] If this contribution is large, please file an Apache\r\n       [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).\r\n\r\n---\r\nReplace finding all `Source` and `Coder` implementations for serialization registration with wrapper classes."},"comment":{"url":"https://api.github.com/repos/apache/incubator-beam/issues/comments/269125130","html_url":"https://github.com/apache/incubator-beam/pull/1674#issuecomment-269125130","issue_url":"https://api.github.com/repos/apache/incubator-beam/issues/1674","id":269125130,"user":{"login":"aviemzur","id":5708714,"avatar_url":"https://avatars.githubusercontent.com/u/5708714?v=3","gravatar_id":"","url":"https://api.github.com/users/aviemzur","html_url":"https://github.com/aviemzur","followers_url":"https://api.github.com/users/aviemzur/followers","following_url":"https://api.github.com/users/aviemzur/following{/other_user}","gists_url":"https://api.github.com/users/aviemzur/gists{/gist_id}","starred_url":"https://api.github.com/users/aviemzur/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aviemzur/subscriptions","organizations_url":"https://api.github.com/users/aviemzur/orgs","repos_url":"https://api.github.com/users/aviemzur/repos","events_url":"https://api.github.com/users/aviemzur/events{/privacy}","received_events_url":"https://api.github.com/users/aviemzur/received_events","type":"User","site_admin":false},"created_at":"2016-12-25T14:36:24Z","updated_at":"2016-12-25T14:36:24Z","body":"After digging deeper into `Spark` serialization code it seems that the issue is simpler than we first thought.\r\n`Spark` serializes closures using Java Serializer, not Kryo. Most of the `Coder` instances in Spark runner are within such closures.\r\nSince `Source` instances are serialized in both bounded and unbounded pipelines, these did require wrappers (`UnboundedSource` was already wrapped in `MicrobatchSource` and `BoundedSource` now has a wrapper named `BoundedSourceWrapper`. The latter might need a better name).\r\nThese `Source` instances often have `Coder` members themselves, but the serialization of these is already taken care of by having the `Source` itself registered to be serialized with `JavaSerializer`."}}
{"actor":{"display_login":"aviemzur"}}