{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/14918","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/14918/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/14918/comments","events_url":"https://api.github.com/repos/apache/spark/issues/14918/events","html_url":"https://github.com/apache/spark/pull/14918","id":174511078,"number":14918,"title":"[SPARK-17360][PYSPARK] Support generator in createDataFrame","user":{"login":"Stibbons","id":133498,"avatar_url":"https://avatars.githubusercontent.com/u/133498?v=3","gravatar_id":"","url":"https://api.github.com/users/Stibbons","html_url":"https://github.com/Stibbons","followers_url":"https://api.github.com/users/Stibbons/followers","following_url":"https://api.github.com/users/Stibbons/following{/other_user}","gists_url":"https://api.github.com/users/Stibbons/gists{/gist_id}","starred_url":"https://api.github.com/users/Stibbons/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Stibbons/subscriptions","organizations_url":"https://api.github.com/users/Stibbons/orgs","repos_url":"https://api.github.com/users/Stibbons/repos","events_url":"https://api.github.com/users/Stibbons/events{/privacy}","received_events_url":"https://api.github.com/users/Stibbons/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":23,"created_at":"2016-09-01T12:31:59Z","updated_at":"2016-10-05T12:45:30Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/14918","html_url":"https://github.com/apache/spark/pull/14918","diff_url":"https://github.com/apache/spark/pull/14918.diff","patch_url":"https://github.com/apache/spark/pull/14918.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nAvoid useless iteration within 'data' structure when creating a data frame from local data (so when createDataFrame uses _createFromLocal) and when 'data' is a Python generator (instead of a 'list' or a dataframe)\r\n\r\n- if schema is provided, only iterate once (in _createFromLocal). In this case we skip 1 struction iteration\r\n- if no schema is provided, we skip the useless 'map' with the 'identify' lambda function\r\n\r\nThere are still serialization to the JVM.\r\n\r\n## How was this patch tested?\r\n\r\nPyspark unit tests and standalone manual pyspark execution"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/251663810","html_url":"https://github.com/apache/spark/pull/14918#issuecomment-251663810","issue_url":"https://api.github.com/repos/apache/spark/issues/14918","id":251663810,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-10-05T12:45:29Z","updated_at":"2016-10-05T12:45:29Z","body":"Merged build finished. Test PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}