{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/22750","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/22750/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/22750/comments","events_url":"https://api.github.com/repos/apache/spark/issues/22750/events","html_url":"https://github.com/apache/spark/pull/22750","id":370686811,"node_id":"MDExOlB1bGxSZXF1ZXN0MjIzMzA4OTQx","number":22750,"title":"[SPARK-25747][SQL] remove ColumnarBatchScan.needsUnsafeRowConversion","user":{"login":"cloud-fan","id":3182036,"node_id":"MDQ6VXNlcjMxODIwMzY=","avatar_url":"https://avatars3.githubusercontent.com/u/3182036?v=4","gravatar_id":"","url":"https://api.github.com/users/cloud-fan","html_url":"https://github.com/cloud-fan","followers_url":"https://api.github.com/users/cloud-fan/followers","following_url":"https://api.github.com/users/cloud-fan/following{/other_user}","gists_url":"https://api.github.com/users/cloud-fan/gists{/gist_id}","starred_url":"https://api.github.com/users/cloud-fan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cloud-fan/subscriptions","organizations_url":"https://api.github.com/users/cloud-fan/orgs","repos_url":"https://api.github.com/users/cloud-fan/repos","events_url":"https://api.github.com/users/cloud-fan/events{/privacy}","received_events_url":"https://api.github.com/users/cloud-fan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-10-16T16:05:47Z","updated_at":"2018-10-16T16:08:02Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/22750","html_url":"https://github.com/apache/spark/pull/22750","diff_url":"https://github.com/apache/spark/pull/22750.diff","patch_url":"https://github.com/apache/spark/pull/22750.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\n`needsUnsafeRowConversion` is used in 2 places:\r\n1. `ColumnarBatchScan.produceRows`\r\n2. `FileSourceScanExec.doExecute`\r\n\r\nWhen we hit `ColumnarBatchScan.produceRows`, it means whole stage codegen is on but the vectorized reader is off. The vectorized reader can be off for several reasons:\r\n1. the file format doesn't have a vectorized reader(json, csv, etc.)\r\n2. the vectorized reader config is off\r\n3. the schema is not supported\r\n\r\nAnyway when the vectorized reader is off, file format reader will always return unsafe rows, and other `ColumnarBatchScan` implementations also always return unsafe rows, so `ColumnarBatchScan.needsUnsafeRowConversion` is not needed.\r\n\r\nWhen we hit `FileSourceScanExec.doExecute`, it means whole stage codegen is off. For this case, we need the `needsUnsafeRowConversion` to convert `ColumnarRow` to `UnsafeRow`, if the file format reader returns batch.\r\n\r\nThis PR removes `ColumnarBatchScan.needsUnsafeRowConversion`, and keep this flag only in `FileSourceScanExec`\r\n\r\n## How was this patch tested?\r\n\r\nexisting tests\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/430297894","html_url":"https://github.com/apache/spark/pull/22750#issuecomment-430297894","issue_url":"https://api.github.com/repos/apache/spark/issues/22750","id":430297894,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMDI5Nzg5NA==","user":{"login":"AmplabJenkins","id":4031291,"node_id":"MDQ6VXNlcjQwMzEyOTE=","avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-10-16T16:08:02Z","updated_at":"2018-10-16T16:08:02Z","author_association":"NONE","body":"Test PASSed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4038/\nTest PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}