{"action":"created","issue":{"url":"https://api.github.com/repos/databricks/spark-csv/issues/387","repository_url":"https://api.github.com/repos/databricks/spark-csv","labels_url":"https://api.github.com/repos/databricks/spark-csv/issues/387/labels{/name}","comments_url":"https://api.github.com/repos/databricks/spark-csv/issues/387/comments","events_url":"https://api.github.com/repos/databricks/spark-csv/issues/387/events","html_url":"https://github.com/databricks/spark-csv/issues/387","id":182902479,"number":387,"title":"In databricks with spark 2.0.1, nullValue param doesn't work","user":{"login":"chouzhupk","id":5733493,"avatar_url":"https://avatars.githubusercontent.com/u/5733493?v=3","gravatar_id":"","url":"https://api.github.com/users/chouzhupk","html_url":"https://github.com/chouzhupk","followers_url":"https://api.github.com/users/chouzhupk/followers","following_url":"https://api.github.com/users/chouzhupk/following{/other_user}","gists_url":"https://api.github.com/users/chouzhupk/gists{/gist_id}","starred_url":"https://api.github.com/users/chouzhupk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chouzhupk/subscriptions","organizations_url":"https://api.github.com/users/chouzhupk/orgs","repos_url":"https://api.github.com/users/chouzhupk/repos","events_url":"https://api.github.com/users/chouzhupk/events{/privacy}","received_events_url":"https://api.github.com/users/chouzhupk/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-10-13T21:13:34Z","updated_at":"2016-10-13T22:22:01Z","closed_at":null,"body":"I notice the document saying that \r\n\r\n> nullValue – sets the string representation of a null value. If None is set, it uses the default value, empty string. Since 2.0.1, this nullValue param applies to all supported types including the string type.\r\n\r\nHowever, when using display(), the null value shown up is 'null' rather than the specified one, for example, you can compare the output with spark 2.0.1 and 2.0.0\r\n\r\n```python\r\ndf = sqlContext.read.csv('/path/to/file', sep='\\t', schema=schema, quote='\\0', nullValue='')\r\ndisplay(df)\r\n```\r\n\r\nMoreover, the fillna() function does not work in this case either, you can once again compare the outputs of\r\n\r\n```python\r\ndf.fillna({'c1': '~'})\r\ndisplay(df)\r\n```\r\n\r\nIs this a bug or the display() hides the representation of nulls?"},"comment":{"url":"https://api.github.com/repos/databricks/spark-csv/issues/comments/253655966","html_url":"https://github.com/databricks/spark-csv/issues/387#issuecomment-253655966","issue_url":"https://api.github.com/repos/databricks/spark-csv/issues/387","id":253655966,"user":{"login":"falaki","id":512364,"avatar_url":"https://avatars.githubusercontent.com/u/512364?v=3","gravatar_id":"","url":"https://api.github.com/users/falaki","html_url":"https://github.com/falaki","followers_url":"https://api.github.com/users/falaki/followers","following_url":"https://api.github.com/users/falaki/following{/other_user}","gists_url":"https://api.github.com/users/falaki/gists{/gist_id}","starred_url":"https://api.github.com/users/falaki/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/falaki/subscriptions","organizations_url":"https://api.github.com/users/falaki/orgs","repos_url":"https://api.github.com/users/falaki/repos","events_url":"https://api.github.com/users/falaki/events{/privacy}","received_events_url":"https://api.github.com/users/falaki/received_events","type":"User","site_admin":false},"created_at":"2016-10-13T22:22:01Z","updated_at":"2016-10-13T22:22:01Z","body":"@chouzhupk I assume you are using spark-csv in Databricks. `display()` is a Databricks notebook function that is not aware of the configuration you used to parse a CSV dataset. Therefore, it does not replace null values with their string representation. "}}
{"actor":{"display_login":"falaki"}}