{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/22273","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/22273/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/22273/comments","events_url":"https://api.github.com/repos/apache/spark/issues/22273/events","html_url":"https://github.com/apache/spark/pull/22273","id":355328258,"node_id":"MDExOlB1bGxSZXF1ZXN0MjExODgzNDU4","number":22273,"title":"[SPARK-25272][PYTHON][TEST] Add test to better indicate pyarrow is installed and related tests will run","user":{"login":"BryanCutler","id":4534389,"node_id":"MDQ6VXNlcjQ1MzQzODk=","avatar_url":"https://avatars3.githubusercontent.com/u/4534389?v=4","gravatar_id":"","url":"https://api.github.com/users/BryanCutler","html_url":"https://github.com/BryanCutler","followers_url":"https://api.github.com/users/BryanCutler/followers","following_url":"https://api.github.com/users/BryanCutler/following{/other_user}","gists_url":"https://api.github.com/users/BryanCutler/gists{/gist_id}","starred_url":"https://api.github.com/users/BryanCutler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/BryanCutler/subscriptions","organizations_url":"https://api.github.com/users/BryanCutler/orgs","repos_url":"https://api.github.com/users/BryanCutler/repos","events_url":"https://api.github.com/users/BryanCutler/events{/privacy}","received_events_url":"https://api.github.com/users/BryanCutler/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":16,"created_at":"2018-08-29T20:51:38Z","updated_at":"2018-08-31T01:47:06Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/22273","html_url":"https://github.com/apache/spark/pull/22273","diff_url":"https://github.com/apache/spark/pull/22273.diff","patch_url":"https://github.com/apache/spark/pull/22273.patch"},"body":"## What changes were proposed in this pull request?\r\nCurrently, Arrow tests in python are skipped if pyarrow is not installed and there is clear Jenkins outputs that these tests were skipped. When it is installed, the tests are run but there will be no output. This adds a test that will be skipped and output a message in the Jenkins log that pyarrow and pandas required versions are installed. This way it is clear that Arrow tests will be run.  The output given when pyarrow and pandas are installed is:\r\n`test_required_pyarrow_pandas_not_installed (pyspark.sql.tests.HaveArrowTests) ... skipped 'Required pyarrow and pandas were found, Arrow tests will run'\r\n\r\n## How was this patch tested?\r\n\r\nAdded new test"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/417523547","html_url":"https://github.com/apache/spark/pull/22273#issuecomment-417523547","issue_url":"https://api.github.com/repos/apache/spark/issues/22273","id":417523547,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNzUyMzU0Nw==","user":{"login":"HyukjinKwon","id":6477701,"node_id":"MDQ6VXNlcjY0Nzc3MDE=","avatar_url":"https://avatars0.githubusercontent.com/u/6477701?v=4","gravatar_id":"","url":"https://api.github.com/users/HyukjinKwon","html_url":"https://github.com/HyukjinKwon","followers_url":"https://api.github.com/users/HyukjinKwon/followers","following_url":"https://api.github.com/users/HyukjinKwon/following{/other_user}","gists_url":"https://api.github.com/users/HyukjinKwon/gists{/gist_id}","starred_url":"https://api.github.com/users/HyukjinKwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HyukjinKwon/subscriptions","organizations_url":"https://api.github.com/users/HyukjinKwon/orgs","repos_url":"https://api.github.com/users/HyukjinKwon/repos","events_url":"https://api.github.com/users/HyukjinKwon/events{/privacy}","received_events_url":"https://api.github.com/users/HyukjinKwon/received_events","type":"User","site_admin":false},"created_at":"2018-08-31T01:47:06Z","updated_at":"2018-08-31T01:47:06Z","author_association":"MEMBER","body":"> Yeah but for people who don't know these tests or how the _have_xxx flags work, having an additional positive confirmation that the Arrow tests are run is nice. We already are doing this for pandas here.\r\n\r\nhmm .. I thought the current information is enough to indicate which Arrow or Pandas we would use and test. For instance,\r\n\r\n```\r\nSkipped tests in pyspark.sql.tests with pypy:\r\n    test_createDataFrame_column_name_encoding (pyspark.sql.tests.ArrowTests) ... skipped 'Pandas >= 0.19.2 must be installed; however, it was not found.'\r\n    ...\r\n\r\nSkipped tests in pyspark.sql.tests with python2.7:\r\n    test_createDataFrame_column_name_encoding (pyspark.sql.tests.ArrowTests) ... skipped 'Pandas >= 0.19.2 must be installed; however, your version was 0.16.0.'\r\n    ...\r\n```\r\n\r\nYea, looks we are already doing this for Pandas .. but actually I would rather remove that test too."}}
{"actor":{"display_login":"HyukjinKwon"}}