{"action":"created","issue":{"url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/issues/476","repository_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic","labels_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/issues/476/labels{/name}","comments_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/issues/476/comments","events_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/issues/476/events","html_url":"https://github.com/jupyter-incubator/sparkmagic/issues/476","id":354891476,"node_id":"MDU6SXNzdWUzNTQ4OTE0NzY=","number":476,"title":"Livy 0.5.0 ","user":{"login":"abronte","id":44644,"node_id":"MDQ6VXNlcjQ0NjQ0","avatar_url":"https://avatars0.githubusercontent.com/u/44644?v=4","gravatar_id":"","url":"https://api.github.com/users/abronte","html_url":"https://github.com/abronte","followers_url":"https://api.github.com/users/abronte/followers","following_url":"https://api.github.com/users/abronte/following{/other_user}","gists_url":"https://api.github.com/users/abronte/gists{/gist_id}","starred_url":"https://api.github.com/users/abronte/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/abronte/subscriptions","organizations_url":"https://api.github.com/users/abronte/orgs","repos_url":"https://api.github.com/users/abronte/repos","events_url":"https://api.github.com/users/abronte/events{/privacy}","received_events_url":"https://api.github.com/users/abronte/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-08-28T20:29:48Z","updated_at":"2018-08-28T20:55:00Z","closed_at":"2018-08-28T20:55:00Z","author_association":"NONE","body":"I recently tried upgrading from AWS EMR version 5.13.0 to 5.16.0 which bumps Livy from 0.4.0 to 0.5.0 and I'm seeing issues in sparkmagic that aren't happening in vanilla spark.\r\n\r\nI'm using sparkmagic 0.12.5.\r\n\r\nIt seems to be around the `sqlContext` object isn't the same anymore. \r\n\r\nWhen I run this in pyspark is works just fine.\r\n```\r\n[root@ip-10-0-0-32 ~]# pyspark\r\nPython 2.7.14 (default, May  2 2018, 18:31:34)\r\n[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux2\r\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\r\n18/08/28 18:50:37 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.\r\nWelcome to\r\n      ____              __\r\n     / __/__  ___ _____/ /__\r\n    _\\ \\/ _ \\/ _ `/ __/  '_/\r\n   /__ / .__/\\_,_/_/ /_/\\_\\   version 2.3.1\r\n      /_/\r\n\r\nUsing Python version 2.7.14 (default, May  2 2018 18:31:34)\r\nSparkSession available as 'spark'.\r\n>>> from pyspark.sql import SQLContext\r\n>>> my_sql_context = SQLContext.getOrCreate(sc)\r\n>>> df = my_sql_context.read.parquet('s3://my-bucket/mydata.parquet')\r\n>>> print(df.count())\r\n67556724\r\n```\r\n\r\nBut the same code executed via jupyter and sparkmagic, I get this.\r\n```\r\nfrom pyspark.sql import SQLContext\r\nmy_sql_context = SQLContext.getOrCreate(sc)\r\ndf = my_sql_context.read.parquet('s3://my-bucket/mydata.parquet')\r\nprint(df.count())\r\n\r\n'JavaMember' object has no attribute 'read'\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/context.py\", line 433, in read\r\n    return DataFrameReader(self)\r\n  File \"/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py\", line 70, in __init__\r\n    self._jreader = spark._ssql_ctx.read()\r\nAttributeError: 'JavaMember' object has no attribute 'read'\r\n```\r\n\r\nIts the same issue with the default sqlContext object. \r\n```\r\ndf = sqlContext.read.parquet('s3://my-bucket/mydata.parquet')\r\n\r\n'JavaMember' object has no attribute 'read'\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/context.py\", line 433, in read\r\n    return DataFrameReader(self)\r\n  File \"/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py\", line 70, in __init__\r\n    self._jreader = spark._ssql_ctx.read()\r\nAttributeError: 'JavaMember' object has no attribute 'read'\r\n```\r\n\r\nHas anyone else run into this problem? I'm not sure if this is a Livy or sparkmagic issue. "},"comment":{"url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/issues/comments/416737176","html_url":"https://github.com/jupyter-incubator/sparkmagic/issues/476#issuecomment-416737176","issue_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/issues/476","id":416737176,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNjczNzE3Ng==","user":{"login":"abronte","id":44644,"node_id":"MDQ6VXNlcjQ0NjQ0","avatar_url":"https://avatars0.githubusercontent.com/u/44644?v=4","gravatar_id":"","url":"https://api.github.com/users/abronte","html_url":"https://github.com/abronte","followers_url":"https://api.github.com/users/abronte/followers","following_url":"https://api.github.com/users/abronte/following{/other_user}","gists_url":"https://api.github.com/users/abronte/gists{/gist_id}","starred_url":"https://api.github.com/users/abronte/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/abronte/subscriptions","organizations_url":"https://api.github.com/users/abronte/orgs","repos_url":"https://api.github.com/users/abronte/repos","events_url":"https://api.github.com/users/abronte/events{/privacy}","received_events_url":"https://api.github.com/users/abronte/received_events","type":"User","site_admin":false},"created_at":"2018-08-28T20:55:00Z","updated_at":"2018-08-28T20:55:00Z","author_association":"NONE","body":"After calling the API directly, I've confirmed this is a Livy issue. https://issues.apache.org/jira/browse/LIVY-504"}}
{"actor":{"display_login":"abronte"}}