{"action":"created","issue":{"url":"https://api.github.com/repos/jupyter/enterprise_gateway/issues/523","repository_url":"https://api.github.com/repos/jupyter/enterprise_gateway","labels_url":"https://api.github.com/repos/jupyter/enterprise_gateway/issues/523/labels{/name}","comments_url":"https://api.github.com/repos/jupyter/enterprise_gateway/issues/523/comments","events_url":"https://api.github.com/repos/jupyter/enterprise_gateway/issues/523/events","html_url":"https://github.com/jupyter/enterprise_gateway/issues/523","id":389144414,"node_id":"MDU6SXNzdWUzODkxNDQ0MTQ=","number":523,"title":"Python + Spark doesnt contain support for AWS S3","user":{"login":"rayh","id":57468,"node_id":"MDQ6VXNlcjU3NDY4","avatar_url":"https://avatars3.githubusercontent.com/u/57468?v=4","gravatar_id":"","url":"https://api.github.com/users/rayh","html_url":"https://github.com/rayh","followers_url":"https://api.github.com/users/rayh/followers","following_url":"https://api.github.com/users/rayh/following{/other_user}","gists_url":"https://api.github.com/users/rayh/gists{/gist_id}","starred_url":"https://api.github.com/users/rayh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rayh/subscriptions","organizations_url":"https://api.github.com/users/rayh/orgs","repos_url":"https://api.github.com/users/rayh/repos","events_url":"https://api.github.com/users/rayh/events{/privacy}","received_events_url":"https://api.github.com/users/rayh/received_events","type":"User","site_admin":false},"labels":[{"id":1088948082,"node_id":"MDU6TGFiZWwxMDg4OTQ4MDgy","url":"https://api.github.com/repos/jupyter/enterprise_gateway/labels/Configuration","name":"Configuration","color":"f7ef7b","default":false},{"id":605725523,"node_id":"MDU6TGFiZWw2MDU3MjU1MjM=","url":"https://api.github.com/repos/jupyter/enterprise_gateway/labels/Infrastructure%20&%20Build","name":"Infrastructure & Build","color":"f9d0c4","default":false},{"id":950929566,"node_id":"MDU6TGFiZWw5NTA5Mjk1NjY=","url":"https://api.github.com/repos/jupyter/enterprise_gateway/labels/Kubernetes","name":"Kubernetes","color":"1d76db","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/jupyter/enterprise_gateway/milestones/27","html_url":"https://github.com/jupyter/enterprise_gateway/milestone/27","labels_url":"https://api.github.com/repos/jupyter/enterprise_gateway/milestones/27/labels","id":3392041,"node_id":"MDk6TWlsZXN0b25lMzM5MjA0MQ==","number":27,"title":"v2.0","description":"Enterprise Gateway v2.0 builds on the features of v1.0 by adding support for Kubernetes clusters - both Spark-enabled and generic.","creator":{"login":"kevin-bates","id":22599560,"node_id":"MDQ6VXNlcjIyNTk5NTYw","avatar_url":"https://avatars0.githubusercontent.com/u/22599560?v=4","gravatar_id":"","url":"https://api.github.com/users/kevin-bates","html_url":"https://github.com/kevin-bates","followers_url":"https://api.github.com/users/kevin-bates/followers","following_url":"https://api.github.com/users/kevin-bates/following{/other_user}","gists_url":"https://api.github.com/users/kevin-bates/gists{/gist_id}","starred_url":"https://api.github.com/users/kevin-bates/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kevin-bates/subscriptions","organizations_url":"https://api.github.com/users/kevin-bates/orgs","repos_url":"https://api.github.com/users/kevin-bates/repos","events_url":"https://api.github.com/users/kevin-bates/events{/privacy}","received_events_url":"https://api.github.com/users/kevin-bates/received_events","type":"User","site_admin":false},"open_issues":10,"closed_issues":40,"state":"open","created_at":"2018-06-01T00:01:48Z","updated_at":"2018-12-10T16:24:07Z","due_on":null,"closed_at":null},"comments":0,"created_at":"2018-12-10T05:17:51Z","updated_at":"2018-12-10T16:24:30Z","closed_at":null,"author_association":"NONE","body":"```\r\nPy4JJavaErrorTraceback (most recent call last)\r\n/usr/local/share/jupyter/kernel-launchers/python/scripts/launch_ipykernel.py in <module>()\r\n----> 1 df = spark.read.format(\"csv\").option(\"header\", \"true\").load(\"s3a://eliiza-sydney-datasets-melb-parking-sensors/parking-sensor-2016.csv\")\r\n\r\n/opt/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py in load(self, path, format, schema, **options)\r\n    164         self.options(**options)\r\n    165         if isinstance(path, basestring):\r\n--> 166             return self._df(self._jreader.load(path))\r\n    167         elif path is not None:\r\n    168             if type(path) != list:\r\n\r\n/opt/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in __call__(self, *args)\r\n   1255         answer = self.gateway_client.send_command(command)\r\n   1256         return_value = get_return_value(\r\n-> 1257             answer, self.gateway_client, self.target_id, self.name)\r\n   1258 \r\n   1259         for temp_arg in temp_args:\r\n\r\n/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py in deco(*a, **kw)\r\n     61     def deco(*a, **kw):\r\n     62         try:\r\n---> 63             return f(*a, **kw)\r\n     64         except py4j.protocol.Py4JJavaError as e:\r\n     65             s = e.java_exception.toString()\r\n\r\n/opt/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)\r\n    326                 raise Py4JJavaError(\r\n    327                     \"An error occurred while calling {0}{1}{2}.\\n\".\r\n--> 328                     format(target_id, \".\", name), value)\r\n    329             else:\r\n    330                 raise Py4JError(\r\n\r\nPy4JJavaError: An error occurred while calling o189.load.\r\n: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found\r\n\tat org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)\r\n\tat org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2654)\r\n\tat org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)\r\n\tat org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)\r\n\tat org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)\r\n\tat org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)\r\n\tat org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)\r\n\tat org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)\r\n\tat org.apache.spark.sql.execution.datasources.DataSource$$anonfun$org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary$1.apply(DataSource.scala:547)\r\n\tat org.apache.spark.sql.execution.datasources.DataSource$$anonfun$org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary$1.apply(DataSource.scala:545)\r\n\tat scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)\r\n\tat scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)\r\n\tat scala.collection.immutable.List.foreach(List.scala:392)\r\n\tat scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)\r\n\tat scala.collection.immutable.List.flatMap(List.scala:355)\r\n\tat org.apache.spark.sql.execution.datasources.DataSource.org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary(DataSource.scala:545)\r\n\tat org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:359)\r\n\tat org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)\r\n\tat org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)\r\n\tat org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:498)\r\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\r\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)\r\n\tat py4j.Gateway.invoke(Gateway.java:282)\r\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\r\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\r\n\tat py4j.GatewayConnection.run(GatewayConnection.java:238)\r\n\tat java.lang.Thread.run(Thread.java:748)\r\nCaused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found\r\n\tat org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)\r\n\tat org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)\r\n\t... 30 more\r\n```\r\n\r\nI suspect I'm missing something obvious here.\r\n"},"comment":{"url":"https://api.github.com/repos/jupyter/enterprise_gateway/issues/comments/445876959","html_url":"https://github.com/jupyter/enterprise_gateway/issues/523#issuecomment-445876959","issue_url":"https://api.github.com/repos/jupyter/enterprise_gateway/issues/523","id":445876959,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NTg3Njk1OQ==","user":{"login":"kevin-bates","id":22599560,"node_id":"MDQ6VXNlcjIyNTk5NTYw","avatar_url":"https://avatars0.githubusercontent.com/u/22599560?v=4","gravatar_id":"","url":"https://api.github.com/users/kevin-bates","html_url":"https://github.com/kevin-bates","followers_url":"https://api.github.com/users/kevin-bates/followers","following_url":"https://api.github.com/users/kevin-bates/following{/other_user}","gists_url":"https://api.github.com/users/kevin-bates/gists{/gist_id}","starred_url":"https://api.github.com/users/kevin-bates/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kevin-bates/subscriptions","organizations_url":"https://api.github.com/users/kevin-bates/orgs","repos_url":"https://api.github.com/users/kevin-bates/repos","events_url":"https://api.github.com/users/kevin-bates/events{/privacy}","received_events_url":"https://api.github.com/users/kevin-bates/received_events","type":"User","site_admin":false},"created_at":"2018-12-10T16:24:30Z","updated_at":"2018-12-10T16:24:30Z","author_association":"COLLABORATOR","body":"Hmm - The [base spark images](https://hub.docker.com/r/elyra/spark-py/) were built using the Spark 2.4.0 image builder script.  Not sure how AWS S3 gets added or made available.\r\n\r\nSomeone will need to dig deeper on this. "}}
{"actor":{"display_login":"kevin-bates"}}