{"action":"opened","issue":{"url":"https://api.github.com/repos/databricks/spark-csv/issues/343","repository_url":"https://api.github.com/repos/databricks/spark-csv","labels_url":"https://api.github.com/repos/databricks/spark-csv/issues/343/labels{/name}","comments_url":"https://api.github.com/repos/databricks/spark-csv/issues/343/comments","events_url":"https://api.github.com/repos/databricks/spark-csv/issues/343/events","html_url":"https://github.com/databricks/spark-csv/issues/343","id":161724591,"number":343,"title":"Error while adding spark-csv library in pyspark through cmd","user":{"login":"mahima221","id":20094297,"avatar_url":"https://avatars.githubusercontent.com/u/20094297?v=3","gravatar_id":"","url":"https://api.github.com/users/mahima221","html_url":"https://github.com/mahima221","followers_url":"https://api.github.com/users/mahima221/followers","following_url":"https://api.github.com/users/mahima221/following{/other_user}","gists_url":"https://api.github.com/users/mahima221/gists{/gist_id}","starred_url":"https://api.github.com/users/mahima221/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mahima221/subscriptions","organizations_url":"https://api.github.com/users/mahima221/orgs","repos_url":"https://api.github.com/users/mahima221/repos","events_url":"https://api.github.com/users/mahima221/events{/privacy}","received_events_url":"https://api.github.com/users/mahima221/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-06-22T16:12:19Z","updated_at":"2016-06-22T16:12:19Z","closed_at":null,"body":"I'm trying to add spark-csv library to use in my pyspark IDE (PyCharm). I have Spark 1.6.1 built for Hadoop 2.4.0 and Scala version 2.10.5. I'm getting following errors when I try to add it through cmd. I want to access spark-csv from IDE directly as well.\r\n\r\n\r\n```\r\n    C:\\spark-1.6.1-bin-hadoop2.4>bin\\pyspark --packages com.databricks:spark-csv_2.10:1.0.3\r\n    Python 2.7.8 (default, Jun 30 2014, 16:03:49) [MSC v.1500 32 bit (Intel)] on win32\r\n    Type \"help\", \"copyright\", \"credits\" or \"license\" for more information.\r\n    Ivy Default Cache set to: C:\\Users\\Mahima\\.ivy2\\cache\r\n    The jars for the packages stored in: C:\\Users\\Mahima\\.ivy2\\jars\r\n    :: loading settings :: url = jar:file:/C:/spark-1.6.1-bin-hadoop2.4/lib/spark-assembly-1.6.1-hadoop2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml\r\n    com.databricks#spark-csv_2.10 added as a dependency\r\n    :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0\r\n            confs: [default]\r\n            found com.databricks#spark-csv_2.10;1.0.3 in central\r\n            found org.apache.commons#commons-csv;1.1 in central\r\n    downloading https://repo1.maven.org/maven2/com/databricks/spark-csv_2.10/1.0.3/spark-csv_2.10-1.0.3.jar ...\r\n            [SUCCESSFUL ] com.databricks#spark-csv_2.10;1.0.3!spark-csv_2.10.jar (833ms)\r\n    :: resolution report :: resolve 6592ms :: artifacts dl 857ms\r\n            :: modules in use:\r\n            com.databricks#spark-csv_2.10;1.0.3 from central in [default]\r\n            org.apache.commons#commons-csv;1.1 from central in [default]\r\n            ---------------------------------------------------------------------\r\n            |                  |            modules            ||   artifacts   |\r\n            |       conf       | number| search|dwnlded|evicted|| number|dwnlded|\r\n            ---------------------------------------------------------------------\r\n            |      default     |   2   |   1   |   1   |   0   ||   2   |   1   |\r\n            ---------------------------------------------------------------------\r\n    :: retrieving :: org.apache.spark#spark-submit-parent\r\n            confs: [default]\r\n            1 artifacts copied, 1 already retrieved (54kB/60ms)\r\n    16/06/22 20:42:02 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\r\n    16/06/22 20:42:03 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path\r\n    java.io.IOException: Could not locate executable null\\bin\\winutils.exe in the Hadoop binaries.\r\n            at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)\r\n            at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)\r\n            at org.apache.hadoop.util.Shell.<clinit>(Shell.java:326)\r\n            at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)\r\n            at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:93)\r\n            at org.apache.hadoop.security.Groups.<init>(Groups.java:77)\r\n            at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:240)\r\n            at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:255)\r\n            at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:232)\r\n            at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:718)\r\n            at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:703)\r\n            at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:605)\r\n            at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2160)\r\n            at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2160)\r\n            at scala.Option.getOrElse(Option.scala:120)\r\n            at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2160)\r\n            at org.apache.spark.SparkContext.<init>(SparkContext.scala:322)\r\n            at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:59)\r\n            at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\r\n            at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)\r\n            at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)\r\n            at java.lang.reflect.Constructor.newInstance(Unknown Source)\r\n            at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)\r\n            at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)\r\n            at py4j.Gateway.invoke(Gateway.java:214)\r\n            at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)\r\n            at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)\r\n            at py4j.GatewayConnection.run(GatewayConnection.java:209)\r\n            at java.lang.Thread.run(Unknown Source)\r\n    16/06/22 20:42:26 WARN : Your hostname, Mahima resolves to a loopback/non-reachable address: fe80:0:0:0:4950:202:cee6:f31f%22, but we couldn't find any external IP address!\r\n    16/06/22 20:42:28 ERROR SparkContext: Error initializing SparkContext.\r\n    java.lang.NullPointerException\r\n            at java.lang.ProcessBuilder.start(Unknown Source)\r\n            at org.apache.hadoop.util.Shell.runCommand(Shell.java:445)\r\n            at org.apache.hadoop.util.Shell.run(Shell.java:418)\r\n            at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)\r\n            at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:873)\r\n            at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:853)\r\n            at org.apache.spark.util.Utils$.fetchFile(Utils.scala:406)\r\n            at org.apache.spark.SparkContext.addFile(SparkContext.scala:1386)\r\n            at org.apache.spark.SparkContext.addFile(SparkContext.scala:1340)\r\n            at org.apache.spark.SparkContext$$anonfun$15.apply(SparkContext.scala:491)\r\n            at org.apache.spark.SparkContext$$anonfun$15.apply(SparkContext.scala:491)\r\n            at scala.collection.immutable.List.foreach(List.scala:318)\r\n            at org.apache.spark.SparkContext.<init>(SparkContext.scala:491)\r\n            at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:59)\r\n            at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\r\n            at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)\r\n            at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)\r\n            at java.lang.reflect.Constructor.newInstance(Unknown Source)\r\n            at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)\r\n            at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)\r\n            at py4j.Gateway.invoke(Gateway.java:214)\r\n            at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)\r\n            at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)\r\n            at py4j.GatewayConnection.run(GatewayConnection.java:209)\r\n            at java.lang.Thread.run(Unknown Source)\r\n    16/06/22 20:42:28 WARN MetricsSystem: Stopping a MetricsSystem that is not running\r\n    Traceback (most recent call last):\r\n      File \"C:\\spark-1.6.1-bin-hadoop2.4\\bin\\..\\python\\pyspark\\shell.py\", line 43, in <module>\r\n        sc = SparkContext(pyFiles=add_files)\r\n      File \"C:\\spark-1.6.1-bin-hadoop2.4\\python\\pyspark\\context.py\", line 115, in __init__\r\n        conf, jsc, profiler_cls)\r\n      File \"C:\\spark-1.6.1-bin-hadoop2.4\\python\\pyspark\\context.py\", line 172, in _do_init\r\n        self._jsc = jsc or self._initialize_context(self._conf._jconf)\r\n      File \"C:\\spark-1.6.1-bin-hadoop2.4\\python\\pyspark\\context.py\", line 235, in _initialize_context\r\n        return self._jvm.JavaSparkContext(jconf)\r\n      File \"C:\\spark-1.6.1-bin-hadoop2.4\\python\\lib\\py4j-0.9-src.zip\\py4j\\java_gateway.py\", line 1064, in __call__\r\n      File \"C:\\spark-1.6.1-bin-hadoop2.4\\python\\lib\\py4j-0.9-src.zip\\py4j\\protocol.py\", line 308, in get_return_value\r\n    py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.\r\n    : java.lang.NullPointerException\r\n            at java.lang.ProcessBuilder.start(Unknown Source)\r\n            at org.apache.hadoop.util.Shell.runCommand(Shell.java:445)\r\n            at org.apache.hadoop.util.Shell.run(Shell.java:418)\r\n            at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)\r\n            at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:873)\r\n            at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:853)\r\n            at org.apache.spark.util.Utils$.fetchFile(Utils.scala:406)\r\n            at org.apache.spark.SparkContext.addFile(SparkContext.scala:1386)\r\n            at org.apache.spark.SparkContext.addFile(SparkContext.scala:1340)\r\n            at org.apache.spark.SparkContext$$anonfun$15.apply(SparkContext.scala:491)\r\n            at org.apache.spark.SparkContext$$anonfun$15.apply(SparkContext.scala:491)\r\n            at scala.collection.immutable.List.foreach(List.scala:318)\r\n            at org.apache.spark.SparkContext.<init>(SparkContext.scala:491)\r\n            at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:59)\r\n            at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\r\n            at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)\r\n            at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)\r\n            at java.lang.reflect.Constructor.newInstance(Unknown Source)\r\n            at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)\r\n            at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)\r\n            at py4j.Gateway.invoke(Gateway.java:214)\r\n            at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)\r\n            at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)\r\n            at py4j.GatewayConnection.run(GatewayConnection.java:209)\r\n            at java.lang.Thread.run(Unknown Source)\r\n    \r\n    >>> from pyspark.sql import SQLContext\r\n    >>> sqlcon = SQLContext(sc)\r\n    Traceback (most recent call last):\r\n      File \"<stdin>\", line 1, in <module>\r\n    NameError: name 'sc' is not defined\r\n    >>> sqlcon = SQLContext()\r\n    Traceback (most recent call last):\r\n      File \"<stdin>\", line 1, in <module>\r\n    TypeError: __init__() takes at least 2 arguments (1 given)\r\n    >>>\r\n```\r\n\r\nThanks in advance."}}
{"actor":{"display_login":"mahima221"}}