{"action":"closed","issue":{"url":"https://api.github.com/repos/nchammas/flintrock/issues/240","repository_url":"https://api.github.com/repos/nchammas/flintrock","labels_url":"https://api.github.com/repos/nchammas/flintrock/issues/240/labels{/name}","comments_url":"https://api.github.com/repos/nchammas/flintrock/issues/240/comments","events_url":"https://api.github.com/repos/nchammas/flintrock/issues/240/events","html_url":"https://github.com/nchammas/flintrock/issues/240","id":306105467,"number":240,"title":"README's instructions for s3a seem incomplete or inaccurate","user":{"login":"qwertystop","id":10714694,"avatar_url":"https://avatars1.githubusercontent.com/u/10714694?v=4","gravatar_id":"","url":"https://api.github.com/users/qwertystop","html_url":"https://github.com/qwertystop","followers_url":"https://api.github.com/users/qwertystop/followers","following_url":"https://api.github.com/users/qwertystop/following{/other_user}","gists_url":"https://api.github.com/users/qwertystop/gists{/gist_id}","starred_url":"https://api.github.com/users/qwertystop/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/qwertystop/subscriptions","organizations_url":"https://api.github.com/users/qwertystop/orgs","repos_url":"https://api.github.com/users/qwertystop/repos","events_url":"https://api.github.com/users/qwertystop/events{/privacy}","received_events_url":"https://api.github.com/users/qwertystop/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-03-16T23:32:15Z","updated_at":"2018-03-17T01:35:22Z","closed_at":"2018-03-17T01:35:22Z","author_association":"NONE","body":"* Flintrock version: 0.9.0\r\n* Python version: 3.5.2\r\n* OS: Linux Mint 16.2\r\n\r\nAccording to the README, to get the cluster to access data on S3 through s3a:\r\n\r\n> We recommend you access data on S3 from your Flintrock cluster by following these steps:\r\n>    1. Setup an IAM Role that grants access to S3 as desired. Reference this role when you launch your cluster using the --ec2-instance-profile-name option (or its equivalent in your config.yaml file).\r\n>    2. Reference S3 paths in your Spark code using the s3a:// prefix. s3a:// is backwards compatible with s3n:// and replaces both s3n:// and s3://. The Hadoop project recommends using s3a:// since it is actively developed, supports larger files, and offers better performance.\r\n>    3. Make sure Flintrock is configured to use Hadoop/HDFS 2.7+. Earlier versions of Hadoop do not have solid implementations of s3a://. Flintrock's default is Hadoop 2.7.4, so you don't need to do anything here if you're using a vanilla configuration.\r\n> With this approach you don't need to copy around your AWS credentials or pass them into your Spark programs. As long as the assigned IAM role allows it, Spark will be able to read and write data to S3 simply by referencing the appropriate path (e.g. s3a://bucket/path/to/file).\r\n\r\nI tried this, and it isn't working; either I'm overlooking something, the documentation skipped a step, or something's wrong in the default configuration. Possibly several of those.\r\n\r\nMy Flintrock configuration is entirely default (as produced by `flintrock configure`), except for filling in key-name, identity-file, and instance-profile-name, and changing the region to `us-east-2`, the AMI to `ami-25615740` (the us-east-2 equivalent of the default), and the instance-type to `t2.micro` (free). But the versions of Spark and HDFS are default (2.2.0 and 2.7.3, from the default download sources). I've reinitialized it (and then made the above specific changes, and no others) to be sure of that.\r\n\r\nI have an IAM role called S3_accessor, with one policy, AmazonS3FullAccess. That's given as the instance-profile-name in Flintrock's configuration.\r\n\r\nThen I run `flintrock launch test-cluster`. No errors are reported. `flintrock login test-cluster`. One message about available updates through `yum`, which I do not apply at this time (for the sake of reproducibility).\r\n\r\nThen, `spark-shell`. Dependency resolution produces no errors. Here's the console after it's done, through to the stack trace when I try to read from S3:\r\n\r\n```\r\nUsing Spark's default log4j profile: org/apache/spark/log4j-defaults.properties\r\nSetting default log level to \"WARN\".\r\nTo adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\r\n18/03/16 23:25:00 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\r\n18/03/16 23:25:10 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException\r\nSpark context Web UI available at http://ec2-18-218-79-104.us-east-2.compute.amazonaws.com:4040\r\nSpark context available as 'sc' (master = local[*], app id = local-1521242701346).\r\nSpark session available as 'spark'.\r\nWelcome to\r\n      ____              __\r\n     / __/__  ___ _____/ /__\r\n    _\\ \\/ _ \\/ _ `/ __/  '_/\r\n   /___/ .__/\\_,_/_/ /_/\\_\\   version 2.2.0\r\n      /_/\r\n         \r\nUsing Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_161)\r\nType in expressions to have them evaluated.\r\nType :help for more information.\r\n\r\nscala> val textFile = sc.textFile(\"s3a://qtest11/realestate.txt\")\r\ntextFile: org.apache.spark.rdd.RDD[String] = s3a://qtest11/realestate.txt MapPartitionsRDD[1] at textFile at <console>:24\r\n\r\nscala> textFile.count()\r\ncom.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS Service: Amazon S3, AWS Request ID: 1A7F70463AA88DFE, AWS Error Code: null, AWS Error Message: Bad Request\r\n  at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798)\r\n  at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421)\r\n  at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)\r\n  at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)\r\n  at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031)\r\n  at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994)\r\n  at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297)\r\n  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2598)\r\n  at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)\r\n  at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2632)\r\n  at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2614)\r\n  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)\r\n  at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)\r\n  at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:256)\r\n  at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:228)\r\n  at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:313)\r\n  at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:194)\r\n  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)\r\n  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)\r\n  at scala.Option.getOrElse(Option.scala:121)\r\n  at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)\r\n  at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)\r\n  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)\r\n  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)\r\n  at scala.Option.getOrElse(Option.scala:121)\r\n  at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)\r\n  at org.apache.spark.SparkContext.runJob(SparkContext.scala:2087)\r\n  at org.apache.spark.rdd.RDD.count(RDD.scala:1158)\r\n  ... 48 elided\r\n\r\nscala> \r\n```\r\nI've confirmed that the file exists; it's a little text file I had lying around, uploaded it with the web interface, downloaded it back with the CLI, all with the same auth as I gave to Flintrock.\r\n\r\nLooking it up, [the only suggestion I found](https://stackoverflow.com/questions/34209196/amazon-s3a-returns-400-bad-request-with-spark#34225930) was to run `sc.hadoopConfiguration.set(\"fs.s3a.endpoint\", \"s3-us-east-2.amazonaws.com\")` before trying to load the text file. This had no effect (identical stack trace).\r\n"}}
{"actor":{"display_login":"qwertystop"}}