title: GoogleHadoopFileSystem cannot be cast to hadoop FileSystem?
body: <p>The original question was <a href="https://stackoverflow.com/questions/31466206/spark-1-4-image-for-google-cloud"> trying to deploy spark 1.4 on Google Cloud</a>. After downloaded and set </p>
  
  <pre><code>SPARK_HADOOP2_TARBALL_URI='gs://my_bucket/my-images/spark-1.4.1-bin-hadoop2.6.tgz'
  </code></pre>
  
  <p>deployment with bdutil was fine; however, when trying to call SqlContext.parquetFile("gs://my_bucket/some_data.parquet"), it runs into following exception:</p>
  
  <pre><code> java.lang.ClassCastException: com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem cannot be cast to org.apache.hadoop.fs.FileSystem
  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2595)
  at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
  at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
  at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:354)
  at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
  at org.apache.hadoop.hive.metastore.Warehouse.getFs(Warehouse.java:112)
  at org.apache.hadoop.hive.metastore.Warehouse.getDnsPath(Warehouse.java:144)
  at org.apache.hadoop.hive.metastore.Warehouse.getWhRoot(Warehouse.java:159)
  </code></pre>
  
  <p>And what confused me is that GoogleHadoopFileSystem should be a subclass of org.apache.hadoop.fs.FileSystem, and I even verified in the same spark-shell instance:</p>
  
  <pre><code>scala&gt; var gfs = new com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem()
  gfs: com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem = com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem@46f105c
  
  scala&gt; gfs.isInstanceOf[org.apache.hadoop.fs.FileSystem]
  res3: Boolean = true
  
  scala&gt; gfs.asInstanceOf[org.apache.hadoop.fs.FileSystem]
  res4: org.apache.hadoop.fs.FileSystem = com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem@46f105c
  </code></pre>
  
  <p>Did I miss anything, any workaround? Thanks in advance!</p>
  
  <p>UPDATE: this is my bdutil (version 1.3.1) setting for deployment:</p>
  
  <pre><code>import_env hadoop2_env.sh
  import_env extensions/spark/spark_env.sh
  CONFIGBUCKET="my_conf_bucket"
  PROJECT="my_proj"
  GCE_IMAGE='debian-7-backports'
  GCE_MACHINE_TYPE='n1-highmem-4'
  GCE_ZONE='us-central1-f'
  GCE_NETWORK='my-network'
  GCE_MASTER_MACHINE_TYPE='n1-standard-2'
  PREEMPTIBLE_FRACTION=1.0
  PREFIX='my-hadoop'
  NUM_WORKERS=8
  USE_ATTACHED_PDS=true
  WORKER_ATTACHED_PDS_SIZE_GB=200
  MASTER_ATTACHED_PD_SIZE_GB=200
  HADOOP_TARBALL_URI="gs://hadoop-dist/hadoop-2.6.0.tar.gz"
  SPARK_MODE="yarn-client"
  SPARK_HADOOP2_TARBALL_URI='gs://my_conf_bucket/my-images/spark-1.4.1-bin-hadoop2.6.tgz'
  </code></pre>