title: java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StorageStatistics
body: <p>I'm trying to run a simple spark to s3 app from a server but I keep getting the below error because the server has hadoop 2.7.3 installed and it looks like it doesn't include the <a href="http://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-common/api/org/apache/hadoop/fs/GlobalStorageStatistics.html" rel="nofollow noreferrer">GlobalStorageStatistics class</a>. I have hadoop 2.8.x defined in my pom.xml file but trying to test it by running it locally. </p>
  
  <p>How can I make it ignore searching for that or what workaround options are there to include that class if I have to go with hadoop 2.7.3?</p>
  
  <pre><code>Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StorageStatistics
      at java.lang.Class.forName0(Native Method)
      at java.lang.Class.forName(Class.java:348)
      at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134)
      at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2099)
      at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
      at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2654)
      at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
      at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
      at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
      at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
      at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
      at org.apache.spark.sql.execution.datasources.DataSource.hasMetadata(DataSource.scala:301)
      at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:344)
      at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:152)
      at org.apache.spark.sql.DataFrameReader.parquet(DataFrameReader.scala:441)
      at org.apache.spark.sql.DataFrameReader.parquet(DataFrameReader.scala:425)
      at com.ibm.cos.jdbc2DF$.main(jdbc2DF.scala:153)
      at com.ibm.cos.jdbc2DF.main(jdbc2DF.scala)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
      at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.StorageStatistics
      at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      ... 28 more
  </code></pre>