title: Hadoop HDFS Write Operation Programmatically
body: <p>I asked a similar question a while back but then I had not idea what I was talking about. I am posting this question with further details and to the point queries.</p>
  
  <p>So I have set up hadoop cluster with namenode and 2 datanodes. I am using hadoop 2.9.0. I ran the command hdfs dfs -put "SomeRandomFile" and it seems to be working ok. The only confusion I have here is why does it store my file to /user/hduser/ path? I didn't specify this path anywhere in configurations so how is it building this path on hdfs?</p>
  
  <p>Furthermore I created a small java program to do the same thing. I created a simple eclipse project and wrote following lines:</p>
  
  <pre><code>public static boolean fileWriteHDFS(InputStream input, String fileName) {   
      try {
          System.setProperty("HADOOP_USER_NAME", "hduser");
  
          //Get Configuration of Hadoop system
          Configuration conf = new Configuration();
          conf.set("fs.defaultFS", "hdfs://localhost:9000");
          //conf.get("fs.defaultFS");     
  
          //Extract destination path
          URI uri = URI.create(DESTINATION_PATH+fileName);
          Path path = new Path(uri);
  
          //Destination file in HDFS
          FileSystem fs = FileSystem.get(uri, conf); //.get(conf);
  
          //Check if the file already exists
          if (fs.exists(path))
          {
              //Write appropriate error to log file and return.
              return false;
          }
  
          //Create an Output stream to the destination path
          FSDataOutputStream out = fs.create(path);
  
          //Copy file from input steam to HDFSs
          IOUtils.copyBytes(input, out, 4096, true);
  
          //Close all the file descriptors
          out.close();
          fs.close();
          //All went perfectly as planned
          return true;    
      } catch (Exception e) {
          //Something went wrong
          System.out.println(e.toString());
          return false;
      }
  }
  </code></pre>
  
  <p>And I added following three hadoop libraries:</p>
  
  <blockquote>
    <p>/home/hduser/bin/hadoop-2.9.0/share/hadoop/common/hadoop-common-2.9.0.jar
    /home/hduser/bin/hadoop-2.9.0/share/hadoop/common/hadoop-common-2.9.0-tests.jar
    /home/hduser/bin/hadoop-2.9.0/share/hadoop/common/hadoop-nfs-2.9.0.jar</p>
  </blockquote>
  
  <p>As you can see my hadoop installation location is /home/hduser/bin/hadoop-2.9.0/... When I run this code it throws an exception. i.e.</p>
  
  <pre><code>Exception in thread "main" java.lang.NoClassDefFoundError: com/ctc/wstx/io/InputBootstrapper
  at com.ws.filewrite.fileWrite.fileWriteHDFS(fileWrite.java:21)
  at com.ws.main.listenerService.main(listenerService.java:21)
  Caused by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper
      at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      ... 2 more
  </code></pre>
  
  <p>Specifically the exception is thrown at the lie: </p>
  
  <blockquote>
    <p>Configuration conf = new Configuration();</p>
  </blockquote>
  
  <p>Am I missing something here? What is causing this problem? I am completely new to HDFS so pardon me it is obvious problem.</p>