title: Wildcard in Hadoop's FileSystem listing API calls
body: <p><strong><em><code>tl;dr:</code></strong> To be able to use wildcards (globs) in the listed paths, one simply has to use <a href="http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#globStatus(org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.PathFilter)" rel="noreferrer"><code>globStatus(...)</code></a> instead of <a href="http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#listStatus(org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.PathFilter)" rel="noreferrer"><code>listStatus(...)</code></a>.</em></p>
  
  <hr>
  
  <h1>Context</h1>
  
  <p>Files on my HDFS cluster are organized in partitions, with the date being the <em>"root"</em> partition. A simplified example of the files structure would look like this:</p>
  
  <pre><code>/schemas_folder
  ├── date=20140101
  │   ├── A-schema.avsc
  │   ├── B-schema.avsc
  ├── date=20140102
  │   ├── A-schema.avsc
  │   ├── B-schema.avsc
  │   ├── C-schema.avsc
  └── date=20140103
      ├── B-schema.avsc
      └── C-schema.avsc
  </code></pre>
  
  <p>In my case, the directory stores <a href="http://avro.apache.org/" rel="noreferrer">Avro</a> schemas for different types of data <em>(A, B and C in this example)</em> at different dates. The schema might start existing, evolve and stop existing... as time passes.</p>
  
  <hr>
  
  <h1>Goal</h1>
  
  <p>I need to be able to get all the schemas that exist for a given type, as quickly as possible. In the example where I would like to get all the schemas that exist for type A, I would like to do the following:</p>
  
  <pre><code>hdfs dfs -ls /schemas_folder/date=*/A-schema.avsc
  </code></pre>
  
  <p>That would give me</p>
  
  <pre><code>Found 1 items
  -rw-r--r--   3 user group 1234 2014-01-01 12:34 /schemas_folder/date=20140101/A-schema.avsc
  Found 1 items
  -rw-r--r--   3 user group 2345 2014-01-02 23:45 /schemas_folder/date=20140102/A-schema.avsc
  </code></pre>
  
  <hr>
  
  <h1>Problem</h1>
  
  <p>I don't want to be using the shell command, and cannot seem to find the equivalent to that command above in the Java APIs. When I try to implement the looping myself, I get terrible performance. I want <em>at least</em> the performance of the command line (around <strong>3 seconds</strong> in my case)...</p>
  
  <hr>
  
  <h1>What I found so far</h1>
  
  <p>One can notice that it prints twice <code>Found 1 items</code>, once before each result. It does not print <code>Found 2 items</code> once at the beginning. That probably hints that wildcards are not implemented on the <code>FileSystem</code> side but somehow handled by the client. I can't seem to find the right source code to look at to see how that was implemented.</p>
  
  <p>Below are my first shots, probably a bit too naïve...</p>
  
  <h2>Using listFiles(...)</h2>
  
  <h3>Code:</h3>
  
  <pre><code>RemoteIterator&lt;LocatedFileStatus&gt; files = filesystem.listFiles(new Path("/schemas_folder"), true);
  Pattern pattern = Pattern.compile("^.*/date=[0-9]{8}/A-schema\\.avsc$");
  while (files.hasNext()) {
      Path path = files.next().getPath();
      if (pattern.matcher(path.toString()).matches())
      {
          System.out.println(path);
      }
  }
  </code></pre>
  
  <h3>Result:</h3>
  
  <p>This prints exactly what I would expect, but since it first lists everything recursively and then filters, the performance is really poor. With my current dataset, it takes almost <strong>25 seconds</strong>...</p>
  
  <h2>Using listStatus(...)</h2>
  
  <h3>Code:</h3>
  
  <pre><code>FileStatus[] statuses = filesystem.listStatus(new Path("/schemas_folder"), new PathFilter()
  {
      private final Pattern pattern = Pattern.compile("^date=[0-9]{8}$");
  
      @Override
      public boolean accept(Path path)
      {
          return pattern.matcher(path.getName()).matches();
      }
  });
  Path[] paths = new Path[statuses.length];
  for (int i = 0; i &lt; statuses.length; i++) { paths[i] = statuses[i].getPath(); }
  statuses = filesystem.listStatus(paths, new PathFilter()
  {
      @Override
      public boolean accept(Path path)
      {
          return "A-schema.avsc".equals(path.getName());
      }
  });
  for (FileStatus status : statuses)
  {
      System.out.println(status.getPath());
  }
  </code></pre>
  
  <h3>Result:</h3>
  
  <p>Thanks to the <code>PathFilter</code>s and the use of arrays, it seems to perform faster (around <strong>12 seconds</strong>). The code is more complex, though, and more difficult to adapt to different situations. Most importantly, the performance is still 3 to 4 times slower than the command-line version!</p>
  
  <hr>
  
  <h1>Question</h1>
  
  <p><strong>What am I missing here? What is the fastest way to get the results I want?</strong></p>
  
  <hr>
  
  <h1>Updates</h1>
  
  <h2>2014.07.09 - 13:38</h2>
  
  <p>The proposed <a href="https://stackoverflow.com/a/24651078/1994863">answer</a> of <a href="https://stackoverflow.com/users/1970125/mukesh-s">Mukesh S</a> is apparently the best possible API approach.</p>
  
  <p>In the example I gave above, the code end-up looking like this:</p>
  
  <pre><code>FileStatus[] statuses = filesystem.globStatus(new Path("/schemas_folder/date=*/A-schema.avsc"));
  for (FileStatus status : statuses)
  {
      System.out.println(status.getPath());
  }
  </code></pre>
  
  <p>This is the best looking and best performing code I could come up with so far, but is still not performing as well as the shell version.</p>