title: How Can I Load Every File In a Folder Using PIG?
body: <p>I have a folder of files created daily that all store the same type of information. I'd like to make a script that loads the newest 10 of them, UNIONs them, and then runs some other code on them. Since pig already has an ls method, I was wondering if there was a simple way for me to get the last 10 created files, and load them all under generic names using the same loader and options. I'm guessing it would look something like:</p>
  
  <pre><code>REGISTER /usr/local/lib/hadoop/hadoop-lzo-0.4.13.jar;
  REGISTER /usr/local/lib/hadoop/elephant-bird-2.0.5.jar;
  FOREACH file in some_path:
      file = LOAD 'file' 
      USING com.twitter.elephantbird.pig.load.LzoTokenizedLoader('\\t') 
      AS (i1, i2, i3);
  </code></pre>