title: Shell script: segregate multiple files
body: <p>I have this in my local directory ~/Report:</p>
  
  <pre><code>Rep_{ReportType}_{Date}_{Seq}.csv
  
  Rep_0001_20150102_0.csv
  Rep_0001_20150102_1.csv
  Rep_0102_20150102_0.csv
  Rep_0503_20150102_0.csv
  Rep_0503_20150102_0.csv
  </code></pre>
  
  <p>Using shell-script, </p>
  
  <ol>
  <li><p>How do I get multiple files from a local directory with a fixed batch size?</p></li>
  <li><p>How do I segregate/group the files together by report type (0001 files are grouped together, 0102 grouped together, 0503 grouped together, etc.)</p></li>
  </ol>
  
  <p>I will generate a sequence file (using forqlift) for EACH group/report type. The output would be Report0001.seq, Report0102.seq, Report0503.seq (3 sequence files). In which I will save to a different directory.</p>
  
  <p>Note: In sequence files, the key is the filename of csv (Rep_0001_20150102.csv), and the value is the content of the file. It is stored as [String, BytesWritable].</p>
  
  <p>This is my code: </p>
  
  <pre><code>1  reportTypes=(0001 0102 8902)
  2
  3  # collect all files matching expression into an array
  4  filesWithDir=(~/Report/Rep_[0-9][0-9][0-9][0-9]_[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]_[0-1].csv)
  5
  6  # take only the first hundred
  7  filesWithDir =( "${filesWithDir[@]:0:100}" )
  8
  9  # files="${filesWithDir[@]##*/}" #### commented out since forqlift cannot create sequence file without the path/to/file
  10 # echo ${files[@]}
  11
  12 shopt -s nullglob
  13
  14 # Line 21 is commented out since it has a bug. It collects files in
  15 # current directory when it should be filtering the "files array" created
  16 # in line 7
  17
  18
  19 for i in ${reportTypes[@]}; do
  20   printf -v val '%04d' "$i"
  21   # files=("Rep_${val}_"*.csv) 
       # solution to BUG: (filter files array)
       groupFiles=( $( for j in ${filesWithDir[@]} ; do echo $j ; done | grep ${val} ) )
  22
  23   # Generate sequence file for EACH Report Type
  24   forqlift create --file="Report${val}.seq" "${groupFiles[@]}"
  25 done
  </code></pre>
  
  <p>(Note: The sequence file output should be in current directory, not in ~/Report)</p>