title: s3distcp copy files and directory from HDFS to S3 in a single command
body: <p>I have below 2 files and 1 directory in HDFS.</p>
  
  <p>-rw-r--r--   1 hadoop hadoop   11194859 2017-05-05 19:53 hdfs:///outputfiles/abc_output.txt<br>
  drwxr-xr-x   - hadoop hadoop          0 2017-05-05 19:28 hdfs:///outputfiles/sample_directory<br>
  -rw-r--r--   1 hadoop hadoop   68507436 2017-05-05 19:55 hdfs:///outputfiles/sample_output.txt  </p>
  
  <p>I want to copy abc_output.txt and sample_directory in gzip format onto S3 from HDFS in a single command. I don't want the files to be combined on S3. </p>
  
  <p>My S3 bucket should contain the following:
  abc_output.txt.gzip
  sample_directory.gzip</p>
  
  <p>I tried the following:</p>
  
  <p>s3-dist-cp --s3Endpoint=s3.amazonaws.com --src=hdfs:///outputfiles/ --dest=s3://bucket-name/outputfiles/ --outputCodec=gzip</p>
  
  <p>But this copies all files and folders from source to destination.</p>
  
  <p>By referring 
  <a href="https://stackoverflow.com/questions/43548832/deduce-the-hdfs-path-at-runtime-on-emr/43641654#43641654">Deduce the HDFS path at runtime on EMR</a> , I also tried the below command:</p>
  
  <p>s3-dist-cp --s3Endpoint=s3.amazonaws.com --src=hdfs:///outputfiles/ --dest=s3://bucket-name/outputfiles/ --srcPattern=.*abc_output.txt.<em>sample_directory.</em> --outputCodec=gzip but this failed.</p>