title: Search for a String in 1000 files and each file size is 1GB
body: <p>I am working on SunOS (which is slightly brain-dead). And below is the Disk Throughput for the above Solaris Machine-</p>
  
  <pre><code>bash-3.00$ iostat -d 1 10
      sd0           sd1           sd2           sd3
  kps tps serv  kps tps serv  kps tps serv  kps tps serv
    0   0    0  551  16    8  553  16    8  554  16    8
    0   0    0  701  11   25    0   0    0  1148  17   33
    0   0    0    0   0    0    0   0    0    0   0    0
    0   0    0    0   0    0    0   0    0    0   0    0
    0   0    0    0   0    0    0   0    0    0   0    0
    0   0    0    0   0    0    0   0    0    0   0    0
    0   0    0    0   0    0    0   0    0    0   0    0
    0   0    0    0   0    0    0   0    0    0   0    0
    0   0    0    0   0    0    0   0    0    0   0    0
    0   0    0    0   0    0    0   0    0    0   0    0
  </code></pre>
  
  <p><strong>Problem Statement</strong></p>
  
  <p>I have around <code>1000 files</code> and each file is of the size of <code>1GB</code>. And I need to find a <code>String</code> in all these <code>1000 files</code> and also which files contains that particular String. I am working with <code>Hadoop File System</code> and all those <code>1000 files</code> are in Hadoop File System. </p>
  
  <p>All the <code>1000 files</code> are under <code>real-time</code> folder, so If I do like this below, I will be getting all the <code>1000 files</code>. And I need to find which files contains a particular String.</p>
  
  <pre><code>bash-3.00$ hadoop fs -ls /apps/technology/b_dps/real-time
  </code></pre>
  
  <p>So for the above problem statement, I am using the below command that will find all the files which contains the particular string-</p>
  
  <pre><code>hadoop fs -ls /apps/technology/b_dps/real-time | awk '{print $8}' | while read f; do hadoop fs -cat $f | grep cec7051a1380a47a4497a107fecb84c1 &gt;/dev/null &amp;&amp; echo $f; done
  </code></pre>
  
  <p>So in the above case it will find all the files which contains this string <strong>cec7051a1380a47a4497a107fecb84c1</strong>. And it is working fine for me and I am able to get the file names which contains the particular string. </p>
  
  <p><strong>My Question is-</strong></p>
  
  <p>But the problem with above command is, it is <strong>very very slow</strong>. So is there any way we can <code>parallelize</code> the above command or make the above command to search the files a lot faster?</p>
  
  <p>Any suggestions will be appreciated.</p>