title: Find and execute line by line before final result in bash
body: <p>I'm trying to get rid of old asset versions. The file naming is <em>strictly</em> as follows:<br />
  <code>&lt;timestamp&gt;_&lt;constant&gt;&lt;version&gt;&lt;assetID&gt;.zip&lt;.extra&gt;</code><br />
  for example, 202201012359_FOOBAR0101234567.zip.done.<br />
  <code>&lt;timestamp&gt;</code> is the datetime when the file has been added to the folder.<br />
  <code>&lt;constant&gt;</code> does not change inside the folder being handled.<br />
  <code>&lt;version&gt;</code> is a two digit number starting from 00, and describes the version of the asset with the <code>&lt;assetID&gt;</code>.<br />
  <code>&lt;extra&gt;</code> is optional, so extension can be .zip, .zip.done, or .zip.somethingelse.<br />
  However, assets may have all three different extensions, and they can exist multiple times with different timestamps. This means that asset may have multiple <em>additional</em> files with the same ID and version number, but timestamp is different.<br />
  The goal would be finding the latest version of each asset with the same ID and removing older versions. The version number is what matters, not the timestamp.<br />
  Assets are all located in one folder without subfoldering.</p>
  <h2>Current solution</h2>
  <p>So far, this is how the goal can be reached:</p>
  <pre><code>#!/bin/bash
  location=&quot;/home/user/FOOBAR&quot;
  echo &quot;Deleting older files...&quot;
  
  # Declare variable to print the outcome of removed asset ID's
  declare -A assetsRemoved
  
  # The main loop which finds all the files in the folder
  find $location -maxdepth 1 -type f -name &quot;*.zip*&quot; -a -name &quot;*FOOBAR*&quot; | while read line; do
      # &lt;timestamp&gt;_FOOBAR&lt;iterator&gt;&lt;assetId&gt;&lt;file-extensions&gt;
      # 20201229104919_FOOBAR0300040682.zip.done
  
      # Separate assetId
      rest=${line#*'.zip'}
      # .done
      pos=$(( ${#line} - ${#rest} - 4 ))
      # 20201229104919_FOOBAR0300040682&lt;^&gt;.zip.done
      assetId=${line:pos-8:8}
      # 20201229104919_FOOBAR03&lt;00040682&gt;.zip.done
  
      # Find all files with same assetId
      assets=&quot;$(find ~+ $location -maxdepth 1 -type f -name &quot;*$assetId.zip*&quot; -a -name &quot;*FOOBAR*&quot;)&quot;
      # Init loop variables
      max=-1
      mostRecent=&quot;&quot;
      cleanedOld=0
  
      # Loop all files with same assetId
      for file in $assets
      do
          # Separate basename without extension
          basenameNoExt=&quot;${file%%.*}&quot;
          # &lt;20201229104919_FOOBAR00300040682&gt;.zip.done
          # Separate iterator, 2 numbers
          iter=${basenameNoExt:${#basenameNoExt}-10:2}
          # 20201229104919_FOOBAR0&lt;03&gt;00040682.zip.done
          if [[ $iter -gt $max ]]
          then
              max=$iter
              if [[ -n $mostRecent ]]
              then
                  rm $mostRecent*
                  cleanedOld=1
              fi
              mostRecent=$basenameNoExt
          elif [[ $iter -lt $max ]]
          then
              [ -f $file ] &amp;&amp; rm $basenameNoExt* 
              cleanedOld=1
          fi
          # $iter == $max -&gt; same asset with different file extension, leave to be
      done
  
      if [[ $max -gt 0  &amp;&amp; cleanedOld -gt 0 ]] 
      then
          assetsRemoved[$assetId]=$max
      fi
  
  done
  
  for a in &quot;${!assetsRemoved[@]}&quot;; do
      echo &quot;Cleaned asset $a from versions lower than ${assetsRemoved[$a]}&quot;
  done
  </code></pre>
  <h2>The problem</h2>
  <p>This solution has serious problem: it is <em>slow</em>.
  As it first finds all the files, takes one and figures out the max version while deleting older versions, the next iteration in the outermost find-loop tries to execute the find-remove-command to assets that may have been already handled or deleted.</p>
  <h2>The question</h2>
  <p>Is there a way to execute commands for each result of <code>find</code> <strong>before</strong> all the results have been gathered? Or is there some other, more efficient way to loop the results? There are over 100k files to handle, and I assume that wildcard <code>rm</code> loops them through when searching relevant files to delete. This would require over 100.000^2 iterations through the files. Is there any way to prevent this?</p>
  <h2>Example</h2>
  <p>Consider a folder with the following files:</p>
  <pre><code>20191229104919_FOOBAR0001234567.zip
  20191229104919_FOOBAR0001234567.zip.done
  20191229104919_FOOBAR0001234567.zip.somethingelse
  20191229104919_FOOBAR0087654321.zip
  20191129104919_FOOBAR0087654321.zip.done
  20191129104919_FOOBAR0087654321.zip.somethingelse
  20191129100000_FOOBAR0187654321.zip
  20191229100000_FOOBAR0187654321.zip.done
  20191229100000_FOOBAR0187654321.zip.somethingelse
  20201229104919_FOOBAR0101234567.zip
  20201229104919_FOOBAR0101234567.zip.done
  20201229104919_FOOBAR0101234567.zip.somethingelse
  20211229104919_FOOBAR0201234567.zip
  20211229104919_FOOBAR0201234567.zip.done
  20211229104919_FOOBAR0201234567.zip.somethingelse
  20221229104919_FOOBAR0201234567.zip
  20221229104919_FOOBAR0201234567.zip.done
  20221229104919_FOOBAR0201234567.zip.somethingelse
  </code></pre>
  <p>The remaining files after cleaning would be:</p>
  <pre><code>20191129100000_FOOBAR0187654321.zip
  20191229100000_FOOBAR0187654321.zip.done
  20191229100000_FOOBAR0187654321.zip.somethingelse
  20211229104919_FOOBAR0201234567.zip
  20211229104919_FOOBAR0201234567.zip.done
  20211229104919_FOOBAR0201234567.zip.somethingelse
  20221229104919_FOOBAR0201234567.zip
  20221229104919_FOOBAR0201234567.zip.done
  20221229104919_FOOBAR0201234567.zip.somethingelse
  </code></pre>
  <p>Notice:<br />
  Newest <strong>version</strong> is what matters. Same asset version&amp;ID with different timestamps and extensions must be preserved</p>