title: How to run a Particle Swarm Optimization using Python and Abaqus in Cluster
body: <p>Regards,</p>
  
  <p>I apologize before hand for the lengthy post:</p>
  
  <p><strong>My question:</strong> <em>How do you modify the loop between * * * in <code>errFunction</code>, the <code>runABQfile</code> function (<code>subprocess.call</code>), and the <code>bash script</code> below so that I can run a PSO optimization in a cluster?</em></p>
  
  <p><strong>The Background:</strong> I am calibrating a model using Particle Swarm Optimization (PSO) written in Python and ABAQUS with VUMAT (user material). A python script updates the input files <strong>N</strong> different ABAQUS models (which correspond to <strong>N</strong> different experiments) for each iteration and should run each of the <strong>N</strong> models until the global error between experiments and models is minimized. I am running this optimization in a cluster where I do not have admin privileges.</p>
  
  <p>Assume I have a working main script <strong><em>main.py</em></strong> that import necessary modules, initiates variables, read the experimental data before calling a function <strong><em>PSO.py</em></strong> using</p>
  
  <p><code>XOpt, FOpt = pso(errFunction, lb, ub, f_ieqcons=mycons, args=args)</code></p>
  
  <p>The target function <code>errFunction</code> to be minimized is to run all <strong>N</strong> models using the <code>runABQfile</code> function and return the global error each iteration to the PSO function. A brief view of the structure of my code is shown below (I left out parts that are not relevant).</p>
  
  <pre class="lang-py prettyprint-override"><code>    def errFunction(param2Calibrate,otherProps,InputFiles,experimentData,otherArgs):
          maxNpts   = otherArgs[0]
          nAnalysis = otherArgs[1]
  
          # Run Each Abaqus Simulation
          inpFile   = [0] * nAnalysis
          abqDisp   = [[0 for x in range(maxNpts)] for y in range(nAnalysis)]
          abqForce  = [[0 for x in range(maxNpts)] for y in range(nAnalysis)]
          iexpForce = [[0 for x in range(maxNpts)] for y in range(nAnalysis)]
  
          # ***********************************#
          # - Update and Run Each Input File - #
          for r in range(nParallelLoops):
              for k in range( r*nAnalysis/nParallelLoops, (r+1)*nAnalysis/nParallelLoops ):
                  # - Write and Run Abaqus INP file - #
                  inpFile[k] = writeABQfile(param2Calibrate,otherProps[k],InputFiles[k])
                  runABQfile(inpFile[k])
                  # - Extract from Abaqus ODB - #
                  abqDisp_, abqForce_ = extraction(inpFile[k])
                  abqDisp[k][0:len(abqDisp_)]   = abqDisp_
                  abqForce[k][0:len(abqForce_)] = abqForce_
  
          # ***********************************#
  
          # - Interpolate Experimental Results to Match Abaqus - #
          for k in range(0,nAnalysis):
              iexpForce_ = interpolate(experimentData[k],abqDisp[:][k])
              iexpForce[k][0:len(abqDisp_)]= iexpForce_
  
          # - Get Error - #
          for k in range(0,nAnalysis):
              Err[k] = Error(iexpForce[:][k],abqDisp[:][k],abqForce[:][k])
      return Err
  </code></pre>
  
  <p>And the <code>runABQfile</code> is setup as follow, where 2 processes are to run in serie:</p>
  
  <pre class="lang-py prettyprint-override"><code>    def runABQfile(inpFile):    
          import subprocess
          import os
  
          # - Run Abaqus - #
          ABQexe = '/opt/abaqus/6.14-1/code/bin/abq6141'
          prcStr1 = (ABQexe+' '+'job='+inpFile+' input='+inpFile+'                  \ 
                     user=$HOME/mPDFvumatNED.f scratch=/scratch/$USER/$SLURM_JOBID  \
                     cpus=12 parallel=domain domains=12 mp_mode=mpi memory=60000mb  \
                     interactive double=both')
  
          prcStr2 = (ABQexe+' '+'cae noGUI='+inpFile+'_CAE.py')
  
          process = subprocess.call(prcStr1,stdin=None,stdout=None,stderr=None,shell=True)
          process = subprocess.call(prcStr2,shell=True)
  </code></pre>
  
  <hr>
  
  <p><strong>Where the problem seem to be:</strong> I have access to maximum 2 nodes with 24 cpus per job (restricted by # of ABAQUS licenses). If I were to run a single analysis, I'd queue the job using SLURM with the following script. </p>
  
  <pre class="lang-bash prettyprint-override"><code>    #!/bin/bash
      #SBATCH --job-name="abaqus"
      #SBATCH --output="abaqus.%j.%N.out"
      #SBATCH --partition=debug
      #SBATCH --nodes=2
      #SBATCH --export=ALL
      #SBATCH --ntasks-per-node=24
      #SBATCH -L abaqus:25
      #SBATCH -t 00:30:00
  
      #Get the env file setup 
      scontrol show hostname &gt; file-list1
      scontrol show hostlist &gt; file-list2
      HOST1=`sed -n '1p' file-list1`
      HOST2=`sed -n '2p' file-list1`
      cat abq_v6.env |sed -e "s/host1/$HOST1/g" &gt; ttt1.env
      cat ttt1.env | sed -e "s/host2/$HOST2/g" &gt; abaqus_v6.env
      rm ttt*env
  
      #Run the executable remotely
      sed "s/DUMMY/$SLURM_JOBID/g" s4b.sh.orig &gt; s4b.sh
      chmod u+x s4b.sh
      export EXHOST=`/bin/hostname`
      ssh $EXHOST $SLURM_SUBMIT_DIR/s4b.sh
  </code></pre>
  
  <p>where <code>s4b.sh.orig</code> looks like this:</p>
  
  <pre class="lang-bash prettyprint-override"><code>    #!/bin/bash -l
      cd /share/apps/examples/ABAQUS/s4b_multinode
      module purge
      module load abaqus/6.14-1
      export EXE=abq6141
      $EXE job=s4b scratch=/scratch/$USER/DUMMY cpus=48 -verbose 3 \
       standard_parallel=all mp_mode=mpi memory=120000mb interactive
  </code></pre>
  
  <p>This script setup is the only way to to submit one ABAQUS job that runs on multiple nodes on that cluster because of problems with the ABAQUS environment file and SLURM (my guess the mp_host_list is not being properly assigned or it is oversubscribed, but honestly I do not understand what could be going on).   </p>
  
  <p>I modified my <code>runABQfile</code> function to use the bash construct when calling <code>subprocess.call</code> to something like this:</p>
  
  <pre class="lang-py prettyprint-override"><code>    prcStr1 = ('sed "s/DUMMY/$SLURM_JOBID/g" s4b.sh.orig &gt; s4b0.sh; \ 
                 sed "s/MODEL/inpFile/g" s4b0.sh &gt; s4b1.sh;           \ 
                 chmod u+x s4b1.sh;                                   \
                 export EXHOST=`/bin/hostname`;                       \
                 ssh $EXHOST $SLURM_SUBMIT_DIR/s4b1.sh' )   
      process = subprocess.call(prcStr1,stdin=None,stdout=None,stderr=None,shell=True)
  </code></pre>
  
  <p>But the optimization never starts and quits right after modifying the first script.</p>
  
  <hr>
  
  <p>Now the question again is <strong>How do you modify the loop between * * * in <code>errFunction</code>, the <code>runABQfile</code> function (<em>subprocess.call</em>), and the bash script so that I can run this optimization?</strong>... I would like to use at least 12 processors per ABAQUS model that is potentially running 4 jobs at the same time. Keep in mind all <strong>N</strong> models need to run and finish before moving to the next iteration.</p>
  
  <p>I will appreciate any help you guys could provide.</p>
  
  <p>Sincerely,</p>
  
  <p>D P.</p>