parallelize a large loop <p>I have a large loop where I am trying to calculate certain attributes for each pixel in a DEM (4800x6000). I am calling a function demPHV in which I've vectorized all calculations that outputs a structure with 26 fields . I have 4 cores but also have access to a multi-core cluster. I would like to speed up the weeks it would take to run this.</p> <p>Z is the dem for this example. R is the spatialref object (a vector for example's sake). latlim and lonlim are vectors of lat and long of the western US coastline (made up pairs in the example). for example:</p> <pre><code>Z=rand(48,60); R=makerefmat(120,40,.5,.5) latlim=[40:60]'; lonlim=[136:(143-136)/(length(latlim)-1):143]'; </code></pre> <p>Then my original loop:</p> <pre><code>for col=11:size(Z,2)-11 for row=11:size(Z,1)-11 dpv=demPHV(Z,R,row,col,latlim,lonlim) fn=fieldnames(dpv); for k=1:length(fieldnames(dpv)) DEM_PHV.(fn{k}).{row,col}=dpv.(fn{k}); end end </code></pre> <p>Loops for parallelizing:</p> <p>option 1:</p> <pre><code>[rows, cols] = meshgrid(12:(size(Z,1)-12), 12:(size(Z,2)-12)); inds = sub2ind(size(Z), rows, cols); inds = inds(:)'; parfor i=inds(1):inds(end) dpv=demPHV(Z,R,i,latlim,lonlim) end </code></pre> <p>This includes <code>[r,c]=ind2sub(size(Z),i)</code> in the function to use in the function demPHV.</p> <p>option 2:</p> <pre><code>parfor col=11:size(Z,2)-11 for row=11:size(Z,1)-11 dpv=demPHV(Z,R,row,col,latlim,lonlim) end end </code></pre> <p>parfor requires consecutive integers hence some of these changes. I have to exclude the bordering 11 rows and columns because my function uses surrounding pixels to calculate some of the attributes.</p> <p>So, my questions:</p> <ol> <li>Would you expect either of these two options to be faster than the other?</li> <li><p>parfor does not allow me to include the second part of my original loop:</p> <p><code>fn=fieldnames(dpv);</code></p> <p><code>for k=1:length(fieldnames(dpv))</code></p> <p><code>DEM_PHV.(fn{k}).{row,col}=dpv.(fn{k});</code></p> <p><code>end</code></p></li> </ol> <p>during which I assign the output structure to another variable. The ultimate goal is to have the variable DEM_PHV have fields for every attribute I need, and every field to be a matrix size(Z) where every cell is the corresponding value for that attribute. I've tried to have my function output the values in the correct cell of the matrix, but then I get a matrix size(Z) with <code>[]</code> everywhere except for the value at location <code>row,col</code>. This seems like a horribly inefficient use of memory... any better suggestions? I hope I covered everything. Thanks for looking!</p>