 <p>So after a long profiling session, with XDEBUG, phpmyadmin and friends, I finally narrowed down what the bottleneck was: the mysql queries</p> <p>I enabled </p> <p><code>define('SAVEQUERIES', true);</code></p> <p>to examine the queries in more detail and I output the queries array to my debug.log file</p> <pre><code>global $wpdb; error_log( print_r( $wpdb-&gt;queries, true ) ); </code></pre> <p>upon examining each INSERT call to the database</p> <ol> <li><p>wp_insert_post()</p> <p>[0] => INSERT INTO <code>wp_posts</code> (<code>post_author</code>,<code>post_date</code>,<code>post_date_gmt</code>,<code>post_content</code>,<code>post_content_filtered</code>,<code>post_title</code>,<code>post_excerpt</code>,<code>post_status</code>,<code>post_type</code>,<code>comment_status</code>,<code>ping_status</code>,<code>post_password</code>,<code>post_name</code>,<code>to_ping</code>,<code>pinged</code>,<code>post_modified</code>,<code>post_modified_gmt</code>,<code>post_parent</code>,<code>menu_order</code>,<code>guid</code>) VALUES (...)</p> <p>[1] => 0.10682702064514</p></li> <li><p>update_post_meta()</p> <p>[0] => INSERT INTO <code>wp_postmeta</code> (<code>post_id</code>,<code>meta_key</code>,<code>meta_value</code>) VALUES (...)</p> <p>[1] => 0.10227680206299</p></li> </ol> <p>I found that each one of those calls to the database costs, on average, ~0.1 s on my localhost server</p> <p>But, since I am updating 6 custom fields per entry, it works out to </p> <p>6 cf insert calls/entry * ~0.1s/cf insert call *20 000 total entries * 1 min/60s = 200 min </p> <p>~2.5 hrs to process the custom fields alone for 20k posts</p> <p>Now, since all the custom fields are inserted into the same table, wp_post_meta, I looked to combine all the update_post_meta calls into one call, and hence, drastically improve the performance in terms of the execution time of this import script.</p> <p>I looked through the update_post_meta function in the wp core and saw there was no native option to pass an array instead of a single cf key and value, so I went through the relevant code to pinpoint the SQL INSERT line and looked how to do something along the lines of:</p> <pre><code>`INSERT INTO `wp_postmeta` (`post_id`,`meta_key`,`meta_value`) VALUES ( $post_id, 'artist_name' , $artist) ( $post_id, 'song_length' , $length ) ( $post_id, 'song_genre' , $genre ) ...` </code></pre> <p>and so forth for all the 6 custom fields in my case, all rolled in inside a single <code>$wpdb-&gt;query();</code> .</p> <p>Tweaking my process_custom_post() function to reflect this, I ended up with:</p> <pre><code>function process_custom_post($song) { global $wpdb; // Prepare and insert the custom post $track = (array_key_exists(0, $song) &amp;&amp; $song[0] != "" ? $song[0] : 'N/A'); $custom_post = array(); $custom_post['post_type'] = 'songs'; $custom_post['post_status'] = 'publish'; $custom_post['post_title'] = $track; $post_id = wp_insert_post( $custom_post ); // Prepare and insert the custom post meta $meta_keys = array(); $meta_keys['artist_name'] = (array_key_exists(1, $song) &amp;&amp; $song[1] != "" ? $song[1] : 'N/A'); $meta_keys['song_length'] = (array_key_exists(2, $song) &amp;&amp; $song[2] != "" ? $song[2] : 'N/A'); $meta_keys['song_genre'] = (array_key_exists(3, $song) &amp;&amp; $song[3] != "" ? $song[3] : 'N/A'); $meta_keys['song_year'] = (array_key_exists(4, $song) &amp;&amp; $song[4] != "" ? $song[4] : 'N/A'); $meta_keys['song_month'] = (array_key_exists(5, $song) &amp;&amp; $song[5] != "" ? $song[5] : 'N/A'); $meta_keys['sample_playlist'] = (array_key_exists(6, $song) &amp;&amp; $song[6] != "" ? $song[6] : ''); $custom_fields = array(); $place_holders = array(); $query_string = "INSERT INTO $wpdb-&gt;postmeta ( post_id, meta_key, meta_value) VALUES "; foreach($meta_keys as $key =&gt; $value) { array_push($custom_fields, $post_id, $key, $value); $place_holders[] = "('%d', '%s', '%s')"; } $query_string .= implode(', ', $place_holders); $wpdb-&gt;query( $wpdb-&gt;prepare("$query_string ", $custom_fields)); return true; } </code></pre> <p>and viola! Instead of 7 INSERT calls to the database per custom post, I end up with 2 making a gynormous difference when attempting to process multiple custom fields while iterating over a large number of posts - in my case now, 20k entries are processed to completion within 15-20min (vs a situation where I would experience a dropout after ~17k posts and a few hrs of processing time)...</p> <p>So the key thing I learned from this whole experience, </p> <p>mind your database calls - they can quickly add up!</p>