title: Improve performance of mysql LOAD DATA / mysqlimport?
body: <p>I'm batching <code>CSV</code> 15GB (30mio rows) into a <code>mysql-8</code> database.</p>
  
  <p>Problem: the task takes about 20min, with approxy throughput of 15-20 MB/s. While the harddrive is capable of transfering files with 150 MB/s.</p>
  
  <p>I have a RAM disk of 20GB, which holds my csv. Import as follows:</p>
  
  <pre><code>mysqlimport --user="root" --password="pass" --local --use-threads=8 mytable /tmp/mydata.csv
  </code></pre>
  
  <p>This uses <code>LOAD DATA</code> under the hood.
  My target table does not have any indexes, but approx 100 columns (I cannot change this).</p>
  
  <p>What is strange: I tried tweaking several config parameters as follows in <code>/etc/mysql/my.cnf</code>, but they did not give any significant improvement:</p>
  
  <pre><code>log_bin=OFF
  skip-log-bin
  innodb_buffer_pool_size=20G
  tmp_table_size=20G
  max_heap_table_size=20G
  innodb_log_buffer_size=4M
  innodb_flush_log_at_trx_commit=2
  innodb_doublewrite=0
  innodb_autoinc_lock_mode=2
  </code></pre>
  
  <p>Question: does <code>LOAD DATA</code> / <code>mysqlimport</code> respect those config changes? Or does it bypass? Or did I use the correct configuration file at all?</p>
  
  <p>At least a select on the variables shows they are correctly loaded by the mysql server. For example <code>show variables like 'innodb_doublewrite'</code> shows <code>OFF</code></p>
  
  <p>Anyways, how could I improve import speed further? Or is my database the bottleneck and there is no way to overcome the 15-20 MB/s threshold?</p>
  
  <p>Update:
  Interestingly if I import my csv from harddrive into the ramdisk, performance is almost the same (just a little bit better, but never over 25 MB/s). I also tested the same amount of rows, but only with a few (5) columns. And there I'm getting to about 80 MB/s. So clearly the number of columns is the bottleneck? But why do more columns slow down this process?</p>