title: Why reading rows is faster than reading columns?
body: <p>I am analysing a dataset having 200 rows and 1200 columns, this dataset is stored in a <code>.CSV</code> file. In order to process, I read this file using R's <code>read.csv()</code> function. </p>
  
  <p>R takes ≈ 600 seconds to read this dataset. Later I got an idea and I transposed the data inside <code>.CSV</code> file and tried to read it again using <code>read.csv()</code> function. I was amazed to see that it only took ≈ 20 seconds. As you can see, it was ≈ 30 times faster.</p>
  
  <p>I verified it for following iterations:</p>
  
  <p></p>
  
  <h3>Reading 200 rows and 1200 columns  (Not transposed)</h3>
  
  <pre><code>&gt; system.time(dat &lt;- read.csv(file = "data.csv", sep = ",", header = F))
  
     user  system elapsed 
   610.98    6.54  618.42 # 1st iteration
   568.27    5.83  574.47 # 2nd iteration
   521.13    4.73  525.97 # 3rd iteration
   618.31    3.11  621.98 # 4th iteration
   603.85    3.29  607.50 # 5th iteration
  </code></pre>
  
  <h3>Reading 1200 rows and 200 columns  (Transposed)</h3>
  
  <pre><code>&gt; system.time(dat &lt;- read.csv(file = "data_transposed.csv",
        sep = ",", header = F))
  
     user  system elapsed 
    17.23    0.73   17.97 # 1st iteration
    17.11    0.69   17.79 # 2nd iteration
    20.70    0.89   21.61 # 3rd iteration
    18.28    0.82   19.11 # 4th iteration
    18.37    1.61   20.01 # 5th iteration
  </code></pre>
  
  
  
  <p>In any data-set we take observations in rows and columns contain variables to-be observed. Transpose changes this structure of data. <strong>Is it a good practice to transpose</strong> the data for processing, even though it makes data look weird?</p>
  
  <p>I am wondering what makes R read datasets fast when I transposed the data. I am sure it is because earlier dimensions were <code>200 * 1200</code> which became <code>1200 * 200</code> after transpose operation. 
  <strong><em>Why R reads data fast when I transpose the data?</em></strong></p>
  
  <hr>
  
  <h3>Update : Research &amp; experiments</h3>
  
  <hr>
  
  <p>I initially asked this question because my RStudio was taking long time to read and compute a highly dimensional dataset (many columns as compare to rows [200 rows, 1200 columns]). I was using built-in R function <code>read.csv()</code>. I read the comments below, as per their suggestions later I experimented with <code>read.csv2()</code> and <code>fread()</code> function they all work well but they perform slowly for my original dataset [200 rows * 1200 columns] and they read transposed data-set faster.</p>
  
  <p>I observed that this is also valid for <strong>MS-Excel</strong> and <strong>Libre office Calc</strong> too. I even tried to open it into <strong>Sublime Text editor</strong> and even for this text editor it was easy(fast) to read transposed data. I am still not able to figure out the reason why all these applications behave so. All these apps get into trouble if your data has many columns as compare to rows.</p>
  
  <p>So to wrap up whole story, I have only 3 question.</p>
  
  <ol>
  <li>What kind of issue is it? Is it related to operating systems or is it application level problem?</li>
  <li>Is it a good practice to transpose the data for processing?</li>
  <li>Why R and/or other apps reads my data fast when I transpose the data?</li>
  </ol>
  
  
  
  <p>My experiments perhaps helped me to rediscover some '<em>already known</em>'
   wisdom, but I couldn't find anything relevant on internet. Kindly
   share such good programming/data analysis practices.</p>