title: Improving time efficiency of code, working with a big Data Set - Python
body: <p>I’m trying to figure out a way to “optimize” my code and reduce (by a lot ideally) the time it takes to run through the whole data set.</p>
  <p>I’m working with a simple .csv file that has 3 columns: <strong>time_UTC</strong>, <strong>vmag2D</strong> and <strong>vdir</strong>. The data set as around 1420000 lines (one million, four hundred and 20 thousand). I wrote this simple for cycle and it took around 15/20 minutes to run. I am running this on my Mac with the M1 processor, so I’m not sure if I complicated something somewhere for it to take so much time to (meaning, I don't believe that the processor is that &quot;bad&quot; and that it is good enough to run this little piece of code faster)
  If someone has any suggestion on how I can improve it, please let me know!</p>
  <pre><code>import pandas as pd
  
  path_data = '&quot; *insert a path here* &quot;'
  file = path_data + ' *name of the .csv file* '
  
  data = pd.read_csv(file)
  
  time_UTC = []
  vmag2D = []
  vdir = []
  
  for i in range(len(data)):
      x = data.iloc[i][0]
      x1 = x.split(' ')
      x2 = x1[1].split(';')
      date = x.split(' ')[0]
      time_UTC.append(x2[0])
      vmag2D.append(x2[1])
      vdir.append(x2[2])
  </code></pre>
  <p>The code is parsing each of the lines in the .csv file, and each of them has the same “template”: <strong>'1994-01-01 00:05:00;0.52;193'</strong></p>
  <p>Thank you for any help!</p>
  <p>Cheers!</p>