title: Python. Filling DataFrame: code optimisation
body: <p>I have big dataset with around 4M rows. I need to clean it by regex and put into Pandas' DataFrame. Here is my code for that:</p>
  
  <pre><code># 1) reading a text file with a dataset, where 4M rows
  orgfile = open("good_dmoz.txt", "r")
  
  # 2) create an empty dataframe
  df0=pd.DataFrame(columns=['url'])
  
  # 3) creating mask for cleaning data
  regex = re.compile(r"(?&lt;=\')(.*?)(?=\')")
  
  # 4) clearing data and put into the dataframe
  for line in orgfile:
      urls = regex.findall(line)
      df0.url = df0.append({"url": urls[0]}, ignore_index=True)
  </code></pre>
  
  <p>The code handles the task in a small fragment, but it takes <strong>too long time</strong> to process full data (4M rows). My question is: is it possible to optimize the code? By optimization, I mean to reduce the speed of code execution.</p>
  
  <p>Thank you!</p>