title: Doing too many operation when creating csv files - Looking for help on optimization
body: <p>I coded a small programm to download financial data, and divide it into csv with the second data, minute data, etc..., the programm is running fine, but I'm creating csv files that I delete later, and it seems like a waste of computing power, I was wondering if there were any ways to make it run quicker. </p>
  
  <p>The programm looks like this: </p>
  
  <pre><code>url = 'https://tickdata.fxcorporate.com/EURUSD/2015/1.csv.gz'
  urllib.request.urlretrieve(url, folder)
  </code></pre>
  
  <p>First I download a csv file from an url, this is the tick data that is necessary, however, this tick data has invalid symbols in it, so I need to correct it: </p>
  
  <pre><code>with gzip.open(folder, 'rb') as f:
      data = f.read()
  
  with gzip.open(file + '.csv.gz', 'wb') as f:
      f.write(data.decode('utf-8').replace('\x00', '').encode('utf-8'))
  </code></pre>
  
  <p>Here, I remove all the invalids <code>/x00</code> characters, so that python can read it fine (if I don't do that, I cannot handle the csv with pandas), thus creating another csv file.</p>
  
  <p>Then, I do all the operation I need to create the several csv file with the data I need using pandas:</p>
  
  <pre><code>tick = pd.read_csv(file + '.csv.gz',compression='gzip')
  tick.index = pd.to_datetime(tick['DateTime'],format="%m/%d/%Y %H:%M:%S.%f")
  sec =  tick['Bid'].resample('1s').ohlc()
  sec = sec.fillna(method="ffill")
  sec.to_csv(file + '_S1.csv',float_format='%.5f')
  ...
  </code></pre>
  
  <p>And at the end, I remove the file I first created downloading the data, and the second one with the invalid characters removed:</p>
  
  <pre><code>os.remove(folder)
  os.remove(file + '.csv.gz')
  </code></pre>
  
  <p>I'm not an expert, but it seems that creating two csv files and then deleting them is a waste of time, I didn't find any solutions to do less steps than I already do.</p>
  
  <p>Would anyone have an idea of how to speed up the process by not writing the extra csv files? </p>
  
  <p>Thanks!</p>