title: How to read and push to an SQL database using pandas a file that is constantly getting data
body: <p>I have a CSV file that is constantly getting new data chugged in (new rows appended at the end) by another program.</p>
  <p>My goal is to read this file in python and push the incoming data on-line in a SQL database. I do not want to wait (or rather I cannot afford) that the file is finished before pushing the data. Once the file is open, the program should constantly wait for new lines to be pushed on the DB.</p>
  <p>Pandas embeds a very useful method <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_sql.html" rel="nofollow noreferrer">to_sql</a> that I am currently using to push the data on the database, which is why I'd love to keep using pandas.</p>
  <p>Pandas' <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html" rel="nofollow noreferrer">read_csv</a> method accepts a <em>chunk_size</em> argument that I tried to use. The behavior is interesting, however the file is <strong>not</strong> reloaded each time a chunk is queried, hence not solving my problem. In this example</p>
  <pre><code>df = pd.read_csv('filename.csv', chunksize=1)
  time.sleep(10)
  df.get_chunk(5)
  </code></pre>
  <p>if the data in the file is modified during the sleep, it will not be catched when the chunks are queried.</p>
  <p>Does anyone has an idea how could I do ?</p>
  <p>Thanks in advance</p>