title: Convert several json files into a single dask dataframe and save this dataframe in a database
body: <p>I have a large number of json files in a folder. I would like to read them and save them in a single database. I had thought of pandas dataframes to do this but due to the large number of files, this operation is very slow. Someone suggested dask, which also has dataframes and seems to be very fast, so I installed it and did some tests. It seems to be confirmed. but unlike pandas, the dataframe just contains the content of each json as text. Can someone tell me how to do this so that I get a single dataframe with all the json files read and with the column names of the keys of each object in each json as well as the values?</p>
  <h2>pandas code</h2>
  <pre class="lang-py prettyprint-override"><code>import pandas as pd
  
  
  data = []
  folder = '20-05-2019'
  json_files = get_json_files(folder)
  for json_file in json_files:
      df_temp = pd.read_json(json_file, encoding='utf-8')
      data.append(df_temp)
  
  df = pd.concat(data)
  
  df.head(10)
  </code></pre>
  <h2>result</h2>
  <p><img src="https://i.imgur.com/NMLH5dj.png" alt="https://i.imgur.com/NMLH5dj.png" /></p>
  <h2>dask dataframe code</h2>
  <p><img src="https://wtf.roflcopter.fr/pics/I8wWAFah/UCtfQymu" alt="code+result" /></p>
  <h2>sample folder with 7000 json files in each folder (3000 folder in total)</h2>
  <p><img src="https://i.imgur.com/JxzYO6X.png" alt="https://i.imgur.com/JxzYO6X.png" /></p>
  <h2>sample json</h2>
  <pre class="lang-json prettyprint-override"><code>[
      {
          &quot;sector&quot;:&quot;AAA&quot;,
          &quot;code&quot;:&quot;0009&quot;,
          &quot;id&quot;:&quot;00000001&quot;,
          &quot;fname&quot;:&quot;FirstName&quot;,
          &quot;lname&quot;:&quot;LastName&quot;,
          &quot;height&quot;:&quot;158&quot;,
          &quot;dob&quot;:&quot;01/03/2006&quot;,
          &quot;din&quot;:&quot;19/05/2019 13:23&quot;,
          &quot;dout&quot;:&quot;19/05/2019 17:46&quot;,
          &quot;type&quot;:&quot;some text&quot;,
          &quot;group&quot;:&quot;2&quot;,
          &quot;dod&quot;:&quot;19/05/2019 13:48&quot;,
          &quot;desc&quot;:&quot;some text&quot;,
          &quot;details&quot;:&quot;some text&quot;,
          &quot;feval&quot;:&quot;Triage 1&quot;,
          &quot;localisation&quot;:&quot;some place&quot;,
          &quot;infop&quot;:&quot;not yet implemented&quot;,
          &quot;is_in&quot;:true,
          &quot;is_op&quot;:false,
          &quot;list_c&quot;:[],
          &quot;list_e&quot;:[],
          &quot;list_a&quot;:[],
          &quot;list_r&quot;:[]
      },
      {
          &quot;sector&quot;:&quot;AAA&quot;,
          &quot;code&quot;:&quot;0009&quot;,
          &quot;id&quot;:&quot;00000001&quot;,
          &quot;fname&quot;:&quot;FirstName&quot;,
          &quot;lname&quot;:&quot;LastName&quot;,
          &quot;height&quot;:&quot;158&quot;,
          &quot;dob&quot;:&quot;01/03/2006&quot;,
          &quot;din&quot;:&quot;19/05/2019 13:23&quot;,
          &quot;dout&quot;:&quot;19/05/2019 17:46&quot;,
          &quot;type&quot;:&quot;some text&quot;,
          &quot;group&quot;:&quot;2&quot;,
          &quot;dod&quot;:&quot;19/05/2019 13:48&quot;,
          &quot;desc&quot;:&quot;some text&quot;,
          &quot;details&quot;:&quot;some text&quot;,
          &quot;feval&quot;:&quot;Triage 1&quot;,
          &quot;localisation&quot;:&quot;some place&quot;,
          &quot;infop&quot;:&quot;not yet implemented&quot;,
          &quot;is_in&quot;:true,
          &quot;is_op&quot;:false,
          &quot;list_c&quot;:[],
          &quot;list_e&quot;:[],
          &quot;list_a&quot;:[],
          &quot;list_r&quot;:[]
      },
      {
          &quot;sector&quot;:&quot;AAA&quot;,
          &quot;code&quot;:&quot;0009&quot;,
          &quot;id&quot;:&quot;00000001&quot;,
          &quot;fname&quot;:&quot;FirstName&quot;,
          &quot;lname&quot;:&quot;LastName&quot;,
          &quot;height&quot;:&quot;158&quot;,
          &quot;dob&quot;:&quot;01/03/2006&quot;,
          &quot;din&quot;:&quot;19/05/2019 13:23&quot;,
          &quot;dout&quot;:&quot;19/05/2019 17:46&quot;,
          &quot;type&quot;:&quot;some text&quot;,
          &quot;group&quot;:&quot;2&quot;,
          &quot;dod&quot;:&quot;19/05/2019 13:48&quot;,
          &quot;desc&quot;:&quot;some text&quot;,
          &quot;details&quot;:&quot;some text&quot;,
          &quot;feval&quot;:&quot;Triage 1&quot;,
          &quot;localisation&quot;:&quot;some place&quot;,
          &quot;infop&quot;:&quot;not yet implemented&quot;,
          &quot;is_in&quot;:true,
          &quot;is_op&quot;:false,
          &quot;list_c&quot;:[],
          &quot;list_e&quot;:[],
          &quot;list_a&quot;:[],
          &quot;list_r&quot;:[]
      },
      {
          &quot;sector&quot;:&quot;AAA&quot;,
          &quot;code&quot;:&quot;0009&quot;,
          &quot;id&quot;:&quot;00000001&quot;,
          &quot;fname&quot;:&quot;FirstName&quot;,
          &quot;lname&quot;:&quot;LastName&quot;,
          &quot;height&quot;:&quot;158&quot;,
          &quot;dob&quot;:&quot;01/03/2006&quot;,
          &quot;din&quot;:&quot;19/05/2019 13:23&quot;,
          &quot;dout&quot;:&quot;19/05/2019 17:46&quot;,
          &quot;type&quot;:&quot;some text&quot;,
          &quot;group&quot;:&quot;2&quot;,
          &quot;dod&quot;:&quot;19/05/2019 13:48&quot;,
          &quot;desc&quot;:&quot;some text&quot;,
          &quot;details&quot;:&quot;some text&quot;,
          &quot;feval&quot;:&quot;Triage 1&quot;,
          &quot;localisation&quot;:&quot;some place&quot;,
          &quot;infop&quot;:&quot;not yet implemented&quot;,
          &quot;is_in&quot;:true,
          &quot;is_op&quot;:false,
          &quot;list_c&quot;:[],
          &quot;list_e&quot;:[],
          &quot;list_a&quot;:[],
          &quot;list_r&quot;:[]
      },
      {
          &quot;sector&quot;:&quot;AAA&quot;,
          &quot;code&quot;:&quot;0009&quot;,
          &quot;id&quot;:&quot;00000001&quot;,
          &quot;fname&quot;:&quot;FirstName&quot;,
          &quot;lname&quot;:&quot;LastName&quot;,
          &quot;height&quot;:&quot;158&quot;,
          &quot;dob&quot;:&quot;01/03/2006&quot;,
          &quot;din&quot;:&quot;19/05/2019 13:23&quot;,
          &quot;dout&quot;:&quot;19/05/2019 17:46&quot;,
          &quot;type&quot;:&quot;some text&quot;,
          &quot;group&quot;:&quot;2&quot;,
          &quot;dod&quot;:&quot;19/05/2019 13:48&quot;,
          &quot;desc&quot;:&quot;some text&quot;,
          &quot;details&quot;:&quot;some text&quot;,
          &quot;feval&quot;:&quot;Triage 1&quot;,
          &quot;localisation&quot;:&quot;some place&quot;,
          &quot;infop&quot;:&quot;not yet implemented&quot;,
          &quot;is_in&quot;:true,
          &quot;is_op&quot;:false,
          &quot;list_c&quot;:[],
          &quot;list_e&quot;:[],
          &quot;list_a&quot;:[],
          &quot;list_r&quot;:[]
      },
      {
          &quot;sector&quot;:&quot;AAA&quot;,
          &quot;code&quot;:&quot;0009&quot;,
          &quot;id&quot;:&quot;00000001&quot;,
          &quot;fname&quot;:&quot;FirstName&quot;,
          &quot;lname&quot;:&quot;LastName&quot;,
          &quot;height&quot;:&quot;158&quot;,
          &quot;dob&quot;:&quot;01/03/2006&quot;,
          &quot;din&quot;:&quot;19/05/2019 13:23&quot;,
          &quot;dout&quot;:&quot;19/05/2019 17:46&quot;,
          &quot;type&quot;:&quot;some text&quot;,
          &quot;group&quot;:&quot;2&quot;,
          &quot;dod&quot;:&quot;19/05/2019 13:48&quot;,
          &quot;desc&quot;:&quot;some text&quot;,
          &quot;details&quot;:&quot;some text&quot;,
          &quot;feval&quot;:&quot;Triage 1&quot;,
          &quot;localisation&quot;:&quot;some place&quot;,
          &quot;infop&quot;:&quot;not yet implemented&quot;,
          &quot;is_in&quot;:true,
          &quot;is_op&quot;:false,
          &quot;list_c&quot;:[],
          &quot;list_e&quot;:[],
          &quot;list_a&quot;:[],
          &quot;list_r&quot;:[]
      },
  ]
  </code></pre>
  <h3>EDIT for further information</h3>
  <blockquote>
  <p>Maybe I didn't explain well enough why I put everything in a dataframe first. In fact, in each json a certain amount of information is stored about people and this data is updated at regular intervals. At the end of the day, the backup operation in the database takes place. So it is sometimes and often a question of duplicate data between files that must be processed before saving everything in the database.</p>
  <p>This means that the files cannot be processed individually. You have to group them all together in a single dataframe (this is the best idea I have at the moment).</p>
  </blockquote>