title: Best way to combine json files into one
body: <p>I have json files with same names in different folders in folder structure mentioned below</p>
  
  <pre><code>folder1/
      file1.json
      file2.json
      file3.json
  folder2/
      file1.json
      file2.json
      file3.json
      file4.json
  folder3/
      file1.json
      file2.json
      file3.json
      file4.json
      file5.json
  ....
  </code></pre>
  
  <p>What is the best way to combine json files available in all the folders to create a single json file. The keys in <code>file1.json</code> is unique across all the folders that it exists in </p>
  
  <p>So far, I could think of the following approach but it feels slow since each json file is around 5 MB.</p>
  
  <pre class="lang-py prettyprint-override"><code>from pathlib import Path
  
  output_dir = Path(location_of_output_folder)
  output_dir.mkdir(parents=True, exist_ok=True)
  
  # find all the folders
  root_dir = Path(root_location_for_folders)
  folders = [fld for fld in root_dir.iterdir() if fld.is_dir()]
  
  # find all the unique file names
  all_filenames = []
  for fld in folders:
      for f in fld.glob('*.json'):
          all_filenames.append(f.name)
  
  
  ## Approach 1
  # Join file that possibly exists across all the folders by creating empty list
  for f in list(set(all_filenames)):
      f_data = []
  
      for fld in folders:
          if (fld / f).is_file():
             with open(fld /f, 'r') as fp:
                 f_data.append(json.load(fp))
  
      with open(output_dir / f, 'w') as fp:
          json.dump(f_data, fp, indent=4)
  
  
  ## Approach 2
  # Join file that possibly exists across all the folders by creating empty dict
  for f in list(set(all_filenames)):
      f_data = {}
  
      for fld in folders:
          if (fld / f).is_file():
             with open(fld /f, 'r') as fp:
                 f_data.update(json.load(fp))
  
      with open(output_dir / f, 'w') as fp:
          json.dump(f_data, fp, indent=4)
  </code></pre>
  
  <p>Is there a better(faster) way to this. I am only worried about the time and not interested in a pythonic solution</p>
  
  <p>Thanks</p>
  
  <p>Update #1: The files that have same filename should be merged. Sorry if I was not clear on that. Each file will have few keys <code>(l1, l2, l3, l4)</code> that are similar with all the files</p>
  
  <p>Example</p>
  
  <p>a. structure of <code>file1.json</code> in <code>folder1</code></p>
  
  <pre><code>{
      k1: {
             l1: 11,
             l2: 12,
             l3: 13,
             l4: 14,
          },
  
      k2: {
             l1: 21,
             l2: 22,
             l3: 23,
             l4: 24,
          }
      .....
  }
  
  </code></pre>
  
  <p>a. structure of <code>file2.json</code> in <code>folder2</code></p>
  
  <pre><code>{
      k8: {
             l1: 41,
             l2: 42,
             l3: 43,
             l4: 44,
          },
  
      k9: {
             l1: 51,
             l2: 52,
             l3: 53,
             l4: 54,
          }
      .....
  }
  
  </code></pre>