title: Merge objects based on a list of dicts which is stored in a nested field
body: <p>I'm using Python 3.9.x - I have a problem where I'd like to merge a list of dicts in an optimal way. However, these aren't simple dicts - there is an ID which is a simple numeric ID; and a list of dicts called &quot;codes&quot; - which is where the list of dicts to merge from are stored.</p>
  <p>An example of raw data is below:</p>
  <pre><code>[
    {
      &quot;id&quot; : &quot;1234&quot;,
      &quot;codes&quot; : [
        {
          &quot;provider&quot; : &quot;provider1&quot;,
          &quot;id&quot; : &quot;1234&quot;,
        },
        {
          &quot;provider&quot; : &quot;provider2&quot;,
          &quot;id&quot; : &quot;AA0001&quot;,
        },
        {
          &quot;provider&quot; : &quot;provider3&quot;,
          &quot;id&quot; : &quot;tt00001&quot;,
        },
        {
          &quot;provider&quot; : &quot;provider4&quot;,
          &quot;id&quot; : &quot;0000-0000-27E0-0000-9-0000-0000-A&quot;,
        }
      ]
    },
    {
      &quot;id&quot; : &quot;12345&quot;,
      &quot;codes&quot; : [
        {
          &quot;provider&quot; : &quot;provider1&quot;,
          &quot;id&quot; : &quot;2345&quot;,
        },
        {
          &quot;provider&quot; : &quot;provider3&quot;,
          &quot;id&quot; : &quot;tt00001&quot;,
        },
        {
          &quot;provider&quot; : &quot;provider4&quot;,
          &quot;id&quot; : &quot;0000-0000-27E0-0000-9-0000-0000-A&quot;,
        }
        {
          &quot;provider&quot; : &quot;provider5&quot;,
          &quot;id&quot; : &quot;F0046872&quot;,
        },
      ]
    },
    {
      &quot;id&quot;: &quot;123456&quot;,
      &quot;codes&quot;: [
          {
              &quot;id&quot;: &quot;0000&quot;, 
              &quot;provider&quot;: &quot;provider6&quot;
          }
      ]
    }
  ]
  </code></pre>
  <p>In the above example, we can see there are two common dicts in the &quot;codes&quot; list for two objects (with IDs: &quot;1234&quot; and &quot;12345&quot;):</p>
  <pre><code>      {
          &quot;provider&quot; : &quot;provider3&quot;,
          &quot;id&quot; : &quot;tt00001&quot;,
        },
        {
          &quot;provider&quot; : &quot;provider4&quot;,
          &quot;id&quot; : &quot;0000-0000-27E0-0000-9-0000-0000-A&quot;,
        }
  </code></pre>
  <p>As long as there is one common provider and id combination - we should trigger a merge. Hence objects with IDs 1234 and 12345 should be merged.</p>
  <p>However, the desired output would preserve all the objects with all IDs, i.e.,:</p>
  <pre><code>[
    {
      &quot;id&quot; : &quot;1234&quot;,
      &quot;codes&quot; : [
        {
          &quot;provider&quot; : &quot;provider1&quot;,
          &quot;id&quot; : &quot;1234&quot;,
        },
        {
          &quot;provider&quot; : &quot;provider2&quot;,
          &quot;id&quot; : &quot;AA0001&quot;,
        },
        {
          &quot;provider&quot; : &quot;provider3&quot;,
          &quot;id&quot; : &quot;tt00001&quot;,
        },
        {
          &quot;provider&quot; : &quot;provider4&quot;,
          &quot;id&quot; : &quot;0000-0000-27E0-0000-9-0000-0000-A&quot;,
        },
        {
          &quot;provider&quot; : &quot;provider5&quot;,
          &quot;id&quot; : &quot;F0046872&quot;,
        }
      ]
    },
    {
      &quot;id&quot; : &quot;12345&quot;,
      &quot;codes&quot; : [
        {
          &quot;provider&quot; : &quot;provider1&quot;,
          &quot;id&quot; : &quot;2345&quot;,
        },
        {
          &quot;provider&quot; : &quot;provider3&quot;,
          &quot;id&quot; : &quot;tt00001&quot;,
        },
        {
          &quot;provider&quot; : &quot;provider5&quot;,
          &quot;id&quot; : &quot;F0046872&quot;,
        },
        {
          &quot;provider&quot; : &quot;provider2&quot;,
          &quot;id&quot; : &quot;AA0001&quot;,
        },
        {
          &quot;provider&quot; : &quot;provider4&quot;,
          &quot;id&quot; : &quot;0000-0000-27E0-0000-9-0000-0000-A&quot;,
        }
      ]
    },
    {
      &quot;id&quot;: &quot;123456&quot;,
      &quot;codes&quot;: [
          {
              &quot;id&quot;: &quot;0000&quot;, 
              &quot;provider&quot;: &quot;provider6&quot;
          }
      ]
    }
  ]
  </code></pre>
  <p>Some code so far where:</p>
  <ul>
  <li>The codes_file has the same data as shown in the raw data above</li>
  <li>The Codes class the raw_data is read into is simply a python dataclass</li>
  </ul>
  <pre><code>    def test_codes_extractor_flattener(self, codes_extractor: CodesExtractor):
          codes_file = os.path.join(
              os.path.dirname(os.path.abspath(__file__)),
              &quot;data/codes_cache_flatten.txt&quot;,
          )
          codes_cache = []
          with open(codes_file, &quot;r&quot;) as in_file:
              data = ast.literal_eval(in_file.read())
              for raw_data in data:
                  for code in raw_data[&quot;codes&quot;]:
                      for raw_data_ in data:
                          for code_ in raw_data_[&quot;codes&quot;]:
                              if code_[&quot;provider&quot;] == code[&quot;provider&quot;] and \
                                      code_[&quot;id&quot;] == code[&quot;id&quot;]:
                                  self.merge(raw_data, raw_data_)
                                  continue
                          continue
                  codes_cache.append(Codes(**raw_data))
          assert len(codes_cache) == 2
          for codes in codes_cache:
              assert len(codes.codes) == 6
  
      def merge(self, code_1, code_2):
          code_2_codes = code_2[&quot;codes&quot;]
          for code_ in code_1[&quot;codes&quot;]:
              if code_ not in code_2_codes:
                  code_2_codes.append(code_)
  </code></pre>
  <p>Note:</p>
  <ul>
  <li>The order of the provider fields does not matter (as shown by the second entry in the desired output</li>
  <li>The &quot;id&quot; field outside the &quot;codes&quot; list must be preserved.</li>
  </ul>