title: difference union vs or-clause
body: <p>in Spark, what is the difference between an <code>union</code> and an <code>or-clause</code> ? </p>
  
  <p>Let's take an example : </p>
  
  <p>Here is my dataframe: </p>
  
  <pre><code>df = spark.createDataFrame(
    [
      ('96','2e63e9f4-27ba-4f50-bc65-a97032a22096' ),
      ('55','4bced1f9-63ad-4ebb-bf34-5fd7ff52d8e2' ),
      ('47','6c5c8151-7891-4567-9d6a-8dace74904bd' ),
      ('90','781eb57d-0774-46c0-9366-13cbab6322c6' ),
      ('27','7eb27670-1e4d-422f-b4f6-f65461bbeda5' ),
      ('259','91646385-3446-42af-a823-33112645024b'),
      ('33','92c77bd9-373d-4d32-9f36-5fa3fc093cd6' ),
      ('96','c6bcc234-7cd7-4134-8f89-b8bb50ae5e0f' ),
      ('55','4ade739d-5115-439c-900e-09fc4cb25293' ),
      ('47','73a2e429-cadc-4afa-ade2-4251e3745a0c' ),
      ('90','c0246074-a899-4437-a461-26c9445822ef' ),
      ('27','a7f6bbfb-fc03-4d04-ab4a-8f58eaf55dd0' ),
      ('259','13bc9ef0-35a0-4f85-8017-55bb8dae6628'),
      ('33','c77c5580-494f-45bf-bb04-6683a9dcc425' ),
    ],
    ["ClientId", "PublicId"]
  )
  </code></pre>
  
  <p>and my filter information : </p>
  
  <pre><code>my_filter = [
    ('33','92c77bd9-373d-4d32-9f36-5fa3fc093cd6' ),
    ('96','c6bcc234-7cd7-4134-8f89-b8bb50ae5e0f' ),
    ('55','4ade739d-5115-439c-900e-09fc4cb25293' ),
  ]
  </code></pre>
  
  <p>If I filter using <code>union</code>, I will do : </p>
  
  <pre><code>from functools import reduce
  
  out_dataframe_1 = reduce(
              lambda a, b: a.union(b),
              (
                  df.where(
                      "ClientId = '{ClientId}' and "
                      "PublicId = '{PublicId}'".format(
                          ClientId=ClientId,
                          PublicId=PublicId,
                      )
                  )
                  for ClientId, PublicId
                  in my_filter
              )
          )
  
  out_dataframe_1.collect()
  </code></pre>
  
  <p>If I do it with <code>or-clause</code>, i will do : </p>
  
  <pre><code>where_clause = ' or '.join(
    "(ClientId = '{ClientId}' and "
    "PublicId = '{PublicId}')".format(
      ClientId=ClientId,
      PublicId=PublicId,
    )
    for ClientId, PublicId
    in my_filter
  )
  
  out_dataframe_2 = df.where(where_clause)
  
  out_dataframe_2.collect()
  </code></pre>
  
  <p>Which one is the best to use ? 
  Is there any other way to perform a succession of filters ? Maybe a join would be the best ?</p>