title: Different output between DISTINCT SQL statement and DISTINCT method from pyspark
body: <p>I'm making some cleanings in a +11M rows dataset that contains entreprises names with so much noise. I want to make some regex cleanings (deleting punctuation, whitespaces, digits, accents and some othe issues) and then apply some <em>approximate string matching</em> algorithm like TF-IDF or BM25, but that's another problem.</p>
  <p><strong>EDIT</strong></p>
  <p>These are my first five rows (including headers).</p>
  <pre><code>FILE_PATH = &quot;my_folder/dataset.csv/&quot;
  spark = SparkSession.builder.getOrCreate()
  sc = spark.sparkContext
  my_rdd = spark.sparkContext.textFile(FILE_PATH)
  my_rdd.take(5)
  
  ['cuit_impor,prove_dest,sim_sptos,porg',
   '34557619099,PUMA SPORTS LA SA                                 ,61099000111N,337',
   '34557619099,PUMA SPORTS LA SA                                 ,61102000110R,306',
   '34557619099,PUMA SPORTS LA SA                                 ,61102000110R,310',
   '34557619099,PUMA SPORTS LA SA                                 ,61103000912D,337']
  </code></pre>
  <p>It is presented as a list of strings. But with the following code I splitted each string into a list of four elements.</p>
  <pre><code>my_rdd_splitted = my_rdd.map(lambda x: x.split(',')).cache()
  my_rdd_splitted.take(5)
  
  [['cuit_impor', 'prove_dest', 'sim_sptos', 'porg'],
   ['34557619099',
    'PUMA SPORTS LA SA                                 ',
    '61099000111N',
    '337'],
   ['34557619099',
    'PUMA SPORTS LA SA                                 ',
    '61102000110R',
    '306'],
   ['34557619099',
    'PUMA SPORTS LA SA                                 ',
    '61102000110R',
    '310'],
   ['34557619099',
    'PUMA SPORTS LA SA                                 ',
    '61103000912D',
    '337']]
  </code></pre>
  <p>My columns, as you can see, are <code>cuit_impor</code>,<code>prove_dest</code>,<code>sim_sptos</code>,<code>porg</code>. The second one, <code>prove_dest</code>, is the one that has the names. To reach my goal of cleaning up the dataset, I first got all the distinct &quot;dirty&quot; names, so I read the dataset with <code>pyspark</code> and ran:</p>
  <pre><code>#in the map method I split the string and select the element in position 1, and 
  #then apply distinct. 
  unique_names = my_rdd.map(lambda x: x.split(&quot;,&quot;)[1]).disinct().cache()
  print(unique_names.count())
  </code></pre>
  <p>The output was 2,292,034.</p>
  <p>To check if everything was ok I looked for distinct values again, using the DB Browser for SQLite, Version 3.11.2, with this sentence:</p>
  <pre><code>CREATE TABLE &quot;dataset&quot; (
      &quot;cuit_impor&quot;    REAL,
      &quot;prove_dest&quot;    TEXT,
      &quot;sim_sptos&quot; TEXT,
      &quot;porg&quot;  INTEGER
  );
  
  SELECT COUNT(DISTINCT prove_dest)
  FROM dataset;
  </code></pre>
  <p>The output was 2,312,252</p>
  <p>The number of rows read in both cases is the same: 11,474,580. That means that in both cases the dataset is the same. There's no problem in data reading.</p>
  <p>It seems that SQLite identifies as distinct values ​​some strings that pyspark does not. Has anyone had this problem at any time? How can I solved this?</p>