title: Getting duplicates on sort with uniq at bash
body: <p>I tried to combine 2 files and do the sort + uniq in it, but I am still getting duplicate values in the output.</p>
  <pre><code>❯ cat new_ids.txt existing_ids.txt|  sort | uniq | head
  009fff77-e6f2-46b1-8353-a271e3e1bdea
  00AB4E27-4D4F-4E4F-B4EC-2BE83B85CEF8
  00a6ceff-cac8-42e3-bdc9-cbf2538e31ea
  00a6ceff-cac8-42e3-bdc9-cbf2538e31ea
  00ab4e27-4d4f-4e4f-b4ec-2be83b85cef8
  00ab4e27-4d4f-4e4f-b4ec-2be83b85cef8
  00c650d7-902a-4223-b11b-c5b41ab8f1f5
  00e55256-981e-4a4a-ad8b-a0a6cb75f47c
  00eead4b-1d8f-418b-80c1-976c55abb629
  00eead4b-1d8f-418b-80c1-976c55abb629
  </code></pre>
  <p>I tried to normalize line delimiters, by</p>
  <pre><code>❯ cat new_ids.txt existing_ids.txt| sed 's/\r$//' | tr -d '\t' | tr -d ' '| sort | uniq | head
  009fff77-e6f2-46b1-8353-a271e3e1bdea
  00AB4E27-4D4F-4E4F-B4EC-2BE83B85CEF8
  00a6ceff-cac8-42e3-bdc9-cbf2538e31ea
  00a6ceff-cac8-42e3-bdc9-cbf2538e31ea
  00ab4e27-4d4f-4e4f-b4ec-2be83b85cef8
  00ab4e27-4d4f-4e4f-b4ec-2be83b85cef8
  00c650d7-902a-4223-b11b-c5b41ab8f1f5
  00e55256-981e-4a4a-ad8b-a0a6cb75f47c
  00eead4b-1d8f-418b-80c1-976c55abb629
  00eead4b-1d8f-418b-80c1-976c55abb629
  </code></pre>
  <p>But I am still getting duplicates. Any help on identifying what is wrong with this is highly appreciated.</p>
  <pre><code>❯ cat new_ids.txt existing_ids.txt| sed 's/\r$//' | tr -d '\t' | tr -d ' '| sort | uniq | head | hexdump -C
  00000000  30 30 39 66 66 66 37 37  2d 65 36 66 32 2d 34 36  |009fff77-e6f2-46|
  00000010  62 31 2d 38 33 35 33 2d  61 32 37 31 65 33 65 31  |b1-8353-a271e3e1|
  00000020  62 64 65 61 0a 30 30 41  42 34 45 32 37 2d 34 44  |bdea.00AB4E27-4D|
  00000030  34 46 2d 34 45 34 46 2d  42 34 45 43 2d 32 42 45  |4F-4E4F-B4EC-2BE|
  00000040  38 33 42 38 35 43 45 46  38 0a 30 30 61 36 63 65  |83B85CEF8.00a6ce|
  00000050  66 66 2d 63 61 63 38 2d  34 32 65 33 2d 62 64 63  |ff-cac8-42e3-bdc|
  00000060  39 2d 63 62 66 32 35 33  38 65 33 31 65 61 0a 30  |9-cbf2538e31ea.0|
  00000070  30 61 36 63 65 66 66 2d  63 61 63 38 2d 34 32 65  |0a6ceff-cac8-42e|
  00000080  33 2d 62 64 63 39 2d 63  62 66 32 35 33 38 65 33  |3-bdc9-cbf2538e3|
  00000090  31 65 61 0d 0a 30 30 61  62 34 65 32 37 2d 34 64  |1ea..00ab4e27-4d|
  000000a0  34 66 2d 34 65 34 66 2d  62 34 65 63 2d 32 62 65  |4f-4e4f-b4ec-2be|
  000000b0  38 33 62 38 35 63 65 66  38 0a 30 30 61 62 34 65  |83b85cef8.00ab4e|
  000000c0  32 37 2d 34 64 34 66 2d  34 65 34 66 2d 62 34 65  |27-4d4f-4e4f-b4e|
  000000d0  63 2d 32 62 65 38 33 62  38 35 63 65 66 38 0d 0a  |c-2be83b85cef8..|
  000000e0  30 30 63 36 35 30 64 37  2d 39 30 32 61 2d 34 32  |00c650d7-902a-42|
  000000f0  32 33 2d 62 31 31 62 2d  63 35 62 34 31 61 62 38  |23-b11b-c5b41ab8|
  00000100  66 31 66 35 0d 0a 30 30  65 35 35 32 35 36 2d 39  |f1f5..00e55256-9|
  00000110  38 31 65 2d 34 61 34 61  2d 61 64 38 62 2d 61 30  |81e-4a4a-ad8b-a0|
  00000120  61 36 63 62 37 35 66 34  37 63 0a 30 30 65 65 61  |a6cb75f47c.00eea|
  00000130  64 34 62 2d 31 64 38 66  2d 34 31 38 62 2d 38 30  |d4b-1d8f-418b-80|
  00000140  63 31 2d 39 37 36 63 35  35 61 62 62 36 32 39 0a  |c1-976c55abb629.|
  00000150  30 30 65 65 61 64 34 62  2d 31 64 38 66 2d 34 31  |00eead4b-1d8f-41|
  00000160  38 62 2d 38 30 63 31 2d  39 37 36 63 35 35 61 62  |8b-80c1-976c55ab|
  00000170  62 36 32 39 0d 0a                                 |b629..|
  00000176
  </code></pre>
  <p>It seems there is an extra '.', but not sure how to remove that.</p>
  <p>(*) edit to update the results with hexdump</p>