title: U-SQL Split a CSV file to multiple files based on Distinct values in file
body: <p>I have the Data in Azure Data Lake Store and I am processing the data present there with Azure Data Analytic Job with U-SQL. I have several CSV files which contain spatial data, similar to this:</p>
  
  <p>File_20170301.csv </p>
  
  <pre><code>    longtitude| lattitude | date         | hour | value1
      ----------+-----------+--------------+------+-------
      45.121    | 21.123    | 2017-03-01   | 01   | 20
      45.121    | 21.123    | 2017-03-01   | 02   | 10
      45.121    | 21.123    | 2017-03-01   | 03   | 50
      48.121    | 35.123    | 2017-03-01   | 01   | 60
      48.121    | 35.123    | 2017-03-01   | 02   | 15
      48.121    | 35.123    | 2017-03-01   | 03   | 80
  </code></pre>
  
  <p>File_20170302.csv </p>
  
  <pre><code>    longtitude| lattitude | date         | hour | value1
      ----------+-----------+--------------+------+-------
      45.121    | 21.123    | 2017-03-02   | 01   | 20
      45.121    | 21.123    | 2017-03-02   | 02   | 10
      45.121    | 21.123    | 2017-03-02   | 03   | 50
      48.121    | 35.123    | 2017-03-02   | 01   | 60
      48.121    | 35.123    | 2017-03-02   | 02   | 15
      48.121    | 35.123    | 2017-03-02   | 03   | 80
  </code></pre>
  
  <p>Each file contains data for a different day and for all longtitude-lattitude combinations.</p>
  
  <p>I would like to merge all files I have and split the data so I can end up with one file for each longtitude-lattitude combination. </p>
  
  <p>So, after looping through all files in my folder and appending all data for all days, i would end up with the following:</p>
  
  <p>File_45_21.csv</p>
  
  <pre><code>    longtitude| lattitude | date         | hour | value1
      ----------+-----------+--------------+------+-------
      45.121    | 21.123    | 2017-03-01   | 01   | 20
      45.121    | 21.123    | 2017-03-01   | 02   | 10
      45.121    | 21.123    | 2017-03-01   | 03   | 50
      45.121    | 21.123    | 2017-03-02   | 01   | 20
      45.121    | 21.123    | 2017-03-02   | 02   | 10
      45.121    | 21.123    | 2017-03-02   | 03   | 50
  </code></pre>
  
  <p>File_48_35.csv</p>
  
  <pre><code>    longtitude| lattitude | date         | hour | value1
      ----------+-----------+--------------+------+-------
      48.121    | 35.123    | 2017-03-01   | 01   | 60
      48.121    | 35.123    | 2017-03-01   | 02   | 15
      48.121    | 35.123    | 2017-03-01   | 03   | 80
      48.121    | 35.123    | 2017-03-02   | 01   | 60
      48.121    | 35.123    | 2017-03-02   | 02   | 15
      48.121    | 35.123    | 2017-03-02   | 03   | 80
  </code></pre>
  
  <p>In theory the following should happen:</p>
  
  <ol>
  <li>Find distinct values for a combination of longtitude and lattitude in the data</li>
  <li>take the above Array of distinct values and create a file for each combination and extract the data to it from the source files, based on the two parameters (longtitude and lattitude)</li>
  </ol>
  
  <p>What I am struggling with is how to get started on the looping and the extraction of data based on two parameters in the source and how to "partition" the data source by the distinct values of the combination of parameters. </p>