title: How to implement grouping and filters inside a nested for loop in pig
body: <p>I am new to pig and stuck with implementation of an algorithm. I have a data set which contains start date , drop date , market and size of an object. </p>
  
  <p>Here is what i need to implement </p>
  
  <ol>
  <li>Find unique start dates (implemented)</li>
  <li>For each unique date filter out the values which have a start date &lt;= unique date &lt;= drop date.</li>
  <li>Group the filtered values by market. </li>
  <li>Find the total size of objects in the market.</li>
  </ol>
  
  <p>Final output will be TotalSize for each market each day. The trick is an object can be alive across days and thus it's size need to be included in sum calculation for each day it is alive. An object dies on the drop date. </p>
  
  <p>I have not been able to figure out a way to implement Step 2,3 and 4 of the algorithm efficiently in pig. </p>
  
  <p>Another question is, if it is at all possible to implement this requirement in pig. </p>
  
  <p>Here is what I have tried::</p>
  
  <pre><code>req_data = FOREACH uniqueStartDays {
                                  temp = FILTER joined_data BY joined_data.startDay &lt;= uniqueStartDays.startDay AND uniqueStartDays.startDay &lt;= joined_data.dropDay;
                                  group_temp = GROUP temp BY market;
                                  FOREACH (group) GENERATE market, startDay, assetSizeInMb;
                          };
  </code></pre>
  
  <p>Any help will be appreciated. </p>
  
  <pre><code>Adding the sample data and the expected result 
  
  Market       Start_Time                 End_Time              Drop_Time               Size 
  Allahabad       2016-08-09 12:00        2016-08-09 1:00       2016-08-13 4:00        120 
  Allahabad       2016-08-10 2:00         2016-08-10 3:00       2016-08-12  5:00        60
  Allahabad       2016-08-09 3:00         2016-08-09 4:30       2016-08-09  5:30        80 
  
  Result:
  Market          Day             TotalSize  
  Allahabad       2016-08-09      200  (120+80)
                  2016-08-10      180  (120+60)
                  2016-08-11      180  (120+60)
                  2016-08-12      180  (120+60)
                  2016-08-13      120
  </code></pre>