title: Spark group by Key and partitioning the data
body: <p>I have a large csv file with data in following format.</p>
  <p>cityId1,name,address,.......,zip</p>
  <p>cityId2,name,address,.......,zip</p>
  <p>cityId1,name,address,.......,zip</p>
  <p>........</p>
  <p>cityIdN,name,address,.......,zip</p>
  <p>I am performing following operation on the above csv file:</p>
  <ol>
  <li><p>Group by cityId as key and list of resources as value</p>
  <p>df1.groupBy($&quot;cityId&quot;).agg(collect_list(struct(cols.head, cols.tail: _*)) as &quot;resources&quot;)</p>
  </li>
  <li><p>Change it to jsonRDD</p>
  <p>val jsonDataRdd2 = df2.toJSON.rdd</p>
  </li>
  <li><p>Iterate through each Partition and upload to s3 per key</p>
  </li>
  </ol>
  <ul>
  <li>I can not use dataframe partitionby write because of business logic constraints (how other services read from S3 )</li>
  </ul>
  <p>My Questions:</p>
  <ul>
  <li>What is the default size of a spark partition?</li>
  <li>Let's say default size of partition is X MBs and there is one large record present in the dataFrame with key having Y MBs of data (Y &gt; X) , what would happen in this scenario?</li>
  <li>Do I need to worry about having the same key in different partitions in that case?</li>
  </ul>