title: How to dynamically chain when conditions in Pyspark?
body: <h3>Context</h3>
  <p>A dataframe should have the <code>category</code> column, which is based on a set of fixed rules. The set of rules becomes quite large.</p>
  <h3>Question</h3>
  <p>Is there a way to use a list of tuples (see example below) to dynamically chain the <code>when</code> conditions to achieve the same result as hard coded solution at the bottom.</p>
  <pre class="lang-py prettyprint-override"><code># Potential list of rule definitions
  category_rules = [
      ('A', 8, 'small'),
      ('A', 30, 'large'),
      ('B', 5, 'small'),
      # Group, size smaller value --&gt; Category
      # and so on ... e.g.,
  ]
  </code></pre>
  <h3>Example</h3>
  <p>Here is a toy example for reproducibility. A dataframe consisting of groups and ids should have the column <code>category</code> added, which depends on the content of the <code>group</code> column. The list of rules is shown in the section above.</p>
  Input data
  <pre class="lang-py prettyprint-override"><code>data = [('A', '45345', 5), ('C', '55345', 5), ('A', '35345', 10), ('B', '65345', 4)]
  df = spark.createDataFrame(data, ['group', 'id', 'size'])
  </code></pre>
  <pre><code>+-----+-----+-----+
  |group|   id| size|
  +-----+-----+-----+
  |    A|45345|    5|
  |    C|55345|    5|
  |    A|35345|   10|
  |    B|65345|    4|
  +-----+-----+-----+
  </code></pre>
  Hard coded solution
  <pre class="lang-py prettyprint-override"><code>df = df.withColumn(
      'category',
      F.when(
          (F.col('group') == 'A')
          &amp; (F.col('size') &lt; 8),
          F.lit('small')
      ).when(
          (F.col('group') == 'A')
          &amp; (F.col('size') &lt; 30),
          F.lit('large')
      ).when(
          (F.col('group') == 'B')
          &amp; (F.col('size') &lt; 5),
          F.lit('small')
      ).otherwise(
          F.lit('unkown')
      )
  )
  </code></pre>
  <pre class="lang-py prettyprint-override"><code>+-----+-----+----+--------+
  |group|   id|size|category|
  +-----+-----+----+--------+
  |    A|45345|   5|   small|
  |    C|55345|   5|  unkown|
  |    A|35345|  10|   large|
  |    B|65345|   4|   small|
  +-----+-----+----+--------+
  </code></pre>
  <p>[Edit 1] Add more complex conditions to explain why chaining is needed.</p>