title: Spark join two dataframes on best .startswith match
body: <p>Two dataframes with phone numbers with ids that could only be matched via regex, e.g.</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>idPrefix</th>
  <th>other columns</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>420</td>
  <td></td>
  </tr>
  <tr>
  <td>42055</td>
  <td></td>
  </tr>
  <tr>
  <td>420551</td>
  <td></td>
  </tr>
  </tbody>
  </table>
  </div><div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>phoneNumber</th>
  <th>other columns</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>420551666</td>
  <td></td>
  </tr>
  <tr>
  <td>421709560</td>
  <td></td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>I would need to join these dataframes on the best match of <code>idPrefix</code> to the <code>phoneNumber</code>, matching the longest starting prefix possible, if there is one. E.g. if there were any option to join on longest <code>idPrefix</code> for <code>phoneNumber.startswith(idPrefix)</code>, that would be great.</p>
  <p>e.g. result would look like
  <a href="https://i.stack.imgur.com/Kx5Tr.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Kx5Tr.png" alt="enter image description here" /></a></p>
  <p>I tried a few UDFs with regex instead of a <code>df.join()</code>, but there does not seem to be a way to have a dataframe as an input to UDF.</p>