title: Unnest a large data table efficiently in R
body: <p>I have a large data frame with about 3.5 million rows and 44 columns. Some of the columns are character class and the others are list. The elements in the list columns do not always share equal length. Below is a representative example.</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th style="text-align: left;">name</th>
  <th style="text-align: left;">id</th>
  <th style="text-align: left;">scores</th>
  <th style="text-align: left;">hobbies</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td style="text-align: left;">A</td>
  <td style="text-align: left;">A102</td>
  <td style="text-align: left;">22</td>
  <td style="text-align: left;">c(sports,music)</td>
  </tr>
  <tr>
  <td style="text-align: left;">B</td>
  <td style="text-align: left;">B098</td>
  <td style="text-align: left;">c(20,71,2)</td>
  <td style="text-align: left;">c(sports,dancing,music)</td>
  </tr>
  <tr>
  <td style="text-align: left;">C</td>
  <td style="text-align: left;">B876</td>
  <td style="text-align: left;">c(66,76)</td>
  <td style="text-align: left;">c(running,dancing)</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>I would like to unnest this table into the following table:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th style="text-align: left;">name</th>
  <th style="text-align: left;">id</th>
  <th style="text-align: left;">scores</th>
  <th style="text-align: left;">hobbies</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td style="text-align: left;">A</td>
  <td style="text-align: left;">A102</td>
  <td style="text-align: left;">22</td>
  <td style="text-align: left;">sports</td>
  </tr>
  <tr>
  <td style="text-align: left;">A</td>
  <td style="text-align: left;">A102</td>
  <td style="text-align: left;">22</td>
  <td style="text-align: left;">music</td>
  </tr>
  <tr>
  <td style="text-align: left;">B</td>
  <td style="text-align: left;">B098</td>
  <td style="text-align: left;">20</td>
  <td style="text-align: left;">sports</td>
  </tr>
  <tr>
  <td style="text-align: left;">B</td>
  <td style="text-align: left;">B098</td>
  <td style="text-align: left;">71</td>
  <td style="text-align: left;">dancing</td>
  </tr>
  <tr>
  <td style="text-align: left;">B</td>
  <td style="text-align: left;">B098</td>
  <td style="text-align: left;">2</td>
  <td style="text-align: left;">music</td>
  </tr>
  <tr>
  <td style="text-align: left;">C</td>
  <td style="text-align: left;">B876</td>
  <td style="text-align: left;">66</td>
  <td style="text-align: left;">running</td>
  </tr>
  <tr>
  <td style="text-align: left;">C</td>
  <td style="text-align: left;">B876</td>
  <td style="text-align: left;">76</td>
  <td style="text-align: left;">dancing</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>I have tried the <code>tidyr:unnest</code> but it takes forever to execute. I have also tried <code>dt_hoist</code> from the <strong>tidyfast</strong> package, but it requires that the columns to unnest must all be the same length when unnested, so it won't work on my dataset. Additionally, I have tried the data table solution posted on R bloggers (see below). However, it requires that each list column is unnested separately, and moreover it doesn't retain other list columns in the data table.</p>
  <pre><code>dt[, list(scores = as.character(unlist(scores))), by = list(name, id)]
  </code></pre>
  <p>Anyone here has any clues as to how to unnest this efficiently? Any help would be greatly appreciated. Thanks in advance.</p>