title: Filtering data over a large combinations of strings
body: <p>I have a dataframe(around 50 gb) and 3 lists (a few hundred elements each with total number of combinations 445005).</p>
  
  <p>I need to check if values from column <code>url</code> match any of the combinations from the 3 lists and return this combinations. I do it this way.</p>
  
  <pre><code>def checkMatch(query1:List[String], query2:List[String], model:List[String]):List[(String, String, String)]= {
  for{
  x &lt;- query1
  y &lt;- query2
  z &lt;- model
  if(url.contains(x) &amp;&amp; url.contains(y) &amp;&amp; url.contains(z))
  } yield (x,y,z)
  }
  </code></pre>
  
  <p>This fails.As in, the aplication doesn't stop, but every task fails with </p>
  
  <pre><code>ExecutorLostFailure (executor 26 exited unrelated to the running tasks) Reason: Container container on host: host was preempted.
  </code></pre>
  
  <p>And the application runs until I kill it, without any task completing.</p>
  
  <p>All I found about the error suggests lack of memory. My configs are </p>
  
  <pre><code>spark-submit \
  --class Main \
  --master yarn \
  --deploy-mode client \
  --num-executors 200 \
  --executor-cores 10 \
  --driver-memory 4G \
  --executor-memory 8G \
  --files hive-site.xml#hive-site.xml \
  --conf spark.task.maxFailures=10 \
  --conf spark.executor.memory=8G \
  --conf spark.app.name=spark-job \
  --conf spark.yarn.executor.memoryOverhead=4096 \
  --conf spark.yarn.driver.memoryOverhead=2048 \
  --conf spark.shuffle.service.enabled=true \
  --conf spark.shuffle.consolidateFiles=true \
  --conf spark.broadcast.compress=true \
  --conf spark.shuffle.compress=true \
  --conf spark.shuffle.spill.compress=true \
  --conf spark.network.timeout=10000000 \
  --conf spark.executor.heartbeatInterval=10000000 \
  </code></pre>
  
  <p>I tried allocating 2-3 times more memory. It didn't help.</p>
  
  <p>What other solutions are there? Andis lack of memory actually the reason?</p>