title: Is there a "hook" for the end of a stage/task in spark?
body: <p>I have a Spark UDF which needs to have a specific file (MATLAB runtime in my case) installed on the local disk of the executor (we are using YARN). As I don't have direct access to the executor machines, I have to find another way to deploy my runtime on the cluster.
  As the file is very large, I cannot install/delete it for every invokation of the UDF, so I thought about the following strategy:</p>
  
  <ul>
  <li>At the beginning of the UDF, I check whether the runtime is available locally under a specific path, e.g. the <code>tmp</code>-directory </li>
  <li>If the runtime is not yet installed, I fetch it from the HDFS and install in in <code>tmp</code>. This is only done the first time the UDF is called on an specific executor.</li>
  </ul>
  
  <p>The above screnario works fine, but leaves my installed runtimes all over the cluster, so I want to delete them afterwards.</p>
  
  <p>There seems to be no way to add shutdown-hooks for the executors (<a href="https://stackoverflow.com/questions/33375080/spark-worker-shutdown-how-to-free-shared-resources">Spark worker shutdown - how to free shared resources</a>), is there another "hook" e.g. end-of-a-task hook?</p>