title: NLTK is called and got error of "punkt" not found on databricks pyspark
body: <p>I would like to call NLTK to do some NLP on databricks by pyspark.
  I have installed NLTK from the library tab of databricks. It should be accessible from all nodes.</p>
  <p>My py3 code :</p>
  <pre><code> import pyspark.sql.functions as F
   from pyspark.sql.types import StringType
   import nltk
   nltk.download('punkt')
   
  
   def get_keywords1(col):
       sentences = []
       sentence = nltk.sent_tokenize(col)
        
  
   get_keywords_udf = F.udf(get_keywords1, StringType())
  </code></pre>
  <p>I run the above code and got:</p>
  <pre><code> [nltk_data] Downloading package punkt to /root/nltk_data...
   [nltk_data]   Package punkt is already up-to-date!
  </code></pre>
  <p>When I run the following code:</p>
  <pre><code> t = spark.createDataFrame(
   [(2010, 1, 'rdc', 'a book'), (2010, 1, 'rdc','a car'),
    (2007, 6, 'utw', 'a house'), (2007, 6, 'utw','a hotel')
   ], 
   (&quot;year&quot;, &quot;month&quot;, &quot;u_id&quot;, &quot;objects&quot;))
   
   t1 = t.withColumn('keywords', get_keywords_udf('objects'))
   t1.show() # error here !
  </code></pre>
  <p>I got error:</p>
  <pre><code> &lt;span class=&quot;ansi-red-fg&quot;&gt;&amp;gt;&amp;gt;&amp;gt; import nltk
  
   PythonException: 
    An exception was thrown from the Python worker. Please see the stack trace below.
   Traceback (most recent call last):
    
   LookupError: 
   **********************************************************************
   Resource punkt not found.
   Please use the NLTK Downloader to obtain the resource:
  
   &gt;&gt;&gt; import nltk
   &gt;&gt;&gt; nltk.download('punkt')
  
   For more information see: https://www.nltk.org/data.html
  
  Attempted to load tokenizers/punkt/PY3/english.pickle
  
  Searched in:
  - '/root/nltk_data'
  - '/databricks/python/nltk_data'
  - '/databricks/python/share/nltk_data'
  - '/databricks/python/lib/nltk_data'
  - '/usr/share/nltk_data'
  - '/usr/local/share/nltk_data'
  - '/usr/lib/nltk_data'
  - '/usr/local/lib/nltk_data'
  - ''
  </code></pre>
  <p>I have downloaded 'punkt'. It is located at</p>
  <pre><code>/root/nltk_data/tokenizers
  </code></pre>
  <p>I have updated the PATH in spark environment with the folder location.</p>
  <p>Why it cannot be found ?</p>
  <p>The solution at <a href="https://stackoverflow.com/questions/55297145/nltk-punkt-not-found">NLTK. Punkt not found</a> and this <a href="https://stackoverflow.com/questions/3522372/how-to-config-nltk-data-directory-from-code/22987374#22987374">How to config nltk data directory from code?</a>
  but none of them  work for me.</p>
  <p>I have tried to updated</p>
  <pre><code> nltk.data.path.append('/root/nltk_data/tokenizers/')
  </code></pre>
  <p>it does not work.
  It seems that nltk cannot see the new added path !</p>
  <p>I also copied punkz to the path where nltk will search for.</p>
  <p>cp -r /root/nltk_data/tokenizers/punkt /root/nltk_data</p>
  <p>but, nltk still cannot see it.</p>
  <p>thanks</p>