<p>I have a spark cluster set up and would like to integrate spark-nlp to run named entity recognition. I need to access the model from disk rather than download it from the internet at runtime. I have downloaded the <code>recognize_entities_dl</code> model from the model download page and placed the unzipped files where spark <em>should</em> be able to access it. When I run the following code:</p>

<pre><code>ner = NerDLModel.pretrained('/path/to/unzipped/files')
</code></pre>

<p>I see the <code>Can not find the model to download please check the name!</code> message, indicating it can't find the files followed by a stacktrace further down in the code. I've also tried the <code>PretrainedPipeline</code> class with similar results.</p>

<p>A few important details for what they're worth:</p>

<p>spark version: 2.4.4</p>

<p>sparknlp version: 2.3.3</p>

<p>Spark is running in a docker container within a kubernetes pod. I can exec into this container and run commands manually to reproduce the problem. It looks like <code>_internal._GetResourceSize</code> is returning a -1, causing the loader to exit. I also get some warnings about http, but all I'm trying to do is access a local file so not sure what that would have to do with things:</p>

<pre><code>&gt;&gt;&gt; _internal._GetResourceSize('/path/in/container/recognize_entities_dl_en_2.1.0_2.4_1562946909722', 'en', remote_loc=None).apply()
19/12/02 20:29:03 WARN ApacheUtils: NoSuchMethodError was thrown when disabling normalizeUri. This indicates you are using an old version (&lt; 4.5.8) of Apache http client. It is recommended to use http client version &gt;= 4.5.9 to avoid the breaking change introduced in apache client 4.5.7 and the latency in exception handling. See https://github.com/aws/aws-sdk-java/issues/1919 for more information
19/12/02 20:29:03 WARN ApacheUtils: NoSuchMethodError was thrown when disabling normalizeUri. This indicates you are using an old version (&lt; 4.5.8) of Apache http client. It is recommended to use http client version &gt;= 4.5.9 to avoid the breaking change introduced in apache client 4.5.7 and the latency in exception handling. See https://github.com/aws/aws-sdk-java/issues/1919 for more information
'-1'
&gt;&gt;&gt;
</code></pre>

<p>You are trying to load a pre-trained pipeline inside an annotator. There are two types of pre-trained resources: model and pipeline. The pre-trained model can be load inside the annotator which later will be used inside of a pipeline, however, the pre-trained pipeline can be just loaded easily and used after.</p>
<ul>
<li>Example of the pre-trained pipeline (online-requires internet):</li>
</ul>
<pre class="lang-scala prettyprint-override"><code>import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline
import com.johnsnowlabs.nlp.SparkNLP

SparkNLP.version()

val testData = spark.createDataFrame(Seq(
(1, &quot;Google has announced the release of a beta version of the popular TensorFlow machine learning library&quot;),
(2, &quot;Donald John Trump (born June 14, 1946) is the 45th and current president of the United States&quot;)
)).toDF(&quot;id&quot;, &quot;text&quot;)

// Pay attention, for loading a pre-trained pipeline we use PretrainedPipeline
val pipeline = PretrainedPipeline(&quot;recognize_entities_dl&quot;, lang=&quot;en&quot;)

val annotation = pipeline.transform(testData)

annotation.show()

/*
import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline
import com.johnsnowlabs.nlp.SparkNLP
2.4.0
testData: org.apache.spark.sql.DataFrame = [id: int, text: string]
pipeline: com.johnsnowlabs.nlp.pretrained.PretrainedPipeline = PretrainedPipeline(entity_recognizer_dl,en,public/models)
annotation: org.apache.spark.sql.DataFrame = [id: int, text: string ... 6 more fields]
+---+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
| id|                text|            document|            sentence|               token|          embeddings|                 ner|       entities|
+---+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
|  1|Google has announ...|[[document, 0, 10...|[[document, 0, 10...|[[token, 0, 5, Go...|[[word_embeddings...|[[named_entity, 0...|[[chunk, 0, 5, Go...|
|  2|Donald John Trump...|[[document, 0, 92...|[[document, 0, 92...|[[token, 0, 5, Do...|[[word_embeddings...|[[named_entity, 0...|[[chunk, 0, 16, D...|
+---+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
*/

annotation.select(&quot;entities.result&quot;).show(false)

/*
+----------------------------------+
|result                            |
+----------------------------------+
|[Google, TensorFlow]              |
|[Donald John Trump, United States]|
+----------------------------------+
*/

</code></pre>
<ul>
<li>Example of the pre-trained pipeline (offline-loading a saved pipeline):</li>
</ul>
<pre class="lang-scala prettyprint-override"><code>import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline
import com.johnsnowlabs.nlp.SparkNLP

SparkNLP.version()

val testData = spark.createDataFrame(Seq(
(1, &quot;Google has announced the release of a beta version of the popular TensorFlow machine learning library&quot;),
(2, &quot;Donald John Trump (born June 14, 1946) is the 45th and current president of the United States&quot;)
)).toDF(&quot;id&quot;, &quot;text&quot;)

// Here we are loading a pre-trained pipeline we already downloaded manually for offline use

val pipeline = PretrainedPipeline.load(&quot;/path/in/container/recognize_entities_dl_en_2.1.0_2.4_1562946909722&quot;)

val annotation = pipeline.transform(testData)

annotation.show()

/*
import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline
import com.johnsnowlabs.nlp.SparkNLP
2.4.0
testData: org.apache.spark.sql.DataFrame = [id: int, text: string]
pipeline: com.johnsnowlabs.nlp.pretrained.PretrainedPipeline = PretrainedPipeline(entity_recognizer_dl,en,public/models)
annotation: org.apache.spark.sql.DataFrame = [id: int, text: string ... 6 more fields]
+---+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
| id|                text|            document|            sentence|               token|          embeddings|                 ner|       ner_converter|
+---+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
|  1|Google has announ...|[[document, 0, 10...|[[document, 0, 10...|[[token, 0, 5, Go...|[[word_embeddings...|[[named_entity, 0...|[[chunk, 0, 5, Go...|
|  2|Donald John Trump...|[[document, 0, 92...|[[document, 0, 92...|[[token, 0, 5, Do...|[[word_embeddings...|[[named_entity, 0...|[[chunk, 0, 16, D...|
+---+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
*/

annotation.select(&quot;entities.result&quot;).show(false)

/*
+----------------------------------+
|result                            |
+----------------------------------+
|[Google, TensorFlow]              |
|[Donald John Trump, United States]|
+----------------------------------+
*/
</code></pre>
<ul>
<li>Example of loading pre-trained model for NerDLModel</li>
</ul>
<pre class="lang-scala prettyprint-override"><code>// Online
val ner = NerDLModel.pretrained(name=&quot;ner_dl&quot;, lang=&quot;en&quot;)
// Offline - manualy downloaded
val ner = NerDLModel.load(&quot;/path/ner_dl_en_2.4.0_2.4_1580251789753&quot;)
</code></pre>
<p>Let me know if you have any questions or problems with your input data and I'll update my answer.</p>
<p><strong>References</strong>:</p>
<ul>
<li><a href="https://github.com/JohnSnowLabs/spark-nlp" rel="nofollow noreferrer">https://github.com/JohnSnowLabs/spark-nlp</a></li>
<li><a href="https://nlp.johnsnowlabs.com/models" rel="nofollow noreferrer">https://nlp.johnsnowlabs.com/models</a></li>
<li><a href="https://github.com/JohnSnowLabs/spark-nlp-workshop" rel="nofollow noreferrer">https://github.com/JohnSnowLabs/spark-nlp-workshop</a></li>
</ul>
