title: Apply spacy Classification pipeline on a large corpus
body: <p>I am trying to apply spacY classification pipeline on a large review corpus.</p>
  <p>The end goal of this is to be able to classify reviews on certain categories, and then filter reviews from this large corpus based on a specific threshold for the classification of categories.</p>
  <p>Please see the following code required to import the pipeline:</p>
  <pre><code>    import spacy
  import classy_classification
  
  data = { 'fast delivery': ['summary',
  'Super fast delivery (came roughly 3 days before projected delivery)',
  'Very fast delivery',
  '&quot;Awesome!!! Great price,  fast delivery!!!!&quot;',
  '... much for the fast delivery and I am very satisfied with my ScanDisk'],
  'bad delivery': ['summary',
  '&quot;Good premise, bad delivery&quot;',
  'good product bad delivery comunication',
  '&quot;Good Product, bad delivery&quot;',
  '&quot;Great scope, but bad delivery&quot;',
  &quot;I hate FedEx Smartpost's very bad delivery service&quot;,
  'Watch out for bad delivery',
  '&quot;Great Product, Really bad delivery time!!!!&quot;',
  'Great Product...bad delivery experience',
  '&quot;good product, bad delivery&quot;',
  '&quot;bad delivery, nice camera&quot;',
  'slow delivery',
  '&quot;Great product, slow delivery&quot;',
  '&quot;Very disappointed, and slow delivery after paying for faster&quot;',
  'slow delivery and no response from the seller',
  '&quot;Great idea, poor delivery!&quot;',
  'poor delivery condition']
  }
  
  nlp = spacy.blank(&quot;en&quot;)
  nlp.max_length = 2_534_045 # or higher
  nlp.add_pipe(
      &quot;text_categorizer&quot;, 
      config={
          &quot;data&quot;: data, 
          &quot;model&quot;: &quot;sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2&quot;,
          &quot;device&quot;: &quot;gpu&quot;
      }
  ) 
  </code></pre>
  <p>And the following code to apply the pipeline on a review/text corpus:</p>
  <pre><code>    text = text = &quot;Worked, for about 10 weeks.\r\nBest buy for the money\r\nBe careful\r\nnice and spacious.....\r\nSeems as good as B+W and looks better on Sigma lenses\r\nAn Incredible Protective Case to safeguard your Tom Tom GPS.\r\nThe Bomb-proof container.\r\nFive Stars\r\nTough as nails and great to fly with.\r\nWorks well, but noisy\r\nPERFECT photo refill pages and we do not use any magnet parts, as they really are not needed unless one wants to overfill album.\r\nKoss PortaPro Headphones\r\nMet my needs\r\namazing view\r\nFive Stars\r\nSamsung LNS4051D 40 Wide LCD HDTV with Integrated ATSC Tuner\r\nCable Does the Job\r\nGreat home router\r\nGet More from your sound system\r\nmy new front lens cap\r\nGood Buy\r\nFive Stars\r\nnot rust-proof\r\nThree Stars\r\nBetter than the stick-on mount from Garmin, but only just...\r\nThis will wake you up!\r\nOverpriced---Don\'t waste your money on this.\r\nDISCREPANCY BETWEEN IMAGE AND PRODUCT DESCRIPTION\r\nSpeakers pack a punch!\r\nSilk Tripod Carrying Case\r\nCase logic make the best cases!\r\nFive Stars&quot;
  
  sentence_model = spacy.blank(&quot;en&quot;)
  
  sentence_model.add_pipe(&quot;sentencizer&quot;)
  
  sentences = sentence_model(text)
  
  final_data[1]
  </code></pre>
  <p>The above code should be able to run just fine. However, I am wondering for a larger text file it is not able to run it. Please see the following line for the error code of this.</p>
  <blockquote>
  <p>ValueError: [E088] Text of length 1198902 exceeds maximum of 1000000. The parser and NER models require roughly 1GB of temporary memory per 100,000 characters in the input. This means long texts may cause memory allocation errors. If you're not using the parser or NER, it's probably safe to increase the <code>nlp.max_length</code> limit. The limit is in number of characters, so you can check whether your inputs are too long by checking <code>len(text)</code>.</p>
  </blockquote>
  <p>As you can see, the nlp.max_length is already set to a certain value, but it does not process this somehow. Is there a method to adjust this? Or perhaps another way to process a larger corpus?</p>
  <p>Looking forward for responses.</p>