title: spacy tokenize apostrophe
body: <p>I am trying to properly split words to fit my corpus. I'm already using <a href="https://stackoverflow.com/questions/51012476/spacy-custom-tokenizer-to-include-only-hyphen-words-as-tokens-using-infix-regex">this approach</a> which fixes hyphenated words, what I can't seem to figure out is <strong>how to keep words with apostrophes for contractions like: <em>can't, won't, don't, he's</em>, etc. together as one token in spacy.</strong></p>
  
  <p><strong>More specifically I am searching how to do this for Dutch words: <em>zo'n, auto's, massa's,</em> etc. but this problem should be language-independent.</strong></p>
  
  <p>I have the following tokenizer:</p>
  
  <pre><code>def custom_tokenizer(nlp):
      prefix_re = compile_prefix_regex(nlp.Defaults.prefixes)
      suffix_re = compile_suffix_regex(nlp.Defaults.suffixes)
      infix_re = re.compile(r'''[.\,\?\:\;\...\‘\’\'\`\“\”\"\'~]''')
  
      return Tokenizer(nlp.vocab, prefix_search=prefix_re.search,
                       suffix_search=suffix_re.search,
                       infix_finditer=infix_re.finditer,
                       token_match=None)
  nlp = spacy.load('nl_core_news_sm')
  nlp.tokenizer = custom_tokenizer(nlp)
  </code></pre>
  
  <p>with this the tokens I get are:</p>
  
  <p>'Mijn','eigen','huis','staat',<strong>'zo',"'",'n'</strong>,'zes','meter','onder','het','wateroppervlak','van','de','Noordzee','.'</p>
  
  <p>...but the tokens I expected should be:</p>
  
  <p>'Mijn','eigen','huis','staat',<strong>"zo'n"</strong>,'zes','meter','onder','het','wateroppervlak','van','de','Noordzee','.'</p>
  
  <p>I know it is possible to add custom rules like:</p>
  
  <pre><code>case = [{ORTH: "zo"}, {ORTH: "'n", LEMMA: "een"}]
  tokenizer.add_special_case("zo'n",case)
  </code></pre>
  
  <p>But I am looking for a more general solution.</p>
  
  <p>I've tried editing the <code>infix_re</code> regex from the other thread, but I doesn't seem to have any impact on the issue. Is there any setting or change I can do to fix this?</p>