title: Strange results with huggingface transformer[marianmt] translation of larger text
body: <p>I need to translate large amounts of text from a database. Therefore, I've been dealing with transformers and models for a few days. I'm absolutely no data science expert and unfortunately I don't get any further.</p>
  <p>The problem starts with longer text. The 2nd issue is the usual-maximum token size (512) of the sequencers. Just truncating is not really an option. <a href="https://towardsdatascience.com/how-to-apply-transformers-to-any-length-of-text-a5601410af7f" rel="nofollow noreferrer">Here</a> I did  find a work-around, but it does not work properly and the result is a word salad on longer texts (&gt;300 sequences)</p>
  <p>Here an Example <em>(please ignore the warnings, this is another issues - which does not hurt currently that much)</em>;</p>
  <p>If i take the Example Sentence 2 (55 seq) or 5 times (163 sequences) - <strong>no issues.</strong></p>
  <p>But it get messed up with e.g. 433 sequences (the 3rd green text block in the screenshot).</p>
  <p><a href="https://i.stack.imgur.com/IzYKf.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/IzYKf.png" alt="enter image description here" /></a></p>
  <p>With more than 510 sequences, I tried to split it up in chunks as in the upper described link. But the result here is as well pretty strange.</p>
  <p>I am pretty sure - that I have more than just one mistake and underestimated this topic.
  But I see no alternative (free/cheap) way for translating big amount of text.</p>
  <p>Can you guys help me out? Which (thinking) errors do you see and how would you suggest to solve the issues? Thank you very much.</p>
  <p><a href="https://i.stack.imgur.com/S8jMW.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/S8jMW.png" alt="enter image description here" /></a></p>
  <pre><code>from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
  import torch
  
  if torch.cuda.is_available():  
    dev = &quot;cuda&quot;
  else:  
    dev = &quot;cpu&quot; 
  device = torch.device(dev)
   
  mname = 'Helsinki-NLP/opus-mt-de-en'
  tokenizer = AutoTokenizer.from_pretrained(mname)
  model = AutoModelForSeq2SeqLM.from_pretrained(mname)
  model.to(device)
  
  chunksize = 512
  
  text_short = &quot;Nach nur sieben Seiten appellierte man an die Wählerinnen und Wähler, sich richtig zu entscheiden, nämlich für Frieden, Freiheit, Sozialismus. &quot;
  text_long = text_short
  #this loop is just for debugging/testing and simulating long text
  for x in range(30):
      text_long = text_long + text_short
  
  tokens = tokenizer.encode_plus(text_long, return_tensors=&quot;pt&quot;, add_special_tokens=True, padding=False, truncation=False).to(device)
  str_len = len(tokens['input_ids'][0])
  
  if str_len &gt; 510:
      # split into chunks of 510 tokens, we also convert to list (default is tuple which is immutable)
      input_id_chunks = list(tokens['input_ids'][0].split(chunksize - 2))
      mask_chunks = list(tokens['attention_mask'][0].split(chunksize - 2))
  
      cnt = 1
      for tensor in input_id_chunks:
          print('\033[96m' + 'chunk ' + str(cnt) + ': ' + str(len(tensor)) + '\033[93m')
          cnt += 1
      
      # loop through each chunk
      # https://towardsdatascience.com/how-to-apply-transformers-to-any-length-of-text-a5601410af7f
      for i in range(len(input_id_chunks)):
          # add CLS and SEP tokens to input IDs
          input_id_chunks[i] = torch.cat([
              torch.tensor([101]).to(device), input_id_chunks[i], torch.tensor([102]).to(device)
          ])
          # add attention tokens to attention mask
          mask_chunks[i] = torch.cat([
              torch.tensor([1]).to(device), mask_chunks[i], torch.tensor([1]).to(device)
          ])
          # get required padding length
          pad_len = chunksize - input_id_chunks[i].shape[0]
          # check if tensor length satisfies required chunk size
          if pad_len &gt; 0:
              # if padding length is more than 0, we must add padding
              input_id_chunks[i] = torch.cat([
                  input_id_chunks[i], torch.Tensor([0] * pad_len).to(device)
              ])
              mask_chunks[i] = torch.cat([
                  mask_chunks[i], torch.Tensor([0] * pad_len).to(device)
              ])
     
      input_ids = torch.stack(input_id_chunks)
      attention_mask = torch.stack(mask_chunks)
      input_dict = {'input_ids': input_ids.long(), 'attention_mask': attention_mask.int()}
      
      outputs = model.generate(**input_dict)
      #this doesnt work - following error comes to the console --&gt; &quot;host_softmax&quot; not implemented for 'Long'
      #probs = torch.nn.functional.softmax(outputs[0], dim=-1)
      # probs
      # probs = probs.mean(dim=0)
      # probs
    
  else:
      tokens[&quot;input_ids&quot;] = tokens[&quot;input_ids&quot;][:, :512] #truncating normally not necessary
      tokens[&quot;attention_mask&quot;] = tokens[&quot;attention_mask&quot;][:, :512]
      outputs = model.generate(**tokens)
  
  decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
  print('\033[94m' + str(str_len))
  print('\033[92m' + decoded)
  </code></pre>
  <p>Remark; following libs are necessary:</p>
  <blockquote>
  <p>pip3 install torch==1.9.0+cu102 torchvision==0.10.0+cu102 torchaudio===0.9.0 -f <a href="https://download.pytorch.org/whl/torch_stable.html" rel="nofollow noreferrer">https://download.pytorch.org/whl/torch_stable.html</a></p>
  </blockquote>
  <blockquote>
  <p>pip install transformers</p>
  </blockquote>
  <blockquote>
  <p>pip install sentencepiece</p>
  </blockquote>