title: TypeError: Int' object is not callable when calling Bert methods for producing embeddings
body: <p>I have the following code and I obtain 'TypeError: 'tuple' object is not callable'(in new_time) but I dont understand why. I wrote it based on this tutorial
  <a href="https://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/" rel="nofollow noreferrer">https://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/</a>  and
  <a href="https://github.com/getalp/Flaubert" rel="nofollow noreferrer">https://github.com/getalp/Flaubert</a></p>
  <p>My code :</p>
  <pre><code>#torch == 1.8.1
  #numpy == 1.20.2
  #pandas == 1.0.3
  #transformers == 4.6.1
  
  
  from transformers import logging
  logging.set_verbosity_warning()
  
  import numpy as np
  import torch
  from transformers import FlaubertModel, FlaubertTokenizer
  
  language_model_dir = 'flaubert/flaubert_small_cased' # version &gt; 2.0.0
  flaubert, info = FlaubertModel.from_pretrained(language_model_dir, output_loading_info=True)
  flaubert_tokenizer = FlaubertTokenizer.from_pretrained(language_model_dir)
  
  # f_verbatim is a &quot; &lt;class 'pandas.core.series.Series'&gt;&quot;, table of sentences
  tokenized = f_verbatim.apply((lambda x: flaubert_tokenizer.encode(x, add_special_tokens=True, max_length=512, padding=True, truncation=True))) 
  #print(tokenized)
  
  #Padding
  max_len = 0
  for i in tokenized.values:
      if len(i) &gt; max_len:
          max_len = len(i)
  padded = np.array([i + [0] * (max_len - len(i)) for i in tokenized.values])
  # set data to tensor format
  input_ids = torch.tensor(padded)
  print(type(input_ids)) #&lt;class 'torch.Tensor'&gt;
  
  attention_mask = np.where(padded != 0, 1, 0)
  print(type(attention_mask))  #&lt;class 'numpy.ndarray'&gt;
  
  # this line is causing the error
  hidden_state = flaubert(input_ids, attention_mask=attention_mask) 
  
  </code></pre>
  <p>Error :  #Stacktrace
  ​</p>
  <pre><code>--------------------------------------------------------------------------
  TypeError                                 Traceback (most recent call last)
  &lt;ipython-input-7-8d53b819c31a&gt; in &lt;module&gt;
        1 print(flaubert)
  ----&gt; 2 hidden_state = flaubert(input_ids, attention_mask=attention_mask)
  
  ~\Anaconda3\envs\bert\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
      887             result = self._slow_forward(*input, **kwargs)
      888         else:
  --&gt; 889             result = self.forward(*input, **kwargs)
      890         for hook in itertools.chain(
      891                 _global_forward_hooks.values(),
  
  ~\Anaconda3\envs\bert\lib\site-packages\transformers\models\flaubert\modeling_flaubert.py in forward(self, input_ids, attention_mask, langs, token_type_ids, position_ids, lengths, cache, head_mask, inputs_embeds, output_attentions, output_hidden_states, return_dict)
      195 
      196         # generate masks
  --&gt; 197         mask, attn_mask = get_masks(slen, lengths, self.causal, padding_mask=attention_mask)
      198         # if self.is_decoder and src_enc is not None:
      199         #     src_mask = torch.arange(src_len.max(), dtype=torch.long, device=lengths.device) &lt; src_len[:, None]
  
  ~\Anaconda3\envs\bert\lib\site-packages\transformers\models\xlm\modeling_xlm.py in get_masks(slen, lengths, causal, padding_mask)
      104 
      105     # sanity check
  --&gt; 106     assert mask.size() == (bs, slen)
      107     assert causal is False or attn_mask.size() == (bs, slen, slen)
      108 
  
  TypeError: 'int' object is not callable
  </code></pre>
  <p>As I understand, the problem is due to a missing comma but I cannot figure it out.
  printing &quot;flaubert&quot; fucntion give :</p>
  <pre><code>(FlaubertModel(
    (position_embeddings): Embedding(512, 512)
    (embeddings): Embedding(68729, 512, padding_idx=2)
    (layer_norm_emb): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
    (attentions): ModuleList(
      (0): MultiHeadAttention(
        (q_lin): Linear(in_features=512, out_features=512, bias=True)
        (k_lin): Linear(in_features=512, out_features=512, bias=True)
        (v_lin): Linear(in_features=512, out_features=512, bias=True)
        (out_lin): Linear(in_features=512, out_features=512, bias=True)
      )
      (1): MultiHeadAttention(
        (q_lin): Linear(in_features=512, out_features=512, bias=True)
        (k_lin): Linear(in_features=512, out_features=512, bias=True)
        (v_lin): Linear(in_features=512, out_features=512, bias=True)
        (out_lin): Linear(in_features=512, out_features=512, bias=True)
      )
      (2): MultiHeadAttention(
        (q_lin): Linear(in_features=512, out_features=512, bias=True)
        (k_lin): Linear(in_features=512, out_features=512, bias=True)
        (v_lin): Linear(in_features=512, out_features=512, bias=True)
        (out_lin): Linear(in_features=512, out_features=512, bias=True)
      )
      (3): MultiHeadAttention(
        (q_lin): Linear(in_features=512, out_features=512, bias=True)
        (k_lin): Linear(in_features=512, out_features=512, bias=True)
        (v_lin): Linear(in_features=512, out_features=512, bias=True)
        (out_lin): Linear(in_features=512, out_features=512, bias=True)
      )
      (4): MultiHeadAttention(
        (q_lin): Linear(in_features=512, out_features=512, bias=True)
        (k_lin): Linear(in_features=512, out_features=512, bias=True)
        (v_lin): Linear(in_features=512, out_features=512, bias=True)
        (out_lin): Linear(in_features=512, out_features=512, bias=True)
      )
      (5): MultiHeadAttention(
        (q_lin): Linear(in_features=512, out_features=512, bias=True)
        (k_lin): Linear(in_features=512, out_features=512, bias=True)
        (v_lin): Linear(in_features=512, out_features=512, bias=True)
        (out_lin): Linear(in_features=512, out_features=512, bias=True)
      )
    )
    (layer_norm1): ModuleList(
      (0): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
      (1): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
      (2): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
      (3): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
      (4): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
      (5): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
    )
    (ffns): ModuleList(
      (0): TransformerFFN(
        (lin1): Linear(in_features=512, out_features=2048, bias=True)
        (lin2): Linear(in_features=2048, out_features=512, bias=True)
      )
      (1): TransformerFFN(
        (lin1): Linear(in_features=512, out_features=2048, bias=True)
        (lin2): Linear(in_features=2048, out_features=512, bias=True)
      )
      (2): TransformerFFN(
        (lin1): Linear(in_features=512, out_features=2048, bias=True)
        (lin2): Linear(in_features=2048, out_features=512, bias=True)
      )
      (3): TransformerFFN(
        (lin1): Linear(in_features=512, out_features=2048, bias=True)
        (lin2): Linear(in_features=2048, out_features=512, bias=True)
      )
      (4): TransformerFFN(
        (lin1): Linear(in_features=512, out_features=2048, bias=True)
        (lin2): Linear(in_features=2048, out_features=512, bias=True)
      )
      (5): TransformerFFN(
        (lin1): Linear(in_features=512, out_features=2048, bias=True)
        (lin2): Linear(in_features=2048, out_features=512, bias=True)
      )
    )
    (layer_norm2): ModuleList(
      (0): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
      (1): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
      (2): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
      (3): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
      (4): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
      (5): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
    )
  ), {'missing_keys': [], 'unexpected_keys': ['pred_layer.proj.bias', 'pred_layer.proj.weight'], 'error_msgs': []})
  
  </code></pre>
  <p>f_verbatim look likes this :</p>
  <pre><code>&lt;class 'pandas.core.series.Series'&gt;
  0      Dans le cadre de l’ATEX, il y a certains types de départ moteur qu’on va mesurer la température de pot du moteur et en cas d’anomalie il faut absolument couper le moteur. 
  1      moi ce qui me dérange. C’est quand on a des enfants en bas âge. C’est dangereux, c’est trop facile                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
  2      par rapport à une… enfin, à ce qui existe actuellement, si on parle du Tesys U… Enfin, sur Canopen, par exemple.                                                                                                                                                                       
  3      Je ne verrais pas ça pour une machine, on va dire, une application. Ce serait pour plusieurs machines.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
  4      Spécifique : Pas n’importe qui pourrait le prendre  
  
  </code></pre>
  <p>attention_mask look like this :</p>
  <pre><code>tensor([[  0, 156,  20,  ...,   0,   0,   0],
          [  0, 253,  45,  ...,   0,   0,   0],
          [  0,  38, 243,  ...,   0,   0,   0],
          ...,
          [  0, 141, 104,  ...,   0,   0,   0],
          [  0,  59, 178,  ...,   0,   0,   0],
          [  0, 141, 432,  ...,   0,   0,   0]], dtype=torch.int32)
  </code></pre>
  <p>input_ids like this :</p>
  <pre><code>[[0 1 1 ... 0 0 0]
   [0 1 1 ... 0 0 0]
   [0 1 1 ... 0 0 0]
   ...
   [0 1 1 ... 0 0 0]
   [0 1 1 ... 0 0 0]
   [0 1 1 ... 0 0 0]]
  
  </code></pre>