text: user734: I was curious to give DNA BERT a try. This is a BERT (Bidirectional Encoder Representations from Transformers) model that was trained on short (k=3,4,5, or 6) k-mers of DNA. Overall, It is exciting work since it bridges NLP approaches and genetics data.
  I span up a pytroch3.6 machine on SageMaker with a GPU and tried to assess the prediction of its pre-trained DNABERT model:
  Preparing the data:
  !git clone https://github.com/jerryji1993/DNABERT
  %cd DNABERT
  !python -m pip install --editable .
  %cd examples
  !python -m pip install -r requirements.txt
  !curl -L https://northwestern.app.box.com/shared/static/g8m974tr86h0pvnpymxq84f1yxlhnvbi --output a6-new-12w-0.zip
  !unzip a6-new-12w-0.zip
  !cp ./sample_data/ft/prom-core/6/dev.tsv /home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0
  
  
  Running prediction based on the pre-trained data:
  
  !python run_finetune.py \
     --model_type=dna \
     --tokenizer_name=dna6 \
     --model_name_or_path=/home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0 \
     --task_name=dnaprom \
     --do_predict \
     --data_dir=/home/ec2-user/SageMaker/DNABERT/examples/sample_data/ft/prom-core/6  \
     --max_seq_length=75 \
     --per_gpu_pred_batch_size=128   \
     --output_dir=/home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0/out \
     --predict_dir=/home/ec2-user/SageMaker/DNABERT/examples/result \
     --n_process=4
  
  Output:
  11/09/2020 04:35:57 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 1, distributed training: False, 16-bits training: False
  11/09/2020 04:35:57 - INFO - transformers.configuration_utils -   loading configuration file /home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0/config.json
  11/09/2020 04:35:57 - INFO - transformers.configuration_utils -   Model config BertConfig {
    "architectures": [
      "BertForMaskedLM"
    ],
    "attention_probs_dropout_prob": 0.1,
    "bos_token_id": 0,
    "do_sample": false,
    "eos_token_ids": 0,
    "finetuning_task": "dnaprom",
    "hidden_act": "gelu",
    "hidden_dropout_prob": 0.1,
    "hidden_size": 768,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "initializer_range": 0.02,
    "intermediate_size": 3072,
    "is_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "layer_norm_eps": 1e-12,
    "length_penalty": 1.0,
    "max_length": 20,
    "max_position_embeddings": 512,
    "model_type": "bert",
    "num_attention_heads": 12,
    "num_beams": 1,
    "num_hidden_layers": 12,
    "num_labels": 2,
    "num_return_sequences": 1,
    "num_rnn_layer": 1,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_past": true,
    "pad_token_id": 0,
    "pruned_heads": {},
    "repetition_penalty": 1.0,
    "rnn": "lstm",
    "rnn_dropout": 0.0,
    "rnn_hidden": 768,
    "split": 10,
    "temperature": 1.0,
    "top_k": 50,
    "top_p": 1.0,
    "torchscript": false,
    "type_vocab_size": 2,
    "use_bfloat16": false,
    "vocab_size": 4101
  }
  
  ============================================================
  <class 'transformers.tokenization_dna.DNATokenizer'>
  11/09/2020 04:35:57 - INFO - filelock -   Lock 140334164036072 acquired on /home/ec2-user/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e.lock
  11/09/2020 04:35:57 - INFO - transformers.file_utils -   https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt not found in cache or force_download set to True, downloading to /home/ec2-user/.cache/torch/transformers/tmpa9hkn2q1
  Downloading: 28.7kB [00:00, 12.5MB/s]                                           
  11/09/2020 04:35:57 - INFO - transformers.file_utils -   storing https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt in cache at /home/ec2-user/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
  11/09/2020 04:35:57 - INFO - transformers.file_utils -   creating metadata file for /home/ec2-user/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
  11/09/2020 04:35:57 - INFO - filelock -   Lock 140334164036072 released on /home/ec2-user/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e.lock
  11/09/2020 04:35:57 - INFO - transformers.tokenization_utils -   loading file https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt from cache at /home/ec2-user/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
  11/09/2020 04:35:57 - INFO - transformers.modeling_utils -   loading weights file /home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0/pytorch_model.bin
  
  However, the result directory is empty and I am not sure what did I miss?
  !ls -lh /home/ec2-user/SageMaker/DNABERT/examples/result
  total 0
  
  
  !ls -l  /home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0/out
  total 0
  
  
  EDIT:
  I have tried also to run this on another ec2 machine and apparently it gets a segmentation fault, so my conclusion is that it's unstable. I will train a RoBERTAa model or similar by myself:
  SageMaker doesn't propagate the segmentation fault to the notebook which is another weird thing about it.
  11/12/2020 14:01:18 - INFO - filelock -   Lock 139866003860056 acquired on /root/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e.lock
  11/12/2020 14:01:18 - INFO - transformers.file_utils -   https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt not found in cache or force_download set to True, downloading to /root/.cache/torch/transformers/tmpwpxfkxlm
  Downloading: 28.7kB [00:00, 20.2MB/s]
  11/12/2020 14:01:18 - INFO - transformers.file_utils -   storing https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt in cache at /root/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
  11/12/2020 14:01:18 - INFO - transformers.file_utils -   creating metadata file for /root/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
  11/12/2020 14:01:18 - INFO - filelock -   Lock 139866003860056 released on /root/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e.lock
  11/12/2020 14:01:18 - INFO - transformers.tokenization_utils -   loading file https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt from cache at /root/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
  11/12/2020 14:01:18 - INFO - transformers.modeling_utils -   loading weights file /home/ec2-user/DNABERT/examples/6-new-12w-0/pytorch_model.bin
  Segmentation fault
  
  
  user3848: I don't why this RNN k-mer Pytorch model is not producing an output. I do know that,
   "rnn_dropout": 0.0,
  
  ... is bad news for any ANN (artificial neural network) let alone LSTM RNN, which is a pretty complex model.
  Firstly, you have to assign a meaningful drop out rate and 0.5 is okay.
  If dropout rate = 0 as an output then the model hasn't worked. A dropout layer is essential to present overfitting. Its a foundational part of deep learning. No-one emperically knows quite why it works, there's a few approximate good guesses, but a dropout is essential.
  
  The other thing to note is what does "n_process" mean? If you are parallelising a GPU across 4-cores, okay its got a lot of layers but even so thats alot of cores. It ain't a CPU and I'd get it working on a single GPU core first.
  
original_text: 734: <p>I was curious to give <a href="https://github.com/jerryji1993/DNABERT" rel="nofollow noreferrer">DNA BERT</a> a try. This is a <a href="https://en.wikipedia.org/wiki/BERT_(language_model)" rel="nofollow noreferrer">BERT</a> (Bidirectional Encoder Representations from Transformers) model that was trained on short (<code>k=3,4,5, or 6</code>) k-mers of DNA. Overall, It is exciting work since it bridges NLP approaches and genetics data.</p>
  <p>I span up a pytroch3.6 machine on SageMaker with a GPU and tried to assess the prediction of its pre-trained DNABERT model:</p>
  <p>Preparing the data:</p>
  <pre><code>!git clone https://github.com/jerryji1993/DNABERT
  %cd DNABERT
  !python -m pip install --editable .
  %cd examples
  !python -m pip install -r requirements.txt
  !curl -L https://northwestern.app.box.com/shared/static/g8m974tr86h0pvnpymxq84f1yxlhnvbi --output a6-new-12w-0.zip
  !unzip a6-new-12w-0.zip
  !cp ./sample_data/ft/prom-core/6/dev.tsv /home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0
  </code></pre>
  <ol start="2">
  <li>Running <a href="https://github.com/jerryji1993/DNABERT#4-prediction" rel="nofollow noreferrer">prediction</a> based on the pre-trained data:</li>
  </ol>
  <pre><code>!python run_finetune.py \
     --model_type=dna \
     --tokenizer_name=dna6 \
     --model_name_or_path=/home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0 \
     --task_name=dnaprom \
     --do_predict \
     --data_dir=/home/ec2-user/SageMaker/DNABERT/examples/sample_data/ft/prom-core/6  \
     --max_seq_length=75 \
     --per_gpu_pred_batch_size=128   \
     --output_dir=/home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0/out \
     --predict_dir=/home/ec2-user/SageMaker/DNABERT/examples/result \
     --n_process=4
  </code></pre>
  <p>Output:</p>
  <pre><code>11/09/2020 04:35:57 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 1, distributed training: False, 16-bits training: False
  11/09/2020 04:35:57 - INFO - transformers.configuration_utils -   loading configuration file /home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0/config.json
  11/09/2020 04:35:57 - INFO - transformers.configuration_utils -   Model config BertConfig {
    &quot;architectures&quot;: [
      &quot;BertForMaskedLM&quot;
    ],
    &quot;attention_probs_dropout_prob&quot;: 0.1,
    &quot;bos_token_id&quot;: 0,
    &quot;do_sample&quot;: false,
    &quot;eos_token_ids&quot;: 0,
    &quot;finetuning_task&quot;: &quot;dnaprom&quot;,
    &quot;hidden_act&quot;: &quot;gelu&quot;,
    &quot;hidden_dropout_prob&quot;: 0.1,
    &quot;hidden_size&quot;: 768,
    &quot;id2label&quot;: {
      &quot;0&quot;: &quot;LABEL_0&quot;,
      &quot;1&quot;: &quot;LABEL_1&quot;
    },
    &quot;initializer_range&quot;: 0.02,
    &quot;intermediate_size&quot;: 3072,
    &quot;is_decoder&quot;: false,
    &quot;label2id&quot;: {
      &quot;LABEL_0&quot;: 0,
      &quot;LABEL_1&quot;: 1
    },
    &quot;layer_norm_eps&quot;: 1e-12,
    &quot;length_penalty&quot;: 1.0,
    &quot;max_length&quot;: 20,
    &quot;max_position_embeddings&quot;: 512,
    &quot;model_type&quot;: &quot;bert&quot;,
    &quot;num_attention_heads&quot;: 12,
    &quot;num_beams&quot;: 1,
    &quot;num_hidden_layers&quot;: 12,
    &quot;num_labels&quot;: 2,
    &quot;num_return_sequences&quot;: 1,
    &quot;num_rnn_layer&quot;: 1,
    &quot;output_attentions&quot;: false,
    &quot;output_hidden_states&quot;: false,
    &quot;output_past&quot;: true,
    &quot;pad_token_id&quot;: 0,
    &quot;pruned_heads&quot;: {},
    &quot;repetition_penalty&quot;: 1.0,
    &quot;rnn&quot;: &quot;lstm&quot;,
    &quot;rnn_dropout&quot;: 0.0,
    &quot;rnn_hidden&quot;: 768,
    &quot;split&quot;: 10,
    &quot;temperature&quot;: 1.0,
    &quot;top_k&quot;: 50,
    &quot;top_p&quot;: 1.0,
    &quot;torchscript&quot;: false,
    &quot;type_vocab_size&quot;: 2,
    &quot;use_bfloat16&quot;: false,
    &quot;vocab_size&quot;: 4101
  }
  
  ============================================================
  &lt;class 'transformers.tokenization_dna.DNATokenizer'&gt;
  11/09/2020 04:35:57 - INFO - filelock -   Lock 140334164036072 acquired on /home/ec2-user/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e.lock
  11/09/2020 04:35:57 - INFO - transformers.file_utils -   https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt not found in cache or force_download set to True, downloading to /home/ec2-user/.cache/torch/transformers/tmpa9hkn2q1
  Downloading: 28.7kB [00:00, 12.5MB/s]                                           
  11/09/2020 04:35:57 - INFO - transformers.file_utils -   storing https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt in cache at /home/ec2-user/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
  11/09/2020 04:35:57 - INFO - transformers.file_utils -   creating metadata file for /home/ec2-user/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
  11/09/2020 04:35:57 - INFO - filelock -   Lock 140334164036072 released on /home/ec2-user/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e.lock
  11/09/2020 04:35:57 - INFO - transformers.tokenization_utils -   loading file https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt from cache at /home/ec2-user/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
  11/09/2020 04:35:57 - INFO - transformers.modeling_utils -   loading weights file /home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0/pytorch_model.bin
  </code></pre>
  <p>However, the <code>result</code> directory is empty and I am not sure what did I miss?</p>
  <pre><code>!ls -lh /home/ec2-user/SageMaker/DNABERT/examples/result
  total 0
  
  
  !ls -l  /home/ec2-user/SageMaker/DNABERT/examples/6-new-12w-0/out
  total 0
  </code></pre>
  <hr />
  <p><strong>EDIT:</strong>
  I have tried also to run this on another ec2 machine and apparently it gets a segmentation fault, so my conclusion is that it's unstable. I will train a RoBERTAa model or similar by myself:</p>
  <p>SageMaker doesn't propagate the segmentation fault to the notebook which is another weird thing about it.</p>
  <pre><code>11/12/2020 14:01:18 - INFO - filelock -   Lock 139866003860056 acquired on /root/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e.lock
  11/12/2020 14:01:18 - INFO - transformers.file_utils -   https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt not found in cache or force_download set to True, downloading to /root/.cache/torch/transformers/tmpwpxfkxlm
  Downloading: 28.7kB [00:00, 20.2MB/s]
  11/12/2020 14:01:18 - INFO - transformers.file_utils -   storing https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt in cache at /root/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
  11/12/2020 14:01:18 - INFO - transformers.file_utils -   creating metadata file for /root/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
  11/12/2020 14:01:18 - INFO - filelock -   Lock 139866003860056 released on /root/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e.lock
  11/12/2020 14:01:18 - INFO - transformers.tokenization_utils -   loading file https://raw.githubusercontent.com/jerryji1993/DNABERT/master/src/transformers/dnabert-config/bert-config-6/vocab.txt from cache at /root/.cache/torch/transformers/ea1474aad40c1c8ed4e1cb7c11345ddda6df27a857fb29e1d4c901d9b900d32d.26f8bd5a32e49c2a8271a46950754a4a767726709b7741c68723bc1db840a87e
  11/12/2020 14:01:18 - INFO - transformers.modeling_utils -   loading weights file /home/ec2-user/DNABERT/examples/6-new-12w-0/pytorch_model.bin
  Segmentation fault
  </code></pre>
   3848: <p>I don't why this RNN k-mer Pytorch model is not producing an output. I do know that,</p>
  <pre><code> &quot;rnn_dropout&quot;: 0.0,
  </code></pre>
  <p>... is bad news for any ANN (artificial neural network) let alone LSTM RNN, which is a pretty complex model.</p>
  <p>Firstly, you have to assign a meaningful drop out rate and 0.5 is okay.</p>
  <p>If dropout rate = 0 as an output then the model hasn't worked. A dropout layer is essential to present overfitting. Its a foundational part of deep learning. No-one emperically knows quite why it works, there's a few approximate good guesses, but a dropout is essential.</p>
  <hr />
  <p>The other thing to note is what does &quot;n_process&quot; mean? If you are parallelising a GPU across 4-cores, okay its got a lot of layers but even so thats alot of cores. It ain't a CPU and I'd get it working on a single GPU core first.</p>
  