text: user9417: I am having strange issues lifting this dataset to hg37 from hg38. I am using the HGDP WGS data from ftp://ngs.sanger.ac.uk/production/hgdp/hgdp_wgs.20190516/. The data set split by chromosome, so I first converted them to BED then merged them to a single file using plink1.9. However, when trying to lift to hg37/19 using either liftOver  or CrossMap v0.5.1 I get strange errors. I tried both the chain from ensemble.org and from UCSC, I tried them both compressed and uncompressed. I am starting to think I need to try the conversion on the VCF files but these are so large I would prefer not to duplicate them unnecessarily.
  Data
  The data looks fine. If we look at the BIM:
  user@desktop:/media/luks8tb1/data/genomics/HGDP$ head hgdp_hg38_chrpos.bim
  1   1:10153 0   10153   G   A
  1   1:10163 0   10163   C   T
  1   1:10180 0   10180   C   T
  1   1:10250 0   10250   C   A
  1   1:10257 0   10257   C   A
  1   1:10291 0   10291   T   C
  1   1:10297 0   10297   T   C
  1   1:10327 0   10327   C   T
  1   1:10330 0   10330   A   C
  1   1:10333 0   10333   T   C
  user@desktop:/media/luks8tb1/data/genomics/HGDP$ tail hgdp_hg38_chrpos.bim
  Y   Y:56887228  0   56887228    C   T
  Y   Y:56887285  0   56887285    C   A
  Y   Y:56887316  0   56887316    A   G
  Y   Y:56887463  0   56887463    A   C
  Y   Y:56887491  0   56887491    A   C
  Y   Y:56887503  0   56887503    T   G
  Y   Y:56887583  0   56887583    G   A
  Y   Y:56887631  0   56887631    T   C
  Y   Y:56887837  0   56887837    A   G
  Y   Y:56887844  0   56887844    C   T
  
  The data are in chrpos variant names here but I had the same result using the original rsid.
  Chains
  Chains look fine, and I am able to read them into R without problems:
  user@desktop:/media/luks8tb1/data/genomics/HGDP$ head GRCh38_to_GRCh37.chain
  chain 1 1 248956422 + 10000 297968 1 249250621 + 10000 267719 2
  167417 80249 50000
  40302
  
  chain 1 1 248956422 + 347968 501617 1 249250621 - 248779253 248932902 3
  153649
  
  chain 1 1 248956422 + 585988 12949384 1 249250621 + 521368 13009210 4
  1044707 1 0
  3709 3 0
  user@desktop:/media/luks8tb1/data/genomics/HGDP$ tail GRCh38_to_GRCh37.chain
  
  chain 1 chrY 57227415 + 56821509 57217415 Y 59373566 + 58967656 59363566 11183
  85168 0 1
  78580 0 1
  63496 0 1
  3158 1 0
  30382 0 1
  54147 0 1
  80974
  
  CrossMap
  user@desktop:/media/luks8tb1/data/genomics/HGDP$ CrossMap.py bed GRCh38_to_GRCh37.chain hgdp_hg38_chrpos.bed hgdp_hg37_chrpos.bed
  @ 2020-08-25 20:04:26: Read the chain file:  GRCh38_to_GRCh37.chain
  Traceback (most recent call last):
    File "/home/user/.local/bin/CrossMap.py", line 166, in <module>
      crossmap_bed_file(mapTree, in_file, out_file)
    File "/home/user/.local/lib/python3.6/site-packages/cmmodule/mapbed.py", line 32, in crossmap_bed_file
      for line in ireader.reader(inbed):
    File "/home/user/.local/lib/python3.6/site-packages/cmmodule/ireader.py", line 27, in reader
      yield l.decode('utf8').strip().replace("\r", "")
  UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdd in position 20: invalid continuation byte
  
  liftOver
  user@desktop:/media/luks8tb1/data/genomics/HGDP$ liftOver hgdp_hg38.bed hg38ToHg19.over.chain hgdp_hg37.bed hgdp_hg37_errors.bed
  Reading liftover chains
  Mapping coordinates
  Data format error: expecting at least 3 fields in BED file (hgdp_hg38.bed)
  
  Trying the PED file with liftOver
  I converted BED file to PED with plink1.9:
  plink --bfile hgdp_hg38 --recode --out hgdp_hg38
  
  Then:
  user@desktop:/media/luks8tb1/data/genomics/HGDP$ liftOver hgdp_hg38.ped hg38ToHg19.over.chain hgdp_hg37.ped hgdp_hg37_errors.ped
  Reading liftover chains
  Mapping coordinates
  invalid unsigned integer: "HGDP00001"
  
  There's a few question on Google about this, but nothing that looks useful.
  CrossMap does not support PED.
  
  user4766: BED format can refer to the ".txt" file format that stores chromosomes and positions of genomic features as well as some other things like name. bed format can also refer to the plink format, which belongs to bed, bim, fam trio. The plink bed format stores all the genetic variants, but the program needs to refer to bim file for their positions.
  Genome liftover obviously only needs the coordination of each SNPs, and it does not need individual genotypes at all. In fact, the bed format in PLINK does not provide any useful information. Knowing this means that providing the plink bed file is wrong. And this is pretty clear from the error message.
  Moreover, since the three plink file always needs to be used together, most programs refer to them as plink file format. So whenever people say BED format, it is the txt file format.
  
original_text: 9417: <p>I am having strange issues lifting this dataset to hg37 from hg38. I am using the HGDP WGS data from <a href="ftp://ngs.sanger.ac.uk/production/hgdp/hgdp_wgs.20190516/" rel="nofollow noreferrer">ftp://ngs.sanger.ac.uk/production/hgdp/hgdp_wgs.20190516/</a>. The data set split by chromosome, so I first converted them to BED then merged them to a single file using plink1.9. However, when trying to lift to hg37/19 using either <a href="http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/" rel="nofollow noreferrer">liftOver</a>  or <a href="https://crossmap.readthedocs.io/en/latest/#" rel="nofollow noreferrer">CrossMap v0.5.1</a> I get strange errors. I tried both the chain from <a href="ftp://ftp.ensembl.org/pub/assembly_mapping/homo_sapiens/" rel="nofollow noreferrer">ensemble.org</a> and <a href="http://hgdownload.cse.ucsc.edu/goldenPath/hg38/liftOver/" rel="nofollow noreferrer">from UCSC</a>, I tried them both compressed and uncompressed. I am starting to think I need to try the conversion on the VCF files but these are so large I would prefer not to duplicate them unnecessarily.</p>
  <h1>Data</h1>
  <p>The data looks fine. If we look at the BIM:</p>
  <pre><code>user@desktop:/media/luks8tb1/data/genomics/HGDP<span class="math-container">$ head hgdp_hg38_chrpos.bim
  1   1:10153 0   10153   G   A
  1   1:10163 0   10163   C   T
  1   1:10180 0   10180   C   T
  1   1:10250 0   10250   C   A
  1   1:10257 0   10257   C   A
  1   1:10291 0   10291   T   C
  1   1:10297 0   10297   T   C
  1   1:10327 0   10327   C   T
  1   1:10330 0   10330   A   C
  1   1:10333 0   10333   T   C
  user@desktop:/media/luks8tb1/data/genomics/HGDP$</span> tail hgdp_hg38_chrpos.bim
  Y   Y:56887228  0   56887228    C   T
  Y   Y:56887285  0   56887285    C   A
  Y   Y:56887316  0   56887316    A   G
  Y   Y:56887463  0   56887463    A   C
  Y   Y:56887491  0   56887491    A   C
  Y   Y:56887503  0   56887503    T   G
  Y   Y:56887583  0   56887583    G   A
  Y   Y:56887631  0   56887631    T   C
  Y   Y:56887837  0   56887837    A   G
  Y   Y:56887844  0   56887844    C   T
  </code></pre>
  <p>The data are in chrpos variant names here but I had the same result using the original rsid.</p>
  <h1>Chains</h1>
  <p>Chains look fine, and I am able to read them into R without problems:</p>
  <pre><code>user@desktop:/media/luks8tb1/data/genomics/HGDP$ head GRCh38_to_GRCh37.chain
  chain 1 1 248956422 + 10000 297968 1 249250621 + 10000 267719 2
  167417 80249 50000
  40302
  
  chain 1 1 248956422 + 347968 501617 1 249250621 - 248779253 248932902 3
  153649
  
  chain 1 1 248956422 + 585988 12949384 1 249250621 + 521368 13009210 4
  1044707 1 0
  3709 3 0
  user@desktop:/media/luks8tb1/data/genomics/HGDP$ tail GRCh38_to_GRCh37.chain
  
  chain 1 chrY 57227415 + 56821509 57217415 Y 59373566 + 58967656 59363566 11183
  85168 0 1
  78580 0 1
  63496 0 1
  3158 1 0
  30382 0 1
  54147 0 1
  80974
  </code></pre>
  <h1>CrossMap</h1>
  <pre><code>user@desktop:/media/luks8tb1/data/genomics/HGDP$ CrossMap.py bed GRCh38_to_GRCh37.chain hgdp_hg38_chrpos.bed hgdp_hg37_chrpos.bed
  @ 2020-08-25 20:04:26: Read the chain file:  GRCh38_to_GRCh37.chain
  Traceback (most recent call last):
    File &quot;/home/user/.local/bin/CrossMap.py&quot;, line 166, in &lt;module&gt;
      crossmap_bed_file(mapTree, in_file, out_file)
    File &quot;/home/user/.local/lib/python3.6/site-packages/cmmodule/mapbed.py&quot;, line 32, in crossmap_bed_file
      for line in ireader.reader(inbed):
    File &quot;/home/user/.local/lib/python3.6/site-packages/cmmodule/ireader.py&quot;, line 27, in reader
      yield l.decode('utf8').strip().replace(&quot;\r&quot;, &quot;&quot;)
  UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdd in position 20: invalid continuation byte
  </code></pre>
  <h1>liftOver</h1>
  <pre><code>user@desktop:/media/luks8tb1/data/genomics/HGDP$ liftOver hgdp_hg38.bed hg38ToHg19.over.chain hgdp_hg37.bed hgdp_hg37_errors.bed
  Reading liftover chains
  Mapping coordinates
  Data format error: expecting at least 3 fields in BED file (hgdp_hg38.bed)
  </code></pre>
  <h1>Trying the PED file with liftOver</h1>
  <p>I converted BED file to PED with plink1.9:</p>
  <pre><code>plink --bfile hgdp_hg38 --recode --out hgdp_hg38
  </code></pre>
  <p>Then:</p>
  <pre><code>user@desktop:/media/luks8tb1/data/genomics/HGDP$ liftOver hgdp_hg38.ped hg38ToHg19.over.chain hgdp_hg37.ped hgdp_hg37_errors.ped
  Reading liftover chains
  Mapping coordinates
  invalid unsigned integer: &quot;HGDP00001&quot;
  </code></pre>
  <p><a href="https://www.google.com/search?q=%22liftover%22%20%22invalid%20unsigned%20integer%22" rel="nofollow noreferrer">There's a few question on Google about this, but nothing that looks useful.</a></p>
  <p>CrossMap does not support PED.</p>
   4766: <p>BED format can refer to the &quot;.txt&quot; file format that stores chromosomes and positions of genomic features as well as some other things like name. <code>bed</code> format can also refer to the plink format, which belongs to bed, bim, fam trio. The plink bed format stores all the genetic variants, but the program needs to refer to bim file for their positions.</p>
  <p>Genome liftover obviously only needs the coordination of each SNPs, and it does not need individual genotypes at all. In fact, the bed format in PLINK does not provide any useful information. Knowing this means that providing the plink bed file is wrong. And this is pretty clear from the error message.</p>
  <p>Moreover, since the three plink file always needs to be used together, most programs refer to them as <strong>plink file format</strong>. So whenever people say BED format, it is the txt file format.</p>
  