text: user12391: I am now trying to find intersect genes of two files: one is .bigbed file (very large, 117GB), and the other is a .bed file.
  At first, I tried to convert bigbed file to bed file and then applied bedtools. I didn't run the bigbedtobed program on the server because I am not allowed to install the bigbedtobed program on our server. Thus I planned to convert the .bb file on my own computer and then upload the converted .bed file to the server, and then apply bedtools. Firstly I tried to convert some smaller .bb data sets (< 10 GB) to .bed and it worked slowly but well on my computer. However, when I tried the 117GB data there was an error:
  Invalid argument, Error reading 9188986873 bytes.
  My code is :
  bigBedToBed JASPAR2022_hg19.bb hg1 9.bed
  Thank you in advance!
  
  Now the bug is solved, and I got many other good suggestions. Below you can find many methods to work with the bigbed file, directly working with it or converting it to bed file.
  
  user3848: No matter how you approach this problem 110+ GB is big:
  Your options are:
  
  Break up the file and perform the intersection on a chunk by chunk basis
  Ask your system administrator to install the program
  Load it into Docker, your systems administrator will understand this. Installing docker is a single line of code (which I forget). Docker is a container which protects the system from the program and no systems administrator in this day and age would refuse a request for Docker installation.
  
  Your.systrm administrator might be concerned about the file sizes involved, but I think a better dialogue with them is the right way forward here.
  
  user12391: The question I asked a couple of days ago has been solved. Here I want to make a summary of this and hope it can help people who also encounter such problems later.
  First, about the way to process the bigbed file. I didn't find a very good way to work directly with bigbed file at that time, so I still firstly converted the bigbed file to the bed file using bigBedTobed program and then apply bedtools. The bigBedTobed program can be downloaded through this link: UCSC programs, and it also contains many useful programs from  UCSC genome browser. I finally converted my data (117 GB bigbed file) on the server, and it took ~2.5h to convert it to bed file. After conversion, the converted bed file is about 550 GB. And it took ~4h to intersect with a 4 Mb bed file using bedtools.
  Second, the bug that I encountered when I tried to convert the bigbed file on my own laptop. The code I used is bigBedToBed JASPAR2022_hg19.bb hg1 9.bed and it has error Invalid argument, Error reading 9188986873 bytes. I reported this error to the UCSC genome browser group and they said it is a MacOS specific bug, reaching a default data limit. They have solved it in their development environment but it still needs some time to reach the public site. The current solution for this is to use a URL to the file instead of a local file allows the program to read appropriately sized pieces of data at a time(-udcDir= in bigBedToBed).
  Finally, this is my first time asking questions here and I really appreciate those good suggestions from @M__ and @ColinD.
  
  Renew: Methods from @malcook can work directly with the bigbed file! For a very large bigbed file, it is indeed a more efficient way.
  
  user1692: You asked
  
  Does anyone know any tools that can intersect two bigbed files
  
  ...but it appears you really wanted to subset a single very large bigbed file based on the contents of a much smaller bed file.
  I disagree with your summation "there is not a very efficient way to work directly with the bigbed file".  I see two options.
  The first is to repeatedly call bigBedTobed once for each row in your .bed file.  If you have multiple cores, this can be parallelized using GNU parallel.  The following does that, emitting each overlapping result with additional column being the index into your .bed file from which the result stems.
  cat your.bed | parallel --lb -j --colsep '\t' bigBedToBed -chrom={1} -start={2} -end={3}  your.bb  /dev/stdout '|' sed -e 's/$/\\t${PARALLEL_SEQ}/'
  
  Depending upon your application, there is a second approach which uses R/BioConductor's rtracklayer::import.bb which can be parallelized across (chunks of) your .bed file.
  I considered these approaches in what I expect is an almost identical situation:
  My bigbed holds genome-wide transcription factor binding site predictions from JASPAR Genome Browser tracks as produced by wassermanlab / JASPAR-UCSC-tracks.
  My bed file holds the loci of ~23K putative enhancers as determined by an analysis of regions of accessible chromatin from ATAC-Seq and various ChIP marks.
  In my case I wished to produce a sparse Matrix with one row for each putative enhancer, and one column for each JASPAR motif (which will form the basis of downstream enrichment analyses).
  bbf<-BigBedFile('path/to/your.bb')
  gr<-import.bed('path/to/your.bed')
  ## I found I had to harmonize seqlevels (e.g. chromosome identifiers) to make this work.  YMMV
  gr<-renameSeqlevels(gr,paste0('chr',gsub('^(K.*)\\.(\\d+)$','Un_\\1v\\2',seqlevels(gr))))
  mm<-motifMatrix(gr,bbf)
  
  which results in a sparse Matrix, viz:
  > rownames(mm)<-as.character(gr) # optional.
  > mm[1:5,1:5]
  5 x 5 sparse Matrix of class "dgCMatrix"
                   MA0002.2 MA0003.4 MA0004.1 MA0006.1 MA0007.3
  chr1:11156-12180       14        .        2       28        2
  chr1:17836-20023       25        .       27       63        2
  chr1:22495-24367       22        1       22      137        2
  chr1:27533-28038        2        2        4        7        .
  chr1:36549-37079        6        .        2       41        .
  
  This depends upon the following definition:
  motifMatrix<-function(gr,bbf,motifID=c('name','TFName')[[1]],mc.jobs=100) {
    ## produce a sparse Matrix with one row for each region in
    ## GenomicRanges gr, and one column for each JASPAR motif indicated
    ## as overlapping <gr> accorinding to BigBedFile <bbf>.  Default to
    ## using the 'name' column as the motif identifier, but allow for
    ## possibly using another, such as 'TFName' (as used by 2022 version
    ## of
    ## https://jaspar.genereg.net/genome-tracks/#ucsc_tracks). Parallelize
    ## into mc.jobs queries, defaulting to 100, which was found suitable
    ## for ~25000 loci in <gr> against JASPAR matches to zebrafish
    ## danRer11 genome running on a few dozen cores.  In general,
    ## knowing how "big" each parallel job *should* be depends on
    ## available RAM and cores, and number of loci in <gr>.
    ##
    ## AUTHOR: malcolm_cook@stowers.org
    ##
    ## NB: you must install a version of `rtracklayer` which addresses
    ## issue [import\.bb returning incorrect ranges & results as
    ## compared with kent tools
    ## bigBedToBed](https://github.com/lawremi/rtracklayer/issues/59)
    ## (such as by `install_github('lawremi/rtracklayer')`)
    si<-parallel::splitIndices(length(gr),mc.jobs)
    mm<-mclapply(si,function(i) {
      gri<-gr[i]
      m<-import.bb(bbf,selection=BigBedSelection(gri, colnames = c(motifID)))
      fo<-findOverlaps(gri,m)
      xt<-
        xtabs(~.,data=list(locus=i[queryHits(fo)],motif=mcols(m[subjectHits(fo)])[[motifID]]),sparse=TRUE) 
      xt
    })
    mm<-
      ## rBind.fill(mm) ## NOT!  It does NOT preserve dimnames and
      ## returns matrix when given Matrix.  Issue reported:
      ## [[https://github.com/cvarrichio/Matrix.utils/issues/5][rBind.fill
      ## on a list of Matrix arguably should return Matrix but returns
      ## matrix #5]]
      Reduce(rBind.fill,mm) # which does preserve dimnames and return Matrix
    mm
  }
  
  NB: If you choose this approach, you MUST install a version of rtracklayer which addresses issue the import.bb returning incorrect ranges & results as compared with kent tools bigBedToBed (such as by install_github('lawremi/rtracklayer'))
  
  user339: My 2p, a little late and maybe not too useful:
  
  I didn't run the bigbedtobed program on the server because I am not allowed to install the bigbedtobed program on our server.
  
  Presumably, you cannot install in /usr/local/bin or some other root directory because you don't have admin rights. This is the common setup on shared servers. However, many programs can be installed in the user's home directory without admin rights. Even better, install conda and mamba first (no need to be admin) and use that to install other programs.
  The problem with docker is that some system administrators are concerned about security and refuse to install it (and docker does need admin rights). Whether such concerns are founded, I don't know...
  
  I tried to convert bigbed file to bed file and then applied bedtools, but I found it was very slow and couldn't make it.
  
  bedtools intersect supports the -sorted options that should speed things up quite a bit - Have you applied it? Your huge bigbed files should be already sorted after you convert to bed.
  
original_text: 12391: <p>I am now trying to find intersect genes of two files: one is .bigbed file (very large, 117GB), and the other is a .bed file.</p>
  <p>At first, I tried to convert bigbed file to bed file and then applied bedtools. I didn't run the bigbedtobed program on the server because I am not allowed to install the bigbedtobed program on our server. Thus I planned to convert the .bb file on my own computer and then upload the converted .bed file to the server, and then apply bedtools. Firstly I tried to convert some smaller .bb data sets (&lt; 10 GB) to .bed and it worked slowly but well on my computer. However, when I tried the 117GB data there was an error:</p>
  <p><code>Invalid argument, Error reading 9188986873 bytes.</code></p>
  <p>My code is :</p>
  <p><code>bigBedToBed JASPAR2022_hg19.bb hg1 9.bed</code></p>
  <p>Thank you in advance!</p>
  <hr />
  <p>Now the bug is solved, and I got many other good suggestions. Below you can find many methods to work with the bigbed file, directly working with it or converting it to bed file.</p>
   3848: <p>No matter how you approach this problem 110+ GB is big:</p>
  <p>Your options are:</p>
  <ol>
  <li>Break up the file and perform the intersection on a chunk by chunk basis</li>
  <li>Ask your system administrator to install the program</li>
  <li>Load it into Docker, your systems administrator will understand this. Installing docker is a single line of code (which I forget). Docker is a container which protects the system from the program and no systems administrator in this day and age would refuse a request for Docker installation.</li>
  </ol>
  <p>Your.systrm administrator might be concerned about the file sizes involved, but I think a better dialogue with them is the right way forward here.</p>
   12391: <p>The question I asked a couple of days ago has been solved. Here I want to make a summary of this and hope it can help people who also encounter such problems later.</p>
  <p>First, about the way to process the bigbed file. I didn't find a very good way to work directly with bigbed file at that time, so I still firstly converted the bigbed file to the bed file using <code>bigBedTobed</code> program and then apply bedtools. The <code>bigBedTobed</code> program can be downloaded through this link: <a href="https://hgdownload.soe.ucsc.edu/admin/exe/" rel="nofollow noreferrer">UCSC programs</a>, and it also contains many useful programs from  UCSC genome browser. I finally converted my data (117 GB bigbed file) on the server, and it took ~2.5h to convert it to bed file. After conversion, the converted bed file is about 550 GB. And it took ~4h to intersect with a 4 Mb bed file using bedtools.</p>
  <p>Second, the bug that I encountered when I tried to convert the bigbed file on my own laptop. The code I used is <code>bigBedToBed JASPAR2022_hg19.bb hg1 9.bed</code> and it has error <code>Invalid argument, Error reading 9188986873 bytes.</code> I reported this error to the UCSC genome browser group and they said it is a MacOS specific bug, reaching a default data limit. They have solved it in their development environment but it still needs some time to reach the public site. The current solution for this is to use a URL to the file instead of a local file allows the program to read appropriately sized pieces of data at a time<code>(-udcDir= in bigBedToBed)</code>.</p>
  <p>Finally, this is my first time asking questions here and I really appreciate those good suggestions from @M__ and @ColinD.</p>
  <hr />
  <p>Renew: Methods from @malcook can work directly with the bigbed file! For a very large bigbed file, it is indeed a more efficient way.</p>
   1692: <p>You asked</p>
  <blockquote>
  <p>Does anyone know any tools that can intersect two bigbed files</p>
  </blockquote>
  <p>...but it appears you really wanted to subset a single very large bigbed file based on the contents of a much smaller bed file.</p>
  <p>I disagree with your summation &quot;there is not a very efficient way to work directly with the bigbed file&quot;.  I see two options.</p>
  <p>The first is to repeatedly call <code>bigBedTobed</code> once for each row in your .bed file.  If you have multiple cores, this can be parallelized using <a href="https://www.gnu.org/software/parallel/" rel="nofollow noreferrer">GNU parallel</a>.  The following does that, emitting each overlapping result with additional column being the index into your .bed file from which the result stems.</p>
  <pre><code>cat your.bed | parallel --lb -j --colsep '\t' bigBedToBed -chrom={1} -start={2} -end={3}  your.bb  /dev/stdout '|' sed -e 's/$/\\t${PARALLEL_SEQ}/'
  </code></pre>
  <p>Depending upon your application, there is a second approach which uses R/BioConductor's <code>rtracklayer::import.bb</code> which can be parallelized across (chunks of) your .bed file.</p>
  <p>I considered these approaches in what I expect is an almost identical situation:</p>
  <p>My bigbed holds genome-wide transcription factor binding site predictions from <a href="https://jaspar.genereg.net/genome-tracks/#ucsc_tracks" rel="nofollow noreferrer">JASPAR Genome Browser tracks</a> as produced by <a href="https://github.com/wassermanlab" rel="nofollow noreferrer">wassermanlab / JASPAR-UCSC-tracks</a>.</p>
  <p>My bed file holds the loci of ~23K putative enhancers as determined by an analysis of regions of accessible chromatin from ATAC-Seq and various ChIP marks.</p>
  <p>In my case I wished to produce a sparse Matrix with one row for each putative enhancer, and one column for each JASPAR motif (which will form the basis of downstream enrichment analyses).</p>
  <pre><code>bbf&lt;-BigBedFile('path/to/your.bb')
  gr&lt;-import.bed('path/to/your.bed')
  ## I found I had to harmonize seqlevels (e.g. chromosome identifiers) to make this work.  YMMV
  gr&lt;-renameSeqlevels(gr,paste0('chr',gsub('^(K.*)\\.(\\d+)$','Un_\\1v\\2',seqlevels(gr))))
  mm&lt;-motifMatrix(gr,bbf)
  </code></pre>
  <p>which results in a sparse Matrix, viz:</p>
  <pre><code>&gt; rownames(mm)&lt;-as.character(gr) # optional.
  &gt; mm[1:5,1:5]
  5 x 5 sparse Matrix of class &quot;dgCMatrix&quot;
                   MA0002.2 MA0003.4 MA0004.1 MA0006.1 MA0007.3
  chr1:11156-12180       14        .        2       28        2
  chr1:17836-20023       25        .       27       63        2
  chr1:22495-24367       22        1       22      137        2
  chr1:27533-28038        2        2        4        7        .
  chr1:36549-37079        6        .        2       41        .
  </code></pre>
  <p>This depends upon the following definition:</p>
  <pre><code>motifMatrix&lt;-function(gr,bbf,motifID=c('name','TFName')[[1]],mc.jobs=100) {
    ## produce a sparse Matrix with one row for each region in
    ## GenomicRanges gr, and one column for each JASPAR motif indicated
    ## as overlapping &lt;gr&gt; accorinding to BigBedFile &lt;bbf&gt;.  Default to
    ## using the 'name' column as the motif identifier, but allow for
    ## possibly using another, such as 'TFName' (as used by 2022 version
    ## of
    ## https://jaspar.genereg.net/genome-tracks/#ucsc_tracks). Parallelize
    ## into mc.jobs queries, defaulting to 100, which was found suitable
    ## for ~25000 loci in &lt;gr&gt; against JASPAR matches to zebrafish
    ## danRer11 genome running on a few dozen cores.  In general,
    ## knowing how &quot;big&quot; each parallel job *should* be depends on
    ## available RAM and cores, and number of loci in &lt;gr&gt;.
    ##
    ## AUTHOR: malcolm_cook@stowers.org
    ##
    ## NB: you must install a version of `rtracklayer` which addresses
    ## issue [import\.bb returning incorrect ranges &amp; results as
    ## compared with kent tools
    ## bigBedToBed](https://github.com/lawremi/rtracklayer/issues/59)
    ## (such as by `install_github('lawremi/rtracklayer')`)
    si&lt;-parallel::splitIndices(length(gr),mc.jobs)
    mm&lt;-mclapply(si,function(i) {
      gri&lt;-gr[i]
      m&lt;-import.bb(bbf,selection=BigBedSelection(gri, colnames = c(motifID)))
      fo&lt;-findOverlaps(gri,m)
      xt&lt;-
        xtabs(~.,data=list(locus=i[queryHits(fo)],motif=mcols(m[subjectHits(fo)])[[motifID]]),sparse=TRUE) 
      xt
    })
    mm&lt;-
      ## rBind.fill(mm) ## NOT!  It does NOT preserve dimnames and
      ## returns matrix when given Matrix.  Issue reported:
      ## [[https://github.com/cvarrichio/Matrix.utils/issues/5][rBind.fill
      ## on a list of Matrix arguably should return Matrix but returns
      ## matrix #5]]
      Reduce(rBind.fill,mm) # which does preserve dimnames and return Matrix
    mm
  }
  </code></pre>
  <p>NB: If you choose this approach, you MUST install a version of <code>rtracklayer</code> which addresses issue the <a href="https://github.com/lawremi/rtracklayer/issues/59" rel="nofollow noreferrer">import.bb returning incorrect ranges &amp; results as compared with kent tools bigBedToBed</a> (such as by <code>install_github('lawremi/rtracklayer')</code>)</p>
   339: <p>My 2p, a little late and maybe not too useful:</p>
  <blockquote>
  <p>I didn't run the bigbedtobed program on the server because I am not allowed to install the bigbedtobed program on our server.</p>
  </blockquote>
  <p>Presumably, you cannot install in <code>/usr/local/bin</code> or some other root directory because you don't have admin rights. This is the common setup on shared servers. However, many programs can be installed in the user's home directory without admin rights. Even better, install conda and mamba first (no need to be admin) and use that to install other programs.</p>
  <p>The problem with docker is that some system administrators are concerned about security and refuse to install it (and docker does need admin rights). Whether such concerns are founded, I don't know...</p>
  <blockquote>
  <p>I tried to convert bigbed file to bed file and then applied bedtools, but I found it was very slow and couldn't make it.</p>
  </blockquote>
  <p>bedtools intersect supports the <code>-sorted</code> options that should speed things up quite a bit - Have you applied it? Your huge bigbed files should be already sorted after you convert to bed.</p>
  