text: user14472: I'd like to try a contact prediction method (like TripletRes, CCMpred etc) using a custom set of sequences.  There are about 1.5M sequences, all very similar to the canonical sequence; their distribution reflects the occurence of mutations "in the wild" (they're not clustered or anything like that).  How should I make a MSA in this case?
  For example in the original TripletRes paper it looks like the MSA comes from DeepMSA - this searches UniClust30 and UniRef90 using hhblits and jackhmmer.  I don't see any way to use DeepMSA to make a MSA from a fully custom set of sequences, and also it's not clear how to combine a custom set with the UniClust/UniRef search results.  I can make an alignment of just my data with jackhmmer, but I don't know if that's what the other tools expect.  For one thing, the alignment is pretty huge compared to the typical inputs (~million vs typical DeepMSA outputs of a few hundred); for another, it is a lot less variable (most sequences differ in just 1-2 aa, vs completely different species).
  
  user14806: That's a LOT of sequences! I'm doubtful you'd be able to get TripletRes or CCMPred to run at all with an MSA depth of 1.5 million. I suggest pruning your dataset to remove redundant sequences (sequences which don't add much information, i.e. the ones which differ only 1 or 2 AA). I'm curious if your Neff (normalized number of effective sequences--as defined in the ResTriplet paper you link) is anywhere close to your number of sequences.
  Also, I think you've misunderstood the use of DeepMSA. DeepMSA's main use is to find sequences to include in an MSA. I believe it uses much simpler programs to construct the MSA itself. So you'd use DeepMSA if you wanted to do 1 query sequence --> an MSA of several hundred sequences. Since you already have the (very complete) set of sequences, it wouldn't make sense to use DeepMSA. You could instead just use something like Clustal Omega to generate the MSA. There's also a pretty easy-to-use web server but it's limited to 4k sequences.
  
  user6628: You might have some luck with Parsnp. From the description:
  
  Parsnp was designed to align the core genome of hundreds to thousands of bacterial genomes within a few minutes to few hours. Input can be both draft assemblies and finished genomes, and output includes variant (SNP) calls, core genome phylogeny and multi-alignments. Parsnp leverages contextual information provided by multi-alignments surrounding SNP sites for filtration/cleaning, in addition to existing tools for recombination detection/filtration and phylogenic reconstruction.
  
  The tldr is that Parsnp is meant for aligning large sets of very closely related microbial sequences. For example, others in my lab have used parsnp to align over a million SARS-CoV-2 genomes. The only potential concern is that the Parsnp workflow also finishes with a non-optional phylogeny estimation step that takes quite long on such large datasets. Someone should really make it so that you can opt out of that step...
  Spoiler/disclaimer, as the current maintainer of Parsnp, that someone is probably me :grimace:
  The output of Parsnp is in XMFA. You're probably looking for a fasta alignment, so you can use HarvestTools to convert them:
  harvesttools -x sars_align/parsnp.xmfa -M sars_align/parsnp.maf
  
  Both parsnp and harvest-tools are available on bioconda, and I'd highly recommend installing them that way.
  
original_text: 14472: <p>I'd like to try a contact prediction method (like TripletRes, CCMpred etc) using a custom set of sequences.  There are about 1.5M sequences, all very similar to the canonical sequence; their distribution reflects the occurence of mutations &quot;in the wild&quot; (they're not clustered or anything like that).  How should I make a MSA in this case?</p>
  <p>For example in the original <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6851483/" rel="nofollow noreferrer"><strong>TripletRes paper</strong></a> it looks like the MSA comes from DeepMSA - this searches UniClust30 and UniRef90 using <a href="https://www.openstructure.org/docs/2.2/bindings/hhblits/" rel="nofollow noreferrer"><strong>hhblits</strong></a> and <a href="http://hmmer.org/" rel="nofollow noreferrer"><strong>jackhmmer</strong></a>.  I don't see any way to use DeepMSA to make a MSA from a fully custom set of sequences, and also it's not clear how to combine a custom set with the UniClust/UniRef search results.  I can make an alignment of just my data with jackhmmer, but I don't know if that's what the other tools expect.  For one thing, the alignment is pretty huge compared to the typical inputs (~million vs typical DeepMSA outputs of a few hundred); for another, it is a lot less variable (most sequences differ in just 1-2 aa, vs completely different species).</p>
   14806: <p>That's a LOT of sequences! I'm doubtful you'd be able to get TripletRes or CCMPred to run at all with an MSA depth of 1.5 million. I suggest pruning your dataset to remove redundant sequences (sequences which don't add much information, i.e. the ones which differ only 1 or 2 AA). I'm curious if your Neff (normalized number of effective sequences--as defined in the ResTriplet paper you link) is anywhere close to your number of sequences.</p>
  <p>Also, I think you've misunderstood the use of DeepMSA. DeepMSA's main use is to find sequences to include in an MSA. I believe it uses much simpler programs to construct the MSA itself. So you'd use DeepMSA if you wanted to do 1 query sequence --&gt; an MSA of several hundred sequences. Since you already have the (very complete) set of sequences, it wouldn't make sense to use DeepMSA. You could instead just use something like <a href="http://www.clustal.org/omega/" rel="nofollow noreferrer">Clustal Omega</a> to generate the MSA. There's also a pretty easy-to-use <a href="https://www.ebi.ac.uk/Tools/msa/clustalo/" rel="nofollow noreferrer">web server</a> but it's limited to 4k sequences.</p>
   6628: <p>You might have some luck with <a href="https://github.com/marbl/parsnp" rel="nofollow noreferrer">Parsnp</a>. From the description:</p>
  <blockquote>
  <p>Parsnp was designed to align the core genome of hundreds to thousands of bacterial genomes within a few minutes to few hours. Input can be both draft assemblies and finished genomes, and output includes variant (SNP) calls, core genome phylogeny and multi-alignments. Parsnp leverages contextual information provided by multi-alignments surrounding SNP sites for filtration/cleaning, in addition to existing tools for recombination detection/filtration and phylogenic reconstruction.</p>
  </blockquote>
  <p>The tldr is that Parsnp is meant for aligning large sets of very closely related microbial sequences. For example, others in my lab have used parsnp to align over a million SARS-CoV-2 genomes. The only potential concern is that the Parsnp workflow also finishes with a non-optional phylogeny estimation step that takes quite long on such large datasets. Someone should really make it so that you can opt out of that step...</p>
  <p>Spoiler/disclaimer, as the current maintainer of Parsnp, that someone is probably me :grimace:</p>
  <p>The output of Parsnp is in <a href="http://darlinglab.org/mauve/user-guide/files.html" rel="nofollow noreferrer">XMFA</a>. You're probably looking for a fasta alignment, so you can use <a href="https://github.com/marbl/harvest-tools" rel="nofollow noreferrer">HarvestTools</a> to convert them:</p>
  <pre><code>harvesttools -x sars_align/parsnp.xmfa -M sars_align/parsnp.maf
  </code></pre>
  <p>Both parsnp and harvest-tools are available on bioconda, and I'd highly recommend installing them that way.</p>
  