text: user3964: I have been experimenting a lot lately with Snakemake, I love it. Recently I also switched to using conda (--use-conda) in the way that is advertised. However, I have some issues with it, mostly related to the way we work. I work in a lab that processes data from our sequencer but also external datasets, moreover, multiple people use my pipeline.
  So, with respect to conda this means: currently I have 5 envs, averaging 1.5 GB per env, this takes about 20 min to set up and it's quite some data to pull down. Especially since it is pulled down every time someone analyses data (sometime multiple times a day). This feels like quite a waste of resources. Also, I fix some package versions in the yaml files, but certainly not all (this may lead to issues when repos get reshuffled I read), so I also can't guarantee the exact environment long term (say over years, dependencies may change/update).
  So I'm considering switching to a central conda env which I version and of which I export lists of all packages installed for reproducibility reasons. This will then contain all tools.
  My question is, is there a nice way to do this? I had issues with putting a "source activate" line in every shell command (and how to do this for Python code under "run" anyway) before, see this: https://stackoverflow.com/questions/54024862/snakemake-conda-env-parameter-is-not-taken-from-config-yaml-file.
  So, is there a way, also when using a qsub cluster, to always use a specific conda env, for everything you do?
  
  user77: Inside your Snakefile you can provide the same conda: directive to each rule. If you install your pipeline in a central location and have everyone use the same conda when running it (hint: use a wrapper script that sets the path to conda) then everyone will be using the same environment. This has the added benefit that you can make this readable to everyone, but disallow them to make any modifications (so users can't mess things up).
  The tack we've taken is to have a basic conda env with snakemake and little else and then separate envs for various pipelines (and versions thereof if software versions in your YAML files change) or parts thereof, in cases where you have rules using tools with conflicting dependencies. This ends up being fairly convenient for more complicated pipelines, though if you can get away with just a single environment then finswimmer's answer is simpler.
  Note that all of this should work with any scheduler, since snakemake will insert the environment activation stuff in the script it submits to your scheduler.
  As an aside, this is exactly how snakePipes works.
  
  user1310: A good practice is to provide yml file along with your Snakefile. This file should contain all the enviroment definitions that are needed to run your pipeline. The minimum content of this file would be to install snakemake itself. Any other program used in your pipeline should be included as well. For reproducibility version numbers must be used.
  So, if someone would like to use your pipeline, (s)he first creates a new enviroment based on your definition:
  $ conda env create -f my_pipeline.yml
  
  Afterwards activate this environment and run the pipeline:
  $ conda activate my_pipeline
  $ snakemake
  
  The conda directive in your Snakefile is only neccessary if you need to use program which dependencies cannot be resolved within the main environment. If you start snakemake with the --use-conda parameter, snakemake will create a new environment based on the provided definition file and activate it for the rule.
  
  Especially since it is pulled down every time someone analyses data (sometime multiple times a day).
  
  The environment creation is only done once. If you run the pipeline a second time snakemake can activate the environment diretcly without downloading and installing something again.
  The tutorial I wrote some time ago on biostars about snakemake and conda might be useful for you.
  
  user1461: Using only the --use-conda directive, snakemake copies environments somewhere in the local .snakemake directory. This is repeated for each directory you run it in and replicates all environments effectively for each dataset.
  You can however tell snakemake to use a shared environment location using --conda-prefix as described here:
  
  Specify a directory in which the ‘conda’ and ‘conda-archive’
    directories are created. These are used to store conda environments
    and their archives, respectively. If not supplied, the value is set to
    the ‘.snakemake’ directory relative to the invocation directory. If
    supplied, the –use-conda flag must also be set. The value may be given
    as a relative path, which will be extrapolated to the invocation
    directory, or as an absolute path.
  
  This way you could only download/copy the environment once and share it, e.g. this way:
  snakemake --use-conda --conda-prefix /path/to/shared/environment
  
  
original_text: 3964: <p>I have been experimenting a lot lately with Snakemake, I love it. Recently I also switched to using conda (--use-conda) in the way that is advertised. However, I have some issues with it, mostly related to the way we work. I work in a lab that processes data from our sequencer but also external datasets, moreover, multiple people use my pipeline.</p>
  
  <p>So, with respect to conda this means: currently I have 5 envs, averaging 1.5 GB per env, this takes about 20 min to set up and it's quite some data to pull down. Especially since it is pulled down every time someone analyses data (sometime multiple times a day). This feels like quite a waste of resources. Also, I fix some package versions in the yaml files, but certainly not all (this may lead to issues when repos get reshuffled I read), so I also can't guarantee the exact environment long term (say over years, dependencies may change/update).</p>
  
  <p>So I'm considering switching to a central conda env which I version and of which I export lists of all packages installed for reproducibility reasons. This will then contain all tools.</p>
  
  <p>My question is, is there a nice way to do this? I had issues with putting a "source activate" line in every shell command (and how to do this for Python code under "run" anyway) before, see this: <a href="https://stackoverflow.com/questions/54024862/snakemake-conda-env-parameter-is-not-taken-from-config-yaml-file">https://stackoverflow.com/questions/54024862/snakemake-conda-env-parameter-is-not-taken-from-config-yaml-file</a>.</p>
  
  <p>So, is there a way, also when using a qsub cluster, to always use a specific conda env, for everything you do?</p>
   77: <p>Inside your Snakefile you can provide the same <code>conda:</code> directive to each rule. If you install your pipeline in a central location and have everyone use the same conda when running it (hint: use a wrapper script that sets the path to conda) then everyone will be using the same environment. This has the added benefit that you can make this readable to everyone, but disallow them to make any modifications (so users can't mess things up).</p>
  
  <p>The tack we've taken is to have a basic conda env with snakemake and little else and then separate envs for various pipelines (and versions thereof if software versions in your YAML files change) or parts thereof, in cases where you have rules using tools with conflicting dependencies. This ends up being fairly convenient for more complicated pipelines, though if you can get away with just a single environment then finswimmer's answer is simpler.</p>
  
  <p>Note that all of this should work with any scheduler, since snakemake will insert the environment activation stuff in the script it submits to your scheduler.</p>
  
  <p>As an aside, this is exactly how <a href="https://snakepipes.readthedocs.io/en/latest/" rel="nofollow noreferrer">snakePipes</a> works.</p>
   1310: <p>A good practice is to provide <code>yml</code> file along with your <code>Snakefile</code>. This file should contain all the enviroment definitions that are needed to run your pipeline. The minimum content of this file would be to install <code>snakemake</code> itself. Any other program used in your pipeline should be included as well. For reproducibility version numbers must be used.</p>
  
  <p>So, if someone would like to use your pipeline, (s)he first creates a new enviroment based on your definition:</p>
  
  <pre><code>$ conda env create -f my_pipeline.yml
  </code></pre>
  
  <p>Afterwards activate this environment and run the pipeline:</p>
  
  <pre><code><span class="math-container">$ conda activate my_pipeline
  $</span> snakemake
  </code></pre>
  
  <p>The <code>conda</code> directive in your <code>Snakefile</code> is only neccessary if you need to use program which dependencies cannot be resolved within the main environment. If you start snakemake with the <code>--use-conda</code> parameter, snakemake will create a new environment based on the provided definition file and activate it for the rule.</p>
  
  <blockquote>
    <p>Especially since it is pulled down every time someone analyses data (sometime multiple times a day).</p>
  </blockquote>
  
  <p>The environment creation is only done once. If you run the pipeline a second time <code>snakemake</code> can activate the environment diretcly without downloading and installing something again.</p>
  
  <p>The <a href="https://www.biostars.org/p/335903/" rel="nofollow noreferrer">tutorial</a> I wrote some time ago on biostars about <code>snakemake</code> and <code>conda</code> might be useful for you.</p>
   1461: <p>Using only the <code>--use-conda</code> directive, snakemake copies environments somewhere in the local <code>.snakemake</code> directory. This is repeated for each directory you run it in and replicates all environments effectively for each dataset.</p>
  
  <p>You can however tell snakemake to use a shared environment location using <code>--conda-prefix</code> as described <a href="https://snakemake.readthedocs.io/en/stable/executing/cli.html#CONDA" rel="nofollow noreferrer">here</a>:</p>
  
  <blockquote>
    <p>Specify a directory in which the ‘conda’ and ‘conda-archive’
    directories are created. These are used to store conda environments
    and their archives, respectively. If not supplied, the value is set to
    the ‘.snakemake’ directory relative to the invocation directory. If
    supplied, the –use-conda flag must also be set. The value may be given
    as a relative path, which will be extrapolated to the invocation
    directory, or as an absolute path.</p>
  </blockquote>
  
  <p>This way you could only download/copy the environment once and share it, e.g. this way:</p>
  
  <pre><code>snakemake --use-conda --conda-prefix /path/to/shared/environment
  </code></pre>
  