text: user208: I have a gene expression count matrix produced from bulk RNA-seq data. I'd like to find genes that were not expressed in a group of samples and were expressed in another group.
  The problem of course is that not all effectively non-expressed genes will have 0 counts due to sequencing errors, or because they were expressed in a small subset of cells.
  I'm interested in solutions using R.
  
  user235: Depending on how much effort you wish to put into this, here is one suggestion I have see used before (uses more than just R). Steps 1-4 come from this paper (see supplimentary material section "Calculation of per gene Local FDR").
  
  Take your gene models and collapse the introns down to maybe 100bp
  Thus if we have a gene with two exons and a 1kb intron eg. exons (1000, 1100) and (2000,2100), we reduce the intron size so that the exons are (1000,1100) and (1200,1300). This is because we need to find sufficient gene free space to fit the null gene in.    
  Shift each gene into the nearest genomic space at least 5 kp from a genomic annotation and free of ESTs.
  Calculate the FPKM distribution of this shifted null set 
  From these two distributions it should be possible to calculate a local FDR for expression.
  You will need to use a "mixing proportion" to do this. The above reference used Qvality to do this.
  Carry out differential expression analysis as described in Konrad's answer
  Subset the differentially expressed genes to only consider those that are not expressed in the control condition
  You might want to also recalculate the FDRs for differential expression as you are only considering a subset of the tests you might have. 
  
  
  user77: A common method is to use zFPKMs, which you can find implemented in R here.
  Having said that, there's an inherent problem in declaring a difference between two things on either side of a given threshold. Given that, I would encourage you to use more than "expressed in one and not in another", likely adding at least a "with a minimal difference of X between them" metric. You may also find the tau metric useful, which is implemented in R here. This is meant to measure tissue-specificity, which is more akin to what you're probably interested in doing.
  
  user29: 
  I'd like to find genes that were not expressed in a group of samples and were expressed in another group.
  
  This is, fundamentally, a differential expression analysis, with a twist. To solve this, you’d first use a differential expression library of your choice (e.g. DESeq2) and perform a one-tailed test of differential expression.
  Briefly, you’d perform the normal setup and then use
  results(dds, altHypothesis = 'greater')
  
  To perform a one-tailed test. This will give you only those genes that are significantly upregulated in one group. Check chapter 3.9 of the vignette for details.
  Of course this won’t tell you that the genes are unexpressed in the other group. Unfortunately I don’t know of a good value to threshold the results; I would start by plotting a histogram of the (variance stabilised) expression values in your first group, and then visually choose an expression threshold that cleanly separates genes that are clearly expressed from zeros:
  vst_counts = assay(vst(dds))
  dens = density(vst_counts[, replicate])
  plot(dens, log = 'y')
  
  (This merges the replicates in the group, which should be fine.)
  Counts follow a multimodal distribution, with one mode for unexpressed and one or more for expressed genes. The expression threshold can be set somewhere between the clearly unexpressed and expressed peaks:
  
  Here I used identify(dens) to identify the threshold interactively but you could also use an analytical method:
  threshold = identify(dens)
  quantile = sum(dens$x < dens$x[threshold]) / length(dens$x)
  
  # Using just one replicate here; more robust would be to use a mean value.
  nonzero_counts = counts(dds, normalized = TRUE)[, replicates[1]]
  nonzero_counts = nonzero_counts[nonzero_counts > 0]
  
  (expression_threshold = quantile(nonzero_counts, probs = quantile))
  
  26.5625%
  4.112033
  
  
original_text: 208: <p>I have a gene expression count matrix produced from bulk RNA-seq data. I'd like to find genes that were <strong>not</strong> expressed in a group of samples and were expressed in another group.</p>
  
  <p>The problem of course is that not all <em>effectively</em> non-expressed genes will have 0 counts due to sequencing errors, or because they were expressed in a small subset of cells.</p>
  
  <p>I'm interested in solutions using <code>R</code>.</p>
   235: <p>Depending on how much effort you wish to put into this, here is one suggestion I have see used before (uses more than just R). Steps 1-4 come from <a href="http://genome.cshlp.org/content/24/12/1918.long#ref-55" rel="nofollow noreferrer">this paper</a> (see supplimentary material section "Calculation of per gene Local FDR").</p>
  
  <ol>
  <li><p><em>Take your gene models and collapse the introns down to maybe 100bp</em><br>
  Thus if we have a gene with two exons and a 1kb intron eg. exons (1000, 1100) and (2000,2100), we reduce the intron size so that the exons are (1000,1100) and (1200,1300). This is because we need to find sufficient gene free space to fit the null gene in.    </p></li>
  <li><p><em>Shift each gene into the nearest genomic space at least 5 kp from a genomic annotation and free of ESTs.</em></p></li>
  <li><p><em>Calculate the FPKM distribution of this shifted null set</em>  </p></li>
  <li><p><em>From these two distributions it should be possible to calculate a <a href="https://cran.r-project.org/web/packages/locfdr/vignettes/locfdr-example.pdf" rel="nofollow noreferrer">local FDR</a> for expression.</em><br>
  You will need to use a "mixing proportion" to do this. The above reference used <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2660870/" rel="nofollow noreferrer">Qvality</a> to do this.</p></li>
  <li><p><em>Carry out differential expression analysis as described in <a href="https://bioinformatics.stackexchange.com/a/690/235">Konrad's answer</a></em></p></li>
  <li><p><em>Subset the differentially expressed genes to only consider those that are not expressed in the control condition</em></p></li>
  <li><p><em>You might want to also recalculate the FDRs for differential expression as you are only considering a subset of the tests you might have.</em> </p></li>
  </ol>
   77: <p>A common method is to use <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3870982/" rel="noreferrer">zFPKMs</a>, which you can find implemented in R <a href="https://github.com/severinEvo/gene_expression/blob/master/zFPKM.R" rel="noreferrer">here</a>.</p>
  
  <p>Having said that, there's an inherent problem in declaring a difference between two things on either side of a given threshold. Given that, I would encourage you to use more than "expressed in one and not in another", likely adding at least a "with a minimal difference of X between them" metric. You may also find the tau metric useful, which is implemented in R <a href="https://github.com/severinEvo/gene_expression/blob/master/tau.R" rel="noreferrer">here</a>. This is meant to measure tissue-specificity, which is more akin to what you're probably interested in doing.</p>
   29: <blockquote>
    <p>I'd like to find genes that were not expressed in a group of samples and were expressed in another group.</p>
  </blockquote>
  
  <p>This is, fundamentally, a differential expression analysis, with a twist. To solve this, you’d first use a differential expression library of your choice (e.g. DESeq2) and perform a one-tailed test of differential expression.</p>
  
  <p>Briefly, you’d perform the normal setup and then use</p>
  
  <pre><code>results(dds, altHypothesis = 'greater')
  </code></pre>
  
  <p>To perform a one-tailed test. This will give you only those genes that are significantly upregulated in one group. Check chapter 3.9 of the vignette for details.</p>
  
  <p>Of course this won’t tell you that the genes are <em>unexpressed</em> in the other group. Unfortunately I don’t know of a good value to threshold the results; I would start by plotting a histogram of the (variance stabilised) expression values in your first group, and then visually choose an expression threshold that cleanly separates genes that are clearly expressed from zeros:</p>
  
  <pre><code>vst_counts = assay(vst(dds))
  dens = density(vst_counts[, replicate])
  plot(dens, log = 'y')
  </code></pre>
  
  <p>(This merges the replicates in the group, which should be fine.)</p>
  
  <p>Counts follow a multimodal distribution, with one mode for unexpressed and one or more for expressed genes. The expression threshold can be set somewhere between the clearly unexpressed and expressed peaks:</p>
  
  <p><a href="https://i.stack.imgur.com/MF0vy.png" rel="noreferrer"><img src="https://i.stack.imgur.com/MF0vy.png" alt="enter image description here"></a></p>
  
  <p>Here I used <code>identify(dens)</code> to identify the threshold interactively but you could also use an analytical method:</p>
  
  <pre><code>threshold = identify(dens)
  quantile = sum(dens$x &lt; dens$x[threshold]) / length(dens$x)
  
  # Using just one replicate here; more robust would be to use a mean value.
  nonzero_counts = counts(dds, normalized = TRUE)[, replicates[1]]
  nonzero_counts = nonzero_counts[nonzero_counts &gt; 0]
  
  (expression_threshold = quantile(nonzero_counts, probs = quantile))
  </code></pre>
  
  
  
  <pre><code>26.5625%
  4.112033
  </code></pre>
  