text: user3119: So I am trying to compute diversity indices for a dataset I have but the math doesn't work out and I don't really understand how to translate Shannon and/or Simpson diversity indices into something that is understandable. Maybe I got everything wrong, this is certainly a possibility. However, this is what I have up to now:
  # load libraries
  library("phyloseq")
  data("GlobalPatterns")
  
  # load data
  globp = GlobalPatterns
  
  # compute transformation according to link below
  simpson_transformed <- 1/(1-estimate_richness(globp, measures="Simpson"))
  shannon_transformed <- exp(estimate_richness(globp, measures="Shannon"))
  
  # print side by side
  print(cbind.data.frame(simpson_transformed, shannon_transformed))
  
  The comparing_indices columns are computed according to what I found at this link (italic is mine):
  
  They [i.e. the values] should be converted to effective number of species,
  which are the true diversities. By referring to Table 1 we see that
  species richness is already a true diversity, so the true diversity
  according to this index is 5.000 species. Table 1 shows that the
  Shannon entropy or Shannon-Wiener index is converted to effective
  number of species or true diversity by taking the exponential:
  exp(1.609) is 5.000, so the true diversity according to the Shannon
  entropy is also 5.000 species. The Gini-Simpson index is converted to
  a  true diversity by subtracting it from unity and inverting:
  1/(1-0.8) = 5.000 species also. So in fact all these indices agree
  that the diversity of this community is 5.000 species.
  
  I also verified how phyloseq is computing the indices. I believe the transformation for Shannon should be the same mentioned above. I have doubts about the Simpson, since they refer to the Gini-Simpson index and not the Simpson itself. However, the documentation for vegan's diversity() function, which is the function that (I believe) is called by phyloseq's estimate_richness() (see docs), mention that the Simpson index is returned as 1-D which seems to be the Gini-Simpson index (see the 1-lambda). In other words, it looks like the transformation I linked above should be exactly the one that are used in the code I provided. But, I was expecting the same values in the two columns. But they are not there and I wonder why this happens. What did I do wrong?
  
  user4282: You are trying to obtain Hill numbers, sometimes called true diversity, and you have different results because you have uneven species frequencies.
  Vegan calculates Shannon index like:
  $$ H' = -\sum_{i=1}^R p_i \ln p_i $$
  And Gini-Simpson like:
  $$ 1 - \lambda = 1 - \sum_{i=1}^R p_i^2 $$
  So the Hill numbers should be:
  $$ H1 = exp(H') $$
  $$ H2 = 1 / 1 - \lambda $$
  With Phyloseq:
  H1 <- 1 / estimate_richness(globp, measures="Simpson")
  H2 <- exp(estimate_richness(globp, measures="Shannon")
  
  
original_text: 3119: <p>So I am trying to compute diversity indices for a dataset I have but the math doesn't work out and I don't really understand how to translate Shannon and/or Simpson diversity indices into something that is understandable. Maybe I got everything wrong, this is certainly a possibility. However, this is what I have up to now:</p>
  <pre><code># load libraries
  library(&quot;phyloseq&quot;)
  data(&quot;GlobalPatterns&quot;)
  
  # load data
  globp = GlobalPatterns
  
  # compute transformation according to link below
  simpson_transformed &lt;- 1/(1-estimate_richness(globp, measures=&quot;Simpson&quot;))
  shannon_transformed &lt;- exp(estimate_richness(globp, measures=&quot;Shannon&quot;))
  
  # print side by side
  print(cbind.data.frame(simpson_transformed, shannon_transformed))
  </code></pre>
  <p>The <strong>comparing_indices</strong> columns are computed according to what I found <a href="http://www.loujost.com/Statistics%20and%20Physics/Diversity%20and%20Similarity/Diversity%20of%20a%20single%20community.htm" rel="nofollow noreferrer">at this link</a> (italic is mine):</p>
  <blockquote>
  <p>They <em>[i.e. the values]</em> should be converted to effective number of species,
  which are the true diversities. By referring to Table 1 we see that
  species richness is already a true diversity, so the true diversity
  according to this index is 5.000 species. Table 1 shows that the
  Shannon entropy or Shannon-Wiener index is converted to effective
  number of species or true diversity by taking the exponential:
  exp(1.609) is 5.000, so the true diversity according to the Shannon
  entropy is also 5.000 species. The Gini-Simpson index is converted to
  a  true diversity by subtracting it from unity and inverting:
  1/(1-0.8) = 5.000 species also. So in fact all these indices agree
  that the diversity of this community is 5.000 species.</p>
  </blockquote>
  <p>I also verified how <strong>phyloseq</strong> is computing the indices. I believe the transformation for Shannon should be the same mentioned above. I have doubts about the Simpson, since they refer to the Gini-Simpson index and not the Simpson itself. However, the <a href="https://www.rdocumentation.org/packages/vegan/versions/2.4-2/topics/diversity" rel="nofollow noreferrer">documentation</a> for <strong>vegan</strong>'s <code>diversity()</code> function, which is the function that (I believe) is called by <strong>phyloseq</strong>'s <code>estimate_richness()</code> (see <a href="https://www.rdocumentation.org/packages/phyloseq/versions/1.16.2/topics/estimate_richness" rel="nofollow noreferrer">docs</a>), mention that the Simpson index is returned as <code>1-D</code> <a href="https://en.wikipedia.org/wiki/Diversity_index#Gini%E2%80%93Simpson_index" rel="nofollow noreferrer">which seems to be the Gini-Simpson index</a> (see the <code>1-lambda</code>). In other words, it looks like the transformation I linked above should be exactly the one that are used in the code I provided. But, I was expecting the same values in the two columns. But they are not there and I wonder why this happens. What did I do wrong?</p>
   4282: <p>You are trying to obtain Hill numbers, sometimes called true diversity, and you have different results because you have uneven species frequencies.</p>
  <p><code>Vegan</code> calculates Shannon index like:</p>
  <p><span class="math-container">$$ H' = -\sum_{i=1}^R p_i \ln p_i $$</span></p>
  <p>And Gini-Simpson like:</p>
  <p><span class="math-container">$$ 1 - \lambda = 1 - \sum_{i=1}^R p_i^2 $$</span></p>
  <p>So the Hill numbers should be:</p>
  <p><span class="math-container">$$ H1 = exp(H') $$</span></p>
  <p><span class="math-container">$$ H2 = 1 / 1 - \lambda $$</span></p>
  <p>With <code>Phyloseq</code>:</p>
  <pre><code>H1 &lt;- 1 / estimate_richness(globp, measures=&quot;Simpson&quot;)
  H2 &lt;- exp(estimate_richness(globp, measures=&quot;Shannon&quot;)
  </code></pre>
  