text: user48: I have perform an enrichment analysis to a cluster of genes. The output is a list of pathways and their p-value (the pathways are selected because p-value < 0.05). The list is still quite long, so I want to reduce it. For that purpose I have a calculated the Dice coefficient of the pathways in a matrix $p$x$p$ where $p$ is the number of pathways in the list. I want both the ones that are more different (they overlap less, their Dice coefficient is lower) and the pathways more representative of the most similar pathways (So if a there is a group of 5 pathways that overlap over 0.8 take just one).
  How can I select the most representatives pathways? 
  There is a similar tool for GO but it relays on discarding not significant GO, while here all the initial pathways are already significant. 
  If I do a clustering of the pathways using the Dice coefficient matrix I don't know where (or how) to cut.
  
  I tried using the height to select the pathways. But I am unsure of the interpretation of height. 
  Some other tools I have seen use a multidimensional scaling plot, but I am not sure if performing it and cutting at certain point of the first dimension would help.
  
  
  user163: If you're happy with a more confident ranking of the most representative gene sets, rather than necessarily cutting down the list, you might try EGSEA. It uses an ensemble approach to give a ranking of the most relevant gene sets, and also produces an interactive HTML output with statistics, heatmaps, pathway maps, summary plots and GO graphs which allows you to examine the output at varying levels of granularity. 
  You can read the paper on bioRxiv or download the package from Bioconductor.
  
  user939: One way to group similar significant pathways, is to quantify how many genes overlap between the pathways, and then use this in clustering (heatmap). I have made a tool in R which calculates the overlap index between GO terms and subsequently clusters them in a heatmap. Overlap index is the fraction of genes that overlap (number between 0-1). Also Pearson correlation can be used for clustering instead of the overlap index. My package (gogadget) works only with  goseq analysis, but you can use goseq also for reactome or kegg data.
  I have used this overlap clustering approach for different data sets now, and usually we can reduce 200-300 GO terms into 10-20 functional groups.
  
  user73: This sounds like something that might be amenable to a clustered heatmap plot, or a correlation matrix plot, or something similar. Have you looked at a correlation matrix of the dice coefficient matrix (or maybe just a heatmap plot of that matrix without the correlation matrix)?
  The corrplot package looks like it might be useful, in particular the hclust / drawing rectangles presentation.
  I can't vouch for this package though; it's just something I found by a search for "R plot correlation matrix".
  
original_text: 48: <p>I have perform an enrichment analysis to a cluster of genes. The output is a list of pathways and their p-value (the pathways are selected because p-value &lt; 0.05). The list is still quite long, so I want to reduce it. For that purpose I have a calculated the Dice coefficient of the pathways in a matrix $p$x$p$ where $p$ is the number of pathways in the list. I want both the ones that are more different (they overlap less, their Dice coefficient is lower) and the pathways more representative of the most similar pathways (So if a there is a group of 5 pathways that overlap over 0.8 take just one).</p>
  
  <p>How can I select the most representatives pathways? </p>
  
  <p>There is a similar <a href="http://revigo.irb.hr/" rel="nofollow noreferrer">tool</a> for GO but it relays on discarding not significant GO, while here all the initial pathways are already significant. </p>
  
  <p>If I do a clustering of the pathways using the Dice coefficient matrix I don't know where (or how) to cut.</p>
  
  <p><a href="https://i.stack.imgur.com/mMdEH.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/mMdEH.png" alt="circular dendrogara"></a></p>
  
  <p>I tried using the height to select the pathways. But I am unsure of the interpretation of height. </p>
  
  <p>Some other tools I have seen use a multidimensional scaling plot, but I am not sure if performing it and cutting at certain point of the first dimension would help.
  <a href="https://i.stack.imgur.com/CeetJ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/CeetJ.png" alt="MDS plot"></a></p>
   163: <p>If you're happy with a more confident ranking of the most representative gene sets, rather than necessarily cutting down the list, you might try EGSEA. It uses an ensemble approach to give a ranking of the most relevant gene sets, and also produces an interactive HTML output with statistics, heatmaps, pathway maps, summary plots and GO graphs which allows you to examine the output at varying levels of granularity. </p>
  
  <p>You can read the paper on <a href="http://biorxiv.org/content/early/2016/07/08/042580" rel="nofollow noreferrer">bioRxiv</a> or download the package from <a href="https://bioconductor.org/packages/release/bioc/html/EGSEA.html" rel="nofollow noreferrer">Bioconductor</a>.</p>
   939: <p>One way to group similar significant pathways, is to quantify how many genes overlap between the pathways, and then use this in clustering (heatmap). I have made a tool in R which calculates the overlap index between GO terms and subsequently clusters them in a heatmap. Overlap index is the fraction of genes that overlap (number between 0-1). Also Pearson correlation can be used for clustering instead of the overlap index. My package (<a href="https://sourceforge.net/projects/gogadget/" rel="nofollow noreferrer">gogadget</a>) works only with  goseq analysis, but you can use <a href="https://bioconductor.org/packages/release/bioc/html/goseq.html" rel="nofollow noreferrer">goseq</a> also for reactome or kegg data.</p>
  
  <p>I have used this overlap clustering approach for different data sets now, and usually we can reduce 200-300 GO terms into 10-20 functional groups.</p>
   73: <p>This sounds like something that might be amenable to a clustered heatmap plot, or a correlation matrix plot, or something similar. Have you looked at a correlation matrix of the dice coefficient matrix (or maybe just a heatmap plot of that matrix without the correlation matrix)?</p>
  
  <p>The <code>corrplot</code> package looks like it might be useful, in particular <a href="https://cran.r-project.org/web/packages/corrplot/vignettes/corrplot-intro.html" rel="nofollow noreferrer" title="corrplot vignette">the <code>hclust</code> / drawing rectangles presentation</a>.</p>
  
  <p>I can't vouch for this package though; it's just something I found by a search for "R plot correlation matrix".</p>
  