text: user3425: I am currently working on a dataset that contains 50 samples (10 samples * 5 blocks). The features of the date set are:
  
  The data is perfectly balanced between blocks, with equal treatment representation in each block. Each block contains 2 control samples (CTL) that are pools of all other treatment samples. These are technical replicates that are identical across multiplexes.
  
  There is a lot of missing data (~40%),
  
  most of the missing data is missing by block, meaning that the protein is present in some blocks, but not others.
  
  When looking at the expression of these missing proteins in the other blocks, they tend to be around the median value for the experiment, which suggests to me that these values are missing completely at random.
  
  
  I could proceed using complete cases, but then I'm missing out on a large portion of data. I am considering using an imputation method, but I'm concerned that (due to the nature of the missing data) that I will just be introducing a block effect into the data.
  Any input on potential methods for imputation would be greatly appreciated.
  Experimental design:
  SampleNumber    Multiplex   Treatment
  1   A   CTL
  2   A   CTL
  3   A   TMT1
  4   A   TMT1
  5   A   TMT2
  6   A   TMT2
  7   A   TMT3
  8   A   TMT3
  9   A   TMT4
  10  A   TMT4
  11  B   CTL
  12  B   CTL
  13  B   TMT1
  14  B   TMT1
  15  B   TMT2
  16  B   TMT2
  17  B   TMT3
  18  B   TMT3
  19  B   TMT4
  20  B   TMT4
  21  C   CTL
  22  C   CTL
  23  C   TMT1
  24  C   TMT1
  25  C   TMT2
  26  C   TMT2
  27  C   TMT3
  28  C   TMT3
  29  C   TMT4
  30  C   TMT4
  31  D   CTL
  32  D   CTL
  33  D   TMT1
  34  D   TMT1
  35  D   TMT2
  36  D   TMT2
  37  D   TMT3
  38  D   TMT3
  39  D   TMT4
  40  D   TMT4
  41  E   CTL
  42  E   CTL
  43  E   TMT1
  44  E   TMT1
  45  E   TMT2
  46  E   TMT2
  47  E   TMT3
  48  E   TMT3
  49  E   TMT4
  50  E   TMT4
  
  The data is in a matrix format, columns = samples and rows = individual proteins, similar to gene expression.
  The controls are technical replicates, but they are pools of all other 40 samples. This is for TMT proteomics, where the pools are used by the MS to identify peptides across the multiplexes, not necessarily for multiplex correction. I've included multiplex (as well as treatment) in the linear model to identify changes associated with the treatment, while removing the effect of the multiplex. I have confirmed (with PCA and PVCA) that the multiplex effect is negligible after including it in the lm. My problem at the moment is the missing data.
  I calculated the missing percentage from the data matrix by taking the number of missing values divided by the total number of potential values. he data is in a matrix format, columns = samples and rows = individual proteins, similar to gene expression. To calculate the missing values, I divided the total number of NA's by the total number of values in the matrix (number of samples * number of proteins). Missing data is common in proteomics, but normally this can be attributed to something, most often being close to the limit of detection. But that is not the case here, if a value is missing, it is missing in the whole multiplex, including the pool, where we know it should be found.
  The missing values in this case are random, and this is definitely due to the MS. It will select the first ~65k peptides that it identifies to trigger an additional MS event, so if it doesn't see that first peptide, it will be missing from the entire multiplex. There are methods for imputation of data that is missing at random, but I'm concerned that it will just be imputing by multiplex.
  
  user3848: 40% missing data is huge. Missing data analysis is complicated on the underlying distribution. If the data set is periodic then missing data periodicity is needed. Non-periodic data can be solved using regression, I think SciPy has an automated method for this, but the volume of missing data makes this approach complex. If you can bring the missing data down to 20% by compartmenatlising the data set that would be the starting point.
  
  user9197: This looks like a TMT proteomics experiment. What tools/pipeline did you use to arrive at your quantitative values?
  There are a variety of tools that are useful for analyzing this type of data and supply appropriate imputation methods. If you are comfortable programming in R, one I would suggest is the MSstatsTMT R package: http://msstats.org/msstatstmt/
  That being said, any ways you can improve the upstream analysis to decrease the amount of missing values will be extremely beneficial.
  
original_text: 3425: <p>I am currently working on a dataset that contains 50 samples (10 samples * 5 blocks). The features of the date set are:</p>
  <ol>
  <li><p>The data is perfectly balanced between blocks, with equal treatment representation in each block. Each block contains 2 control samples (CTL) that are pools of all other treatment samples. These are technical replicates that are identical across multiplexes.</p>
  </li>
  <li><p>There is a lot of missing data (~40%),</p>
  </li>
  <li><p>most of the missing data is missing by block, meaning that the protein is present in some blocks, but not others.</p>
  </li>
  <li><p>When looking at the expression of these missing proteins in the other blocks, they tend to be around the median value for the experiment, which suggests to me that these values are missing completely at random.</p>
  </li>
  </ol>
  <p>I could proceed using complete cases, but then I'm missing out on a large portion of data. I am considering using an imputation method, but I'm concerned that (due to the nature of the missing data) that I will just be introducing a block effect into the data.</p>
  <p>Any input on potential methods for imputation would be greatly appreciated.</p>
  <p>Experimental design:</p>
  <pre><code>SampleNumber    Multiplex   Treatment
  1   A   CTL
  2   A   CTL
  3   A   TMT1
  4   A   TMT1
  5   A   TMT2
  6   A   TMT2
  7   A   TMT3
  8   A   TMT3
  9   A   TMT4
  10  A   TMT4
  11  B   CTL
  12  B   CTL
  13  B   TMT1
  14  B   TMT1
  15  B   TMT2
  16  B   TMT2
  17  B   TMT3
  18  B   TMT3
  19  B   TMT4
  20  B   TMT4
  21  C   CTL
  22  C   CTL
  23  C   TMT1
  24  C   TMT1
  25  C   TMT2
  26  C   TMT2
  27  C   TMT3
  28  C   TMT3
  29  C   TMT4
  30  C   TMT4
  31  D   CTL
  32  D   CTL
  33  D   TMT1
  34  D   TMT1
  35  D   TMT2
  36  D   TMT2
  37  D   TMT3
  38  D   TMT3
  39  D   TMT4
  40  D   TMT4
  41  E   CTL
  42  E   CTL
  43  E   TMT1
  44  E   TMT1
  45  E   TMT2
  46  E   TMT2
  47  E   TMT3
  48  E   TMT3
  49  E   TMT4
  50  E   TMT4
  </code></pre>
  <p>The data is in a matrix format, columns = samples and rows = individual proteins, similar to gene expression.</p>
  <p>The controls are technical replicates, but they are pools of all other 40 samples. This is for TMT proteomics, where the pools are used by the MS to identify peptides across the multiplexes, not necessarily for multiplex correction. I've included multiplex (as well as treatment) in the linear model to identify changes associated with the treatment, while removing the effect of the multiplex. I have confirmed (with PCA and PVCA) that the multiplex effect is negligible after including it in the lm. My problem at the moment is the missing data.</p>
  <p>I calculated the missing percentage from the data matrix by taking the number of missing values divided by the total number of potential values. he data is in a matrix format, columns = samples and rows = individual proteins, similar to gene expression. To calculate the missing values, I divided the total number of NA's by the total number of values in the matrix (number of samples * number of proteins). Missing data is common in proteomics, but normally this can be attributed to something, most often being close to the limit of detection. But that is not the case here, if a value is missing, it is missing in the whole multiplex, including the pool, where we know it should be found.</p>
  <p>The missing values in this case are random, and this is definitely due to the MS. It will select the first ~65k peptides that it identifies to trigger an additional MS event, so if it doesn't see that first peptide, it will be missing from the entire multiplex. There are methods for imputation of data that is missing at random, but I'm concerned that it will just be imputing by multiplex.</p>
   3848: <p>40% missing data is huge. Missing data analysis is complicated on the underlying distribution. If the data set is periodic then missing data periodicity is needed. Non-periodic data can be solved using regression, I think <code>SciPy</code> has an automated method for this, but the volume of missing data makes this approach complex. If you can bring the missing data down to 20% by compartmenatlising the data set that would be the starting point.</p>
   9197: <p>This looks like a TMT proteomics experiment. What tools/pipeline did you use to arrive at your quantitative values?</p>
  <p>There are a variety of tools that are useful for analyzing this type of data and supply appropriate imputation methods. If you are comfortable programming in R, one I would suggest is the MSstatsTMT R package: <a href="http://msstats.org/msstatstmt/" rel="nofollow noreferrer">http://msstats.org/msstatstmt/</a></p>
  <p>That being said, any ways you can improve the upstream analysis to decrease the amount of missing values will be extremely beneficial.</p>
  