text: user4783: My data = data
  Model   A1BG    A1CF    A2M A2ML1   A3GALT2 A4GALT  foldchange
  IM00499 N   L   L   G   N   N   0.285744
  IM00499.1   L   N   D   G   D   D   0.005046
  IM00980 N   N   N   N   N   N   -0.068949
  IM01529 N   N   L   N   N   N   0.065914
  IM01624 A   N   L   N   N   N   0.302100
  IM01240 N   D   N   D   N   G   -0.428324
  IM00807 N   L   A   N   D   N   0.257222
  IM00999 N   N   N   N   N   D   0.056268
  IM00983 N   N   N   N   N   N   0.107456
  IM01076 N   N   D   A   N   N   0.218301
  IM00803 N   D   N   N   N   D   -0.479368
  IM00125 N   N   A   N   N   A   0.270428
  IM00998 D   N   N   N   G   N   0.142748
  IM00997 N   L   G   G   N   G   -0.154540
  IM00996 N   L   N   N   N   L   -1.302023
  
  The above data frame is divided into 5 categories = A, N, L, G and D
  feature = ['A1BG', 'A1CF', 'A2M', 'A2ML1', 'A3GALT2', 'A4GALT']
  Feature_level_partitions are arrays of combination of classification values
  feature_level_partitions = ['A', 'N' , 'L',  'G',  'D'], ['D'], ['L', 'G']
  I was using the below list comprehension to calculate Hedge's G.
          hedges_g = np.array(
              [
                  Hedges_g(data[feature].isin(feature_level_partition), y)
                  for feature_level_partition in feature_level_partitions
              ],
              dtype=np.float64,
          )
  
  Hedges_g is defined as follows:
  def Hedges_g(x, y):
      difference_of_means = abs(np.mean(y[x])[0] - np.mean(y[~x])[0])
      s_pooled_weighted = math.sqrt((((n1 - 1) * s1 ** 2) + ((n2 - 1) * s2 ** 2)) / (n1 + n2 - 2))
      return difference_of_means / s_pooled_weighted
  
  I am interested in knowing how many models there are within each category. So for example if I divide up the data into ["D","L"] on one side, and the rest on the other, I want to know how many models have either "D" OR "L" and how many models have any of the other categories and then don't calculate Hegde's G if the number on either side is less than 2.
  To answer the above query I tried the following code
  [data[feature].isin(feature_level_partition).sum()
      for feature_level_partition in feature_level_partitions]
  
  The above code gave me the number of models with the feature_level_partitions
  However I also want to know the number of models without the feature_level_partitions
  I also want to create a python function which calculates both above numbers in one line and fit this code as a if statement in my above list comprehension function
  
  user3848: What I'm not clear about is the definition of y - thats crucial to understanding what the code does. Just to note, although you've called the dataframe data, I've called it df (traditional shorthand). In terms of speed there will not be much in it between vectorised pandas and numpy. Explicitly using iterrows or else apply is when it slows down a lot.
  Possibilities of the definition of y.
  1.
  y = df.drop(['Model', 'foldchange'], axis=1)
  
  I think the above is more likely, or ...
  2.
  y = df['fold change']
  
  What I think is occurring is np.mean is performing a boolean operation on the feature columns. What I didn't understand though was this ...
  abs(np.mean(y[x])[0] - np.mean(y[~x])[0])
  
  .. this is because for example model 'IM00980' albeit the mean is 0 this will always be 1 under the above equation. This is because y[~x] will be 1 for all values and thus abs(0 - 1) is 1. Equally if all columns in a row were 1 the output would also be 1 because abs(1-0) is 1. Anyway, what I would do is explicitly perform the operation within pandas rather than coercing np into a boolean operation. This is
  import pandas as pd
  
  df = pd.read_csv(r'/pathtofile/megha.csv', delimiter="\t")
  
  feature = ['A1BG', 'A1CF', 'A2M', 'A2ML1', 'A3GALT2', 'A4GALT']
  df=df.drop(['foldchange'],axis=1)
  feature_level_partitions = ['L', 'G']
  filtered = df[feature].isin(feature_level_partitions).eq(True).mul(1)
  df2 = df['Model'].to_frame().join(filtered)
  df2['mean'] = df2[feature].mean(axis=1)
  print(df2)
  
  Here ~x would simply 1 - mean as a new pandas column, i.e. df2['~mean'] = df2['mean'] - 1
  The output of that code is:
  
  
  
  
  -
  Model
  A1BG
  A1CF
  A2M
  A2ML1
  A3GALT2
  A4GALT
  mean.
  
  
  
  
  0
  IM00499
  0
  1.
  1.
  1.
  0.
  0.
  0.500000
  
  
  1
  IM00499.1
  1
  0
  0
  1
  0
  0
  0.333333
  
  
  2
  IM00980
  0
  0
  0
  0
  0
  0
  0.000000
  
  
  3
  IM01529
  0
  0
  1
  0
  0
  0
  0.166667
  
  
  4
  IM01624
  0
  0
  1
  0
  0
  0
  0.166667
  
  
  5
  IM01240
  0
  0
  0
  0
  0
  1
  0.166667
  
  
  6
  IM00807
  0
  1
  0
  0
  0
  0
  0.166667
  
  
  7
  IM00999
  0
  0
  0
  0
  0
  0
  0.000000
  
  
  8
  IM00983
  0
  0
  0
  0
  0
  0
  0.000000
  
  
  9
  IM01076
  0
  0
  0
  0
  0
  0
  0.000000
  
  
  10
  IM00803
  0
  0
  0
  0
  0
  0
  0.000000
  
  
  11
  IM00125
  0
  0
  0
  0
  0
  0
  0.000000
  
  
  12
  IM00998
  0
  0
  0
  0
  1
  0
  0.166667
  
  
  13
  IM00997
  0
  1
  1
  1
  0
  1
  0.666667
  
  
  14
  IM00996
  0
  1
  0
  0
  0
  1
  0.333333
  
  
  
  
  If this is correct it provides the basis to move forward for all other operations. Basically, what I'm saying is I think is good to break open the numpy into explicit pandas operations, what is happening is there's a mix of pandas and numpy and I would recommend going down one route or the other.
  
original_text: 4783: <p>My data = <code>data</code></p>
  <pre><code>Model   A1BG    A1CF    A2M A2ML1   A3GALT2 A4GALT  foldchange
  IM00499 N   L   L   G   N   N   0.285744
  IM00499.1   L   N   D   G   D   D   0.005046
  IM00980 N   N   N   N   N   N   -0.068949
  IM01529 N   N   L   N   N   N   0.065914
  IM01624 A   N   L   N   N   N   0.302100
  IM01240 N   D   N   D   N   G   -0.428324
  IM00807 N   L   A   N   D   N   0.257222
  IM00999 N   N   N   N   N   D   0.056268
  IM00983 N   N   N   N   N   N   0.107456
  IM01076 N   N   D   A   N   N   0.218301
  IM00803 N   D   N   N   N   D   -0.479368
  IM00125 N   N   A   N   N   A   0.270428
  IM00998 D   N   N   N   G   N   0.142748
  IM00997 N   L   G   G   N   G   -0.154540
  IM00996 N   L   N   N   N   L   -1.302023
  </code></pre>
  <p>The above data frame is divided into 5 categories = A, N, L, G and D</p>
  <p><code>feature = ['A1BG', 'A1CF', 'A2M', 'A2ML1', 'A3GALT2', 'A4GALT']</code></p>
  <p>Feature_level_partitions are arrays of combination of classification values</p>
  <p><code>feature_level_partitions = ['A', 'N' , 'L',  'G',  'D'], ['D'], ['L', 'G']</code></p>
  <p>I was using the below <code>list comprehension</code> to calculate <code>Hedge's G</code>.</p>
  <pre><code>        hedges_g = np.array(
              [
                  Hedges_g(data[feature].isin(feature_level_partition), y)
                  for feature_level_partition in feature_level_partitions
              ],
              dtype=np.float64,
          )
  </code></pre>
  <p><code>Hedges_g</code> is defined as follows:</p>
  <pre><code>def Hedges_g(x, y):
      difference_of_means = abs(np.mean(y[x])[0] - np.mean(y[~x])[0])
      s_pooled_weighted = math.sqrt((((n1 - 1) * s1 ** 2) + ((n2 - 1) * s2 ** 2)) / (n1 + n2 - 2))
      return difference_of_means / s_pooled_weighted
  </code></pre>
  <p>I am interested in knowing how many models there are within each category. So for example if I divide up the data into [&quot;D&quot;,&quot;L&quot;] on one side, and the rest on the other, I want to know how many models have either &quot;D&quot; OR &quot;L&quot; and how many models have any of the other categories and then don't calculate <code>Hegde's G</code> if the number on either side is less than 2.</p>
  <p>To answer the above query I tried the following code</p>
  <pre><code>[data[feature].isin(feature_level_partition).sum()
      for feature_level_partition in feature_level_partitions]
  </code></pre>
  <p>The above code gave me the number of models with the <code>feature_level_partitions</code>
  However I also want to know the number of models without the <code>feature_level_partitions</code></p>
  <p>I also want to create a python function which calculates both above numbers in one line and fit this code as a <code>if statement</code> in my above <code>list comprehension</code> function</p>
   3848: <p>What I'm not clear about is the definition of <code>y</code> - thats crucial to understanding what the code does. Just to note, although you've called the dataframe <code>data</code>, I've called it <code>df</code> (traditional shorthand). In terms of speed there will not be much in it between vectorised <code>pandas</code> and <code>numpy</code>. Explicitly using <code>iterrows</code> or else <code>apply</code> is when it slows down a lot.</p>
  <p>Possibilities of the definition of <code>y</code>.</p>
  <p>1.</p>
  <pre><code>y = df.drop(['Model', 'foldchange'], axis=1)
  </code></pre>
  <p>I think the above is more likely, or ...</p>
  <p>2.</p>
  <pre><code>y = df['fold change']
  </code></pre>
  <p>What I think is occurring is <code>np.mean</code> is performing a boolean operation on the feature columns. What I didn't understand though was this ...</p>
  <pre><code>abs(np.mean(y[x])[0] - np.mean(y[~x])[0])
  </code></pre>
  <p>.. this is because for example model 'IM00980' albeit the mean is 0 this will always be 1 under the above equation. This is because <code>y[~x]</code> will be 1 for all values and thus abs(0 - 1) is 1. Equally if all columns in a row were 1 the output would also be 1 because abs(1-0) is 1. Anyway, what I would do is explicitly perform the operation within <code>pandas</code> rather than coercing <code>np</code> into a boolean operation. This is</p>
  <pre><code>import pandas as pd
  
  df = pd.read_csv(r'/pathtofile/megha.csv', delimiter=&quot;\t&quot;)
  
  feature = ['A1BG', 'A1CF', 'A2M', 'A2ML1', 'A3GALT2', 'A4GALT']
  df=df.drop(['foldchange'],axis=1)
  feature_level_partitions = ['L', 'G']
  filtered = df[feature].isin(feature_level_partitions).eq(True).mul(1)
  df2 = df['Model'].to_frame().join(filtered)
  df2['mean'] = df2[feature].mean(axis=1)
  print(df2)
  </code></pre>
  <p>Here ~x would simply <code>1 - mean</code> as a new pandas column, i.e. <code>df2['~mean'] = df2['mean'] - 1</code></p>
  <p>The output of that code is:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>-</th>
  <th>Model</th>
  <th>A1BG</th>
  <th>A1CF</th>
  <th>A2M</th>
  <th>A2ML1</th>
  <th>A3GALT2</th>
  <th>A4GALT</th>
  <th>mean.</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>0</td>
  <td>IM00499</td>
  <td>0</td>
  <td>1.</td>
  <td>1.</td>
  <td>1.</td>
  <td>0.</td>
  <td>0.</td>
  <td>0.500000</td>
  </tr>
  <tr>
  <td>1</td>
  <td>IM00499.1</td>
  <td>1</td>
  <td>0</td>
  <td>0</td>
  <td>1</td>
  <td>0</td>
  <td>0</td>
  <td>0.333333</td>
  </tr>
  <tr>
  <td>2</td>
  <td>IM00980</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0.000000</td>
  </tr>
  <tr>
  <td>3</td>
  <td>IM01529</td>
  <td>0</td>
  <td>0</td>
  <td>1</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0.166667</td>
  </tr>
  <tr>
  <td>4</td>
  <td>IM01624</td>
  <td>0</td>
  <td>0</td>
  <td>1</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0.166667</td>
  </tr>
  <tr>
  <td>5</td>
  <td>IM01240</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>1</td>
  <td>0.166667</td>
  </tr>
  <tr>
  <td>6</td>
  <td>IM00807</td>
  <td>0</td>
  <td>1</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0.166667</td>
  </tr>
  <tr>
  <td>7</td>
  <td>IM00999</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0.000000</td>
  </tr>
  <tr>
  <td>8</td>
  <td>IM00983</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0.000000</td>
  </tr>
  <tr>
  <td>9</td>
  <td>IM01076</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0.000000</td>
  </tr>
  <tr>
  <td>10</td>
  <td>IM00803</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0.000000</td>
  </tr>
  <tr>
  <td>11</td>
  <td>IM00125</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0.000000</td>
  </tr>
  <tr>
  <td>12</td>
  <td>IM00998</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>1</td>
  <td>0</td>
  <td>0.166667</td>
  </tr>
  <tr>
  <td>13</td>
  <td>IM00997</td>
  <td>0</td>
  <td>1</td>
  <td>1</td>
  <td>1</td>
  <td>0</td>
  <td>1</td>
  <td>0.666667</td>
  </tr>
  <tr>
  <td>14</td>
  <td>IM00996</td>
  <td>0</td>
  <td>1</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>1</td>
  <td>0.333333</td>
  </tr>
  </tbody>
  </table>
  </div><hr />
  <p>If this is correct it provides the basis to move forward for all other operations. Basically, what I'm saying is I think is good to break open the <code>numpy</code> into explicit <code>pandas</code> operations, what is happening is there's a mix of <code>pandas</code> and <code>numpy</code> and I would recommend going down one route or the other.</p>
  