# HEWL S-SAD Merging Statistics

Merging statistics are a useful means to assess data quality in crystallography. However, each statistic has inherent shortcomings. For example, R-merge will appear inflated if the multiplicity is high, and the Pearson correlation coefficients used for $CC_{1/2}$ are very sensitive to outliers. 

Most scaling and merging programs output multiple merging statistics to get around these shortcomings. However, one can imagine that it could also be useful to customize certain parameters, such as how many resolution bins are used. Or, perhaps a better statistic will be developed that is worth implementing. 

In this notebook, ``reciprocalspaceship`` is used to compute half-dataset correlation coefficients ($CC_{1/2}$ and $CC_{anom}$) for a dataset collected from a tetragonal hen egg-white lysozyme (HEWL) crystal at 6550 eV. 
These data are unmerged, but were scaled in AIMLESS. 
They contain sufficient sulfur anomalous signal to determine a solution by the SAD method. 
This illustrates the use of ``rs`` to implement a merging routine, create a custom analysis, and could be useful as a template for other exploratory crystallographic data analyses. 
As an example, we will compare half-dataset correlations computed using both Pearson and Spearman correlation coefficients.


```python
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_context("notebook", font_scale=1.3)
import numpy as np
```


```python
import reciprocalspaceship as rs
```


```python
print(rs.__version__)
```

    0.9.2


---
### Load scaled, unmerged data  

This data has been scaled in AIMLESS. The data includes the image number and the scaled **I** and **SIGI** values. 


```python
hewl = rs.read_mtz("data/HEWL_unmerged.mtz")
```


```python
hewl.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th></th>
      <th></th>
      <th>BATCH</th>
      <th>I</th>
      <th>SIGI</th>
      <th>PARTIAL</th>
    </tr>
    <tr>
      <th>H</th>
      <th>K</th>
      <th>L</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th rowspan="5" valign="top">0</th>
      <th rowspan="5" valign="top">0</th>
      <th>4</th>
      <td>137</td>
      <td>696.5212</td>
      <td>87.83294</td>
      <td>False</td>
    </tr>
    <tr>
      <th>4</th>
      <td>520</td>
      <td>710.6812</td>
      <td>88.107025</td>
      <td>False</td>
    </tr>
    <tr>
      <th>4</th>
      <td>856</td>
      <td>672.05634</td>
      <td>87.75671</td>
      <td>False</td>
    </tr>
    <tr>
      <th>4</th>
      <td>1239</td>
      <td>642.47485</td>
      <td>87.90302</td>
      <td>False</td>
    </tr>
    <tr>
      <th>4</th>
      <td>2160</td>
      <td>655.71783</td>
      <td>87.74394</td>
      <td>False</td>
    </tr>
  </tbody>
</table>
</div>




```python
print(f"Number of observed reflections: {len(hewl)}")
```

    Number of observed reflections: 816804


---
### Merging with Inverse-Variance Weights

Since the input data are unmerged, we will implement the inverse-variance weighting scheme used by AIMLESS to merge the observations. 
The weighted average is a better estimator of the true mean than the raw average, and this weighting scheme corresponds to the maximum likelihood estimator of the true mean if we assume that the observations are normally-distributed about the true mean. 

The merged intensity for each reflection, $I_h$, can be determined from the observed intensities, $I_{h,i}$, and error estimates, $\sigma_{h,i}$, as follows:

\begin{equation}
I_h = \frac{\sum_{i}w_{h,i} I_{h,i}}{\sum_{i} w_{h,i}}
\end{equation}

where the weight for each observation, $w_{h,i}$ is given by:

\begin{equation}
w_{h,i} = \frac{1}{(\sigma_{h,i})^2}
\end{equation}

The updated estimate of the uncertainty, $\sigma_{h}$, is given by:

\begin{equation}
\sigma_{h} = \sqrt{\frac{1}{\sum_{i} w_{h,i}}}
\end{equation}

Let's start by implementing the above equations in a function that will compute the merged $I_h$ and $\sigma_h$. We will use the [Pandas groupby](https://pandas.pydata.org/pandas-docs/stable/user_guide/groupby.html) methods to apply this function across the unique Miller indices in the ``DataSet``. If we group Friedel pairs together, we will refer to the quantity as **IMEAN**, and if we keep the Friedel pairs separate we will refer to the quantities as **I(+)**  and **I(-)**.


```python
def merge(dataset, anomalous=False):
    """
    Merge dataset using inverse-variance weights.
    
    Parameters
    ----------
    dataset : rs.DataSet
        DataSet to be merged containing scaled I and SIGI
    anomalous : bool
        If True, I(+) and I(-) will be reported. If False,
        IMEAN will be reported
        
    Returns
    -------
    rs.DataSet
        Merged DataSet object
    """
    ds = dataset.hkl_to_asu(anomalous=anomalous)
    ds["w"]  = ds['SIGI']**-2
    ds["wI"] = ds["I"] * ds["w"]
    g = ds.groupby(["H", "K", "L"])
    
    result = g[["w", "wI"]].sum()
    result["I"] = result["wI"] / result["w"]
    result["SIGI"] = np.sqrt(1 / result["w"])
    result.merged = True
    
    if anomalous:
        result = result.loc[:, ["I", "SIGI"]].unstack_anomalous()
        return result[["I(+)", "SIGI(+)", "I(-)", "SIGI(-)"]]
    else:
        return result[["I", "SIGI"]]
```

Using `anomalous=False`, this function can be used to compute **IMEAN** and **SIGIMEAN** by including both Friedel pairs:


```python
result1 = merge(hewl, anomalous=False)
```


```python
result1.sample(5)
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th></th>
      <th></th>
      <th>I</th>
      <th>SIGI</th>
    </tr>
    <tr>
      <th>H</th>
      <th>K</th>
      <th>L</th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>15</th>
      <th>13</th>
      <th>17</th>
      <td>9.901793</td>
      <td>1.084705</td>
    </tr>
    <tr>
      <th>30</th>
      <th>20</th>
      <th>0</th>
      <td>532.1264</td>
      <td>25.592989</td>
    </tr>
    <tr>
      <th>20</th>
      <th>13</th>
      <th>9</th>
      <td>282.86536</td>
      <td>3.382065</td>
    </tr>
    <tr>
      <th>24</th>
      <th>13</th>
      <th>11</th>
      <td>110.44999</td>
      <td>2.107947</td>
    </tr>
    <tr>
      <th>23</th>
      <th>15</th>
      <th>1</th>
      <td>413.71066</td>
      <td>5.939175</td>
    </tr>
  </tbody>
</table>
</div>



Using `anomalous=True`, this function can be used to compute **I(+)**, **SIGI(+)**, **I(-)**, and **SIGI(-)** by separating Friedel pairs:


```python
result2 = merge(hewl, anomalous=True)
```


```python
result2.sample(5)
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th></th>
      <th></th>
      <th>I(+)</th>
      <th>SIGI(+)</th>
      <th>I(-)</th>
      <th>SIGI(-)</th>
    </tr>
    <tr>
      <th>H</th>
      <th>K</th>
      <th>L</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>30</th>
      <th>19</th>
      <th>5</th>
      <td>214.14368</td>
      <td>5.669856</td>
      <td>227.93652</td>
      <td>5.321779</td>
    </tr>
    <tr>
      <th>36</th>
      <th>10</th>
      <th>3</th>
      <td>303.7916</td>
      <td>9.099753</td>
      <td>287.29083</td>
      <td>7.715621</td>
    </tr>
    <tr>
      <th>31</th>
      <th>8</th>
      <th>0</th>
      <td>49.594536</td>
      <td>1.023669</td>
      <td>49.594536</td>
      <td>1.023669</td>
    </tr>
    <tr>
      <th>18</th>
      <th>4</th>
      <th>11</th>
      <td>1108.2372</td>
      <td>18.975866</td>
      <td>1118.1456</td>
      <td>21.075169</td>
    </tr>
    <tr>
      <th>30</th>
      <th>18</th>
      <th>3</th>
      <td>204.25981</td>
      <td>5.025567</td>
      <td>192.05464</td>
      <td>5.023432</td>
    </tr>
  </tbody>
</table>
</div>



A variant of the above function is implemented in `rs.algorithms`, and we will use that implementation in the next section for computing merging statistics. This function computes **IMEAN**, **I(+)**, **I(-)**, and associated uncertainties for each unique Miller index.


```python
result3 = rs.algorithms.merge(hewl)
```


```python
result3.sample(5)
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th></th>
      <th></th>
      <th>IMEAN</th>
      <th>SIGIMEAN</th>
      <th>I(+)</th>
      <th>SIGI(+)</th>
      <th>I(-)</th>
      <th>SIGI(-)</th>
      <th>N(+)</th>
      <th>N(-)</th>
    </tr>
    <tr>
      <th>H</th>
      <th>K</th>
      <th>L</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>35</th>
      <th>8</th>
      <th>2</th>
      <td>280.71973</td>
      <td>4.633233</td>
      <td>274.86194</td>
      <td>6.7137227</td>
      <td>286.0464</td>
      <td>6.4021378</td>
      <td>32</td>
      <td>36</td>
    </tr>
    <tr>
      <th>20</th>
      <th>5</th>
      <th>18</th>
      <td>23.973345</td>
      <td>1.2019373</td>
      <td>24.840725</td>
      <td>1.7172365</td>
      <td>23.140327</td>
      <td>1.6828763</td>
      <td>32</td>
      <td>32</td>
    </tr>
    <tr>
      <th>22</th>
      <th>13</th>
      <th>5</th>
      <td>120.70732</td>
      <td>1.5874561</td>
      <td>118.58768</td>
      <td>2.2071195</td>
      <td>122.979004</td>
      <td>2.2849045</td>
      <td>56</td>
      <td>52</td>
    </tr>
    <tr>
      <th>31</th>
      <th>17</th>
      <th>7</th>
      <td>884.2669</td>
      <td>14.877254</td>
      <td>885.66364</td>
      <td>21.056253</td>
      <td>882.8747</td>
      <td>21.023014</td>
      <td>32</td>
      <td>32</td>
    </tr>
    <tr>
      <th>19</th>
      <th>8</th>
      <th>8</th>
      <td>1436.745</td>
      <td>18.876888</td>
      <td>1430.5896</td>
      <td>26.704302</td>
      <td>1442.8926</td>
      <td>26.687607</td>
      <td>52</td>
      <td>52</td>
    </tr>
  </tbody>
</table>
</div>



---
### Merging with 2-fold Cross-Validation  

To compute correlation coefficients we will repeatedly split our data into half-datasets. We will do this by randomly splitting the data using the image number. These half-datasets will be merged independently and used to determine uncertainties in the correlation coefficients. We will first write a method to randomly split our data, and then we will then write a method that automates the sampling and merging of multiple half-datasets. 


```python
def sample_halfdatasets(data):
    """Randomly split DataSet into two equal halves by BATCH"""
    batch = data.BATCH.unique().to_numpy(dtype=int)
    np.random.shuffle(batch)
    halfbatch1, halfbatch2 = np.array_split(batch, 2)
    half1 = data.loc[data.BATCH.isin(halfbatch1)]
    half2 = data.loc[data.BATCH.isin(halfbatch2)]
    return half1, half2
```


```python
def merge_dataset(dataset, nsamples):
    """
    Merge DataSet using inverse-variance weighting scheme. This represents the 
    maximum-likelihood estimator of the mean of the observed intensities assuming 
    they are independent and normally distributed with the same mean. 
    
    Sample means across half-datasets can be used to compute the merging statistics CC1/2 and CCanom.
    """
    dataset = dataset.copy()
    samples = []
    for n in range(nsamples):
        half1, half2 = sample_halfdatasets(dataset)
        mergedhalf1 = rs.algorithms.merge(half1)
        mergedhalf2 = rs.algorithms.merge(half2)
        result = mergedhalf1.merge(mergedhalf2, left_index=True, right_index=True, suffixes=(1, 2))
        result["sample"] = n
        samples.append(result)
    return rs.concat(samples).sort_index()
```

---
### Merge HEWL data

We will now merge the HEWL data, repeatedly sampling across half-datasets in order to assess the distribution of correlation coefficients.


```python
merged = merge_dataset(hewl, 15)
```


```python
merged
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th></th>
      <th></th>
      <th>IMEAN1</th>
      <th>SIGIMEAN1</th>
      <th>I(+)1</th>
      <th>SIGI(+)1</th>
      <th>I(-)1</th>
      <th>SIGI(-)1</th>
      <th>N(+)1</th>
      <th>N(-)1</th>
      <th>IMEAN2</th>
      <th>SIGIMEAN2</th>
      <th>I(+)2</th>
      <th>SIGI(+)2</th>
      <th>I(-)2</th>
      <th>SIGI(-)2</th>
      <th>N(+)2</th>
      <th>N(-)2</th>
      <th>sample</th>
    </tr>
    <tr>
      <th>H</th>
      <th>K</th>
      <th>L</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th rowspan="5" valign="top">0</th>
      <th rowspan="5" valign="top">0</th>
      <th>4</th>
      <td>653.37506</td>
      <td>35.868774</td>
      <td>653.37506</td>
      <td>35.86877</td>
      <td>653.37506</td>
      <td>35.86877</td>
      <td>6</td>
      <td>6</td>
      <td>666.0466</td>
      <td>27.759663</td>
      <td>666.0466</td>
      <td>27.75966</td>
      <td>666.0466</td>
      <td>27.75966</td>
      <td>10</td>
      <td>10</td>
      <td>0</td>
    </tr>
    <tr>
      <th>4</th>
      <td>658.2116</td>
      <td>26.48474</td>
      <td>658.2116</td>
      <td>26.48474</td>
      <td>658.2116</td>
      <td>26.48474</td>
      <td>11</td>
      <td>11</td>
      <td>668.0805</td>
      <td>39.243835</td>
      <td>668.0805</td>
      <td>39.243835</td>
      <td>668.0805</td>
      <td>39.243835</td>
      <td>5</td>
      <td>5</td>
      <td>1</td>
    </tr>
    <tr>
      <th>4</th>
      <td>644.7756</td>
      <td>35.824974</td>
      <td>644.7756</td>
      <td>35.824974</td>
      <td>644.7756</td>
      <td>35.824974</td>
      <td>6</td>
      <td>6</td>
      <td>671.236</td>
      <td>27.780022</td>
      <td>671.236</td>
      <td>27.780024</td>
      <td>671.236</td>
      <td>27.780024</td>
      <td>10</td>
      <td>10</td>
      <td>2</td>
    </tr>
    <tr>
      <th>4</th>
      <td>670.15765</td>
      <td>27.77585</td>
      <td>670.15765</td>
      <td>27.77585</td>
      <td>670.15765</td>
      <td>27.77585</td>
      <td>10</td>
      <td>10</td>
      <td>646.55707</td>
      <td>35.833927</td>
      <td>646.55707</td>
      <td>35.833927</td>
      <td>646.55707</td>
      <td>35.833927</td>
      <td>6</td>
      <td>6</td>
      <td>3</td>
    </tr>
    <tr>
      <th>4</th>
      <td>668.8312</td>
      <td>25.352604</td>
      <td>668.8312</td>
      <td>25.352604</td>
      <td>668.8312</td>
      <td>25.352604</td>
      <td>12</td>
      <td>12</td>
      <td>638.73004</td>
      <td>43.888813</td>
      <td>638.73004</td>
      <td>43.888813</td>
      <td>638.73004</td>
      <td>43.888813</td>
      <td>4</td>
      <td>4</td>
      <td>4</td>
    </tr>
    <tr>
      <th>...</th>
      <th>...</th>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th rowspan="5" valign="top">45</th>
      <th rowspan="5" valign="top">10</th>
      <th>2</th>
      <td>15.318142</td>
      <td>3.9687188</td>
      <td>15.318142</td>
      <td>3.9687188</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>3</td>
      <td>0</td>
      <td>30.170218</td>
      <td>7.3511124</td>
      <td>30.170218</td>
      <td>7.3511124</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>1</td>
      <td>0</td>
      <td>9</td>
    </tr>
    <tr>
      <th>2</th>
      <td>24.119883</td>
      <td>4.656634</td>
      <td>24.119883</td>
      <td>4.656634</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>2</td>
      <td>0</td>
      <td>11.665072</td>
      <td>5.2794247</td>
      <td>11.665072</td>
      <td>5.2794247</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>2</td>
      <td>0</td>
      <td>10</td>
    </tr>
    <tr>
      <th>2</th>
      <td>11.665072</td>
      <td>5.2794247</td>
      <td>11.665072</td>
      <td>5.2794247</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>2</td>
      <td>0</td>
      <td>24.119883</td>
      <td>4.656634</td>
      <td>24.119883</td>
      <td>4.656634</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>2</td>
      <td>0</td>
      <td>12</td>
    </tr>
    <tr>
      <th>2</th>
      <td>18.59031</td>
      <td>8.728156</td>
      <td>18.59031</td>
      <td>8.728156</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>1</td>
      <td>0</td>
      <td>18.685297</td>
      <td>3.8105915</td>
      <td>18.685297</td>
      <td>3.8105915</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>3</td>
      <td>0</td>
      <td>13</td>
    </tr>
    <tr>
      <th>2</th>
      <td>7.669444</td>
      <td>6.6297607</td>
      <td>7.669444</td>
      <td>6.6297607</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>1</td>
      <td>0</td>
      <td>22.89468</td>
      <td>4.1084785</td>
      <td>22.89468</td>
      <td>4.1084785</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>3</td>
      <td>0</td>
      <td>14</td>
    </tr>
  </tbody>
</table>
<p>187288 rows × 17 columns</p>
</div>



---
### Compute $CC_{1/2}$ and $CC_{anom}$  

We will first assign each reflection to a resolution bin and then we will compute the correlation coefficients


```python
merged, labels = merged.assign_resolution_bins()
```


```python
groupby1 = merged.groupby(["sample", "bin"])[["IMEAN1", "IMEAN2"]]
pearson1 = groupby1.corr(method="pearson").unstack().loc[:, ("IMEAN1", "IMEAN2")]
pearson1.name = "Pearson"
spearman1 = groupby1.corr(method="spearman").unstack().loc[:, ("IMEAN1", "IMEAN2")]
spearman1.name = "Spearman"
results1 = rs.concat([pearson1, spearman1], axis=1)
results1 = results1.groupby("bin").agg(["mean", "std"])
```


```python
results1.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead tr th {
        text-align: left;
    }

    .dataframe thead tr:last-of-type th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr>
      <th></th>
      <th colspan="2" halign="left">Pearson</th>
      <th colspan="2" halign="left">Spearman</th>
    </tr>
    <tr>
      <th></th>
      <th>mean</th>
      <th>std</th>
      <th>mean</th>
      <th>std</th>
    </tr>
    <tr>
      <th>bin</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.997022</td>
      <td>0.000454</td>
      <td>0.998628</td>
      <td>0.000141</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.997647</td>
      <td>0.000331</td>
      <td>0.999001</td>
      <td>0.000135</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.998144</td>
      <td>0.000435</td>
      <td>0.999532</td>
      <td>0.000039</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0.999188</td>
      <td>0.000219</td>
      <td>0.999657</td>
      <td>0.000039</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.999198</td>
      <td>0.000093</td>
      <td>0.999626</td>
      <td>0.000030</td>
    </tr>
  </tbody>
</table>
</div>




```python
plt.figure(figsize=(8, 4))
plt.errorbar(results1.index, results1[("Pearson", "mean")], 
             yerr=results1[("Pearson", "std")], label=r"$CC_{1/2}$ (Pearson)")
plt.errorbar(results1.index, results1[("Spearman", "mean")], 
             yerr=results1[("Spearman", "std")], label=r"$CC_{1/2}$ (Spearman)")
plt.xticks(results1.index, labels, rotation=45)
plt.ylabel("Correlation Coefficient")
plt.xlabel(r"Resolution Bin ($\AA$)")
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
plt.show()
```

It is important to note the scale on the y-axis -- this dataset is edge-limited, and as such the $CC_{1/2}$ is very high across all resolution bins. The Spearman CC appears higher across all resolution bins except at high resolution, and overall has a lower standard deviation among samples.
This is consistent with our expectation that Spearman CCs are a more robust estimator of correlation than Pearson CCs.

Let's now repeat this for the anomalous data, computing $CC_{anom}$:


```python
merged["ANOM1"] = merged["I(+)1"] - merged["I(-)1"]
merged["ANOM2"] = merged["I(+)2"] - merged["I(-)2"]

groupby2 = merged.groupby(["sample", "bin"])[["ANOM1", "ANOM2"]]
pearson2 = groupby2.corr(method="pearson").unstack().loc[:, ("ANOM1", "ANOM2")]
pearson2.name = "Pearson"
spearman2 = groupby2.corr(method="spearman").unstack().loc[:, ("ANOM1", "ANOM2")]
spearman2.name = "Spearman"
results2 = rs.concat([pearson2, spearman2], axis=1)
results2 = results2.groupby("bin").agg(["mean", "std"])
```


```python
results2.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead tr th {
        text-align: left;
    }

    .dataframe thead tr:last-of-type th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr>
      <th></th>
      <th colspan="2" halign="left">Pearson</th>
      <th colspan="2" halign="left">Spearman</th>
    </tr>
    <tr>
      <th></th>
      <th>mean</th>
      <th>std</th>
      <th>mean</th>
      <th>std</th>
    </tr>
    <tr>
      <th>bin</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.403517</td>
      <td>0.068201</td>
      <td>0.494861</td>
      <td>0.033880</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.292030</td>
      <td>0.081911</td>
      <td>0.413485</td>
      <td>0.033760</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.304523</td>
      <td>0.052493</td>
      <td>0.363430</td>
      <td>0.039869</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0.377034</td>
      <td>0.057670</td>
      <td>0.427256</td>
      <td>0.030954</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.488685</td>
      <td>0.070847</td>
      <td>0.539804</td>
      <td>0.018448</td>
    </tr>
  </tbody>
</table>
</div>




```python
plt.figure(figsize=(8, 4))
plt.errorbar(results2.index, results2[("Pearson", "mean")], 
             yerr=results2[("Pearson", "std")], label=r"$CC_{anom}$ (Pearson)")
plt.errorbar(results2.index, results2[("Spearman", "mean")],
             yerr=results2[("Spearman", "std")], label=r"$CC_{anom}$ (Spearman)")
plt.xticks(results2.index, labels, rotation=45)
plt.ylabel("Correlation Coefficient")
plt.xlabel(r"Resolution Bin ($\AA$)")
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
plt.show()
```

There is significant anomalous signal across all but the highest resolution bins. The Spearman CCs have smaller error bars than the corresponding Pearson CCs and the Spearman CCs are also higher in most bins. These differences highlight the influence of outlier measurements on the different correlation coefficients.

---
### Summary  

We have used `reciprocalspaceship` to merge a dataset using inverse-variance weights. As part of this analysis, we performed repeated 2-fold cross-validation to compute Pearson and Spearman correlation coefficients and associated uncertainties. This relatively simple procedure to obtain uncertainty estimates for correlation coefficients is seldom done when analyzing merging quality. However, we can see that the standard deviation for computed correlation coefficients can be as high as $\pm0.1$ for quantities such as $CC_{anom}$. This is worth keeping in mind when analyzing SAD experiments because this dataset is very high quality (edge-limited) when one considers the $CC_{1/2}$. Putting these two correlation coefficients on the same axes emphasizes this point:


```python
plt.figure(figsize=(9, 6))
plt.errorbar(results1.index, results1[("Pearson", "mean")], 
             yerr=results1[("Pearson", "std")], color='k', label=r"$CC_{1/2}$ (Pearson)")
plt.errorbar(results1.index, results1[("Spearman", "mean")], 
             yerr=results1[("Spearman", "std")], color='gray', label=r"$CC_{1/2}$ (Spearman)")
plt.errorbar(results2.index, results2[("Pearson", "mean")], 
             yerr=results2[("Pearson", "std")], color='k', linestyle="--", label=r"$CC_{anom}$ (Pearson)")
plt.errorbar(results2.index, results2[("Spearman", "mean")], 
             yerr=results2[("Spearman", "std")], color='gray', linestyle="--", label=r"$CC_{anom}$ (Spearman)")
plt.xticks(results2.index, labels, rotation=45)
plt.ylabel(r"Correlation Coefficient")
plt.xlabel(r"Resolution Bin ($\AA$)")
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
plt.tight_layout()
plt.grid(linestyle='--')
```

Even a simple change to the procedure for computing merging statistics, such as substituting a Spearman correlation coefficient for a Pearson one, can alter the apparent quality of a dataset. By lowering the barrier to implementing new analyses, we hope that `reciprocalspaceship` can encourage the development of more robust indicators of crystallographic data quality.

16f578f602478e2081e84772c9f82708847c9b06
ipynb
Jupyter Notebook
docs/examples/2_mergingstats.ipynb
apeck12/reciprocalspaceship
ef00e76172699a1dd6f5ce6a128ecac16ba43eec
MIT
docs/examples/2_mergingstats.ipynb
apeck12/reciprocalspaceship
ef00e76172699a1dd6f5ce6a128ecac16ba43eec
MIT
docs/examples/2_mergingstats.ipynb
apeck12/reciprocalspaceship
ef00e76172699a1dd6f5ce6a128ecac16ba43eec
MIT
Qwen/Qwen-72B
1. YES
2. YES