# Digital agro

# Sensitivity analysis of models

# Seminar - 24.04.2020

## Mikhail Gasanov

### The scheme and main blocks of the plant growth model

## git clone

```python
!git clone https://github.com/mishagrol/Seminar_Sobol.git
!cp -r ./Seminar_Sobol/* .
```

    Cloning into 'Seminar_Sobol'...
    remote: Enumerating objects: 213, done.
    remote: Counting objects: 100% (213/213), done.
    remote: Compressing objects: 100% (163/163), done.
    remote: Total 213 (delta 72), reused 184 (delta 47), pack-reused 0
    Receiving objects: 100% (213/213), 33.39 MiB | 27.20 MiB/s, done.
    Resolving deltas: 100% (72/72), done.

# Running PCSE/WOFOST with custom input datф

This Jupyter notebook will show you how to read inputs from files for running PCSE/WOFOST.

thanks to **Allard de Wit**

**Prerequisites for running this notebook**

Several packages need to be installed for running PCSE/WOFOST:

  1. `PCSE` and its dependencies. See the [PCSE user guide](http://pcse.readthedocs.io/en/stable/installing.html) for more information;
  2. The `pandas` module for processing and storing WOFOST output;
  3. The `matplotlib` module for generating charts

## Introduction

For running PCSE/WOFOST (and PCSE models in general) with your own data sources you need three different types of inputs:

 1. Model parameters that parameterize the different model components. These parameters usually consist of a set of crop parameters (or multiple sets in case of crop rotations), a set of soil parameters and a set of site parameters. The latter provide ancillary parameters that are specific for a location, for example the initial amount of moisture in the soil.
 2. Driving variables represented by weather data which can be derived from various sources.
 3. Agromanagement actions which specify the farm activities that will take place on the field that is simulated by PCSE.

For this example we will run a simulation for sugar beet in Wageningen (Netherlands) and we will read the input data step by step from several different sources instead of using the pre-configured start_wofost() script. For the example we will assume that data files are in the `data` directory within the directory where this notebook is located. This will be the case if you downloaded the notebooks from github.

## Importing the relevant modules

```python
!pip install pcse
```

    Requirement already satisfied: pcse in /usr/local/lib/python3.6/dist-packages (5.4.2)
    Requirement already satisfied: SQLAlchemy>=0.8.0 in /usr/local/lib/python3.6/dist-packages (from pcse) (1.3.19)
    Requirement already satisfied: traitlets-pcse==5.0.0.dev in /usr/local/lib/python3.6/dist-packages (from pcse) (5.0.0.dev0)
    Requirement already satisfied: numpy>=1.6.0 in /usr/local/lib/python3.6/dist-packages (from pcse) (1.18.5)
    Requirement already satisfied: PyYAML>=3.11 in /usr/local/lib/python3.6/dist-packages (from pcse) (3.13)
    Requirement already satisfied: xlrd>=0.9.3 in /usr/local/lib/python3.6/dist-packages (from pcse) (1.1.0)
    Requirement already satisfied: pandas>=0.20 in /usr/local/lib/python3.6/dist-packages (from pcse) (1.0.5)
    Requirement already satisfied: requests>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from pcse) (2.23.0)
    Requirement already satisfied: xlwt>=1.0.0 in /usr/local/lib/python3.6/dist-packages (from pcse) (1.3.0)
    Requirement already satisfied: decorator in /usr/local/lib/python3.6/dist-packages (from traitlets-pcse==5.0.0.dev->pcse) (4.4.2)
    Requirement already satisfied: ipython-genutils in /usr/local/lib/python3.6/dist-packages (from traitlets-pcse==5.0.0.dev->pcse) (0.2.0)
    Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from traitlets-pcse==5.0.0.dev->pcse) (1.15.0)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.20->pcse) (2018.9)
    Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.20->pcse) (2.8.1)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0.0->pcse) (2.10)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0.0->pcse) (2020.6.20)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0.0->pcse) (1.24.3)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0.0->pcse) (3.0.4)

```python
%matplotlib inline
import sys, os
import matplotlib
matplotlib.style.use("ggplot")
import matplotlib.pyplot as plt
import pandas as pd
import yaml
import numpy as np

data_dir = os.path.join(os.getcwd(), "data")
import pcse
print("This notebook was built with:")
print("python version: %s " % sys.version)
print("PCSE version: %s" %  pcse.__version__)
```

    This notebook was built with:
    python version: 3.6.9 (default, Jul 17 2020, 12:50:27)
    [GCC 8.4.0]
    PCSE version: 5.4.2

## Reading model parameters
### Crop parameters

The crop parameters consist of parameter names and the corresponding parameter values that are needed to parameterize the components of the crop simulation model. These are crop-specific values regarding phenology, assimilation, respiration, biomass partitioning, etc. The parameter file for sugar beet is taken from the crop files in the WOFOST Control Centre.

As many crop models in Wageningen were written in FORTRAN, the crop parameters for many models in Wageningen are often provided in the CABO format that could be read with the TTUTIL FORTRAN library. This CABO format will be gradually phased out and PCSE will move to a new format based on YAML, see [here for an example](https://github.com/ajwdewit/WOFOST_crop_parameters/blob/master/wheat.yaml). However, PCSE tries to be backward compatible as much as possible and provides the `CABOFileReader` for reading parameter files in CABO format. The `CABOFileReader` returns a dictionary with the parameter name/value pairs:

```python
from pcse.fileinput import CABOFileReader
cropfile = os.path.join(data_dir, 'crop', 'SUG0601.crop')
cropdata = CABOFileReader(cropfile)
```

```python
#potato
from pcse.fileinput import CABOFileReader
cropfile_potato = os.path.join(data_dir, 'crop', 'POT701.CAB')
cropdata_potato = CABOFileReader(cropfile_potato)
```

```python
# Number of parameters for our crop
len(cropdata_potato)
```

    63

### Soil parameters
The soildata dictionary provides the parameter name/value pairs related to the soil type and soil physical properties. The number of parameters is variable depending on the soil water balance type that is used for the simulation. For this example, we will use the water balance for freely draining soils and use the soil file for medium fine sand: `ec3.soil`. This file is also taken from the soil files in the [WOFOST Control Centre](http://www.wageningenur.nl/wofost).

```python
soilfile = os.path.join(data_dir, 'soil', 'ec3.soil')
soildata = CABOFileReader(soilfile)
print(soildata)
```

    ** $Id: ec3.new 1.2 1997/09/18 17:33:54 LEM release $
    **
    ** SOIL DATA FILE for use with WOFOST Version 5.0, June 1990
    **
    ** EC3-medium fine
    ------------------------------------
    SMW: 0.104 <class 'float'>
    SMFCF: 0.3 <class 'float'>
    SM0: 0.41 <class 'float'>
    CRAIRC: 0.06 <class 'float'>
    K0: 25.586 <class 'float'>
    SOPE: 1.47 <class 'float'>
    KSUB: 1.47 <class 'float'>
    SPADS: 0.1 <class 'float'>
    SPODS: 0.03 <class 'float'>
    SPASS: 0.2 <class 'float'>
    SPOSS: 0.05 <class 'float'>
    DEFLIM: -0.3 <class 'float'>
    RDMSOL: 120 <class 'int'>
    SOLNAM: EC3-medium fine <class 'str'>
    SMTAB: [-1.0, 0.41, 1.0, 0.398, 1.3, 0.389, 1.491, 0.38, 2.0, 0.34, 2.4, 0.287, 2.7, 0.241, 3.4, 0.148, 4.204, 0.104, 6.0, 0.09] <class 'list'>
    CONTAB: [0.0, 1.408, 1.0, 0.167, 1.3, -0.215, 1.491, -0.638, 1.7, -0.854, 2.0, -1.155, 2.4, -1.796, 2.7, -2.26, 3.0, -2.745, 3.4, -3.357, 3.7, -3.824, 4.0, -4.276, 4.204, -4.678] <class 'list'>

### Site parameters

The site parameters provide ancillary parameters that are not related to the crop or the soil. Examples are the initial conditions of the water balance such as the initial soil moisture content (WAV) and the initial and maximum surface storage (SSI, SSMAX). Also the atmospheric $CO_{2}$
concentration is a typical site parameter. For the moment, we can define these parameters directly on the Python commandline as a simple python dictionary. However, it is more convenient to use the `WOFOST71SiteDataProvider` that documents the site parameters and provides sensible defaults:

```python
from pcse.util import WOFOST71SiteDataProvider
sitedata = WOFOST71SiteDataProvider(WAV=100, CO2=360)
print(sitedata)
```

    {'IFUNRN': 0, 'NOTINF': 0, 'SSI': 0.0, 'SSMAX': 0.0, 'WAV': 100.0, 'SMLIM': 0.4, 'CO2': 360.0}

### Packaging all parameters
Finally, we need to pack the different sets of parameters into one variable using the `ParameterProvider`. This is needed because PCSE expects one variable that contains all parameter values. Using this approach has the additional advantage that parameter value can be easily overridden in case of running multiple simulations with slightly different parameter values:

```python
from pcse.base import ParameterProvider
parameters = ParameterProvider(cropdata=cropdata, soildata=soildata, sitedata=sitedata)
```

## Agromanagement
The agromanagement inputs provide the start date of the agricultural campaign, the start_date/start_type of the crop simulation, the end_date/end_type of the crop simulation and the maximum duration of the crop simulation. The latter is included to avoid unrealistically long simulations for example as a results of a too high temperature sum requirement.

The agromanagement inputs are defined with a special syntax called [YAML](http://yaml.org/) which allows to easily create more complex structures which is needed for defining the agromanagement. The agromanagement file for sugar beet in Wageningen `sugarbeet_calendar.agro` can be read with the `YAMLAgroManagementReader`:

```python
from pcse.fileinput import YAMLAgroManagementReader
#crop rotation for Moscow region
agromanagement_file = os.path.join(data_dir, 'agro', 'sugarbeet_calendar_Moscow_short.agro')
#agromanagement_file = os.path.join(data_dir, 'agro', 'sugarbeet_calendar.agro')
agromanagement = YAMLAgroManagementReader(agromanagement_file)
print(agromanagement)
```

    !!python/object/new:pcse.fileinput.yaml_agro_loader.YAMLAgroManagementReader
    listitems:
    - 2019-06-01:
        CropCalendar:
          crop_end_date: 2019-10-15
          crop_end_type: harvest
          crop_name: sugar-beet
          crop_start_date: 2019-06-02
          crop_start_type: emergence
          max_duration: 300
          variety_name: sugar-beet-601
        StateEvents: null
        TimedEvents:
        - comment: All fertilizer amounts in kg/ha
          event_signal: apply_npk
          events_table:
          - 2019-06-22:
              K_amount: 122
              N_amount: 128
              P_amount: 25
          name: Timed N/P/K application table

We can create a crop rotation in the model

```python
"""
- 2001-03-01:
    CropCalendar:
        crop_name: maize
        variety_name: fodder-maize
        crop_start_date: 2001-04-15
        crop_start_type: sowing
        crop_end_date:
        crop_end_type: maturity
        max_duration: 200
    TimedEvents:
    -   event_signal: irrigate
        name:  Timed irrigation events
        comment: All irrigation amounts in cm
        events_table:
        - 2001-06-01: {irrigation_amount: 2.0}
        - 2001-07-21: {irrigation_amount: 5.0}
        - 2001-08-18: {irrigation_amount: 3.0}
        - 2001-09-19: {irrigation_amount: 2.5}
    -   event_signal: apply_npk
        name:  Timed N/P/K application table
        comment: All fertilizer amounts in kg/ha
        events_table:
        - 2001-05-25: {N_amount : 50, P_amount: 25, K_amount: 22}
        - 2001-07-05: {N_amount : 70, P_amount: 35, K_amount: 32}
    StateEvents:null
    """
```

```python
K_kg = 60
P_kg = 60
N_kg = 120
year_date_1 = 2000
year_date=2017
yaml_agro = f"""
- {year_date}-05-01:
    CropCalendar:
        crop_name: 'sugar-beet'
        variety_name: 'sugar-beet-601'
        crop_start_date: {year_date}-05-20
        crop_start_type: sowing
        crop_end_date:
        crop_end_type: maturity
        max_duration: 250
    TimedEvents:
    -   event_signal: apply_npk
        name:  Timed N/P/K application table
        comment: All fertilizer amounts in kg/ha
        events_table:
        - {year_date}-06-22: {{N_amount : {N_kg}, P_amount: {P_kg}, K_amount: {K_kg}}}
    StateEvents: null
"""
agromanagement = yaml.safe_load(yaml_agro)
print(yaml_agro)
#crop_end_date: {year_date_1}-11-15
```

    - 2017-05-01:
        CropCalendar:
            crop_name: 'sugar-beet'
            variety_name: 'sugar-beet-601'
            crop_start_date: 2017-05-20
            crop_start_type: sowing
            crop_end_date:
            crop_end_type: maturity
            max_duration: 250
        TimedEvents:
        -   event_signal: apply_npk
            name:  Timed N/P/K application table
            comment: All fertilizer amounts in kg/ha
            events_table:
            - 2017-06-22: {N_amount : 120, P_amount: 60, K_amount: 60}
        StateEvents: null

## Daily weather observations
Daily weather variables are needed for running the simulation. There are several data providers in PCSE for reading weather data, see the section on [weather data providers](http://pcse.readthedocs.io/en/stable/reference_guide.html#weather-data-providers) to get an overview.

For this example we will use weather data from an excel file which provides daily weather data for Wageningen for the period 2004 to 2008. We will read the data from the file using the ExcelWeatherDataProvider:

### NASA Weather Data Provider from NASA [DataBase](https://power.larc.nasa.gov/)

```python
#NASA Weather system

#Sometimes it doesnot work, upload excel file
from pcse.db import NASAPowerWeatherDataProvider
```

```python
#export weather to list

weather = NASAPowerWeatherDataProvider(51, 5, force_update=True)

```

## Importing, initializing and running a PCSE model

Internally, PCSE uses a simulation engine to run a crop simulation. This engine takes a configuration file that specifies the components for the crop, the soil and the agromanagement that need to be used for the simulation. So any PCSE model can be started by importing the engine and initializing it with a given configuration file and the corresponding parameters, weather data and agromanagement.

However, as many users of PCSE only need a particular configuration (for example the WOFOST model for potential production), preconfigured Engines are provided in `pcse.models`. For the sugarbeet example we will import the WOFOST model for water-limited simulation under freely draining soil conditions:

```python
from pcse.models import Wofost71_WLP_FD
wofsim = Wofost71_WLP_FD(parameters, weather, agromanagement)
wofsim.run_till_terminate()
df_results = pd.DataFrame(wofsim.get_output())
df_results = df_results.set_index("day")
df_results.tail()
```

      DVS
      LAI
      TAGP
      TWSO
      TWLV
      TWST
      TWRT
      TRA
      RD
      SM
      WWLOW

      day

      2017-10-18
      1.974014
      1.835032
      15026.929031
      11425.236324
      1576.867308
      2024.825399
      1510.637632
      0.077443
      120.0
      0.285257
      34.230800

      2017-10-19
      1.982186
      1.767827
      15124.263881
      11512.837689
      1581.734050
      2029.692142
      1512.295944
      0.113898
      120.0
      0.284645
      34.157386

      2017-10-20
      1.991271
      1.706132
      15205.833376
      11586.250234
      1585.812525
      2033.770616
      1513.771929
      0.071547
      120.0
      0.283741
      34.048913

      2017-10-21
      1.998536
      1.642232
      15243.799727
      11620.419951
      1587.710843
      2035.668934
      1514.503668
      0.126996
      120.0
      0.284684
      34.162043

      2017-10-22
      2.000000
      1.579179
      15324.917499
      11693.425945
      1591.766731
      2039.724823
      1516.143676
      0.071848
      120.0
      0.284029
      34.083500

We can then run the simulation and retrieve the time series of daily simulation output using the get_output() method on the WOFOST object. Finally, we convert the simulation results to a pandas dataframe:

```python
summary_output = wofsim.get_summary_output()
```

```python
wofsim.get_summary_output()
```

    [{'CTRAT': 17.589144084399933,
      'DOA': datetime.date(2017, 7, 8),
      'DOE': datetime.date(2017, 5, 27),
      'DOH': None,
      'DOM': datetime.date(2017, 10, 22),
      'DOS': datetime.date(2017, 5, 20),
      'DOV': None,
      'DVS': 2.0,
      'LAIMAX': 2.621559537235757,
      'RD': 120.0,
      'TAGP': 15324.917498780354,
      'TWLV': 1591.7667312492454,
      'TWRT': 1516.143676336077,
      'TWSO': 11693.425944916902,
      'TWST': 2039.7248226142065}]

```python
msg = "Reached maturity at {DOM} with total biomass {TAGP} kg/ha "\
"and a yield of {TWSO} kg/ha."
print(msg.format(**summary_output[0]))
```

    Reached maturity at 2017-10-22 with total biomass 15324.917498780354 kg/ha and a yield of 11693.425944916902 kg/ha.

# Sensitivity analysis

```python
!pip install SALib
```

    Collecting SALib
      Downloading https://files.pythonhosted.org/packages/f7/33/cee4d64f7c40f33c08cf5ef5c9b1fb5e51f194b5deceefb5567112800b70/SALib-1.3.11.tar.gz (856kB)
         |████████████████████████████████| 860kB 3.3MB/s
    Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from SALib) (1.18.5)
    Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from SALib) (1.4.1)
    Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from SALib) (3.2.2)
    Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from SALib) (1.0.5)
    Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->SALib) (2.8.1)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->SALib) (2.4.7)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->SALib) (0.10.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->SALib) (1.2.0)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas->SALib) (2018.9)
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil>=2.1->matplotlib->SALib) (1.15.0)
    Building wheels for collected packages: SALib
      Building wheel for SALib (setup.py) ... done
      Created wheel for SALib: filename=SALib-1.3.11-py2.py3-none-any.whl size=729665 sha256=93e720543725a20b1f31d6b4dcd5681574a59bb5c49e364a65b935ee9a92a0b0
      Stored in directory: /root/.cache/pip/wheels/62/ed/f9/a0b98754ffb2191b98324b96cbbeb1bd5d9598b39ab996b429
    Successfully built SALib
    Installing collected packages: SALib
    Successfully installed SALib-1.3.11

```python
from SALib.sample import saltelli
from SALib.analyze import sobol
from SALib.test_functions import Ishigami
import numpy as np
```

__Docs [SALib](https://salib.readthedocs.io/en/latest/#)__

In this example, we will perform a Sobol’ sensitivity analysis of the _Ishigami_ function, shown below. The _Ishigami_ function is commonly used to test uncertainty and sensitivity analysis methods because it exhibits strong nonlinearity and nonmonotonicity.

$f(x)=\sin \left(x_{1}\right)+ \text{a}\, \operatorname{sin}^{2}\left(x_{2}\right)+ \text{b}\,  x_{3}^{4} \sin \left(x_{1}\right)$

```python
problem = {
  'num_vars': 3,
  'names': ['x1', 'x2', 'x3'],
  'bounds': [[-np.pi, np.pi]]*3
}
```

```python
# Generate samples
param_values = saltelli.sample(problem, 10, calc_second_order=True)
param_values.shape
```

    (80, 3)

Here, `param_values` is a NumPy matrix. If we run `param_values.shape`, we see that the matrix is **8000 by 3**. The Saltelli sampler generated 8000 samples. The Saltelli sampler generates $N∗(2D+2)$ samples, where in this example $N$ is 1000 (the argument we supplied) and $D$ is 3 (the number of model inputs). The keyword argument `calc_second_order=False` will exclude second-order indices, resulting in a smaller sample matrix with $N∗(D+2)$ rows instead.

```python
# Run model (example)
Y = Ishigami.evaluate(param_values)

# Perform analysis
Si = sobol.analyze(problem, Y, print_to_console=True)
# Returns a dictionary with keys 'S1', 'S1_conf', 'ST', and 'ST_conf'
# (first and total-order indices with bootstrap confidence intervals)
T_Si, first_Si, (idx, second_Si) = sobol.Si_to_pandas_dict(Si)
```

    Parameter S1 S1_conf ST ST_conf
    x1 -0.203434 0.660158 1.345837 2.206907
    x2 0.838328 0.650453 0.716922 0.567090
    x3 -0.592087 0.390549 0.459872 0.420236

    Parameter_1 Parameter_2 S2 S2_conf
    x1 x2 0.208665 1.159449
    x1 x3 1.165212 0.917190
    x2 x3 0.268558 0.759311

Consider the model output as
\begin{eqnarray*}
Y=f(X)=f\left(X_{1}, \ldots, X_{p}\right),
\end{eqnarray*}
where $f$ in our case part of agro-model simulator, $X$ are $p$ varied input parameters and $Y$ is the predicted output. Following the techniques by Sobol we represent the multi-variate random function $f$ using Hoeffding decomposition:
\begin{equation}
f(X_1,\dots,X_p) = f_0 + \sum_i^p f_i + \sum_i^p\sum_{j>i}^p f_{ij} + \dots + f_{1\dots p},
\end{equation}
where $f_0$ is a constant term, $f_i = f_i(X_i)$ denotes main effects, $f_{ij} = f_{ij}(X_i, X_j)$ and others describe higher-order interactions. These terms can be written as
\begin{equation*}
\begin{split}
f_0 &= E(Y),\\
f_i &= E_{X_{\sim i}}(Y | X_i) - E(Y),\\
f_{ij} &= E_{X_{\sim ij}}(Y | X_i, X_j) - f_i - f_j - f_0,\\
\dots
\end{split}
\end{equation*}
where $E$ is mathematical expectation and $X_{\sim i}$ denotes all parameters except $i^\text{th}$. Under the assumption that the input parameters are independent, total variance $V(Y)$ of the crop yield can be decomposed as follows:
\begin{equation*}
V(Y) = \sum_i^p V_i + \sum_i^p\sum_{j>i}^p V_{ij} + \dots + V_{12\dots p},
\end{equation*}
where partial variances are
\begin{equation*}
\begin{split}
V_i &= V[f_i(X_i)] = V_{X_i}\left[E_{X_{\sim i}}(Y | X_i)\right],\\
V_{ij} &= V[f_{ij}(X_i,X_j)] = V_{X_iX_j}\left[E_{X_{\sim ij}}(Y | X_i, X_j)\right] - V_i - V_j,\\
\dots
\end{split}
\end{equation*}

## Sobol index (first order, second order, total index)

This way, sensitivity indices (SI) can be introduced as
\begin{equation}
\Large
S_i = \frac{V_i}{V(Y)},~S_{ij} = \frac{V_{ij}}{V(Y)},~\dots
\end{equation}
In order to incorporate all of the interactions for a particular parameter, one can compute the total effect index:
\begin{equation}
S_{T_i} = \frac{E_{X_{\sim i}}\left[V_{X_i}(Y|X_{\sim i})\right]}{V(Y)} = 1 - \frac{V_{X_{\sim i}}\left[E_{X_i}(Y | X_{\sim i})\right]}{V(Y)}
\end{equation}

From this assumption we can conclude:
\begin{equation}
\Large
0 \leq S_i \leq S_{T_i} \leq 1
\end{equation}

## Sobol’ Sequences versus Random numbers and regular grid

## Modern Quasi-Random Sequences

More -
* [Wiki](https://en.wikipedia.org/wiki/Sobol_sequence)
* [Habr](https://habr.com/ru/post/440892/)
* Feature selection [Skoltech ML 2020](https://github.com/adasegroup/ML2020_lectures/blob/master/lecture9/Lecture_9_Model_Feature_Selection_Sensitivity.pdf)

# Sensitivity analysis of WOFOST model

## Install modules

```python
  from SALib.sample import saltelli
  from SALib.analyze import sobol
  from SALib.test_functions import Ishigami
  import numpy as np
  import pandas as pd
```

## Parameters

```python
NPK = {
    'num_vars':3,
    'names':['N_kg', 'P_kg', 'K_kg'],
    'bounds':[[30., 60.],
              [60., 90.],
              [100., 130.]]
}
```

```python
Soil_parameters = {
  'num_vars':5,
    'names':['SMV', 'SMFCF', 'SM0', 'CRAIRC', 'K0'],
    'bounds':[[0.7, 1.3],
              [0.1, 0.5],
              [0.2, 0.6],
             [0.04, 0.08],
             [22.5, 27.5]]}
```

## Generate input parameters

```python
param_values = saltelli.sample(Soil_parameters, 10)
```

$n = N \times (D \times 2 +2)$

```python
param_values.shape
```

    (120, 5)

## Loop for yield prediction

```python
from pcse.fileinput import YAMLAgroManagementReader
agromanagement_file = os.path.join(data_dir, 'agro', './sugarbeet_calendar.agro')
agromanagement = YAMLAgroManagementReader(agromanagement_file)
#print(agromanagement)
Soil_parameters = {
  'num_vars':5,
    'names':['SMV', 'SMFCF', 'SM0', 'CRAIRC', 'K0'],
    'bounds':[[0.7, 1.3],
              [0.1, 0.5],
              [0.2, 0.6],
             [0.04, 0.08],
             [22.5, 27.5]]}
param_values = saltelli.sample(Soil_parameters, N=10, calc_second_order=True)
```

Soil parameters in [PCSE model](https://pcse.readthedocs.io/en/stable/code.html?highlight=K0#pcse.soil.WaterbalanceFD)

```python
def sensitivity_soil(soil_parameters):
    SMV, SMFCF, SM0, CRAIRC, K0 = soil_parameters
    soildata['SMV'] = SMV
    soildata['SMFCF'] = SMFCF
    soildata['SM0'] = SM0
    soildata['CRAIRC'] = CRAIRC
    soildata['K0'] = K0
    parameters = ParameterProvider(cropdata=cropdata, soildata=soildata, sitedata=sitedata)
    wofsim = Wofost71_WLP_FD(parameters, wdp, agromanagement)
    wofsim.run_till_terminate()
    #df_results = pd.DataFrame(wofsim.get_output())
    #df_results = df_results.set_index("day")
    #df_results.tail()
    summary_output = wofsim.get_summary_output()
    yield_list.append(summary_output[0]['TWSO'])
```

```python
%%time
yield_list = []
param_values = saltelli.sample(Soil_parameters, 10, calc_second_order=True)
for step in range(len(param_values)):
    sensitivity_soil(param_values[step])
    print(param_values[step])
```

    [ 0.83183594  0.13867188  0.40742187  0.06707031 23.90136719]
    [ 1.24433594  0.13867188  0.40742187  0.06707031 23.90136719]
    [ 0.83183594  0.11835938  0.40742187  0.06707031 23.90136719]
    [ 0.83183594  0.13867188  0.55976563  0.06707031 23.90136719]
    [ 0.83183594  0.13867188  0.40742187  0.06003906 23.90136719]
    [ 0.83183594  0.13867188  0.40742187  0.06707031 22.84667969]
    [ 0.83183594  0.11835938  0.55976563  0.06003906 22.84667969]
    [ 1.24433594  0.13867188  0.55976563  0.06003906 22.84667969]
    [ 1.24433594  0.11835938  0.40742187  0.06003906 22.84667969]
    [ 1.24433594  0.11835938  0.55976563  0.06707031 22.84667969]
    [ 1.24433594  0.11835938  0.55976563  0.06003906 23.90136719]
    [ 1.24433594  0.11835938  0.55976563  0.06003906 22.84667969]
    [ 1.13183594  0.33867188  0.20742188  0.04707031 26.40136719]
    [ 0.94433594  0.33867188  0.20742188  0.04707031 26.40136719]
    [ 1.13183594  0.31835938  0.20742188  0.04707031 26.40136719]
    [ 1.13183594  0.33867188  0.35976562  0.04707031 26.40136719]
    [ 1.13183594  0.33867188  0.20742188  0.04003906 26.40136719]
    [ 1.13183594  0.33867188  0.20742188  0.04707031 25.34667969]
    [ 1.13183594  0.31835938  0.35976562  0.04003906 25.34667969]
    [ 0.94433594  0.33867188  0.35976562  0.04003906 25.34667969]
    [ 0.94433594  0.31835938  0.20742188  0.04003906 25.34667969]
    [ 0.94433594  0.31835938  0.35976562  0.04707031 25.34667969]
    [ 0.94433594  0.31835938  0.35976562  0.04003906 26.40136719]
    [ 0.94433594  0.31835938  0.35976562  0.04003906 25.34667969]
    [ 1.28183594  0.23867188  0.50742187  0.07707031 25.15136719]
    [ 0.79433594  0.23867188  0.50742187  0.07707031 25.15136719]
    [ 1.28183594  0.21835938  0.50742187  0.07707031 25.15136719]
    [ 1.28183594  0.23867188  0.25976562  0.07707031 25.15136719]
    [ 1.28183594  0.23867188  0.50742187  0.05003906 25.15136719]
    [ 1.28183594  0.23867188  0.50742187  0.07707031 26.59667969]
    [ 1.28183594  0.21835938  0.25976562  0.05003906 26.59667969]
    [ 0.79433594  0.23867188  0.25976562  0.05003906 26.59667969]
    [ 0.79433594  0.21835938  0.50742187  0.05003906 26.59667969]
    [ 0.79433594  0.21835938  0.25976562  0.07707031 26.59667969]
    [ 0.79433594  0.21835938  0.25976562  0.05003906 25.15136719]
    [ 0.79433594  0.21835938  0.25976562  0.05003906 26.59667969]
    [ 0.98183594  0.43867188  0.30742188  0.05707031 22.65136719]
    [ 1.09433594  0.43867188  0.30742188  0.05707031 22.65136719]
    [ 0.98183594  0.41835937  0.30742188  0.05707031 22.65136719]
    [ 0.98183594  0.43867188  0.45976563  0.05707031 22.65136719]
    [ 0.98183594  0.43867188  0.30742188  0.07003906 22.65136719]
    [ 0.98183594  0.43867188  0.30742188  0.05707031 24.09667969]
    [ 0.98183594  0.41835937  0.45976563  0.07003906 24.09667969]
    [ 1.09433594  0.43867188  0.45976563  0.07003906 24.09667969]
    [ 1.09433594  0.41835937  0.30742188  0.07003906 24.09667969]
    [ 1.09433594  0.41835937  0.45976563  0.05707031 24.09667969]
    [ 1.09433594  0.41835937  0.45976563  0.07003906 22.65136719]
    [ 1.09433594  0.41835937  0.45976563  0.07003906 24.09667969]
    [ 0.90683594  0.28867188  0.25742188  0.05207031 23.27636719]
    [ 1.16933594  0.28867188  0.25742188  0.05207031 23.27636719]
    [ 0.90683594  0.26835938  0.25742188  0.05207031 23.27636719]
    [ 0.90683594  0.28867188  0.20976563  0.05207031 23.27636719]
    [ 0.90683594  0.28867188  0.25742188  0.05503906 23.27636719]
    [ 0.90683594  0.28867188  0.25742188  0.05207031 25.97167969]
    [ 0.90683594  0.26835938  0.20976563  0.05503906 25.97167969]
    [ 1.16933594  0.28867188  0.20976563  0.05503906 25.97167969]
    [ 1.16933594  0.26835938  0.25742188  0.05503906 25.97167969]
    [ 1.16933594  0.26835938  0.20976563  0.05207031 25.97167969]
    [ 1.16933594  0.26835938  0.20976563  0.05503906 23.27636719]
    [ 1.16933594  0.26835938  0.20976563  0.05503906 25.97167969]
    [ 1.20683594  0.48867187  0.45742187  0.07207031 25.77636719]
    [ 0.86933594  0.48867187  0.45742187  0.07207031 25.77636719]
    [ 1.20683594  0.46835938  0.45742187  0.07207031 25.77636719]
    [ 1.20683594  0.48867187  0.40976563  0.07207031 25.77636719]
    [ 1.20683594  0.48867187  0.45742187  0.07503906 25.77636719]
    [ 1.20683594  0.48867187  0.45742187  0.07207031 23.47167969]
    [ 1.20683594  0.46835938  0.40976563  0.07503906 23.47167969]
    [ 0.86933594  0.48867187  0.40976563  0.07503906 23.47167969]
    [ 0.86933594  0.46835938  0.45742187  0.07503906 23.47167969]
    [ 0.86933594  0.46835938  0.40976563  0.07207031 23.47167969]
    [ 0.86933594  0.46835938  0.40976563  0.07503906 25.77636719]
    [ 0.86933594  0.46835938  0.40976563  0.07503906 23.47167969]
    [ 1.05683594  0.18867188  0.35742188  0.04207031 27.02636719]
    [ 0.71933594  0.18867188  0.35742188  0.04207031 27.02636719]
    [ 1.05683594  0.16835938  0.35742188  0.04207031 27.02636719]
    [ 1.05683594  0.18867188  0.50976562  0.04207031 27.02636719]
    [ 1.05683594  0.18867188  0.35742188  0.06503906 27.02636719]
    [ 1.05683594  0.18867188  0.35742188  0.04207031 24.72167969]
    [ 1.05683594  0.16835938  0.50976562  0.06503906 24.72167969]
    [ 0.71933594  0.18867188  0.50976562  0.06503906 24.72167969]
    [ 0.71933594  0.16835938  0.35742188  0.06503906 24.72167969]
    [ 0.71933594  0.16835938  0.50976562  0.04207031 24.72167969]
    [ 0.71933594  0.16835938  0.50976562  0.06503906 27.02636719]
    [ 0.71933594  0.16835938  0.50976562  0.06503906 24.72167969]
    [ 0.75683594  0.38867188  0.55742187  0.06207031 24.52636719]
    [ 1.01933594  0.38867188  0.55742187  0.06207031 24.52636719]
    [ 0.75683594  0.36835938  0.55742187  0.06207031 24.52636719]
    [ 0.75683594  0.38867188  0.30976562  0.06207031 24.52636719]
    [ 0.75683594  0.38867188  0.55742187  0.04503906 24.52636719]
    [ 0.75683594  0.38867188  0.55742187  0.06207031 27.22167969]
    [ 0.75683594  0.36835938  0.30976562  0.04503906 27.22167969]
    [ 1.01933594  0.38867188  0.30976562  0.04503906 27.22167969]
    [ 1.01933594  0.36835938  0.55742187  0.04503906 27.22167969]
    [ 1.01933594  0.36835938  0.30976562  0.06207031 27.22167969]
    [ 1.01933594  0.36835938  0.30976562  0.04503906 24.52636719]
    [ 1.01933594  0.36835938  0.30976562  0.04503906 27.22167969]
    [ 0.73808594  0.17617188  0.21992188  0.05832031 25.62011719]
    [ 0.85058594  0.17617188  0.21992188  0.05832031 25.62011719]
    [ 0.73808594  0.48085937  0.21992188  0.05832031 25.62011719]
    [ 0.73808594  0.17617188  0.52226562  0.05832031 25.62011719]
    [ 0.73808594  0.17617188  0.21992188  0.04128906 25.62011719]
    [ 0.73808594  0.17617188  0.21992188  0.05832031 24.56542969]
    [ 0.73808594  0.48085937  0.52226562  0.04128906 24.56542969]
    [ 0.85058594  0.17617188  0.52226562  0.04128906 24.56542969]
    [ 0.85058594  0.48085937  0.21992188  0.04128906 24.56542969]
    [ 0.85058594  0.48085937  0.52226562  0.05832031 24.56542969]
    [ 0.85058594  0.48085937  0.52226562  0.04128906 25.62011719]
    [ 0.85058594  0.48085937  0.52226562  0.04128906 24.56542969]
    [ 1.03808594  0.37617188  0.41992188  0.07832031 23.12011719]
    [ 1.15058594  0.37617188  0.41992188  0.07832031 23.12011719]
    [ 1.03808594  0.28085938  0.41992188  0.07832031 23.12011719]
    [ 1.03808594  0.37617188  0.32226562  0.07832031 23.12011719]
    [ 1.03808594  0.37617188  0.41992188  0.06128906 23.12011719]
    [ 1.03808594  0.37617188  0.41992188  0.07832031 27.06542969]
    [ 1.03808594  0.28085938  0.32226562  0.06128906 27.06542969]
    [ 1.15058594  0.37617188  0.32226562  0.06128906 27.06542969]
    [ 1.15058594  0.28085938  0.41992188  0.06128906 27.06542969]
    [ 1.15058594  0.28085938  0.32226562  0.07832031 27.06542969]
    [ 1.15058594  0.28085938  0.32226562  0.06128906 23.12011719]
    [ 1.15058594  0.28085938  0.32226562  0.06128906 27.06542969]
    CPU times: user 1min 21s, sys: 229 ms, total: 1min 21s
    Wall time: 1min 21s

```python
np_yield = np.array(yield_list)
Si = sobol.analyze(Soil_parameters, np_yield, print_to_console=False)
```

```python
Si_dict = dict(Si)
Si_df = pd.DataFrame()
Si_df = Si_df.append(pd.Series(Si_dict['S1']), ignore_index=True)
Si_df = Si_df.append(pd.Series(Si_dict['ST']), ignore_index=True)
Si_df = Si_df.append(pd.Series(Si_dict['S1_conf']), ignore_index=True)
Si_df = Si_df.append(pd.Series(Si_dict['ST_conf']), ignore_index=True)
Si_df = Si_df.T
Si_df.columns = ['Si', 'ST', 'Si_conf', 'ST_conf']
Si_df.rename(index={0:'SMV',1:'SMFCF', 2:'SM0', 3:'CRAIRC', 4:'K0'}, inplace=True)
Si_df
```

      Si
      ST
      Si_conf
      ST_conf

      SMV
      0.000000
      0.000000
      0.000000
      0.000000

      SMFCF
      -0.077417
      0.284878
      0.839456
      0.638290

      SM0
      0.267504
      0.381437
      0.693409
      0.688329

      CRAIRC
      0.000000
      0.000000
      0.000000
      0.000000

      K0
      0.000000
      0.000000
      0.000000
      0.000000

Is it ok?

\begin{equation}
\Large
0 \leq S_i \leq S_{T_i} \leq 1
\end{equation}

### For 5 years

```python
def sensitivity_weather(year):
    K_kg = 60
    P_kg = 60
    N_kg = 120
    year_date=year
    print(year)
    print(year_date)
    yaml_agro = f"""
    - {year_date}-06-01:
        CropCalendar:
            crop_name: 'sugar-beet'
            variety_name: 'sugar-beet-601'
            crop_start_date: {year_date}-06-02
            crop_start_type: emergence
            crop_end_date: {year_date}-10-15
            crop_end_type: harvest
            max_duration: 300
        TimedEvents:
        -   event_signal: apply_npk
            name:  Timed N/P/K application table
            comment: All fertilizer amounts in kg/ha
            events_table:
            - {year_date}-06-22: {{N_amount : {N_kg}, P_amount: {P_kg}, K_amount: {K_kg}}}
        StateEvents: null
        """
    agromanagement = yaml.safe_load(yaml_agro)
    parameters = ParameterProvider(cropdata=cropdata, soildata=soildata, sitedata=sitedata)
    wofsim = Wofost71_WLP_FD(parameters, moscow_weather, agromanagement)
    wofsim.run_till_terminate()
    summary_output = wofsim.get_summary_output()
    yield_list_weather.append(summary_output[0]['TWSO'])
```

## Visualizing simulation results

Finally, we can generate some figures of WOFOST variables such as the development (DVS), total biomass (TAGP), leaf area index (LAI) and root-zone soil moisture (SM) using the MatPlotLib plotting package:

```python
fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(12,10))
for var, ax in zip(["DVS", "TWSO", "LAI", "SM"], axes.flatten()):
    ax.plot_date(df_results.index, df_results[var], 'b-')
    ax.set_title(var)
fig.autofmt_xdate()
```

# Visualization for sensitivity analysis

Plots by [Water programming group](https://waterprogramming.wordpress.com/2019/08/27/a-python-implementation-of-grouped-radial-convergence-plots-to-visualize-sobol-sensitivity-analysis-results/)

How to repeat: [Repo of SampleVIS](https://github.com/charlesrouge/SampleVis)

```python
import numpy as np
from SALib.analyze import sobol
from SALib.sample import saltelli
from fishery import fish_game
import matplotlib.pyplot as plt
import itertools
import math
```

### Why number of samples is important?

```python
# Set up dictionary with system parameters
problem = {
  'num_vars': 6,
  'names': ['a', 'b', 'c','h',
            'K','m'],
  'bounds': [[ 0.002, 2],
             [0.005, 1],
             [0.2, 1],
             [0.001, 1],
             [100, 5000],
             [0.1, 1.5]]
}

# Array with n's to use
nsamples = np.arange(50, 500, 50)

# Arrays to store the index estimates
S1_estimates = np.zeros([problem['num_vars'],len(nsamples)])
ST_estimates = np.zeros([problem['num_vars'],len(nsamples)])

# Loop through all n values, create sample, evaluate model and estimate S1 & ST
for i in range(len(nsamples)):
    print('n= '+ str(nsamples[i]))
    # Generate samples
    sampleset = saltelli.sample(problem, nsamples[i],calc_second_order=False)
    # Run model for all samples
    output = [fish_game(*sampleset[j,:]) for j in range(len(sampleset))]
    # Perform analysis
    results = sobol.analyze(problem, np.asarray(output), calc_second_order=False,print_to_console=False)
    # Store estimates
    ST_estimates[:,i]=results['ST']
    S1_estimates[:,i]=results['S1']

np.save('ST_estimates.npy', ST_estimates)
np.save('S1_estimates.npy', S1_estimates)

S1_estimates = np.load('S1_estimates.npy')
ST_estimates = np.load('ST_estimates.npy')

# Generate figure showing evolution of indices
fig =  plt.figure(figsize=(18,9))
ax1 = fig.add_subplot(1,2,1)
handles = []
for j in range(problem['num_vars']):
    handles += ax1.plot(nsamples, S1_estimates[j,:], linewidth=5)
ax1.set_title('Evolution of S1 index estimates', fontsize=20)
ax1.set_ylabel('S1', fontsize=18)
ax1.set_xlabel('Number of samples (n)', fontsize=18)
ax1.tick_params(axis='both', which='major', labelsize=14)
ax2 = fig.add_subplot(1,2,2)
for j in range(problem['num_vars']):
    ax2.plot(nsamples, ST_estimates[j,:], linewidth=5)
ax2.set_title('Evolution of ST index estimates', fontsize=20)
ax2.set_ylabel('ST', fontsize=18)
ax2.tick_params(axis='both', which='major', labelsize=14)
ax2.set_xlabel('Number of samples (n)', fontsize=18)
fig.legend(handles, problem['names'], loc = 'right', fontsize=11)
plt.show()
#plt.savefig('indexevolution.png')

# Calculate parameter rankings
S1_ranks = np.zeros_like(S1_estimates)
ST_ranks = np.zeros_like(ST_estimates)
for i in range(len(nsamples)):
    orderS1 = np.argsort(S1_estimates[:,i])
    orderST = np.argsort(ST_estimates[:,i])
    S1_ranks[:,i] = orderS1.argsort()
    ST_ranks[:,i] = orderST.argsort()

# Generate figure showing evolution of ranks
fig =  plt.figure(figsize=(18,9))
ax1 = fig.add_subplot(1,2,1)
handles = []
for j in range(problem['num_vars']):
    handles += ax1.plot(nsamples, S1_ranks[j,:], linewidth=3)
ax1.set_title('Parameter ranking based on S1', fontsize=20)
ax1.set_ylabel('S1', fontsize=18)
ax1.set_xlabel('Number of samples (n)', fontsize=18)
ax1.set_yticklabels(np.arange(problem['num_vars']+1, 0, -1))
ax1.tick_params(axis='both', which='major', labelsize=14)
ax2 = fig.add_subplot(1,2,2)
for j in range(problem['num_vars']):
    ax2.plot(nsamples, ST_ranks[j,:], linewidth=3)
ax2.set_title('Parameter ranking based on ST', fontsize=20)
ax2.set_ylabel('ST', fontsize=18)
ax2.set_yticklabels(np.arange(problem['num_vars']+1, 0, -1))
ax2.tick_params(axis='both', which='major', labelsize=14)
ax2.set_xlabel('Number of samples (n)', fontsize=18)
fig.legend(handles, problem['names'], loc = 'right', fontsize=14)
#plt.show()
#plt.savefig('rankingevolution.png')
```

## Radial plot for SA

```python
import numpy as np
import itertools
import matplotlib.pyplot as plt
import seaborn as sns
import math
from numpy import genfromtxt
import matplotlib.patches as mpatches
import matplotlib.pyplot as plt
sns.set_style('whitegrid', {'axes_linewidth': 0, 'axes.edgecolor': 'white'})
```

## Plot function

```python
def is_significant(value, confidence_interval, threshold="conf"):
    if threshold == "conf":
        return value - abs(confidence_interval) > 0
    else:
        return value - abs(float(threshold)) > 0

def grouped_radial(SAresults, parameters, radSc=2.0, scaling=1, widthSc=0.5, STthick=1, varNameMult=1.3, colors=None, groups=None, gpNameMult=1.5, threshold="conf"):
    # Derived from https://github.com/calvinwhealton/SensitivityAnalysisPlots
    fig, ax = plt.subplots(1, 1)
    color_map = {}

    # initialize parameters and colors
    if groups is None:

        if colors is None:
            colors = ["k"]

        for i, parameter in enumerate(parameters):
            color_map[parameter] = colors[i % len(colors)]
    else:
        if colors is None:
            colors = sns.color_palette("deep", max(3, len(groups)))

        for i, key in enumerate(groups.keys()):
            #parameters.extend(groups[key])

            for parameter in groups[key]:
                color_map[parameter] = colors[i % len(colors)]

    n = len(parameters)
    angles = radSc*math.pi*np.arange(0, n)/n
    x = radSc*np.cos(angles)
    y = radSc*np.sin(angles)

    # plot second-order indices
    for i, j in itertools.combinations(range(n), 2):
        #key1 = parameters[i]
        #key2 = parameters[j]

        if is_significant(SAresults["S2"][i][j], SAresults["S2_conf"][i][j], threshold):
            angle = math.atan((y[j]-y[i])/(x[j]-x[i]))

            if y[j]-y[i] < 0:
                angle += math.pi

            line_hw = scaling*(max(0, SAresults["S2"][i][j])**widthSc)/2

            coords = np.empty((4, 2))
            coords[0, 0] = x[i] - line_hw*math.sin(angle)
            coords[1, 0] = x[i] + line_hw*math.sin(angle)
            coords[2, 0] = x[j] + line_hw*math.sin(angle)
            coords[3, 0] = x[j] - line_hw*math.sin(angle)
            coords[0, 1] = y[i] + line_hw*math.cos(angle)
            coords[1, 1] = y[i] - line_hw*math.cos(angle)
            coords[2, 1] = y[j] - line_hw*math.cos(angle)
            coords[3, 1] = y[j] + line_hw*math.cos(angle)

            ax.add_artist(plt.Polygon(coords, color="0.75"))

    # plot total order indices
    for i, key in enumerate(parameters):
        if is_significant(SAresults["ST"][i], SAresults["ST_conf"][i], threshold):
            ax.add_artist(plt.Circle((x[i], y[i]), scaling*(SAresults["ST"][i]**widthSc)/2, color='w'))
            ax.add_artist(plt.Circle((x[i], y[i]), scaling*(SAresults["ST"][i]**widthSc)/2, lw=STthick, color='0.4', fill=False))

    # plot first-order indices
    for i, key in enumerate(parameters):
        if is_significant(SAresults["S1"][i], SAresults["S1_conf"][i], threshold):
            ax.add_artist(plt.Circle((x[i], y[i]), scaling*(SAresults["S1"][i]**widthSc)/2, color='0.4'))

    # add labels
    for i, key in enumerate(parameters):
        ax.text(varNameMult*x[i], varNameMult*y[i], key, ha='center', va='center',
                rotation=angles[i]*360/(2*math.pi) - 90,
                color=color_map[key])

    if groups is not None:
        for i, group in enumerate(groups.keys()):
            print(group)
            group_angle = np.mean([angles[j] for j in range(n) if parameters[j] in groups[group]])

            ax.text(gpNameMult*radSc*math.cos(group_angle), gpNameMult*radSc*math.sin(group_angle), group, ha='center', va='center',
                rotation=group_angle*360/(2*math.pi) - 90,
                color=colors[i % len(colors)])

    ax.set_facecolor('white')
    ax.set_xticks([])
    ax.set_yticks([])
#     ax.
    plt.axis('equal')
    plt.axis([-2*radSc, 2*radSc, -2*radSc, 2*radSc])
    #plt.show()

    return fig
```

## Range of soil parameters

```python
problem = {
    'num_vars':6,
    'names':['SOC', 'Sand', 'Clay', 'pH', 'CN', 'BD'],
    'bounds':[[2.58, 6.20],
              [0.01, 0.30],
              [0.01, 0.30],
              [4.6, 6.9],
              [10.9, 12.4],
              [900, 1350]]
}
```

```python
#names for csv files
list_of_csv=['soybean-000-2015.csv', 'sugar-beet-2011.csv', 'sugar-beet-2017.csv',
'spring-barley-2012.csv', 'sugar-beet-2014.csv']
list_of_names=['soybean-000-2015', 'sugar-beat-2011', 'sugar-beat-2017',
'spring-barley-2012', 'sugar-beat-2014']
list_of_totals=['total_SI_'+x for x in list_of_names]
list_of_first=['fisrt_SI_'+x for x in list_of_names]
list_of_second=['second_SI_'+x for x in list_of_names]
list_of_SI=['SI_'+x for x in list_of_names]
```

```python
for j, i in enumerate(list_of_csv):
        all_data_csv = genfromtxt('./'+str(i), delimiter=',')
        output = all_data_csv[:,2]
        print(i)
        list_of_SI[j] = sobol.analyze(problem, output, calc_second_order=True, conf_level=0.95, print_to_console=False)
```

    soybean-000-2015.csv
    sugar-beet-2011.csv
    sugar-beet-2017.csv
    spring-barley-2012.csv
    sugar-beet-2014.csv

```python
groups={"Soil physics" : ["Sand", "Clay", "BD"],
        "Soil chemistry" : ["pH", "SOC", "CN"]}

fig = grouped_radial(list_of_SI[4], ['BD', 'Sand', 'Clay', 'pH', 'CN', 'SOC'], groups=groups, threshold=0.001)
red_patch = mpatches.Patch(color='red', label='The red data')
plt.title(list_of_names[4], loc='left')

plt.show()
```

## Homework

__[Tasks](https://skoltech-my.sharepoint.com/:w:/g/personal/mikhail_gasanov_skoltech_ru/EeTPQxbrzVdPqnSENKYyoTUBay1RDYgMMW3GO3qFT2ge5g?e=4hk45V)__

Usefull

__SA and UQ__

1) [Rhodium project](https://github.com/Project-Platypus/Rhodium.git)

2) [SALib](https://github.com/SALib/SALib)

__Model__

3) [PCSE](https://pcse.readthedocs.io/en/stable/index.html)

4) How to install PCSE at local machine
    `conda env create -f` [py3_pcse.yml](https://github.com/mishagrol/Seminar_Sobol/blob/master/py3_pcse.yml)

Аny questions  -

Telegram - `@misha_grol`

Part 1 – Crop Yield Prediction (PCSE, MONICA)

You can use the seminars’ colab:

“https://colab.research.google.com/drive/1j4AHD8KkTRThPuNsQzQFWYSJtptQ6bUA”

1) Assess the yield of one of the crops for the Moscow region over several years (potatoes, sugar beets for 2-3 years)

    Crop  - (https://github.com/mishagrol/Seminar_Sobol/tree/master/data/crop)

    Soil - (https://github.com/mishagrol/Seminar_Sobol/tree/master/data/soil)

    Weather  - NASAdataprovider in PCSE (https://pcse.readthedocs.io/en/stable/code.html?highlight=NASA#pcse.db.NASAPowerWeatherDataProvider)

    Agromanagement - (https://github.com/mishagrol/Seminar_Sobol/blob/master/data/agro/sugarbeet_calendar_Moscow_short.agro)

Part 2 – Sensitivity Analysis (SALib)

1) Perform sensitivity analysis of one of the model blocks (crop, soil, agromanagement *) with SALib. You can choose one of the methods that you consider necessary (Sobol, FAST, ...).

    Generate samples – In report provide the size of the resulting matrix and the sample size (N)

    Conduct parameter sensitivity analysis  - In report provide S1 and ST indices.

2) Generate plots (Hist, Radial convergence plot, etc.)

*3) Estimate the required number of simulations to obtain reliable values of the sensitivity indices. Try to estimate the sample size at the confidence interval of the sensitivity indices.

* Please note that working with discrete data can cause certain difficulties.

#Agro Hack https://agro-code.ru/

### bonus

__Morris method__

Generate a sample using the Method of Morris

Three variants of Morris' sampling for elementary effects is supported:

- Vanilla Morris
- Optimised trajectories when ``optimal_trajectories=True`` (using
    Campolongo's enhancements from 2007 and optionally Ruano's enhancement
    from 2012; ``local_optimization=True``)
- Groups with optimised trajectories when ``optimal_trajectories=True`` and
    the problem definition specifies groups (note that ``local_optimization``
    must be ``False``)

At present, optimised trajectories is implemented using either a brute-force
approach, which can be very slow, especially if you require more than four
trajectories, or a local method based which is much faster. Both methods now
implement working with groups of factors.

Note that the number of factors makes little difference,
but the ratio between number of optimal trajectories and the sample size
results in an exponentially increasing number of scores that must be
computed to find the optimal combination of trajectories.  We suggest going
no higher than 4 from a pool of 100 samples with the brute force approach.
With local_optimization = True (which is default),
it is possible to go higher than the previously suggested 4 from 100.

```python
import sys

from SALib.analyze import morris
from SALib.sample.morris import sample
from SALib.test_functions import Sobol_G
from SALib.util import read_param_file
from SALib.plotting.morris import horizontal_bar_plot, covariance_plot, \
    sample_histograms
import matplotlib.pyplot as plt

#sys.path.append('../..')

# Read the parameter range file and generate samples
#problem = read_param_file('/Users/mikhailgasanov/Documents/GIT/SALib/src/SALib/test_functions/params/Sobol_G.txt')
# or define manually without a parameter file:
problem = {
 'num_vars': 8,
 'names': ['x1', 'x2', 'x3', 'x4', 'x5', 'x6', 'x7', 'x8'],
 'groups': None,
 'bounds': [[0.0, 1.0],
            [0.0, 1.0],
            [0.0, 1.0],
            [0.0, 1.0],
            [0.0, 1.0],
            [0.0, 1.0],
            [0.0, 1.0],
            [0.0, 1.0]]
}
# Files with a 4th column for "group name" will be detected automatically, e.g.
# param_file = '../../src/SALib/test_functions/params/Ishigami_groups.txt'

# Generate samples
param_values = sample(problem, N=1000, num_levels=4,
                      optimal_trajectories=None)

# To use optimized trajectories (brute force method),
# give an integer value for optimal_trajectories

# Run the "model" -- this will happen offline for external models
Y = Sobol_G.evaluate(param_values)

# Perform the sensitivity analysis using the model output
# Specify which column of the output file to analyze (zero-indexed)
Si = morris.analyze(problem, param_values, Y, conf_level=0.95,
                    print_to_console=True,
                    num_levels=4, num_resamples=100)
# Returns a dictionary with keys 'mu', 'mu_star', 'sigma', and 'mu_star_conf'
# e.g. Si['mu_star'] contains the mu* value for each parameter, in the
# same order as the parameter file

fig, (ax1, ax2) = plt.subplots(1, 2)
horizontal_bar_plot(ax1, Si, {}, sortby='mu_star', unit=r"tCO$_2$/year")
covariance_plot(ax2, Si, {}, unit=r"tCO$_2$/year")

fig2 = plt.figure()
sample_histograms(fig2, param_values, problem, {'color': 'y'})
plt.show()
```

```python

```


# Training a neural network on QM9

This tutorial will explain how to use SchNetPack for training a model
on the QM9 dataset and how the trained model can be used for further applications.

First, we import the necessary modules and create a new directory for the data and our model.

```python
import os
import schnetpack as spk
from schnetpack.datasets import QM9
import schnetpack.transform as trn

import torch
import torchmetrics
import pytorch_lightning as pl

qm9tut = './qm9tut'
if not os.path.exists('qm9tut'):
    os.makedirs(qm9tut)
```

## Loading the data

As explained in the [previous tutorial](tutorial_01_preparing_data.ipynb), datasets in SchNetPack are loaded with the `AtomsLoader` class or one of the sub-classes that are specialized for common benchmark datasets.
The `QM9` dataset class will download and convert the data. We will only use the inner energy at 0K `U0`, so all other properties do not need to be loaded:

```python
%rm split.npz

qm9data = QM9(
    './qm9.db',
    batch_size=100,
    num_train=1000,
    num_val=1000,
    transforms=[
        trn.ASENeighborList(cutoff=5.),
        trn.RemoveOffsets(QM9.U0, remove_mean=True, remove_atomrefs=True),
        trn.CastTo32()
    ],
    property_units={QM9.U0: 'eV'},
    num_workers=1,
    split_file=os.path.join(qm9tut, "split.npz"),
    pin_memory=True, # set to false, when not using a GPU
    load_properties=[QM9.U0], #only load U0 property
)
qm9data.prepare_data()
qm9data.setup()
```

    rm: cannot remove 'split.npz': No such file or directory

    /home/kschuett/anaconda3/envs/spkdev/lib/python3.8/site-packages/pytorch_lightning/core/datamodule.py:73: LightningDeprecationWarning: DataModule property `train_transforms` was deprecated in v1.5 and will be removed in v1.7.
      rank_zero_deprecation(
    /home/kschuett/anaconda3/envs/spkdev/lib/python3.8/site-packages/pytorch_lightning/core/datamodule.py:77: LightningDeprecationWarning: DataModule property `val_transforms` was deprecated in v1.5 and will be removed in v1.7.
      rank_zero_deprecation(
    /home/kschuett/anaconda3/envs/spkdev/lib/python3.8/site-packages/pytorch_lightning/core/datamodule.py:81: LightningDeprecationWarning: DataModule property `test_transforms` was deprecated in v1.5 and will be removed in v1.7.
      rank_zero_deprecation(
    /home/kschuett/anaconda3/envs/spkdev/lib/python3.8/site-packages/pytorch_lightning/core/datamodule.py:114: LightningDeprecationWarning: DataModule property `train_transforms` was deprecated in v1.5 and will be removed in v1.7.
      rank_zero_deprecation(
    100%|██████████| 10/10 [00:14<00:00,  1.40s/it]
    /home/kschuett/anaconda3/envs/spkdev/lib/python3.8/site-packages/pytorch_lightning/core/datamodule.py:133: LightningDeprecationWarning: DataModule property `val_transforms` was deprecated in v1.5 and will be removed in v1.7.
      rank_zero_deprecation(
    /home/kschuett/anaconda3/envs/spkdev/lib/python3.8/site-packages/pytorch_lightning/core/datamodule.py:152: LightningDeprecationWarning: DataModule property `test_transforms` was deprecated in v1.5 and will be removed in v1.7.
      rank_zero_deprecation(

The dataset is downloaded and partitioned automatically. PyTorch `DataLoader`s can be obtained using `qm9data.train_dataloader()`, `qm9data.val_dataloader()` and `qm9data.test_dataloader()`.

Before building the model, we remove offsets from the energy for good initial conditions. We will get this from the training dataset. Above, this is done automatically by the `RemoveOffsets` transform.
In the following we show what happens under the hood.
For QM9, we also have single-atom reference values stored in the metadata:

```python
atomrefs = qm9data.train_dataset.atomrefs
print('U0 of hyrogen:', atomrefs[QM9.U0][1].item(), 'eV')
print('U0 of carbon:', atomrefs[QM9.U0][6].item(), 'eV')
print('U0 of oxygen:', atomrefs[QM9.U0][8].item(), 'eV')
```

    U0 of hyrogen: -13.613121032714844 eV
    U0 of carbon: -1029.863037109375 eV
    U0 of oxygen: -2042.611083984375 eV

These can be used together with the mean and standard deviation of the energy per atom to initialize the model with a good guess of the energy of a molecule. When calculating these statistics, we pass the atomref to take into account, that the model will add these atomrefs to the predicted energy later, so that this part of the energy does not have to be considered in the statistics, i.e.
\begin{equation}
\mu_{U_0} = \frac{1}{n_\text{train}} \sum_{n=1}^{n_\text{train}} \left( U_{0,n} - \sum_{i=1}^{n_{\text{atoms},n}} U_{0,Z_{n,i}} \right)
\end{equation}
for the mean and analogously for the standard deviation. In this case, this corresponds to the mean and std. dev of the *atomization energy* per atom.

```python
means, stddevs = qm9data.get_stats(
    QM9.U0, divide_by_atoms=True, remove_atomref=True
)
print('Mean atomization energy / atom:', means.item())
print('Std. dev. atomization energy / atom:', stddevs.item())
```

    Mean atomization energy / atom: -4.247325399125455
    Std. dev. atomization energy / atom: 0.18015809859127724

## Setting up the model

Next, we need to build the model and define how it should be trained.

In SchNetPack, a neural network potential usually consists of three parts:

1. A list of input modules that prepare the batched data before the building the representation.
   This includes, e.g., the calculation of pairwise distances between atoms based on neighbor indices or add auxiliary
   inputs for response properties.
2. The representation which either constructs atom-wise features, e.g. with SchNet or PaiNN.
3. One or more output modules for property prediction.

Here, we use the `SchNet` representation with 3 interaction layers, a 5 Angstrom cosine cutoff with pairwise distances
expanded on 20 Gaussians and 50 atomwise features and convolution filters, since we only have a few
training examples. Then, we use an `Atomwise` module to predict the inner energy $U_0$ by summing over atom-wise
energy contributions.

```python
cutoff = 5.
n_atom_basis = 30

pairwise_distance = spk.atomistic.PairwiseDistances() # calculates pairwise distances between atoms
radial_basis = spk.nn.GaussianRBF(n_rbf=20, cutoff=cutoff)
schnet = spk.representation.SchNet(
    n_atom_basis=n_atom_basis, n_interactions=3,
    radial_basis=radial_basis,
    cutoff_fn=spk.nn.CosineCutoff(cutoff)
)
pred_U0 = spk.atomistic.Atomwise(n_in=n_atom_basis, output_key=QM9.U0)

nnpot = spk.model.NeuralNetworkPotential(
    representation=schnet,
    input_modules=[pairwise_distance],
    output_modules=[pred_U0],
    postprocessors=[trn.CastTo64(), trn.AddOffsets(QM9.U0, add_mean=True, add_atomrefs=True)]
)
```

The last argument here is a list of postprocessors that will only be used if `nnpot.inference_mode=True` is set.
It will not be used in training or validation, but only for predictions.
Here, this is used to deal with numerical accuracy and normalization of model outputs:
To make training easier, we have subtracted single atom energies as well as the mean energy per atom
in the preprocessing (see above).
This does not matter for the loss, but for the final prediction we want to get the real energies.
Additionally, we have removed the energy offsets *before* casting to float32 in the preprocessor.
This avoids loss of numerical precision.
Analog to this, we also have to first cast to float64, before re-adding the offsets in the post-processor

The output modules store the prediction in a dictionary under the `output_key` (here: `QM9.U0`), which is connected to
a target property with loss functions and evaluation metrics using the `ModelOutput` class:

```python
output_U0 = spk.task.ModelOutput(
    name=QM9.U0,
    loss_fn=torch.nn.MSELoss(),
    loss_weight=1.,
    metrics={
        "MAE": torchmetrics.MeanAbsoluteError()
    }
)
```

By default, the target is assumed to have the same name as the output. Otherwise, a different `target_name`
has to be provided.
Here, we already gave the output the same name as the target in the dataset (`QM9.U0`).
In case of multiple outputs, the full loss is a weighted sum of all output losses.
Therefore, it is possible to provide a `loss_weight`, which we here just set to 1.

All components defined above are then passed to `AtomisticTask`, which is a sublass of
[`LightningModule`](https://pytorch-lightning.readthedocs.io/en/latest/common/lightning_module.html).
This connects the model and training process and can then be passed to the PyTorch Lightning `Trainer`.

```python
task = spk.task.AtomisticTask(
    model=nnpot,
    outputs=[output_U0],
    optimizer_cls=torch.optim.AdamW,
    optimizer_args={"lr": 1e-4}
)
```

## Training the model

Now, the model is ready for training. Since we already defined all necessary components, the only thing left to do is
passing it to the PyTorch Lightning `Trainer` together with the data module.

Additionally, we can provide callbacks that take care of logging, checkpointing etc.

```python
logger = pl.loggers.TensorBoardLogger(save_dir=qm9tut)
callbacks = [
    spk.train.ModelCheckpoint(
        inference_path=os.path.join(qm9tut, "best_inference_model"),
        save_top_k=1,
        monitor="val_loss"
    )
]

trainer = pl.Trainer(
    callbacks=callbacks,
    logger=logger,
    default_root_dir=qm9tut,
    max_epochs=3, # for testing, we restrict the number of epochs
)
trainer.fit(task, datamodule=qm9data)
```

    GPU available: True, used: False
    TPU available: False, using: 0 TPU cores
    IPU available: False, using: 0 IPUs
    /home/kschuett/anaconda3/envs/spkdev/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1582: UserWarning: GPU available but not used. Set the gpus flag in your trainer `Trainer(gpus=1)` or script `--gpus=1`.
      rank_zero_warn(
    /home/kschuett/anaconda3/envs/spkdev/lib/python3.8/site-packages/pytorch_lightning/core/datamodule.py:469: LightningDeprecationWarning: DataModule.setup has already been called, so it will not be called again. In v1.6 this behavior will change to always call DataModule.setup.
      rank_zero_deprecation(

      | Name    | Type                   | Params
    ---------------------------------------------------
    0 | model   | NeuralNetworkPotential | 16.4 K
    1 | outputs | ModuleList             | 0
    ---------------------------------------------------
    16.4 K    Trainable params
    0         Non-trainable params
    16.4 K    Total params
    0.066     Total estimated model params size (MB)

    Validation sanity check: 0it [00:00, ?it/s]

    /home/kschuett/anaconda3/envs/spkdev/lib/python3.8/site-packages/pytorch_lightning/trainer/data_loading.py:116: UserWarning: The dataloader, val_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 8 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
      rank_zero_warn(
    /home/kschuett/anaconda3/envs/spkdev/lib/python3.8/site-packages/pytorch_lightning/utilities/data.py:59: UserWarning: Trying to infer the `batch_size` from an ambiguous collection. The batch size we found is 100. To avoid any miscalculations, use `self.log(..., batch_size=batch_size)`.
      warning_cache.warn(
    /home/kschuett/anaconda3/envs/spkdev/lib/python3.8/site-packages/pytorch_lightning/trainer/data_loading.py:116: UserWarning: The dataloader, train_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 8 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
      rank_zero_warn(
    /home/kschuett/anaconda3/envs/spkdev/lib/python3.8/site-packages/pytorch_lightning/trainer/data_loading.py:412: UserWarning: The number of training samples (10) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
      rank_zero_warn(

    Training: 0it [00:00, ?it/s]

    Validating: 0it [00:00, ?it/s]

    Validating: 0it [00:00, ?it/s]

    Validating: 0it [00:00, ?it/s]

The `ModelCheckpoint` of SchNetPack is equivalent to that in PyTorch Lightning,
except that we also store the best inference model. We will show how to use this in the next section.

You can have a look at the training log using Tensorboard:
```
tensorboard --logdir=qm9tut/default
```

## Inference

Having trained a model for QM9, we are going to use it to obtain some predictions.
First, we need to load the model. The `Trainer` stores the best model in the model directory which can be loaded using PyTorch:

```python
import torch
import numpy as np
from ase import Atoms

best_model = torch.load(os.path.join(qm9tut, 'best_inference_model'))
```

We can use the test dataloader from the QM( data to obtain a batch of molecules and apply the model:

```python
for batch in qm9data.test_dataloader():
    result = best_model(batch)
    print("Result dictionary:", result)
    break
```

    Result dictionary: {'energy_U0': tensor([11774.3995, 10674.7092, 10336.3100, 11225.4174,  9808.5495, 10313.0941,
            10737.9138, 10822.9320, 10336.9593, 11225.9045,  8719.2882, 11773.0849,
             9763.9021,  9786.9482, 10675.5382, 11837.0301, 12219.8479, 11818.1505,
            11689.9811, 10381.7697, 10780.0383, 10292.8703, 10255.7486, 11354.0601,
            10372.0447, 11328.8663, 10825.3296, 10380.5685, 11374.1378, 10231.7369,
            11774.1743, 12240.5745, 11668.0797, 11331.6547, 12197.7266, 11817.8596,
            11795.9724, 10802.3730, 12643.4294, 12198.9642,  9808.1227, 11203.0182,
            13805.0691, 10360.3273, 11797.6667, 12431.5858, 11796.2583, 10379.8109,
            11370.1223, 10335.8519,  9736.0116, 11225.1377, 11352.5657, 10800.0178,
            10802.7024, 11396.5917, 11351.0837, 12831.2083, 12368.0770, 11372.5354,
            11248.3640, 11795.8115, 13213.4434,  9387.1987, 10781.3485, 12325.7788,
            11668.8270, 10231.1422, 13235.5105,  9319.7437,  5194.6013, 10297.2585,
            11670.1327, 11648.3546, 10358.1942,  9387.4924, 12345.5267, 11639.6398,
            11224.0048,  9552.0740, 12240.1212, 10518.4759, 11754.8642, 11648.4826,
            10328.6224, 11118.9744, 11690.7071, 12197.8641, 11394.3093, 11371.5435,
            10358.5896, 10802.8291, 11246.9137, 10335.4797, 11395.6447, 10780.8476,
            11775.4573, 12791.9836,  9312.3371, 13322.4199], dtype=torch.float64,
           grad_fn=<SubBackward0>)}

If your data is not already in SchNetPack format, a convenient way is to use ASE atoms with the
provided `AtomsConverter` and (optionally) the `SpkCalculator`:

```python
converter = spk.interfaces.AtomsConverter(neighbor_list=trn.ASENeighborList(cutoff=5.), dtype=torch.float32)
calculator = spk.interfaces.SpkCalculator(model=best_model, converter=converter)
```

```python
numbers = np.array([6, 1, 1, 1, 1])
positions = np.array([[-0.0126981359, 1.0858041578, 0.0080009958],
                      [0.002150416, -0.0060313176, 0.0019761204],
                      [1.0117308433, 1.4637511618, 0.0002765748],
                      [-0.540815069, 1.4475266138, -0.8766437152],
                      [-0.5238136345, 1.4379326443, 0.9063972942]])
atoms = Atoms(numbers=numbers, positions=positions)
```

```python
inputs = converter(atoms)

print('Keys:', list(inputs.keys()))

pred = best_model(inputs)

print('Prediction:', pred[QM9.U0])
```

    Keys: ['_n_atoms', '_atomic_numbers', '_positions', '_cell', '_pbc', '_idx_i_local', '_idx_j_local', '_offsets', '_idx_m', '_idx_j', '_idx_i']
    Prediction: tensor([1064.6565], dtype=torch.float64, grad_fn=<SubBackward0>)

Alternatively, one can use the `SpkCalculator` as an interface to ASE:

```python
calculator = spk.interfaces.SpkCalculator(model=best_model, converter=converter, energy=QM9.U0, energy_units="eV")
atoms.set_calculator(calculator)
print('Prediction:', atoms.get_total_energy())
```

    Prediction: 1064.6565392017365

The calculator automatically converts the prediction of the given unit to internal ASE units, which is `eV`
for the energy.
Using the calculator interface makes more sense if you have trained SchNet for a potential energy surface.
In the next tutorials, we will show how to learn potential energy surfaces and forces field as well as performing
molecular dynamics simulations with SchNetPack.