01_bcce_2024.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>\n",
    "    <img src=\"./images/molssi_ai.png\"\n",
    "         alt=\"MolSSI-AI Logo\"\n",
    "         width=400 \n",
    "         height=250\n",
    "    />\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authors\n",
    "\n",
    "* **Mohammad Mostafanejad**, Molecular Sciences Software Institute, Blacksburg, VA, USA\n",
    "* **Ashley Ringer McDonald**, California Polytechnic State University, San Luis Obispo, CA, USA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Learning Objectives\n",
    "- Perform exploratory data analysis (EDA), data preprocessing and featurization\n",
    "- Create and train a linear regression model and apply it to solve a practical cheminformatics problem\n",
    "- Evaluate the performance of the trained models on test data\n",
    "- Determine the importance of cross-validation on the quality of the trained models\n",
    "- Use PyCaret to automate the process of creating, training and evaluating the regression models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we start, let's make sure we have the necessary libraries ready for use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-26T16:51:46.314577Z",
     "start_time": "2024-06-26T16:51:46.301095Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd                 # for data manipulation\n",
    "import seaborn as sns               # for data visualization\n",
    "import matplotlib.pyplot as plt     # for data visualization\n",
    "import numpy as np                  # for numerical operations\n",
    "\n",
    "from sklearn.preprocessing import StandardScaler            # for scaling the data\n",
    "from sklearn.model_selection import train_test_split        # for splitting the data into training and testing sets\n",
    "from sklearn.model_selection import cross_val_score, KFold  # for K-fold cross-validation\n",
    "from sklearn.linear_model import LinearRegression           # for creating a linear regression model\n",
    "from sklearn.dummy import DummyRegressor                    # for creating a base regressor to compare the model with\n",
    "from sklearn.metrics import mean_squared_error, r2_score    # for evaluating the model\n",
    "from sklearn.pipeline import make_pipeline                  # for building operational pipelines\n",
    "\n",
    "import sweetviz as sv               # for automatic EDA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "    <b>Note</b>\n",
    "    We have added comments to clarify the purpose of each imported library.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem Statement"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we will be working on a **supervised learning** problem which involves both featurized data and their corresponding labels. The goal is to predict the solubility of a molecule based on its chemical structure using regression models. We are going to use the [Delaney's solubility dataset](https://doi.org/10.1021/ci034243x). It contains the chemical structures of 1144 compounds along with their experimentally measured solubility in mol/L. We provide a preprocessed version of the [dataset](https://github.com/MolSSI-Education/bcce-2024/tree/main/data) in the comma-separated value (CSV) format.\n",
    "\n",
    "Let's load the solubility data using the ``pandas`` library \n",
    "and take a look at a few samples in the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-26T16:49:13.095479Z",
     "start_time": "2024-06-26T16:49:13.088117Z"
    }
   },
   "outputs": [],
   "source": [
    "# Path to the preprocessed data file\n",
    "data_path = \"./data/solubility-processed.csv\"\n",
    "\n",
    "# Read the data into a DataFrame\n",
    "df = pd.read_csv(data_path,\n",
    "                 dtype={\"MolLogP\": \"float64\", \n",
    "                        \"MolWt\": \"float64\",\n",
    "                        \"NumRotatableBonds\": \"int\",\n",
    "                        \"AromaticProportion\": \"float\",\n",
    "                        \"logS\": \"float64\"\n",
    "                        })\n",
    "\n",
    "# Display the first few rows of the DataFrame\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset contains the following columns:\n",
    "- **MolLogP**: solubility values estimated by [Daylight CLOGP version 4.72](https://www.daylight.com/dayhtml/doc/clogp/index.html)\n",
    "- **MolWt**: molecular weight\n",
    "- **NumRotatableBonds**: number of rotatable bonds\n",
    "- **AromaticProportion**: the portion of heavy atoms that are in an aromatic ring\n",
    "- **logS**: the solubility of the molecule in mol/L measured at 25 $\\degree C$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sidebar: A Brief Note on Solubility"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-26T16:49:44.121462Z",
     "start_time": "2024-06-26T16:49:44.025680Z"
    }
   },
   "source": [
    "The solubility of a molecule is a critical property in medicinal chemistry, drug discovery, and agrochemistry as it determines the bioavailability of a drug or pesticide.\n",
    "**Partition coefficient**, $P$, measures the propensity of a neutral compound to dissolve in a mixture of two immiscible solvents, often water and octanol. In simple terms, it determines how much of a solute dissolves in the aqueous phase versus the organic portion.\n",
    "\n",
    "The partition coefficient is defined as:\n",
    "\n",
    "$$\n",
    "P = \\frac{[Solute]_{\\text{organic}}}{[Solute]_{\\text{aqueous}}}     \n",
    "$$\n",
    "\n",
    "where $[Solute]_{\\text{organic}}$ and $[Solute]_{\\text{aqueous}}$ are the concentrations of the solute in the organic and aqueous phases, respectively.\n",
    "Instead of $P$, we often use the logarithm of the partition coefficient, $\\log P$, which is more convenient to work with. A negative value of $\\log P$ indicates that the solute is more soluble in water, while a positive value indicates that the solute is more soluble in the organic phase. When $\\log P$ is close to zero, the solute is equally soluble in both phases. Although $\\log P$ is a constant value, its magnitude is dependent on the measurement conditions and the choice of the organic solvent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\"> \n",
    "    <b>Exercise</b>\n",
    "    What does logP value of 1 mean?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "source": [
    "#### Solution\n",
    "\n",
    "A $\\log P$ value of 1 indicates that the solute is 10 times more soluble in the organic phase than in the aqueous phase."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploratory Data Analysis (EDA)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step after stating the problem is to perform exploratory data analysis (EDA) on raw data. The EDA is crucial for data preprocessing pipelines as it helps us understand the nature of our data, identify the key patterns and relationships, and detect anomalies. The EDA involves summarizing the main characteristics of the data, often using visual methods. \n",
    "\n",
    "Loading the data into a Pandas DataFrame provides a convenient way to perform EDA. Let's start by getting a high-level overview of the dataset\n",
    "using the ``info`` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    " # Analyse Dataset\n",
    "report = sv.analyze(df)\n",
    "\n",
    "# View and Save\n",
    "report.show_notebook()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking for Missing Values\n",
    "Since the data is already preprocessed, the number of ``Missing`` values for all features should be zero. The experimentally measured solubility in the last column ``logS`` is the target variable and the remaining columns are the features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\"> \n",
    "    <b>Exercise</b>\n",
    "    Are there any missing values in the data set?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Statistical Summary of the Data\n",
    "We also see a statistical summary of each numerical features in our dataset. The provided statistics include the value counts, mean, standard deviation, minimum, 25th percentile, median, average, 75th percentile, and maximum values for each feature.\n",
    "\n",
    "This information is extremely useful for understanding the data and the distribution of the features. It helps in identifying anomalies in the data or if our data requires any preprocessing. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\"> \n",
    "    <b>Exercise</b>\n",
    "    What is the range of the logS values in this dataset?\n",
    "    </div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Skewness\n",
    "The skewness of the data, which is determined by comparing the mean and median values, is also provided.\n",
    "\n",
    "The numerical values of skewness can be interpreted using the following rules:\n",
    "- The skewness value of zero indicates a perfect symmetrical distribution,\n",
    "- a skewness between -0.5 and 0.5 indicates an approximately symmetric distribution,\n",
    "- a skewness between -1 and -0.5 (or 0.5 and 1) indicates a moderately skewed distribution,\n",
    "- a skewness between -1.5 and -1 (or 1 and 1.5) indicates a highly skewed distribution, and\n",
    "- a skewness less than -1.5 (or greater than 1.5) indicates an extremely skewed distribution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\"> \n",
    "    <b>Exercise</b>\n",
    "    Use the skewness rules and the histograms to categorize the skewness of all five properties based on the calculated skewness values.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Correlation of Features with Target Variable\n",
    "The correlation matrix between the features and the target variable provides insights into the relationships between the features and the target variable(s). You may need to click on the \"Associations\" button in order to see the correlation matrix.\n",
    "\n",
    "A correlation value close to 1 indicates a strong positive relationship, while a correlation value close to -1 indicates a strong negative relationship. A correlation value close to 0 indicates no relationship between the features.\n",
    "\n",
    "The coloring scheme makes it easy to uncover the relationships between the features. The darker the color, the stronger the correlation. The diagonal line represents the correlation of each feature with itself, which is always 1. The blue color indicates a positive correlation, while the red color signifies a negative correlation (based on the provided key)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\"> \n",
    "    <b>Exercise</b>\n",
    "    Based on the correlation plot, which feature is most strongly correlated with the logS value? And is it a negative or positive correlation?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Splitting the Data Into Training and Testing Sets\n",
    "\n",
    "Once we have a good understanding of the data, we can move on to the next step, which is splitting the data into a training set and a testing set. The training set is used to train the model, and the testing set is used to evaluate the model's performance. This process allows you to test the model's accuracy on unseen data and ensures that the model can generalize well to new data.\n",
    "\n",
    "It is extremely important to split the data first and then perform subsequent feature engineering steps. Feature engineering prior to splitting the data can cause a <b>data leakage</b> problem, allowing the model to \"see\" the testing data in the training phase. This violates our intention to treat the test data as a good representative sample of the real-world data. Data leakage leads to a model that performs well in training and testing but that performs poorly when given novel data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create our training and testing data sets, we will use the ``train_test_split`` function from the ``sklearn.model_selection`` module to split the data. The training set will be used to train the model, while the testing set will be used to evaluate the model's performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Useful information about the train test split process:\n",
    "- ``x`` generally denotes the input variables (the data the model will use to make predictions)\n",
    "- ``y`` is often used for target variable (what we are trying to predict)\n",
    "- ``test_size`` is used to assign the percentage of the data set aside for the testing set\n",
    "- ``random_state`` controls the random number generator used to shuffle the data before splitting it. In other words, it ensures that the same randomization is used each time you run the code, resulting in the same splits of the data. This is especially useful if you want to compare the performance of multiple models\n",
    "- ``shuffle = True`` ensures that the data is split randomly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the feature matrix (X), feature vector (x), and the target vector (y)\n",
    "X = df.drop(columns=['logS'])\n",
    "x = X[\"MolLogP\"]\n",
    "y = df['logS']\n",
    "\n",
    "# Split the data into training and testing sets (80% training, 20% testing)\n",
    "x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=123, shuffle=True)\n",
    "\n",
    "# Reshape the data into 2D arrays of shape (n_samples, 1)\n",
    "# (if working with only one input feature)\n",
    "x_train = x_train.values.reshape(-1,1)\n",
    "x_test = x_test.values.reshape(-1,1)\n",
    "y_train = y_train.values.reshape(-1,1)\n",
    "y_test = y_test.values.reshape(-1,1)\n",
    "\n",
    "# Display the shapes of the training and testing sets\n",
    "x_train.shape, x_test.shape, y_train.shape, y_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "    <b>Note</b>\n",
    "    Note the use of the <b>x</b> vector in the <b>train_test_split</b> function as initially we will train a model using only one input feature.<br> If we instead want to use all the available features we would need to use the <b>X</b> matrix that we have defined.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have a good understanding of the data, we can move on to the next step, which is feature engineering. Feature engineering is the process of transforming the raw data into a format that is suitable for machine learning models. Feature engineering often involves creating new features, selecting the most important features, and transforming the existing features in order to improve the model's performance.\n",
    "\n",
    "After splitting our data, we need to scale our train and test features. Scaling is a crucial step in the data preprocessing pipeline as it ensures that all features have the same scale, since many machine learning models are sensitive to the scale of the input features. We will use the ``StandardScaler`` from the ``sklearn.preprocessing`` module to scale our features. ``StandardScaler`` transforms the data in such a manner that it has mean value of 0 and a standard deviation value of 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the standard scaler object\n",
    "scaler = StandardScaler()\n",
    "\n",
    "# Fit and transform the training feature vector x_train\n",
    "x_train_scaled = scaler.fit_transform(x_train)\n",
    "\n",
    "# Transform the test feature vector x_test\n",
    "x_test_scaled = scaler.transform(x_test)\n",
    "\n",
    "# Make sure the training data is scaled correctly\n",
    "print(f\" Training feature mean: {x_train_scaled.mean():.5f}\")\n",
    "print(f\" Training feature standard deviation: {x_train_scaled.std():.5f}\\n\")\n",
    "\n",
    "# Print the scaler statistics on the test data\n",
    "print(f\" Testing feature mean: {x_test_scaled.mean():.5f}\")\n",
    "print(f\" Testing feature standard deviation: {x_test_scaled.std():.5f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "    <b>Reminder:</b>\n",
    "    It is extremely important to split the data first and then fit the scaler on the training data, only. Fitting the scaler on the entire data and then splitting it causes the <b>data leakage</b> problem which violates our intention to treat the test data as a good representative sample of the real-world data.  \n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building and Training a Linear Regression Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dummy Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step after the data preparation is to build and train our model. We will build a simple linear regression model which focuses on the relationship between a single feature (``MolLogP``) and the target variable (``logS``). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\"> \n",
    "    <b>Exercise</b>\n",
    "    What is the reason behind choosing <b>MolLogP</b> as our main feature in the linear regression model?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to evaluate the performance of our model, we can first create a dummy \"model\" using the ``DummyRegressor`` class from the ``sklearn.dummy`` module. This class provides a simple way to create a model that calculates the mean value of the target feature and predicts this mean value for each observation. The ``fit`` method is used to train the model on the training data. Once the model is trained, we can use the ``predict`` method to make predictions on the test data. Note that the ``DummyRegressor`` is not for solving real problems!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-26T16:51:57.478588Z",
     "start_time": "2024-06-26T16:51:57.469581Z"
    }
   },
   "outputs": [],
   "source": [
    "# Create a dummy model using the mean value of the target property\n",
    "dummy_model = DummyRegressor(strategy=\"mean\")\n",
    "\n",
    "# Fit the model to the training data\n",
    "dummy_model.fit(x_train_scaled, y_train)\n",
    "\n",
    "# Make predictions on the testing data\n",
    "y_pred_dummy = dummy_model.predict(x_test_scaled)\n",
    "\n",
    "# Calculate the performance metrics and store them in a DataFrame\n",
    "results = pd.DataFrame({\n",
    "    \"Coefficients\": [np.array(dummy_model.constant_)],   # the regression coefficient\n",
    "    \"MSE\": mean_squared_error(y_test, y_pred_dummy),     # the mean squared error\n",
    "    \"R2\": r2_score(y_test, y_pred_dummy)                 # the coefficient of determination\n",
    "}, index=[\"Dummy\"])\n",
    "                            \n",
    "\n",
    "# Set the formatting style\n",
    "results.style.format(\n",
    "    {\n",
    "        \"MSE\": \"{:.3f}\",\n",
    "        \"R2\": \"{:.2f}\"\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\"> \n",
    "    <b>Exercise</b>\n",
    "    Do we want to maximize or minimize the MSE value? What about R2?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linear Regression Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's next build and train a single-feature input linear regression model. We will use the ``LinearRegression`` class from the ``sklearn.linear_model`` module to create the model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a simple linear regression model\n",
    "simple_reg_model = LinearRegression()\n",
    "\n",
    "# Fit the model to the training data\n",
    "simple_reg_model.fit(x_train_scaled, y_train)\n",
    "\n",
    "# Make predictions on the test data\n",
    "y_pred_simple = simple_reg_model.predict(x_test_scaled)\n",
    "\n",
    "# Calculate the performance metrics\n",
    "simple_model_results = pd.DataFrame({\n",
    "    \"Coefficients\": [np.array(simple_reg_model.coef_)],   # the regression coefficient\n",
    "    \"MSE\": mean_squared_error(y_test, y_pred_simple),     # the mean squared error\n",
    "    \"R2\": r2_score(y_test, y_pred_simple)                 # the coefficient of determination\n",
    "}, index=[\"Simple-Linear-Regression\"])\n",
    "\n",
    "# Store the results into results DataFrame\n",
    "results = pd.concat([results, simple_model_results])\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\"> \n",
    "    <b>Exercise</b>\n",
    "    Which model is best at predicting the value of logS?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing Model Performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-26T16:52:00.069272Z",
     "start_time": "2024-06-26T16:51:59.817331Z"
    }
   },
   "outputs": [],
   "source": [
    "# Create a plot object \n",
    "fig, ax = plt.subplots(figsize=(5, 5))\n",
    "\n",
    "# Plot the test data\n",
    "ax.scatter(x_test_scaled, y_test, color='blue', label='Test Data')\n",
    "\n",
    "# Plot the simple linear regression model\n",
    "ax.plot(x_test_scaled, y_pred_simple, color='red', label='Simple Linear Regression')\n",
    "\n",
    "# Plot the baseline model\n",
    "ax.plot(x_test_scaled, y_pred_dummy, \"g--\", label=\"Dummy\")\n",
    "\n",
    "# Create the legends\n",
    "fig.legend(facecolor='white')\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multiple Linear Regression Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the ``MolLogP`` feature with highest correlation with the target variable ``logS`` is a good starting point. The reason is that we knew beforehand that the CLOGP model would be a great predictor of solubility. However, we can improve our model by including other features that show some correlation with the target variable. This is where multiple linear regression comes in.\n",
    "\n",
    "Let's build a multiple linear regression model using all the features in our dataset. The process is very similar to building a single-feature linear regression model: Once again, we need to scale the data, train the model on the scaled training data using the ``fit`` method, and make predictions on the test data using the ``predict`` function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "    <b>Note</b>\n",
    "    Note the usage of <b>X</b> matrix instead of <b>x</b> vector in the <b>train_test_split</b> function, since now we want to use multiple features to predict the target variable.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-26T16:52:01.901353Z",
     "start_time": "2024-06-26T16:52:01.889727Z"
    }
   },
   "outputs": [],
   "source": [
    "# Split the data into training and testing sets (80% training, 20% testing)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=123, shuffle=True)\n",
    "\n",
    "# Create the standard scaler object\n",
    "scaler = StandardScaler()\n",
    "\n",
    "# Fit and transform the training feature matrix X_train\n",
    "X_train_scaled = scaler.fit_transform(X_train)\n",
    "\n",
    "# Transform the test feature matrix X_test\n",
    "X_test_scaled = scaler.transform(X_test)\n",
    "\n",
    "# Create a linear regression model\n",
    "multi_feature_model = LinearRegression()\n",
    "\n",
    "# Fit the model to the training data\n",
    "multi_feature_model.fit(X_train_scaled, y_train)\n",
    "\n",
    "# Make predictions on the test data\n",
    "y_pred_multi_feature = multi_feature_model.predict(X_test_scaled)\n",
    "\n",
    "# Calculate the performance metrics\n",
    "multi_feature_model_results = pd.DataFrame({\n",
    "    \"Coefficients\": [np.array(multi_feature_model.coef_)],       # the regression coefficients\n",
    "    \"MSE\": mean_squared_error(y_test, y_pred_multi_feature),     # the mean squared error\n",
    "    \"R2\": r2_score(y_test, y_pred_multi_feature)                 # the coefficient of determination\n",
    "}, index=[\"Multi-Linear-Regression\"])\n",
    "\n",
    "# Store the results into results DataFrame\n",
    "results = pd.concat([results, multi_feature_model_results])\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\"> \n",
    "    <b>Exercise</b>\n",
    "    Which model is best at predicting the value of logS?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Better Model Evaluation with Cross-Validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We evaluate the performance of our trained model on the test dataset. However, the measured performance depends on the quality of the data in the splits (train/validation/test).\n",
    "In order to ameliorate this issue, we can use a technique called $k$-fold cross-validation. The $k$-fold cross-validation method splits the data into $k$ subsets,\n",
    "trains the data on the union of $k-1$ sets and measures the performance of the trained model on the $k$-th set, and repeats the process $k$ times to cover all subsets.\n",
    "The performance score is reported as the average score of $k$ experiments.\n",
    "\n",
    "<div style=\"text-align:center;\">\n",
    "  <img src=\"./images/K-fold_cross_validation_EN.svg\" style=\"display: block; margin: 0 auto; max-height:400px;\">\n",
    "</div>\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "Here, we use the ``cross_val_score`` function from the ``sklearn.model_selection`` module to perform a 5-fold cross validation experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a standard scaler object\n",
    "scaler = StandardScaler()\n",
    "\n",
    "# Create a linear regression model\n",
    "multi_feature_model = LinearRegression()\n",
    "\n",
    "# Create a pipeline object\n",
    "pipeline = make_pipeline(scaler, multi_feature_model)\n",
    "\n",
    "# Create a KFold object with 5 folds (defined using n_splits)\n",
    "kf = KFold(n_splits=5, shuffle=True, random_state=123)\n",
    "\n",
    "# Perform cross-validation\n",
    "cv_results = cross_val_score(pipeline, X, y, cv=kf, scoring=\"neg_mean_squared_error\")\n",
    "\n",
    "# Calculate the mean and standard deviation of the cross-validation results\n",
    "print(f\"CV Results Mean MSE: {-cv_results.mean():.5f}\")\n",
    "print(f\"CV Results STD MSE: {cv_results.std():.5f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\"> \n",
    "    <b>Bonus Exercise</b>\n",
    "    Getting help from the code above, perform a 5-fold cross-validation using a simple linear regression model and compare the results with the previous model evaluation based on the R**2 metric.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# perform a 5-fold cross-validation for a simple linear regression model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution to Bonus Exercise\n",
    "\n",
    "```python\n",
    "\n",
    "    from sklearn.model_selection import cross_val_score, KFold\n",
    "    from sklearn.pipeline import make_pipeline\n",
    "\n",
    "    # Create a standard scaler object\n",
    "    scaler = StandardScaler()\n",
    "\n",
    "    # Create a linear regression model\n",
    "    simple_linear_model = LinearRegression()\n",
    "\n",
    "    # Create a pipeline object\n",
    "    pipeline = make_pipeline(scaler, simple_linear_model)\n",
    "\n",
    "    # Create a KFold object\n",
    "    kf = KFold(n_splits=5, shuffle=True, random_state=123)\n",
    "\n",
    "    # Perform cross-validation   Note we use x instead of X as the input data\n",
    "    cv_results = cross_val_score(pipeline, x.values.reshape(-1,1), y, cv=kf, scoring=\"r2\")\n",
    "\n",
    "    # Calculate the mean and standard deviation of the cross-validation results\n",
    "    print(f\"CV Results Mean MSE: {cv_results.mean():.5f}\")\n",
    "    print(f\"CV Results STD MSE: {cv_results.std():.5f}\")\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Is Linear Regression Really the Best Model for This Task? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You have learned about the basics of linear regression,\n",
    "exploratory data analysis, feature engineering, and model evaluation. However, the process of building\n",
    "and evaluating multiple machine learning models to find the one best suited for your purposes can be time-consuming and repetitive.\n",
    "\n",
    "**PyCaret** is an open-source, low-code machine learning library written in Python that automates building \n",
    "the end-to-end machine learning processes. It provides a simple and efficient way to build, evaluate, and \n",
    "compare machine learning models. PyCaret is designed to help data scientists and machine learning engineers\n",
    "focus on the data and the problem at hand rather than the code.\n",
    "\n",
    "Let's see how we can use PyCaret to build and evaluate a wide range of machine learning models with just a\n",
    "few lines of code. First, we need to import the regression modules from PyCaret\n",
    "\n",
    "Before we move forward, we need to clean up the memory by restarting the kernel,\n",
    "import the necessary libraries for PyCaret and load the data into a Pandas\n",
    "DataFrame again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython import get_ipython\n",
    "\n",
    "if get_ipython():\n",
    "    get_ipython().kernel.do_shutdown(restart=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-26T16:51:46.314577Z",
     "start_time": "2024-06-26T16:51:46.301095Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd                 # for data manipulation\n",
    "import seaborn as sns               # for data visualization\n",
    "import matplotlib.pyplot as plt     # for data visualization\n",
    "\n",
    "from pycaret.regression import *    # for comparing the models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-26T16:49:13.095479Z",
     "start_time": "2024-06-26T16:49:13.088117Z"
    }
   },
   "outputs": [],
   "source": [
    "# Path to the preprocessed data file\n",
    "data_path = \"./data/solubility-processed.csv\"\n",
    "\n",
    "# Read the data into a DataFrame\n",
    "df = pd.read_csv(data_path,\n",
    "                 dtype={\"MolLogP\": \"float64\", \n",
    "                        \"MolWt\": \"float64\",\n",
    "                        \"NumRotatableBonds\": \"int\",\n",
    "                        \"AromaticProportion\": \"float\",\n",
    "                        \"logS\": \"float64\"\n",
    "                        })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After importing PyCaret's regression modules, we can create a PyCaret experiment using the ``setup`` function.\n",
    "The ``setup`` function automatically preprocesses the data, splits it into training and testing sets, \n",
    "and sets up the environment for building and evaluating machine learning models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = setup(data = df, target = 'logS', session_id = 123, fold = 5, n_jobs = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can compare the performance of different machine learning models using the\n",
    "``compare_models`` function. The function trains and evaluates a wide range of \n",
    "machine learning models on the input data and displays a table of the models' \n",
    "performance metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "best = compare_models()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Which Model is Best?\n",
    "\n",
    "- PyCaret shows which model it has determined is \"best\"\n",
    "- There are many metrics for scoring model performance. One good source for exploring other metrics is the ``sklearn.metrics`` section of the scikit-learn documentation (https://scikit-learn.org/stable/modules/model_evaluation.html)\n",
    "- Required computing time is also an important metric\n",
    "- Explainability is also important, especially if you are working with stakeholders who are not experts in machine learning\n",
    "- You need to decide which model is the best for your purposes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Identifying the Most Important Features\n",
    "\n",
    "Feature importance measures how much each feature contributes to a specific build of a model, and many models allow you to display the feature importances after training a model. PyCaret can show the feature importance for the best model.\n",
    "\n",
    "Let's take a look at the feature importance plot for the best performing model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_model(best, plot='feature')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which features are most important in your plot?\n",
    "\n",
    "- They can change between different training instances (especially if random state not set)\n",
    "- They are often different for different types of models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other options for the ``plot_model`` function to explore on your own:\n",
    "    \n",
    "* ``pipeline`` - Schematic drawing of the preprocessing pipeline\n",
    "* ``residuals_interactive`` - Interactive Residual plots\n",
    "* ``residuals`` - Residuals Plot\n",
    "* ``error`` - Prediction Error Plot\n",
    "* ``cooks`` - Cooks Distance Plot\n",
    "* ``rfe`` - Recursive Feat. Selection\n",
    "* ``learning`` - Learning Curve\n",
    "* ``vc`` - Validation Curve\n",
    "* ``manifold`` - Manifold Learning\n",
    "* ``feature`` - Feature Importance\n",
    "* ``feature_all`` - Feature Importance (All)\n",
    "* ``parameter`` - Model Hyperparameter\n",
    "* ``tree`` - Decision Tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bonus Section: EDA Using Python Code\n",
    "\n",
    "We used the SweetViz library to do all our exploratory data analysis (EDA) in just a few lines of code. Sometimes, however, it is desirable to do additional exploration on your own. Below is the python code necessary to do the same analysis that SweetViz performed. This code provides a good set of templates for doing your own customized EDA."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get a high-level overview of the dataset using the ``info`` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-26T16:49:45.746932Z",
     "start_time": "2024-06-26T16:49:45.735372Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the data is already preprocessed, the value of ``Non-Null Count`` columns for all features should be equal to the total number of samples in the dataset, *i.e.,* 1144. All features are of numerical type ``float64``. The experimentally measured solubility in the last column ``logS`` is the target variable and the remaining columns are the features.\n",
    "\n",
    "In addition to ``info()``, Pandas provides another useful function called ``describe`` which provides a statistical summary of all numerical features in our dataset. The provided statistics involve the count, mean, standard deviation, minimum, 25th percentile, median, 75th percentile, and maximum values for each feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This information is extremely useful for understanding the data and the distribution of the features. It helps in identifying any missing values or anomalies in the data or if our data requires any preprocessing. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualizing the data can also be helpful. For example, the skeweness of the data, which is determined by comparing the mean and median values, can be visualized using Pandas's ``hist`` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.hist(figsize=(8,8), edgecolor='black', grid=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly, the ``MolLogP`` solubility values are normally distributed while other features are more or less positively or negatively skewed. The Pandas library helps us quantify these observations for each feature using the ``skew`` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.skew()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The numerical values of skewness can be interpreted using the following rules:\n",
    "- The skewness value of zero indicates a perfect symmetrical distribution,\n",
    "- a skewness between -0.5 and 0.5 indicates an approximately symmetric distribution,\n",
    "- a skewness between -1 and -0.5 (or 0.5 and 1) indicates a moderately skewed distribution,\n",
    "- a skewness between -1.5 and -1 (or 1 and 1.5) indicates a highly skewed distribution, and\n",
    "- a skewness less than -1.5 (or greater than 1.5) indicates an extremely skewed distribution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we use Pandas's ``corr`` function to calculate the correlation matrix between the features and the target variable. The correlation matrix provides insights into the relationships between the features and the target variable(s). A correlation value close to 1 indicates a strong positive relationship, while a correlation value close to -1 indicates a strong negative relationship. A correlation value close to 0 indicates no relationship between the features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.corr(method='pearson')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An easy way to examine the relationships between the features and the target variable is to use a heatmap correlation matrix plot. \n",
    "We use the Seaborn library to create a heatmap plot of the correlation matrix as shown below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a plot object\n",
    "plt.figure(figsize=(8,6))\n",
    "\n",
    "# Calculate the correlation matrix\n",
    "corr = df.corr()\n",
    "\n",
    "# Create a heatmap\n",
    "sns.set(font_scale=1.1)\n",
    "sns.heatmap(corr, annot=True, cmap=\"coolwarm\", fmt=\".2f\")\n",
    "\n",
    "# Display the heatmap\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our coloring scheme makes it easy to uncover the relationships between the features. The darker the color, the stronger the correlation. The diagonal line represents the correlation of each feature with itself, which is always 1. In this plot, the red color indicates a positive correlation, while the blue color signifies a negative correlation."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "bcce",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}