# Adaptive Market Planning

## Narrative:

There is a broad class of problems that involves allocating some resources to meet an uncertain (
sometimes unobservable ) demand . Example: 

**Stocking a perishable inventory (e.g. fresh fish) to meet a demand where leftover inventory cannot be held for the future.** \
**We have to allocate annual budgets for activities such as marketing. Left-over funds are returned to the company**

"newsvendor problem", as canonical problem in optimization under uncertainity.

### Newsvendor problem

$$
\max_{x} \; \mathbf{E} \; F(x, W) =  \mathbf{E} \{pmin\{x,W\}-cx\} \\
$$

* where $x$ is our decision variable that detrmines the amount of resources to meet task
* where $W$ is uncertain demand for the resource
* We assume that we "purchase" our resource at a unit cost of $c$
* we sell the smaller of $x$ and $W$ at a price $p$
* $p$ is assumed greater than than $c$

If $W$ was deterministic (and if $p$>$c$), then the solution is easily verified to be $x$=$W$

Now imagine that $W$ is a random variable with:   

* probality distribution: $f^W(W)$

* cumulative distribution: $F^W(w) = Prob[W<w]$

* Then we can compute:

* $F(x) = \mathbf{E} \; F(x, W)$

Then, the optimal solution $x^{*}$ would satisfy:

$$
\frac{dF(x)}{dx} = 0 \\
x=x^*
$$

Now consider the stochastic gradient, where we take the derivative of $F(x, W)$ assuming we know 
$W$, which is given by:

$$
\begin{equation}
  \frac{dF(x,W)}{dx} =\left\{
  \begin{array}{@{}ll@{}}
    p-c, & x\leq  W\  \\
    -c, & x > W
  \end{array}\right.
\end{equation} 
$$

* Taking Expectation:

$$
\begin{equation}
\mathbf{E}\frac{dF(x,W)}{dx} = (p-c)Prob[x \leq W] - cProb[x>W] \\
= (p-c)(1-F^W(x))-cF^W(x) \\
= (p-c) - pF^W
= 0 \; For \; x = x^*
\end{equation}
$$

We can now solve for $F^W(x^*)$ giving:

$$
F^W(x^*) = \frac{p-c}{p}
$$

**This Chapter**: we are going to addressing the single-dimensional newsvendor, where: \
we assume that the demand distribution $W$ is unknown, but can be observed.

## Basic Model

We are going to solve problem using a classical method, called a stochastic gradient algorithm. The sequence is like:

we going to pick $x^n$, then observe $W^{n+1}$ and then compute $\nabla F (x^n, W^{n+1})$- As:

$$
x^{n+1} = x^{n} + \alpha_{n} \nabla F(x^n, W^{n+1)}
$$

whre $\alpha_n$ is known as a stepsize. Where it has been shown that we obtain asymptotic optimality:

$$
\lim_{n \to +\infty} x^n = x^*,
$$

If the stepsize $\alpha_n$ satisfie:

$$
\alpha_n \ > \ 0, \\
\sum_{n=1}^{\infty}\alpha_n = \infty, \\
\sum_{n=1}^{\infty}(\alpha_n)^2 < \infty,
$$

The second equation ensures that the stepsizes do not shrink so quickly that we stall out on the way to the optimum, the last equation has a effect ofinsuring that the variance of our estimate $x^*$ does shrink to zero.


```python

```

### State variable:

For our stochastic gradient algorithm, our state variable is given by:

$$
S^n = (x^n)
$$

### Decision Variable:

### harmonic stepsize rule:

***Step size Policy***

As with all of our sequential decision problems, the decision ( that is, the stepsize) is determined by what is typically referred to as a stepsize rule, but is sometimes called a stepzie policy that we denote by $\alpha^{\pi}(S^n)$ 

$$
\alpha^{harmonic} (S^n|\theta^{step}) = \frac{\theta^{step}}{\theta^{step} + n -1}
$$

### Exogenous information

The exogenous information is the random demand $W^{n+1}$ for the resource (product, time or money) that we are trying to meet with our supply of product $x^n$.<br> We may assume that we observe $W^{n+1}$ directly.

### Transition Function

$$
x^{n+1} = x^n + \alpha_n \nabla_xF(x^n, W^{n+1})
$$

### Objective Function

***Net benefit at each iteration***

$$
F(x^n, W^{n+1}) = p \ min \{x^n, W^{n+1} \} -cx
$$

***Maximize the total reward over some horizon***

$$
max \ \mathbf{E} \{ \sum_{n=0}^{N-1}F(X^{\pi}(S^n), W^{n+1})| S^0)\} \\
\pi
$$

$$
S^{n+1} = S^M(S^n, X^{\pi}(S^n), W^{n+1})
$$

## Coding Part


```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
```

### Chnaging the path to function folder 


```python
import os
os.chdir("/home/peyman/Documents/PhD_UiS/seqdec_powell_repo/Chap3_Adap_Mar_Plan/functions")
```


```python
from AdaptiveMarketPlanningModel import AdaptiveMarketPlanningModel
from AdaptiveMarketPlanningPolicy import AdaptiveMarketPlanningPolicy
```


```python
os.chdir("/home/peyman/Documents/PhD_UiS/seqdec_powell_repo/Chap3_Adap_Mar_Plan/data")
```


```python
raw_data= pd.read_excel("Base_parameters.xlsx", sheet_name="parameters", usecols=["Parameter", "Value"])
```


```python
raw_data
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Parameter</th>
      <th>Value</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>cost</td>
      <td>20</td>
    </tr>
    <tr>
      <th>1</th>
      <td>trial_size</td>
      <td>100</td>
    </tr>
    <tr>
      <th>2</th>
      <td>price</td>
      <td>26</td>
    </tr>
    <tr>
      <th>3</th>
      <td>theta_step</td>
      <td>50</td>
    </tr>
    <tr>
      <th>4</th>
      <td>T</td>
      <td>24</td>
    </tr>
    <tr>
      <th>5</th>
      <td>reward_type</td>
      <td>Terminal</td>
    </tr>
  </tbody>
</table>
</div>




```python
cost=raw_data["Value"][0] 
trial_size = raw_data["Value"][1]
price = raw_data["Value"][2]
theta_step = raw_data["Value"][3]
T = raw_data["Value"][4]
reward_type = raw_data["Value"][5]
```


```python
if __name__ == "__main__":
# this is an example of creating a model and running a simulation for a certain trial size

# define state variables
    state_names = ['order_quantity', 'counter']
    init_state = {'order_quantity': 0, 'counter': 0}
    decision_names = ['step_size']
    
    # read in variables from excel file
    #file = 'Base parameters.xlsx'
    #raw_data = pd.ExcelFile(file)
    raw_data= pd.read_excel("Base_parameters.xlsx", sheet_name="parameters", usecols=["Parameter", "Value"])
    cost=raw_data["Value"][0] 
    trial_size = raw_data["Value"][1]
    price = raw_data["Value"][2]
    theta_step = raw_data["Value"][3]
    T = raw_data["Value"][4]
    reward_type = raw_data["Value"][5]
    
    
    # initialize model and store ordered quantities in an array
    M = AdaptiveMarketPlanningModel(state_names, decision_names, init_state, T,reward_type, price, cost)
    P = AdaptiveMarketPlanningPolicy(M, theta_step)

    rewards_per_iteration = []
    learning_list_per_iteration = []
    for ite in list(range(trial_size)):
        print("Starting iteration ", ite)
        reward,learning_list = P.run_policy()
        M.learning_list=[]
        #print(learning_list)
        rewards_per_iteration.append(reward)
        learning_list_per_iteration.append(learning_list)
        print("Ending iteration ", ite," Reward ",reward)


    nElem = np.arange(1,trial_size+1)

    rewards_per_iteration = np.array(rewards_per_iteration)
    rewards_per_iteration_sum = rewards_per_iteration.cumsum()
    rewards_per_iteration_cum_avg = rewards_per_iteration_sum/nElem

    if (reward_type=="Cumulative"):
        rewards_per_iteration_cum_avg = rewards_per_iteration_cum_avg/T
        rewards_per_iteration = rewards_per_iteration/T

    optimal_order_quantity = -np.log(cost/price) * 100
    print("Optimal order_quantity for price {} and cost {} is {}".format(price,cost,optimal_order_quantity))
    print("Reward type: {}, theta_step: {}, T: {} - Average reward over {} iteratios is: {}".format(reward_type,theta_step,T,trial_size,rewards_per_iteration_cum_avg[-1]))
    
    ite = np.random.randint(0,trial_size)
    order_quantity = learning_list_per_iteration[ite]
    print("Order quantity for iteration {}".format(ite))
    print(order_quantity)

    #Ploting the reward
    fig1, axsubs = plt.subplots(1,2,sharex=True,sharey=True)
    fig1.suptitle("Reward type: {}, theta_step: {}, T: {}".format(reward_type,theta_step,T) )

    axsubs[0].plot(nElem, rewards_per_iteration_cum_avg, 'g')
    axsubs[0].set_title('Cum_average reward')
  
    axsubs[1].plot(nElem, rewards_per_iteration, 'g')
    axsubs[1].set_title('Reward per iteration')
    #Create a big subplot
    ax = fig1.add_subplot(111, frameon=False)
    # hide tick and tick label of the big axes
    plt.tick_params(labelcolor='none', top=False, bottom=False, left=False, right=False)
    ax.set_ylabel('USD', labelpad=0) # Use argument `labelpad` to move label downwards.
    ax.set_xlabel('Iterations', labelpad=10)
    plt.show()
    



    # ploting the analytical sol
    plt.xlabel("Time")
    plt.ylabel("Order quantity")
    plt.title("Analytical vs learned ordered quantity - (iteration {})".format(ite))
    time = np.arange(0, len(order_quantity))
    plt.plot(time, time * 0 - np.log(cost/price) * 100, label = "Analytical solution")
    plt.plot(time, order_quantity, label = "Kesten's Rule for theta_step {}".format(theta_step))
    plt.legend()
    plt.show()
```

4b296002eef414848967c2b1ba37cde75ab5ed15
ipynb
Jupyter Notebook
Chap3_Adap_Mar_Plan/Chap3_Modeling_Theory.ipynb
Peymankor/seqdec_powell_repo
d3a03399ca7762821e80988d112f98bad5adefd8
MIT
2021-04-21T19:21:53.000Z
2021-04-21T19:21:53.000Z
Chap3_Adap_Mar_Plan/Chap3_Modeling_Theory.ipynb
Efsilvaa/seqdec_powell_repo
d3a03399ca7762821e80988d112f98bad5adefd8
MIT
Chap3_Adap_Mar_Plan/Chap3_Modeling_Theory.ipynb
Efsilvaa/seqdec_powell_repo
d3a03399ca7762821e80988d112f98bad5adefd8
MIT
2021-04-21T19:21:45.000Z
2021-06-24T18:19:17.000Z
Qwen/Qwen-72B
1. YES
2. YES