formulated as: $$T_{shuffle} = \beta_{0f} + \beta_6S_d + \beta_7M_t
\label{eq:shufflephase}$$

### **Custom Reduce**

For each key in a data partition, the reduce function is executed. We use the formula below to approximate the time it takes for the reduce phase to complete.

$$T_{reduce} = \frac{\sum_{i=1}^{n}T_{key}}{R_t}/{N_c}
 \label{eq:reducephase}$$

Here, $N_c$ is the total number of containers.

### **Write**

This is the last phase of the MapReduce pipeline. The output of custom reduce function is collected and written to HDFS. To model this phase, first, we modelled the total data that the reducer task emits, i.e., $$R_d = S_{d} * R_{sel} 
\label{eq:reducedata}$$ where $S_d$ is the total shuffle data fed to a reduce task and $R_{sel}$ is the ratio of input and output sizes. Using the relations in Equation , we can, therefore, define the cost model of the write as a linear function of the reduce output data size, which is: $$T_{write} = f(R_d) = \beta_{0w} + \beta_8R_d
\label{eq:writephase}$$

**Combining it All Together**
-----------------------------

Now that we have proposed the cost model for the individual phases of the MapReduce pipeline. Putting them all together for the two main phases, we have: $$T_{mt} = T_{read} + T_{map} + T_{collect} + T_{spill} + T_{merge}
