**2. Targets for remineralization**

When designing remineralization systems and determining the appropriate water quality targets, a handful of key parameters are taken into consideration: alkalinity, calcium hardness and pH. The importance of these parameters are discussed further in detail in the following sections.

#### **2.1 Importance of alkalinity**

Alkalinity is one of the most important water quality parameter due to its ability to maintain a stable and buffered system, as well as its effectiveness in protecting against various mechanisms of corrosion. In general terms, alkalinity is defined as the capacity of an aqueous solution to accept a proton, or rather to neutralize an acid. Although other systems can contribute to alkalinity, for drinking water processes, it is only the carbonate system that is taken into account to define alkalinity. This is due to its predominance over other buffering systems in the pH ranges usually associated with water chemistry. Alkalinity for drinking water is therefore expressed as the following:

$$\text{Alkalinity} = 2\left[\text{CO}\_3\text{}^{2-}\right] + \left[\text{HCO}\_3^-\right] + \left[\text{OH}^-\right] \text{-} \left[\text{H}^+\right] \tag{1}$$

As seen in Eq. (1), alkalinity is determined primarily by the concentration of carbonate and bicarbonate ions, and to a lesser extent by the concentration of hydroxide and hydrogen ions, which also define the pH. Bicarbonate ions are particularly important, due to their ability to consume both hydrogen and hydroxide ions, and therefore provide a buffer to pH shifts in both directions.

In addition to providing a buffer to protect against shifts in pH, alkalinity is one of the most important parameters in corrosion control. The World Health Organization recommends high alkalinity levels as a suitable technique for preventing many mechanisms of corrosion in their Guidelines to Drinking Water Quality [7]. This is backed up by a multitude of studies into corrosion control and experts responsible for setting water quality targets.

More particularly, alkalinity is essential in controlling the corrosion of many metal materials of construction. One of the best methods of controlling iron, copper, zinc or galvanized iron corrosion, is the precipitation of the respective carbonates, such as siderite (FeCO3), basic copper carbonate and basic zinc carbonate, for the formation of a passivation layer on the surface of the material [8]. A higher alkalinity content is therefore imperative to ensure sufficient carbonate species are available for the formation of these compounds. These protective layers are also an effective strategy against microbiologically induced pitting [9]. Lead on the other

hand, is released into the water either directly from the pipe, or from leadcontaining corrosion products that are formed on the pipe surface. In terms of lead corrosion products, the most common of these is lead carbonate whose propensity of lead (II) carbonates to dissolve into the water stream is directly related to the concentration of carbonate species already within the water. Finally, alkalinity is essential to prevent the degradation of concrete and cement-based systems that in many cases are used as a lining to protect large bore conduits constructed from mild steel or ductile iron.


2.1.2. Materials

The AM tools are made from Accura Bluestone by the company 3D Systems®, which is processed on a Viper si2 System (SLA). Accura Bluestone is a ceramic filled epoxy composite, which is characterized by relatively high heat deflection temperature of up to 284 ◦C and a tensile strength about 8000 MPa [47]. Due to its material characteristics, high imaging accuracy and low surface roughness, Kampkar et al.[21] recommend Accura Bluestone as an AM tool material.

The thermal conductivity coefficient of Accura Bluestone was determined for this investigation at approximately 0.781 W/mK, which is 50 times lower than the thermal conductivity of conventional steel (34.5–49.3 W/mK) [48] and 150 of aluminum (130–160 W/mK) [49]. Aside from the cooling system, all other tools are identical in their geometry and design. Table 1 shows a brief comparison of the material properties of the different tool materials. As a reference, the same tools are conventionally milled from aluminum (EN AW-7075) and steel (C45 U).

**Table 1.** Comparison of the material properties of the different tool materials (Bluestone, aluminum, steel).

To increase the heat deflection temperature, the Accura Bluestone mold insert is tempered with a temperature profile as recommended by 3D Systems. This is done through thermal post curing for 2 h at 120 ◦C, which increases the deflection temperature from 65–66 ◦C to 267–284 ◦C [47].

Representative for the most used LFT, the glass fiber-reinforced PP injection molding material STAMAX from SABIC is used in different fiber weight percentages of 10 wt.%, 20 wt.%, 40 wt.%, and 60 wt.%. The specific fiber contents are chosen in accordance with the results of Goris [50], which showed the highest contrast and a clear differentiation for interpretation. Ratios of 30 wt.% and 50 wt.% were not considered to keep the experimental volume adequate. STAMAX is a certified product series for the automotive industry and is already available pre-mixed in many different glass fiber contents. Accordingly, the mixing ratios 20 wt.%, 40 wt.%, and 60 wt.% for the investigations can be obtained directly. The mixing ratio 10 wt.% is gravimetrically prepared from STAMAX 20 wt.% (20YM240) and pure PP, which the manufacturer SABIC uses as a base matrix material. In addition to its wide industrial usage, the material is characterized by its particularly good processability. PP has a wide

processing temperature range with a low melt viscosity, melt temperature, and adhesivity. This is beneficial to reduce the thermal and mechanical stress on the polymer tools and to ensure the possibility to produce enough samples for the evaluation. For the specimen fabrication, a Boy 25 E injection molding machine from Dr. Boy GmbH & Co. KG is chosen.

## *2.2. Experiment Methodolgy*

## 2.2.1. Mechanical Properties

To analyze the mechanical properties, tensile testing after DIN EN ISO 527 is performed. The results are analyzed toward tensile strength, elongation, and Young's modulus. For the calculation of the standard deviation, five samples of each specimen type are analyzed as recommended by DIN EN ISO 527. Nominal fiber contents of 10 wt.%, 20 wt.%, 40 wt.%, and 60 wt.% are analyzed. In accordance with DIN EN ISO 527, a testing speed of 1 mm/min is used for determination of the elastic properties and, more specifically, the determination of the Young's modulus. For the deformation properties, the testing speed is increased to 50 mm/min for the practical purpose of decreasing testing time. Although not in accordance with the norm, this is a common approach in the field of tensile testing. However, this effect must be considered for the discussion of the test results.

#### 2.2.2. Fiber Length Analysis: Epoxy Plug Method

The epoxy plug method is centered around a down-sampling step after Kunc [40], which reduces the fiber count from <1,000,000 to a representative amount around 15,000–60,000 fibers. The detailed experiments steps are depicted in Figure 4.

As can be seen in Figure 4, the matrix material of the sample is initially removed. A sample diameter of at least twice the initial fiber length must be chosen to avoid measuring fibers that crossed the cutting plane during extraction. An initial fiber length of 15 mm results in a diameter of 30 mm. The matrix removal is performed by pyrolysis at 500 ◦C for 8 h in an industrial oven. Rohde et al. [46] studied the impact of matrix removal by pyrolysis or chemical decomposition on the morphology of single fibers. The results show that performing pyrolysis at 500 ◦C for 2 h is optimal for a PP sample. However, initial test runs showed that the pyrolysis time needed to be extended to 8 h to su fficiently remove the matrix.

For this investigation, the selection of the experimental fiber analysis test method was based on two factors: to provide su fficient repeatability and comparability toward previous investigations, as well as being able to generate an adequate output with a reasonable experiment duration. Therefore, the developed method by Goris et al. [17] (compare Section 1.2) was chosen. Based on the investigations of Rohde et al. [46], a thermal matrix removal is chosen instead of a chemical removal, since the fiber length could be negatively altered. Next, the down-sampling step is performed. A defined amount of UV-activated epoxy B0027N07MM liquid-glue by the company BONDIC (Aurora, Canada) is injected in the exposed fiber bed. The diameter of the injected epoxy varies from ca. 4–7 mm. After UV-curing of the glue and careful removing of non-attached fibers, a second pyrolysis is performed at 500 ◦C for 8 h. The subsequent fiber dispersion step is performed within a dispersion chamber, which can be seen in Figure 5. The turbulent dispersion is performed through small amounts of pressured air, performed 3–4 times at 1 bar for around 0.5 s. The fibers are than dispersed on a 210 mm × 255 mm × 4 mm glass plate, which is positioned on an EPSON Perfection V800 Photo scanner of the company Seiko Epson (Suwa, Japan) to create a digital image at 2400 dpi. For image enhancement and threshold, Adobe ® Photoshop ™ is used. Threshold levels are approximately 40 at the image center and 60 at the edges. This variation is necessary due to inhomogeneous illumination. Then, a MATLAB-based algorithm is used for fiber detection, which was developed at the Polymer Engineering Center (Madison, WI, USA) [51] and is based on the work of Wang [52]. The fiber detection is automated and even detects stacked and bent fibers.

**Figure 5.** Dispersion chamber after Goris adapted from [50].

Two di fferent averages are calculated for investigation of the fiber length: number average and length average. The number average is calculated using (compare nomenclature Table 2)

$$L\_N = \frac{\sum\_{i=1}^n (N\_{i\cdot} l\_i)}{\sum\_{i=1}^n (N\_i)} \tag{4}$$

and the length average (compare nomenclature Table 2)

$$L\_{\mathcal{W}} = \frac{\sum\_{i=1}^{n} \left( N\_{i} \cdot l\_{i}^{2} \right)}{\sum\_{i=1}^{n} \left( N\_{i} \cdot l\_{i} \right)} \tag{5}$$

**Table 2.** Nomenclature.

Long fibers have a more significant impact on the length average than short fibers. This e ffect is described as nonuniformity, of which high values are usually preferable in technical applications. Another e ffect that must be considered during the epoxy plug method is the preferred pickup of long fibers during the down-sampling step. This phenomenon can lead to a distorted average fiber length and is schematically represented in Figure 6. Five fibers are visible, from which only four are picked up by the down-sampling region. The fifth does not contribute to the experimental analysis aside from a similar alignment of the center of mass of each fiber. Therefore, Kunc et al. [40] introduced the so-called Kunc correction, which determines a corrected fiber frequency *Ni* in favor of shorter fibers. It is calculated as:

$$N(L\_i) = \theta(L\_i) \cdot \left(1 + \frac{4 \cdot L\_i}{\pi d}\right) \tag{6}$$

**Figure 6.** Illustration of the Kunc correction procedure, adapted after [40].

For this investigation, the analysis of the fiber length focuses on the analysis of the fiber-reinforced discs. Three segments—A, B, and C—with a diameter of 30 mm ± 0.5 mm are therefore removed by sawing from discs as depicted in Figure 7. The diameter is chosen based on the initial fiber length of 15 mm for STAMAX pellets. For the calculation of the standard deviation, three samples were analyzed for each location. Nominal fiber contents of 10 wt.%, 20 wt.%, 40 wt.%, and 60 wt.% are analyzed.

**Figure 7.** Disc segments for fiber length analysis.

#### 2.2.3. Fiber Content Analysis: Pyrolysis

The analysis of the fiber content is determined gravimetrically. For this, a pyrolysis in accordance with DIN EN ISO 3451 is performed for each sample within an industrial oven. Each sample is kept at 100 ◦C for 30 min before the matrix is removed at 625 ◦C for 3 h. This represents a less gentle and therefore faster temperature program compared to the pyrolysis steps of the epoxy plug method, since the fiber quality is not important for this experiment. The fiber content is then calculated in accordance with DIN EN ISO 3451 as (compare nomenclature Table 3):

$$
\alpha = \frac{m\_F}{m\_{Total}} \cdot 100 \text{ wt\%} \tag{7}
$$

**Table 3.** Nomenclature.

For this investigation, the analysis of the fiber content focuses on the analysis of the fiber-reinforced discs. The discs are therefore quartered and segmented in 33 squares with an edge length of 11.5 mm ± 0.5 mm as depicted in Figure 8. The samples were then cut out with scissors, resulting in 22 segments with identical in volume for each nominal fiber content, as well as 11 segments from the disc edges with varying volume and sample shape. The varying shape does not constitute a problem for later calculation of the fiber content, since the results are normalized by the individual total sample mass. For the calculation of the standard deviation, three samples were analyzed for each location. Nominal fiber contents of 10 wt.%, 20 wt.%, 40 wt.%, and 60 wt.% are analyzed.

**Figure 8.** Segmentation of fiber-reinforced discs for fiber content analysis.

2.2.4. Fiber Orientation Analysis: Micro-Computed Tomography

For fiber orientation analysis, μCT scans are used, which are performed with a GE v|tome|x m 240/180 by the company General Electric (Boston, MA, USA). The scan parameters can be seen in Table 4.

**Table 4.** μCT scan parameters GE v|tome|x m 240/180.

The analysis of the μCT data is performed with VGSTUDIO MAX 3.3.0 64 Bit by the company Volume graphics GmbH (Heidelberg, Germany). The software was chosen in accordance with a comparative study by Goris [17], which compared results of the conventional method of ellipses to μCT-data analyzed with three di fferent algorithms: VGSTUDIO MAX 3.3.0 64 Bit, slit method (SM) algorithm, and Mimics (proprietary to SABIC and Materialise MV). The results showed that a minimal resolution of 19 ± 1 μm could be used to su fficiently analyze the fiber orientation using VGSTUDIO MAX. For this investigation, the analysis of isotropic fiber orientation is performed with plane projection. The direction of the normal vector is identical to the thickness direction. Table 5 shows the detailed analysis parameters. As can be seen, the fiber material is defined by specific threshold, which is identical for all samples.

**Table 5.** Analysis parameters VGSTUDIO MAX 3.3.0 64 Bit.

For this investigation, the analysis of the fiber orientation focuses on the analysis of the fiber-reinforced discs. As depicted in Figure 9, a rectangular sample of 18 mm × 70 mm (±0.5 mm) is removed with a saw from each disc. Three squares—A, B, and C—with an edge length of 12 mm ± 0.5 mm are then analyzed within VGSTUDIO MAX. Nominal fiber contents of 10 wt.%, 20 wt.%, 40 wt.%, and 60 wt.% are analyzed. Due to limitations for the experimental amount, only one sample of each fiber content and tool type is analyzed.

**Figure 9.** Segmentation of the fiber-reinforced discs for fiber orientation analysis; Segments A, B, and C (blue) are analyzed within VGSTUDIO MAX.

#### *2.3. Processing Parameters and Tool Design*

The processing parameters for STAMAX are based on the processing guidelines [53] recommended by SABIC. Within these guidelines, a distinction is made between minimum, moderate, and maximum parameters. For this investigation, moderate parameters are targeted to generate comparability to the results of Goris [50]. In accordance with the guidelines, the injection pressure is chosen at 800 bar with linear decline down to 700 bar for steel tooling, with moderate injection speeds of 70 ccm/s to 60 ccm/s. The holding pressure is recommended at 50–80% of the injection pressure. Therefore, a holding pressure of 400 bar is chosen with a linear decline to 350 bar. Since aluminum and Bluestone provide lower mechanical properties than steel, the processing parameters including the clamping force for these tools were lowered accordingly. Table 6 gives a detailed overview of the processing parameters of each tooling type.

The melt temperature is chosen at 250 ◦C with a flat temperature profile within the screw cylinder. As can be seen, the mold temperature is set to 30 ◦C. The cooling process is accomplished through a water-cooling cycle of the same temperature. As soon as the implemented temperature sensors near the cavity reach the targeted temperature, a new cycle can be started.

As described before, geometrical limitations for the cooling channels are small due to additive manufacturing. This is represented in Figure 10, which shows the difference in possible cooling cycle alignment for ejector half metal disc tools (a) and additively manufactured disc tools (b).

**Figure 10.** Ejector half disc tool designs: aluminum and steel (**a**) and Bluestone (**b**). Ejector pins are marked (A).

As can be seen, sealing plugs were used to close the boreholes of the metal tools. For the fixed mold half, the cooling path is analogue to the ejector half. The cooling channel geometry was determined through iterative calculation steps based on fundamental theorems of heat transfer and through calculation of the divergence in mold temperature. Moldflow simulations were used to guarantee a homogenic average mold temperature of the developed concepts, to minimize the effect of different cooling channel geometries on the results. As an example, the simulative results of Bluestone tensile specimen-tools are depicted in Figure 11.

**Figure 11.** Moldflow simulation results of the average temperature for tensile specimen-tools out of Bluestone at 30 ◦C coolant temperature.

As can be seen, the average temperature ranges from approximately 30 ◦C up to 125 ◦C. The highest temperatures can be seen in the stagnation points of the melt at the tensile specimen shoulders. Since for Bluestone, a uniform temperature could not be guaranteed within the whole part, the focus lied on creating a homogenous average temperature within the functional regions, which is the gauge for tensile specimens. The stagnation points (red) were defined as critical regions, which will be discussed further in the following chapter. In case of the tensile specimens, parabolic runners with a diameter of 5 mm and a non-specified gate are used only on the ejector half. For the disc tools, the melt enters the cavity directly through the nozzle without the use of runners or a gate. The striking point of the melt also represents a critical region, especially for disc tools.


*5.1. Illustrate the E*ff*ectiveness of the NSGA-II*

The parameters of the genetic algorithm, multi-objective particle swarm optimization (MOPSO), NSGA-II, and optimized NSGA-II are set in Table 2. Verifying the validity and stability of NSGA-II based on constrained processing is a crucial part of optimizing the feasible search space. Since NSGA-II belongs to a posteriori algorithm, the effectiveness of the multi-objective reservoir scheduling is verified by selecting the 1981 data, and the comparison results are shown in Table 3.

**Table 2.** Parameter setting of the genetic algorithm (GA) and the improved NSGA-II.

*Water* **2020** , *12*, 915

In order to determine the accuracy and stability of the GA, MOPSO, NSGA-II, and the improved NSGA-II algorithm, take the result of running five times as an example, in which the different indexes of each algorithm are obtained. The upper and lower limits of the optimal value of power generation and water shortage, the mean value, the standard deviation, and the average running time are shown in Table 3. It can be seen that:

(1) The upper limit of the power generation of optimized NSGA-II is 10.315 billion kW·h. Compared with NSGA-II and MOPSO, however, both of which improved by 1 million kW·h, there is almost no change relative to the GA. The lower limit of the power generation of optimized NSGA-II is the same as that of NSGA-II and MOPSO.

(2) From the view of ecological water shortage, the lower limit of ecological deficiency is consistent. The upper limit of ecological water shortage for optimized NSGA-II is 506 million m3, which is higher than NSGA-II but lower than MOPSO.

(3) From the stability of results, the maximum and minimum standard deviation of ICS optimization results is 0.03, which is lower than the other three algorithms. In addition, the first results of MOPSO significantly premature convergence, the lower limit of the amount of water is non-zero. Meanwhile, the second result of NSGA-II has local convergence, and the power generation does not reach the maximum. Therefore, the results of optimized NSGA-II are relatively stable.

(4) From the optimization time, NSGA-II is longer than GA, and different optimization strategies will make the algorithm optimization time longer. While optimized NSGA-II is shorter than NSGA-II, which reflects that the improved strategy proposed in this paper accelerates the population convergence under the premise of ensuring a certain accuracy. Overall, the optimized NSGA-II is superior to GA, NSGA-II, and MOPSO, indicating that the optimization strategy is effective.

#### *5.2. Analysis of the Impact of Ecological Goals on Dispatching Results*

For the single-objective model with the maximum power generation as the target, a strong ecological constraint is added to form Model 2. The maximum degree of influence of ecological goals on the scheduling results can be found by comparing Model 1 with Model 2. Among the long series of results, the total power generation was 8.759 billion kW · h, which was 0.87% lower than the 8.836 billion kW · h of Model 1. However, the ecological guarantee rate of the Huayuankou section increased by 18.75%, which met the planned 90% guarantee rate. It shows that using ecological goals as a constraint can significantly increase the guarantee rate of ecological water demand, but the power generation benefits are slightly lost.

Model 2 is a single-objective model with maximum power generation after adding strong ecological constraints, and the impact of ecological objectives on the dispatching results can be compared between Model 1 and Model 2. Select 1981–1982, 1960–1961, and 1972–1973 as wet, normal, dry. Among these dates, July–October is the flood season, November–March is the dry season, and April–June is the water-supply period. The process of reservoir water level, reservoir discharge, and the power generation of Model 1 and Model 2 are compared and analyzed, as shown in Figure 6, Figure 7, and Figure 8.

**Figure 6.** Water level change of the Xiaolangdi reservoir in each typical year.

**Figure 7.** Variation of discharge from the Xiaolangdi reservoir in each typical year.

**Figure 8.** Variation of the monthly generating capacity of Xiaolangdi in each typical year.

(1) The water level of each month in Model 2 is basically lower than Model 1, especially in the non-flood period of the wet year, the normal, and the dry year, and the water level of the reservoir is obviously maintained at the low water level. This is because, in order to meet the downstream ecological water storage, the reservoir needs to increase the discharge, which causes the water level to drop. Therefore, considering the ecological constraints, the water level during the water supply period of the Xiaolangdi dam during the dispatching period has been changed.

(2) Restricted by ecological constraints, the monthly outflow of the Xiaolangdi dam in Model 2 basically meets the ecological requirements. In Model 1, the discharge flow of individual months is lower than the ecological water demand and cannot meet the ecological requirements. This is because, in order to maximize the power generation, Model 1 needs to discharge as little and as much water as possible during the storage period and store it at the normal storage level as soon as possible. As a result, the amount of water discharged during the storage period cannot meet the ecological water demand. The destruction of the month occurred in the dry season. The number of damaging months is 2, 4, and 4, respectively, and the degree of damage for the wet year, normal year, and dry year are as follows.

(3) The power generation of Xiaolangdi Hydropower Station shows the same variation in all typical years: the flood season is large, the dry season is small, and the water supply period is large. In Model 2, the power generation in the dry season is obviously larger than Model 1. The main reasons are as follows: In Model 1, the destruction of ecological water demand are all in the dry season. In order to ensure the ecological flow, Model 2 runs at the low water level during the dry year, which increases the discharge and power generation.

Table 4 shows the annual electricity generation and the number of eco-friendly months of different models in each typical year. As can be seen from Table 4, the monthly number of ecological guarantees in each typical year of Model 2 was increased by 2, 4, and 4 months, while the power generation decreased by 0.33, 0.86 and 1.27 billion kW·h. The ecological target has a great impact on power generation during the year and has the greatest impact on power generation in the dry year.

Therefore, adding Model 2 with strong ecological constraints, there is a clear change in the operation process and the magnitude of the reservoir during the dispatch cycle, and Model 2 sacrifices more power generation to complete the ecological target.

**Table 4.** Analysis of power generation and ecological guarantee in each typical year.

#### *5.3. Result Analysis of the Multi-Objective Model*

In this paper, we use the optimized NSGA-II to solve the multi-objective Model 3. The results of the multi-objective optimization scheduling model are plotted as a Pareto-front curve, as shown in Figure 9.

**Figure 9.** Multi-objective Pareto-front curves in each typical year.

In the Pareto-front curve of multi-objective optimal scheduling, the annual variation in the annual water power generation and the comprehensive water shortage in the wet year are 9.564–9.597 billion kW·h and 0–0.530 billion m3, respectively; the annual variation in the annual water power generation and the comprehensive water shortage in normal year are 6.707–6.795 billion kW·h and 1.547–3.246 billion m3, respectively; and the annual variation in the annual water power generation and the comprehensive water shortage in dry year are 5.875–6.002 billion kW·h and 0–2.835 billion m3, respectively.

It can be seen that with the decrease in typical annual runoff that the decrease in hydropower generation and the increase in ecological comprehensive water shortage make the contradiction between ecological target and power generation target aggravating. The total water shortage in normal year is larger than that of the dry year, which is mainly caused by the annual runoff process and the integrated water demand process.

Due to the different water requirement processes in typical years, Model 1 and Model 2 are different in Pareto-front curves:

(1) The annual generation capacity of the model 1 is 9.597 billion kW·h, and the comprehensive water shortage is 0.530 billion m3, which is located at the right extreme value of the Pareto-front curve; the annual generation capacity of Model 2 is 9.564 billion kW·h, and the comprehensive water shortage is 0 billion m3, which is located at the left extreme value of the Pareto-front curve.

(2) The annual generation capacity of Model 1 and Model 2 in the normal water year is 6.795 billion kW·h and 6.707 billion kW·h, respectively, and the comprehensive water shortage is 3.246 billion m3 and 1. 547 million m3, respectively.

(3) The annual generation capacity of Model 1 and Model 2 in the dry water year is 6.002 billion kW·h and 5.875 billion kW·h, respectively, and the comprehensive water shortage is 2.835 billion m3 and 0 million m3, respectively.

(4) With the increase in the total amount of incoming water, the power generation in the dry year, the normal year, and the wet year increase sequentially, and the water shortage also increases in turn. There is a lot of water in the normal year, but it is restricted by the constraints of poor distribution

in the water year, and the ecological water demand is greater than in the dry year, which causes the maximum water shortage to increase.

It can be seen that the single-objective calculation results fall on the Pareto-front curve of the global equilibrium solution. With the decrease in incoming water, the power generation decreased drastically, the water deficit increased significantly, and the contradiction between the target of ecological comprehensive water demand and the power generation goal was aggravated.

On the Pareto-front curve of the multi-objective global equilibrium solution, we select the power generation index, and generate Schemes 1 to 5 shown from small to large (the position is shown in Figure 9). The outflow of Xiaolangdi in each typical year is shown in Figures 10–12. The scheduling results of each scheme are shown in Table 5.

**Figure 10.** Comparative analysis of the outflow of Xiaolangdi in the wet year.

**Figure 11.** Comparative analysis of the outflow of Xiaolangdi in the normal year.

**Figure 12.** Comparative analysis of the outflow of Xiaolangdi in the dry year.

**Table 5.** Comparative analysis of various schemes in typical years.

Table 5 shows that in wet year, the multi-objective scheme can meet the ecological requirements under the condition of minimum power generation. From Schemes 1–5, the power generation increased from 9.565 billion kW·h to 9.595 billion kW·h, and the outflow decreased during the dry year. From the ecological destruction duration, the monthly number of guarantees is reduced from 12 months to 9 months; from the depth of ecological destruction, the ecological water shortage increased from 0 to 0.462 billion m3, and Schemes 2–5 failed to meet the ecological requirements.

Compared with the wet year, the power generation Schemes 1–5 consist of 6.709–5 kW·h increased to 6.792 billion kW·h, from the diachronic perspective of ecological destruction, which dropped from 11 months to 8 months. From the depth of ecological damage, the ecological water shortage of 1.601 billion m3 increased to 2.957 billion m3.

Like the normal year, the power generation of each scheme is reduced to a minimum value of 5.875 billion kW·h, which can basically guarantee the minimum ecological runoff process in the dry year. The ecological flow of Scheme 2 to Scheme 5 has been destroyed in varying degrees. Under the condition of a continuous decrease in incoming water, the contradiction between power generation target and ecological target is aggravated day by day.

From the above analysis, the following conclusions are drawn:

(1) Due to the difference in incoming water and ecological water demand, the Pareto-front optimal solution set shows the solution spatial distribution and the dimension change of each typical single-multi-objective model. The results of the single-objective Model 1 or Model 2 are basically on the curve, and the results of Model 1 are close to the maximum target of power generation, and Model 2- s results are close to the minimum target of water shortage.

(2) The location of the single-objective result in the curve shows that it has some limitations. The multi-objective model considers the power generation and ecological targets and gives a multi-objective global equilibrium solution set. The multi-objective model has superiority that can fully meet the ecological flow requirements. It provides a multi-objective dispatching solution for decision-makers in reservoir operation and watershed management.

(3) In the multi-objective model, except for Scheme 1, other schemes cannot meet the downstream ecological objective. The safety of ecological flow in the lower reaches of the Yellow River is serious, and as the inflow decreases, the amount of power generation will be correspondingly reduced, the number of ecological assurances and the amount of ecological water deficit will increase, and the ecological targets will be harder to meet. The contradiction between ecological and power generation targets is increasingly aggravated—it is bound to sacrifice the efficiency of power generation in exchange for ecological benefits.

### *5.4. Sensitivity Analysis*

In order to reveal the law of mutual transformation between power generation and ecology, this paper analyzes the sensitivity of various dispatching schemes in each typical year. Using the dimensionless method, three new indicators are creatively proposed: the ratio of power generation to the installed capacity is defined as the coefficient of elasticity (*f* 1); the ratio of the difference in the total water requirement and water deficit to the total water demand is defined as the ecological elastic coefficient (*f* 2), as shown in Table 5.

*Water* **2020**, *12*, 915

The increase in power generation benefit when reducing the 1% ecological benefit is the ratio of ecological- and power-loss benefits (*k*), just like the Formula (4). The greater the *k*, the smaller the impact of ecological benefits on power generation benefits, and the greater the overall benefits of the scheme.

$$k(i) = \left\{f\_1(i) - f\_1(i-1)\right\} / \left\{f\_2(i) - f\_2(i-1)\right\} \tag{10}$$

Based on the Scheme 1, the k of each typical year can be obtained as shown in Figures 12 and 13.

**Figure 13.** Correlation between power generation and ecological elasticity in each typical year.

It can be seen from Figure 14 that the linear fitting relationship between *f* 1 and *f* 2 in each typical year is better, and the linear slopes of the typical years are 15.03, 14.70, and 22.45, respectively, showing a consistent and significant increasing trend. In response to the sensitivity of power generation and ecology, the normal year is the smallest, followed by the wet year and the dry year. That is, with the decrease in inflow, the amount of power generation, eco-elastic coefficient, and the number of ecological guarantee months are all reduced, the ecological water deficit is greatly increased, the restrictive relationship between power generation and ecological targets is strengthened, and the sensitivity between the two is enhanced. The decrease in the ecological elasticity coefficient will make the restoration of ecological environment more severe; the reservoir should take the ecological as the main goal in the dry year and take the phased measures to alleviate the ecological deterioration situation.

**Figure 14.** *k* value of different scheme intervals in each typical year.

It can be seen from Figure 13 that from Scheme 1 to Scheme 5, the k value in each typical year shows a continuous decreasing trend, and the maximum value of *k* in each typical year is 0.082, 0.109, and 0.071, respectively. The corresponding optimal mediation scheme all are Schemes 1–2 intervals. The recommended best coordination solution can maximize the comprehensive benefit.


**1. Introduction**

Today, 82 percent of Indians are stressed, as per the Cigna 360 Well-being study [1]. Rising stress levels even led to India being named the world's most depressed country a few years ago. In our society, the word "stress" has become overused. There are many more people suffering from stress in India than in any Western nation, despite the fact that the issue has long been dismissed as a "western" problem. More and more Indians are experiencing stress, depression, anxiety, and other related conditions due to factors such as work pressure, life's challenges, relationships, financial stress, or mental overload. The fact that stress is one of the main factors in cardiac arrests, blood pressure increases, and an increased risk of chronic diseases at an early age is even more shocking. A recent study by the World Health Organization and the Global Burden of Disease Study found that since the COVID-19 pandemic hit, there has been an increase in stress and anxiety among people [2]. Our problems have only gotten worse as a result of the global pandemic. Experts are concerned about a recent development that suggests we may be on the verge of a terrible mental illness pandemic—not to mention the emotional and financial toll the COVID-19 crisis has taken on many. There is still a lack of mental health awareness and

**Citation:** Gupta, M.V.; Vaikole, S.; Oza, A.D.; Patel, A.; Burduhos-Nergis, D.P.; Burduhos-Nergis, D.D. Audio-Visual Stress Classification Using Cascaded RNN-LSTM Networks. *Bioengineering* **2022**, *9*, 510. https://doi.org/10.3390/ bioengineering9100510

Academic Editors: Pedro Miguel Rodrigues, João Paulo do Vale Madeiro and João Alexandre Lobo Marques

Received: 13 September 2022 Accepted: 23 September 2022 Published: 27 September 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

a continuing stigmatization of mental illnesses in our country. Despite the internet and open conversations initiated by celebrities, there are many myths, taboos, and pieces of misinformation surrounding mental health. It is important to remember that going to therapy, seeing a psychiatrist, or being open about your feelings does not make you weak or bad. Mental illnesses should not be stigmatized, and getting the right help at the right time can prevent a lot of problems.

#### *1.1. Causes of Mental Stress*

Stress is a condition of mental pressure for individuals facing problems relating to environmental and social well-being which leads to many diseases. It was discovered that academic exams, human relationships, interpersonal difficulties, life transitions, and career choices all contribute to stress. Such stress is commonly associated with psychological, physical, and behavioral issues [3].

According to Lazarus and Folkman (1984), "stress is a mental or physical phenomenon formed through one's cognitive appraisal of the stimulation and is a result of one's interaction with the environment". The existence of stress depends on the existence of the stressor. Feng (1992) and Volpe (2000) defined a stressor as "anything that challenges an individual's adaptability or stimulates an individual's body or mentality". Stress can be caused by environmental factors, psychological factors, biological factors, and social factors, as shown in Table 1.

**Table 1.** Major causes of stress.

#### *1.2. Importance of Mental Stress Detection*

Human stress represents an imbalanced state [4] of an individual and is triggered when environmental demands exceed the regulatory capacity of the individual [5]. Because of its unhealthy effects [6], stress detection is an ongoing research topic among both psychologists and engineers, and has been applied to lie detection tests [7], emergency call identification [8], and the development of better human–computer interfaces [9]. People experience stress because of the requests and pressures put on them. The situation becomes more difficult when they perceive the circumstances to be overpowering and believe that it will be difficult to adapt [10]. Three levels of stress can be distinguished depending on the time of exposure to stressors. Acute stress is the innate "flight-or-fight" response in the face of short-lasting exposure to stressors, and it is not considered harmful. Episodic stress appears when stressful situations occur more frequently, but they cease from time to time. It is associated with a very stressful and chaotic life [11]. Finally, chronic stress, which is the most harmful, takes place when stressors are persistent and long-standing, such as family problems, job strain, or poverty [12]. To prevent stress reaching the highest level and help diminish the risks [13], it is necessary to detect and treat it in its earlier stages, i.e., when it is still acute or episodic stress. Stress identification has gained remarkably high attention in various fields in the last two decades. These fields include the medical sector, forensics, smart environments, teaching, learning and education, human–computer interactions, the emergency services, and of course real-time situations, which are the most crucial [14]. The identification of stress is a standout among the best research topic points for psychologists

as well as engineers. Stress management should begin before stress starts to cause medical problems. This is where stress monitoring can help. In recent years, interest in artificial intelligence-aided health monitoring or psychological counseling systems has increased due to the convenience and efficiency of machine learning-based algorithms. To provide appropriate services in these areas, the mental state of the user must be detectable. Among various emotional states, we focus on a methodology to detect the user's stress status.

#### *1.3. Role of Emotion in Stress Identification*

Emotions are present in almost every decision and moment of our lives. Thus, recognizing emotions awakens interest, since knowing what others feel helps us to interact with them more effectively. Emotions are considered a psychological state [15]. In the process of detecting stress using the audio-visual approach, it is important to detect the emotional state of the person. Hence, emotion recognition should be performed to decide on the stress level. The field of emotion recognition (ER) is a part of human–computer interaction, and this field has evolved very rapidly in the last decade. Several works have been performed on emotion recognition using audio and video; however, recent work has been completed on the fusion of the different modalities. Expressing emotions while interacting with others has always been a major part of communication among humans. Emotions are reflected through voice, facial expressions, and hand gestures, and can easily transcend the boundary of languages. A lot of work has been performed over the decades on automatic emotion recognition as part of human–computer interaction.

It must be considered that emotions are subjective to an individual, i.e., each subject may experience a different emotion in response to the same stimuli. Thus, emotions can be classified into two different models—the discrete model and the dimensional model. The discrete model includes basic emotions such as happiness, sadness, fear, disgust, anger, surprise, and mixed emotions such as motivation (thirst, hanger, pain, mood), selfawareness (shame, disgrace, guilt), etc. The dimensional model is expressed in terms of two emotions, valence (disgust, pleasure) and arousal (calm, excitement). The various emotions experienced by a human can be represented through the Plutchik wheel of emotion [16], as shown in Figure 1.

**Figure 1.** Plutchik wheel of emotion.

Several researchers have analyzed human stress using basic emotions. It is possible to map emotions with the stress level. Stress can be detected based on emotions obtained from the audio-visual data. Human emotions are expressed in the voice as well as on the face. The emotional state is extracted from the audio-visual data first. Positive emotions such as happiness, joy, love, pride, and pleasure can have a positive effect, such as improving daily work performance, and negative emotions such as anger, terrible, sad and, disgust can have

a negative impact on the health of a person. Positive and negative emotions are represented in Figure 2. Emotional signs such as depression, terrible, unhappiness, anxiety, agitation, and anger are responsible for stress. Stress can be detected from the two emotional states of anger and disgust.

**Figure 2.** Positive and negative emotions.

The valence–arousal space, as illustrated in Figure 3, can be subdivided into four quadrants, namely low arousal/low valence (LALV), low arousal/high valence (LAHV), high arousal/low valence (HALV), and high arousal/high valence (HAHV) [17].

**Figure 3.** Arousal–valence model for emotion representation.

Our model reached an accuracy of 91% on the 'The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS), outperforming some of the previous solutions evaluated in similar conditions. As far as we know, our study also represents the first attempt to combine speech and facial expressions to recognize the eight emotions in RAVDESS and finally conclude on a stressed or relaxed state.

The rest of the paper is organized as follows: Section 2 describes preceding research studies related to our proposal. Section 3 presents feature extraction, elaborating on the facial action coding system (FACS) and OpenFace. Section 4 summarizes the dataset and the proposed methodology. Section 5 describes the main results obtained and performance analysis. Finally, in Section 6, we discuss the main conclusions of our study and propose future research lines.

#### **2. Literature Review**

By outlining some of the difficulties that these systems encountered, we present earlier automatic stress detection techniques here. We describe the stress-inducing stimuli that were employed, how stress was measured, the signals that were gathered, and the machine learning techniques that were applied in these studies.

#### *2.1. Stress Detection Using Speech Signal*

Stress detection from speech signals has many applications. It is used in psychology to monitor the different stress levels of patients with different stress conditions and provide necessary treatments. The safety and security of a system can be established by monitoring the different stress levels of pilots, deep sea divers and military officials undertaking law enforcement. Stress detection is also useful in speaker identification, deception detection and identification of threatening calls in a few cases of crimes [18]. In order to effectively express his or her message, a person must choose the words to use at each stage of speaking. These choices, as well as the language, syntax, and timing of speech, can all be impacted by stress [19,20]. These changes in wording, grammar, and timing can then be employed as vocal cues to indicate stress. Other changes are also brought on by stress, though. For instance, in order to create sound waves during speaking, the body modifies the tension of many muscles to push air through the vocal folds and out of the vocal tract [21]. Stress raises the breathing rate and muscle tension, which alters speech mechanics and, as a result, the way speech sounds [22,23]. A voice-based stress detection system, named StressSense [24], was implemented on Android phones to detect the stress levels from human voice. The stress model was developed in several contexts, testing various speakers and events.

Kevin Tomba et al. [25] worked on the Berlin Emotional Database (EMO-DB), the Keio University Japanese Emotional Speech Database (KeioESD) and the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS). SVM and ANN algorithms were used. It was found that mean energy, mean intensity, and Mel frequency cepstral coefficients proved to be good features for speech analysis. However, only audio signals were considered, and not audio-visual data. N.P. Dhole and S.N. Kale investigated RNN classification and used it on the BERLIN and HUMAINE Datasets. They also used Audacity software to build real datasets for recurrent neural network applications. Despite being effective for audio signals, the efficiency percentage was not calculated [14]. Audio-visual data were not considered. Mansouri et al. [26] used a wavelet and neural network to create and implement an emotion identification system from speech signals. EMO-DB and SAVEE were used. The accuracy was judged to be satisfactory. The procedure, however, was time-consuming. The detection of stress was not taken into account.

#### *2.2. Stress Detection Using Audio-Visual Data*

Speech and facial expression are two natural and effective ways of expressing emotions when human beings communicate with each other. During the last two decades, audio-visual emotion recognition integrating speech and facial expression has attracted extensive attention owing to its promising potential applications in human–computer interaction [27,28]. However, recognizing human emotions with computers is still a challenging task because it is difficult to extract the best audio and visual features characterizing human emotions.

G. Giannakakis et al. [29] recorded videos using a camera. The videos' facial cues were used to identify signs of anxiety and stress. This method achieved good classification accuracy. However, a 1 min video duration could yield more reliable estimates. Kah Phooi et al. [30] used the eNTERFACE and RML (RAVDESS) datasets. They used a combined rule-based and machine learning approach for emotion recognition using audio-visual data. Anupam Agrawal and Nayaneesh Kumar Mishra [31] used SAVEE and created their own dataset. They worked on emotion classification based on the fusion of audio and visual data. However, it was found that the results can be improved using deep learning techniques. Noroozi, F et al. [32] worked on audio-visual emotion recognition in video clips. The datasets used were SAVEE, eNTERFACE'05, and RML.

Audio-visual data were not considered for the stress detection, and only audio signals were used. Moreover, the accuracy of the results can be improved. Although there is much research discussing the recognition and analysis of the six basic emotions, i.e., anger, disgust, fear, happiness, sadness, and surprise, considerably less research has focused on stress and anxiety detection from audio visuals, as these states are considered as complex emotions that are linked to basic emotions (e.g., fear). The results of emotion state recognition from audio-visual data can be improved using deep learning techniques, which can be further used to detect stress.

#### *2.3. Analysis*

Overall, this seems to be an interesting area of research, and the analysis of the existing work would help in carrying out future research. Table 2 provides an overview of numerous studies reflecting the same area of interest, together with the datasets they employed, the techniques used, the pros of these techniques, and the scope for advancement.

#### **Table 2.** Analysis of earlier work in the same field of study.

To sum up, despite the fact that other works in the literature also performed multimodal emotion recognition on RAVDESS, such as Wang et al. [33], who used facial images to generate spectrograms, which were then used for data augmentation to improve the SER model performance in six emotions, our work is the first that, to our knowledge, detects the stressed and relaxed state using the audio-visual information of RAVDESS by means of aural and facial emotion recognition using the eight emotions.

#### *2.4. Transition from Holistic Facial Recognition to Deep Learning Based Recognition*

In the 1990s and 2000s, the face recognition community was dominated by holistic techniques. Faces are represented using holistic approaches utilising the complete facial region. Many of these approaches function by projecting facial photographs into a lowdimensional space that eliminates unimportant features and variances. PCA is one of the most prominent techniques in this field. Deep neural networks trained with extremely huge datasets have lately supplanted older approaches based on hand-crafted features and typical machine learning techniques. Deep face recognition algorithms, which employ hierarchical design to learn discriminative face representation, have significantly enhanced state-of-the-art performance and spawned a multitude of successful real-world applications. Deep learning employs many processing layers to discover data representations with numerous feature extraction levels [34,35].

#### **3. Feature Extraction from Facial Expressions**

### *3.1. Facial Action Coding System*

Eckman and Friesen [36] created the FACS technique to analyze facial microexpressions and identify the emotions of the persons being studied. It is predicated on the notion that various facial muscle patterns can be linked to various emotions, and that the face areas where these muscles are active can be used to identify an individual's emotion. The fundamental benefit of FACS over other face analysis techniques is that it can detect concealed emotions, even when the person is attempting to imitate other emotions. People's emotions are frequently assessed using FACS.

FACS divides the face into 46 action units (AUs), as shown in Figure 4, which can be either nonadditive (an AU's activity is unrelated to the activity of other AUs) or additive (when one AU is activated, it causes another AU or group of AUs to activate). The Action units along with their pictorial representation is depicted in Table 3 below. The Facial Action Coding System (FACS) is used to classify human facial movements according to how they appear on the face. FACS encodes the movements of specific facial muscles from slight instantaneous changes in facial appearance.

**Figure 4.** Action units.

#### **Table 3.** Action units.

Almost any anatomically conceivable facial expression can be coded using FACS, which breaks it down into the specific action units (AUs) that give rise to the expression, as shown in Table 4. FACS is a widely used method for accurately describing facial expressions. Facial expressions are considered as the signal, and emotions as the message.

**Table 4.** List of AUs involved in basic expressions.

#### *3.2. OpenFace*

OpenFace is a tool intended for computer vision and machine learning researchers, the affective computing community, and people interested in building interactive applications based on facial behavior analysis. OpenFace is the first toolkit capable of facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation with available source code for both running and training the models. Specifically, OpenFace can identify AUs 1, 2, 4, 5, 6, 7, 9, 10, 12, 14, 15, 17, 20, 23, 25, 26, 28 and 45.

There are two ways to categorize AUs: intensity and presence. Presence (for instance, AU01 c) indicates whether an AU is visible on the face. On a scale of 1 to 5, intensity indicates the degree of AU intensity (min to max). Both of these scores are presented by OpenFace.

These two scores are provided by OpenFace. The output file's column AU01 c encodes 0 as not present and 1 as present for the presence of AU 1. The output file's column AU01 r has continuous values in the range of 0 (not present), 1 (present at minimum intensity), and 5 (present at maximum intensity) for the intensity of AU 1.

#### **4. Proposed Method to Classify Mental Stress**

Our proposed stress detection framework includes two systems: a speech emotion recognizer and a face emotion recognizer. The outputs of these subsystems were integrated to identify the dominant emotion and eventually result in a stressed or unstressed state. In the current research, we made a point to highlight a novel method of implementing two different algorithms to function better than any single algorithm working individually. The proposed algorithm not only improves the overall accuracy in determining emotions but also is faster than each individual algorithm, as it uses the advantages of each algorithm and eliminates the disadvantages or time-consuming processes of each of them. Further, the work may seem complicated at the first glance; however, the accuracy improvement in the field of mental stress determination is what we are looking for, and our set objectives for the research work are met through the approach.

#### *4.1. The RAVDESS Dataset*

The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) is licensed under CC BY-NA-SC 4.0. The paper by Livingstone SR and Russo FA (2018) described the construction and validation of the dataset.

There are 7356 files in the RAVDESS. Each file was rated ten times for emotional validity, intensity, and authenticity. A group of 247 people who were typical untrained adult research participants from North America provided ratings. The second group of 72 people provided test–retest data. Emotional validity, interrater reliability, and test–retest intra-rater reliability were all reported to be high.

#### 4.1.1. Description

The dataset included all 7356 RAVDESS files in their entirety (total size: 24.8 GB). The three modality formats for each of the 24 actors were audio-only (16 bit, 48 kHz.wav), audiovideo (720p H.264, AAC 48 kHz,.mp4), and video-only (480p H.264, AAC 48 kHz,.mp4) (no sound). Please take note that Actor 18 did not have any song files.

#### 4.1.2. Data

A total of 4948 samples were used for this task. Audio files were extracted from videoaudio files using the "mp4 to wav" algorithm. The filenames for each of the 7356 RAVDESS files were distinctive. A seven-part numerical identifier comprised the filename (e.g., 02-01-06-01-02-01-12.mp4). These codes specified the properties of the stimulus:

The filename identifiers used are illustrated in Table 5 below.

#### **Table 5.** Identifiers of RAVDESS filenames.

Taking the example of the RAVDESS filename 02-01-06-01-02-01-12.mp4: Video-only (02) Speech (01) Fearful (06) Normal intensity (01) Statement "dogs" (02) 1st Repetition (01) 12th Actor (12) Female, as the actor ID number is even.

*4.2. Proposed System*

4.2.1. Why RNN?

ANN and/or CNN have been presented before in the literature, and an accuracy of around 80% has been reported for them. In our literature review, we did not find any

individual algorithm which would improve the accuracy of prediction beyond 90%. So, we needed a different approach wherein we combined two relatively less processor-heavy algorithms to work on and improve the accuracy and simultaneously work at a faster rate. However, as rightly pointed out by the reviewer, in our continued plan for our research work, we will make a point to work on ANN- and CNN-based algorithms to either present a comparative analysis or to cascade them as per our intended method to verify their performance for the said cause. Recurrent neural networks (RNNs) have been successfully applied to sequence learning issues such as action identification, scene labeling, and language processing. An RNN has a recurrent connection, unlike feed-forward networks such as convolutional neural networks (CNNs), where the previous hidden state is an input to the subsequent state. An enhanced RNN, or sequential network, called a long short-term memory network, allows information to endure. It is capable of resolving the RNN's vanishing gradient issue. Persistent memory is achieved via a recurrent neural network or RNN. Let us imagine that when reading a book or viewing a movie, you are aware of what happened in the preceding scene or chapter. RNNs function similarly; they retain the knowledge from the past and apply it to process the data at hand. Due to their inability to remember long-term dependencies, RNNs have this drawback. Long-term dependency issues are specifically avoided when designing LSTMs.

In our case, RNN is used to classify data of facial landmark position with respect to time for visual data analysis and to classify the pitch of different frequencies of the audio signal with respect to time to determine the emotions.

Speech and facial expressions are used to detect users' emotional states. These modalities are combined by employing two independent models connected by a novel approach. By merging the information from aural and visual modalities, audio-visual emotion identification is vital for the human–machine interaction system. We propose a cascaded RNN-LSTM approach for audio-visual emotion recognition through correlation analysis. The emotions will finally be categorized as a stressed mental state or a relaxed mental state. We use the RAVDESS dataset for the verification of the proposed algorithm.

#### 4.2.2. Speech-Based Stress Detection

The flowgraph for stress recognition using speech signals is shown in Figure 5. In the proposed approach, two closely related ML algorithms viz. RNN and LSTM are cascaded together, as shown in the flowchart (Figure 6). Cascading improves the convergence time of the combined algorithm. The Mel-frequency cepstral coefficient (MFCC) is the most well-known spectral feature, since it is used to model the human auditory perception system. Here, speech signals are pre-processed and filtered using MFCC. The features of the input signal are extracted at this stage. These features are sent to 4 neurons RNN and 10 neuron LSTM working in parallel with each other. The RNN module which does not have a cell state generates the required labels for these features while the LSTM module is used for emotion prediction only. The LSTM module receives labels from the RNN module as its first input and the extracted features from MFCC as its second input. This combined approach reduces the size of the LSTM module by reducing the number of neurons required for emotion prediction, e.g., a 40-neuron LSTM module is replaced by a 10-neuron LSTM module with a 4-neuron RNN module to achieve the same result at a faster rate. Moreover, to prevent the model from overfitting, the LSTM module employs a dropout layer by randomly setting other edges of the hidden layer to zero. This reduces the convergence time to about 3/4 of the traditional approach.

**Figure 5.** Flowgraph for stress recognition using speech signals.

**Figure 6.** Speech Signals based Human Stress detection.

4.2.3. Proposed Method for Audio-Visual Based Stress Detection

Our deep learning model contains two individual input streams, i.e., the audio network processing audio signals with the cascaded RNN-LSTM model, and the visual network processing visual data with the hybrid RNN-LSTM model. The flowchart for the algorithm is in Figure 7 below.

In the proposed algorithm, audio files are extracted from the video files and processed separately. Librosa is used to process audio files while OpenFace is used to process video files. Overall, 66% of samples are used for training purposes, while the rest are used for testing the algorithm. In the algorithm, RNN and LSTM work parallelly to improve the speed of the feature extraction process. Audio signals need 20 neurons in the LSTM network while video signals need 40 neurons due to their signal processing requirements. MFCC is used as a filter for feature extraction. Dropout layers are used to prevent data from overfitting. Max pooling with convolution creates the final 8 required labels from the features. A dense sigmoid function is used for the final classification of the output with 10 neurons each. The separate outputs of both audio and video files are compared on a common platform to improve the accuracy by matching the missing labels. The following emotions are predicted in this model: "neutral": "01", "calm": "02", "happy": "03", "sad": "04", "angry": "05", "fearful": "06", "disgust": "07", "surprised": "08". Finally, 8 emotions are classified into 2 mental states—stressed and relaxed. First of all, we chose the method of comparing both audio and video files to avoid any misrepresentation of emotions due to the use of only one kind of file. In a scenario where the classification of both files is different, the average sum of scores of each signal will determine the probability of the inclination of the signals to a particular emotion. However, such a scenario has not yet occurred in our work, and hence the algorithm has not yet been validated.

**Figure 7.** Proposed system workflow.

#### *4.3. Analysis*

We used the Jupyter interface to run the program. LibROSA, a python package, was used for music and audio analysis, while the OpenFace package was used for facial motion tracking.

We plotted the signal from a random file with audio and facial recognition separated as shown in Figure 8 below.

**Figure 8.** Audio signal.

Two facial recognition examples are illustrated in Figures 9 and 10 for frames 36 and 16 respectively.

**Figure 9.** Facial recognition example 1.

**Figure 10.** Facial recognition example 2.

#### **5. Experimental Results**

NumPy array was created for extracting Mel-frequency cepstral coefficients (MFCCs), while the classes for prediction were extracted from the name of the file.

To apply the cascaded RNN-LSTM method effectively, we need to expand the dimensions of our array, adding a third one using the NumPy "expand\_dims" feature.

Layer (type) Output Shape Param #

======================================================

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

conv1d\_1 (Conv1D) (None, 40, 128) 768

activation\_1 (Activation) (None, 40, 128) 0

dropout\_1 (Dropout) (None, 40, 128) 0.1

max\_pooling1d\_1 (MaxPooling1 (None, 5, 128) 0

conv1d\_2 (Conv1D) (None, 5, 128) 82,048

activation\_2 (Activation) (None, 5, 128) 0

dropout\_2 (Dropout) (None, 5, 128) 0.5

flatten\_1 (Flatten) (None, 640) 0

dense\_1 (Dense) (None, 10) 6410

activation\_3 (Activation) (None, 10) 0

======================================================

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Total params: 89,226 Trainable params: 89,226 Non-trainable params: 0

The model loss of epochs based on training and test data is shown in the Figure 11 below. Figure 12 indicates the accuracy of the model.

**Figure 12.** Accuracy of the Model.

To understand the errors of the top solution, we extracted the confusion matrix of the SVM, LSTM, and RNN-LSTM approaches with an accuracy of 76%, 82%, and 91%, respectively. The confusion matrix displayed in the Figures 13–15 below is the rounded

average value of the errors and the correct predictions obtained from the folds of the 5-CV. This matrix will display an average of 288 samples (1440/5).

**Figure 13.** Average confusion matrix for the SVM algorithm. Accuracy = 76%.

**Figure 14.** Average confusion matrix for the LSTM Algorithm. Accuracy = 82%.

Figure 15 reveals that the RNN-LSTM approach showed a good performance, except for some samples. The 'Sad' class contained the highest number of errors, mistaking this class in most cases for other emotions such as 'Disgusted' or 'Fearful', although it also confused this emotion with 'Calm', which may be caused by the low arousal level of both emotions.

**Figure 15.** Average confusion matrix for RNN-LSTM approach. Accuracy = 91%.

The proposed algorithm is compared with the conventional ones and the performance analysis is presented the Table 6.

**Table 6.** Performance analysis of the proposed system on RAVDESS.

Final Output:

1633/1633 [==============================]—0s 125s/step Accuracy: 91.00%

The existing work was focused on either audio or facial images. In audio-visual data, the separate output of audio and video files was compared on a common platform to improve accuracy by matching the missing labels. In order to enhance the accuracy further, we increased the dimensions of the dataset, as LSTM works better with more data. The accuracy for prediction for the proposed algorithm for the RAVDESS dataset is 91%.

#### **6. Our Contributions**

Only image-based classification may give polarized results in cases where the image under processing lacks the overall gesture being conveyed. Moreover, using audio and visual signals will help to improve the emotion classification accuracy, which is needed to determine whether the algorithm further needs to be fully developed for the medical determination of mental stress. Although we used well-established packages for our work, we made several changes to the algorithm to make it work and provide novelty. The changes in the algorithm include cascading or the parallel operation of algorithms (which usually runs sequentially), the addition of dropout layers to adjust the blank values and to

avoid overfitting of the data, and processing of both audio and video files to compare and improve classification accuracy. We would like to state that this method of implementing the algorithm has never been reported in the literature before.

#### **7. Conclusions and Future Scope**

Detecting stress is essential before it turns chronic and leads to health issues. The current paper suggests that audio-visual data have the potential to detect stress. In our society, stress is becoming a major concern, and modern employment challenges such as heavy workloads and the need to adjust to ongoing change only make the situation worse. In addition to severe financial losses in businesses, people are experiencing health issues related to excessive amounts of stress. Therefore, it is crucial to regularly check your stress levels to detect stress in its preliminary stages and prevent harmful long-term consequences. The necessity for individuals to handle chronic stress gave rise to the concept of stress detection. The accuracy of the cascaded RNN-LSTM approach for the RAVDESS dataset is 91%. The obtained results are 15–20% better than those of other conventional algorithms. The proposed method is an excellent starting point to work towards mental health by detecting stress and improving one's quality of life.

The evaluation of the test results showed that the successful detection of stress is achieved, although further improvements and extensions can be made. The implementation of this system can be improved by using more efficient data structures and software to reduce delays and achieve real-time requirements.

**Author Contributions:** M.V.G. writing—original draft preparation, methodology, investigation; S.V. investigation, conceptulization, methodology; A.D.O. writing—original draft preparation, data curation, formal analysis; A.P. investigation, writing—original draft preparation; D.P.B.-N. writing review and editing, resources, conceptulization. D.D.B.-N. writing—review and editing, resources, methodology. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was supported by Gheorghe Asachi Technical University of Ia¸si—TUIASI-Romania, Scientific Research Funds, FCSU-2022.

**Institutional Review Board Statement:** The study was conducted in accordance with the Declaration of Helsinki, and approved by the Research Advisory Committee of Datta Meghe College of Engineering (protocol code RAC/006/2022 and date of approval-01/06/2022).

**Informed Consent Statement:** This research work has been done on the synthetic dataset. Hence no real patients are involved.

**Data Availability Statement:** Not applicable.

**Acknowledgments:** The authors would like to gratefully acknowledge the support of Datta Meghe College of Engineering, Airoli, Navi Mumbai, and New Horizon Institute of Technology and Management, Thane for facilitating a conducive environment to perform our research work.

**Conflicts of Interest:** The authors declare that they have no conflict of interest.

#### **References**


**1. Introduction**

The human coronavirus SARS-CoV-2 or COVID-19 that emerged in late 2019 causes a respiratory tract infection of the COVID-19 disease and, according to the official website of the Ministry of Health, which reports World Health Organization data, has currently resulted in more than 627 million confirmed cases and over 6.58 million deaths worldwide up to October 2022.

The characteristic symptoms presented by patients affected by SARS-CoV-2 led to the belief that it was a pneumonia with an interstitial component, very often bilateral, associated with respiratory symptoms which in the early phase are generally limited, but which can subsequently lead to progressive clinical instability with respiratory failure. The phenomenon of the so-called "silent hypoxemia", characterized by low blood oxygenation values in the absence of subjective feeling of dyspnea, is characteristic of this phase of the disease. This scenario, in a number of people, can evolve towards a worsening clinical picture dominated by a cytokine storm, the excessive immune response from the uncontrolled release of a series of interleukins, chemokines, interferons, and tumour necrosis factors and the consequent hyperinflammatory state, which determines local and systemic consequences. Such a response represents a negative prognostic factor producing, at the pulmonary level, pictures of arterial and venous thrombi of small vessels and evolution towards severe and sometimes permanent pulmonary lesions (pulmonary fibrosis) [1]. In particular, vascular permeability is increased, resulting in a large amount of fluid and blood cells entering the alveoli, causing dyspnoea and even respiratory failure, desquamation

**Citation:** Traina, G. The Connection between Gut and Lung Microbiota, Mast Cells, Platelets and SARS-CoV-2 in the Elderly Patient. *Int. J. Mol. Sci.* **2022**, *23*, 14898. https://doi.org/ 10.3390/ijms232314898

Academic Editor: Maria Teresa Mascellino

Received: 24 October 2022 Accepted: 25 November 2022 Published: 28 November 2022

**Copyright:** © 2022 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

of alveolar cells and hyaline membrane formation. A mass of fluid similar to mucus accumulates in the lungs, and this accumulation is caused by an excessive immune response due to signalling molecules, in particular interleukin-(IL)-6, IL-8, and tumour necrosis factor (TNFα) [1–4]. Cytokine overproduction and cytokine storm induce clinically relevant extrapulmonary effects on various key organs such as heart, kidney, liver and intestine and dysbiosis [3]. The final stages of this very severe clinical picture can lead to multi-organ failure, with cardiovascular, gastrointestinal, haematological, respiratory, neurological and renal complications [5].


**Anja Sundal \* and Helge Hellevang**

Department of Geosciences, University of Oslo, 0371 Oslo, Norway<EMAIL_ADDRESS>**\*** Correspondence<EMAIL_ADDRESS>Tel.: +47-22-85-66-52

Received: 9 September 2019; Accepted: 24 October 2019; Published: 31 October 2019

**Abstract:** Reservoir characterization specific to CO2 storage is challenging due to the dynamic interplay of physical and chemical trapping mechanisms. The mineralization potential for CO2 in a given siliciclastic sandstone aquifer is controlled by the mineralogy, the total reactive surface areas, and the prevailing reservoir conditions. Grain size, morphologies and mineral assemblages vary according to sedimentary facies and diagenetic imprint. The proposed workflow highlights how the input values for reactive mineral surface areas used in geochemical modelling may be parameterized as part of geological reservoir characterization. The key issue is to separate minerals both with respect to phase chemistry and morphology (i.e., grain size, shape, and occurrence), and focus on main reactants for sensitivity studies and total storage potentials. The Johansen Formation is the main reservoir unit in the new full-value chain CO2 capture and storage (CCS) prospect in Norway, which was licenced for the storage of CO2 as of 2019. The simulations show how reaction potentials vary in different sedimentary facies and for different mineral occurrences. Mineralization potentials are higher in fine-grained facies, where plagioclase and chlorite are the main cation donors for carbonatization. Reactivity decreases with higher relative fractions of ooidal clay and lithic fragments.

**Keywords:** CCS; CO2 storage; mineralization; carbonatization; mineral trapping; mineral sequestration; Johansen Formation; North Sea; sedimentary facies


**6. Pipe wear**

An unavoidable consequence of flows containing particles is wear. Enormous costs to industry are attributable to wear, but costs can be reduced by careful design. Since 2017, many advances have been made in the study of erosion-corrosion in experimental and *in-situ* monitoring of wear, condition monitoring, the monitoring of wall erosion and computational modeling to name but three.

Designers of slurry systems will often reach for answers from Computational Fluid Dynamics (CFD). There is certainly a promising future for CFD erosion modeling, but that advance is not yet with us. Despite improved particle impact modeling, dynamically deforming geometry and improved particle tracking, the accuracy of erosion predictions is poor. There is hope of improvement from two comparatively new approaches explored by authors in this volume. The Discrete Phase Model (DPM) plots the path of a group of dynamically similar particles through a continuous liquid, while the Discrete Element Model (DEM) tracks individual particles and

solves the equations of motion as they interact with each other. In theory, the accuracy of the DPM approach can therefore be improved by a coupling with DEM. A CFD-DPM-DEM combination has been used to model air-particle flows, but no attempt at extending the technique to solid-liquid flows has been reported to date. There is great potential for this combination but a disincentive to its development is the very large computational cost that would be incurred. One of the most significant improvements in the application of CFD is the Moving-Deforming-Mesh (MDM) approach. The computational mesh is updated at regular intervals based on local erosion rate and the geometry of the wall. In this way erosive deformations are integrated into the flow field and secondary erosion patterns can be predicted.

Very little research has been carried out on the influence of pipe layout. on the life of vulnerable components. Pipe geometry upstream of a vulnerable pipe section can have a significant effect on erosion rates. For example, the swirl pipe design described above can show significant wear reductions in adjacent downstream pipe components. Unsurprisingly internal surface finish (particularly in pipe elbows) has a disproportionate effect on the erosion pattern suffered in service. Comparatively minor melioration in this area could be influential in increasing component lifetimes.


**2. Results**

## *2.1. Haematology/Peripheral Leucocyte Responses*

In experiment one, total WBC and RBC as well as MCH did not change significantly among different dietary treatments at both 1 and 2 weeks of feeding (Table 1). However, fish fed D2 presented a higher haemoglobin (Hb) concentration than D1 and D3 fed fish (Table 1). Differential leucocyte counts showed different modulation patterns between dietary treatments regardless of the sampling point (Table 2). For instance, the D1 fed group showed lower lymphocyte numbers at both 1 and 2 weeks, when compared to the other dietary treatments (Table 2). Whereas peripheral neutrophils increased in D2 fed fish compared to those fed CTR (Table 2). Circulating monocytes were not significantly modulated by dietary treatments at either 1 or 2 weeks of feeding.

**Table 1.** Haemoglobin, mean corpuscular haemoglobin (MCH), red blood cells (RBC) and white blood cells (WBC) in gilthead seabream juveniles after 1 and 2 weeks of feeding (experiment one). Data are the mean ± SEM (*n* = 12).

Different capital letters represent significant differences in time regardless of diet and between diets regardless of time (*p* < 0.05), ns (not significant).

**Table 2.** Absolute values of peripheral blood leucocytes (thrombocytes, Lymphocytes, monocytes and neutrophils) in gilthead seabream juveniles after 1 and 2 weeks of feeding (experiment one). Data are the mean ± SEM (*n* = 12).

Different capital letters represent significant differences in time regardless of diet and between diets regardless of time (*p* < 0.05), ns (not significant).

After the inflammatory insult (experiment two), Hb increased at 24 h following inoculation with the inactivated bacteria, while MCH, total WBC and RBC remained unchanged

(Table 3). Peripheral lymphocyte numbers decreased at 24 h compared to 0 h, returning to resting values at 48 h (Table 4). Circulating neutrophil levels increased at 24 h and 48 h following pathogen inoculation compared to time 0 h (Table 4). Total thrombocyte and monocyte concentrations were unaffected (Table 4).