According Figure 5, we can see how the variation of the economic and environmental functions occurs for each case studied. From these results, we can verify among cases B, C, and D what the best decision related to the wind farm unit to be installed is in comparison with the results obtained in case A, considering the interest of the decision-maker in obtaining the reduction of the cost associated with the economic function (*Fe* + *Fw*) or reduction of the emission associated with the environmental function (*Fa*).

Table 1 presents the goal-exceeded values for each case (A, B, C, and D), where only the variables with positive deviations assume values greater than zero.

**Table 1.** Deviation variable values from weighted goal programming (WGP) for cases A, B, C, and D.

The values presented in Table 1 show how much the values found for the (*Fe* + *Fw*) and *Fa* functions of the solution chosen by the decision-maker are above the set goal.

For case A, where we do not have the wind farm insertion, according to the deviation *p*1, the value of the *Fe* + *Fw* function exceeded 7.27% of the goal set ((*Fe* + *Fw*)*min* = 840.08); according to the deviation *p*2, the value of *Fa* exceeded 23.79% of the goal set ((*Fa*)*min* = 235.69). For the cases where we have the wind farm insertion in case B, according to the deviation *p*1, the function *Fe* + *Fw* exceeded 8.59% ((*Fe* + *Fw*)*min* = 769.85) and, according to the deviation *p*2, *Fa* exceeded 18.59% of the goal set ((*Fa*)*min* = 205.97). In case C, according to the deviation *p*1, the value of *Fe* + *Fw* function exceeded 6.24% ((*Fe* + *Fw*)*min* = 863.31), and according to the deviation *p*2, the value of *Fa* exceeded 18.44% of the goal set ((*Fa*)*min* = 202.55). In case D, according to the deviation *p*1, the value of the *Fe* + *Fw* function exceeded 8.20% ((*Fe* + *Fw*)*min* = 804.35), and according to the deviation *p*2, the value of *Fa* exceeded 21.89% of the goal set ((*Fa*)*min* = 198.24). It is the decision-maker who verifies which of the solutions found are satisfactory and meets the production interest of the power plant, according to the results obtained by the proposed problem.

For determining which wind farm will be deployed, an analysis was made of the solutions chosen by the decision-maker for each case investigated. The comparison of the values found was made in relation to the solution obtained by exploring the WGP technique, which is the solution that is nearest to the ideal solution. Table 2 presents the solution found in case A with the weights *w*1 = 0.35 and *w*2 = 0.15, and for cases B, C, and D with the weights *w*1 = 0.40 and *w*2 = 0.10, the weight values for each case are inserted in the Table A3. According to the *w*1 weights defined for all cases, greater emphasis was given to the (*Fe* + *Fw*) objective, thus giving preponderance to the economic objective.

**Table 2.** Value of the cost and emissions found by WGP for cases A, B, C, and D.

The values presented in Table 2 show that the insertion of wind power into the power system reduces pollutant emissions. With the insertion of wind farm unit 1 (case B), there was a reduction of 47.53 Kg/h with relation to the emission of pollutants found without the insertion of wind farm units (Case A), corresponding to a reduction of 16.29% in the emission of polluting gases through the burning of fossil fuels. In case C, when inserting the wind farm unit 2, the reduction was 51.87 Kg/h compared to case A, referring to 17.78%, and in the implantation of wind farm unit 3 (case D), the reduction was 50.13 Kg/h, corresponding to the 17.18% reduction in pollutant emissions relative to case A.

Comparing the results obtained in Table 2 for cases B, C, and D as compared to case A, if the decision-maker's interest is to obtain the best pollutant reduction, the option would be to install wind farm 2, corresponding to case C, with a 17.18% emission reduction and with the lowest operating cost reduction, which is 1.41%. On the other hand, if the option were to improve cost reduction, the choice would be the implantation of wind farm 1 for case B, with 7.24% cost reduction and 16.29% polluting reduction, which is the worst result for emission reduction. An intermediate and balanced choice would be to deploy wind farm 3 in case D, respectively, with an emission reduction of 17.18% and cost reduction of 3.43%.

Therefore, from the results obtained, it can be seen that the addition of a wind farm in all investigated cases (B, C, and D) ensures the reduction of pollutant emissions and operating costs when compared with case A without the insertion of wind farm. The results show the importance and feasibility of exploiting the wind energy in the power generation system in terms of improving the overall sustainability of power systems.

#### **4. Conclusions**

This paper has implemented a novel methodology based on a combination of goal programming and the progressive bounded constraints as well as weighted goal programming in order to resolve the economic and environmental dispatch problem for a combination of thermal and wind power units. This has allowed the quantification of the benefits of adding wind power units into a conventional power system. The results in Figures 1–5 show that:

The above results should give encouragement to managers and stakeholders of solely thermal power systems to consider inserting wind-based units into their systems if the conditions allow. Equally, designers of new systems should consider the appropriate inclusion level of wind power in their systems. In both cases, this paper has presented a methodology that can be used to give a quantification of the environmental and economic benefits of wind power inclusion.

This paper has examined one pair of energy sources (thermal and wind) with two sustainability objectives (economic and environmental). Potential further work could involve the addition of more conventional and intermittent energy sources as well as additional objectives.

**Author Contributions:** Conceptualization of the new multi-objective model and of the new solution methodology, A.C.d.S.M., A.R.B., and D.J.; methodology, A.C.d.S.M., A.R.B., D.J., and L.N.; software learning and application, A.C.d.S.M., E.M.S., and E.C.B.; validation of results, A.C.d.S.M., L.N., E.M.S., and E.C.B.; formal analysis, A.C.d.S.M., A.R.B., and D.J.; investigation, A.C.d.S.M., A.R.B., and D.J.; writing—original draft preparation, A.C.d.S.M., D.J., and A.R.B.; writing—review and editing, D.J. and A.C.d.S.M.; supervision, A.R.B.; project administration, A.R.B., A.C.d.S.M., D.J., and L.N. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was financed in part by PROPG-UNESP (PGEE/2018).

**Acknowledgments:** The authors thank the financial support of the FAPESP (Project n. 2014/20853-0, 2013/18036-1) and CNPq (Projects n. 305548/2019-0, 313495/2017-3, and 309780/2017-9).

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Appendix A**

The cost coefficients of the generating units that are presented in Table A1 can be found in Gonçalves et al. [21] and Ravi, Chakrabarti, and Choudhuri [34].

**Table A1.** Thermal generator data.

The wind farm data that are given in Table A2 can be found in CRESESB [35] and ONS [36].

The values of the weights used in each case are given in Table A3.

**Table A3.** The weights used in each case.

#### **References**

c 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **Quantifying the Sustainability of Products and Suppliers in Food Distribution Companies**

#### **Marina Segura 1,\*, Concepción Maroto 1 and Baldomero Segura 2**

Received: 14 September 2019; Accepted: 22 October 2019; Published: 23 October 2019

**Abstract:** Supplier evaluation is a relevant task of supply chain management where multicriteria methods make great contributions to manufacturing industries. This is not the case in food distribution companies, which have a key role in providing safe and affordable food to society. The purpose of this research is to measure the sustainability of products and suppliers in food distribution companies through a multiple criteria approach. Firstly, the system proposed provides indicators to qualify products and assess the food quality, using the compensatory Multi-Attribute Utility Theory (MAUT) model. Secondly, these indicators are included in supplier evaluation, which takes economic, environmental, and social criteria into account. MAUT and Preference Ranking Organisation Method for Enrichment Evaluation (PROMETHEE), a non-compensatory method, are used for supplier evaluation. This approach has been validated for fresh food in a supermarket chain, mainly using historical data. Partial indicators, such as food safety scores, together with global indicators of suppliers, inform the most appropriate decisions and the most appropriate relations between companies and providers. Poor performance in food safety can lead to the disqualification of some suppliers. MAUT is good for qualifying products and is easy to apply at the operational level in logistic platforms, while PROMETHEE is more suitable for supplier segmentation, as it helps to identify supplier strengths and weaknesses.

**Keywords:** supplier evaluation; supplier segmentation; multi-attribute utility theory; preference ranking organisation method for enrichment evaluation; quality indicator; food safety; fresh food; sustainable supply chain; multicriteria

#### **1. Introduction**

In recent decades, sustainability has been an increasing concern for a society that involves everything from countries to companies and consumers. Sustainability can be assessed in relation to different scopes, such as a sector, supply chain, company, supplier, and/or product. Many decision-making problems deal with this concept from different perspectives. Life Cycle Assessment (LCA) is a common approach in the scientific literature dealing with product sustainability evaluation. Nevertheless, LCA is not an appropriate tool to support decision making in Sustainable Supply Chain Management (SSCM) [1]. The Multiple Criteria Decision Making (MCDM) approach is suitable to aggregate different dimensions of sustainability and provides a wide range of methods, which allow us to tackle a wide range of issues and support decision making [2].

Literature offers a huge number of research works focused on supplier evaluation, as well as well-structured reviews on this topic. The main approaches are Data Envelopment Analysis (DEA), Analytic Hierarchy Process (AHP), Analytic Network Process (ANP), Technique for Order Performance by Similarity to Ideal Solution (TOPSIS), Simple Multi-Attribute Rating Technique (SMART), Preference

Ranking Organisation Method for Enrichment Evaluation (PROMETHEE), ELimination Et Choix Traduisant la REalité (ELECTRE) and Linear, Mixed Integer, Goal, and Multiobjective Programming. The applications are concentrated in various manufacturing sectors, such as the automotive, electrical, and electronics sectors, among others [3–10].

The SSCM concept involves integrating all three dimensions of sustainability. Nevertheless, published research has mainly been focused on economic and environmental criteria, with scarce contributions that included social criteria. It is also interesting to point out that research has solved the supplier selection problem for one product only, although this situation is not frequent [11–19]. In practice, companies buy several products from several suppliers and the same supplier provides several products. Moreover, the majority of case studies used data based on opinions, judgements, and/or direct ratings from experts or managers [20–22]. There is a proposal for supplier evaluation for several products, applied to the food industry, in which decision making integrates historical data and expert knowledge [23].

There are many applications of MCDM methods to supplier evaluation for selection and segmentation in manufacturing companies, some in food industries [6–20]. Nevertheless, this situation contrasts with the lack of proposals for the food distribution business, such as supermarket chains. The food supply chain is different from other supply chains when it comes to sustainable management, due to the continuous change in the quality of products until the point of final consumption. In addition to sustainability, consumers demand quality and safety as important food attributes [24,25]. Lau et al. (2017) carried out the only research that applies MCDM to fresh food supplier evaluation in a supermarket chain in Australia. They applied Fuzzy AHP to obtain the weights of criteria, ELECTRE to rank the supplier performance with respect to food safety, as a non-compensatory criterion, and TOPSIS to rank suppliers qualified previously by the safety criterion [26]. Evaluating food suppliers from a sustainable perspective requires the definition of specific criteria and indicators, such as those that are applied to measure food safety [27]. This criterion can be included in the social dimension [3], although some authors propose defining food sustainability based on five dimensions: social, environmental, economic, health and ethical [28].

The aim of this research is to measure the sustainability of products and suppliers in food distribution companies through a multiple criteria approach. For this purpose, a decision support system was designed in order to first obtain product indicators, which allow the companies both to accept/reject products and to assess the quality of food. These product indicators are subsequently included in a global supplier evaluation from a sustainable perspective, which takes into account economic, environmental, and social criteria.

The main contribution of this paper is related to product and supplier evaluation in the last part of the fresh food supply chain, based on multiple criteria techniques, integrating historical and objective data on product and supplier performance rather than the judgements and direct rating from managers alone. This approach also allows us to classify suppliers by analyzing the historical and reliable indicators needed in a decision support system to inform decision making at operational, tactical, and strategic levels. This research innovates in proposing and validating a model to evaluate an important part of the fresh food supply chain from a sustainable perspective, based on historical and objective data. To our knowledge, this is a new and original proposal that allows us to generate information useful for making sustainable decisions on a regular basis in supermarket chains. This supply chain differs from others due to the change in the quality of products until they are consumed. In addition, new social criteria related to food safety are proposed in a novel and unified approach to dealing with sustainable fresh food and supplier evaluation in distribution companies, such as supermarket chains.

The rest of the paper is organized as follows: Section 2 briefly presents the methodology used and the decision support system proposed for food and supplier evaluation, which is based on several compensatory and non-compensatory multiple criteria techniques. The system proposed has been validated in a real company, a supermarket chain, as explained in Sections 3 and 4. In Section 3, criteria related to fresh food (fruits) are defined and their utility functions established in order to obtain quality indicators for products by applying Multi-Attribute Utility Theory (MAUT) methodology. The case study allows validation of the model, which is based on data about different varieties of yellow peach and from several suppliers. Section 4 presents the results of sustainable evaluation of suppliers, by integrating quality indicators of products with food safety, logistic, commercial, and environmental criteria. This sustainable evaluation of suppliers is carried out through PROMETHEE and MAUT. Outcomes from these two methodologies are compared, highlighting their key strengths and weaknesses. Finally, a discussion of results and conclusions is presented in the last two sections, respectively.

#### **2. A Multiple Criteria System for Sustainable Evaluation of Food and Suppliers in Distribution Companies**

Multiple Criteria Decision Making (MCDM) techniques stand out among the huge volume of literature focused on supplier evaluation [3,5,6,8]. When sustainability is taken into account, the MCDM approach is essential to integrate conflicting objectives in the analysis appropriately. Although there is quite a number of well-established methods, selection of the most suitable is a relevant issue. The nature of the real problem to be solved, data availability and type (qualitative/quantitative), uncertainty and decision makers, as well as the chance of success in actual implementation are some of the basic issues to consider. In addition, real problem solving often requires innovative contributions in developing models for decision making.

The system proposed to evaluate fresh food and suppliers is based on at least two multiple criteria methods: MAUT and PROMETHEE, which are adequate and meet the needs for sustainable decision making in fresh food supply chain management. On the one hand, MAUT is easier for managers and decision makers in companies to understand and apply than PROMETHEE. On the other, MAUT is a compensatory method and PROMETHEE is non-compensatory, which is essential in aspects such as food safety, in addition to providing other advantages. For a detailed explanation of both methods, see Belton and Stewart [29]. The seminal work of Keeney and Raiffa [30] contributed the foundations and operational mechanics of MAUT. In addition, Brans and Mareschal [31] provides all versions of the PROMETHEE method, which has been applied to many fields [32,33], but only some articles to supply chain management [20,23,34–36]. AHP is the discrete multiple criteria method most applied on this topic so far, to both elicit weights of criteria and select suppliers, with a fuzzy version also used to deal with uncertainty [4,5,8,37,38]. In contrast, MAUT barely appears in the literature, although it is appropriate for qualifying products and suppliers [3,23].

Figure 1 shows the flowchart of the multiple criteria system for sustainable evaluation of food and suppliers in distribution companies proposed in this research. The first step consists of defining product categories, such as fruits, vegetables or meat. Then, focused on a specific category of homogenous products, the second step is to establish the evaluation criteria for products and suppliers. In food distribution companies, the departments involved are quality, purchasing and logistics, among others, depending on organisational structure. The weights of criteria play a very important role in many MCDM methods, and in particular in MAUT and PROMETHEE. Thus, the weights should take the company objectives and strategy into account. There are several ways to assign the weights of criteria. Determining weights by consensus among managers from the main departments or management board involved is one proposal that increases the real implementation of results in decision making. The AHP method, another multiple criteria approach, provides a robust mechanism to elicit the criteria weight collaboratively [39–41].

**Figure 1.** Flowchart of multiple criteria system for sustainable evaluation of food and suppliers in distribution companies.

After defining the criteria and grouping them in a hierarchy, as well as determining their weights, the second step is to qualify products. Food distribution companies have logistic platforms where products are checked to decide whether to accept or reject them according to conformity to specifications following established protocols. MAUT is the multiple criteria method to qualify and evaluate the quality of products. Therefore, the values of the criteria for each product are required, as well as the utility function for each criterion. Experts provide the information needed to draw up the specific utility functions. The global score of an alternative obtained by MAUT depends on its own performance, while the score from PROMETHEE is based on the pairwise comparisons among alternatives. As the quality of a product only depends on its performance according to the criteria, PROMETHEE is not applicable in this phase, as an objective quality of a product is not related to the quality of other products.

The third step is supplier evaluation, as shown in Figure 1. The scores of products provided by suppliers to assess are aggregated and then, together with other groups of criteria, considered for supplier evaluation. These additional criteria are food safety, cost and logistical criteria, commercial criteria, and environmental criteria. Food safety is a social criterion with an increasingly important role in the food supply chain, which can be measured by using sub criteria. The same applies for other criteria of suppliers, as shown in the case study in Section 4.

Global indicator of suppliers and their ranking are obtained by applying MAUT and PROMETHEE. Both have strengths and weaknesses, as they show complementary perspectives of the evaluation. Both require the weights of criteria; MAUT needs the utility functions that measure the supplier's own performance, while PROMETHEE uses pairwise comparison between suppliers through preference functions. Being easy for managers to understand is one of the main advantages of MAUT. In addition,

it does not present the rank reversal problem, while PROMETHEE does. The compensatory nature of MAUT is its principal disadvantage, whereas the outranking method PROMETHEE is a non-compensatory approach. This is linked with the more discriminant power of the latter, which allows a segmentation mechanism that is very useful in supply chain management. Finally, this system provides indicators of suppliers by product, by food safety, as well as segmentation of suppliers by specific criteria, which is essential for decision making about the best relationship between the company and the supplier, such as for disqualifying suppliers, partnerships, and long-term contracts.

#### **3. Results of the Quality Evaluation of Products**

The multiple criteria approach explained in the previous section was applied to evaluate fresh fruits and suppliers in a supermarket chain with 740 stores located in Spain in order to improve the operations management of the company. This section shows the indicators of the quality of yellow peach provided by suppliers in the whole season from May to October 2018, whose data distribution from controls in the logistic platform is in Table 1. There are 10 varieties of yellow peach from six suppliers.

**Table 1.** Data distribution of yellow peach specifications by suppliers and varieties. Season 2018.

Following a protocol, the company carries out regular inspections and measures the conformance to specification through three characteristics from ten pieces per batch: the quantity of sugars (Brix degrees), firmness (kg/cm2) and size (mm). Due to the large amount of data, Figure 2 only shows some of them, together with minimum and maximum values for these specifications (supplementary material provides complete data). Brix degrees should be 9 as the minimum level to be acceptable, firmness should be between 2 and 5 kg/cm2, and peach size between 65 and 80 mm. Based on the compliance level, yes/no for each characteristic, the company assigns a 1/0 indicator, respectively. The global indicator for the fruit inspected is 1 when the three specifications have a value of 1, and zero otherwise. If more than 20% of the fruits score zero, then the batch is rejected. This procedure does not assess the fruit quality. Therefore, it does not distinguish among peaches with minimum, maximum or optimum firmness, which is a criterion whose performance changes over time until the fruit is consumed. The system proposed in the previous section overcomes this weakness by applying a multicriteria approach to decide whether to accept/reject the fruit batch and gauge the product quality. The latter should provide important data for supplier evaluation.

The evaluation table used to apply MAUT and obtain a robust indicator for peach quality was built with data shown in Figure 2 and the supplementary material. In addition to the evaluation table, MAUT needs the weights of criteria, as shown in Figure 3, which were set by consensus in the quality department.

**Figure 2.** Some data of the evaluation table to apply Multi-Attribute Utility Theory (MAUT) for product assessment by using D-Sight software.

**Figure 3.** Criteria and weights to measure the quality of fruits: yellow peach.

Finally, it also necessary to establish the utility function for each criterion through the information provided by experts from the supermarket chain quality department. They provided direct assessment of partial value function for each criterion in several interviews. Using a global scale from zero to 100, the experts were questioned about assessing the value from the performance of products against each criterion [29]. Figure 4 shows the utility functions of criteria to evaluate the quality of fresh yellow peach. Brix degrees quantify sugars in the fruit and have great influence on its flavor. As shown in utility function (a), the company specifies a minimum value for this criterion, for which a greater value is ideal. When Brix degrees are very high, increases in their value provide fewer quality improvements than when the Brix degrees are lower.

**Figure 4.** Utility functions of criteria for fresh fruit evaluation. (**a**) Brix degrees. (**b**) Firmness. (**c**) Size.

Figure 4b represents the utility function of firmness and (c) the utility function of size. In both cases, specifications set minimum and maximum values. Thus, values outside of this range have a score of zero. Inside this range, there are two sections. In the first one, the function increases until the most preferred value is reached. The utility function decreases more for greater criterion values.

Applying MAUT with all this real information, indicators for the quality of yellow peach are obtained. Figure 5 illustrates the results of the global score and the individual criterion contribution, represented by different colours. Brix degree is the most important criterion, as it has the highest weight (50%), followed by firmness (30%) and size (20%). The usefulness of these results is twofold. Firstly, they are useful at an operational level in the logistic platform, in order to reject those batches with the worst scores. Secondly, they are useful at a tactical/strategic level, when the company is evaluating and monitoring supplier performance.

**Figure 5.** Quality indicators of yellow peach obtained by MAUT. All suppliers, 2018.

Figure 6 presents the same information as Figure 5 for only one supplier (S3). In this case, it is easier to note that in general the worst scores are for products that do not meet the specification for at least one criterion, whose value is below minimum or above maximum. As MAUT compensates for the performance of all criteria, an acceptable score is also feasible when some criteria have a very good performance, but one criterion does not satisfy the technical specifications. In this case, the company indicator would be zero.

**Figure 6.** Quality indicators of yellow peach obtained by MAUT. Supplier 3, 2018.

The Global Visual Analysis is a very useful tool for the analyst, as it provides a different perspective on the results (Figure 7). The points represent products and the criteria are the green axes. Close points represent products with similar quality levels. Products with good performance with regard to a criterion are located in the direction of the axis of this criterion.

**Figure 7.** Global Visual Analysis of yellow peach obtained by MAUT. Supplier 3, 2018.

#### **4. Results of Sustainable Supplier Evaluation: Comparing MAUT and PROMETHEE Approaches**

#### *4.1. Criteria for Sustainable Supplier Evaluation of Fresh Fruits*

The performance of fresh food suppliers in supermarket chains depends on the quality of products provided by suppliers, in addition to other criteria needed to assess them from a sustainable perspective. In particular, the company assesses the fruit suppliers based on three main groups of criteria: logistic, commercial and "product quality". Logistic criteria are stockout, rejections (in logistic platform), and service capability. Commercial criteria include product innovation, conflict resolution, collaboration, and administrative functions. The "product quality" that a company evaluates is a subjective assessment of product quality made by a manager, followed by withdrawal when food safety problems occur. These criteria have some weaknesses; for example, they do not consider objective measures of the quality of products, but only consider the judgements/opinions of managers about the general performance of suppliers with respect to this issue. The manager evaluations for all criteria are measured on a scale from zero to 10. The company wants to improve its procedure to assess the food safety of suppliers and this paper provides new criteria for this purpose. In addition, another shortcoming of the company procedure consists of not including environmental criteria, with the exception of environmental certification or HACCP implementation as a prerequisite.

Sustainability requires taking into account economic, environmental, and social factors. Figure 8 presents a criteria hierarchy for sustainable supplier evaluation in fresh fruits that contributes significantly to both the literature and company management. To select the criteria, the authors have taken a literature review into account [26,42], as well as the results of a survey with managers of supermarket chains and interviews with experts. The survey included questions about the current system for evaluating fresh products and their suppliers and distinguished between criteria already applied and those that would be of interest to apply in the future. The criteria selection also considered the availability of reliable data in practice. The aggregated indicator of the quality of products provided by a supplier is the average of all product indicators from this supplier plus the standard deviation.

**Figure 8.** Hierarchy of criteria for sustainable supplier evaluation of fresh fruits. Weights of criteria are expressed in percentages.

Food safety is the most relevant criterion in food supply chain management. In general, food distribution companies qualify suppliers if they have food safety certifications. Nevertheless, from time to time crises arise due to food safety issues, which are difficult to manage by the companies affected and have great impacts from social and economic standpoints. Thus, the indicator of food safety from suppliers is measured through four sub criteria: food safety certification, withdrawal volume due to food safety issues, number of withdrawals, and withdrawals impact. If a supplier has at least one food safety certification, it receives the best value (100) in the evaluation table. When the supplier has HACCP implemented, the value is 50 and zero when it has neither of these options. HACCP implementation is necessary when the provider does not have any food safety certification. Withdrawal volume is expressed in percentage with respect to total quantity provided by the supplier. Finally, the following scale is used to assess the impact: none (100), low (75), medium (30), and high (0).

The model proposes as environmental criteria: environmental certification, product origin, and organic product. These criteria are qualitative, and are translated into numerical values for the multicriteria evaluation table as follows. If the supplier has environmental certification, the value assigned is 100 and is zero otherwise. The origin criterion distinguishes among local, national, or international sources (100, 75, 50). Organic products have better value than non-organic products, scoring 100 and 75, respectively.

The logistic criteria are the product number, product volume, stockout, rejection, and service capability. The number of varieties of yellow peach provided by a supplier is the number of products in this case study. All these data are quantitative and from databases of the company related to supplier performance in the logistic platform. Stockout and rejection criteria are measured in percentages over the quantity of product. The company monitors the degree of compliance with the day and time in order to measure the service capability of suppliers, which is the average of both percentages.

Finally, the commercial criteria group includes product innovation, conflict resolution, collaboration and administrative functions of suppliers. Managers of the company rate the supplier performance for these criteria using a scale from zero to ten.

Following the flowchart in Figure 1, the second step is to carry out the supplier evaluation taking into account the scores (indicators) of all products provided by each supplier, as well as other relevant supplier criteria for the company. Figure 8 shows all criteria included in sustainable evaluation, as well as their weights, for applying MAUT and PROMETHEE.

#### *4.2. Supplier Performance and Contribution of Criteria*

A company provided the data to complete the evaluation table with the performance of three suppliers, S1, S3, and S5, for all criteria, as well as information to build the utility functions to apply MAUT and the preference functions in PROMETHEE. This company assesses all the supplier performance criteria on a scale from zero to ten. In particular, the evaluation of the subjective quality from company is zero for S1, nine for S3, and eight for S5. The system proposed based on a MAUT methodology gives the following scores for the objective product quality with a scale from zero to 100: 64.16 for S1, 84.71 for S3, and 85.86 for S5.

This section presents the main results on supplier performance and the contribution of criteria to the global score of suppliers obtained by MAUT and PROMETHEE. It is necessary to highlight that the multicriteria approach to evaluate suppliers included the same four commercial criteria as the company used, with the same performance data. Nevertheless, the proposal presented considers ten new criteria: product quality, four criteria related to food safety, three environmental criteria, and two more logistic criteria in addition to stockout, rejection, and service capability. These new criteria allow the assessment of suppliers from a sustainability perspective.

Figure 9 shows the global indicators derived from multicriteria approach obtained by D-Sight software. The score value represents the supplier performance measured in a scale from zero to 100. MAUT has greater values than those obtained from PROMETHEE. The best supplier is the same in both multicriteria methods, S3 with a score of 85.86 from MAUT and 63.65 in PROMETHEE. Nevertheless, the ranking is different, as S1 is the second and S5 is the third in MAUT, while the order is S5 and S1 in PROMETHEE. The company attributes the following scores to these suppliers in a scale from zero to ten with the criteria indicated previously: 8.58 for S3, 5.70 for S1, and 5.30 for S5. Thus, supplier S3 is the best supplier according to company procedure and both multicriteria approaches, mainly because of the good logistic performance and the weight of this criterion (40% in a company system and 30% in

**Figure 9.** Global indicators of suppliers: (**a**) MAUT (**b**) Preference Ranking Organisation Method for Enrichment Evaluation (PROMETHEE).

Figure 9 also shows the relative contribution of each group of criteria to the global score of suppliers that present important differences. This is mainly noticeable in food safety and product quality. The compensatory nature of MAUT, in which bad performance in one criterion can be compensated for by good scores in other criteria, explains these results. In contrast, PROMETHEE, a non-compensatory method, is based on pairwise comparisons of suppliers.

Figure 9 also highlights the greater discriminant power of the PROMETHEE method, especially for food safety and product quality. Figure 10 shows the profiles of suppliers using a different graph, which illustrate how S3 dominates other suppliers in almost all criteria. Figures 9–11 offer results from multicriteria analysis, which are easy to understand for managers and facilitate decision making.

**Figure 10.** Performance of suppliers by a group of criteria. (**a**) MAUT. (**b**) PROMETHEE.

**Figure 11.** Spider web graph of supplier profiles. (**a**) MAUT (**b**) PROMETHEE.

#### *4.3. Capabilities for Supplier Segmentation of MAUT and PROMETHEE*

D-Sight software [43], used to apply MAUT and PROMETHEE in this research, provides an interesting representation of results, named Global Visual Analysis, which corresponds to the GAIA plane developed for the PROMETHEE method by Brans and Mareschal [31]. In Figure 12b green axes represent the main groups of criteria and the red stick indicates the best alternative. The longer the axis, the more discriminant the criterion. S3 is the best supplier and it is especially good for logistics, as its point is located in the direction of the logistic axis. In addition, this representation highlights that S1 has a bad performance in product quality, as it appears in the opposite direction to the criterion axis. For the same reason, S5 shows weaknesses in food safety. This information is shown clearly in the graph from PROMETHEE results, but this is not clear in the graph from MAUT (Figure 12a).

**Figure 12.** Global Visual Analysis of suppliers. (**a**) MAUT (**b**) PROMETHEE.

Global Visual Analysis of suppliers is very useful for analysts, but is more difficult to understand for managers. In practice, bubble graphs such as those in Figure 13 are more useful for individual and collaborative decision making in companies interested in developing the best relationship with suppliers, as recommended by the latest trends in supply chain management.

**Figure 13.** Supplier segmentation by food safety and product quality. Scenario 1. (**a**) MAUT (**b**) PROMETHEE.

Figure 13 represents the suppliers according to food safety and product quality scores. The global score of suppliers is taken into account in the size of the bubble. Comparing results from MAUT and PROMETHEE, the ranking is different, as explained in Section 4.2. This figure highlights the higher discriminant power of PROMETHEE, which allows for segmentation of suppliers based on groups of criteria and is becoming a very important practical tool for decision making in companies. To analyze the sensitivity of the scores to changes in criteria weights, Figure 14 shows the results for another scenario, scenario 2, in which the weight of product quality has been increased to 40%, while the weights of logistics and commercial criteria are 20% and 10%, respectively. These changes do not modify the segmentation of suppliers.

**Figure 14.** Supplier segmentation by food safety and product quality. Scenario 2. (**a**) MAUT. (**b**) PROMETHEE.

#### **5. Discussion**

Product qualifying and supplier evaluation are critical supply chain management issues in food distribution companies, mainly due to food safety incidents that can cause negative effects on consumer health, sales, and brand image, among others. Environmental criteria are also increasingly important for consumers and economic criteria are relevant in order to provide profits for companies and affordable food to society. If a product is good enough for the consumer, this is related to its own performance on criteria selected to qualify the product. Thus, the MAUT method provides an appropriate approach to carry out food product evaluation. In addition, it is easy to use and apply at an operational level in companies, as shown in Section 3.

Evaluating suppliers from a sustainable perspective can be done by both MAUT and PROMETHEE. The main advantages and drawbacks of these approaches derive from the compensatory nature of MAUT and the non-compensatory character of PROMETHEE, based on pairwise comparisons of the supplier performance with respect to all criteria. Section 4 has shown the better suitability of the latter to identify strengths and weaknesses of each supplier. Similar results were obtained in the decision support system related to supplier segmentation validated in a manufacturing company, which makes intermediate products for other companies of food, pharmaceuticals, and chemicals sectors [23]. In this case, the evaluation was focused on chemical and packaging goods, the products and supplier criteria were grouped into critical and strategic criteria, and sustainability was not the main issue studied. Therefore, the criteria considered were quite different to those proposed in this research for fresh fruits in supermarket chains.

The proposal by Lau et al. for fresh food included three multicriteria techniques [26]. Fuzzy AHP to determine the weights of criteria, while we propose AHP or consensus among managers from all departments involved. These authors first applied the non-compensatory ELECTRE method to qualify suppliers from the food safety perspective, followed by TOPSIS to evaluate only the suppliers that verified safety specifications. TOPSIS is a classical multicriteria method with good theoretical properties that provides quantitative indicators based on the distance of the alternatives with respect to the ideal point. Nevertheless, the current trends in supply chain management encourage evaluating and monitoring suppliers systematically, ensuring that they satisfy legal regulations and technical specifications, in order to develop the most appropriate relations with suppliers according to company strategy and preferences [3,23,31].

Regarding the specific measures to evaluate fresh food suppliers, Lau et al. included 10 main criteria: product, quality, food safety, price, delivery, serviceability, commercial position, supplier relationship, risk factor, and Corporate Social Responsibility (CSR). Each main criterion includes between five and nine sub criteria, with more than 60 in total. The qualitative or quantitative nature of many of them is unclear, as is whether the data are sourced from manager evaluations or from quantitative values of databases. The weights of the main criteria vary from 4% for CSR to 16.5% for quality and food safety, which includes certifications, audits, and traceability. Our proposal includes food safety certifications and HACCP qualifications, as well as three new criteria which are the withdrawal volume, number, and impact of withdrawals.

There are some remarkable differences between the approach of Lau et al. [26] and the system proposed in this research. The perspective of our evaluation is global sustainability, while Lau et al. do not consider environmental criteria specifically. These authors only include eco-labeling as a sub criterion in the CSR criterion with very little weight. They also include supplier relationship as a main criterion, whereas this is something to be monitored and reported on from the results of supplier segmentation in our proposal. For example, the supplier is one with a long-term contract, it is advisable to maintain this relationship, or the company should set out new measures if poor performance is found in the period evaluated.

The selection and number of criteria, a clear definition and how to measure their performance, are key issues in the real implementation of a decision support system based on MCDM. In particular, the multicriteria model should be completed. This means that it includes all relevant criteria according

to the company's strategic objectives. At the same time, the selected criteria should have reliable and available data to measure their performance. Many of the case studies on supplier evaluation are mainly based on the manager's assessment. Our proposal is to balance criteria number and model completeness together with data quality and availability. Sustainable supply chain trends in the food sector, such as traceability, facilitate the collection of data needed to implement decision support systems to evaluate and monitor products and suppliers.

The system proposed, based on both MAUT and PROMETHEE, is easy to understand and implement as the core of a decision support system useful for product and supplier assessment in food distribution companies. MAUT is appropriate to qualify fresh food, but has less power to rank and classify suppliers than PROMETHEE. The power of outranking methods, such as PROMETHEE and ELECTRE, comes from their focus on pairwise comparison of alternatives. Nevertheless, ELECTRE is more complex and difficult to understand and apply, as more inputs and parameters are required. On the one hand, MAUT does not have the same rank reversal problems as TOPSIS, which only needs the weights of criteria and is based on an easy algorithm, but it is not appropriate for sustainable supplier evaluation. On the other hand, PROMETHEE has demonstrated the power of graphical tools to carry out more in-depth studies with Global Visual Analysis (GAIA plane), as well as its graphical capabilities to detect strengths and weaknesses of the suppliers' performance and design supplier segmentation to inform strategic decision making.

Finally, it is necessary to highlight that LCA is a common approach in the literature dealing with product sustainability. Nevertheless, LCA is not as appropriate a tool to qualify, evaluate, and monitor food products and their suppliers as the MCDM methods are, as MCDM methods facilitate decision making in companies at the operational, tactical, and strategic level [1].

#### **6. Conclusions**

Qualifying and evaluating suppliers are two common and relevant supply chain management tasks in large companies. There is a huge number of articles on this topic, but they are focused on selecting providers for one product only in manufacturing industries. Nevertheless, companies work in more complex and changing contexts, where selecting a small number of suppliers per product is the trend in supply chain management. In addition, there is a lack of literature on supplier evaluation in food distribution companies, which have an increasing global impact from an economic, environmental, and social point of view.

This article proposes a system based on two multicriteria methods in order to first qualify food and then evaluate its quality, whose results are used in a second step, along with other criteria from providers, focused on sustainability evaluation of fresh food suppliers. The system combines a compensatory multicriteria technique, MAUT, which is easy to use and appropriate to qualify and obtain a quality indicator for products, with PROMETHEE, a non-compensatory method. The models proposed have been validated for fresh food in a supermarket chain, mainly using historical data on fruits rather than opinions, judgements and/or direct ratings from managers. This research represents the first time that both approaches, suitable for supplier evaluation, are applied to this purpose in distribution companies. The system includes economic, environmental, and social criteria to develop sustainability indicators for suppliers, which allow them to improve food safety, one of the most relevant social criteria.

Global and partial indicators from suppliers provide information on the most appropriate decisions and relations between distribution companies and their providers. An example of this is whether to accept or reject a fresh food batch when it is checked at the logistical platform, as well as evaluating the quality of fresh food. Poor performance in food safety can lead to some suppliers being disqualified. As MAUT is easy to apply and interpret, it can be used successfully in real implementations at an operational level in logistic platforms. MAUT is also easy to apply and is useful for monitoring global performance of suppliers in a season, which can be carried out by PROMETHEE. In addition, this latter

method is more suitable for supplier segmentation and provides additional information to analyze their performance in depth, as it discriminates more among supplier strengths and weaknesses.

Finally, in future research it would be interesting to extend this proposal to a large number of fruits and vegetables, including organic products in the evaluation. Likewise, developing appropriate criteria and models to evaluate the food safety and sustainability of fresh meat and its suppliers could contribute significantly to improving not only food distribution supply chain management, but also providing benefits for companies and people. To achieve this end, multicriteria models must be based on transparent and objective data for clearly defined criteria.

**Supplementary Materials:** The following are available online at http://www.mdpi.com/2071-1050/11/21/5875/s1.

**Author Contributions:** All author designed the model and interpreted the data and results of the models. M.S. and C.M. carried out literature review, multicriteria analysis, model validation and wrote the manuscript.

**Funding:** This research was funded by the Regional Ministry of Education, Research, Culture and Sport of the Autonomous Government of the Valencian Region, Spain, grant number AICO/2017/066.

**Acknowledgments:** The authors would like to thank the Regional Ministry of Education, Research, Culture and Sport for funding the research project "Sustainability of the food value chain from production to responsible consumption". The authors thank the company, a supermarket chain, for providing real problems and data and the quality department personnel for helping us to understand the complex nature of fresh food and supplier evaluation. We also thank the editors and reviewers for their suggestions on improving the paper.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, and in the decision to publish the results.

#### **References**

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **Building a Composite Indicator to Measure Environmental Sustainability Using Alternative Weighting Methods**

#### **José A. Gómez-Limón 1, Manuel Arriaza 2 and M. Dolores Guerrero-Baena 1,\***

Received: 30 April 2020; Accepted: 23 May 2020; Published: 27 May 2020

**Abstract:** Environmental sustainability in agriculture can be measured through the construction of composite indicators. However, this is a challenging task because these indexes are heavily dependent on how the individual base indicators are weighted. The main aim of this paper is to contribute to the existing literature regarding the robustness of subjective (based on experts' opinions) weighting methods when constructing a composite indicator for measuring environmental sustainability at the farm level. In particular, the study analyzes two multi-criteria techniques, the analytic hierarchy process and the recently developed best-worst method, as well as the more straightforward point allocation method. These alternative methods have been implemented to empirically assess the environmental performance of irrigated olive farms in Spain. Data for this case study were collected from a panel of 22 experts and a survey of 99 farms. The results obtained suggest that there are no statistically significant differences in the weights of the individual base indicators derived from the three weighting methods considered. Moreover, the ranking of the sampled farms, in terms of their level of environmental sustainability measured through the composite indicators proposed, is not dependent on the use of the different weighting methods. Thus, the results support the robustness of the three weighting methods considered.

**Keywords:** agricultural sustainability; environmental performance; sustainability indices; multi-criteria analysis; analytic hierarchy process; best-worst method; irrigated olive groves; Spain

#### **1. Introduction**

There is a broad consensus about the definition of 'sustainable agriculture' as an activity that satisfies the following requirements for an indefinite period of time [1,2]: a) it protects biodiversity and natural resources and prevents environmental degradation, b) it is economically viable, and c) it is socially acceptable. Taking these requirements into consideration, agricultural sustainability can be defined as a concept that encompasses three main dimensions:

• *Socio-cultural sustainability*. Sustainable agriculture must be socially and culturally beneficial, i.e., it should ensure food security and the fair and equitable distribution of the wealth it generates, as well as contribute to the viability of rural communities.

This paper is focused on the measurement of environmental sustainability. To date, various indicator-based methods have been developed for this purpose, constructed using a wide range of agri-environmental indicators (AEIs) (for a review of the approaches proposed, interested readers can consult Bockstaller et al. [3]). All these proposals are based on large sets of AEIs aimed at assessing the multidimensional environmental impacts of agricultural activity and the provision of ecosystem services (natural resources, biological processes, biodiversity, etc.) (e.g., [4–6]). However, the quantification of environmental sustainability through AEIs has been criticized for several reasons. On the one hand, there are technical problems related to data and measurability (qualitative aspects that are hard to quantify) issues and a lack of sound ecological models that enable the interpretation of the indicators (e.g., a lack of reference levels or thresholds for reversibility processes), the multiple spatial scales needed for an overall assessment or the appropriate time horizon required (extended monitoring is needed for long-term environmental changes) [7,8]. On the other hand, there are also operational concerns related to the interpretation of the whole set of indicators required for such analyses, which is an obstacle to their use as a practical decision-support tool. In order to deal with the latter problem, composite indicators or indexes have been proposed as a means of summarizing the information provided by multiple indicators into an overall assessment of environmental performance (see, for instance, [9,10]).

In order to construct a composite environmental indicator, specific methodological approaches for the normalization, weighting, and aggregation of base AEIs must be selected from several alternatives (see further details in Section 3). All these choices have a significant effect on the overall composite indicator built. The inherent subjectivity of the choice of these approaches is behind most of the criticisms leveled at the different sustainability indexes proposed in the empirical studies carried out to date [11]. This issue has prompted an academic debate on the robustness and sensitivity of the methodological approaches used in the construction of composite indicators [12,13].

The objective of this paper is to contribute to the existing literature regarding the robustness of three alternative weighting methods [14]. For this purpose, we build a composite environmental indicator using three different methods to assess the relative importance of the individual base AEIs; then, by comparing the results obtained from a real-world case study, we provide further insights into the robustness of the weighting methods implemented.

In this context, it is worth mentioning that two of the weighting methods used in this paper are at the core of the multi-criteria decision-making (MCDM) paradigm, since they are used as key tools to provide information about the relative importance of the different criteria considered in these kinds of problems. For this reason, MCDM weighting procedures have commonly been used to build composite indicators [15], with the analytic hierarchy process (AHP) being the most popular one (e.g., [16–18]). Moreover, other more consistent and less time-consuming weighting methods have recently been developed, with the notable among them being the best-worst method (BWM) [19]. As this is a new technique, there have only been a few applications to date, and it has not yet been used in the construction of composite indicators. In this paper, we use the two aforementioned multi-criteria methods, in addition to the more straightforward point allocation (PA) method, to weight the base AEIs that are to be included in an index. We then compare the results obtained from the three methods in an empirical case study. To the best of our knowledge, this is the first study focused on the analysis of the robustness of weighting methods when constructing a composite environmental indicator, although there have been several empirical studies comparing some alternative weighting methods in different types of composite indicators (e.g., [20,21]). The comparative analysis proposed will enable us to draw useful conclusions about the construction of sound composite indicators.

The empirical case study considered is the assessment of the environmental sustainability of olive groves in Spain. For this purpose, the composite environmental indicator built (*ENV*\_*SUST*) relies on previous research focused on the selection of relevant AEIs for this particular agricultural system [22], and uses primary data gathered from Spanish olive farms to calculate the whole set of selected AEIs at the farm level [23]. In any case, it is worth noting that, as far as the authors are aware, this paper is the first to develop a single composite environmental index for an overall assessment of environmental performance in olive growing. Thus, this paper also contributes to the existing literature by providing a sound instrument that is particularly useful for designing targeted policy interventions aimed at promoting sustainable olive farming.

#### **2. Olive Farming in Spain: Environmental Sustainability Assessment**

#### *2.1. Recent Developments of Olive Groves in Spain*

The current area of olive groves in Spain has reached its historically highest level, with around 2.5 million hectares (14% of the country's utilized agricultural area). In fact, Spain is the world's leading olive-producing country, accounting for one-third of the total olive grove area worldwide and half of the total olive oil production.

Spain's accession to the EU in 1986 allowed the Spanish olive sector to benefit from the implementation of the Common Agricultural Policy (CAP), which granted olive growers coupled subsidies that made olive farming more profitable than other types of agriculture (i.e., extensive herbaceous crops). The CAP subsidies encouraged these farmers to increase their olive grove area and olive production. This process of expansion has been also supported by new growing techniques aimed at production intensification, such as irrigation (olive has traditionally been a non-irrigated crop) and higher tree density (increasing from traditional densities of around 100 olive trees per hectare to 300–500 trees per hectare in 'intensive' orchards, or even more than 1000 trees per hectare in 'super-intensive' ones). As a result of these changes, Spain has increased its olive grove area by 25% between 1990 and 2020, and doubled its production of olive oil.

However, this expansion and intensification of Spanish olive groves has given rise to a number of environmental problems [23,24]:

and diuron) have been banned, water quality could be further improved by modifying some of the current olive farming practices [29].

4. *Overexploitation of water resources*. Before the 1990s, most olive trees in Spain were rain-fed, but the intensification of the crop has seen the emergence of 800,000 hectares of irrigated olive groves. Although olive trees have low water requirements and are usually irrigated using highly efficient irrigation systems (water consumption using drip irrigation is around 1500 m3/ha·year), there is substantial pressure on water resources [30]. Increasing water extraction not only causes the overexploitation of water resources but also jeopardizes the ability to meet other water demands in basins with a higher degree of water scarcity [31].

It is worth noting that several policy initiatives have been implemented in recent years to (partially) solve all these relevant environmental problems related to olive groves, by encouraging farmers to adopt various biodiversity-friendly and resource conservation practices. Along with the rational use of agrochemicals and water, these include some compatible soil conservation practices, such as disposing of olive-desuckering debris without burning, which helps mitigate climate change; the shredding of olive-pruning debris for use as soil cover, improving soil texture and reducing the impact of rain and water run-offs; and the use of cover crops under mower control as a sustainable practice in terms of soil protection [26,32,33]. In addition, some practices related to functional elements (hedgerows, riparian vegetation, plots margins, etc.) have proven effective in enhancing biodiversity as well as having a positive effect on other ecosystem services, such as landscape aesthetics [34]. Yet there is still plenty of room for further improvement. To effectively guide the future development of olive groves, more in-depth analyses are required, especially those that provide a quantitative assessment of the environmental performance of individual olive farms.

#### *2.2. Indicators Measuring Environmental Sustainability of Olive Farming*

Although there are several methodological frameworks for the quantitative assessment of the environmental sustainability of agricultural systems, there is widespread scientific agreement that constructing and calculating environmental indicators is the most suitable approach for this purpose [35,36]. Thus, the present study relies on the set of AEIs proposed by Gómez-Limón and Riesgo [22] to evaluate the sustainability of olive farms in Spain.

In order to evaluate the environmental sustainability of agriculture, the approach followed is founded on the SAFE (*Sustainability Assessment of Farming and the Environment*) analytical framework [37]. SAFE is based on a hierarchical structure with three levels: i) principles, ii) criteria and iii) indicators. *Principles* are general conditions for achieving environmental sustainability related to the ecological functions of the agro-ecosystems. In this sense, the environmental sustainability of agricultural systems centers on two principles regarding the protection of (a) biodiversity and (b) natural resources. *Criteria* are the resulting states of agricultural systems when their related principles are respected. For the particular case of olive groves, biodiversity is considered to be protected when the following elements are guaranteed: a1) olive grove genetic diversity (trees in the orchard), a2) biological diversity (a range of different species within the farm boundaries, from 0.1 km2 to 10 km2), and a3) habitat diversity (a range of different habitats within the landscape unit, from 10 km2 to 1000 km2). In addition, natural resources conservation is achieved when: b1) soil erosion is minimized, b2) soil fertility is protected or enhanced, b3) soil and water quality are maintained or improved, b4) water extraction is minimized, and b5) the energy balance (primary energy supply minus primary energy used per cultivation unit) is optimized. Lastly, *indicators* are variables that can be assessed to measure compliance with a criterion, thus producing a representative picture of the environmental sustainability of the agricultural system under analysis. Taking the former criteria into account, a set of 11 AEIs was selected on the basis of analytical soundness, measurability and policy relevance [38,39], as shown in Table 1. Technical details on why each indicator was chosen, how it was calculated at farm level and how its value should be interpreted can be found in Gómez-Limón and Arriaza [23].

**Table 1.** Principles, criteria and indicators of the environmental sustainability of olive groves.

Source: Gómez-Limón and Riesgo [22] and Gómez-Limón and Arriaza [23].

#### **3. Building a Composite Indicator to Measure Environmental Sustainability**

#### *3.1. The Methodological Approach for Building Composite Indicators*

The literature contains a plethora of techniques for building environmental sustainability indices. In any case, the Organization for Economic Co-operation and Development (OECD) and the Joint Research Centre of the European Commission (JRC) [11] provide guidance on the transparent construction of composite indicators, identifying the steps that analysts should follow:

and 1 is assigned to all cases where the AEI value is better than or equal to a 'desired level of sustainability' (i.e., the best environmental performance).

Having made the decisions explained above, the composite indicator measuring the environmental sustainability of olive farms (*ENV*\_*SUST*) can be calculated as a function of the normalized values of the 11 AEIs taken into account (*Ik*), the weights assigned to each of these indicators (*wk*) and the compensation parameter (λ), following the expression:

$$ENV\\_SUST = (1 - \lambda) \cdot \left[ \underset{k}{\text{Min}} (w\_k \cdot I\_k) \right] + \lambda \cdot \sum\_{k=1}^{k=11} w\_k I\_k. \tag{1}$$

The parameter λ ranges between 0 and 1, thus affecting the degree of compensability among the indicators. Here we consider five values of the compensation parameter (λ = 0, λ = 0.25, λ = 0.5, λ = 0.75 and λ = 1), which gives us the three abovementioned possibilities: (a) total compensability (λ = 1), (b) various degrees of partial compensability (0 < λ < 1) and (c) zero compensability (λ = 0).

#### *3.2. Alternative Techniques for Weighting Indicators*

As has been previously stated, this paper is focused on the role of weighting techniques when constructing composite indicators, in order to provide further empirical insights about their robustness. For this reason, in this paper we use three exogeneous weighting methods (i.e., based on expert

opinion) to determine the priorities (global weights) (*wk*) of the whole set of AEIs used to construct the *ENV*\_*SUST* index: AHP, BWM, and PA.

The AHP was initially developed as a decision-support tool for making complex decisions [47], but it was subsequently adapted to index construction; this technique is also particularly useful for weighting sustainability attributes when constructing composite indicators [13,48]. The implementation of this method involves the following steps: first, the weighting problem is structured as a tree-based hierarchy, where the overall goal of the problem (in our paper, the environmental sustainability of irrigated olive grove) is at the top of the hierarchy. Decision criteria contributing to the main goal are placed at an intermediate level (i.e., biodiversity protection and conservation of natural resources in our case) and decision subcriteria are positioned at the lowest level (the base AEIs in our case). Second, experts individually perform pairwise comparisons at each node of the hierarchy, expressing their preferences as to how much one (sub)criterion should be valued over another, following Saaty's fundamental scale (from 1 −equal importance− to 9 −extreme importance of one (sub)criterion over another). Based on these expert judgments, reciprocal square matrices can be built for each node. Third, the local weights of the sets of criteria (biodiversity protection and conservation of natural resources) and subcriteria (the base AEIs) are calculated using the main eigenvector method proposed by Saaty [49]: *AW* = λ*maxW*, where λ*max* is the maximum eigenvalue of *A* and *W* is the vector of local weights. AHP allows some degree of inconsistency in the decision maker's judgments, measured using a consistency ratio that must not exceed predefined values [50]. Fourth, global weights (*wk*) of the base AEIs are calculated by multiplying the local weight of each subcriterion (AEI) by the priority of its parent node (its related principle).

The BWM is a novel multi-criteria decision-making technique [19]. Like AHP, this method is suitable for weighting attributes, and although it has only recently been developed, it has already been applied to the construction of composite indicators [51,52]. BWM requires fewer pairwise comparisons than AHP (in the AHP method, the number of comparisons is *n* (*n* − 1)/2, while for the BWM, the number of comparisons is 2*n* − 3), which may lead to more consistent and reliable results. In order to derive the global weights of the base AEIs, BWM entails the following steps: first, as in AHP, the problem is structured as a tree-based hierarchy (overall goal, decision criteria, and decision subcriteria). Second, the best (sub)criterion (i.e., the most important) and the worst (sub)criterion (i.e., the least important) of the set of (sub)criteria are identified by the expert. Third, the preference for the best (sub)criterion over all the other (sub)criteria is determined using a number between 1 and 9, similar to Saaty's fundamental scale. The expert's preferences are then used to generate the Best-to-Others vector: *AB* = (*aB*1, ... , *aBk*, ... , *aBK*), where *aBk* shows the preference for the best (sub)criterion *B* over (sub)criterion *k*, and *aBB* = 1. Fourth, the preferences of all the (sub)criteria over the worst (sub)criterion are determined using a number between 1 and 9, as in the previous step. This information enables the construction of the Others-to-Worst vector: *AW* = (*a*1*W*, ... , *akW*, ... , *aKW*) *T*, where *akW* shows the preference for the (sub)criterion *k* over the worst (sub)criterion *W*, and *aWW* = 1. Finally, the local weights of decision (sub)criteria and the corresponding indicator of the consistency of responses (ξ*L*) are obtained by solving the following linear programming model:

$$\begin{array}{c} \min \xi^{\mathbb{L}\_{\prime}}\_{\prime} \text{ s.t.}\\ |w\_{B} - a\_{Bk}w\_{k}| \le \xi^{\mathbb{L}\_{\prime}}\_{\prime} \text{ for all } k\\ |w\_{k} - a\_{kW}w\_{W}| \le \xi^{\mathbb{L}\_{\prime}}\_{\prime} \text{ for all } k\\ \sum\_{k} w\_{k} = 1, \ w\_{k} \ge 0, \text{ for all } k \end{array} \tag{2}$$

Using ξ*L*, it is possible to calculate a consistency ratio (CR), which must not be higher than 0.25. As in AHP, global weights (*wk*) of the base AEIs are calculated by multiplying the local weight of each AEI by the priority of its associated principle.

The PA method is a straightforward weighting technique that has proved valuable for determining the priorities of the different attributes of sustainability composite indicators [40,53]. In this method, the expert is asked to directly allocate a fixed number of points (e.g., 10 or 100) among the multiple

criteria (indicators in our case) considered in a decision problem to establish the weight of each criterion [54]. In this paper, we apply this method as follows: first, each expert has to distribute 100 points between the two principles related to the environmental sustainability of irrigated olive grove (biodiversity protection and conservation of natural resources); second, experts allocate another 100 points among the five AEIs linked to biodiversity protection and an additional 100 points among the six AEIs related to the conservation of natural resources. Third, the global weights of the AEIs (*wk*) are obtained (in percentage terms) by multiplying the priority of each AEI by the priority of its related principle.

#### *3.3. Data Collection for the Empirical Assessment of Environmental Sustainability*

The empirical assessment of the environmental sustainability of the irrigated olives groves in Spain relies on two data gathering sources: a survey of farmers to collect the farm-level technical data needed to calculate the AEIs considered (*Ik*), and a survey of experts to obtain the weights assigned to each of these indicators (*wk*).

#### 3.3.1. Farmer Survey

Due to the dispersion of the olive orchards in Spain, we carried out multi-stage cluster sampling to obtain a representative sample of irrigated olive farms. First, following a random selection of agricultural regions proportional to the total area of olive groves, six agricultural regions in Andalusia were selected (Andalusia accounts for more than 80% of total Spanish olive oil production). Figure 1 shows their location on a map. Second, a number of farms proportional to the area of olive groves in the agricultural region were selected through quota sampling, taking into account the farm size. Third, the selection of olive growers to be interviewed was determined using random route sampling. This procedure yielded a final sample consisting of a total of 480 olive farms. Further details about the sampling procedure can be seen in [23].

**Figure 1.** Location of the six agricultural regions included in the survey.

The data collection was carried out via face-to-face interviews with the farmers (lasting around 35 min each) using a structured questionnaire with nine blocks: (1) farm characteristics (location, area of irrigated and rainfed crops, ownership type and farm labor); (2) olive growing characteristics (varieties, plantation age, tree density, type of management: conventional, integrated or organic– and yield); (3) soil and weed management (agricultural practices, weed control—tilled or not tilled and use of cover crops); (4) olive-pruning (date and desuckering debris management); (5) irrigation system and use of water-soluble fertilizers; (6) fertilization (fertilizers and dosage); (7) crop protection

(chemicals, dosage, and management plans); (8) olive harvest (manual or mechanical); and (9) farmer's socio-economic characteristics (gender, age, professional experience, family size, income share from agriculture, education level, membership of producer organizations, and generational renewal). This information allowed us to calculate the corresponding AEIs (*Ik*) at the farm level.

The sample of 480 farms is divided into four types of olive orchards: traditional mountain olive groves (rain-fed), traditional low-medium slope olive groves (rain-fed), semi-intensive irrigated olive groves and other types. In this study, the assessment of environmental sustainability focuses on the semi-intensive irrigated type and is based on a subsample of 99 olive farms. The Table 2 summarizes farms' and farmers' characteristics in this subsample:

**Table 2.** Descriptive statistics from the farmer survey (n = 99).

Source: Own elaboration based on farmer survey.

#### 3.3.2. Expert Survey

In order to weight the contribution of each AEI to the composite indicator measuring the environmental sustainability of olive farms (*ENV*\_*SUST*), a multidisciplinary group of 22 experts was selected following a judgmental sampling method [55]. The expert panel was primarily composed of scientists from universities and research centers (15), but also contained specialists from the Regional Administration (3) and technical services firms (4). Although it is a non-probability sampling technique, the nature of the technical information required and the homogeneity of the group (in terms of their expertise) suggest that the data gathering is reliable and bias-free [56].

The survey was based on one-to-one interviews, and two sessions were conducted with each expert. After an introduction about the objective of the study and the assessment methods, the questionnaires designed for each of those methods (AHP, BWM and PA) were administered. Each interview took approximately 20–30 min. Since the AHP questionnaire was slightly longer than the one for BWM, the short questionnaire conducted for the PA method was included in the BWM session. To avoid order effects (period and carryover effects), a counterbalanced Latin square design was followed [57,58]. Thus, half of the experts were given the AHP questionnaire first, whereas the other half began with the BWM and PA methods (controlling for the period effect). Then, two days later, the experiments were reversed for each group of experts (controlling for the carryover effect).

Finally, it is worth pointing out that from these questionnaires we obtained the individual AEI weights according to each of the three methods used, for each expert in the panel (*wkj*), with the subscript *k* denoting the base indicator and the subscript *j* denoting the expert considered. However, the weights to be included in Equation (1) are the result of the synthesis of the panel's weights. In this regard, we follow Forman and Peniwati [59], who suggest that group decision-making should be performed by aggregating individual weights using the geometric mean for every weighting method (AHP, BWM and PA):

$$w\_k = \sqrt[l]{\prod\_{j=1}^{j-l} w\_{kj}}.\tag{3}$$

#### **4. Results**

#### *4.1. Indicator Weighting*

The values of the consistency ratios for AHP and BWM, although not reported here due to space constraints, do not exceed the permissible threshold levels and, hence, provide evidence of the high degree of reliability of the experts' responses.

Table 3 shows the summary statistics of the global weights obtained by the three methods implemented (AHP, BWM and PA).

Source: Own elaboration based on expert survey.

There is consensus about the most important AEI for the protection of biodiversity; namely, the index of biological diversity (*DIVERSIND*). This is followed by the pesticide risk (*PESTRISK*) for AHP, and by the percentage of non-cultivated land (*NONCULTIV*) for BWM and PA. Regarding the second principle, the conservation of natural resources, there is also consensus about the most important indicator, soil erosion (*EROSION*), followed by soil organic matter (*ORGMAT*). It is worth noting that the lowest variability of AEI weights is found with the PA method.

#### 4.1.1. Inter-rater Reliability

Before undertaking the comparison of the weighting methods, we assess the degree of agreement between the experts' AEI weights, using the intraclass correlation coefficient (ICC). Unlike the traditional correlation coefficient, based on paired observations, the ICC simultaneously considers the group agreement. Higher ICC values indicate higher inter-rater reliability, with 1 indicating perfect agreement and 0 only indicating random agreement [60]. There are 10 different forms of ICC depending on: (1) the statistical model; one-way or two-way models, according to whether the source of variation comes from objects or subjects (raters), respectively; (2) whether raters are considered as random or fixed effects (two-way random-effects model or two-way mixed-effects model, respectively); and (3) the type of agreement: absolute agreement (for the same object, similar scores among raters) or consistency (for the same object, similar ranking among raters) [61,62].

Since the experts are not randomly selected and we are interested in assessing whether or not the AEI weights are equal within each weighting method, we estimate the absolute agreement among experts using a two-way mixed-effects model. The resulting ICCs for AHP, BWM and PA are presented in Table 4.

**Table 4.** Intraclass correlation coefficients and 95% confidence intervals for the three methods.

Source: Own elaboration based on expert survey.

To interpret ICC, Cicchetti [63] gives some guidelines: <0.40, 0.40–0.59, 0.60–0.74, and >0.74 for poor, fair, good and excellent reliability, respectively. Additionally, Koo and Li [64] give slightly different intervals: <0.5, 0.5–0.75, 0.75–0.9 and >0.9 for poor, moderate, good and excellent reliability, respectively. In our case, the degree of agreement can be regarded as good to excellent in all three methods. Thus, despite the sample size limitation, all three methods produce consistent assessments of AEI weights. Notwithstanding these outcomes, as commented above, PA produces the least variability in experts' assessments in terms of the coefficient of variation (see Table 3).

#### 4.1.2. Multivariate Comparison of Weights from the Three Methods

In order to compare the AEI weights from the three methods, a within-subjects multivariate analysis of variance (MANOVA) design was implemented. The MANOVA not only reduces the chance of Type I error but can also account for the correlation among the dependent variables [65], and therefore has more power to detect differences among groups [66,67]. The experimental design met the assumptions relating to the measurement of the dependent variables at interval scale, the independence of observations, and adequate sample size (more observations than the number of dependent variables; in this case, 22 observations vs. 11 variables).

Regarding the additional assumptions of the MANOVA, we conclude: (1) the visual check of scatterplots suggests that the condition of linearity (no curvilinear pattern between all pairs of indicators) is met; (2) the conditions of homogeneity of variance-covariance matrices (Box'M statistic = 124.0, *p*-value = 0.686) and homogeneity of variances (minimum Levene statistic = 0.895, *p*-value = 0.414) are fulfilled; (3) there is no multicollinearity among the AEIs (maximum r = 0.580 < 0.90 [67]); (4) none of the three methods satisfy the multivariate normality assumption, however, since we have equal group sizes, the MANOVA is robust given the absence of multivariate outliers [68,69]; and (5) no multivariate outliers were identified using Mahalanobis distance (minimum probability equals 0.066). According to this evidence, the MANOVA can be applied to check for significant differences of means among weighting methods. The results of the MANOVA are shown in Table 5.

As Table 5 shows, three out of four multivariate criteria test statistics suggest there is no statistically significant difference in means. Furthermore, as Kuhfeld [70] points out, in the event of a discrepancy between Roy's Largest Root and the other three test statistics, the effect should be considered to not

be significant. In summary, the results suggest that the AEI weights do not depend on the weighting method (AHP, BWM, and PA).

**Table 5.** Multivariate comparison of AEI weights by weighting method.

Source: Own elaboration based on expert survey.

It is subject to discussion whether the consistency of the three methods depends on the use of the same group of experts. Although further research would be needed in this regard, a random subsampling of the 22 experts was carried out (assigning 8 to AHP, 7 to BWM, and 7 to PA) to compare the AEI weights, yielding the same conclusion based on the MANOVA test (*p*-values: 0.190, 0.256, 0.338 and 0.158, respectively).

#### *4.2. Assessing the Environmental Performance of Irrigated Olive Farms*

Table 6 shows the main descriptive statistics of the 15 distributions of the composite indicator *ENV\_SUST* (3 weighting methods × 5 values of lambda) obtained for the 99 olive farms sampled. As can be clearly observed, the index values calculated vary more due to the compensation parameter λ than due to the weighting method. In fact, while there is no statistically significant difference among the means and the variances of the *ENV\_SUST* distributions obtained for every single λ, it can be proved that the average values significantly decrease as the compensation parameter decreases.

**Table 6.** Descriptive statistics of the composite indicator (*ENV\_SUST*) by compensation parameter (λ) and weighting method.

Source: Own elaboration based on expert and farmer surveys.

Although the discussion about the most suitable value of the compensation parameter to measure farms' environmental performance is beyond the scope of this paper, it is worth pointing out that composite indicators based on complete compensability (i.e., λ = 1) have been criticized because trade-offs between base indicators could be considered incompatible with the concept of sustainability [13,40]. It is thus reasonable to opt for indexes that allow partial compensability (i.e., *ENV\_SUST* for 0 < λ < 1). In any case, the selection of the most suitable value of λ is an issue that remains open for discussion in future studies [18].

The assessment of the rankings of farms produced by the composite indicator *ENV*\_*SUST* using the three weighting methods, for each of the five values considered for the compensation parameter (λ = 0, λ = 0.25, λ = 0.5, λ = 0.75 and λ = 1), is carried out using the Kendall's coefficient of concordance, or Kendall's W [71], mathematically:

$$\mathcal{W} = \frac{12S^2}{m^2(k^3 - k)} - \frac{3(k+1)}{k-1},\tag{4}$$

where *m* = 3 (AHP, BWM, and PA), *k* = 99 (number of farms), *rij* = ranking of farm *i* by method *j*, *Ri* = *m j*=1 *rij* and *S*2 = *k i*=1 *R*2 *i* . To test the null hypothesis of no agreement among the methods, that is *W* = 0, the statistic to be used is *m*(*k* − 1)*W* ~ χ2 *k*−1 .

As the results in Table 7 show, the ranking of the irrigated olive farms based on their environmental performance (i.e., values of composite indicator *ENV*\_*SUST*), for any lambda considered, does not depend on the weighting method used (AHP, BWM, or PA). Furthermore, when considering the 15 rankings simultaneously, the overall Kendall's *W* indicates a strong level of concordance (Kendall *W's* = 0.705). This indicates that, regardless of the weighting method or the compensation parameter, all *ENV*\_*SUST* measurements provide similar rankings of the sampled irrigated olive farms.

**Table 7.** Kendall's coefficient of concordance of farms' sustainability ranking with the three weighting methods.

Source: Own elaboration based on experts' and farmers' survey.

#### **5. Conclusions**

Measuring environmental sustainability in agriculture through the construction of composite indicators is a widespread practice, although it is a tough task. An especially challenging aspect is the choice of the most appropriate methods to normalize, weight and aggregate the large set of base agri-environmental indicators usually considered. In particular, the results of environmental composite indicators are heavily dependent on how the base indicators are weighted (i.e., if the indicator weights can accurately synthetize the relative importance of each AEI included in the index built). For this reason, our main aim in this paper was to analyze the robustness of three alternative weighting methods (AHP, BWM and PA). To that end, we consulted the opinions of a panel of experts and compared the results obtained for an environmental index implemented in the real-world case study of irrigated olive farms in Spain.

In light of the results, we can identify three main findings: first, there is a high level of consistency in experts' assessments of AEI weights derived from the three weighting methods; second, there are no statistically significant differences in the means of the AEI weights estimated with the three methods; and third, the values of the composite indicator built (*ENV*\_*SUST*) using the three alternative weighting techniques produce similar rankings of the irrigated olive farms in terms of their environmental performance. Further evidence regarding the consistency of the weighting methods is needed to confirm whether this finding is generalizable. In any case, it can be hypothesized that similarly consistent results can be expected whenever the composite indicator construction (i.e., weighting) is focused on the assessment of technical concepts, where expert opinion is rooted in empirical knowledge.

Overall, these findings provide useful empirical insights into the robustness of the two multicriteria methods, AHP and BWM, and the more straightforward PA, as weighting techniques to be used when constructing composite environmental indicators. However, although the three methods are valid, feasible tools to determine the weights of the individual base indicators, it is worth noting that the PA could be cumbersome if there is a large number of indicators to be included in a single index (e.g., more than six) [44].

Beyond the methodological focus, it is also worth pointing out that the *ENV*\_*SUST* composite indicator implemented in this study is sufficiently stable and methodologically sound to be used in the design of targeted policy interventions aimed at measuring the environmental sustainability of farms. In this context, there is still room for research on the practical implementation of this environmental index to analyze the heterogeneous environmental performance among farms (i.e., determining reference and threshold values) and track changes in agricultural practices (i.e., irrigation or fertilization). This practical information could be useful for policy-makers in the design of the results-based agri-environmental programs, to set the level of payments to be granted to each particular farm or any other policy instruments with a similar purpose (i.e., fiscal or qualitative rewards).

**Author Contributions:** Conceptualization, J.A.G.-L.; methodology, J.A.G.-L. and M.A.; formal analysis, M.A., J.A.G.-L., and M.D.G.-B.; writing—original draft preparation, J.A.G.-L., M.A., and M.D.G.-B.; writing—review and editing, J.A.G.-L., M.A., and M.D.G.-B.; funding acquisition, J.A.G.-L. and M.D.G.-B. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was partially funded by the Andalusian Department of Economy and Knowledge and the European Regional Development Fund (ERDF) through the research project FINAGUA (UCO-1264548). These funding institutions had no involvement in the conduction of the research nor the preparation of the paper.

**Acknowledgments:** The authors are gratefully to anonymous reviewers for their comments and suggestions, which have substantially improved the quality of the paper. We are also grateful for the collaboration of the 22 academics and professionals who took part in the panel of experts weighting the AEIs (Francisco Bruno, Juan Cano, Juan Castro, Sergio Colombo, Juan Domínguez, Manuel Estévez, José A. Gómez, Emilio González, Rubén Granado, Carlos Gutiérrez, Javier Hidalgo, Lorenzo León, Salvador Nadal, Ángel Navas, Rafaela M. Ordóñez, Luis Padilla, José M. Penco, Macario Rodríguez, Raúl de la Rosa and Anastasio J. Villanueva; the two remaining experts prefer to remain anonymous). The authors also thank the participation of the 480 farmers sampled for providing actual information about the agricultural management of their olive groves.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **Sustainability in Forest Management Revisited Using Multi-Criteria Decision-Making Techniques**

**Marta Ezquerro 1,\*, Marta Pardos 2 and Luis Diaz-Balteiro 1**

Received: 30 May 2019; Accepted: 28 June 2019; Published: 2 July 2019

**Abstract:** Since its origins, the idea of sustainability has always been linked to forest management. However, nowadays, sustainable forest management has usually been approached by defining a set of criteria and indicators. This paper aims to address sustainability in forest management including a set of criteria encompassing the most common decisions: whether the stands are even or uneven-aged, and the optimal silviculture that should be applied in each stand. For this purpose, a lexicographic goal programming model with two priority levels has been defined, into which six different criteria are integrated. Each criterion corresponds to a particular pillar (economic, technical, or environmental). Furthermore, also incorporated into the model are the preferences of diverse stakeholders, both for the criteria considered in the analysis and for the most suitable silvicultural alternatives to be applied in each stand. This methodology has been applied to a case study in Spain, and the results show much more attractive solutions than the current forest management planning, allowing the obtainment of multi-aged systems that could be favourable for other ecosystem services.

**Keywords:** goal programming; interactive methods; forest planning; Green-Tree Retention

#### **1. Introduction**

The concept of sustainability in a forestry perspective has a development history that dates over 300 years back in time [1], without the author himself having supplied any definition of this term [2]. In some respects, this idea at that time was the first conservation principle ever formulated [3]. In recent decades, it has spread in an exceptional manner to other areas [4]. To some authors, the sustainability concept is currently belonging to the field of social ethics [5]. Nowadays, it continues to be a forceful message with a wide representation in diverse disciplines [6]. In synthesis, von Carlowitz's premise promoted the concept of not exhausting resources in order to bequeath them to future generations, safeguarding finite natural resources for future generations [7].

In order to integrate sustainability into forest management, it has been customary to define a multidisciplinary ensemble of criteria and indicators in order to arrive at a consensus as to what sustainable forest management should be like [8], although sometimes it is not easy to transfer this idea to a strategic level [9]. It has been successfully applied on different spatial scales, using more or less aggregated information. Thus, there are indicators for sustainable forest management focused at a national or supranational level [10,11], or at a management unit level [12,13]. Further, in order to include this concept of sustainability in forest management, tools like certification systems aimed at ensuring sustainable management in forests have been developed. Thus, the systems most used nowadays simply require compliance with a series of indicators [14]. However, although it is commonly believed that a certification scheme implies a sustainability attribute, this direct relationship is not always clear [15].

On the basis of the literature consulted, it has been verified that the definition of sustainable forest management was usually of a static nature. That is to say, the studies in question contained only one measurement of the indicators, without taking into account their temporal evolution, in spite of this component being included in the definition of the word sustainability [16]. However, only in some studies has it been proposed to measure a set of criteria over the entire planning horizon covering forest management in a certain case study [17–19]. Given the extensive length of rotations in many forest systems, it is necessary to have a set of indicators available whose values are known in each period or, at least, are quantifiable at the end of the planning horizon.

Given the multidimensional nature intrinsic to the concept of sustainability, and as has been mentioned previously, the need to apply multifunctional management requires the use of Multi-Criteria Decision-Making (MCDM) techniques. These methodologies have been widely employed to solve typical forest management problems [20–22]. Moreover, multicriteria methods have been applied prolifically to tackle aspects related to sustainability [23–26], even dealing with aggregation problems and dynamic sustainability indicators [27]. These methodologies are also recommendable for integrating different ecosystem services into the decision-making process [28–30]. Finally, it is fitting to insist that these methods permit the solution of the emergent problem of how to aggregate the indicators on which the idea of sustainability is based [31].

Bearing in mind that sustainability is difficult to define in precise terms [32,33], the main objective of this study is to present a flexible model, based on multicriteria techniques. These kinds of models permit the redefinition of sustainability in a forest system, not at one specific moment, but through the evolution of the indicators considered throughout the planning horizon. This new view of sustainability should address the multifunctionality of forest systems, which implies defining a diverse set of criteria and indicators, choosing the best silvicultural alternative for each stand, and enabling the stakeholders involved to have their opinion integrated into it. It should be emphasized that some authors have affirmed that the rigidity of the concept of a normal forest does not ensure the idea of sustainability in force today [34], which would justify using the approach proposed here in order to adapt this notion to the present-day context. Finally, in the following Sections, we will introduce terms like "criterion", "objective", "indicator" and "index". In order to avoid misinterpretations, and following [35], under an MCDM umbrella, criteria are the objectives or goals to be considered relevant for a certain decision-making situation. However, if applicable, we have considered a hierarchy between criteria and indicators. Indicators are parameters or sub-criteria which can be measured, and which correspond to a particular criterion. Besides, some sustainability measurements can be defined as a synthetic, aggregate, or composite index, and the value achieved by this index is a proxy of the respective sustainability goodness.

#### **2. Theoretical Background**

#### *2.1. Framework*

The stages of the model proposed are condensed in Figure 1A. The first step involves defining both the sustainability indicators selected for the planning horizon chosen, and the different silvicultural alternatives put forward. Applying both the indicators and the silvicultural alternatives to the case study, precise information is made available for the formulation of a strategic forest planning model. If each criterion considered is optimized separately, the pay-off matrix can be set up and the degree of conflict between the different criteria verified [35]. If no solution is supplied by the pay-off matrix, coming from the optimization of a single criterion is acceptable. The following step is to construct an MCDM model, integrating the opinions of the stakeholders (DMs) on some aspects. In that way, compromise solutions that could be accepted by the decision-maker will be obtained.

As previously mentioned, the methodology to be applied is based on MCDM techniques, because the latter are highly appropriate when facing this type of problem. However, although, under the MCDM umbrella, a large set of techniques co-exist [36], due to the nature of the problem, only those that can solve continuous-type problems will be apt. This circumstance implies that those techniques that aim to choose the best solution out of a finite set of alternatives would not be valid for this case. From the MCDM techniques that fulfil the above condition, goal programming has been selected. This methodology adds flexibility to forest management models [37], and it has been used quite frequently to tackle this type of problem [31]. The basics of goal programming can be seen in [38–40].

Among the variants of this technique typically employed [39], we chose Lexicographic Goal Programming (LGP), also known as Non-Archimedean or pre-emptive goal programming [41]. The selection of this technique is justified because it allows the decision-making centre to show an order of preference for each of the goals defined. In addition, the idea of defining sustainability by means of a conceptual hierarchy [42] fits the LGP notion perfectly. This hierarchy manifests itself in the definition of different levels of priority to which each goal is attached. This means that the preferences are initially assigned to each goal following an established order, and that the achievement of a goal situated at a certain level of priority is infinitely preferred to the fulfilment of one located at a lower priority level. In short, this type of modelling implies accepting that there are no finite trade-offs among the goals placed at one priority level with respect to goals placed at lower ones [35]. Consequently, the results obtained for the goals situated at lower priority levels cannot worsen those of the goals at higher priority levels [43]. The mathematical formulation can be seen in Appendix B**.** Once the goals are defined [44], the variant of the weighted goal programmed is selected to solve the problem at each priority level [45], this being the most common option when electing this goal programming variant [46].

With the methodology shown in Appendix B, a model is obtained which permits the evaluation of the different criteria or indicators considered for defining sustainability over the defined time period, in the context of strategic forest planning [9]. The preferences introduced into the model will only take into account the opinion of the stakeholders and exclude intergenerational justice components from the analysis [47]. In addition, only criteria for which values can be obtained throughout the entire planning horizon are considered [31]. That is to say, if the evolution over time of a certain criterion cannot be modelled, it is not included in the analysis. Further, it is assumed that the silvicultural alternatives that can be applied in each of the stands are known with certainty. The model proposed is totally deterministic, and the introduction of elements of risk and/or uncertainty is not considered [48–50]. Finally, although, in previous work, diverse climate change scenarios have been analyzed in the same case study [51,52], the latter will not be included in this work, although the results for some ecosystem services like carbon sequestration could be altered according to the different scenarios contemplated [53].

The next step is to design a strategic forest planning model (see Appendix B). In order to assess the degree of conflict between the criteria considered, a pay-off matrix was developed starting from the results of the strategic forest planning model. This is a usual step in the application of multicriteria techniques in forest management problems [54,55]. The pay-off matrix is a square matrix, whose dimension coincides with the number of objectives considered. This matrix is obtained by optimizing the six objectives separately and computing the values of the criteria for each of the corresponding optimal solutions, thus having a dimension equal to the number of objectives that exist [35]. The main diagonal includes the optimum values that each objective can reach, known as the ideal point. By definition, this vector in the objective space is infeasible, but plays a decisive role as a point of reference [56]. Other cells of the matrix contain the anti-ideal (nadir) points, which would be the worst results obtained for each of the objectives. Concerning the criteria considered, it is appropriate to clarify that *VOL*, *NPV*, *IM* and *C* are criteria belonging to the type "the more, the better", while *NF* and *G* are criteria belonging to the type "the less, the better", following the term used in [35].

#### *2.2. Theory Review*

The primordial concept of sustainability has marked the development of forest science both in Europe and in the U.S.A. [57], and it has become a leading component of forest management over the last 250 years [58]. Similarly, it has been considered to be an integral part of forest management [33] and has been accepted almost unanimously by foresters [59]. Generally, in the past, sustainability was based on a few basic principles in accordance with the type of forest management practiced at the time, which was centered on the "normal forest" or "normality" concept [60], established at the end of the 18th century by German foresters [61]. In fact, it was characterized by being a mono-objective management, focused only on timber production, usually in regular stands, in which one or a few main species prevailed [62], and to which a silviculture with a fixed rotation was applied [63]. In addition, a single decision-maker made the corresponding decisions with the aim of perpetuating the idea of a fully-regulated forest [57,64]. Finally, as a result, and unlike what happens nowadays [65], restrictions to final harvest were very lenient.

The concept of sustainability has been modified and updated due, among other reasons, to the changes in the role played by forests in society [66,67]. Besides that, the demand for goods and services coming from forest systems has been continuously increasing [68], meaning that, today, forest management is much more polyhedral (or multi-faceted, according to [69]) than it was three centuries ago, and includes other ecosystem services not contemplated in the original idea of sustainability, such as in the case of biodiversity [70]. Indeed, some authors have recently gone beyond the theory of von Carlowitz and defined forests as a "multifunctional bioeconomic system" [71]. Thus, numerous authors maintain that the new paradigms associated with sustainability should promote multifunctionality in forest management [72]. At the same time, the idea of sustainability in forest management has been linked to that of resilience [73]. In this regard, although the expression "sustainable" has for many years been on a par with forest management [74], nowadays it has been assumed that the latter should achieve certain goals with respect to economic, social and environmental pillars and, also, balancing present and future demands [75]. Some authors even propose the concept of "ecologically sustainable forest management" to give prominence to a type of forest management that perpetuates ecosystem integrity [76].

The way forest management has evolved has required the reformulation of the original sustainability concept [1] into the idea of a "regulated forest" (the organization of a forest property to provide a sustained yield of timber products forms [57]), by integrating the abovementioned pillars, but taking into account that, on some occasions, the trade-offs between different criteria are not fully known [77]. That is why sustainability should not be defined by only focusing on the conception of its being concentrated on "sustained yield", one of the objectives most sought after in forest management [78], or on other similar concepts [47]. Namely, this idea should be generalized starting from a case in which not all the forest should be associated with the idea of even-aged stands into which different silvicultural alternatives are established [63]; other ecosystem services are integrated, as well as timber production; and the preferences of the different stakeholders in the decision-making process are included. In effect, a multi-aged structure in forest stands is associated with the idea of long-term sustainability [79], incorporating multiple uses into different stands even with variable rotations [32]. Lastly, the inclusion of different silvicultural alternatives in the analysis is basically justified for two reasons. First, because different forest management strategies promote biodiversity [80], and, second, because silviculture as a science has to respond to the demands of society and not only concentrate on timber production [81].

#### **3. Materials and Methods**

#### *3.1. Case Study*

The case study corresponds to Valsaín forest, which is in the Central Mountain Range of Spain. The forest area covers about 7206 ha and mainly composed of pure Scots pine (*Pinus sylvestris* L.) stands, although there are other species less represented in the area such as *Quercus pyrenaica* L. and *Ilex aquifolium* L. The first management plan was developed in 1889, and the main objective was timber production. Since 1940, in order to achieve this objective, a uniform shelterwood system with a rotation of 120 years and a 20 years regeneration period has been proposed [52]. However, in recent years, this management has become more multifunctional [25], and the silvicultural treatment was turned

into a shelterwood group system [52]. Nowadays, other aspects such as biodiversity conservation are of great importance. Thus, there are several endangered species in the area, like the black vulture (*Aegypius monachus* (Linnaeus, 1766)). The number of nests of this species is around 131, and harvest limitations have been introduced in order to protect them. In short, around each nest, a restricted area without management of 3.14 ha has been established as a buffer protection. Finally, there are several protection features in this forest, such as the declaration of the Sierra de Guadarrama National Park in 2013, where commercial cuttings have been forbidden, and which affects 3326 ha.

#### *3.2. Silvicultural Alternatives*

In order to deal with this complexity, seven silvicultural strategies have been proposed to maintain the adaptability of forests to different conditions [81], and to provide different ecosystem services. The set of possible treatments considers the nature of the stands forming the case study, and all of them have been discussed and approved by the forest manager. A priori, the seven treatments (see Figure 1B and Appendix A, Table A1) could be applied to every stand.

**Figure 1.** Framework for integrating sustainability into forest planning (**1A**). Sustainability criteria and silvicultural alternatives proposed (**1B**).

The first alternative is based on a shelterwood group system. The treatment spans 40 years, cutting every ten years, and it would be the "business as usual" alternative (BAU). After the final cut, 2% of the stand volume remains as dispersed trees for at least one rotation. In order to model this alternative, we have considered the schedule and cutting intensity traditionally used in Valsaín forest [82]. BAU has been divided into two new alternatives: BAU1, embracing light thinning, and a more intensive option with strong thinning (BAU2). The third alternative, called Group selection (GS), consists of 2 cuttings with medium patch sizes of 1 hectare. After final cutting, 5% of the stand volume remains as residual trees.

Another two silvicultural alternatives are based on the idea of Green Tree Retention (GTR). Two GTR alternatives with different retention percentages are proposed, before applying BAU1 or BAU2. In short, either 15% of the volume is maintained in an aggregated way as mature forest patches (GTR1), or 30% of it (GTR2). These figures are similar to those proposed for this species in Spain [83]. These patches could simulate the protection buffers discussed above for the protection of the black vulture species.

For the area now included in the National Park, we have proposed an alternative (TNP) similar to BAU, but with a main difference: the final cut is excluded, due to prohibitions in all Spanish National Parks. Moreover, in this case, the rotation has been increased (up to 140 years) in relation to the other treatments, as a conservation measure [84]. Finally, a no management alternative (NOM) has been incorporated in order to facilitate the possibility of creating mature forest stands in the future, given that these stands are allowed to grow naturally, thus forming reserve areas.

#### *3.3. Criteria Considered*

As mentioned above, the criteria to be introduced should evaluate sustainability based on the multifunctionality intrinsic to forest systems. First, two technical criteria have been defined. The first of these corresponds to a synthetic index that encompasses the classic idea of a normal forest (NF). In short, this criterion embraces tree indicators: one associated with the notion of even-flow policy, another with the idea of regulation or regulated forest [57], and, finally, a third one related to a suitable ending forest inventory, which ensures the perpetuation of the forest. In order to aggregate the above indicators under the normal forest concept, the procedure used in [85] was followed. In addition, a criterion called Ideal Management (*IM*) has been introduced. It can be quantified by the amount of area (in hectares) to which the silviculture suitable for each stand, according to the criterion optimized in each case, is applied. Therefore, calculating this criterion consists of minimizing the deviations of the silviculture used, compared to that which would be preferred in terms of the characteristics of the stands and in accordance with the preferences reflected by a decision-maker.

On another side, two environmental criteria have been defined. The first one refers to the possibility that, at the end of the planning horizon, there is a multi-aged forest. To enable a comparison of the degree of irregularity between different prescriptions, the Gini index (*G*) was employed. The latter is frequently used to measure the degree of inequality in different variables, generally with economic data, although applications in many fields have been described [86]. It supplies values in the interval [0 1] and has been defined in such a way that values close to 0 imply the existence of multi-aged stands, whereas values close to 1 ensure an even-aged structure. The second environmental criterion considered was the carbon balance (*C*), defined as the difference between the estimated gains and losses of carbon due to the growth of trees and their removal (default method, following [87]) throughout the planning horizon. This criterion will be computed in physical units (tons of carbon).

With respect to the production criteria, the Net Present Value (*NPV*) and the volume (*VOL*) associated with the final cuttings (cubic meters) over the planning horizon have been considered. Regarding *NPV* (€), the incomes provided referred only to timber sales. We have considered a fixed maintenance cost (11.6 €/ha), and different harvesting costs in relation to the silvicultural treatment applied. Following data provided from the current forest management plan, the timber price considered was 28 €/m3. We have considered a real discount rate of 2%. Concerning the criteria considered, it

is appropriate to clarify that *VOL*, *NPV*, *IM* and *C* are criteria belonging to the type "the more, the better", while *NF* and *G* are criteria belonging to the type "the less, the better", following the terms used in [35]. Figure 1B shows the six criteria considered with their acronyms, meanings, and the units in which they are expressed.

#### *3.4. Multi-Criteria Model*

When analyzing the solutions shown in the Pay-off matrix, if no solution is preferable, it is necessary to look for a satisficing compromise solution, making the harvest scheduling more sustainable. As we have said before, the technique chosen was Lexicographic Goal Programming (LGP), the second most used GP variant in forest management applications [88].

The first issue to decide in this type of model is how to select the number of priority levels that one wants to establish. Starting from the premise that it should be a limited number and that there should not be as many priorities as the number of criteria [89], in this case, two priority levels have been fixed. In the first one, three goals most directly related to silvicultural aspects have been included, while in the second, *NF*, *IM* and *G* are considered. Namely, given that there are no finite trade-offs among goals placed at the two priority levels [88], it is assumed to be a priority to achieve the goals at the first level before optimizing the criteria situated at the second priority level [90]. The target for each goal was established at 70% with respect to the ideal value obtained in the pay-off matrix. This percentage has been employed in the literature [91], and its use avoids the drawback of not obtaining optimal alternatives if the targets are closer to ideal values [92]. Finally, the complete model can be found in Appendix B, and for its resolution the software LINGO 13.0 [93] has been used.

#### *3.5. Interactive Process*

In this study, we interacted with two decision-makers (DM1 and DM2). First, as mentioned above (Equations (A2) and (A3)), it was considered appropriate to introduce preferential weights for each of the six goals contemplated in the analysis. For this purpose, the interaction was with the forest manager (DM1), and his opinion was asked on the six criteria using a survey based on pairwise comparisons, with Saaty's verbal scale [94] (see Appendix A, Table A3). This way of obtaining the decision-maker's preferences is recommended for integrating them into GP models [95]. These pairwise comparisons are typically used in the forest context [96].

On another side, and with the aim of obtaining the IM criterion, the engineer in charge of carrying out the last forest management planning (DM2) was contacted. Taking into account the silvicultural alternatives considered (see Appendix A, Table A1) the objective was to establish the suitability of each one for being applied to each stand. For this purpose, the 288 stands in the forest have been grouped into seven categories generated as a combination of a series of factors: production, protection, National Park, districts with nests or the presence of mixed stands. Once the categories were defined, in a first contact with DM2, we selected a set of 2–3 preferred alternatives for each stand. Later, after a second interaction with the same expert, some weights were obtained to weight the different silvicultural alternatives for their suitability, attending to the characteristics and nature of the stand in which they were found (see Appendix A, Table A2). To obtain the weights assigned to each silvicultural alternative in each stand, the same procedure as that employed with DM1 was used (see Appendix A, Table A3).

#### **4. Results**

Beginning with the pay-off matrix, Table 1 shows the results of the optimization of the six criteria separately. In Table 1, ideal values (principal diagonal of the matrix) are shown in bold type, while anti-ideal (or least desired ones) are underlined. The main diagonal includes the maximum values that each criterion can reach, known as the ideal point. Other parts of the matrix contain the anti-ideal points, which would be the worst results obtained for each of the criteria. Finally, to increase the informative character of this matrix, several additional rows were included by measuring the area occupied when each criterion is optimized by each of the silvicultural treatments described, as well as the average rotation.

**Table 1.** Pay-off matrix (ideal values in bold; anti-ideal values underlined).

*NF*: Normal Forest structure, *IM*: Ideal management (ha), *G*: Gini index, *C*: Carbon balance (103 tC), *VOL*: Harvest volume (103 m3), *NPV*: Net Present Value (103 €). area-BAU1: area under BAU1 treatment (ha), area-BAU2: Area under BAU2 treatment (ha), area-GS: Area under GS treatment (ha), area-GTR1: Area under GTR1 treatment (ha), area-GTR2: Area under GTR2 treatment (ha), area-TNP: Area under TNP treatment (ha), area-NOM: Area under NOM treatment (ha). *Average rotation* (years).

The results given in this table show the ideal and anti-ideal values reached by the criteria, according to the optimized criterion (per columns). First, it can be seen that some objectives do not reach ideal values (*NF*, *IM*, *G*). Then, the level of conflict existing between the different criteria, mainly between *NF* and *IM*, and between *C* and *G*, can be observed, as well as the fact that the results are fairly similar in relation to carbon balance, because the percentage for this criterion varies the least compared to the rest. In synthesis, it can be noted that there is no solution corresponding to the optimization of a single criteria that appears to be sufficiently attractive. In short, the results of this pay-off matrix justify the setting up of an MCDM model in order to achieve solutions that are more balanced from the point of view of sustainability.

However, before revealing the results of those models, the result of the first interaction with DM1 to obtain the preferential weights associated with the criteria is shown (Table 2). Note that the sum of the weights for each priority level is equal to 1. Finally, Appendix A (Table A2) contains the weights obtained throughout the second interactive process with DM2 for the seven categories, in which the stands and the silvicultural alternatives associated with them have been grouped.

**Table 2.** Preferential weights associated with the criteria after DM1 interaction.

*NF*: Normal Forest structure, *IM*: Ideal management (ha), *G*: Gini index, *C*: Carbon balance (tC), *VOL*: Harvest volume (m3), *NPV*: Net Present Value (€).

Next, Table 3 displays the solutions obtained for the two priority levels proposed. It can be noted that the solutions at the second priority level (*U***2**) are the same as the values found in the preceding solution (*U***1**), whereas the values of the three criteria included in *U***2** improve in comparison with those obtained in *U***1**. Further, to facilitate comparisons, the same auxiliary rows have been included. In general, it can also be seen that the areas assigned to each silvicultural alternative are notably modified when moving from *U***1** to *U***2**.

**Table 3.** Results obtained in the resolution of model LGP (in bold the results for the six goals considered).

*NF*: Normal Forest structure, *IM*: Ideal management (ha), *G*: Gini index, *C*: Carbon balance (103 tC), *VOL*: Harvest volume (103 m3), *NPV*: Net Present Value (103 €). area-BAU1: area under BAU1 treatment (ha), area-BAU2: Area under BAU2 treatment (ha), area-GS: Area under GS treatment (ha), area-GTR1: Area under GTR1 treatment (ha), area-GTR2: Area under GTR2 treatment (ha), area-TNP: Area under TNP treatment (ha), area-NOM: Area under NOM treatment (ha). Average rotation (years).

In the auxiliary rows in Table 3, the area selected by the model for each silvicultural alternative is indicated. Also, this solution of the LGP model can be compared with the value proposed by the DM2 (the area pointed out as being the one preferred for applying each silviculture), information which is available on Appendix A (Table A2). In Figure 2 the difference (in surface) between both areas (applied or selected by the model and preferred) can be noted. However, it should be taken into account that the area preferred exceeds the value of the total forest area, because, for each stand, a set of 2–3 possible silvicultural treatments were proposed.

**Figure 2.** Relation between the silviculture applied in the forest and the deviation of when it would be preferred, according to the categories defined.

#### **5. Discussion**

The method here proposed for evaluating sustainability in forest management has the advantage of following the evolution of the criteria over the planning horizon [19], but it could also be adapted when including spatial constraints [97]. In that case, it could be suitable to design a model simultaneously integrating strategic and tactical planning [98]. Furthermore, the methodology proposed could be extended to incorporate the preferences of a group of stakeholders for addressing sustainability [99,100]. Some authors suggest compiling the forest managers' preferences on silvicultural aspects in a goal programming model [101], although the way these individual preferences would be aggregated is still an open question [102]. Finally, although the inclusion of dynamic aspects could trigger highly specific models, limiting their transference to others [103], we do not believe that this could happen in the proposal presented in this study due to its potential adaptation to other works, with other criteria and indicators.

The results presented in Table 3 allow us to observe the differences in relation to the values achieved for the indicators situated at the first priority level, at forest level. Thus, Figure 3 shows how the values of the criteria *NF*, *IM* and *G* obtained in the pay-off matrix are modified (Table 1), with respect to those obtained in the solution of the LGP model (Table 3).

**Figure 3.** Comparison between the results of the pay-off matrix and those obtained in the LGP model, for the criteria *NF*, *IM* and *G*.

As could be expected, in the comparison between models (Figure 3), the values of the criteria obtained in the LGP model are lower than the ideal ones reached in the pay-off matrix. These reductions indicate the opportunity cost associated with the simultaneous consideration of the six criteria defined in this model, instead of the values proposed in the individual optimization of each criterion. Thus, taking into account that the *NF* and *G* values (located at each end of the graphic) are adimensional and that they represent indicators of the less-the-better type, in the LGP solution their values are 2.7 and 1.6 times worse, respectively, compared to that obtained in the pay-off matrix. Finally, the *IM* criterion's value of 5358 ha, which represents the area managed by the most suitable treatment in each management unit (according to DM2), was reduced to 3750 ha in the LGP solution; i.e., about 30% of the area under ideal management (*IM*) is taken advantage of in LGP, using a different silvicultural treatment from the one that a priori was preferred for application in these areas.