In contrast, for the locations there are some statistically significant differences (*p* = 0.0106) for the average number of restoration motives, but not for the average number of restoration rationales advanced (Figure 5, panel C). That the restoration motives are different across the locations is vital to underscore, and such differences lie at two levels as suggested by the post hoc comparison tests. First, across all the five TAs, farmers in Vuso Jere consider fewer motives (<2) than their peers from the other four TAs (≥2), and this is strongly apparent when compared with TA Nthondo. The second difference centers on how some motives varied in importance from one location to another (Appendix A Table A2). One prominent example is the "leadership, encouragement, and support of local authority (M3)" when compared between TA Nthondo in Ntchisi District and TAs Kachindamoto and Kamenyagwaza in Dedza District (Figure 6, panel C). During the discussions in Ntchisi, farmers from Nthondo emphasized how their traditional authority encourages them and holds strong and respected leadership for addressing environmental degradation.

#### 3.2.2. Farmers' Restoration Benefits and Incentives

Discussion participants underscored several restoration benefits and incentives. Benefits were mainly economic (honey produced and sold from beehives put in trees, timbers, selling of crops yield surplus), environmental (moisture and nutrient added to the soil, fresh air and temperature regulation, good and reliable rains), and altruistic (care for future generation and natural trees). Other benefits are food-related (fruits from trees, high crops yield), non-economic and utility-oriented (poles for constructions, coffins, firewood for cooking, and medicinal plants), and socio-cultural (sharing of tree seedlings, access to free firewood during funerals). Illustrative perspectives include:

*"* ... *trees are used as poles for construction and are used for domestic activities such as firewood for cooking, and this reduces pressure on state-owned forest reserves."* (FGD, Bwanali Community, Kachindamoto, Dedza)

*"Through conservation-agriculture practices, higher yields are harvested since mulching conserves moisture and adds nutrients to the soil."* (FGD, Kapenuka Community, Kamenyagwaza, Dedza)

*"Forest and trees regulate temperature; it feels cold in hot season and it feels fresh always. I have a water point that runs from my forest. We receive reliable rains because we have more trees. [Also,] trees improve soil fertility in our fields and conserve moisture. We get manure from the tree leaves. If we plant trees, vetiver, and make contour ridges we make our lands fertile. [* ... *] Trees act as wind break and our houses are protected from severe winds. In the end, the benefits of trees are what motivate us."* (FGD, Ntchisi)

Farmers participating in collective-level restoration in Kabulika community noted economic benefits and explained that *"we also sell poles from the forest and we do this in collaboration with the chief. This is about our village forest which has both exotic and natural trees; people also get timber from the forest at agreed fees, poles are also harvested at a fee."* In reference to social benefits, some illustrated that *"we collect firewood from the forest when there is a funeral; for instance, there was a funeral in our village and women came to ask me as chairperson of the tree nursery management committee if they can go into the forest to collect firewood [and] I told the chief and we granted them permission to go and collect firewood."* Another farmer added they also *"share tree seedlings that [they] plant on [their] farms and house compounds, and people are motivated."* In Ntchisi, farmers bundled restoration benefits that motivate them, as one illustrates:

*"We do receive good rains because our forest is intact and that is one reason that motivates us. [Also], we get firewood for cooking. Other people who do not have trees in their areas use clothes and sacks for heating and cooking. [Moreover], we are motivated by a lot of water that our mountain and forest conserve and we use the water for irrigation."* (FGD, Ntchisi)

Incentives and rewards in the areas varied. For individual-level restoration, many farmers indicated being recipients of both in-kind and money-based incentives from NGOs and government-led programs. One explained: *"at first, people were receiving food and materials such as soybean flower, cooking oil and beans, tree seeds, hoes, and such, and after some time United Purpose started giving us money."* Adding to that, another farmer noted: *"MASAF was also giving us money, MK7200 [*≈*\$9.52] for two weeks; then payment rose from MK7200 [*≈*\$9.52] to MK21600 [*≈*\$28.57]."* Many considered the training they received on restoration matters and the perception of restoration practices as non-labor intensive, as forms of incentives.

*"We have been trained before in environment restoration [and] these restoration activities are cheap. For instance, manure making does not demand monetary costs as compared to inorganic fertilizers. [* ... *] Mulching also reduces labor; we do not make ridges and we do not go and weed because weeds do not grow in a field where ridges were not made. [* ... *]"* (FGDs, Bwanali and Kapenuka Communities, Dedza)

*"We received training in beekeeping, and we are making money out of bee farming. Discovery project trained us in cookstoves making; we have also been trained in briquettes making and we are making money from those activities. We received training in bamboo planting although not enough. We received training from World Vision, EU, CADECOM in environment management and restoration. For example, CADECOM trained us in local tree seedlings production and we can produce own seedlings locally."* (FGD, Kabulika Community, Kachindamoto, Dedza)

Beside these testimonials, farmers argued why training is a critical incentive, as one farmer wrapped it up:

*"For us to do our work properly we need to be trained and when we are trained, we can easily pass knowledge to all the people in the community. Training is very important for a common approach and strategy to restoration activities. We can all have one common idea of the technology and there cannot be confusion. Local knowledge is not universal; some people know things di*ff*erently from other people."* (FGD, Bwanali Community, Kachindamoto, Dedza)

Restoration incentives are not always required by farmers, and some indicated that their ongoing restoration efforts precede the renewed momentum in forest-agricscape restoration.

*"* ... *some of us started doing restoration activities in 2015 and 2016 when there were no projects [while] others started a long time ago when there was a project by ICRAF. Now people are used to these restoration activities and we are doing these without being paid or expecting to be paid."* (FGD, Bwanali Community, Kachindamoto, Dedza)

In contrast, the lack of incentives is associated with complaints. A farmer participating in collective action restoration in Ntchisi expressed long-standing resentment at such lack of incentives:

*"We do lots of work to conserve the forest and people come to see it because we are managing it perfectly. Unfortunately, communities are not benefiting. For instance, we do not have potable water in our community yet our forest conserves water. People from Blantyre, Mzuzu, and all over the country and even abroad come here to see this place just because the forest is well-conserved and managed, but the communities are not benefiting anything apart from good rains we receive. [* ... *] We work in dangerous environments with no protective gear, yet we do not receive anything."* (FGD, Ntchisi)

Hence, many farmers made a case for incentives. They also suggested operational resources and materials as well as training and exchange visits for restoration activities as incentives. For instance, during the discussion in Kapenuka in Dedza, one farmer elaborated that "*we don't receive incentives, but we would prefer being given operational resources"*; another that *"there has to be incentives such as fertilizer."* In Ntchisi, a farmer was also specific: *"incentives we need are soft loans with very low interest [and] energy-saving stoves that reduce pressure on forest resources."* Other forms of suggested incentives include money as allowance for collective action tasks, such as patrolling, or in the form of recurrent salary, an important element for restoration sustainability.

*"With the amount of work that we do, we are supposed to receive something in the form of money. I want to remove some perception that organizations have* ... *they think on our behalf and they think we cannot conserve or manage forests if they give us money. [* ... *] The money should be in the form of allowance and not salary. Whenever you go to patrol or do forest management activities, you should get an allowance, which is MK900 [*≈*\$1.19] according to the government's rate."* (FGD, Ntchisi)

Here also, benefits and incentives for both individual farmer and collective action are intertwined and folded into the restoration motives. For instance, one farmer from the discussion in Kapenuka in Dedza advanced that *"we are also motivated by the high yields we are harvesting from fields where we are practicing soil-conservation technologies. For instance, where we apply manure and use fertilizer trees, we harvest more."* Some farmers from the discussions in Bwanali and Ntchisi also stressed that *"benefits we realize from restoration activities are already incentives on their own [* ... *] for example, food, firewood and water supply [* ... *] and also the activities such as money from sales of trees and from yield surplus."* An unusual perspective was underscored as follows, citing group learning and labor pooling: *"working in a group is incentive already because we learn from the group and implement on our farms; for instance, planting trees and making contour ridges. It is also less laborious when working in a group [because] it reduces time and labor demand as we share responsibilities and knowledge in a group. We can easily make manure and apply on our farms."*

Importance and Differences in Restoration Benefits and Incentives by Gender, Education, and Location

While substantiating the qualitative findings, the survey data depict the relative importance farmers attribute to restoration benefits and incentives. On average, farmers considered 3 ± 1 benefits and ≈1 ± 1 incentive. Specifically, crop yield improvement (96%), sustainable supply of biomass energy (firewood provision and charcoal production) (69%), positive environmental effects of reforestation (66%), and adaptation strategies to climate change impacts (59%) are the most critical benefits cited. Sustainable provision of non-timber forest products (NTFPs) was mentioned less (11%). When asked what incentives they received, *"no incentives"* was the most reported response (54%) by farmers, confirming the limited existence of restoration incentives in the area. The two most important incentives mentioned were knowledge and information support from extension services such as training on sustainable land management practices and supply of information on restoration matters (36%), and free or subsidized inputs such as tree seedlings and agricultural fertilizers (25%). Incentive schemes such as cash for work and credit/loan provision were less cited (6%).

The mean number of restoration benefits was significantly different (*p* = 0.0001) between men and women, but restoration incentives were not (Appendix A Table A3). On average, male farmers reported enjoying more benefits (>3) than female farmers (<3) (Figure 7, panel A).

Among the typical benefits described above, men perceived very strongly the positive environmental effects of reforestation and were more likely to report sustainable firewood provision and charcoal production than women. Men also referred to the sustainable provision of NTFPs as part of restoration benefits, which women barely pointed to (Figure 8, panel A). Likewise, the average number of restoration benefits was statistically different (*p* = 0.0003) by location, but restoration incentives were not (Figure 7, panel C). Farmers from TA Nthondo reported fewer benefits from restoration (=2) than farmers from the other TAs, who reported more (≥3). Restoration as a coping strategy for climate change and as contributing to a sustainable supply of biomass energy was not emphasized in the TAs of Ntchisi as it was in the TAs of Dedza (Figure 8, panel C). In contrast, no statistically significant difference is observed for the number of restoration benefits and incentives across education levels (Figure 7, panel B). Both types of benefits and incentives are similar across the different education levels (Figure 8, panel B).

**Figure 8.** Types of restoration benefits and incentives by gender, education, and location. Notes: Radar charts comparing the mean value (on a scale of 0–1) of each type of benefit/incentive among the different groups/factors. **Benefits:** B1 = Crops yield improvement; B2 = Reforestation and its environmental benefits; B3 = Sustainable provision of firewood and production of charcoal; B4 = Sustainable provision of NTFPs; B5 = Cope with climate change impacts; B6 = Increase tourism/income source/other benefits. **Incentives:** I1 = Cash for work incentives schemes and credit/loan provision schemes; I2 = Free improved cookstoves; I3 = Free/subsidized seedlings for nurseries and agricultural fertilizers; I4 = Training on SLM practices and supply of agricultural information; I5 = Granted land/tree use rights/trainingallowances/in-kindincentives/otherincentives;I6=Noincentive.

#### *3.3. Farmers' Restoration Decision-Making Processes*

We followed two steps to develop the overarching restoration decision-making processes in a table/matrix format. These include the depiction and description of the goal frames, and the identification and characterization of the different rules.

#### 3.3.1. Restoration Decision-Making Goal Frames

We identified a total of 17 goal frames representing different categories of the registered factors (Table 4). We first identified 10 goal frames from the discussions. Further, we drew 15 goal frames from the survey that included eight of the previous goal frames uncovered from the qualitative data, thus adding seven new goal frames to the list. Together, the 17 goal frames indicate a mixture of the restoration rationales, motives, benefits, and incentives described above, to which new elements, indicating factors reflecting concerns and constrains, are added. We report their respective incidences in individual and collective-level restoration.

#### 3.3.2. Restoration Decision-Making Rules

We draw out 21 distinctive rules for individual-level restoration and 19 rules for collective-level restoration. All but two rules from the discussions match those from the survey. The rules reflect how the goal frames are ordered. The highly occurring rules are displayed in Table 5 (full list in Tables A4 and A5).

Following their root goal frames, the rules that influence farmers' restoration behaviors are: problem-solving oriented, resources/materials-constrained, benefits-driven, incentive-driven, peers/ leaders-influenced, knowledge-dependent, altruistic-oriented, rules/norms-constrained, economic capacity-dependent, awareness-dependent, and risk averse-oriented. The main root goal frames for rules at the individual level were problem solving, resource/material constraints, incentives, knowledge, and benefits. When considering rules in collective actions, leadership of local authority, problem solving, benefits, and incentives stood out as highly critical root goal frames, followed by resource/material constraints.

simplegoalregisteredgroup percentagedenotinggoalthe survey respondents; Bold: The most important goal frames. Italics: Common goal frames in the discussions and the survey.

*Sustainability* **2020** , *12*, 5380

individual level, did not provide any factors and rules; # Two respondents,

 among the 268 who claimed to restore land at the collective level, did not provide any factors and rules.

#### 3.3.3. Restoration Decision-Making Processes

We construct the processes as a matrix table linking each rule with the vegetation-based and/or non-vegetation-based restoration practices/activities (Tables 6 and 7; see details of their occurrences in Appendix A Tables A6 and A7). On average, the total number of restoration practices farmers applied is 3-4 (mean 3.59) and 4–5 (mean 4.53) for individual-level and collective-level restoration, respectively. Many restoration practices/activities are common across the rules.

At the individual farm-household level, common vegetation-based restoration activities include agroforestry, farmer-managed natural regeneration (FMNR), and vetiver grass (*Chrysopogon zizanioides*) planting. Farmers aligned with rules #8 implement vegetation-based restoration to a lower extent. Those applying rules #12 and #18 implemented only one practice, FMNR and vetiver grass planting, respectively. Farmers associated with rules #11, #13, #20, and #21 engage with only two of the vegetation-based practices. Most recurrent non-vegetation-based restoration activities are manure application, mulching, and construction of contours ridges. For farmers following rules #8, #18, and #20 mulching comes first ahead of manure application, in contrast to the common trend. Secondary implemented practices include intercropping, swales making, markers/box ridges construction, and minimum or no tillage. Farmers utilizing rules #12, #16, and #19 strongly engage with intercropping, and a fair number of farmers following rules #11 and #18 construct swales.

At the collective-action level, vegetation-based restoration activities center on tree planting and natural regeneration in forest areas. All types of rules greatly reflect those restoration activities, except rule #13 which leads to less engagement in such activities. Farmers following rule #19 plant trees also for riverbank protection. The non-vegetation-based restoration activities most encountered are activities involving firebreaks in communally held forest areas, awareness against tree cutting and deforestation in the community, and other forest protection activities such as patrolling and monitoring. Construction of swales is observed to a lower extent on communal lands. Additionally, farmers following rule #19 are involved in gully reclamation.

**Table 6.** Decision-Making Processes for Individual Restoration of Forest-Agricscapes in the Study Areas.

Bolded letters refer to restoration practices applied by at least half of the respondents who display the specific rule; letters in italics represent related restoration practices with very low occurrence (<20%).

**Table 7.** Decision-Making Processes for Collective Restoration of Forest-Agricscapes in the Study Areas.

Bolded letters refer to restoration activities applied by at least 75% of the respondents who display the specific rule; letters in italics represent related restoration activities with a relatively low occurrence (<40%). The specific restoration activities and their magnitude could not be elicited for Rule#6 as this rule was identified only from the games and not from the survey.

*Key:* Vegetation-based restoration practices: *a* = Agroforestry on farms or active tree planting in forest areas; *b* = Farmer-managed natural regeneration on farms or Natural Regeneration in forest areas; *c* = Pits planting *(Zai)* on farms or on communal lands; *d* = Vetiver grass/other grasses on farms or on communal lands; *e* = Improved fallow on farms; *q* = Riverbank planting for protection of rivers/streams.

Non-vegetation-based restoration practices: *f* = Crop association/intercropping on farms; *g* = Mulching + crop residue incorporation on farms; *h* = No or minimum tillage on farms, *i* = Manure making and application on farms or on communal lands; *j* = Swales on farms or on communal lands; *k* = Contour ridges on farms; *l* = Marker/box ridges on farms or on communal lands; *m* = Rotation on farms; *n* = Fire-breaks in forest areas; *o* = Awareness against tree cutting and deforestation in the community; *p* = Other forest protection activities in forest areas; *r* = Gully reclamation on community lands.

#### **4. Discussion**

#### *4.1. Local Perspectives on Landscape Restoration*

Our findings on the local farmers' perspectives on landscape restoration reflect concerns over both adverse ecological and socioeconomic/livelihood conditions that need concurrent improvement. This dual consideration shapes the practices and activities used for restoration. These findings align with the perspective of the contemporary restoration movement, especially codified in the forest landscape restoration (FLR) paradigm being promoted widely in SSA, including Malawi. Indeed, the new wave of restoration has departed from pure eco-centric perspectives, as seen with ecological restoration, to accommodate more social dimensions, especially with regard to socio-economic well-being to which ecosystem services are a major contributor [40]. As reflected in the commonly used definition of FLR (a planned process aimed at regaining ecological integrity and functions to enhance human well-being in deforested or degraded forest landscapes, [41]), emphasis is put on social, economic, and ecological goals and outcomes.

Moreover, in a previous review, Djenontin et al. [6] pointed to considerations of local contextual factors and aspirations in defining the objectives and goals of such landscape-scale restoration as well as in choosing and promoting technological packages. In that regard, our findings can inform efforts to operationalize FLR goals and practices in mosaic forest-agricscapes, without obscuring the embedded socio-cultural context, to achieve more meaningful and sustainable outcomes. Puspitaloka et al. [42] recently attempted such contextual operationalization by rearticulating restoration in peatland ecosystems in Indonesia, following an empirical assessment of the definitions, goals, and practices of peatland restoration across four restoration interventions in Central Kalimantan.

#### *4.2. Landscape Restoration Rationales, Motives, Benefits, and Incentives*

Our study reveals that while the boundary between restoration rationales and motives remains fuzzy, their nature differs. The former follows the line of logical/causal reasoning while the latter follows affective/emotional action. The distinction is worth making as it has implications on the proper specification of behavioral paradigms and subsequent parameterizations of decision rules in modeling restoration behaviors, as we discuss below. Moreover, our findings provide justification for treating benefits and incentives as extensions of restoration motives. Benefits and incentives were regarded as personal advantages and gains, and they considerably shape restoration behaviors at individual farm-household and collective-action levels. This study confirms, empirically, the need to consider them in restoration programs and policies as observed with benefits/incentives-based interventions in enhancing restoration behaviors.

Furthermore, the motives and benefits vary significantly by location and gender. Notably, having strong leadership from local authorities who support and encourage their community members through self-engagement and exemplary actions has emerged as an integral motivational element. Ntchisi district, exemplified by TA Nthondo, holds that advantage over Dedza district. This finding confirms previous evidence of the importance of strong, engaged, and inspiring leadership in collectively addressing environmental degradation [43,44]. Further, while there were many types of perceived or actual benefits from restoring forest-agricscapes, the gender and locational differences regarding environmental and energy supply-related benefits are important to note in planning restoration interventions. Specifically, this finding can inform the design of restoration programs as to what place-specific and gender-inclusive motives to leverage, and benefits to enhance or promote.

#### *4.3. Landscape Restoration Decision Making and Behavior*

Our findings indicate that restoration rules show various combinations of goal frames, the most important ones making up the roots for the rules. This nature of rules confirms that environmental behaviors are the result of multiple goal frames with some dominating ones. This means that one or more goal frames shift to become focal goal(s) over the others in the process of cognitively constructing the decision-making rules. This corroborates the postulate of the GFT [14,15].

Furthermore, insights on the defining constituents of the goal frames themselves indicate that incentives, benefits, and knowledge could be assimilated with the gain and hedonic goal framings, following [14]. These factors also compare with some of the factors influencing land-use and environmental-management decision making, notably the consideration of economic and non-economic benefits [11]. In our study areas, incentives and benefits were primarily not expressed in monetary terms, but more as in-kind (both soft, like training incentives, and concrete, like crop yields, poles, and other tree-resource benefits), a departure from the widespread over-reliance on cash incentives in collective conservation or restoration policies and strategies (see [44]).

However, problem solving, resource constraints, and leadership of local authorities stand out as contextual types of goal frames associated with restoration behaviors. From the perspective of the GFT [14], these elements do not reflect nor qualify as normative, gain, or hedonic goal framings. Rather, they refer to environmental concerns, capability, and political impulsion. The latter is of interest as the leadership of local authorities greatly influences collective action restoration decisions. These findings underscore the need to ground understanding of restoration behaviors within their local contexts.

#### *4.4. Representation of Restoration Decision Rules in a LUCC-ABM*

We have developed a conceptual approach to depict restoration decision-making processes based on our effort to empirically depict farmers' decision-making rules and the corresponding vegetation-based and non-vegetation-based restoration practices and activities in Central Malawi. This effort is similar to Keshavarz and Karami's [45] effort to identify farmers' decision making and actions for drought management in Iran. It can serve as an input into future ABM-based restoration

modeling aimed at simulating farmers' restoration behaviors, underlying decisions, and the spatially explicit socio-ecological outcomes at a higher (aggregate) forest-agricscape scale in Central Malawi. This approach can also be replicated elsewhere for similar studies attempting to ground social actors' decision rules in empirical data.

Significantly, we have uncovered that farmers incorporate both rationally grounded and motive-based goal frames in their critical decision making on using different restoration practices or getting involved in collective action restoration activities. This implies that when exploring landscape restoration using an ABM with a focus on farmers' restoration decision rules, one cannot solely espouse a rational behavior paradigm. Rather, one should also consider the role of motivational factors, including benefits and incentives. Groeneveld et al. [11] argued that the importance of such non-rational factors influencing land-use and environmental management decisions had significantly been overlooked in LUCC-ABMs, whereas they are appropriately emphasized in behavioral economics. In this regard, our findings support the adoption of a bounded rational behavior paradigm in such future modeling of restoration behaviors.

In addition, we encourage adopting a process-based decision-making model to represent human decision making and circumvent the difficulties of applying theories following previous studies [26,46]. Insights suggest that blending ideas of experience/preference-based decision models with empirical/ heuristic decision rules [26] would be an appropriate decision framework for exploring the future impact of landscape restoration in Central Malawi. Thus, our empirically grounded restoration-decision rules and processes are relevant to inform the parameterization of decision rules and representation of restoration actions for such an ABM. The occurrence of the rules could inform their respective probability distribution in the ABM and the restoration practices and activities would inform the actual actions taken by the farmer-agents.

Finally, methodologically, this study contributes to testing or putting into practice relevant recommendations on data gathering methods and processes to improve the representation of human decision making in empirical ABMs and enhance their analytical and policy value [11,18,23–25,47]. Adopting mixed social science data collection methods, including semi-structured interviews through discussions and games, and a structured interview via surveys, appears useful. They can enhance understanding of the decision making underlying farmers' behaviors to characterize decision rules for further modeling processes.

#### **5. Conclusions**

This study sought to analyze the nature of the decisions to engage in forest-agricscape restoration through individual and collective actions in Central Malawi using a mixed-method approach to data collection and analysis. The study uncovers local farmers' perceptions of forest-agricscape restoration, and the nature of the influential factors considered when deciding to engage in restoration efforts. Furthermore, it reveals how these factors generate diverse goal frames determining restoration decision making, and ultimately the observed restoration practices and activities. Specifically, the study finds that the decision-making rules leading to restoration behaviors appear to be very diverse. They are made of goal frames that reflect nuanced considerations of environmental problems, livelihood needs and gains, constraints, socio-political influences, morals, values, and attitude to risks; all featured in diverse combinations. These restoration goal frames are categorized as problem-solving oriented, resources/materials-constrained, benefits-oriented, incentive-based, peers/leaders-influenced, knowledge/skill-dependent, altruistic-oriented, rules/norms-constrained, economic capacity-dependent, awareness-dependent, and risk averse-oriented. Improved understanding of the goal frames underscoring restoration decision-making rules is critical to inform potential management and policy mechanisms to boost restoration delivery. Finally, the study contributes a conceptual approach, with methodological application, to elicit restoration decision-making processes that associate various decision rules with the vegetation-based and non-vegetation-based restoration practices and activities undertaken by farmers. This will contribute to

empirically ground the design and parameterization of farmers' restoration behaviors in an ABM that will explore effective governance modalities and spatially explicit policy options to boost landscape restoration in Malawi. Beyond that, this approach can be replicated elsewhere for similar studies attempting to ground social actors' decision rules in empirical data.

**Author Contributions:** I.N.S.D. and L.C.Z. conceived of the study; I.N.S.D. designed the data collection, with constructive suggestions from L.C.Z., and A.L.-Z.; I.N.S.D. collected the data, with initial supervision by L.C.Z.; I.N.S.D. analyzed the data; A.L.-Z. and I.N.S.D. elaborated the analytical framework; All I.N.S.D., L.C.Z., and A.L.-Z. wrote the paper. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was partly funded by the Research Scholars Award from the College of Social Sciences of Michigan State University (MSU). It was also financially supported by the Environmental Science and Policy Program of MSU and the Graduate Women International (GWI) under the GWI Recognition Award.

**Acknowledgments:** We are grateful to the Department of Geography, Environment, and Spatial Sciences of MSU for its various institutional support. The first author thanks particularly all members of her research team in Malawi for their invaluable support in collecting the data for this research in the framework of her PhD dissertation.

**Conflicts of Interest:** The authors declare that they have no conflict of interest.

#### **Appendix A**

#### **Table A1.** Sample Characteristics Across the Study Area.

**Table A2.** Differences in the Number of Restoration Rationales and Motives by Gender, Education, and Location.

<sup>a</sup> Student's t-test or one-way ANOVA test of variables "Restoration Rationales" and "Restoration Motives" with either equal or unequal variances; <sup>b</sup> Variable violates the Bartlett's test (homogeneity of variances): assumption of homogeneity of variance is not met. Only the Games and Howell post comparison test is robust then, compared to the Scheffe Test. We consider the multiple comparison tests only for variables with significant t or F test.

**Table A3.** Differences in the Number of Restoration Benefits and Incentives by Gender, Education, and Location.

<sup>a</sup> Student's t-test or one-way ANOVA test of variables "Restoration Benefits" and "Restoration Incentives" with either equal or unequal variances; <sup>b</sup> Variable violates the Bartlett's test (homogeneity of variances): assumption of homogeneity of variance is not met. Only the Games and Howell post comparison test is robust then, compared to the Scheffe Test. We consider the multiple comparison tests only for variables with significant t or F test.

*Sustainability* **2020**, *12*, 5380

individual level, did not provide any factors and rules.

*Sustainability* **2020**, *12*, 5380

a

level, did not provide any factors and rules.

restoration activities and their occurrence could not be elicited for Rule#6 as this rule was depicted from the games only and not from the survey.

### **References**

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **Human Simulation and Sustainability: Ontological, Epistemological, and Ethical Reflections**

**F. LeRon Shults 1,\* and Wesley J. Wildman 2,3,4**

Received: 25 October 2020; Accepted: 26 November 2020; Published: 1 December 2020

**Abstract:** This article begins with a brief outline of recent advances in the application of computer modeling to sustainability research, identifying important gaps in coverage and associated limits in methodological capability, particularly in regard to taking account of the tangled human factors that are often impediments to a sustainable future. It then describes some of the ways in which a new transdisciplinary approach within "human simulation" can contribute to the further development of sustainability modeling, more effectively addressing such human factors through its emphasis on stakeholder, policy professional, and subject matter expert participation, and its focus on constructing more realistic cognitive architectures and artificial societies. Finally, the article offers philosophical reflections on some of the ontological, epistemological, and ethical issues raised at the intersection of sustainability research and social simulation, considered in light of the importance of human factors, including values and worldviews, in the modeling process. Based on this philosophical analysis, we encourage more explicit conversations about the value of naturalism and secularism in finding and facilitating effective and ethical strategies for sustainable development.

**Keywords:** computer modeling; human simulation; social simulation; sustainability; development studies; assemblage theory; ontology; epistemology; ethics

#### **1. Introduction**

This Special Issue highlights the potential of computational modeling and simulation (M&S) to contribute to research on human–environment interactions. Studying the intricate, multi-dimensional, and nonlinear dynamics that characterize these interactions requires powerful analytic and synthetic tools suited to understanding complex adaptive systems. This article explores some of the challenges and opportunities involved in the computational modeling of sustainability from the perspective of "human simulation," a new transdisciplinary and collaborative approach to M&S, and offers some reflections on philosophical issues surrounding the task of finding and facilitating more ethical, effective, and efficient modes of adaptation and sustainable development in our increasingly pluralistic, globalizing, and ecologically fragile environment.

In the next section, we describe some recent advances in sustainability modeling that use M&S techniques. Here, we are using the term sustainability in a broad sense, inclusive of the wide range of themes and tasks identified in the United Nations' Sustainable Development Goals [1]. Problems related to sustainability may be explored from the perspective of ecological, climate, biological, economic, cultural, or social systems. Here, we explore ways in which M&S can help provide answers to questions about the reciprocal causal relationships between human behaviors and the natural systems within which they are entangled at all these levels. These methodological tools have been growing rapidly in popularity in the social sciences more broadly [2–4], but their potential relevance for disciplines

such as development studies, which are increasingly organized around issues of sustainability [5], has not yet been fully appreciated. In particular, we identify gaps in coverage and associated limits in methodological capability in regard to taking account of human (all too human) factors, biases, and tendencies that are often impediments to a sustainable future. The third section introduces "human simulation," a collaborative approach to M&S that engages subject-matter experts, policy professionals, and stakeholders at all stages of the model construction process (including design, implementation, and dissemination of results). With its explicit inclusion of insights from cognitive science, moral psychology, and a rich variety of human sciences, human simulation opens up new vistas for sustainability research, especially through its use of multi-agent artificial intelligence modeling techniques, which are better capable of incorporating human factors relevant for navigating toward a sustainable future into their computational architectures. The fourth section identifies and discusses some of the key ontological, epistemological, and ethical topics that are raised at the intersection of human simulation and sustainability research, focusing on the values and worldviews that tend to advance or impede sustainability. In this connection, we show how approaching sustainability research through human simulation highlights the positive potential of naturalistic explanations and secularistic strategies in our shared struggle to adapt in the Anthropocene—that is, the current geological age, during which human activity has become a dominant influence on the environment.

#### **2. Advances in Sustainability Modeling**

In the broadest sense, all research on sustainability involves modeling—the construction and testing of coordinated concepts and theories that aim to make sense of what is happening (or may happen) in complex social systems. Nobel prize winner Elinor Ostrom, for example, developed an influential theoretical model of the conditions for the sustainability of socio-ecological systems (SESs), in which she put a heavy focus on the complexity and heterogeneity of such systems [6]. Such theoretical models are obviously useful, but in this context, we are interested specifically in computational models based on the formalization of theories such that they can be implemented in computer programs. In fact, Ostrom herself worked with computer scientists to explore the usefulness of agent-based models for analyzing the robustness of SESs [7,8].

There are pragmatic reasons for emphasizing the value of such models for sustainability research. Computational models are uniquely able to express the dynamics of complex adaptive systems, including the behaviors and interactions of agents within the simulated time and space of a virtual ecological environment. Multidisciplinary research teams have developed social simulations aimed at addressing a wide range of societal challenges that affect the sustainability of human groups, including the COVID-19 pandemic [9,10] and the integration of economically, ethnically, and ideologically diverse populations [11]. In this context, we focus on the application of M&S to more classical themes in sustainable development.

Given the potential of M&S approaches, it is not surprising that they were applied relatively early in their development to the task of understanding the conditions under which human societies would remain sustainable in the face of major climate change. The (in)famous Club of Rome system-dynamics models in the 1960s [12], for example, illustrated the limitations as well as the opportunities involved in modeling sustainable futures. Over the decades, the accuracy, explanatory power, and insight-generating capacity of computational models of climate change have increased rapidly [13], as have the diversity of approaches to modeling human–environment interactions that address the growing challenges of the Anthropocene [14–17]. The importance of linking models of human behavior and models of climate has been poignantly brought home by a recent article showing the significant extent to which including social processes (such as shifts in the perception of risk) can alter projections in climate change modeling [18].

This suggests the need for renewed attention to the task of developing models that incorporate the complexity and mutuality of human–environment interactions as we explore the challenges and opportunities for transitioning toward more sustainable societies. In fact, M&S has been increasingly used in recent years to study sustainability transitions at a variety of scales from one (relatively) stable socio-ecological equilibrium to another. For example, M&S has been employed to study the importance of local communities and the role of local leaders in encouraging or discouraging low-carbon energy transitions [19], and the importance of broader, collaborative, multi-level adaptive strategies for managing regional and international sustainability transitions [20]. Other models have been able to simulate some of the major historical shifts in human civilizational forms, such as the transition from hunter-gathering to sedentary agriculture [21], the transition from pre-Axial to Axial Age civilizations across the globe during the first millennium BCE [22], and the ongoing modern transition from cultures in which supernatural beliefs play a dominant role in maintaining social cohesion to more naturalistic and secular cultures [23].

The academic community is increasingly realizing the potential of agent-based models (ABMs) in particular as a consolidated "bottom-up" multidisciplinary approach for studying the emergence of collective responses to climate change policies and related issues by taking into account both "adaptive behavior and heterogeneity of the system's components" [24] (p. 17). One of the distinctive features of ABMs is the capacity for linking the micro-, meso-, and macro-level factors at work in complex adaptive systems. Appropriately verified and validated ABMs can shed light both on the conditions under which and the causal mechanisms by which social (and other) macro-phenomena emerge from micro-level behaviors and meso-level network interactions. This ability to "grow" social phenomena within artificial societies has led some scholars to refer to such methods as generative social science [25]. Although it is still not uncommon to find system dynamics modeling techniques used to study the sustainability of such systems [26,27], ABMs seem to be gaining ground as the computational methodology of choice especially when the primary purpose of the model is to inform policy discussions in a way that takes human behavior into account.

ABMs have been utilized for research on a wide variety of issues related to the sustainability of social systems, including the way in which: cultures arise, develop, and evolve through time [28], norms emerge and shift within populations [29], cooperation is maintained and enhanced within pluralistic contexts [30], diverse agents impact land-use and land-cover change [31], and different values interact with the treatment of and public attitudes toward refugees [32]. The influential MedLand model links an ABM to a landscape evolution model in order to provide a platform for "experimental socioecology," i.e., a platform for studying the interactions among social and biophysical processes in order to inform decision-making strategies for land-use related to issues such as farming and herding [33,34]. Multi-agent models have also been developed to study other probabilistic causal processes in coupled human and natural systems (CHANS), including how interactions between climate, built environment, and human societies could lead to permafrost thawing in Boreal and Arctic regions, thereby greatly impacting human migratory movements [35]; and how interactions between human decisions and natural systems could lead to the long-term sustainability of forest ecosystems in areas such as north-central Texas [36]. Given the potential of ABM techniques to shed light on the reciprocal relations between human behavior and ecological change, it is not surprising that calls for their use in research on the sustainability of economic systems are increasing [37,38].

A recent state-of-the-art survey of methodologies and models for studying SESs summarized the achievements of the field and articulated persistent challenges, such as the need to represent human decision-making more accurately in light of knowledge from social psychology [39]. Other scholars have called for more precise and formal ontologies in SES models that pay more attention to the pivotal role that human actors play in such systems, which is often underrepresented in SES modeling [40]. One recently proposed framework calls for comparing ways in which different behavioral theories (e.g., rational actor, bounded rationality) impact SES outcomes. This opens up the possibility for "sensitivity analysis of human behavior within a natural resource management context [;] such an analysis enables assessing the robustness of the performance of policy options to different assumptions of human behavior" [41] (p. 33); see also [42]. Understanding such human factors requires special attention to insights from biological and cultural evolution into the ways in which humans have adapted (and

might adapt) to changes in their environment [43]. This is why several of the leading scholars in the application of ABMs to sustainability issues have attempted to develop models with agents who are capable of adapting to (simulated) climate change challenges [44,45] and called for even more behaviorally realistic agent architectures [46].

Most SES models that have explicitly tackled challenges related to sustainability have focused on particular regions and specific policy intervention options such as pastoralist conflict in eastern Africa [47,48], decision-making related to a watershed in Kansas [49], and sustainable practices for grassland consumption in Mongolia [50]. ABMs have also been used to evaluate policies for promoting "climate clubs" oriented toward facilitating cooperation among institutions and states in response to the challenges of the Anthropocene [16,51–53]. Social simulation techniques are increasingly applied in the study of sustainability transitions such as shifts toward low-carbon energy [19,54]. For reviews of other uses of ABMs to address issues related to human adaptation and sustainability, see [17,24,55,56].

This growing interest among sustainability researchers and policy stakeholders in using ABMs and other computational modeling techniques to identify "plausible and desirable futures in the Anthropocene" [57] is obviously good news. Such tools are being used to pursue an "experimental socioecology" that can enhance diverse decision-making strategies in cooperative processes such as land-use [33], to optimize cooperation and information exchange [58], and to facilitate the construction of "future safe and just operating spaces" for human societies [26]. Scholars in this sub-field of M&S are quite aware of the limitations in the use of ABMs to study the sustainability of CHANS or SESs, especially challenges related to verification and validation, but are striving to improve the state of the art [39].

#### **3. The "Human Simulation" Approach**

This brief review of the literature documents growing interest in the use of M&S methodologies to address concerns related to our capacity to attain and maintain sustainable human–environment interactions. However, it also suggests that those who engage in this endeavor are faced with at least two major challenges. First, as we have seen, many scholars in this field are acutely aware of the difficulty (and the importance) of constructing more cognitively and sociologically realistic simulated agents and artificial societies in order to take account of the human factors that are important in moving our civilizations toward a sustainable future. Computer modelers have been working on this for decades and several models with relatively psychologically plausible and behaviorally realistic agent architectures have recently been developed for simulating human decision-making in complex environments [41,59,60]. Simulation results from one such "socio-climate" model designed to chart optimal pathways to climate change mitigation suggest that the most efficient route to a reduced peak temperature anomaly is to focus first on increasing social learning (better informing the population about climate change) prior to reducing mitigation costs [61]. However, as the authors of that model acknowledge, their approach assumed homogeneous agents in a socially unstructured artificial society, and they point out the need for new models that involve heterogeneous agents nested in hierarchically complex social structures.

Second, despite extensive efforts in recent decades to engage policy professionals, public stakeholders, change agents, and subject matter experts in the process of model design, simulation implementation, and interpretation of results, there is a broad consensus among computational social scientists that we need to invest more energy into improving our strategies for (and practices in) participatory modeling [27,62–66], including the modeling of CHANS [67,68]. This challenge grows ever more salient as the ethical implications of the development and use of artificial intelligence for human life and social order increasingly become a part of public policy discourse. This is not merely a matter of training simulation engineers to be more sensitive and better able to engage and elicit wisdom from stakeholders, though that will certainly help. It is also about paying attention to the limitations of overly abstract sustainability simulations that seem irrelevant to stakeholders who spend every day thinking about the complexity of human responses to societal instability in the face of ecological

change. In other words, it is about registering within simulation models the very human factors that experts know are crucially important in determining whether or not positive change occurs in our struggle to forge a sustainable future.

The approach outlined and illustrated in *Human Simulation: Perspectives, Insights, and Applications* [69] is intended as a response to these (and related) challenges. The introduction to that volume articulates a transdisciplinary approach that is committed to involving scholars from the humanities and the social sciences, along with subject matter experts, policy professionals, and other stakeholder change-agents in the process of constructing computational simulations that address pressing societal challenges. Human simulation strives to incorporate vital human factors into agent architectures and the artificial societies they inhabit in order to account more adequately for the cognitive and cultural complexity of our species within its models, as well as for the role of actual humans in the ethical production and use of computational models. The overarching vision for the participatory approach in human simulation is depicted in Figure 1.

**Figure 1.** Components of the human simulation participatory approach.

All of these components are oriented toward understanding and solving societal problems (e.g., cultural integration, religious conflict, pandemics, economic exploitation). The top circle is data, indicating the seriousness with which the human-simulation approach takes the need for empirical validation when modeling for explanatory or forecasting purposes. The upper-right circle represents the wealth of resources that M&S methodologies can bring to the task. The lower-right circle indicates the vital role of subject matter experts (SMEs), including from relevant disciplines in the social sciences and the humanities (even philosophers, as we will see below). The lower-left circle points to the critical role of policy professionals in informing research and translating its results into effective change at the policy level. Finally, the upper-left circle expresses the importance of rigorous engagement with stakeholders and change agents who may not be directly involved in public policy but can create the conditions for sustainable development from the bottom-up. All five components of this approach to problem-solving are crucial within human simulation.

The *Human Simulation* volume offers several methodological chapters as well as substantive examples of collaborative social simulation involving a wide array of disciplines, including the development of more cognitively and sociologically realistic computational models designed to study human phenomena such as cultural integration, ritual participation, religious systems, and empathic cooperation. They also show that not all five components (the outer circles in Figure 1) are equally important in every application of the approach since their relevance depends on the problem and the audience to which a model is directed. Typically, however, the pressing problems associated with modeling sustainability require attention to all five components.

The human-simulation approach has also guided the development of several earlier models that incorporate human values or worldviews and are relevant for studying sustainability. We have found that multi-agent artificial intelligence (MAAI) models, which incorporate more psychologically realistic and socially networked simulated agents than traditional ABMs [70], are particularly useful when factors related to human cognition and culture play a significant role in the target phenomenon. For example, one MAAI model simulated the causal relationship between mortality salience and religiosity within an artificial society whose networked agents had cognitive architectures informed by the literature on terror management theory, and who were confronted by environmental threats (including contagion and natural hazards) to which humans have evolved to react with anxiety [71]. A later model, which incorporated additional variables within the artificial society, further informing the agent architectures with social identity theory and identity fusion theory, was able to simulate the conditions under which—and the mechanisms by which—mutually escalating conflict between religious groups tends to increase (or decrease) in a population [72]. Another MAAI model that utilized the human-simulation approach was able to simulate the expansion of secularism in the populations of 22 countries. Utilizing more psychologically realistic simulated agents in social networks, it was able to "grow" macro-level secularization in artificial societies from micro-level agent behaviors and interactions [73].

All of these models were constructed through extended collaboration among the various kinds of experts described above, following a particular procedure (depicted in Figure 2).

**Figure 2.** Navigation of insight space in the human-simulation approach.

This procedure involves the navigation of a conceptual and computational model's "insight space," which comprehends both its problem space and its solution space (see [69] for further details). Under each of the steps in Figure 2, key tasks are listed on the left, and examples of human factors that need to be considered are described on the right. Note that these examples of human factors are relevant not only in the insight space of the model but also in the workflow of the modeling team. The human-simulation methodology calls for managing human factors in the modeling *process* with every bit as much energy and creativity as it seeks to incorporate human factors into simulated agent minds and artificial societies.

Ideally, the types of expertise represented in the outer circles of Figure 1 will be included in the key processes related to the design and implementation of models, as well as the dissemination of their results. This makes sense not only because the SMEs, policy professionals, stakeholders, and change agents have the knowledge necessary to help the simulation engineers construct more realistic simulated agents and social networks but also because their participation helps to ensure that the problem and solution spaces are as relevant as possible for addressing real-world societal challenges. The human-simulation approach requires sensitivity and openness on the part of all collaborators, and this sort of experimentation with transdisciplinary teams continues to lead to new insights for participatory modeling [74,75].

#### **4. Ontological, Epistemological, and Ethical Reflections**

The human-simulation approach also takes seriously the ontological, epistemological, and ethical issues associated with the process of model development and deployment [69] (p. 11). Many sustainability researchers and M&S practitioners may find this sort of philosophical discussion unfamiliar or even irrelevant. However, we argue that, because human values and worldviews do in fact impact behavior in profound ways, it is worthwhile engaging in philosophical reflection on these issues as part of the process of applying the human-simulation approach to sustainability modeling and informing the discovery of more effective socio-ecological adaptive strategies for transitioning toward sustainable human societies in our current pluralistic and ecologically fragile environment.

One can think of a person's *ontology* as his or her inventory list of existing entities and relationships. Assumptions about what belongs on this list may be more or less implicit, but they shape one's understanding and anticipation of potential causal interactions in the world. Computer modeling forces us to make these assumptions explicit, that is, to articulate the ontologies of the artificial societies we construct [40]. Some scholars in sustainability studies have expressed concern about the relative lack of attention traditionally given in the field to ontology, a lacuna increasingly filled by growing attention to and application of social scientific theories such as "assemblage theory", which make explicit claims about what really exists [76–78]. Assemblage thinking has recently been applied, for example, to the analysis of policies related to promoting sustainability in urban and rural areas [79–81] and to the study of aquaculture and agriculture sustainability [82–84]. Some scholars have even applied the concept of assemblage to the process of sustainability policy formulation and implementation, using phrases such as "response assemblages" [76,85] or "adaptation assemblages" [86,87] to refer to the way in which humans attempt to engage and alter the socio-ecological systems in which they live.

M&S methodologies provide sustainability researchers with tools that encourage and enable a more robust articulation of the ontological (including causal) assumptions that are shaping their hypotheses and predictions. By forcing the breakdown of complex causal processes into component parts, human simulation naturally surfaces assumptions latent within politico-economic practices and renders them subject to more rigorous discussion and critique. Other modelers can substitute alternative assumptions, which could potentially lead to very different outcomes in complex social systems. The formalization of theories of sustainability in computational architectures provides what we might call *artificial ontologies* that can then be tested and validated in relation to empirical data in the real world. The credibility of these ontologies (including the causal relations implemented in the model) depends on the extent to which simulation experiments can generate or "grow" the relevant *macro*-level social "wholes" from the local interaction of *micro*-level "parts." We have argued elsewhere that the successful development of MAAI and other simulation technologies lends plausibility to a form of metaphysical *naturalism* involving what philosophers call "weak emergence" (for a fuller statement of this aspect of the argument, see [88]).

Insofar as social simulation through computer modeling can actually generate wholes (e.g., more or less sustainable artificial societies) with emergent properties, tendencies, and capacities that arise solely from the interaction of parts (e.g., physical resources, individual humans), it strengthens the claim that the mechanisms of morphogenesis that explain the causal forces at work in socio-ecological systems are wholly *naturalistic*. The possible implications of these developments for the articulation of a fully immanent metaphysics have been spelled out elsewhere [89,90]. Our point here is that if something goes wrong with the simulation, simulation engineers do not hypothesize a ghost in the machine; rather, they check the code, review the data, and run new simulations. Similarly, in sustainability research, scholars *qua* scholars do not include hypotheses about supernatural entities (such as sea gods, Jesus, or Allah) as potentially responsible for climate events such as hurricanes or tsunamis. In other words, sustainability scholars typically exhibit methodological naturalism: a preference for academic arguments that optimize the use of theories, hypotheses, methods, evidence, and interpretations that do not appeal to supernatural agents. Their causal explanations are only populated with "natural" entities or processes susceptible to empirical analysis such as geological forces, biological organisms, electricity, or climate change.

The success of M&S lends credence to such naturalistic explanations of emergent phenomena. Manuel DeLanda, one of the most well-known developers of "assemblage theory" as well as a computer modeler [91,92], argues that computer simulations "are partly responsible for the restoration of the legitimacy of the concept of emergence [in science and philosophy] because they can stage interactions between virtual entities from which properties, tendencies, and capacities *actually emerge*. Since this emergence is *reproducible* in many computers, it can be probed and studied by different scientists as if it were a laboratory phenomenon. In other words, *simulations* can play the role of laboratory experiments in the study of emergence complementing the role of mathematics in deciphering the structure of possibility spaces. Furthermore, *philosophy* can be the mechanism through which these insights can be synthesized into an emergent *materialist* world view that finally does justice to the creative powers of matter and energy" [92] (p. 6), emphases added.

For DeLanda, assemblage theory is not only explicitly tied to metaphysical naturalism [93] but also to epistemological insights linked to computer modeling and simulation [92]. We can think of a person's *epistemology* as his or her assumptions, again more or less implicit, about what counts as knowledge and how to acquire it. As is the case in all scientific disciplines in the contemporary academy, sustainability researchers and computer modelers are usually methodologically *secularistic* as well as methodologically naturalistic. By methodologically secularistic, we mean that they (as scholars) have a preference for academic practices that optimize the use of scholarly strategies that are not tied to the idiosyncratic interests of a religious coalition. This includes not making appeals to supernatural authorities to defend their knowledge claims. Most scientists assume (while doing science) not only that the ontological components and explanatory causes in their theories are naturalistic but also that the best way to acquire knowledge is through scientific methods that are not dependent on supernatural revelation or the religious authorities of any specific ingroup. While not all scientists live up to this ideal, the secular academy generally values arguments based on evidence that is accessible to any research group across cultures, and not only to those who believe in the supernatural revelation of a particular religious coalition. At least in part, this valuation is based on the fact that methodological secularism has funded the most productive and progressive research programs in science.

The processes by which M&S practitioners clarify, calibrate, verify, and validate the scientific knowledge, hypotheses, and theories formalized within their computational architectures through simulation experiments render their methodological secularism more explicit. What we might call the *artificial epistemologies* built into explanatory computer models are also tested and validated in relation to empirical data in the real world. Given the complexity of socio-ecological systems and the diversity of challenges related to achieving the UN Sustainability Development Goals (SDGs), we agree with the call by Sonetti et al. for a "transdisciplinary epistemology" that includes insights from the humanities as well as the natural and social sciences within sustainability research [94]. As we noted in Section 3, commitment to such an epistemology is at the heart of the human-simulation approach. However, Sonetti et al. also briefly allude to the papal encyclical *Laudato Si*, which they argue promotes an attitude of humility rather than domination and sets up "a sort of democracy of all God's creatures" [94] (p. 11). They do not point out that this encyclical is based on unfalsifiable assumptions about disembodied supernatural entities, knowledge of which can only be allegedly acquired and authorized by imaginatively engaging in the causally opaque rituals led by authorized officers of a particular religious ingroup. While allusions to supernatural authorities might feel inspiring to members of some religious coalitions, it is hard to see what epistemological relevance they have for the increasingly urgent task of finding efficient strategies for adapting to the Anthropocene and pursuing the SDGs here and now. At best, they can engender the support of religious-coalition members for sustainability initiatives articulated and tested within the scientific domain.

However, accommodating supernaturalist epistemologies within the scientific discourse about sustainability could be problematic for other reasons. Empirical research from a wide array of disciplines that contribute to the bio-cultural study of religion has shown that cognitive and coalitional biases related to supernatural worldviews *interfere* with sustainability movements, even when their proponents intend to help, by promoting superstitious beliefs and segregative behaviors that exacerbate rather than ameliorate the deleterious psychological and socio-economic conditions and can promote intellectual obstruction and moral paralysis in the face of globally relevant societal challenges such as climate change [95–103]. Supernatural beliefs and the ritual behaviors associated with them likely played a crucial role in helping humans survive in early ancestral environments, but today they have become maladaptive. Attempting to debunk claims about the existence or causal relevance of supernatural agents is not likely the best strategy here, not only because such claims cannot be definitively disproven (only rendered less plausible) but also because such attempts typically activate confirmation biases and other forms of motivated reasoning that only makes things worse. These are some of the all-too-human factors that impede change and need to be incorporated into computational simulations. As social psychologists and policy-oriented climate scientists are discovering, prebunking strategies that attempt to inoculate individuals against misinformation and superstitious reasoning are more likely to succeed [104].

Thinking about ontology and epistemology may be fun (for philosophers at least), but what ought we to *do* about all this? This brings us to the importance of ethical reflection before, during, and after the process of developing computer models and experiments in human simulation. We can describe a person's *ethics* as his or her assumptions, more or less explicitly articulated, about what (if any) moral dispositions or behaviors are normative or commendable. Sustainable adaptation to the challenges of the Anthropocene, as well as success in achieving the SDGs, will require the widescale emergence of norms and behaviors that promote global sustainability, many of which do not come naturally to most members of our species. We humans are not "rational actors" who calmly calculate utility functions. On the contrary, our moral reasoning is surreptitiously shaped by cognitive and coalitional biases that all too easily activate superstitious inferences, ingroup preferences, and other biases of the sort that promote resistance to naturalistic scientific explanations of (and secularistic policies for responding to) sustainability crises. This is why it is important to incorporate insights from the sciences of bio-cultural evolution about our phylogenetically inherited moral equipment into the ethical (and metaethical) frameworks that guide reflections around the design and implementation of, as well as the dissemination of results from, human-simulation modeling [105].

As the potential (and actual) impact of artificial intelligence (AI) on human life becomes increasingly clear, computer modelers, philosophers, and others are highlighting the importance of discussions about *artificial ethics*. The approach to human simulation described above is heavily invested in

these debates. In fact, the focus on MAAI modeling and other social simulation techniques can help bring a distinctive focus on artificial *social* ethics into the conversation [106]. By constructing and validating "digital twins" of real-world societies, populated by simulated agents and groups with divergent and changing norms, *multi-agent* AI can provide tools for testing hypotheses about the impact of policy proposals and environmental changes on human social behavior. In fact, whether or not one accounts for the actual divergent norms within the pluralistic cultures one is attempting to model has a significant effect on the outcomes of social simulation experiments [107]. This is why the human-simulation approach puts so much attention on minding morality, that is, on incorporating variables such as shared norms and divergent worldviews into the computational architectures of artificial societies [11,108].

Ethical reflection on the application of human simulation to sustainability research is tied to the issues related to naturalism and secularism discussed above. We illustrate this briefly with reference to Rumy Hasan's transdisciplinary analysis of *Religion and Development in the Global South* [109]. As Hasan notes, "adherence to religious doctrines is necessarily in tension with cognitive thinking ... Criticism, curiosity, critiquing, hypothesizing, theorizing, experimentation and the search for evidence all appear to be suppressed or discouraged" (p. 198). The capacities and skills of contemporary human populations are linked to the presence (or absence) of naturalistic education about the actual (non-supernatural) causal mechanisms at work in the world. Hasan concludes that high levels of religious beliefs in the Global South suppress the capacities and skills needed for sustainable development and exacerbate conflict, a claim that is widely supported in the relevant literature [110–113].

The extent to which our ontological, epistemological, and ethical reflections have explicitly highlighted the value of methodological naturalism and secularism in sustainability research, and explicitly criticized the obstructive effects of "religion," may come as a surprise. The goal here has not been to provide a comprehensive philosophical argument designed to compel scholars to take a more aggressive public posture against supernaturalist worldviews and the unsustainable parochial behaviors they promote, though it should be obvious that we think doing so might be a condition for finding and implementing more effective adaptation strategies in the Anthropocene. We have confined ourselves here to the more limited claim that there are at least some good reasons to include philosophical reflection about issues such as metaphysical naturalism and secularism as part of broader conversations among computer modelers, sustainability researchers, and the wider public.

#### **5. Conclusions**

We began this article by sketching some recent developments within the application of computer modeling to sustainability research, noting the pressing need to take better account of human factors. We went on to describe some of the ways in which the "human simulation" approach can contribute to the further advancement of sustainability modeling, especially through its emphasis on stakeholder, policy professional, and subject matter expert participation and its focus on constructing more realistic cognitive architectures, both of which help to facilitate the incorporation of human factors (such as values and worldviews) into the modeling process. Finally, we offered some philosophical reflections on the ontological, epistemological, and ethical issues raised at the intersection of sustainability research and computer simulation. We stressed the importance of having more explicit conversations about the relevance of metaphysical naturalism and secularism in finding and facilitating effective and efficient strategies for sustainable development.

Like all methodologies, computational tools and techniques have their limitations. However, we have attempted to show some of the reasons for hoping that the application of human simulation within sustainability research will continue to be increasingly useful for the purpose of analyzing and forecasting changes in socio-ecological systems as we attempt to respond to the challenges of the Anthropocene.

**Author Contributions:** Conceptualization, F.L.S. and W.J.W.; original draft preparation, F.L.S.; review and editing, W.J.W. and F.L.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

113. Bormann, N.-C.; Cederman, L.-E.; Vogt, M. Language, Religion, and Ethnic Civil War. *J. Confl. Resolut.* **2017**, *61*, 744–771. [CrossRef]

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **A Usability Study of Classical Mechanics Education Based on Hybrid Modeling: Implications for Sustainability in Learning**

**Rosanna E. Guadagno 1,\*, Virgilio Gonzenbach 2, Haley Puddy 3, Paul Fishwick 4, Midori Kitagawa 5, Mary Urquhart 5, Michael Kesden 5, Ken Suura 5, Baily Hale 5, Cenk Koknar 4, Ngoc Tran 6, Rong Jin <sup>7</sup> and Aniket Raj <sup>8</sup>**

**Abstract:** A usability study evaluated the ease with which users interacted with an author-designed modeling and simulation program called STEPP (Scaffolded Training Environment for Physics Programming). STEPP is a series of educational modules for introductory algebra-based physics classes that allow students to model the motion of an object using Finite State Machines (FSMs). STEPP was designed to teach students to decompose physical systems into a few key variables such as time, position, and velocity and then encourages them to use these variables to define states (such as running a marathon) and transitions between these states (such as crossing the finish line). We report the results of a usability study on high school physics teachers that was part of a summer training institute. To examine this, 8 high school physics teachers (6 women, 2 men) were taught how to use our simulation software. Data from qualitative and quantitative measures revealed that our tool generally exceeded teacher's expectations across questions assessing: (1) User Experience, (2) STEM-C Relevance, and (3) Classroom Applicability. Implications of this research for STEM education and the use of modeling and simulation to enhance sustainability in learning will be discussed.

**Keywords:** modeling; hybrid modeling; hybrid simulation; usability; sustainability; high school education; physics education; user experience

#### **1. Introduction**

The United States is a global leader in research and development across the sciences. Unfortunately, this position is threated by an increasing shortage of qualified workers. Indeed, recent data demonstrate this trend—a 2015 study reported that the US ranks low compared to other countries in terms of high school students' science, technology, engineering, and math (STEM) proficiency [1]. To help address this gap, the authors of the present manuscript obtained federal funding to develop a Scaffolded Training Environment for Physics Programming (STEPP) environment for use in high school physics classes. STEPP was designed to be an enjoyable and easy to use modeling and simulation program

139

**Citation:** Guadagno, R.E.; Gonzenbach, V.; Puddy, H.; Fishwick, P.; Kitagawa, M.; Urquhart, M.; Kesden, M.; Suura, K.; Hale, B.; Koknar, C.; et al. A Usability Study of Classical Mechanics Education Based on Hybrid Modeling: Implications for Sustainability in Learning. *Sustainability* **2021**, *13*, 11225. https://doi.org/10.3390/su132011225

Academic Editors: Philippe J. Giabbanelli, Arika Ligmann-Zielinska and Jordi Colomer Feliu

Received: 9 June 2021 Accepted: 28 September 2021 Published: 12 October 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

that would increase student's physics knowledge, computational thinking, and interest in pursuing a career in STEM. The present paper reports the results of a usability study conducted on a sample of high school physics teachers as a means to understand whether the people likely to use STEPP in their classrooms see STEPP as beneficial to their teaching and likely to meet the aforementioned goals. Please see Supplementary Materials for more details on STEPP.

#### *1.1. Physics Teaching Challenges*

Students often have misconceptions of physics [2], and these misconceptions are difficult to correct [3]. Although introductory college physics courses are prerequisites for students to enter many STEM fields, college physics professors are often dissatisfied with the preparation that students have received in high school [4]. The performance of U.S. 12th grade physics students was below the international average and among the lowest of the 16 nations that administered this same physics assessment to a comparable population of their students [5]. High school computer science education is in worse condition. Despite the importance of computing in American society, the positive impact this industry has on our economy, and the focus of national, state, and local policy makers on STEM education in the U.S., computer science education in secondary schools has not been growing. Only 47% of all high schools in the United States teach computer science [6]. Among the STEM fields, computer science is the sole subject in the 21st century United States that has actually shown a decrease in the percentage of high school graduates earning credits: down from 25% in 2000 to 19% in 2009 [6]. Furthermore, between 2005 and 2013, the number of secondary schools that offered introductory computer science courses decreased by 4% while the number of secondary schools that offered the Advanced Placement (AP) Computer Science course increased by 6% [7]. In 2011, only 5% of the high schools in the US were certified to teach the AP Computer Science course and only 22,176 students took the AP Computer Science exam nationally [8], while nearly 20 times that number of students took the Calculus AP exams and 6 times that number of students took the Physics AP exams [9]. This concerning trend in computer science education in secondary school is attributed to the certification process and lack of adequate computer science background for computer science teachers across the nation, which is considered confounding [10].

#### *1.2. Overview of STEPP*

The overarching goal of our NSF-funded STEM + Computing research project is to develop a synergistic scaffolded learning environment in which students learn physics and computational thinking by creating dynamic representations for physics concepts using a 3D video game engine. This paper reports the results of our first usability test of our modules with high school physics teachers.

The modeling methodology used within the implementation of STEPP falls within the domain of hybrid modeling and simulation [11–13]. The hybrid nature of STEPP is in a model that is executed at two levels: the high discrete-event, finite state machine model, and the lower continuous layer beneath each state. States capture the time-based qualitative behavior, and events are Boolean expressions of equational variables for kinematics. Our research plan aligns thematically with previous experimental studies in the simulation community [14]. Rechowicz et al. [15] employ a survey research methodology. The importance of having a human-subject empirical study within model-based systems is emphasized by Giabbanelli et al. [16].

Students often perceive physics as a subject that is too abstract to understand. This is because physics provides knowledge about matter and motion abstracted from our physical world and expressed as mathematical equations [17]. By encoding mathematical equations in a language that computers can process to produce real-time graphical displays, students can turn abstract knowledge back into observable phenomena. Similarly, the game engine provides interactivity and realistic graphics that break down the 3D environment into 1D and 2D environments in a scaffolded approach and incentivizes students to relate physics concepts to real-world phenomena. Scaffolding will allow students to focus on those aspects of dynamic modeling that are reflexive with physics learning.

Modeling, based on the Finite State Machine (FSM) [18], comes from theoretical computer science as well as state charts used in the Unified Modeling Language (UML) [19]. Both the FSM mathematical structure and the software engineering-based UML serve to emphasize conceptual computer science. Our use of modeling, rather than programming in a language such as Python, is hypothesized to teach the student computational [20] and systems [21] thinking rather than lower-level semantics associated with coding (e.g., text-based programming).

To address these issues, we have developed a Scaffolded Training Environment for Physics Programming (STEPP; see Figure 1), and our research team has partnered with local high school teachers to test three scaffolded tutorial modules that consist of instructions, tutorials, and sample programs that can be incorporated in existing high school physics courses. We maintain that the scaffolded nature of STEPP combined with the method of multiple representations, including color-coded dynamic motions and iterative actions (within modules and between modules) that are in real-time with the simulation, creates a context for sustainable learning.

**Figure 1.** Screenshot of STEPP Module 1, Level 4 focused on 1D acceleration as used at the June 2019 summer institute for high school physics teachers.

Our simulation is based on introductory-level classical mechanics in which student users are presented with a graphical simulation containing hybrid discrete-event and continuous models. Users solve specific physics problems that rely on a combination of computational thinking through hybrid modeling, knowledge of Newton's laws of motion, and algebraic problem solving. Our expectation is that when fully deployed, STEPP will increase students' physics knowledge, computational thinking, and interest in a STEM career. Computational thinking is operationally defined as "In the form of a taxonomy of four practices—data practices, modeling and simulation practices, computational problem-solving practices, and systems thinking practices—focusing on the application of computational thinking to mathematics and science" [22]. This important conceptual skill provides people in STEM fields the ability to think like computer scientists [23].

While much of the literature reviewed above pertains to student learning, there is another important group of users that interact with physics learning in US high schools—the teachers. Thus, it was important to assess the usability of STEPP among teachers as well as students. As a result, in June 2019, the STEPP project offered a summer institute at which in-service and pre-service physics teachers would have the first opportunity to learn

and experience STEPP and assist in its further development for use in their classrooms. See Figure 1 for an example screenshot from our modules.

#### *1.3. Sustainability in Learning through Modeling and Simulation*

What does STEPP and the teaching of introductory physics have to do with sustainability? Sustainability by definition is based on abstract idea of a "system" [24]. System thinking broadens computational thinking to a transferable skill in which automata theory can be applied across disciplines. Other work discusses this need to bridge computational and systems thinking as a path toward sustainability [25].

A system is sustainable if it can maintain its present state and thus avoid a potentially undesirable transition. Although usually associated with ecosystems and the environment, we argue that even elementary mechanics can embody these essential qualities of sustainability. Consider Newton's first law of motion: an object at rest will stay at rest, and an object in motion will stay in motion, unless acted on by an external force.

STEPP is designed to teach the generalizable skills of decomposition and state-based modeling, both of which are critical for building simulations of sustainable systems in a wide variety of contexts. Through a scaffolded instructional environment focused on sustained learning of physics through computer science structures, students are supported in their development of understanding in an accessible environment that contributes to their learning in both disciplines. The sustainable learning goals of STEPP are envisioned to be achieved through the empowerment of users to develop the skills and conceptual understanding necessary for broader self-directed learning and transferability to future applications. The hybrid architecture employed by STEPP in which systems are described by both higher- and lower-level states is described in greater detail in Section 2.3 below. By using STEPP to model the motion of objects with FSMs, we expect that students will build skills in decomposition and state-based modeling that can later be transferred to the modeling and simulation of a variety of sustainable systems.

Overall, the evidence suggests that the current American educational system has not been successful in providing sufficient opportunities for high school students to learn physics or computer science. In recent years, multi-sensory experiential learning methods have begun to be incorporated into educational and school programs. Rather than having students learn only through traditional lecture-based methods, teachers are beginning to incorporate simulations, computer games, and more educational software items into their lesson plans. Computer simulations, games, and models can present difficult topics to students that may not be presentable in a traditional lecture-based format [26]. Research has also shown that combining traditional instruction with computer simulations on science teaching and learning is beneficial in promoting and developing students' content knowledge, science process skills, and coping skills with more complex tasks. For instance, Smetana and Bell [27] argue that computer simulations are successful in promoting learning because they allow students greater flexibility in exploring ideas and encourage them to justify their actions and answers in a timely manner. Review of the literature on modeling and simulation software and long-term learning outcomes suggests that simulation games in engineering education can promote the transferability of academic knowledge to industry. In other words, when an engineering simulation game is applied correctly, students will be more likely to sustain that knowledge and use it in their future in the industry [26]. These prior results suggest that the use of STEPP in the classroom has the potential to facilitate sustainable learning of physics and computational thinking—a supposition we tested after the present usability study was conducted.

#### *1.4. Usability Testing and Educational Software*

Part of the software development process involves the usability testing of early versions of the software with a small sample of target users. Developed in the 1980s, the primary function of usability testing is to examine the extent to which a product—in this case, software designed to increase physics learning and computational thinking—is easy

to use by the targeted users [28]. To assess usability, researchers observe the user experience (i.e., observe users sampled from the target audience as they learn to use the software) to determine the extent to which a software product can be used in the manner anticipated by the designers. This typically involves assessing the extent to which users find the product easy to use, whether users see the software as effective in facilitating the intended goals (i.e., in the case of educational software, whether it meets the learning goals), and identifying any design issues and/or bugs that interfere with usability of the program [29].

While there are many different ways to assess usability, the present research utilized the user-based method, a theoretical perspective on usability testing in which users' experiences using a software program are assessed via survey [28,30]. This theoretical perspective on usability testing typically assess the extent to which users' find a software program easy to use by asking participants to use the software and report on their experiences so that the researchers can identify problems users have with the software. As part of this process, user's actions and their overall satisfaction with the program are measured via survey. Because educational software varies by intended audience, the skills taught, and the overall subject matter, it is typical for usability studies to tailor the items they assess to the specific features of the software being tested.

Prior research on the usability of physics educational software has demonstrated the utility of usability testing and shown that these programs, when properly designed (i.e., are easy to use, intuitive, and focused on teaching specific concepts) can facilitate student learning [29,31]. While the majority of published usability studies have only assessed student responses after using educational software, there is limited evidence establishing the importance of assessing teacher feedback on educational software. The limited existing scholarship suggests that obtaining teacher buy in when introducing educational software in the classroom [32] as teachers have the final say in how such programs are utilized by their students [33].

To examine the usability of STEPP, we surveyed high school physics teachers who participated in a multi-day STEPP training workshop on their experiences using the software. This study also fills a gap in the literature in that usability studies in educational software largely examine students' perceptions, while neglecting the user experiences of teachers. The present study is one of the few studies to assess usability from the perspective of teachers—an often-overlooked group that is also important to study, as this group of users often selects the educational software that their students will use. In the case of STEPP, teachers were also expected to utilize the modules in class as part of their lectures. As a result, it is important to solicit feedback from this group of users regarding STEPP's ease of use, applicability to increasing student interest in STEM + Computing (STEM+C), and usefulness in the classroom.

#### *1.5. Overview of the Present Study*

The present study describes the results of our usability survey to determine whether the high school physics teachers attending the summer institute felt that the software was easy and enjoyable to use, would facilitate student physics and STEM-C learning, and would be useful in a high school physics classroom. These conceptual categories were selected based on the literature reviewed above on usability testing in educational software and the specific types of usability feedback needed during the software development process. Based on the prior research on usability testing of educational software reviewed above, we predicted that, relative to expectations, our participants would evaluate the modules more favorably across these measures after exposure to STEPP. We also sought feedback on any design issues that could hinder student ease of use, learning, or overall interest in potential STEM-C careers.

#### **2. Materials and Methods**

#### *2.1. Participants and Design*

A total of 12 high school physics teachers participated in an NSF-funded 4-day training institute held by the authors at a large southwestern university. The research team advertised this training institute to qualified physics high school teachers throughout the university's greater metropolitan area. Our usability study was incorporated into the planning of the institute, with teachers completing the time 1 survey at the beginning of the institute and time 2 at the end. Thus, the experimental design was a pre-posttest such that participants were asked to report their expectations for using STEPP, and then were asked to report on their actual experiences using STEPP.

Unfortunately, data from 4 participants were excluded owing to their failure to complete the pre- or post-test survey, reflecting a 33% attrition rate (i.e., 67% of the sample filled out both surveys) leaving us with a final sample of 8 teachers. The results below reflect input from the 8 teachers who completed both the time 1 and time 2 surveys.

The final sample consisted of 8 participants 8 (6 women, 2 men). They ranged in age from 21 to 47 (M = 33.8, SD = 7.8). Ethnicity was reported as follows: 4 (50%) White/Caucasian, 2 (25%) Asian, 1 (12.5%) Hispanic, and 1 (12.5%) Biracial/Multiracial. Their amount of time teaching Physics ranged as follows: 1 (12.5%) indicated 1–2 years, 2 (25%) indicated 3–5 years, 2 (25%) indicated 6–10 years, and 3 (37.5%) indicated 11–15 years.

The experimental design was a within subject design in which participants provided feedback on their expectations for using STEPP (time 1) and were once again sampled at the end of the institute (time 2).

#### *2.2. Procedure*

Participants were invited to apply for the institute via a flyer that described an opportunity to be on the cutting edge of teaching by learning the STEPP modules, which combined the power of Computer Science with Physics curriculum through a game engine. STEPP was described as a tool that could teach their students the basics of Newtonian mechanics through an interactive environment to build simulations. Only physics teachers who accurately followed our application instructions and were in the same metropolitan area as the university were selected for participation. This approach provided an educational opportunity to local teachers and promoted recruitment of these teachers for future field testing of our modules.

The research team received IRB approval to conduct a usability study as part of the teaching institute. Teachers enrolled in the institute received information on the usability study, and those willing to participate provided their informed consent. Consenting participants voluntarily completed the pre-test (time 1) questionnaire on the first day of the institute and the post-test (time 2) on the final day. In between the two assessments, participants learned to use the STEPP prototype modules. The institute was held for 7 h a day across the 4-day period. During this time, participants used STEPP for roughly 12 h.

#### *2.3. Software*

STEPP was designed to provide users with the opportunity to decompose word problems typically found in the teaching of introductory mechanics applicable to its three modules. First, a user selects a physics problem, such a standard word problem found in a textbook on the curriculum covered by the STEPP modules and levels. The motion of the object described in the problem is then decomposed into discrete states. The user then creates FSM states in STEPP corresponding to each state of motion and programs the transitions which cause the FSM to switch between these states. Inputs for the states and transitions vary by module and level. In module 1, level 1, users engage with a simple interface based on natural language. Subsequent levels and modules increase in complexity. User-programmed inputs include variables such as position, displacement, velocity, acceleration, and mass, with transitions defined by a user-determined final position, final time, or final velocity for each state. Incorrect input can create simulations

that fail to describe the desired motion, however, each simulation must be physically selfconsistent in order to run. Motion in a new state must begin at the same time and position where a prior state ended, so the initial starting values of a new state are automatically created from the final values of the previous state. States can only be deleted in the reverse order in which they were created to prevent the introduction of unphysical errors in the programming of the simulation. If the states or transitions as programmed do not model a self-consistent physical system, the STEPP simulation will not run and an error message is returned to the user with information on how to correct the issue. Thus, students are provided a scaffolded environment in which they can safely take a trial-and-error approach to problem solving. By using scaffolded FSMs to program representations of word problems encountered in the context of the physics classroom, students and teachers can engage in creation of their own simulations of motion without risk of either software failure or the accidental modeling of unphysical situations.

The STEPP environment includes sub-windows and graphic icons and diagrams. The diagram at the bottom of the main window is the Finite State Diagram (i.e., Machine) window which controls the physical motion. Each discrete higher-level state of the FSM is a partition of a lower-level state space characterized by continuous values of time, displacement, velocity, and acceleration. This architecture is hybrid in that continuous (e.g., lower-level state space) and discrete (higher-level state space) are combined and integrated. An example task would be a ball moving up and then down. When the ball moves up, this can be considered one higher-level discrete state labeled "up". Within "up", there are lower-level partitions of the continuous space corresponding to motion that matches the quantitative definition of "up", i.e., positive vertical motion. Similar reasoning can be applied for the state "down." See Figure 2 for screenshots of the final product, informed by the results of the present study, which capture the STEPP modules' scaffolding as well as its multiple representations, color-coded actions, and animations.

**Figure 2.** Screenshots of the current version of STEPP depicting (**A**) the STEPP Module-selection menu; (**B**) the STEPP level-selection menu within Module 1; and (**C**) STEPP Module 1, Level 4 focused on 1D acceleration as improved by the results of this usability study of high school physics teachers.

For an example of a concrete task solved via STEPP, we turn to "Exercise 2.43 Launch Failure" in a typical introductory physics textbook [34]:

"A 7500-kg rocket blasts off vertically from the launch pad with a constant upward acceleration of 2.25 m/s2. and feels no appreciable air resistance. When it has reached a height of 525 m, its engines suddenly fail; the only force acting on it now is gravity.

We can solve this problem by modeling it as a hybrid-level finite-state machine within STEPP. We can describe the evolution of this physical system be decomposing it into three high-level states: (1) the rocket rises with constant upward acceleration before its engines fail, (2) the rocket continues to ascend until it comes to rest at the apex of its flight, and (3) the rocket falls back to Earth and crashes on the launch pad. Each of these discrete high-level states can be further partitioned into the low-level variables of time, position, velocity, and acceleration that evolve continuously between the transitions at the start and end of each high-level state.

State 1 "Powered ascent" is characterized by a constant upward acceleration of 2.25 m/s2. It begins with the rocket launch at *t*1i = 0 s, y1i = 0 m, v1i = 0 m/s, where the number of the subscript corresponds to the high-level state and the letter ("i" or "f") corresponds to the initial or final value at the transition that begins/ends the high-level state. State 1 ends at the transition corresponding to engine failure (at a height of y1f = 525 m). State 2 "Unpowered ascent" is characterized by an upward (positive) velocity and gravitational acceleration of g = −9.8 m/s2. It begins with the same values of the low-level variables as state 1 ends (no "time travel" -> time is continuous, no "teleportation" -> y is continuous, conservation of linear momentum -> v is continuous). State 2 ends at the transition corresponding to rest at the apex of the flight (v2f = 0 m/s). State 3 "Descent" is characterized by a downward (negative) velocity and gravitational acceleration of g = −9.8 m/s2. It begins at the end of state 2 and ends when the rocket crashes on the launch pad (x3f = 0 m).

Once the student has decomposed the problem into these three high-level states and determined the values of the low-level variables that describe the transitions, they can program STEPP to produce a simulation of the rocket's flight, with an animation and real-time graphing providing feedback in multiple representations. STEPP also provides warnings if the student programs unphysical transitions. For example, if the student had forgotten that the gravitational acceleration was negative and had incorrectly inputted g = +9.8 m/s<sup>2</sup> for state 2, the rocket would accelerate upwards and never come to rest. In this case, the transition v2f = 0 m/s could not be realized and the simulation would never end. The current version of STEPP provides the error message "You'll never reach that velocity. The acceleration and change in velocity do not agree." This message is intended to guide the student to correct their mistake without becoming discouraged. Figure 3 shows a screen capture from the end of the STEPP simulation. States 1, 2, and 3 are shown in blue, orange, and green, respectively, with state 1 opened to show the low-level variables describing its initial and final transitions. By opening the other states or reading from directly from the graphs, the student can solve the problem: (a) the rocket reaches a maximum height of 646 m at the end of state 2/beginning of state 3 (orange/green peak in top graph), (b) the rocket comes crashing down to the launch pad t3f–t1f = 16.4 s after engine failure at a final velocity of −113 m/s (right end of the green line segment in middle graph), and (c) STEPP has conveniently provided us with the desired graphs.

**Figure 3.** Screenshot of the end of the simulation describing the rocket flight in "Exercise 2.43 Launch Failure" [34]. The animation shows the rocket's flight, depicted in STEPP as the Apollo lunar module rising and falling from the Moon's surface. The state diagram shows the three high-level states: state 1 "Powered ascent", state 2 "Unpowered ascent", and state 3 "Descent" (shown in blue, orange, and green, respectively). The graphs show how the low-level variables (position, velocity, and acceleration) evolve with time, color-coded by the high-level state. The open panel shows the acceleration and initial and final time, position, and velocity of state 1 "Powered ascent".

#### *2.4. Measures*

The authors developed a short survey with 13 continuous items assessing participants attitudes toward STEPP modules across three different conceptual categories assessed used in usability studies and relevant to our specific software tool: User Experience (e.g., "I like using STEPP"), STEM-C relevance (e.g., "STEPP is useful for teaching Physics"), and Classroom Applicability (e.g., STEPP is valuable for students learning physics") on Likert scales ranging from 1 = "strongly disagree" to 7 = "strongly agree" with 4 as the midpoint labeled "neutral.". See Figure 4 for abbreviated descriptions of each item and Table 1 below for a copy of the full scale along with the observed correlations between time 1 and time 2.

To collect more nuanced feedback on our software, we also asked participants to provide written responses to several open-ended questions that asked participants to provide more detail regarding their impressions of STEPP. These items allowed participants to provide detailed feedback on what they liked and did not like about STEPP, and what they found easy to use and thought could help them teach physics. At time one, these items assessed participants expectations about using STEPP and were phrased to indicate that we were interested in understanding their expectations. At time two, these items assessed participants' actual experiences after using STEPP throughout the teacher training.

Data were collected via Qualtrics survey software. At time 1, we asked a total of 13 quantitative questions and 5 opened-ended questions. At time 2, we re-assessed the same 13 quantitative items and added 2 additional open-ended responses to gather additional feedback on recommended changes after participants had used STEPP.

**Table 1.** The Quantitative items used to assess teacher's usability experience with STEPP along with the correlation between time 1 and time 2 for each pair of questions. Time 1 wording appears in brackets.

Notes: all items were assessed on a continuous scale ranging from 1 ("strongly disagree") to 7 ("strongly agree") with a midpoint of 4 ("neutral"). Phrases above in brackets provide the wording for the pre-test survey. Statistically significant correlations are denoted as follows: \* *p* < 0.05; \*\* *p* < 0.01; \*\*\* *p* < 0.001.

**Figure 4.** Mean evaluations at pre- and post-test on the quantitative items.

#### **3. Results**

We analyzed changes in attitude in three conceptual topics: (1) User Experience, (2) STEM-C Relevance, and (3) Classroom Applicability. To compare changes in teachers' attitude before and after using STEPP, we used within subjects' *t*-tests and report the means, standard deviations, mean difference, correlations between time 1 and time 2 measures, *t*-test statistic, Cohen's d effect size, *p*-value, and observed statistical power for each analysis reported below. See Figure 3 for a graphical display of pre-post means for all measures. Note that while each item was analyzed individually but the reliability for the combined items for each the conceptual topic at both Time 1 and Time 2 are reported below. The correlations between pre- and post-test items is reported in Table 1 above.

An anonymous reviewer asked us to reanalyze the data using a non-parametic equivalent of the within subjects' *t*-test, the Wilcoxon Signed Rank Test. We re-ran the analyses and did not find our results changed for any of the items. Also note that the inclusions of scale reliabilityies for the three conceptual categories was based on similar reviewer feedback.

#### *3.1. User Experience (Time 1 Chronbach's α = 0.88; Time 2 Chronbach's α = 0.57)*

After using STEPP, teachers found it easier to use than expected, (Time 1 M = 4.5, SD = 0.33; Time 2 M = 5.75, SD = 0.25, D = −1.25), *t*(7) = 5, d = 1.71, *p* = 0.002, Obs 1−β = 0.56. Teachers also liked the look of the modules more than they expected, (Time 1 M = 5.125, SD = 1.61; Time 2 M = 6, SD = 0.33; D = −0.875), *t*(7) = 2.83, d = 1.55, *p* = 0.026, Obs 1−β = 0.91. STEPP exceeded teacher's pre-test expectations on how much they would like using the modules, (Time 1 M = 4.875, SD = 0.96; Time 2 M = 6.125, SD = 0.17; D = −1.25), *t*(7) = 3.03, d = 1.5, *p* = 0.019, Obs 1−β = 0.85. There were no significant differences, however, between the degree to which teachers reported wanting to use STEPP and to their expectations of STEPP, (Time1M= 5.125, SD = 1.61; Time 2 M = 6, SD = 0.73; D = −0.875), *t*(7) = 2.2, d = 0.84, *p* = 0.064, Obs 1−β = 0.59. Teachers became confident in using the STEPP modules more than they expected, (Time1M= 4.75, SD = 0.62; Time2M= 5.875, SD = 0.49; D = −1.125), *t*(7) = 4.97, d = 1.3, *p* = 0.002, Obs 1−β = 0.28.

#### *3.2. STEM-C Relevance (Time 1 Chronbach's α = 0.79; Time 2 Chronbach's α = 0.77)*

After use, teachers judged that STEPP would increase interest in Physics more than they expected, (Time1M= 4.625, SD = 1.27; Time2M= 5.625, SD = 2.88; D = −1), *t*(7) = 2.65, d = 0.83, *p* = 0.033, Obs 1−β = 0.44. Teachers' judgement of how much STEPP would increase interest in Computer Science also exceeded their expectation prior to use, (Time 1 M = 5.125, SD = 1.61; Time 2 M = 6.125, SD = 0.48; D = −1), *t*(7) = 2.65, d = 0.99, *p* = 0.033, Obs 1−β = 0.58. Teachers did not differ in how much they believed STEPP could increase interest in STEM fields before and after using the modules, (Time 1 M = 5.125, SD = 1.61; Time2M= 5.5, SD = 2.04; D = −0.375), *t*(7) = 0.81, d = 0.32, *p* = 0.442, Obs 1−β = 0.93. Teachers did, however, report a strong change in their appreciation for STEPP as a tool for teaching Physics, (Time 1 M = 4.375, SD = 1.99; Time 2 M = 6.375, SD = 0.07; D = −2), *t*(7) = 7.48, d = 1.18, *p* < 0.001, Obs 1−β = 0.14.

#### *3.3. Classroom Applicability (Time 1 Chronbach's α = 0.83; Time 2 Chronbach's α = 0.75)*

After use, teachers reported wanting to use STEPP in the classroom at a higher rate after use, (Time 1 M = 5, SD = 1.31; Time 2 M = 6.125, SD = 0.17; D = −1.125), *t*(7) = 4.97, d = 1.01, *p* = 0.002, Obs 1−β = 0.13. Teachers evaluated STEPP to have a higher value to students after having used the program themselves, (Time1M= 4.875, SD = 0.48; Time 2 M = 6, SD = 0.33;D= −1.125), *t*(7) = 9, d = 1.38, *p* < 0.001, Obs 1−β = 0.23. Concept Maps [27] were used in the summer institute/curriculum design as a bridge from a more common classroom tool to the unfamiliar State Diagrams. Teachers did not find the Concept Maps in STEPP to be useful in breaking down physics more after using STEPP, (Time 1 M = 4.875, SD = 4.51; Time 2 M = 5.375, SD = 8.06; D = −0.5), *t*(7) = 1.08, d = 0.31, *p* = 0.316, Obs 1−β = 0.46. However, there was a large difference between how useful teachers found the State Diagrams in STEPP after using the modules relative to before

use, (Time 1 M = 5, SD = 0.33; Time 2 M = 6.25, SD = 0.25; D = −1.25), *t*(7) = 3.42, d = 1.71, *p* = 0.011, Obs 1−β = 0.87.

#### *3.4. Qualitative Feedback*

Participants generally reported that they expected STEPP to be more coding-focused and difficult to use but found instead that it was more physics focused instead. As one participant reported: "I initially envisioned more of a direct coding component, but this was not present in STEPP as STEPP instead introduced CS conceptually through state diagrams".

In terms of what participants liked most about STEPP, their responses uniformly focused on the ability of STEPP to break problems down into discrete, logical steps. Participants unanimously reported minor bugs as the issue they liked least about STEPP. Similarly, they all focused on how the ability to break problems down into discrete steps would be the biggest benefit of using STEPP to teach physics in their classrooms. When asked about how to make STEPP easier to use, participants focused on more detailed instructions. For instance, one participant stated: "Provide more user guidance (maybe through a tutorial at first use?)" When asked about STEPP's usefulness in teaching computational thinking to students, the teachers also focused on the discrete steps approach to problem solving as the modules' best contributing factor.

For the post-test only items, participants suggested including more instructions in STEPP. The final question asked for any additional input and their feedback split into two equal categories: an expression of enthusiasm for using it in their classrooms and a desire for more teaching resources to help support them as they implement the modules into their classes. As one participant expressed: "Keep on keepin' on! This is promising software".

#### **4. Discussion**

Overall, our results supported our predictions in that teachers' expectations for STEPP were exceeded after using it. Furthermore, their overall feedback suggests that these software modules are easy and enjoyable to use, can facilitate interest in and learning of physics and STEM-C, and teachers see the utility of these modules for use in the classroom. This was illustrated by both the quantitative and qualitative responses from our sample of teachers. The qualitative items in particular also captured important user feedback and suggestions which may be capitalized upon in the further development of the STEPP modules. Thus, the user feedback we received was not only consistent with our predictions, but also helpful for the future development of STEPP, which may include bugs fixes and adding a tutorial mode.

These results further support the notion that, as with other types of educational physics software [27,29,30,35], STEPP has the potential to facilitate student understanding of modeling, simulation, and the transitions between different states. These concepts are also useful for understanding the impact of sustainability in physics education. Furthermore, this study adds to the growing literature on the usability of physics educational software by assessing usability from the perspective of teachers—an often overlooked group. This is also important because of teachers' role in selecting the educational software used by both students and teachers inside and outside the classroom. Thus, our results also highlight the importance of the teacher's perspective on educational software in the classroom.

While the research team found the results of the present study promising, this research is not without its limitations. For instance, while we assessed usability and perceived relevance to STEM-C, we did not assess learning. This is because (a) it would make little sense to assess the sample of physics teachers in this manner in the context of a 4-day summer institute; (b) we planned to test student learning after completing our usability studies. Furthermore, the present study has a small sample size—an issue typically problematic in research. However, this is not unusual for usability research where small samples are the norm. Illustrative of this, research indicates that 80% of usability problems are detected with samples as small as 4–5 participants [36]. We intentionally designed a within-subjects usability test as this type of design is also known to accommodate small sample sizes [37]. Based on the generally large effects [38] and high statistical power observed in our results, we maintain that our sample size was adequate for our purposes.

It is notable that we did not conduct multivariate analyses because some of the scaled reliabilities decreased at time 2. This may reflect the notion that the teacher's expectations were more conceptually similar while their hands-on experience with STEPP produced more variability in their post-test responses. Finally, we need to acknowledge the limited generalizability of this research, as these results are specific to our specific software and may not be applicable to other types of users and other types of educational software.

In the future, our research will examine the usability of our software from the perspective of students as well. However, this initial test provides some evidence that STEPP may be an easy tool to increase student interest in and commitment to STEM careers.

#### **5. Conclusions**

Overall, this research addresses a gap in the literature by examining usability from the perspective of teachers. Furthermore, this work suggests that simulations may have a promising role in the future of physics education, modeling, simulation, computational thinking, and the sustainability of systems. While people may generally think of sustainability as referring to some ecological result, such as zero energy or agriculture. Thus, the present research applies sustainability to education by considering the role that educational software plays in long-term learning outcomes. We posit that multiple representations and scaffolding are two educational principles which lead to a more diverse and robust system (STEPP). Having these two principles in action may make STEPP more accessible with broad impact.

#### **Supplementary Materials:** More details on STEPP are available online at https://stepp.utdallas.edu.

**Author Contributions:** Conceptualization, R.E.G., V.G., P.F., M.K. (Midori Kitagawa), M.U. and M.K. (Michael Kesden); methodology, R.E.G. and V.G.; software, P.F., M.K. (Midori Kitagawa), M.K. (Michael Kesden), C.K., N.T., R.J. and A.R.; data collection, R.E.G., V.G., M.U., K.S. and B.H.; data analyses, R.E.G. and V.G.; project administration, M.K. (Midori Kitagawa), K.S. and C.K.; writing—original draft preparation, R.E.G., V.G., H.P., P.F., M.U., M.K. (Midori Kitagawa) and M.K. (Michael Kesden); writing—review and editing, H.P., M.K. (Midori Kitagawa), P.F., M.U. and M.K. (Michael Kesden); funding acquisition, M.K. (Midori Kitagawa), P.F., M.K. (Michael Kesden), M.U. and R.E.G. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the National Science Foundation grant number DRL-1741756.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Institutional Review Board of the University of Texas at Dallas (protocol # 19MR0153, approved 19 June 2019).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data reported is this manuscript will be made available on our website.

**Acknowledgments:** The authors wish to thank the high school teachers who participated in this study and the many students who helped build this software.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel. +41 61 683 77 34 Fax +41 61 302 89 18 www.mdpi.com

*Sustainability* Editorial Office E-mail<EMAIL_ADDRESS>www.mdpi.com/journal/sustainability

MDPI St. Alban-Anlage 66 4052 Basel Switzerland

Tel: +41 61 683 77 34 Fax: +41 61 302 89 18

www.mdpi.com

ISBN 978-3-0365-2809-0