# Sustainability in Computing Education: A Systematic Literature Review

Anne-Kathrin Peters<EMAIL_ADDRESS>KTH Royal Institute of TechnologySweden ,
Rafael Capilla<EMAIL_ADDRESS>Rey Juan Carlos UniversitySpain ,  Vlad
Constantin Coroamă<EMAIL_ADDRESS>Technische Universität BerlinGermany
,  Rogardt Heldal<EMAIL_ADDRESS>Western Norway University of Applied
SciencesNorway ,  Patricia Lago<EMAIL_ADDRESS>Vrije Universiteit AmsterdamThe
Netherlands ,  Ola Leifler<EMAIL_ADDRESS>Linköping UniversitySweden ,
Ana Moreira<EMAIL_ADDRESS>NOVA LINCS, School of Science and Technology, NOVA
University LisbonPortugal ,  João Paulo Fernandes<EMAIL_ADDRESS>LIACC &
DEI-FEUP, University of PortoPortugal ,  Birgit Penzenstadler
<EMAIL_ADDRESS>Chalmers University of TechnologySweden ,  Jari Porras
<EMAIL_ADDRESS>Lappeenranta-Lahti University of Technology LUTFinland and
Colin C. Venters<EMAIL_ADDRESS>University of HuddersfieldU.K

###### Abstract.

Research shows that the global society as organized today, with our current
technological and economic system, is impossible to sustain. We are living in
the Anthropocene, an era in which human activities in highly industrialized
countries are responsible for overshooting several planetary boundaries, with
poorer communities contributing least to the problems but being impacted the
most. At the same time, technical and economic gains fail to provide society
at large with equal opportunities and improved quality of life. This paper
describes approaches taken in computing education to address the issue of
sustainability. It presents results of a systematic review of literature on
sustainability in computing education. From a set of 572 publications
extracted from six large digital libraries plus snowballing, we distilled and
analyzed the 90 relevant primary studies. Using an inductive and deductive
thematic analysis, we study 1) conceptions of sustainability, computing, and
education, 2) implementations of sustainability in computing education, and 3)
research on sustainability in computing education. We present a framework
capturing learning objectives and outcomes as well as pedagogical methods for
sustainability in computing education. These results can be mapped to existing
standards and curricula in future work. We find that only a few of the
articles engage with the challenges as calling for drastic systemic change,
along with radically new understandings of computing and education. We suggest
that future research should connect to the substantial body of critical theory
such as feminist theory of science and technology. Existing research on
sustainability in computing education may be considered as rather immature as
the majority of articles are experience reports with limited empirical
research.

sustainability, computing, education, higher education, pedagogy, equality

††ccs: Social and professional topics Computing education

##  1\. Introduction

The overall integrity of the biosphere on Earth is under serious threat. There
are several tipping points for the Earth’s climate system that we have already
exceeded (IPCC:2022, 84). The accelerating socio-economic trends of the past
century (Steffen:2015aa, 194) have put us on a path toward undermining four of
the nine global planetary boundaries that define the basic conditions for
human existence on Earth (steffen2015planetary, 195). According to the
assessment of the Intergovernmental Panel on Climate Change (IPCC) from 2022,
human-caused climate change has led to a range of strongly adverse effects.
These effects include disproportionate warming of the arctic regions,
acidified oceans that threaten coral reefs and those who depend on them
worldwide, glacial retreat that threatens freshwater supplies for billions of
people, sea-level rise that threatens coastal communities worldwide, and with
extreme weather events that primarily affect poorer nations (IPCC:2022, 84).
The report describes that many of the impacts of global warming are now
already ”irreversible”, placing over 40% of the world’s population at high
risk of the effects of climate change. Access to food, energy, and water is
globally predicated on unsustainable practices, will be made even harder to
sustain with the unfolding climate crisis, and must change fundamentally in
the coming years to support the human population (c.f. (Hanjra:2010ud, 74)).
In parallel to these massive challenges to the biosphere that supports human
life, social issues are just as pressing with more violent conflicts and
democracy in global decline (House:2022uf, 78). Addressing our current
unsustainable situation thus requires “rapid, far-reaching and unprecedented
changes in all aspects of society” (IPCC2018, 85).

In this context, Information Technology (IT) is prominently but ambiguously
positioned on these issues, with respect to, for instance, rapid decoupling of
the economy from ecological footprints or strengthening democracies. Education
is seen as having a great potential for change towards sustainability
(UNESCO:2014, 217). Existing approaches to education for sustainability,
however, are also limited in how they allow for change and provide agency to
teachers and students to contribute to a better world (holfelder2019towards,
77, 62).

With the present work, we explore the potential of information technology for
sustainability as it is conveyed in education. Its aim has been to
systematically review the existing work on sustainability in computing
education, a field of research that has grown and developed into a research
discipline in the past years. We seek to collect and synthesise existing work
and provide threads for which to further advance the work on sustainability in
computing education. Another goal has also been to bring together relevant
work from different research communities engaging with the topics of
sustainability, computing, and education and to consider various forums for
publication. We analyzed existing research publications on sustainability and
computing and education with the goal to understand:

  1. (1)

Conceptions of sustainability, of the relationship between computing and
sustainability, and of education for sustainability

  2. (2)

Implementations of education including educational content, pedagogical
approaches, and effects of education

  3. (3)

Research design conducted into sustainability education, including its quality

We develop our analysis as follows: Section 2 provides various perspectives on
the role of IT for sustainability and an overview of sustainability as an
emerging field of research in computing. It also includes different
perspectives on the role of education for change and previous research
reviewing computing education for sustainability, explaining the gap in
research that this work fills. Thereafter, in Section 3, we present the
research questions for this systematic review along with the research method.
Three sections follow, each of them presenting results to one research
question, synthesizing takeaway lessons at the end of the section. In Section
7, we narrate the findings of our results, especially providing themes or
questions for future research and education development, while also discussing
the threats to validity of our study. Section 8 concludes this systematic
review.

##  2\. Background: Sustainability, Computing, and Education

In the following, we provide a brief overview of the evolution of research on
sustainability in computing generally, and in computing education
specifically. We refrain from providing nuanced conceptualisations as they are
part of the results of our analysis.

###  2.1. Sustainability and Computing

The verb ‘to sustain’ and the noun ‘sustainability’ stem from the Latin
‘sustenere’, which means both ‘to endure’ and ‘to uphold’ something
(Venters2019, 221). Accordingly, ‘sustainability’ refers to the capacity of a
_system to endure_ for a _certain amount of time_ (Venters2019, 221). Within
the well-known current conceptualisation of sustainability, put forward by the
Brundtland commission in 1987 (Brundtland1987, 225), the system in question is
Earth itself, which needs to be preserved for an unspecified period of time,
but which extends many generations into the future. The Brundtland definition
thus encompasses two aspects: distributive intragenerational justice (“the
essential needs of the world’s poor, to which overriding priority must be
given”) (Brundtland1987, 225, 37), but also intergenerational justice, for
which the preservation of the biosphere is a prerequisite.

Given the scale and urgency of various environmental and societal issues, and
the fact that computing technology protrudes ever more parts of our societies
and economies, the idea to deploy computing technology to address questions of
sustainability reaches relatively far back. Early efforts were concerned
mainly with the environmental sustainability of computing itself (i.e.,
resource and energy efficiency), in a paradigm that has often been called
“green IT” (murugesan2008harnessing, 134). The broader, and arguably
potentially more impactful paradigm, is to use computing to address energy and
resource conservation in all the other domains of our societies; a paradigm
known as “green by IT” (hilty2012energy, 75). Digital transformations are
assumed to free the economy from the shackles of the physical world and make
economic growth fair and ethereal (c.f. (Sachs:2019, 183)). However,
cryptocurrencies, for example, have such energy-devouring designs that they
are far from ethereal (Digiconomist:2022vq, 46). And while there are examples
of successful and dematerialising digital substitutions such as partly
virtualised conferences (coroama2012_twin-conference, 35), the substitution of
e-goods for physical goods has not proven to yield an overall reduction of
energy consumption (Court:2020aa, 37). In contrast, Court and Sorrell
(Court:2020aa, 37) question such purported savings: “we cannot conclude that
e-materialisation has delivered significant energy savings to date or is
likely to do so in the future” as they depend on the embedding of computing in
social and narrative structures as well as on values that underpin attempts at
sustainability in computing (Knowles:2014aa, 99).

Sustainability in computing is not limited to the environmental dimension. One
example is the role of IT for strengthening or undermining democracy. Digital
connectivity is seen as key to increased participation in 21st-century
democracies through e-voting and citizen participation (Commission:ub, 60).
However, the primary mechanisms for online connectivity are mediated through
largely unregulated companies that promote toxic content and socially divisive
business models that undermine the foundations for democracy (Zuboff:2015tj,
234). These fundamental tensions illustrate conflicting views about the nature
and future of IT and indicate potential tensions about what it would mean for
IT education to contribute to a societal transformation towards sustainable
and just societies.

The Karlskrona Manifesto for Sustainability Design (becker2012:karlskrona-
short, 14, 13) aimed to provide a focal point for establishing a common ground
for the computing community to engage with sustainability by advocating a set
of fundamental principles and commitments that underpin sustainability design.
The principles stress the importance of recognising that sustainability is an
explicit consideration, even if the primary focus of the system under design
is not sustainability. Building on Penzenstadler (penzenstadler2014infusing,
153), it also advocates that sustainability should be viewed as a construct
across five dimensions – environmental, economic, individual, social, and
technical – and considers the potential long-term effects of systems. Pham et
al. (pham2020, 164) proposed alternative dimensions including purpose, design
aesthetics, integrative and legal aspects. With the exception of the legal
aspects, it is an open question whether these are dimensions equivalent to the
ones listed above.

Individual sustainability connects to education as it is about reaching
personal sustainability goals including sustainability skills and competencies
transforming the role of individuals in their current life. As stated in
(mammas-and-pappas, 148): “ _Sustainable individuals are characterized by
creating harmony, interconnection, and relatively high levels of self-
awareness in their values, thoughts, behaviors, and actions as well as
cultivating continued individual growth in their physical, emotional, social,
philosophical, and intellectual abilities._ ” For instance, developing
individual sustainability with engineering students (barrella2018, 12) aims at
change as a combination of harmony, awareness, and behaviour to engage with
sustainability principles (shields2002, 191) and create a sustainable
personality (mammas-and-pappas, 148). Thereby, emotional and intellectual
sustainability are key factors supporting the abilities of individuals across
a wide variety of disciplines (UNESCO:2017aa, 216). While a focus on
individual sustainability was until recently less common, this started to
change by both analysing how the other sustainability dimensions can create a
positive effect on individual and group well-being (penzenstadler2021take,
159), and also inversely how human factors impact, for example, software
sustainability (imran2021, 82).

There are endless possibilities to engage with sustainability in computing and
the field of research on sustainability and computing is growing and has
matured. It covers a wide range of Green IT aspects but also IT supported
sustainability initiatives (lago2015:cacm, 107). There have been
methodological efforts to understand and assess the economy- and society-wide
indirect effects of computing, which are broad, subtle, and can be both
beneficial and detrimental to sustainability (coroama2020:assessing, 34, 15).
Terms used to refer to the work, besides “green IT” and “green by IT”, are
“Digital Sustainability”, “Sustainable Computing” or “ICT for Sustainability”
(ICT4S) (hilty2015ict, 76).

###  2.2. Education for Sustainability in Computing

Reflecting the growth of computing and sustainability into an established
field of research, there have been several efforts to also develop and
describe education on sustainability in computing. Existing publications,
however, are scattered across numerous disciplines and venues. Some work has
been done from within the _computing and sustainability_ community. By
contrast, _computing education_ , which is its own research field (simon2015,
192), paid so far relatively little attention to sustainability (pollock2019,
167). _Engineering education_ is another separate field of research, in which
sustainability is being discussed, sometimes also including perspectives on
computing.

Yet another field of research in which computing and computing education could
possibly be discussed and that computing education could build on, is the
field of _education for sustainable development_ (ESD), which provides
conceptions of education and its role for sustainability. Different analyses
are made in this field as to what should change in education. The United
Nations see the need for substantial changes to curricula and learning
objectives, requiring a “profound transformation of how we think and act”
(UNESCO:2017aa, 216, p. 7), and a reorientation towards key sustainability
competencies such as normative thinking, anticipatory thinking, and systems
thinking in a wider sense (c.f. (wiek2011key, 226, 216)). Other work suggests
making education less focused on training and detailed learning objectives and
curricula (Osberg2020, 140, 77). An alternative role of education could be to
become a democratic political space in which alternative futures are imagined
(OsbergBiesta:2008, 141, 6).

Work within education for sustainable development are critically discussed in
a research field called “critical ESD” (getzin2019shifting, 69). Recently, for
example, Stein et al. (stein2022:end-of-world, 196) propose a shift from
“education for sustainable development” to “education for the end of the world
as we know it”. Another example is Holfelder (holfelder2019towards, 77), who
criticises the focus on defining competencies in ESD as maintaining the status
quo. As expressed by (UNESCO:2017aa, 216), developing education for
sustainability is thus far more complex than integrating sustainability into
existing curricula. The ways in which existing studies build on education
research generally, and ESD specifically, is part of this review.

Within engineering education, which only sometimes explicitly discusses
computing education, calls have been made for doubling down on traditional
engineering subject matter and problem-solving methods such as optimal
resource use, Green IT, and educating future engineers on the minimization of
harm from engineering (c.f. (murugesan2008harnessing, 134, 130)). At the same
time, others have called for engineering education to be more challenge-based
and trans-disciplinary (c.f. (Ashford:2004aa, 9, 129)).

###  2.3. Other Reviews on Education for Sustainability in Computing

Only a few surveys have so far attempted to map the existing proposals of
education for sustainability in computing, and their scopes have been rather
limited. The authors in (Fisher2016, 64) provide some examples of the
integration of environmental and societal sustainability into computer science
curricula at eight different universities, and (Boyle2004, 20) describes a
theoretical analysis of what sustainability education for engineering studies
should consist of, given third-party literature. Neither, however, reviews the
current state of practice in computing education in general. Three studies
provide perspectives specific to national contexts: (Arefin2021, 7) provides
an Australian perspective and proposes guiding questions for program design,
(Leifler2020, 108) presents Swedish data on engineering degree program
directors and their motivations for and challenges in integrating
sustainability, and (Sanchez2020, 186) presents a Spanish survey and map of
sustainability integration, including in computing. (Andara2018, 180) presents
a roadmap of activities for integrating sustainability in engineering
education, based on a sustainability literacy survey, a literature review, and
an expert panel on pathways for integrating sustainability. (pollock2019, 167)
reviewed the current state of research addressing the issue of climate change
in computing education by searching the ACM and IEEE databases, conducted
expert interviews and collected data from the computing education community
during a conference. Finally, (thurer:2018wu, 207) outlines a research agenda
for studying sustainability integration in engineering in general, centering
on four main themes: (a) implemented practice, (b) values and norms among
teachers (often seen as “subjects”), and students (often seen as “objects”),
(c) the roles of other stakeholders and (d) the assessments of outcomes or
results.

In our study, we address all four themes through the contributions 1-3 listed
above in the Introduction. In relation to the reviews above, our contribution
is the first comprehensive study that addresses both the conceptions,
implementations, and research designs that have been used in prior studies on
sustainability integration in computing, with a broad definition of
sustainability, learning objectives, and pedagogical methods that include
values and norms, the roles of stakeholders and the assessment of outcomes.

##  3\. Research Method

This section details the research questions, the methods we applied, and the
process we followed. We use the term ‘computing’ as proposed by the ACM (2021)
referring to an overarching discipline consisting of five sub-disciplines (ACM
2021): computer science, computer engineering, software engineering,
information systems, and information technology. In this study, we follow the
general guidelines for systematic literature reviews by Kitchenham & Charters
(kitchenham2007guidelines, 95), and the protocol for snowballing as presented
by Wohlin (wohlin2014guidelines, 228). The general process is summarized in
Figure 2.

###  3.1. Setting the Research Questions and Themes of the Study

The following three research questions structured this work, each being
specified through sub-research questions, as follows:

  * RQ 1

How does current research conceptualize sustainability?

    * SRQ 1.1

How is sustainability being perceived?

    * SRQ 1.2

How is the relationship between computing and sustainability understood?

    * SRQ 1.3

How is sustainability education being understood?

  * RQ 2

How is the implementation of sustainability education being presented?

    * SRQ2.1

What content is sustainability education about, i.e. what are the learning
objectives and the topics being taught?

    * SRQ2.2

How is sustainability education being organized in terms of teaching practices
and modules or curricula?

    * SRQ2.3

What are the effects of teaching sustainability?

  * RQ 3

What type of research design has been conducted on sustainability and
computing education?

    * SRQ3.1

What epistemological stance has been adopted as part of the research design?

    * SRQ3.2

What research methods have been adopted as part of the research design?

    * SRQ3.3

What threats to validity are reported?

Table 1 presents the themes and keywords that were identified from our
research questions. They were used to create the following general search
query. Different truncation (wild card) symbols were used in the searches to
capture different word forms:

(“Teach*” OR “Educat*”) AND (“Computer Science” OR “Software Engineering” OR
“Computing”) AND (“Sustainable” OR “Sustainability” OR “Green” OR “Climate
change” OR “Global warming”).

Theme | Proposed keywords
---|---
Education | Education, Teaching, Learning
Computing discipline | Computer science, Software Engineering, Computing
Sustainability | Sustainable, Sustainability, Green, Climate Change, Global warming
Table 1. Themes included in our study with the selected keywords

###  3.2. Defining Search Strategies and Selecting Research Sources

In November 2020, we identified relevant papers by searching through the
following databases, ACM, IEEE, Web of Science (WoS), and Scopus. We also
considered the Emerald database, but due to constraints in the search tool, we
only considered the Journal of Sustainability in Higher Education. We also
searched the Springer database. However, with limited options to specify the
search, the number of results was too large. In all cases, we searched the
fields title, abstract, and author keywords for our specified keywords. To
mitigate the limitations of the search, we did backward- and forward-
snowballing (wohlin2014guidelines, 228) to identify further relevant papers.
We also decided to include relevant papers from the sustainability workshop of
the ICT4S conference as the topics relevant to this study have been covered
there. Table 2 lists the searched databases, the search string used, some
limitations, the number of retrieved publications, and the number of selected
primary studies for data extraction and analysis. We repeated the scan of
literature in March 2022 to identify additional papers published since the
first search.

Database |  Search string |  Other limitations | Results | Incl.
---|---|---|---|---
ACM |  [[Abstract: teach*] OR [Abstract: educat*]] AND [[Abstract: ”computer science”] OR [Abstract: ”software engineering”] OR [Abstract: ”computing”]] AND [[Abstract: ”sustainable”] OR [Abstract: ”sustainability”] OR [Abstract: ”green”] OR [Abstract: ”climate change”] OR [Abstract: ”global warming”]] |  AND [Publication Date: (01/01/2000 TO 27/3/2022)] | 81 | 9
IEEE |  ((”Abstract”:teach* OR ”Abstract”:educat*) AND (”Abstract”:”computer science” OR ”Abstract”:”software engineering” OR ”Abstract”:computing) AND (”Abstract”:sustainable OR ”Abstract”:sustainability OR ”Abstract”:green OR ”Abstract”:”climate change” OR ”Abstract”:”global warming”)) |  Timespan: 2000-2022 | 199 | 19
WoS |  TITLE: ((teach* OR educat*) AND (computer science OR software engineering OR ’computing’) AND (sustainable OR sustainability OR green OR climate change OR global warming)) |  Timespan: 2000-2022 | 33 | 4
Scopus |  TITLE-ABS-KEY-AUTH ( ( teach* OR educat* ) AND ( ’computer AND science’ OR ’software AND engineering’ OR ’computing’ ) AND ( sustainable OR sustainability OR green OR ’climate AND change’ OR ’global AND warming’ ) ) |  Timespan: 2000-2022 | 14 | 1
Emerald |  ((teach* OR educat*) AND (”computer science” OR ”software engineering” OR computing) AND (sustainable OR sustainability OR green OR ”climate change” OR ”global warming”)) |  International Journal of Sustainability in Higher Education | 142 | 10
ICT4S |  Papers in the proceedings of the ICT4S conference series |  International workshop on Computing+Sustainability+Education (CompSusTech) | 4 | 3
Table 2. Digital databases, limitations, retrieved publications and selected
primary studies

###  3.3. Selecting Studies Based on Inclusion and Exclusion Criteria

The search results were collected in a spreadsheet, noting metadata such as
title, authors, abstract, publication date. Study selection was done in two
phases. First excluding papers by using the first five exclusion criteria and
then including only papers fulfilling the inclusion criteria. Exclusion of the
papers based on the exclusion criteria EC1-5 is simple as one does not have to
focus on the contents. Only EC6 requires knowledge of the contents of the
papers and as such, it was implemented as a part of the inclusion phase.

  * •

EC1: Duplicates from different data sources

  * •

EC2: Papers not written in English

  * •

EC3: Papers not accessible by the researchers

  * •

EC4: Short papers less than 4 pages long

  * •

EC5: Non-scientific and peer-reviewed publications, posters, thesis, or books

  * •

EC6: Papers from the same authors presenting the same results

In the inclusion phase the search results were categorized for relevance
(quality criteria towards RQs). Two researchers rated the results
independently, marking the results with either 1 (include), 0 (not included),
or 0.5 (undecided). Undecided papers were further checked by a third
researcher. Articles with a sum of at least 1 were included in our study. As a
result, we included papers based on the following inclusion criteria.

  * •

IC1: Papers about computing and sustainability and education. We understand
computing in a broad sense, including fields such as computer science,
software engineering, and other related engineering disciplines addressing any
sustainability dimension. (Papers that do not address education in computer
science and other related engineering disciplines or don’t include
sustainability topics were not included.)

  * •

IC2: Papers in which sustainability is a goal for education and teaching. (If
papers used sustainable teaching only to refer to the workloads, they were not
included.)

The original search in the databases yielded 473 results. Reading the metadata
(abstract, title, etc.), we found 46 articles relevant. Two of those articles
were excluded after more detailed readings by several of the authors,
resulting in 44 articles.

###  3.4. Study extension by snowballing

Backward and forward snowballing (wohlin2014guidelines, 228) with the selected
46 articles yielded another 99 articles that were identified as potentially
relevant. Those articles were then checked for relevance by another
researcher, resulting in 44 more relevant articles. Thus the final count of
primary studies selected for our study was 90. Table LABEL:long in the
appendix includes a list of primary studies, the type of publication venue and
focus of the venue (education, computing, or sustainability). There is an
increasing number of relevant papers over the years, with a majority of search
results being less than 10 years old, see Fig. 1.

Figure 1. The selected publications per year.

###  3.5. Data collection

To answer the research questions, we identified a set of themes to analyse the
collected data, through a joint analysis of a small number of studies. We
tested those themes, captured in a spreadsheet for the analysis of the paper,
by analyzing the first papers and then discussing the use of the themes among
the group. We then agreed on the following themes to analyse all the
identified papers:

  * •

Sustainability definition (perception of sustainability) [SRQ 1.1 and SRQ 1.2]

  * •

Understanding of what education of sustainability is [SRQ 1.3]

  * •

Learning objectives (goals) [SRQ2.1]

  * •

Best practices (means) [SRQ 2.2]

  * •

Implementation level (single course, minor, programme, curriculum, …) [SRQ
2.2]

  * •

Implementation techniques used in relation to recommendations for learning for
sustainable development (see (UNESCO:2017aa, 216, 223)). Such techniques refer
to learning methods reported to be helpful in developing strategic
competencies such as normative thinking, systems thinking [SRQ 2.2]

  * •

Learning outcomes (students) [SRQ2.3]

  * •

Study outcomes (long-term attitude changes, awareness change, feasibility of
subject integration, faculty acceptance, faculty competence improvement)
[SRQ2.3]

  * •

Study objectives [SRQ 3]

  * •

How and where were the objectives integrated into the curricula [Meta-
analysis]

Figure 2. The literature review process followed in this study.

###  3.6. Data Analysis

We performed a thematic analysis on the extracted results per theme and then
consulted the referenced primary and secondary literature for more details and
evidence as well as quotes. The data were analyzed in three smaller groups of
authors that developed a coding scheme in several iterations of refinement.
The findings were discussed in subgroups and synthesized into a narrative for
reporting. During the whole analysis process, we have had weekly meetings to
report and consolidate the results.

To answer RQ 1, we developed the coding scheme through an inductive approach,
identifying and revising themes capturing the different meanings in the data.
We analyzed different ways of conceiving of sustainability, computing and
sustainability, and education for sustainability in the papers. Oftentimes, we
found such conceptions in the motivations for teaching and learning about
sustainability in computing education.

With regard to RQ2, we investigated how sustainability education has been
implemented in CS and IT courses. We found the implementation described in
terms of learning objectives, topics taught, pedagogical approaches and
learning outcomes, for which we identified sub-themes in various iterations.

For RQ3, we adopted a deductive approach to analyzing the kind of empirical
research and the research quality in the primary studies In order to evaluate
the quality of the studies according to the research method used, we adopted
the ACM taxonomy of empirical standards (Ralph2020, 173) to compare each
primary study versus the different categories of empirical standards. As a
result, we classified the different studies in any of the existing categories
based on a number of essential attributes each type of paper must fulfill.
Apart from this we also analyzed the epistemological stance of the approaches
and if the the threats to validity found in the studies report incorrect
conclusions. In this way we can identify the quality of the empirical studies
reporting a particular case study or if they remain just only as proposals or
vision papers.

In a further step, we synthesized the acquired results into an overall
narrative and shed light on where the results are aligned and form a
homogeneous picture and where it is heterogeneous and sometimes misaligned. We
also identified and sorted out the overlap between the findings in the groups.
Furthermore, we discussed the current state of practice in relation to what we
know about best practices for education and education research for
sustainability.

##  4\. Conceptions of sustainability, computing, and education (RQ1)

In RQ1, the conceptions of sustainability and education were analyzed with
respect to (i) general conceptions of the current state of the world and what
a sustainable development would mean, (ii) how computing relates to the
current state of the world and potential directions for a sustainable
development, and finally (iii) how education for sustainable development is
conceived of in relation to the first two. Not all papers explicitly mention
(i) and (iii), but naturally all have conceptions of IT in relationship to
sustainability. Conceptions are also part of some of the motivations in the
papers, about why sustainability education in computing is important. For
example, sustainability education in computing is described as important
(SB-16, 224, 200), as something that should be uncontroversial (ACM-01, 59).
Mann (SB-05, 116) argues that computing and education are two disciplines with
high leverage for change with respect to sustainability. This implies a view
of three disciplines that are to be connected for change.

We found several categories useful for distinguishing the conceptions
described in the papers. However, they are neither intended to be exclusive
nor comprehensive. A single paper may be included in several of the
categories, so inclusion in one does not exclude from inclusion in others. Not
all papers were explicit about their conceptions of sustainability, computing
or education, so the list does not comprehensively capture conceptions in all
papers. In some instances, a paper could feature several perspectives in one
of the categories as well. However, these categories provide a lens through
which we could understand the different conceptions and uncover the important
differences.

###  4.1. Conceptions of Sustainability

We identified six categories in our analysis of conceptions of sustainability.
In some cases, those categories were explicitly referred to in the papers but
in other cases, it was us as authors who interpreted the positioning of the
papers from their identified descriptions in the papers.

The categories are:

  1. (1)

Temporality: present vs. future

  2. (2)

Value transparency: implicit vs. explicit

  3. (3)

Originality: common definitions vs. synthesis

  4. (4)

Justification: anthropocentric vs. ecocentric

  5. (5)

Relativity: weak vs. strong

  6. (6)

Responsibility: systemic and global vs. individual and local

The temporality category makes a distinction between papers that primarily
considered issues of sustainability-related to the present, and awareness of
issues to address (SB-89, 199, 178, 125), versus those that considered future
states to be avoided (such as societal or ecosystem collapses) or desired
(such as fulfilling the goals in Agenda 2030) (SB-05, 116). For example,
(SB-89, 199) noted that challenges of the present tend to be so-called wicked
problems, not amenable to fixed solutions and optimal outcomes, but rather to
be managed as social issues to be re-negotiated. This concerns the state of
affairs in the present and issues we face. In such papers, awareness of the
current state of affairs was considered important: “it is necessary that
engineering students of all engineering disciplines be aware of environmental
issues” (SB-07, 178), or ”green thinking and the broad adoption of green
software in computer science curriculum can greatly benefit our environment,
society, and students” (ACM-New-2, 187). Alternatively, some papers focus on
actions for future directions that require radical changes to society:
“Sustainability is considered here in terms of system change actions resulting
in restorative socio-ecological transformation” (SB-05, 116).

The value transparency category distinguishes between implicit, technical
definitions of preserving the function of a system (SB-01, 144, 158, 25),
versus explicit value-based reasoning on world views, the rights or needs of
people living now and those who live in the future, as well as other life on
the planet (SB-01, 144, 119). By technical definitions, we mean definitions
that make value concerns invisible. They are also more technical in their
choice of terminology. (SB-01, 144) provides a technical definition and
describes that sustainability means to “use in a ternary relation as the use
of a system S with regard to a function F and a time horizon L, which means
using S in a way that does not compromise its ability to fulfill F for a
period L” (SB-01, 144). This follows from the tradition of System Dynamics and
modeling the world in terms of systems of interacting components with feedback
loops that can cause stabilizing, runaway or generally unpredictable behavior
(see e.g., (Ramage:2020aa, 174)). However, as noted by systems thinkers such
as Ulrich, values always inform the way we choose factual statements and
system boundaries (ulrich1987critical, 215). The value-based definitions in
the papers included (SB-01, 144, 119) mention of the New Ecological Paradigm
(NEP) scale (Dunlap:aa, 48) and the worldviews assumed to be associated with
it: “we are particularly interested in understanding the deeper worldviews of
the students —a focus on the affective attributes of values, attitudes and
beliefs” (SB-31, 119).

Regarding originality, we saw differences between the re-iteration of common
definitions in general literature on sustainability, and a synthesis of
definitions produced by the authors themselves. Here, the definition from Our
Common Future, also referred to as ”Brundtland definition”, appeared most
often (SB-01, 144, 25, 154, 66, 205, 171, 128, 121, 23, 189) planetary
boundaries (Rockstrom:2009aa, 179) appeared as a concept in two papers
(IEEE-175, 170, 55), the Triple Bottom Line in one (Mann2011, 118) (a term the
original author has since tried to revoke (Elkington:2018aa, 51)), Agenda 2030
(:2015ab, 211) appears in two papers (IJSHE-26, 8, 23), and the three
dimensions of economic, ecological and social concerns were mentioned in
another three papers (CompSusEd-02, 219, 4, 189). There were several examples
of extensions regarding the relationship to technology, however. (SB-40, 49)
provides a definition that specifically relates sustainability to the
construction and use of technology: “Sustainability is an emergent phenomenon
from the interaction between (very large numbers of) people, and the ways in
which we build and use technology”. Others referred to the three dimensions of
sustainability but also add individual and technical concerns (IEEE-172, 171,
28) (penzenstadler2013generic, 155), or add a fourth ethical pillar (IEEE-
New-4, 137).

In distinguishing different justifications of sustainability, we could see
both anthropocentric and ecocentric perspectives (Imran:2014aa, 83). From an
anthropocentric perspective, sustainability is justified by the desires and
needs of people. Environmental and social concerns are valid to the extent
they concern the interest of human beings. An ecocentric perspective on the
other hand justifies sustainability from the perspective of the inherent value
of all life on the planet, treating life as sacred on its own terms. The nine
papers that adopt the definition from Our Common Future (SB-01, 144, 25, 154,
66, 205, 171, 128, 121, 23) and the three that mention Agenda 2030 and the
Sustainable Development Goals (IJSHE-26, 8, 23, 71) can be considered
anthropocentric as Our Common Future centers on the needs of humans, 15 of the
17 SDGs concern humans, and only SDG 14 & 15 list goals that pertain to all
other life (Adelman:2018vi, 2). Two papers adopted an ecocentric perspective
on sustainability (SB-05, 116, 119) as they describe sustainability as
“restorative socio-ecological transformation” (SB-05, 116) and an “ecological
worldview” (SB-31, 119).

The relativity category contrasts strong (IEEE-175, 170, 55, 178, 150, 56)
versus weak (SB-01, 144, 232) (“vanilla”) sustainability (neumayer2003weak,
135). The concept of strong sustainability is associated with respecting
absolute, unconditional planetary boundaries and social foundations must be
respected to avoid societal or ecosystem-wide collapse. The papers that
mention the planetary boundaries framework and global limits to human
activities were considered to adhere to the notion of strong sustainability.
For instance, in (SB-62, 56) the authors write that “We have overstepped
several planetary boundaries and risk overstepping several more
(steffen2015planetary, 195). We are about to reach limits as to the resources
we can extract from the earth (bardi2014extracted, 11), and the changes
wreaked are by now so profound that they will likely last for a geological
period of time”. Weak sustainability instead focuses on the relative balance
between economic, social and environmental concerns. (IJSHE-13, 232) writes
that “sustainability is about giving equal consideration to the economic,
environmental and social aspects. There is growing consensus among researchers
that conceptions of sustainability must include consideration of
environmental, economic, and social factors” (IJSHE-13, 232). An example of a
study that explicitly compares these two perspectives is (SB-19, 150), which
states that vanilla sustainability concerns ”picking low-hanging fruit in the
form of energy efficiency, incremental technological innovations, or by
applying human ingenuity [that] will allow us to continue to live as we do
today” (SB-19, 150). They argue that the strong form of sustainability,
however, is required to face serious problems “such as climate change,
planetary boundaries, future scarcity of nonrenewable resources (fossil
energy, minerals), and the consequent challenges this scarcity will pose for
our economic system.” (SB-19, 150). “Strong sustainability” here represents a
realignment of IT with a new societal project, one that assumes that
fundamental changes of contemporary western lifestyles are needed to avert the
worst of climate crisis, resource depletion, and ecosystem destruction.

Finally, the responsibility category concerns whether sustainability is
considered something that must be understood and addressed as something
systemic and global (SB-01, 144, 116) or individual and local (ACM-06, 158).
Here, there is not necessarily so much a conflict between different
conceptions of what sustainability is, but rather what makes most sense to
direct the attention of teachers and students to in the context of education.
For example, (SB-01, 144) takes a systemic and global perspective and mentions
that “The key challenges of the 21st century can mainly be characterized by
their global impacts” (SB-01, 144). (ACM-06, 158) on the other hand, begins by
outlining global issues such as prolonged droughts that are the results of the
climate crisis, but continue to focus more on individual action: “To become
more resilient to future climate conditions (adger2005successful, 3), create a
greener society, save money, and improve food quality (national2014garden,
10), many individuals are practicing home horticulture” (ACM-06, 158).

Of the papers that make it apparent how they conceive of sustainability, we
notice that the most prevalent conception is of sustainability as synonymous
with Agenda 2030 and the definition in Our Common Future, which are
anthropocentric and established in related literature. Ecocentric
perspectives, in contrast, are much less common, although a number of
different perspectives exist in individual papers.

###  4.2. IT and Sustainability

During the analysis, we found three perspectives on the role of IT in
sustainability: (i) an incremental perspective where the focus is on adjusting
computing to reduce its footprint, (ii) an enabling perspective where IT is
the means to address society-wide sustainability issues, and (iii) a
disruptive perspective, which acknowledges the severe sustainability
challenges we are facing that require —as also argued e.g. by the
IPCC111https://www.annualreviews.org/doi/abs/10.1146/annurev-
environ-012220-011104— major, structural societal changes, and where also IT
norms, practices and processes must change to align with and support the core
principles of sustainability.

Several primary studies focus on incremental changes within computing,
adjusting the production of software and hardware in order to reduce their
environmental footprint. Also beyond the education literature covered in our
study, this paradigm has often been called “green IT” (coroama2009energy, 32,
76) or “green computing”. This is in accordance with (SB-01, 144) that
advocates the green use, disposal, design and manufacturing of IT systems as
proposed by (murugesan2008harnessing, 134). Several of the analyzed studies
(SB-01, 144, 154, 25, 205, 53, 86, 55, 90, 71, 187) acknowledge the IT
footprint and suggest adjustments to how we currently work. They acknowledge
that energy and material consumption represent a problem and stress the impact
that ICT infrastructures have on the environment due to their high energy
demand (“3% of global electricity consumption” (SB-01, 144), greenhouse gasses
(ACM-10, 1, 154), raw materials required that also contribute to toxic waste
(ACM-10, 1, 154), and e-waste produced (IEEE-163, 205)). Adjusting IT to
contribute to sustainability is the direction of some other studies that
suggest handling sustainability as yet another quality attribute, e.g. (WoS-2,
146, 154).

We further found studies recognizing the value of computer science or IT as an
enabler to address many of the society-wide sustainability issues. This
paradigm, often addressed as “green by IT” (hilty2015ict, 76), “I(C)T enabling
effect” (malmodin2014life, 115) or occasionally also as “ICT handprint”
(WOS-5, 117), aims at using ICT to support sustainability throughout the
entire society and economy; it applies computing to contribute to societal
change towards sustainability; without, however, expecting radical changes of
the society or economy themselves. Sometimes this paradigm of greening outside
IT is also called “green computing”, e.g. in (ACM-12, 25), although such
terminology can easily lead to confusion with the previous category, “green
IT”, which addresses greening within IT. The enabling paradigm widens the
focus from the IT industry itself towards society and economy at large,
promoting a “new sustainable model by mutually advancing economic,
environmental, and social goals” (ACM-12, 25). Similarly, (CompSusEd-03, 106)
(p.2) notes that “we extend the focus of sustainability-related competencies
and skills from the underlying digital technology to the needs of a digital
society”. Also, (IJSHE-26, 8) recognizes the value of IT as an enabling
technology, stating that “Computer science or IT, does not explicitly
correspond to any of the SDGs but is nevertheless considered to be crucial for
the fulfillment of each of them”. Additionally, (ACM-10, 1) uses IT as an
enabler when discussing how IT can “look outward to solve sustainability
problems that IT isn’t actually causing” (e.g., lights and room temperatures
managed by IT systems, or transportation and logistics systems) and as a
reporter when using data management and visualization to support those
systems.

The third case focuses on a disruptive perspective that puts sustainability
first and entails the realignment of IT with a new societal project, demanding
for radical changes in our professional role through some or all of the
following characteristics: responsibility, accountability, systems thinking,
and affective reasoning, and perhaps activism (also supported by
(schendler2009getting, 188)) and challenging political structures and norms.
This is in accordance with the idea of transformation mindset by (WOS-5, 117)
arguing that a transformation cannot be “met through marginal lifestyle
changes (placet2005strategies, 166), instead we need urgent and ambitious
changes (thogersen2009simple, 206)”. So, a change in professional behavior is
in order. As (IEEE-New-3, 110) formulates it, IT professionals should have
abilities to “understand the principles [of] complex systems, […] reason about
systems of values […], and to deal with conflicts between the values of
different stakeholders” while also having effective competencies for the
“engagement, commitment and behavioral change towards sustainability” (IEEE-
New-3, 110). In particular, several papers emphasize responsibility,
accountability, and the skills to contribute to solve complex problems with
conflicting goals, e.g. (SB-34, 81) stresses the need for creative solutions
to complex problems and the importance of engineers being able to critically
assess the implications of their professional actions. Such skills go beyond
traditional computational thinking, demanding the ability to understand the
circular nature of the world we live in. Systems thinking is also addressed by
several authors (e.g. (SB-01, 144, 49)) as the approach that “provides the
necessary bridge from computational thinking to sustainability practice […]”.
It ”provides a domain ontology for reasoning about sustainability, a
conceptual basis for reasoning about transformational change, and a set of
methods for critical thinking about the social and environmental impacts of
technology” (SB-40, 49). Such disruptive perspective might mean challenging
established computing norms, practices and processes and aligning them with
the core principles of sustainability (SB-07, 178, 144, 110) (e.g., reduce,
reuse, and recycle), “finding innovative ways to use ICT in business processes
to deliver sustainability benefits across the enterprise and beyond” (SB-01,
144), and also considering the Karlskrona
Manifesto222https://www.sustainabilitydesign.org (IEEE-New-3, 110).

###  4.3. Education and Computing Education for Sustainability

In the following, we describe how sustainability education in computing is
understood and presented. We identify two categories for describing this
understanding:

  1. (1)

Context: computing vs sustainability as the main context of learning

  2. (2)

Purpose: training vs emancipation

The first category context reminds of the previous discussion of
sustainability as something that is added to computing or as something that is
treated as the context to understand computing within (see Section 4.2).
Similarly, we find views of computing education as the main context of
learning, having a right in its own, and sustainability education being
“added” to computing education. We find many examples of incremental changes
to education described in (ACM-01, 59, 25, 205, 231, 112, 213, 156, 181, 8,
224, 150, 204, 119, 55, 201, 103, 202, 56, 126, 146, 105). For example,
sustainability education can be included in lectures on software engineering
(IEEE-Penz2011CSEET, 156), where e.g. sustainability can be discussed as a yet
another quality attribute of a software system (WoS-1, 126). Another example
is promoting sustainability competencies (wiek2011key, 226, 145, 87), which is
argued for in (SB-54, 103, 119). A full description of sustainability content,
which is included in computing education is presented in Sections 5.1 and 5.2.
On the other side of the spectrum, computing education as a whole is
reimagined, realigned, or transformed to be about sustainability education, as
explained by (sterling2004higher, 197). That means that computing education is
re-contextualised, sustainability is the core value of computing education
(ACM-14, 121), it is at the “nexus” of computing education (SB-21, 120). Such
a change of education to address sustainability is called “transformative”
(ACM-12, 25). There are few examples of work with such an idea of
sustainability education. An investigation from 2007 finds that only one
engineering programme in the USA had been redesigned (SB-34, 81). Samuel Mann
with colleagues develop computing education that centres around the learning
objective to become a sustainable practitioner (SB-05, 116, 117, 120, 119).
Porras et al. (IEEE-153, 169, 171, 170) as well as Lago and de Boer
(CompSusEd-03, 106) present master programs that were developed from the
ground centering around sustainability.

Transformative change is difficult to achieve in the current system (SB-45,
55). Adding courses or modules to the existing education is an easier and more
straightforward approach to implementing sustainability (ACM-12, 25). Yet,
inserting sustainability into a crowded computing curriculum can be
experienced as a challenge (ACM-12, 25), which means that the insertion of
sustainability can become more of an injection, a “patch-on” (SB-45, 55),
sustai ability is “squeezed in” (SB-62, 56). It requires balancing different
stakeholder interests (SB-30, 204, 213). However, even smaller modules such as
a lecture, may lead to more transformative changes of computing education
later on (IEEE-Penz2011CSEET, 156).

The second category purpose is about what education aims to achieve. Education
with a focus on training centers around equipping the students with certain
skills, knowledge, or competencies, which also include attitudes or values.
Holfelder calls this conception of education “training” and argues that
education here is an instrument for attaining some specified objective
(holfelder2019towards, 77, p. 944) Most papers engage with education as
training. Section 5.1 provides a detailed description of the learning
objectives as they are specified in the papers. A few studies reflect about
education as something different, which we call emancipation, a term also used
in the literature on education for sustainability (kaufmann2019building, 92,
69). Education here supports an inner journey and personal growth to forge the
future with one’s unique personal experiences and background. Samuel Mann
argues for such an understanding of education (SB-05, 116): “Rather than
specifying a predetermined set of behaviors to describe sustainability within
a discipline, instead we aim to take students on a journey of themselves
identifying what it means for them to think and act as a sustainable
practitioner”. Mann describes two student examples and suggests that technical
content can be learnt “on-demand”, as needed by the student or the project.
Such an idea of education would imply a change of power relation from lecturer
as experts deciding on how to expose the student to sustainability and crisis,
to lecturer and learner as more equal and co-learners (SB-05, 116, 106). Lago
and de Boer write about co-creation of real-world examples for local
communities (CompSusEd-03, 106). Mann et al. (ACM-14, 121) state a vision that
graduates, practitioners and academics understand concepts of sustainability
so that they can “evaluate, question and discuss their role in the world and
to enable them to make changes where and when appropriate”. This vision
emphasizes the individuals’ unique possibilities in the world, if they are
seen and given a space in education. The Vrije University of Amsterdam, lifts
three core values, that could be interpreted as emancipatory views of
education (CompSusEd-03, 106): (1) students’ responsibility for their
education, (2) the interdisciplinary and open character of the programme, and
(3) students’ responsibility for issues such as “sustainability, ethics,
privacy”.

###  4.4. Takeaways for RQ1

This section provides categories that help conceive of sustainability,
computing and sustainability, as well as education for sustainability in
computing education. For each category, we contrasted different values such as
ecocentric vs. anthropocentric, or computing first vs. sustainability first.
Those values should not be seen as opposites or binaries that are clearly
separated, but rather reflective of the various, partly overlapping, sometimes
difficult to match, perspectives found in the papers.

Given this plethora of conceptions of sustainability that differ along various
dimensions, it is perhaps surprising that they ultimately boil down to only
three main possible roles that IT could take in sustainability: an incremental
perspective focusing on reducing the direct environmental footprint of
computing, an enabling perspective in which IT is the means to addresses
society-wide sustainability issues, and a disruptive perspective, which
acknowledges and responds to the need of unprecedented change.

Our review of conceptions of education suggests that most primary studies view
sustainability as an important concern for computing professionals, but not in
such a way that it would require radically new degree programs. A few authors,
however, see that education needs to be reconceptualised to support change
towards sustainable development, which we here presented in terms of
emancipatory education.

##  5\. Implementation of education for sustainability in computing (RQ2)

In RQ2, we investigate how education for sustainability in computing is being
implemented. We do so by researching the following aspects: (i) the intended
learning objectives (i.e., what the students should know at the end of a
course, see Section 5.1), (ii) the topics, or contents through which the
learning objectives are being realized (Section 5.2, i.e., the content of
education), (iii) the organization of education in terms of pedagogical
approaches and educational structures (Section 5.3), and finally (iv) the
reported effects (i.e., the achieved learning outcomes and other effects,
Section 5.4). We close with a summary of the main takeaways (Section 5.5).

The systematic literature review results related to the aspects mentioned
above are explained in the following sections and summarized in Figure 3. In
this figure, the horizontal (white) pillars represent the types of intended
learning objectives that were elicited from the primary studies. The topics
are used to concretize learning objectives and are represented by the first
three vertical pillars: they can be computing-specific (i.e., ICT topics where
sustainability provides the application context), sustainability-specific
(i.e., topics covering sustainability knowledge), or a combination thereof
(i.e., topics that really mix both domains and as such cannot be classified as
one or the other). Finally, the last two vertical pillars represent how
education is being organized.

Figure 3. An overview of the different educational components in education for
sustainability in computing

###  5.1. Intended Learning Objectives (SRQ2.1)

The intended learning objectives identified in the primary studies are what
the students are expected to learn in a course or teaching intervention, i.e.,
the competencies the students should have acquired, or, in Bloom’s terms,
“what students should know or be able to do at the end of the course that they
couldn’t do before” (Bloom, 19). We organized the learning objectives in four
broad categories: Acquiring topic-specific competencies, Acquiring cross-
cutting competencies, Gaining awareness, and Gaining practical experience. In
the following, we explain each category and map it on the corresponding Dublin
descriptor(s)333The Dublin Descriptors are general statements about the
ordinary outcomes that are achieved by students after completing a curriculum
of studies and obtaining a qualification. They are neither meant to be
prescriptive rules, nor do they represent benchmarks or minimal requirements,
since they are not comprehensive. The descriptors are conceived to describe
the overall nature of the qualification. Furthermore, they are not to be
considered disciplines and they are not limited to specific academic or
professional areas. The Dublin Descriptors consist of the following elements:
Knowledge and understanding; Applying knowledge and understanding; Making
judgments; Communication skills; Learning skills. The European Higher
Education Area http://ehea.info (Gudeva2012-mo, 72). This way, given the
widespread adoption of Dublin descriptors in educational frameworks
especially, but not exclusively, in the European Union, we aim to ease their
use.

Acquiring topic-specific competencies. Topic-specific competencies are
specific to the domain of computing and sustainability. In any educational
component (module or curriculum), the intended learning objectives and the
related topics are intertwined (as the second is instrumental to achieve the
first). Therefore, the topics elicited from the studies are presented in
Section 5.2. Depending on the types of competencies covered by the study,
acquiring topic-specific competencies can be generally mapped on Dublin
descriptors “Knowledge and understanding”, and “Applying knowledge and
understanding’.

Acquiring cross-cutting competencies. These competencies are relevant across
applications and domains and allow people to act effectively in the target
application or domain. Previous work has formulated such competencies in terms
of “key sustainability competencies” which include, e.g., systems thinking and
integrative problem solving (wiek2011key, 226, 145). We see reference to the
work by Wiek et al. (wiek2011key, 226) in (SB-54, 103, 172, 89, 103, 128, 126,
227). Mann et al. (SB-31, 119) link to action competence444I.e., an
individual’s capacity of critically selecting and conducting possible actions
that may solve societal problems through democratic mechanisms
(https://wikieducator.org/Learning_and_Teaching_in_Practice/Action_competence).
in environmental education (jensen1997action, 87). In the primary studies, we
identified the following list of cross-cutting competencies:

  * •

Systems thinking. Systems thinking is a holistic approach to thinking in
systems and included in various ways and contexts: for climate change and
gamification (ACM-01, 59, 54), in general to address sustainability in broad
contexts (CompSusEd-02, 219, 144); for problem solving in a systemic way
(IEEE-161, 233, 112), to include environmental-friendliness in computing
processes (IEEE-167, 213); and for balancing ICT, economics and sustainability
(IEEE-171, 113, 133, 219). Also, systems thinking is mentioned as an
instrument to acquire a reflective, holistic way of thinking (IJSHE-6, 193).
Easterbrook (SB-40, 49) proposes it in the form of a Toolkit with a Domain
Ontology for Sustainability Thinking, Theories of Change, and Tools for
Critical Analysis. Hilty and Huber (IJSHE-11, 79) conclude that “teaching
systems thinking (…) plays an important role in teaching sustainability, in
particular when the fields of ICT and Sustainable Development (SD) are to be
combined”. Specifically the “rebound effect”, which can be used as a potential
introductory example to systems thinking, provoked students to think about
technical efficiency in a new way.

  * •

Multi-perspective thinking. Multi-perspective thinking uses systems thinking
to especially consider the concerns from various parties. For example,
Krogstie and Krogstie (SB-54, 103) promote it as way to address the concerns
of individuals versus those of social communities, e.g., by using personas and
associated scenarios to explore different needs.

  * •

Transformation mindset. Acquiring a transformation mindset is proposed as “a
way of thinking that leads to transformational acts resulting in socio-
ecological restoration” (WOS-5, 117).

  * •

Judgments. Making judgments is another frequently-mentioned cross-cutting
skill for many sustainability-related purposes: we make judgments in ethics
(CompSusEd-03, 106); we make judgments on the societal role of informatics in
shaping the future, and in doing so we assume an active role as software
professionals (IEEE-New-3, 110); we use judgments to reflect on (i) ethical
and sustainability aspects of software systems, and (ii) choices in terms of
both product impacts and processes (IJSHE-12, 24). We also use judgments for
critical thinking (e.g., for exploring the use of technology to protect the
environment (IJSHE-5, 44); and for learning and reflectivity (SB-99, 227)).

  * •

Creativity. Creativity is reported as a key competence in software engineering
for sustainability (IEEE-161, 233); for which it found as especially necessary
“[…] for developing new insight, making new connections, and identifying new
solutions”.

  * •

Communication. Sustainability in given subjects requires communication skills,
included in primary studies (IJSHE-12, 24, 44) as a skill to report orally
and/or in writing. One paper uses specifically academic writing to communicate
basic tenets of sustainability and IT in society, responsibilities of IT
professionals, and the effects that a particular IT system has from a systems
perspective (IEEE-New-3, 110).

  * •

Ecological approach. The ability of applying an ecological approach to
computing is proposed by some studies as a learning objective instrumental,
e.g., to the creation, research and exploitation of ICT (IEEE-167, 213), to
create eco-friendly technology and processes (IEEE-164, 231), and in general
to develop STEM reasoning skills in socio-technical areas (IJSHE-10, 133). A
special angle is addressed by Qureshi (IJSHE-8, 172), to develop the knowledge
of how to develop and improve sustainable living (where ICT is a supporting
instrument).

  * •

Meta-learning. “Learning to learn” is a skill to “support students’
development in a range of analytical, communication and learning skills
appropriate to the subject matter and level of the course” (IJSHE-5, 44).

  * •

Research methods. Research methods that specifically benefit ICT for
Development (ICT4D) and the SDGs are taught to the students. For example, Lago
and de Boer (CompSusEd-03, 106) propose Green Labs and ICT4D Field Work in
their preview of a Master program geared around Digital Society and
Sustainability. Müller et al. (IJSHE-7, 131) propose a model combining
education on sustainability with thorough training in scientific research
methods, including example projects from an existing social science curriculum
and its integration into a real-world laboratory on sustainable energy use.

  * •

Critical thinking. The concept of critical thinking goes back to the American
philosopher John Dewey (1910), who defined it as “active, persistent and
careful consideration of any belief or supposed form of knowledge in the light
of the grounds that support it, and the further conclusions to which it tends”
(Dewey1910, 45). Critical thinking methods are important for all students such
that “the next generation of professionals may be more likely to give
consideration to issues (e.g., the environment, social equity, biodiversity)
that they otherwise would have ignored” (SB-34, 81). Critical thinking methods
have gained popularity (CompSusEd-03, 106, 56, 57). Another example of
critical thinking is proposed by (IEEE-165, 112), which uses Felber’s model
for the common good matrix to help elicit values in an engineering project.

Depending on the types of competencies covered by the study, Acquiring cross-
cutting competencies can be generally mapped on Dublin descriptors “Make
judgment”, “Learning skills” and “Communication”.

Gaining awareness. The need to create awareness, in the students and various
stakeholders, of the role of software (and ICT in general) for sustainability
is common to many studies. Awareness is often mentioned with a broad
connotation, as stimulating the reflection on the role of software (ACM-10, 1,
121, 209, 23); the role of informatics in society, legal issues and ethical
professional responsibility (SB-82, 27); and the role of ICT industry and its
societal and environmental implications (IEEE-171, 113); or the conceptual
understanding and awareness of environmental issues (SB-07, 178). Lopez et al.
(IEEE-165, 112) propose a framework to help the students think about the
sustainability of their own thesis project.

Further, many studies aim at creating awareness of the specific types of
effects. For example, the mentioned enabling effects (i.e., SE for
sustainability) include the impact of ICT on society (CompSusEd-03, 106), and
case-study specific ICT impact on sustainability (IEEE-Penz2013cseet, 157).
Studies that explicitly target both direct (e.g., software energy footprint)
and enabling effects include, e.g., user participation and inclusivity
(CompSusEd-02, 219), creation of ICT solutions in resource-scarce contexts
(CompSusEd-03, 106), the reflection on ethical aspects of the profession
(IJSHE-12, 24), green and sustainable software engineering (SB-87, 210); and
the environmental impact of software, networks and services (SB-14, 97). Some
studies add the sustainability of the SE process (IEEE-153, 169, 170,
171)(IEEE-164, 231)(WoS-1, 126).

Gaining awareness can be generally mapped onto the Dublin descriptor
“Knowledge and Understanding”.

Gaining practical experience A significant number of studies propose letting
the students gain practical experience in various forms: by means of
integrative and transformative projects (ACM-12, 25); by facilitating
integrative learning through reflective writing (SB-89, 199); and by exploring
different ICT perspectives (IEEE-153, 169, 170, 171). More traditional
approaches include project-centered learning (SB-71, 128)(IEEE-161, 233),
case-based learning (IEEE-Penz2013cseet, 157), practical student assignments
(CompSusEd-02, 219); and using real-world sustainability problems, hard data
and technical methods (SB-92, 109).

Depending on the education level covered by the study, gaining practical
experience can be generally mapped on Dublin descriptors “Applying Knowledge
and Understanding” and “Communication”.

###  5.2. Topics being taught (SRQ2.1)

The topics being taught refer to the context of computing and sustainability.
In particular, we observe that teaching topics may be a collection, or even a
combination, of computing-specific topics and sustainability-specific ones.
Accordingly, computing and sustainability provide the context on which
teaching topics are organized.

Following this line of thought, the Computing-specific topics identified from
the studies entail the phases (e.g., from the software life cycle, like
requirements elicitation) and the contents (e.g., software models)
specifically related to the computing field. Sustainability-specific topics,
in turn, entail notions of sustainability and in particular how it is being
considered in a given educational setting (see Section 4 for different
conceptualisations). Finally, we have also identified a third type of somehow-
hybrid topics, which stem from the combination of computing and sustainability
and as such do not belong to one or the other but lie in their intersection
(e.g., green computing techniques). We call these Combined Computing-
Sustainability topics. The following reports on the primary studies addressing
these three types of topics.

Computing-specific topics. These most prominently regard the focus in a given
module, specifically on software development project management (ACM-10, 1),
user experience/human-computer interaction (UX/HCI) (CompSusEd-02, 219) or
measuring the energy footprint of ICT (IEEE-167, 213).

Sustainability-specific topics We use the PESTLE classification as a general
mapping of how the topic of sustainability is being understood in learning
objectives.555PESTLE stands for Political, Economic, Social, Technological,
Legal and Environmental. A PESTLE analysis aims to identify potential threats
and weaknesses from the perspective of the six factors above.
https://doi.org/10.1002/joe.21935 Accordingly, we identified studies that are
specific to various notions of sustainability: the environmental (IEEE-164,
231)(IEEE-167, 213)(IJSHE-5, 44), the social (social relevance of programming
(SB-53, 201)), and the socio-technical (the impact of inclusive HCI (SB-53,
201) or social learning (SB-99, 227)). Some others are all encompassing, by
addressing dimensions from the whole PESTLE classification (e.g., (IJSHE-12,
24)(Scopus-1, 47)) to sustainability in general (e.g., (SB-77, 53)). In
addition, under this umbrella we identified the following topics:

  * •

Climate Change. This content topic is addressed in many universities, some of
them more in depth using Systems Thinking approaches (CompSusEd-01, 54, 49,
103), and some of them more exploratory in projects (IJSHE-21, 181, 109). The
main teaching point is that climate change is present, complex, and immediate.
Krogsties and Krogsties include sustainability content in assignments, for
example by means of including personas with differing views on the climate
measures (SB-54, 103).

  * •

Sustainable Development Concepts and the SDGs. Several studies used the SDGs
as a source of inspiration to choose projects for students to work on, or
contextualize, the application of IT in a meaningful way. Argento et al.
(IJSHE-26, 8) found projects on-campus related to SDGs. Crompton et al.
(IJSHE-5, 44) helped students understand how the use of technology to meet
material needs of humans contributes to environmental effects. Stone [SB-8]
created introductory programming projects focused on sustainability topics,
such that students were exposed to the general concepts and terminology
involved. This is expanded in (SB-53, 201) with reflective writing. Koniukhov
and Osadcha (SB-46, 102) implemented ESD (Education for Sustainable
Development, UNESCO) principles in the process of training future software
engineers.

  * •

Ethics and Social Good. Several studies mentioned the importance of including
ethics and social good as context for IT projects. For example, Goldweber
(SB-03, 70) reports on 14 introductory projects in that context, from
radioactive mice to water pollution to voting simulation. Polmear et al.
(SB-15, 168) compare macroethics in teaching across cultures that can be drawn
from Hazas and Nathan (SB-62, 56) position students becoming ambassadors for
change.

Combined Computing-Sustainability topics. Some studies report on combined
contents providing competencies like green computing techniques (ACM-12, 25),
technology assessment to analyze technological developments and their
consequences (SB-48, 123) and how to help minimizing the energy footprint of
software solutions (SB-14, 97); or how to work in interdisciplinary settings
(ACM-06, 158) while fostering creativity (IEEE-161, 233). Under this umbrella,
we identified the following topics:

  * •

Environmental assessment, LCA and technology assessment. These topics entail
standards as well as concepts for environmental assessment (EA); life cycle
assessment (LCA); design for environment (DfE); and environmental decision
making (SB-30, 204). These topics include systems and sustainability
perspectives, systems performance analysis, and economic decision-making tools
as well as project evaluation. Furthermore, industrial ecology contributes to
the lens of Earth systems engineering and management, supplemented by LCA and
material flow analysis (SB-16, 224). Technology assessment considers how to
evaluate the outcomes and impacts of large-scale complex engineering systems
from a decision-maker’s perspective. There are legal, political, economic,
environmental and social frameworks that can be used to assess technology and
its impacts (SB-48, 123).

  * •

Green IT and Sustainable Computing. Green IT is the study and practice of
designing, manufacturing, using, and disposing efficiently of computers,
servers, and associated subsystems like monitors, networking and
communications systems (SB-29, 90) (see also the incremental perspective in
Section 4.2). This topic area includes energy-aware services, investments in
Green IT, measurements of energy efficiency and of software qualities, and
“Green Labs” (SB-04, 105). It can be expanded towards a circular economy,
smart systems, as well as Quality of Service and environmental requirements
and contracts (SB-14, 97).

  * •

Sustainability in/by IT. This topic area covers the three perspectives
described in Section 4.2. For Sustainability in and by IT, Hilty defines three
impact dimensions of ICT (first, second, and third order effects)
(hilty2015ict, 76). These impact dimensions are used in several publications
to explain the connection between ICT and sustainability (SB-01, 144)(SB-05,
116).

The biggest potential to motivate students to engage with the topic, according
to Hilty and Huber (IJSHE-11, 79), is an understanding that ICT is part of the
problem and the solution. They give examples for recycling of ICT, using ICT
to reduce greenhouse gas emissions, seemingly dematerialized economies, and
rebound effects that lead to increasing demand for goods or services.

Penzenstadler et al. (IEEE-Penz2011CSEET, 156) argue for making the link
between IT and sustainability explicit, and offer ways of how to teach about
it. (ACM-06, 158) and (IEEE-Penz2013cseet, 157) show examples of how to design
the requirements for a system such that they address the impact of
sustainability requirements on IT (i.e., requirement analysis).

###  5.3. Educational Organization (SRQ2.2)

In the following, we present “how” sustainability education in computing is
being implemented in terms of two elements: (i) pedagogical approaches, or the
teaching methods that are used to realize the learning objectives described
before, and (ii) the “building blocks” in which sustainability education is
organized, i.e., modules and curricula.

####  5.3.1. Pedagogical Approaches

We could trace two aspects in the presentation of pedagogical approaches: (i)
practices to integrate different fields of knowledge, and (ii) pedagogical
methods in sustainability education. They are described in the following.

Practices to integrate different fields of knowledge. Several of the primary
studies include reflections on pedagogical approaches to integrate different
fields of knowledge (ACM-10, 1, 154, 168, 224, 204, 119, 125, 123, 56, 214,
27, 219, 113). Multiple papers argue that the students’ discipline needs to be
combined with domain knowledge of the various dimensions of sustainability
(SB-30, 204, 224, 168), and that specifically, ethics (SB-30, 204, 119, 27) or
other subjects from the humanities (SB-33, 125) are central to the integration
of holistic perspectives on engineering and management. Two such approaches
emerged from our analysis:

  * •

Multidisciplinary student groups. This approach is about the composition of
student groups. Multidisciplinary groups of students are advocated, allowing
for a more holistic understanding of sustainability issues (CompSusEd-02, 219,
123). One example is a collaboration between IT and Material & Earth Sciences
(IJSHE-12, 24). Another one (IJSHE-21, 181) uses a question-based approach to
select topics that then require looking into different disciplines.

  * •

Engaging external actors. Several of the studies report on the value of
engaging external actors (SB-01, 144, 105, 157, 81, 24, 219, 38, 97, 127,
205). Collaboration with external stakeholders is either reported as being
done through entrepreneurial approaches (IJSHE-12, 24), through cooperation
with the industry or public sector (CompSusEd-02, 219, 38, 97, 127), or
through the local community (IEEE-163, 205, 24). Representatives from these
sectors can give guest lectures, as e.g., industry representatives in
(IEEE-172, 171). Students become active agents of change in the local
community when working with external stakeholders, as e.g., expressed in the
following: “Entrepreneurial experiences is an educational framework
encouraging students to have the competencies to make a positive change in
their own context, the ability to assess opportunities and resources for
making a difference and the courage to act accordingly” (IJSHE-12, 24).

Pedagogical methods in sustainability education. The primary studies report on
different methods, that is different ways of engaging students in learning
activities and promoting skills, attitudes and behaviors assumed to be
associated with learning for a sustainable development:

  * •

General learning approaches. These are meant to help students become active
participants in their modules, which are not necessarily specific to learning
for sustainable development. In this category, the studies report on using
student-centered active learning (SB-34, 81), discussions and reflections
(CompSusEd-02, 219, 172, 144, 171), preparation and classification of ready-
made project ideas on computing for the social good (SB-03, 70), experiential
and reflective learning activity and reflective reports (ACM-11, 154), a
“Sustainable programming pedagogical model” with reflections (SB-89, 199), and
flipped-classroom for critical thinking.

  * •

Games, simulations or role-playing exercises. Such exercises for
sustainability were described in five studies (CompSusEd-01, 54, 110, 123,
151, 59). Some use custom board games designed to build experiences related to
understanding norms, values or complex systems (SB-64, 151, 110); some use
simpler playbook formats featuring smaller exercises fit for classroom use
(ACM-01, 59, 54); and one reports on role-playing and “structured academic
controversy forum” (SB-48, 123). Eriksson et al. (CompSusEd-01, 54) use in
their CS module the systems thinking games Living Loop and Harvest from the
“Systems Thinking Playbook for climate change” (Sweeney2010, 203), to let
students experience the concepts detailed in the preceding lectures.
Easterbrook (SB-40, 49) reports on a seminar series where the students
“collectively tease out the systems concepts exhibited by the game, along with
their own reactions to the overall behavior of the system”.

  * •

Problem-based learning, course projects and hands-on experience. These
pedagogical methods were described in fifteen of the primary studies (SB-01,
144, 116, 224, 125, 81, 56, 127, 146, 158, 133, 24, 181, 88, 205, 171). In
these methods, the students engage with and take responsibility for real-world
issues; they also take responsibility for their own learning while working on
the issues. The computing students practise what is emphasised as important
for engineers by Hazas and Nathan (SB-62, 56), i.e., taking responsibility,
Corporate Social Responsibility, and communication. Miñano Rubio et al.
(SB-33, 125) describe course projects as an “important leverage point for the
integration of sustainability”. The projects are contextualised in different
ways, e.g., as a part of a longer education and vertically integrated through
longer periods of studies (IJSHE-10, 133), or integrated with other
disciplines in interdisciplinary projects (ACM-06, 158, 181). The concept
“learning by doing” is used and extended into “learning by living”. The
students apply and learn about software technology in specific applications,
e.g., gardening (growing your own food) (ACM-06, 158). In one example,
students changed lifestyle for a shorter period of time and reflected on the
outcome, i.e., the students explore the notion of sustainable living in their
everyday life, and experience first-hand the related choices and challenges
(IJSHE-8, 172). In addition, connecting computing to social sustainability can
bring sustainability to life (IJSHE-6, 193, 70). Mann (SB-05, 116) proposes an
educational model for transformative action, so that students act consciously
and responsibly within their professional practice.

####  5.3.2. Building Blocks: Modules and Curricula

The realization of sustainability in computing education is discussed in terms
of the changes required to modules or curricula. Some authors aspire for
radical change (SB-01, 144, 116) and propose to rethink education as whole, as
“the solution is not to try and cram more material into an already crowded
curriculum, but rather to see sustainability as the context, a basis for
deeper learning, or even a reason for learning” (SB-05, 116), and therefore
“rather than carefully aligning sustainability with the discipline, we need to
be pointing out how much change is required” (SB-05, 116). Teaching in ways
that allow for and encourage activism, also expressed as “development
activism”, may be one approach to achieving radical change through education,
for example as in “Dissent 101” (Huish2013, 80). The scope of the studies
varies from individual modules to the design of full curricula.

Modules. The primary studies report on specific modules through which
sustainability education is implemented, e.g., teaching general sustainability
competencies (ACM-11, 154) or cross-cutting skills like systems thinking
(SB-40, 49). The studies provide guidance for module design, for instance
thesis projects about sustainable engineering (IEEE-165, 112). Even though
many of the primary studies do not inform about the curricular level in which
their proposal is being taught, most works seem to focus on the BSc and master
levels. The extent to which the integration of sustainability was performed
varies. In a lighter mode, we found efforts to teach sustainability in IT via
one or several workshops (IEEE-156, 38) or seminars (SB-87, 210, 154, 25),
assignments (CompSusEd-02, 219), projects (SB-92, 109, 200, 116, 125), or
through modules in modules (SB-07, 178, 210, 199, 201, 157). Penzenstadler et
al. (IEEE-Penz2011CSEET, 156) describe how integrating sustainability can be
achieved through minor changes, e.g., in seminars, and how those minor changes
can be a seed for promoting the value of sustainability to students, which
might lead to more transformational changes of computing education later on.
Easterbrook (SB-40, 49) informs about their determination to educate all their
MSc and BSc students about sustainability and reports on how they increase
sustainability-related aspects in different modules, particularly in those of
their Human-Technology Interaction MSc programme.

A few primary studies propose whole new modules such as a module on green
computing (ACM-12, 25, 59, 54, 205, 146, 214, 149, 55, 90, 44), introductory
modules on software & sustainable development (SB-75, 57, 23, 204, 150), final
year project module on sustainability (IEEE-165, 112), “software engineering
sustainability” (IEEE-167, 213, 110), and ethics (SB-82, 27). Easterbrook
(SB-40, 49) proposes three modules, on climate informatics, on the social and
environmental impacts of the internet, and on systems thinking for global
problems.

Curricula. Curricula embed sustainability either as a central focus e.g.,
(CompSusEd-03, 106), or as general design guidelines e.g., (IEEE-147,
218)(IEEE-164, 231). The level of the curricula at which sustainability
education in computing is implemented varies from K-12 (IEEE-156, 38), to BSc
and MSc (SB-40, 49), to PhD degrees (Scopus-1, 47). Several primary studies
report on how sustainability is being implemented across the curriculum, or
programme (ACM-10, 1, 218, 231, 113, 127, 128, 224, 125, 81, 104, 49).
Examples include the integration of “Sustainability and Social Commitment”
skills (SSC) in different degree modules in a 4-year Computing Engineering
degree (IEEE-171, 113, 171, 157), “greenifying curricula” (IJSHE-6, 193, 231,
126, 168, 97, 105, 8), a 1-year IS Master program “with a special focus on the
Digital Society and Sustainability” (CompSusEd-03, 106), or the blueprint of
the Erasmus Mundus Joint Master Degree in Pervasive Computing and
Communications for Sustainable Development (IEEE-172, 171, 170). Miñano Rubio
et al. (SB-33, 125) propose a model for the holistic and systematic
integration of sustainability competencies into engineering curricula that can
serve as guidance.

Finally, Mann et al. (ACM-14, 121) develop recommendations for policies that
can catalyse the integration of sustainability in all computing programs
across institutions. The vision of this work is to create “a philosophy of
Computer Education for Sustainability” that “will be enhanced if undertaken
within a context of institutional operational practice” (p.191).

###  5.4. Effects of Sustainability Education (SRQ 2.3)

The effects of teaching sustainability have been studied from two different
perspectives: (i) what researchers/educators reported that their students
demonstrated to have learned (i.e, the achieved learning outcomes), and (ii)
other effects that were reported as a consequence of implementing their
pedagogical initiatives.

####  5.4.1. The Achieved Learning Outcomes

The achieved learning outcomes were categorised in the themes shown in Table 3
together with a description of their meanings and the primary studies in which
they were witnessed. For each theme of learning outcomes, the first column
(and the descriptions below) refers to the categories of learning objectives
that they respond to (see horizontal pillars in Figure 3). Following the
principle of constructive alignment (Biggs2003, 18), our aim is to help
educators align the intended learning objectives and the achieved learning
outcomes. (See also the discussion in Section 5.4).

Achieved cross-cutting competencies. The primary studies report on the
achieved learning outcomes that can be interpreted as cross-cutting
competencies:

  * •

Increased Awareness. The most documented effect of sustainability education in
computing is an increased awareness of what sustainability is and the
sustainability issues humanity is facing. For example, Stone reveals that “a
majority of participating students reported that the course and its
assignments helped them have a greater understanding and appreciation for
sustainability” (SB-08, 200), an observation similarly made in (SB-53, 201,
86, 110). Furthermore, (SB-92, 109) reports that one student “claimed not to
have understood the full implications and scope of climate change before, and
said that his understanding had improved much”. The students seemed surprised
how much they learned and how little they knew before (IJSHE-13, 232).
Students also enter and select sustainability education with an awareness and
desire to learn and care for the environment and state they become even more
aware during the education (IJSHE-5, 44). In (SB-57, 149) one can read how
some students went from doubting climate change to expressing deep concerns
for the future of mankind.

  * •

Restlessness. Teaching sustainability was sometimes reported as raising on
students an overall feeling of restlessness (SB-19, 150, 149, 27). Students
expressed deep concerns, anxiety and fear about our global future. Concrete
evidence suggests that being thoroughly exposed to real-world sustainability
issues can lead to uncertainty on whether it will be possible to address those
issues: “The revelations have made me more anxious about life and our future.
It’s good stuff but I would probably be happier without it. Not knowing is
bliss.” (SB-19, 150). Further evidence reports even more profound impacts,
where several students have expressed feeling almost depressed halfway through
the course: “The course can give a feeling of hopelessness.” (SB-57, 149). As
discussed in (SB-57, 149), this effect suggests that the students were
emotionally highly engaged in the sustainability topic.

  * •

Mindset Change. In addition to an increased awareness of the topic of
sustainability, the primary studies present a change in the mindset of the
students as a learning effect (IJSHE-13, 232, 149, 86, 27, 109). Issa et al.
(SB-69, 86) received formal and informal feedback from their students and
concluded that their unit shifted students’ mindset. The students recognised
the potential damage of IT systems and were oriented to “using IT wisely”
including minimizing IT usage. In (SB-82, 27), one can find direct feedback
from students that sustainability education has changed their view of the
field of Informatics and made them reason about aspects that never crossed
their minds. Similarly, one student admitted to thinking about the people
talking about climate change as extremists before taking the course, a vision
that changed after taking it (SB-92, 109). When asked to write a newspaper
headline that would reflect any change in their thinking about sustainability
as a result of attending the course, the students described in (IJSHE-13, 232)
submitted headlines that reflect how confronting the course was for some of
them; a concrete example being: “I am Lost!! – Please Show Me the Way to
Sustainability”.

  * •

Increased Motivation. Students have demonstrated an increased motivation to
adopt a more sustainable practice or attitude as an effect of sustainability
education (ACM-12, 25, 232, 201, 149, 86). We find expressive statements such
as the “most significant [change this course has brought] has been the
reawakening of an ambition to make a difference” (IJSHE-13, 232). The increase
in motivation expressed by students also encompasses their professional
practice: students who concluded a course on green computing mentioned a
strong enthusiasm for green computing as something they will apply in their
profession (ACM-12, 25). Finally, one can read from a student that: “I was
interested in marketing and stuff like that before the course, but now I feel
like doing something that is more beneficial to humanity” (SB-57, 149).

  * •

Multidimensionality of the Problem. One of the challenges of addressing
sustainability is that it entails multiple dimensions that need to be
addressed in an holistic way. One of the reported effects of teaching
sustainability was that students learned the multidimensionality of the
problem at hand (SB-19, 150, 149, 232). In fact, besides understanding the
multiple dimensions of sustainability, students have also shown to understand
that the different dimensions are often in conflict with each other,
specifically with the economic dimension.

Pillar |  Theme |  Meaning of the category |  Primary studies
---|---|---|---
|  (achieved learning objectives) |  Adding sustainability to education … |
Gain awareness |  Increased awareness |  …improved the awareness among students of the global sustainability issues the world faces. |  (SB-07, 178, 201, 86, 110, 109, 1, 154, 25, 232, 44, 200, 149, 187)
Gain awareness, Acquire cross-cutting competencies |  Restlessness |  …led students to express deep concerns, even anxiety, about the current (lack of) sustainability of the global trends. |  (SB-19, 150, 149, 27)
Acquire cross-cutting competencies |  Mindset Change |  …led students to radically change their perspective about sustainability, its causes or potential solutions. |  (SB-69, 86, 27, 109, 232, 149, 187, 110)
Acquire cross-cutting competencies |  Increased Motivation |  …led students to express their willingness to adopt a more sustainable practice or attitude. |  (SB-53, 201, 86, 25, 232, 200, 149)
Acquire cross-cutting competencies |  Multidimensionality of the Problem |  …led sustainability to be recognised as a multidimensional problem, and one whose dimensions are often in conflict. |  (SB-19, 150, 232, 149)
Acquire cross-cutting competencies |  Negative Impact of ICT Systems |  …linked the computing discipline and its impact on climate change and other sustainability challenges. Students understand how computing can harm sustainability. |  (SB-69, 86, 1, 172, 55, 110)
Acquire topic-specific competencies, Gain practical experience |  Positive Impact of ICT Systems |  …showed the potential of computing for addressing sustainability challenges. Students understand how computing and how their computing skills can contribute to improving sustainability. |  (SB-53, 201, 103, 86, 110, 109, 1, 25, 172, 55, 187, 28)
Gain practical experience |  Lifestyle Change |  …made students change their personal, or their community’s, practices and attitudes into more sustainable ones. |  (SB-53, 201, 44, 172, 55)
Table 3. The achieved learning objectives mapped on the pillars of Figure 3
and general themes

Topic-Specific Competencies. The primary studies reported that the students
learnt about the following topics:

  * •

Negative Impact of ICT Systems (or the use thereof). The instructors of a
Master degree course on sustainability and Green IT describe that “Students
were stunned by the damages of information technology usage locally and
globally” (SB-69, 86). Similarly, having analysed the answers of the
retrospective survey of a course addressing Sustainability for media
technology students, authors acknowledge that they were successful in
explaining the connection between ICT and climate change (SB-45, 55).
Furthermore, in another retrospective survey, students assessed the statement
“Information Technology […] causes new problems to arise that must be
addressed.” with an average of 4.62 on a 1-to-5 Likert scale (ACM-10, 1).
Finally, in their reflections over the post-course feedback provided by
students, the instructors of (IJSHE-8, 172) note that, even if reported in
limited cases, students demonstrate having understood that the convenience
provided by technology can lead to unsustainable habits–one clear example
being online food ordering systems.

  * •

Positive Impact of ICT Systems. We have also found numerous examples that show
that students learned about the potential positive impact of ICT systems,
i.e., about the potential contributions that these can have to address
sustainability issues (ACM-10, 1, 25, 172, 55, 201, 103, 86, 110, 109).
Concrete feedback from students includes “This unit is very important for
future students to make the changes in the IT world” (SB-69, 86); and that the
use of climate models provided them with new perspectives on what their
contribution as future engineers can be (SB-92, 109). Similarly, we observed
that students acknowledged that sustainability is a concern for both the
consumers and the designers of IT systems (IEEE-New-3, 110). The majority of
respondents on a retrospective survey also agreed to strongly-agreed that
thanks to the course they understand the applicability of programming to solve
complex social problems (SB-53, 201). Concurrently, students strongly agreed
that “IT makes potential solutions to the problems of sustainability available
[…]” (ACM-10, 1). The use of technology has also been described by students as
a way to assist them in living more sustainably, namely by using mobile
applications that track their travelling carbon footprint (IJSHE-8, 172).

Practical Experience in the form of Lifestyle Change. The primary studies
report that the students learn about and test concrete lifestyle changes
(IJSHE-5, 44, 172, 55, 201). This could be interpreted as the cross-cutting
competence “gaining practical experience”. Interestingly, while the education
was focused on the relationship between computing and sustainability, the
changes most often refer to daily activities and/or community practices
instead. In (IJSHE-5, 44), one can find various concrete statements along
these lines, and in particular regarding:

  * •

Travel patterns. “I now cycle to work”, “Cut down on unnecessary journeys”,
“Have scrapped our second vehicle and use public transport or walk instead”,
and even “Moved house to reduce travel to and from work”.

  * •

Consumption of energy, materials and food. “We now recycle cardboard and
plastic as well as glass and cans. We shop for food now with an awareness of
’food miles’ and unnecessary packaging and what’s in-season”, “Buy only two
newspapers per week, […], compost all relevant household waste, change all
lighting to low energy”. Concurrent observations in consumption patterns can
also be found in (SB-45, 55) and (SB-53, 201). Further evidence of improved
practices can also be found in (IJSHE-8, 172), most of which reflect
incremental (instead of radical) changes. We should mention that lifestyle
changes are sometimes acknowledged as not being easy to achieve: (IJSHE-8,
172) describes situations in which students admitted to struggling to adjust
to new lifestyle choices, but eventually persisted on them and were successful
in taking (self-admitted) small steps. Finally, the majority of respondents of
a survey in a study that used sustainability-themed projects in introductory
programming modules have also agreed/strongly-agreed they can now see the
potential application of sustainability practices in their own communities
which was facilitated by the assignments and activities of the module (SB-53,
201).

####  5.4.2. Other Effects of Sustainable Computing Education

The effects we describe in this section are effects that we extracted from the
primary studies but go beyond the intended- or achieved learning outcomes.
Similarly to the previous Table 3, we extracted and classified such effects
into general themes (see Table 4) described in the following.

Changing student demographics. Integrating sustainability into computing
education makes computing education attractive to very diverse groups of
prospective students. For example, while computing disciplines have
traditionally been rather male dominated, this is changing when sustainability
related contents are emphasized (IEEE-175, 170). Stone (SB-08, 200) reports
that “the inclusion of socially-relevant projects and course themes has been
shown to assist in attracting students to Computer and Information Science,
and practical, problem-based applications have also been shown to attract
females and underrepresented groups to the discipline”.

Pillar |  Theme |  Meaning of the category |  Examples of primary studies
---|---|---|---
(educational organization) |  (achieved learning objectives) |  Adding sustainability to education … |
Pedagogical approaches |  Changing student demographics |  …attracts new types of students and may change the demographics of the student groups. |  (SB-08, 200, 170)
|  New career opportunities |  …provides new opportunities in work life |  (SB-32, 96, 171, 170, 56)
Modules |  New proposals and openings |  …will eventually lead to new modules, curricula and even policies. |  (SB-21, 120, 144, 127, 171, 170, 113, 133, 121, 125, 210)
Table 4. Other effects mapped on general themes

New career opportunities. The integration of sustainability in computing
subjects has been shown to provide students with new skills and competencies
(SB-71, 128, 214, 106). As such, they also create new career opportunities as
shown in (IEEE-175, 170) and (IEEE-172, 171). These new careers can be linked
with the general evolution of society under the Sustainable Development
Challenges and Climate Change in addition to required 21st century skills
(CompSusEd-03, 106).

New proposals and openings. The successful integration of sustainability into
a subset of computing courses can eventually lead to new proposals and
openings that may change a discipline. An example can be found in the
evolution of the Educational framework for the Information Sciences masters
programs, which was revised in 2016 to include sustainability (MSIS2016, 208).
In some of the primary studies, the proposals focus on identifying elements
needed for a core sustainability course (SB-01), curricula structure or
content (SB-33, 125, 127, 133) or even a policy/approach for including
sustainability into education (SB-21, 120, 121).

###  5.5. Takeaways for RQ2

Building upon the results presented in Section 5, we observe that Figure 3
suggests a Framework for Implementing Educational Components. As illustrated,
sustainability in computing education can be implemented by drawing on the
horizontal pillars (which summarize the identified intended learning
objectives), and the vertical pillars (which summarize the topics pertaining
sustainability in computing, and the educational organization).

In reflection, we argue that educators could build upon Figure 3 and use it as
a framework to design their educational components by identifying the
ingredients from each of the vertical and the horizontal pillars. For the
vertical pillars, we could start defining the ingredients for the desired
educational organization (Are we designing a full curriculum or a single
module? Which pedagogical approach(es) do we find the most appropriate?), and
then identify the specific topics we want to teach for the computing- and
sustainability learning objectives (Which phase(s) do we want to cover? Which
computing-, sustainability-, and combined topics? And for which notion of
sustainability?). For the horizontal pillars, we could ask ourselves which
generic competencies the students should acquire. Our work and the
illustration could also inform an analysis of existing programs.

Once an educational activity is being conducted, we can assess the achieved
learning outcomes by evaluating the coverage of the various pillars. We can
also use the figure to reflect on the implementation of education in a larger
sense, e.g., reflecting on what modules are missing or the pedagogical
approaches that are lacking. Such a framework could also be an instrument to
accommodate change in the way we design and run education.

##  6\. Research Design (RQ3)

A study’s research design provides the blueprint for the collection,
measurement, and analysis of data, and aims to reduce the bias and increase
the trust and reliability in the accuracy of the data collected for the
research phenomenon or problem under investigation (Coe2021, 29). In RQ3, we
investigated the research design being conducted by the community against the
proposed ACM Empirical Standards (Ralph2020, 173) guidelines with respect to
the overall research design including the following sub-research questions:
(i) the epistemological stance underpinning the research design, (ii) the
research method(s) used and to what extent the method(s) included core
attributes of the approach, and (iii) what threats to validity were
considered, which potentially jeopardize the reliability of the results and
the confidence in the conclusions that can be drawn.

###  6.1. Epistemology (RQ 3.1) and Threats to Validity (RQ 3.3)

Based on the results of our analysis we consider these two sub-research
questions together rather than separately as a result of the brevity of the
data that was identified. Sub-research question 3.1 investigated the
epistemological stance adopted in the selected research literature.
Epistemology is the “theory of knowledge” and influences how that knowledge is
collected and from which sources” (Creswell2018b, 43). In research terms, it
is argued that the philosophical standpoint should be made clear from the
beginning, as a researcher’s view of the world and knowledge strongly
influences the choice of methodology and methods, and interpretation of data
(kelly2012, 93). Within epistemology, there are several approaches and
branches, such as positivism and interpretivism, which are diametrically
opposed. A thorough treatment of the relationship between ontology,
epistemology, methodology, and methods is beyond the scope of this paper, and
readers are referred to Bridges (Bridges2017, 22) for a more in-depth
discussion of the relationship and importance of epistemological stance and
research design. During the analysis, we found no studies that [explicitly]
stated an epistemological stance, which is considered a key founding principle
in guiding research in education, underpinning the consequences for knowledge
construction within the research community (Elmore2012, 52). This leads to the
question of whether it is possible to derive an implicit epistemological
stance through a reverse mapping between method and epistemology; however,
that is beyond the scope of this already extensive work. The absence of an
[explicit] epistemological stance raises important questions regarding
research design overall, and the reliability and generalizability of the
results as it is concerned with all aspects of the validity, scope, and
methods of acquiring knowledge. In addition, it influences how researchers
frame their research in their attempts to discover knowledge and the extent to
which its transferability can be assessed.

Sub-research question SRQ3.3 investigated the threats to validity in the
identified research literature. Threats to validity refer to specific reasons
as to why researchers can draw incorrect conclusions when they make an
inference in an experiment because of covariance, causation constructs, or
whether the causal relationship holds over variations in persons, setting,
treatments, and outcomes (Creswell2020, 42). While the concept of threats to
validity has evolved over the years from the initial discussions by Campbell
and Stanley (Campbell1963, 26), Shadish, Cook, and Campbell (Shadish2002, 190)
identified the potential threats to validity, and the statistical procedures
of educational experiments, that are generally considered in modern-day
research including statistical conclusion validity, construct validity,
internal validity, and external validity. From an educational research
perspective, Creswell (Creswell2018a, 41) considers two primary threats
critical to consider: internal validity and external validity, where internal
threats relate to drawing appropriate inferences to the actual design and
procedures used in an experiment, while threats to external validity are
problems that threaten a researcher’s ability to draw correct inferences from
the sample data to other persons, settings, treatment variables, and measures.
According to Cook and Campbell (Cook1979, 31), three threats may affect this
generalizability: the interaction of selection and treatment, the interaction
of setting and treatment, and the interaction of history and treatment. During
the analysis, we found N studies that explicitly considered threats to
validity in their research design, which are considered key characteristics of
experimental research in education (Creswell2020, 42). The general absence of
threats to validity raises important questions regarding research design
overall and the reliability and generalizability of the results.

###  6.2. Methods (RQ 3.2)

Sub-research question 3.2 investigated the methods adopted in the primary
studies and to what extent they aligned with the essential attributes
characterizing each type of empirical study. Table 5 shows the number of
studies that belonged to each type of empirical study and the number of
essential attributes we used to evaluate the quality of each paper. We used
the classification from the ACM (Ralph2020, 173) to categorize the type of
empirical studies such as shown in Table 5.

Type of Empirical Study | Number of Studies | Number of Essential Attributes
---|---|---
General standard | 59 | 16
Systematic review | 2 | 13
Questionnaire survey | 11 | 18
Interview study | 5 | 10
Case study | 13 | 12
Table 5. Number and type of studies classified according to ACM Empirical
Standard (Ralph2020, 173)

General standard This general standard applies to all studies that collect and
analyze data. The results of our analysis identified that the majority of the
studies were classified into the General standard category as reported in
Table 5. Focusing on the research type of the papers within the category, we
classified them into the following sub-categories: (i) a vision/proposal paper
describing an educational plan for the future, (ii) a position paper
recommending a course of action, (iii) an experience report, and (iv) a
proposal of something already happening. From our analysis, we found that the
majority of the studies classified in this General category can be considered
“Experience reports” (ACM-11, 154, 25, 54, 169, 189) or “Proposals”
(CompSusEd-02, 219, 38, 218).A small number of papers were classified as
“Vision” papers, such as (IEEE-161, 233, 150). With regards to the attributes
that describe the purpose and objective of the research questions we found
some diversity. For example, in the experience papers the studies report that
the most common problems belong to the following topics: (i) study the impact
of sustainability software engineering education (ACM-11, 154, 170, 133), (ii)
develop green computer courses (ACM-12, 25, 205, 105), (iii) studies
suggesting system thinking games, creativity and practical assignments to
introduce sustainability (CompSusEd-01, 54, 157), and (iv) curriculum design
to introduce sustainability competencies in software engineering or computer
science courses (SB-21, 120, 81). For proposal papers, we also found similar
topics to that of the experience report papers such as assignments related to
various sustainability topics ((CompSusEd-02, 219)), curriculum design and
competencies ((IEEE-147, 218)) as well as proposals suggesting how to teach
sustainability and social skills ((IEEE-171, 113)), green computing models for
sustainable development (IEEE-164, 231, 105, 90), and introducing
sustainability into bachelor courses (IEEE-Penz2011CSEET, 156, 144, 117). The
vision paper (SB-19, 150) discussed future ways to teach sustainability based
on an introductory course in sustainability and ICT, while another vision
paper (WoS-new-1, 162) describes an educational plan of sustainability in
computing for the future.

Another essential attribute we analyzed was if the methodology was appropriate
for the goal or problem stated. A significant number of studies did not
clearly report the methods used for experience report papers (CompSusEd-01,
54, 120) or were “not applicable” in the case of position papers (IEEE-147,
218, 169). The reason is that we noted that some vision or proposal papers
under this category did not carry out any research, and consequently including
a research method does not make sense. Other papers partially report the
methods used somewhere in the paper (ACM-11, 154, 25) but there is no explicit
methodology section that describes the methods employed for data collection
and analysis. Finally, there are a set of papers that clearly include a clear
methodology section used in the studies (IEEE-Penz2013cseet, 157, 133, 172,
97).

With regards to the what, where, when, and how data were collected, only some
studies in this category indicated fully or partially how and where the data
was collected (ACM-11, 154, 25, 54, 106, 97, 120), while the others did not
provide such information or was “not applicable” according to the study. Also,
many of the studies provided the number of subjects that participated in the
experiments or surveys ranging from few number of subjects (e.g. 16 in (SB-14,
97) or 20 in (ACM-11, 154)) to a huge number participants (e.g. 1270 in
(IEEE-175, 170) or 2420 in (IEEE-164, 231)). Other studies did not indicate
the number of subjects. Surprisingly, for the rest of the 11 essential
attributes defined for this category (e.g. discuss threats to validity,
clarify the role of the researchers, innovates the research methodology or
discuss statistical power or saturation), in the majority of the papers
analyzed we could not find this information or this attribute was “not
applicable” based on the type of study being reported, e.g. experience.

Systematic review: In this category, we defined 13 essential attributes but we
only found two studies (IEEE-145, 88, 96). (IEEE-145, 88) study is a
systematic literature review (SLR) that investigates research trends on
sustainable development for training software engineers on sustainable
principles, while (SB-32, 96) is a survey that provides an educational
landscape of green ICT and sustainable computing. Both studies partially
describe the problem objective or research question and both outline an
appropriate methodology. Only (SB-32, 96) provides the information regarding
data collection. The rest of the essential attributes defined in the ACM
standard have not been included in (IEEE-145, 88). However, in the case of
(SB-32, 96) the authors mention the detailed steps and replication package of
the process, the inclusion and exclusion criteria, and the data extracted from
the primary studies.

Questionnaire survey: In this category we classified eight studies where open-
ended questions are systematically analyzed. Several types of goals are
investigated. For instance, in (SB-15, 168) the authors mix a questionnaire
survey with qualitative questions about ethical aspects that include
sustainability. Ethical aspects are also investigated in (IEEE-170, 16) to
illustrate how sustainability practices include ethics into teaching. The
study uses open-ended questions to extract additional thoughts from the
participants. In (ACM-14, 121) a survey compares the understandings of
sustainability using different benchmarks and discusses the attitudes of the
participants towards integrating sustainability in Computing. Another work
(ACM-New-2, 187) investigates the use of green computing techniques and how
green IT topics are taught in CS courses. About the rest of the essential
attributes that characterize this category, only (ACM-14, 121) did not include
the research questions while (ACM-07, 209, 16, 121, 168, 119) included these.
Additionally, all of the studies indicated an appropriate research
methodology, described how and where the data was collected and how the data
was analyzed (except (ACM-14, 121)). We did not find evidence for the rest of
the essential attributes (e.g. how responses were monitored or managed,
analysis of response rates, or measuring the constructs using validated
scales). Only some of the studies partially indicated that, for instance, the
threats to validity (SB-15, 168), described systematically the replication of
the study (ACM-14, 121), or described how the participants were recruited
(IEEE-170, 16). Only two studies (ACM-14, 121, 209) detailed how the
questionnaire instrument was created.

Interview study: In this category we only found two studies (IJSHE-6, 193,
178). The first study reports semi-structured interviews from programme
leaders and teachers about how social sustainability can be integrated and
taught, while the second combines quantitative and qualitative analysis to
explore the awareness and conceptual understanding of environmental issues in
students enrolled in an introductory computing course. The interview study
investigated how social sustainability is operationalized in engineering
programs and the resources needed to support social sustainability educators
as a learning objective. In this second study, two groups of 137 CS/IT
students were asked about 20 questions on student’s computer use and green
computing. Both studies indicated the research questions and the methodology
while partially indicated the details of the data analysis process and the
selection process of the interviewees. Only (SB-07, 178) described the
demographics of the participants while (IJSHE-6, 193) included the experience
of the subjects. Finally, none of the studies discussed how saturation was
achieved or included the threats to validity. Notwithstanding, in (SB-07, 178)
we found some discussion around the possible biases reflected by the
researchers.

Case study: In this last category we found eleven studies that cover different
topics. For instance, (SB-08, 200) is a two-year case study aimed to create
introductory courses focused on sustainability topics, while (SB-16, 224) uses
a tool to assess universities’ curricula in sustainability development and two
surveys were conducted to investigate the results of the assessment of
sustainability courses in environmental engineering curriculum at a university
in the US and the students perceptions of the contribution of such courses to
sustainability. Other studies analyze the effect of teaching sustainability
topics in regular courses, such as (SB-30, 204) which reports the results of a
sustainability course taught along 3 years and the effectiveness for teaching
sustainability, while (SB-33, 125) analyzes the learning guidelines (i.e. a
document that describes the course details) of degree programs in informatics
and industrial engineering in 25 Spanish universities and proposes a
curriculum model to embed sustainability into engineering education. Other
studies like (IEEE-New-1, 28) analyze 45 project proposals to identify social
and technical sustainability aspects and how sustainability topics are
introduced in customer’s project descriptions.

From our analysis of the rest of the essential attributes characterizing case
studies, we found that 3 out of 11 studies (IJSHE-11, 79, 232, 200) included
the research questions but none of the studies provided examples for the
questions investigated. Conversely, all the studies indicate, fully or
partially, a proper research method for each case study and all except two
(ACM-10, 1, 204) indicated how and where the data was collected. Compared to
the other categories, the studies indicated in most cases that the rest of the
essential attributes were covered or partially covered excepting some case
studies that did not include some of these attributes. For instance, studies
like (ACM-06, 158) did not include the demographics of the participants and
study (IJSHE-11, 79) did not describe the site in rich detail. Others such as
(SB-30, 204) and (SB-33, 125) did not justify the selection of the case study
while only three studies (ACM-06, 158, 79, 200) included some description for
the threats to validity. For the rest of the studies, most of the essential
attributes were detailed.

Summary: Less than half of the papers we analyzed actually present empirical
research. It is challenging to summarize the quality of the studies for each
category and for many different essential attributes that should characterize
each of the studies. Apart from a few papers that could belong to two
different categories, we found some common patterns in our analysis revealing
the quality of the studies analyzed. There are a significant number of studies
that, according to ACM guidelines (Ralph2020, 173), do not describe the
majority of the essential attributes. This is especially bad in the General
standard and Questionnaire survey categories. However, we cannot provide
significant conclusions for Systematic reviews and Interview studies as we
only found a small number of papers in these two categories. On the other
hand, the Case study category exhibits good results for the papers classified.
We found most of them describe the majority of the essential attributes so it
seems more easy to classify papers under this category than for the others. We
statistically summarize in Figure 6.1 our findings attending to the quality of
the studies and the number of essential attributes supported and not
supported. It should be noted that we did not include in the count the
attributes labeled as “not applicable”. Also, in the left side of Figure 6.1
we show the ratio of attributes supported per study type with respect to the
number of papers for each category.

Figure 4. The number of essential attributes covered for the different types
of research methods.

###  6.3. Takeaway Lessons (RQ3)

This section provided an analysis of the types of studies that have been
conducted with an emphasis of the specific design of the research including
the procedures involved in them, such as sampling, data collection instruments
and protocols, etc. In the absence of any generally agreed standard for
evaluating the quality of the research design, we classified each study using
the ACM Empirical Standards (Ralph2020, 173) as a framework to aid in the
identification of the essential, desirable, and extraordinary attributes for
empirical research for a particular method. Alternative perspectives on
evaluating the quality of research have been proposed by Lincoln (Lincoln1995,
111), Tuckman (Tuckman1999, 212), Richardson (Richardson2000, 177), and
Creswell and Miller (Creswell2000, 39). In combination, they offer
philosophical, procedural, and reflexive criteria to use in evaluating
research design. However, Creswell (Creswell2014, 40) argues that as a result
of the fundamental differences between methodological approaches to research,
each merits its own criteria for evaluation. Because of diversity in the
various approaches and subject areas, there is no single Gold standard for
determining the quality of research design but there is a clear need for the
research community in this area to consider moving towards adopting or
developing a common standard by which research output can be transparently and
consistently evaluated against. Our analysis highlights that the majority of
studies published are experience reports that incorporate a range of different
methods. These studies are underpinned by a pragmatic and ad-hoc approach
rather than a theoretically pure and rigorous approach to research design. As
a result, the published output exhibits a similar alignment of shortcomings in
research design to those reported by Hall, Ward, and Corner (Hall1988, 73).
While the papers do not follow a rigorous approach to the design of the
research from a theoretical perspective, the results provide an insight into
how sustainability can be integrated into the computing curricula at different
levels of granularity. As a result, it is strongly recommended that
researchers investigating sustainability in computing education adopt a more
meticulous and repeatable approach to the quality of the research being
conducted. This is imperative if any improvements or recommendations on
educational practice, the quality of policy debates on educational provision,
and the advancement of knowledge in the field are to be achieved that are
supported by sound empirical evidence. To improve the overall quality of the
research design, and the validity and reliability in data-gathering
procedures, we strongly recommend: clearly stating an explicit epistemological
stance; adopting a research design appropriate to address the problem under
investigation; clearly describing the data-gathering methods and adopting an
appropriate sampling strategy where applicable; using appropriate methods to
analyse data; explicitly stating assumptions; and framing the limitations of
the study around threats to validity.

##  7\. Discussion

This study provides understanding of existing work on sustainability education
in computing and its limitations. The review by Pollock et al. from 2019
(pollock2019, 167) suggests that sustainability or climate change has been
incorporated into computing education primarily at an “aspirational level,
without many tangible results to date” (p. 16). In the present systematic
review, spanning a wider set of digital libraries, we analyzed 90 primary
studies (compared to 11 papers in (pollock2019, 167)). Together, the articles
present different perspectives on how sustainability education can be, or
already is, implemented and researched on different scales (seminars, courses,
programmes) and on different levels (K-12 to PhD education).

###  7.1. Limitations of existing work and directions for future work

Most of the existing literature does not address the severity of challenges
and demand for unprecedented change. Stein et al. (stein2022:end-of-world,
196) argue that education needs to be disruptive, it needs to be about
“growing up” and showing up differently, facing the “impossibility of
sustaining our contemporary modern-colonial habits of being, which are
underwritten by racial, colonial, and ecological violence” (p. 275). We did
identify work that engages with conceptions of computing as disruptive
(Section 4.2) and have also pointed to studies that argue for radical changes
to computing education. As discussed in (SB-45, 55), approaches for curriculum
change vary from the incremental and integrated to the transformative and
critical. Such transformative changes rely on the idea that promoting a just
and sustainable society must be the goal, not computing itself
(sterling2004higher, 197). We find that the majority of studies present
smaller and incremental changes to a curriculum to include sustainability