inherent in this problem incorporated into analyses that use these images?
Careful consideration of such questions is crucial for attaching accurate
statements of uncertainty to final estimates.

####  7.4.2 Organizational Challenges

  1. 1.

Accessible publishing of methods. Advances in statistical theory and methods
abound in the literature of that field, but it is often presented in a highly
formalized mathematical manner, which obscures the aspects of most importance
to potential users. This creates a barrier to the appropriate use of these
methods in astronomy. The greater involvement of data scientists in
collaborations will help to bridge this divide, and enable these individuals
to make significant contributions. This will require appropriate professional
recognition for this effort, including encouraging the publication of
methodology papers in astronomical journals by data scientists (see Section
8.6 for related workforce issues).

  2. 2.

Avoiding the “algorithm trap." Astronomical inference problems are of
sufficient complexity that full use of the data requires analysis methods to
be adapted and tailored to the specific problem. For this reason,
statisticians prefer to not think of an analysis as the application of a
ready-made “algorithm.” By contrast, astronomers are generally more interested
in the result of the analysis, so are attracted to well-separated “algorithms”
they can apply to a problem. This difference in perspective only increases the
need to have data scientists deeply involved in the collaborative process.

  3. 3.

Reducing barriers for statisticians. From the other side, data scientists face
challenges in applying analysis techniques astronomical data. This is partly
due to technical difficulties like unique file formats and data access issues.
But it is also because deeply understanding the science is frequently crucial
to building methods tailored to the problem, as outlined above. More effort
needs to be placed on reducing these barriers. For example, astronomers can
work to isolate important statistical aspects of larger problems and create
user-friendly descriptions and datasets to allow statisticians to more quickly
learn and focus on making a contribution. At the same time, embedding
statisticians and data scientists close to astronomers will help bring the
former to a better understanding of the astronomy perspective.

###  7.5 The state of algorithms: developments in the last decade

A key development that has enabled science in this past decade has been the
development of a number of general purpose algorithms that can be applied to a
variety of problems. These algorithms, irrespective of what programming
language the implementation is in, have made astrophysical research more
repeatable and reproducible, and less dependent on human tuning.

For example, PSF kernel convolution has enabled time-domain astrophysics, and
is a key component of difference imaging pipelines, but is also used to
generate deep stacks of the static sky, allowing us to find ever more distant
galaxies. These developments in turn have spurred the development of new
algorithms. In roughly 20 years, the field has moved from Phillip Massey’s
guide to doing aperture photometry by hand with IRAF for small, classically
scheduled programs, to completely automated surveys that optimize their
observing schedule in real-time, record data, detrend the observations, and
perform automated PSF photometry of billions of deblended sources.

As with statistics, the distinction between algorithms, and the software
implementation of algorithms is blurry within the community. In many
situations, we now use algorithms without any knowledge of how they work For
example, we can now expect to sort tables with millions of rows on multiple
keys, without knowing the details of sorting algorithms, precisely because
these details have been abstracted away. We note that the many widely used
algorithms, such as affine-invariant Markov Chain Monte Carlo techniques are
widely used precisely because the algorithm is implemented as a convenient
software package. Community-developed software packages such as scikit-learn,
astropy, and the IDL Astronomy Library have increased the community’s exposure
to various algorithms, and the documentation of these packages has in many
cases supplanted implementation-oriented resources such as Numerical Recipes.

At the same time in the broader world, a class of algorithms is being used to
execute tasks for which an explicit statistical forward model is too complex
to develop, and correlations within the data itself is used to generate
actionable predictions. These AI techniques include machine learning models,
which have been used to replace humans for tasks as varied as identifying
artifacts in difference images, to categorizing proposals for time allocation
committees. These AI techniques, in particular deep learning methods, are
increasingly viewed as a solution to specific petabyte scale problems, as they
have been successfully deployed in the commercial sector on these scales. We
anticipate increasing adoption of these algorithms, as user friendly
implementations such as pyTorch and Keras become more well known, and data
volumes grow. It is also likely that the algorithms that are used to train
these machine learning methods, including techniques like stochastic gradient
descent, will find more use within the astronomical community.

Machine learning algorithms are necessary but not sufficient to continue the
progress in astrophysical research that is driven by algorithms. In
particular, machine learning methods are often not-interpretable, and while
their output can be used effectively, those outputs are not true
probabilities. The scientific method fundamentally involves the generation of
a testable hypothesis that can be evaluated given data, and is therefore
inherently statistical. As data volumes grow, the dimensionality of models
grows, and there is increasing recognition that the model structure is
hierarchical or multi-level. While we see increasing adoption of hierarchical
models for Bayesian inference, there remains much to do to increase awareness
of algorithms to effectively evaluate these models, including probabilistic
programming - algorithms that are used to build and evaluate statistical
models in a programmatic manner.

As in the previous section, we now separately consider some of the specific
technical and organizational challenges in the area of algorithms.

####  7.5.1 Technical challenges

  1. 1.

Both algorithms and models need to be trustworthy and interpretable. It’s easy
to throw a dataset into a neural net or ensemble classifier and overfit. Tools
need to be developed that recognize these traps and in large-scale datasets,
and bring them to the attention of the user.

  2. 2.

Many algorithms, especially in the machine learning space, require labeled
data that may not be available at sufficient volumes, or at all.

  3. 3.

The reproducibility of results derived from algorithms needs to be improved.
This is especially important with machine learning models where black-box
optimization is often used because it is an easy-to-provide feature. Such
reproducibility improvements could be as simple as defining standardized
formats for how we document the model learning parameters, but could also be
more complex, including building out tools that are designed specifically for
reproducibility (e.g. Data reduction pipelines with built-in provenance, or
Jupyter notebooks that download their own data).

  4. 4.

Scalability of newly-developed algorithms. With the data volumes of the
petabyte era, efficiency in all parts of the stack is necessary. Such
optimizations are usually possible, but require investment of time (often by
different people than those who develop the first iterations of the
algorithm).

  5. 5.

Astronomy data has some differences that can expand current algorithmic
development at large. This particularly includes use of measurement
uncertainties, as general-use algorithms often make assumptions that work for
other fields that are homoscedastic or Gaussian which fail in Astronomy. There
is also a need for more algorithms that account for posteriors, a particularly
strong need in astronomy because its domain of “the universe as a whole” means
that algorithms applied to one dataset need their outputs to be considered by
another.

  6. 6.

Significant work is still needed in adapting and improving the current space
of existing algorithms: optimizing traditional astronomy algorithms, adapting
them for a cloud setting, or even making small accuracy improvements.

####  7.5.2 Organizational challenges

  1. 1.

It is difficult to get the necessary expertise onto all missions that will
need it both in terms of developing the expertise internally (due to the fast
pace of change in the space) and hiring in experts.

  2. 2.

There is currently no established marketplace/mechanism for matching difficult
problems in the astronomy domain to relevant experts outside an astronomer’s
network. This is particularly acute given the discussion above about the
growing importance of statistical and data science expertise.

  3. 3.

There is a missing component in the conduit of moving new algorithms developed
in academia into robust, usable, finished products. See Section 6 for
additional discussion in this area.

  4. 4.

We need standardized processes for publishing algorithms and machine learning
models such that the results obtained with these algorithms/models are:
broadly accessible, discoverable, fully reproducible (including archiving the
model parameters), and easily comparable with other algorithms in the problem
space.

  5. 5.

We need to define and fund a process for continually modernizing/upgrading
algorithms as the broader environment changes (new languages, new libraries,
new computational architectures, shift to cloud computing, etc). See Section
6.6 for a broader discussion of mechanisms and recommendations for this.

###  7.6 Emerging trends in industry and other fields

Over the past two decades, the wider industry has also seen a shift in
development approaches and computational techniques that can be adopted by the
astronomical community. As noted in Section 6 open source software has become
a new normal with communities sharing their investment in software
development. When considered along with the industry’s shift toward cloud
computing and software as a service, astronomy can benefit from the new scale
and availability of off-the-shelf solutions for computation and storage.
Astronomers no longer need to focus significant portions of time on the low-
level technical details in running dedicated banks of computers to support
each survey.

This service model is being extended beyond software deployments and starting
to push into algorithms as a service. Cloud machine learning services provide
a portfolio of general algorithms. Instead of worrying about the specifics of
the algorithm development, users focus only on model specification. This
requires a shift in how we think about new algorithm development. Instead of
focusing on the details such as implementation, optimization, and numerical
accuracy, the practitioner focuses primarily on the high level model
specification. Due to a series of recent successes, a significant focus within
hosted machine learning services has been on deep neural networks (DNNs). NNs
have shown remarkable success across a variety of tasks. Further new
developments such as convolutional neural networks and recurrent neural
networks have extended the power of this technique.

Another area of focus within the field of machine learning is blackbox
optimization. Techniques such as Gaussian decision processes, allow algorithms
to jointly model and optimize unknown functions. These techniques can be
applied to a range of problems from optimizing real-world, physical processes
to optimizing the parameters of a machine learning system (e.g. AutoML).

The ultimate goal of algorithms as a service can be seen in the advancements
in AutoML. AutoML systems aim to abstract away not just the algorithm’s
implementation details, but also the need to manually tune model parameters.
For example, recent work in Neural Architecture Search (NAS), allows the
AutoML system to handle such development decisions as choosing the structure
of the network (number and width of layers) as well as the learning
parameters. While this automation greatly simplifies the problem of
constructing accurate models, it does move the practitioner one step further
from understanding the full details of the model.

There is an opportunity for astronomy to both benefit from and help drive new
advances in the emerging industries. As noted above, astronomy can benefit
from the shift from individually developed and maintained systems to hosted
platforms that allow more effort to be spent on the data analysis itself.
Moreover, the shape and size of science data serve as a driver for the
development of new algorithms and approaches. We expect many of the upcoming
advancements to be driven by real-world problems—machine learning will rise to
the challenge of solving new, open problems. The recommendations in this
chapter aim to ensure some of these problems and solutions are in the
astornomy domain.

###  7.7 Enhancing Interdisciplinary Programs and Collaborations

The past decade has been a period of rapid change in the the multi-dimensional
landscape of algorithms, computing, and statistics. We have seen the rise of
new “standard” programming languages and libraries (e.g. Python, astropy,
scikit-learn). There has been a proliferation of new algorithmic and
statistical techniques — from improvements in image processing and compression
to the rise of deep neural networks as a powerful tool from machine learning.
We have seen the rise of new computational modalities, such as cloud computing
and software as a service. New distributed compute frameworks such as Dask and
Spark are emerging to process and analyze large and complex datasets. Even the
basic mechanics of computation is undergoing a shift with the availability of
specialized hardware such as GPUs and TPUs, requiring a new domain of
knowledge to efficiently deploy solutions. There is no reason to expect the
pace of innovation to drop off anytime soon.

This rapid pace of advancement means that it is no longer possible for a
single astronomer or even a small team of astronomers to build the necessary
depth of expertise in all of these areas. However, these technologies are
already proving critical for maximizing the scientific reach of new research.
Robust methodologies that can scale to the expected size and complexity of the
data from new astronomical surveys and experiments will need to be accessible
and usable by a broad section of our community. As new technologies spring up
quickly, the astronomical community will need to balance the cost of learning
the new technologies with the benefits they provide. It is not reasonable to
expect every astronomer to keep up with all of the advances. A number of new
ad hoc collaborations or collectives have sprung up to bring together
astrophysicists and deep learning experts, such as the Deep Skies
Lab222222deepskieslab.ai and Dark Machines232323darkmachines.org.

In cases where collaborations exist today, there can be a variety of
complicating challenges. There is currently no established marketplace for
matching difficult problems in the astronomy domain to relevant experts
outside an astronomer’s network (see also §7.6). The resulting in-depth
collaborations have start up overhead as the external experts learn enough
about the problem domain to be helpful. Short-term engagements can suffer from
a lack of depth or insufficiently productionized solutions. Even in longer
term engagements, there can be misalignment between the parties due to the
different incentives. For example, statisticians and computer scientists in
academia are primarily recognized for only the novel contributions to their
own fields. Papers that apply existing methodologies to new problems are not
considered significant contributions to their fields. Similarly, members of
the astronomy community are not fully recognized for their algorithmic
contributions.

There are many opportunities for astrophysics to benefit from these
investments in technology and computational algorithms. However, requires that
we change how astronomy engages with experts in other fields. The exact shape
of this engagement can take a variety of forms. Examples include:

  1. 1.

Provide funding for astronomical missions to engage with external experts
(academic or industrial) via consulting, co-funded research, or
subcontracting.

  2. 2.

Encourage a robust community of volunteers via open source contributions and
engagement.

  3. 3.

Create forums for external methodological experts to engage in astronomical
projects and analyses. Data challenges and hack sessions can be used to
encourage engagement, but they require sufficient organization and
communication (i.e. funded effort) to ensure they can engage software
engineers at an appropriate level.

  4. 4.

Encourage recognition of interdisciplinary contributions within academic areas
(e.g. career progression for statisticians that enable new astronomy without
necessarily creating new statistics).

  5. 5.

Organize workshops that bring together members of these different fields and
can facilitate matching along problem domain.

  6. 6.

Provide funding for astronomical programs to hire full time experts to be
embedded within the mission. It is important to note that this approach comes
with challenges in recruiting (both these areas are in high demand), costs of
attracting high quality personnel, and in stability for the team members (the
algorithmic / statistical workload might not be consistent throughout the life
of a project).

  7. 7.

Implement reverse sabbaticals where experts from industry can embed in
projects for short intervals (a few months).

  8. 8.

Train astronomers in these fields to become resident experts. Encourage
mobility of these experts to provide support for new missions.

  9. 9.

Establish a center for algorithmic and statistical development in astronomy
(centralized or virtual) that employs full time experts in fields such as
algorithms, statistics, and machine learning. This center would be a community
resource that provides support to individual programs via deep engagement.

The goals of these interactions are not to provide programming support for
projects but to develop a base of expertise built from academic and industrial
experts that can help to define, design, and guide the development of
computational and statistical projects within astronomy. The form and depth of
the engagement will naturally be project dependent. Experimental and
privately-funded interdisciplinary centers e.g. the Moore-Sloan Data Science
Environments at Berkeley, New York University (NYU) and the University of
Washington, or the Simon’s Flatiron Institute have demonstrated how expertise
in data science can advance a broad range of scientific fields. Access to the
resources at these centers is, however, limited to researchers at these
privileged institutions. The challenge we face is how to scale these
approaches to benefit our community as a whole.

###  7.8 Education and training

Training a workforce that can address the algorithmic and statistical
challenges described in this Chapter will require a significant change in how
we educate and train everyone in our field, from undergraduate students to
Principle Investigator (PI)’s. The discussion in this section is complementary
to and aligned with that found in Section 8 and Section 9. The traditional
curricula of physics and astronomy departments do not map easily to the skills
and methodologies that are required for complex and/or data intensive
datasets. This is a rapidly changing field, and will remain so for at least a
decade. However, a strong foundation in Bayesian statistics, data structures,
sampling methodologies, and software design principles would enable
professional astronomers to take advantage to big data in the next decade.
Bridging this gap between the skills we provide our workforce today and the
ones they might need to succeed in the next decade should be a priority for
the field.

In the previous decade there was substantial progress in creating material to
support the teaching of statistics and machine learning in astronomy. This
includes the publication of introductory textbooks (Ivezić et al., 2014; Kohl,
2015; Hornik, 2018), the creation of common software tools and environments
(Astropy Collaboration et al., 2013), the development of tutorials, and a
growing focus on software documentation (Astropy Collaboration, 2019). The
emergence of Jupyter (Kluyver et al., 2016) as a platform for publishing
interactive tutorials and Github and Gitlab for hosting these tutorials and
associated code has simplified the process of sharing material. To date,
however, there has been little coordination in this effort. The coverage of
topics in the available material is not uniform. Moreover, the underlying
principles and foundations of statistics are often not covered in favor of the
introduction of commonly used software tools and algorithms. For the case of
algorithmic design and optimization there has been substantially less progress
in training the community. Instead, the primary focus being the development of
introductory materials such as the Software and Data carpentry (von Hardenberg
et al., 2019; Wilson, 2013).

We have started to make progress in providing an educational foundation in
statistics and algorithms, but it is not uniformly available across our
community — with significantly less access at smaller colleges and in
underrepresented communities. We, therefore, recommend the development and
support of a common and open set of educational resources that can be used in
teaching statistics, and algorithms, and machine or computational learning.
Determining what constitutes an appropriate curriculum will be a balance
between providing the foundations of statistics and algorithmic design
appropriate for the broader science community and teaching specialized skills
(e.g. optimization, compilers) that may benefit a smaller, but crucial, set of
researchers who will engage in the development and implementation of computing
and software frameworks.

This will likely require a coordinated effort to integrate current resources
within a broader curriculum and to make them easily accessible — in a manner
where anyone, from astronomer to an entire educational institution, can create
custom courses tailored to their needs. Given the rapid evolution in
algorithms and in the ecosystem of tools over the last decade, and looking to
the future, this curriculum will need to be able to evolve.

##  8 Workforce & Career Development

Contributors: Dara Norman<EMAIL_ADDRESS>Kelle Cruz, Vandana Desai, Britt
Lundgren, Eric Bellm, Frossie Economou, Arfon Smith, Amanda Bauer, Brian Nord,
Chad Schafer, Gautham Narayan, Ting Li, Erik Tollerud

Note: If you have come directly to this chapter we suggest you please read at
least the Introduction in Section 1 before delving further.

###  8.1 The growing importance of a tech-savvy workforce of astronomers

In the rapidly approaching era of large surveys, experiments, and datasets, we
will only reach our scientific goals if we train and retain highly capable
scientists, who are also engaged with technological advances in computing.
With the goal of advancing scientific discovery through the collection and
analysis of data, we must commit and dedicate resources to building both the
skills and competencies of this workforce. We define the workforce as those
who derive science from the data, as well as those who build the
infrastructure to collect and prepare the data for discovery analyses; one
cannot happen without the other. The areas and skill sets in which our teams
need training are software carpentry, algorithms, statistics, the use of
software tools and user services; and software engineering effective
practices, data management and access (for support staff). In addition, all
members of the workforce, including those who provide infrastructure, will
require professional development opportunities and support for their career
advancement allows them to remain as productive and respected members of the
field.

In this chapter we discuss the activities needed to build, support, and
advance the scientific workforce that will take the petabytes of data
collected to scientific discoveries over the next decade. In particular,
Section 8.2 discusses the current demographics of the data science support
mission while (Section 8.3) exemplifies the scope of training that is needed
to build this workforce. Section 8.4 focuses on training for researchers who
are more accurately described as “users.” In Section 8.5, we discuss modern
challenges for these career paths, as well as how to address them. Finally, in
Section 8.6.2, we identify metrics that we should be using for training in
career development and for reviews in career advancement.

###  8.2 Demographics - who is this workforce and where are they now

Data support roles are spread throughout the astronomy and astrophysics
(hereafter, “astronomy”) scientific community, and encompass people with a
variety of job types and descriptions at levels from post-baccalaureate to
PhD. A range of experience with either topics of astronomy or computing also
differentiate roles. This range of data support positions requires a diversity
of opportunities for training to work at the various levels. In addition,
different career development and advancement suited to those career tracks is
also required. For example, positions for those with PhDs are significantly
different from those that require only a post-bac degree, and thus the metrics
used to support and determine career advancement must also be different. It
has only recently been recognized that this role should be trained for and
tracked independently of scientific interests and other professional duties.
Consequently, the community has not adequately tracked the quantity and
demographics of astronomy researchers currently engaged in science data
support roles.

Instead of quoting statistics here, we present exemplar descriptions of
current job titles and roles. Many of the people engaged in science data
support hold PhDs in astronomy, astrophysics or physics. These researchers may
be employed at colleges, universities, data centers or observatories, national
laboratories. They may hold a leveled academic title (e.g. Professor,
Astronomer, Scientist, etc.), as well as an additional functional job position
in centers or programs with names “Data Science Mission Office,” “Community
Science and Data Center,” “Infrared Processing and Analysis Center.” Despite
the distinct nature of the work for scientists who hold these multiple
roles/titles, the same metrics (e.g. numbers of published papers, h-value,
etc.) for career advancement through titled positions (i.e. assistant,
associate, full, etc.) are used for them as for faculty with more typical
duties and responsibilities. More discussion is in Section 8.5.

There are also many other science data support roles, in which staff have
degrees at the BS, MS, or PhD level with position titles like “research and
instrument associate,” “research and instrument scientist,” “mission systems
scientist,” “archive scientist.” These staff are often responsible for coding
and database support. Below, we discuss the resources and cultural changes
needed to support the career trajectories of this workforce, to slow the
threat of “brain drain” from the field, and to develop a workforce that can
thrive in academia, industry, or government lab positions.

###  8.3 Training to contribute to software development: Building the next
generation

Astronomers have a long history of developing useful software, but software
development itself has not been considered a core component of the astronomy
training curriculum. The expectation of petascale datasets in the 2020’s
provides a strong motivation to increase familiarity with effective practices
in software development, as well as with existing frameworks that are widely
used in the commercial sector. This cultural change will lead to better
software in astronomy and more innovative scientific discovery. It will also
provide astronomers with invaluable training that will increase their
familiarity with (and marketability to) work in industry.

Currently, effective practices include using version control (e.g. GitHub),
maintaining documentation and unit tests with code, and employing continuous
integration methodologies, in which code is built and executed in shared
repositories, allowing teams to identify issues early. Analysis in the 2020s
will involve many pieces of software that are integrated into complex
pipelines, processing ever-larger volumes of data. Astronomical projects are
now comparable in scale to large industrial software development projects.
Consequently, the gap between these effective practices and the modern
cultural norm in astronomy and astrophysics must be reduced as the field
transitions to increasingly large collaborations.

The increasingly critical role of software development in astronomy clearly
indicates it is crucial that software development become part of the core
graduate curriculum alongside typical coursework, like mathematics and
observing techniques. Such coursework will also help reduce the disparity
between students from diverse backgrounds, some of whom may never have been
exposed to software development, or even coding, as undergraduates. This
course material complements, but is distinct from, training in data science
and scientific computing techniques, which are increasingly being incorporated
into Astronomy coursework. Developing the course material for data science
work is likely beyond the scope of most departments, but vital steps have
already been taken by several groups. Notably, the LSST Data Science
Fellowship Program has already developed materials to expose students to best
practices for software development. Curating these materials, and augmenting
them with information for distribution on widely-used platforms will reduce
the barrier to adopting such coursework or integrating it into existing
classes.

Besides developing course material, there are several other challenges to
supporting scientific software training in a university setting. Another
challenge is the lack of access to state-of-the-art technologies. Because the
landscape of coding and software development changes rapidly as coding
languages come and go, workflow best practices continually evolve, and new
platforms emerge and gain wide acceptance. For principal investigators and
project managers to make informed decisions and guide their teams, there must
be opportunities for them to stay abreast of these developments and to
evaluate their utility even if they are not the ones actually using the
various tools.

Yet another challenge resides in the structure and processes of university
departments. Many computer science departments do not teach the programming
skills necessary for scientists. Current faculty, especially at smaller,
under-resourced schools, do not have the resources to develop these. Thus, the
burden of developing more appropriate materials is fractured and currently
falls upon individual instructors. The field needs dedicated staffing to
develop curriculum materials for computational training. A fundamental barrier
to the development of reliable, curated, and widely shared software in
astronomy is the lack of incentives for this work and the dominance of the
“publish or perish” mentality. Changing this cultural norm requires that our
community incentivize — both within scientific projects and across the field
at the employment level — work in developing good software and in educating
people to build good software. Two key updates must be made to our community’s
value structure to change cultural norms and prepare for the software
challenges of projects in the 2020’s: 1) software development work should be a
part of assessments in service and research; 2) software that is widely used
for scientific work should be well-written and -documented. A full solution
cannot be realized through universities alone, and partnerships with data
centers, observatories, national labs, and professional societies are crucial.

The clear successes and popularity of the existing training programs, which
grew organically out of the community, attest to the need for additional and
more advanced training resources. While there are several successful programs
that address some of these concerns, they are insufficient to meet the needs
of the larger community. For example, the Software Carpentry
curriculum242424https://software-carpentry.org/lessons/ is limited to the very
basics of version control and collaborative software development but does not
cover topics, like performance optimization, continuous integration and
testing, and documentation. Furthermore, most of these workshops are targeted
to senior graduate students, with a few targeting very early-career
scientists, and they are not designed to meet the needs or concerns of mid-
career scientists and managers. Thus, these programs are currently limited to
a very small portion of the community and are currently unable to provide the
needed training to people in multiple sectors of our community who need and
want these opportunities.

Staff at Data Centers may themselves currently lack up-to-date data science
skills and knowledge. Funding to support career development for current staff
and to provide resources for centers to hire staff that have data science
expertise is critical to building workforce capacity in the 2020s.

Fundamental coding and software development skills are becoming increasingly
necessary for success in every aspect of Astronomy. However, acquiring
professional training in these skills is rare and inaccessible or impractical
for many members of our community. Students and professionals alike have been
expected to learn these skills on their own, outside of their formal classroom
curriculum or work duties. Despite the recognized importance of these skills,
there is little opportunity to learn and build them — even for interested
researchers. To have a workforce capable of taking advantage of the
computational resources and data coming in the next decade, we must find and
support ways to make coding and software development training widely
accessible to community members at all levels.

REC-18 Programs to cultivate the next generation.
Area: Workforce. Audience: Agency . Term: Long
Agencies should fund more and large-scale programs that cultivate the next
generation of researchers versed in both astrophysics and data science,
similar to smaller and over-subscribed programs like Software and Data
Carpentry, LSSTC Data Science Fellowship/ La Serena Data School for Science,
Penn State Summer School in Statistics for Astronomers.

nrecommendations REC-8.3 Programs to cultivate the next generation.
Area: Workforce. Audience: Agency . Term: Long
Agencies should fund more and large-scale programs that cultivate the next
generation of researchers versed in both astrophysics and data science,
similar to smaller and over-subscribed programs like Software and Data
Carpentry, LSSTC Data Science Fellowship/ La Serena Data School for Science,
Penn State Summer School in Statistics for Astronomers..
See page 8.3.

REC-19 Support to produce training materials.
Area: Workforce. Audience: Agency . Term: Short
Provide funding to data and computational centers to produce modular and re-
usable training resources to the community. These resources should be designed
to be used by individuals, integrated into formal classes, and used as part of
professional development training.

nrecommendations REC-8.3 Support to produce training materials.
Area: Workforce. Audience: Agency . Term: Short
Provide funding to data and computational centers to produce modular and re-
usable training resources to the community. These resources should be designed
to be used by individuals, integrated into formal classes, and used as part of
professional development training..
See page 8.3.

REC-20 Long-term curation of materials.
Area: Workforce. Audience: Agency . Term: Long
Funding must be provided to host and support educational materials in a long-
term, stable, scalable place. Provides stability and improves discoverability
if materials can live in a centralized location.

nrecommendations REC-8.3 Long-term curation of materials.
Area: Workforce. Audience: Agency . Term: Long
Funding must be provided to host and support educational materials in a long-
term, stable, scalable place. Provides stability and improves discoverability
if materials can live in a centralized location..
See page 8.3.

REC-21 Funding for innovative partnerships.
Area: Workforce. Audience: Agency . Term: Medium
Incentives should be provided to launch opportunities to harness partnerships
between data centers, universities and industry through funding. For example,
support for sabbatical programs at the data centers where teaching faculty can
learn skills, develop educational materials for community use, and bring back
to their home institutions.

nrecommendations REC-8.3 Funding for innovative partnerships.
Area: Workforce. Audience: Agency . Term: Medium
Incentives should be provided to launch opportunities to harness partnerships
between data centers, universities and industry through funding. For example,
support for sabbatical programs at the data centers where teaching faculty can
learn skills, develop educational materials for community use, and bring back
to their home institutions..
See page 8.3.

REC-22 Software training as part of science curriculum.
Area: Workforce. Audience: Astronomer, Educator, University . Term: Medium
Individuals, departments, and professional societies should encourage
educational programs to incorporate software training skills into their
existing courses and programs.

nrecommendations REC-8.3 Software training as part of science curriculum.
Area: Workforce. Audience: Astronomer, Educator, University . Term: Medium
Individuals, departments, and professional societies should encourage
educational programs to incorporate software training skills into their
existing courses and programs..
See page 8.3.

###  8.4 Training to take advantage of big data for research: Bytes to Science

Astronomers who came of age before the era of Big Data require training to
take advantage of astronomical “Big Data” in the 2020s. They also need these
skills to mentor students, who are simultaneously learning both astrophysics
and the uses of data for research. It is crucial that access to this training
be made widely available to professionals who come from a variety of science
backgrounds and are based at a broad range of institutions (e.g. universities,
data centers, etc.). This is especially important, considering these
professionals will be cultivating their students and the next generation of
scientists, as well as making decisions about which technologies to invest in.
If access to advancing data skills remains difficult to obtain, we will fail
to build a diverse workforce equipped to answer the most pressing questions in
astronomical research. Data Centers could play an important role in providing
this training.

New, freely accessible open source code and Jupyter frameworks like
SciServer.org and NOAO Data Lab enable anyone with a web browser to quickly
and easily analyze vast stores of professional astronomy datasets via web-
based notebooks. These cloud-based platforms can democratize educational
access by providing a scale of computing power and data storage that was
previously reserved for students and faculty at well-resourced research
institutions, where high-performance computing access and support are
abundant. A small number of astronomers in higher education are already
developing instructional activities for these platforms. These instructional
materials train students and other users to explore and analyze large
professional astronomy datasets with ease and to equip users with the
computational foundation needed to pursue advanced independent research
projects.

Jupyter notebooks in particular hold enormous potential for training the
current and next generation of astronomy professionals. However, currently,
the development of standardized curricular activities is performed in an
entirely ad-hoc manner. Limited resources (funding and time) lead to very
little deliberate coordination amongst various astronomy faculty who produce
such materials, and these products are not sufficiently discoverable (e.g.
accessible through a common repository).

The establishment of Community Science Centers hosted by Data Centers (like
NOAO) can be a hub (clearing house) to bring information to the community
about opportunities for the kind of resources and training that allow a broad
group of researchers to go from petabytes to publications.

In order to provide the most useful training, data centers need a clear view
of user needs. This information is provided by advisory committees, like “User
Panels.” However, these panels are traditionally populated by astronomers
based at R1 institutions and other data centers. Data Centers should ensure
that their User Panels include representatives from small and under-resourced
institutions; this will provide a clearer picture of the unique training needs
and challenges that must be addressed for these researchers. In addition
community surveys that reach astronomers who do not currently use data centers
should be undertaken to better understand what barriers exist.

REC-23 Training activities and materials.
Area: Workforce. Audience: Agency . Term: Short
Agencies must ensure that big data science training activities and materials
for PROFESSIONALS (as well as students) are included as part of the federally
funded data center’s mission and deliverables.

nrecommendations REC-8.4 Training activities and materials.
Area: Workforce. Audience: Agency . Term: Short
Agencies must ensure that big data science training activities and materials
for PROFESSIONALS (as well as students) are included as part of the federally
funded data center’s mission and deliverables..
See page 8.4. REC-24 Change advisory board representation.
Area: Workforce. Audience: Agency . Term: Medium
Federally (and privately?) funded science centers should include
representatives from small and under-resourced institutions to provide a broad
and clear picture of need in the community. The collection of information,
perhaps through surveys, to better understand the barriers to access that
exist for astronomers at these institutions should be undertaken by data
centers and others.

nrecommendations REC-8.4 Change advisory board representation.
Area: Workforce. Audience: Agency . Term: Medium
Federally (and privately?) funded science centers should include
representatives from small and under-resourced institutions to provide a broad
and clear picture of need in the community. The collection of information,
perhaps through surveys, to better understand the barriers to access that
exist for astronomers at these institutions should be undertaken by data
centers and others..
See page 8.4.

###  8.5 Identifying career paths around scientific software development & big
data science

The key skills necessary for data-intensive scientific research are also
highly valued in industry, government, and media/communication sectors.
Astronomy training can serve as a stepping stone to fulfilling careers in a
wide variety of fields, and astronomers should support and encourage those who
transition to jobs in non-academic science, because ties with industry can
strengthen and leverage our partnership opportunities. However, we need
informed people on both sides: in many cases, challenging and uncertain career
paths in astronomy push the best and brightest towards careers where their
contributions are more readily appreciated. This “brain drain” siphons away
the very researchers most needed to tackle the most pressing science questions
of the 2020s.

####  8.5.1 Universities

In the university context, tenure-track faculty positions remain the gold
standard for stability, compensation, and prestige. However, despite the
fundamental role of software in scientific discovery, it remains difficult to
receive credit towards tenure and promotion for developing software and
services. Section 8.6.2 offers more specific recommendations for improving
recognition for these contributions.

Even with appropriate credit for software contributions, faculty positions
will continue to carry expectations of leadership, grant-writing, teaching,
mentorship, and service, as is appropriate. Furthermore, driven by ongoing
changes in the landscape of higher education, tenure-track hiring continues to
flatten. To benefit from the opportunities of large datasets, universities
also need the ability to support and retain technically capable faculty and
staff, who have expertise and a longevity that typically cannot be matched by
graduate students or postdocs. These “Research Software Engineers” (Research
Software Engineers International (2018)) would provide a technical core for
data-intensive research groups, just as opto-mechanical and electrical
engineers are vital to the success of instrumentation labs.

Stable funding is the largest need for the success of staff Research Software
Engineers (Geiger et al., 2018). A patchwork of 2-3-year soft-money grants is
insufficient to retain highly-capable professionals, especially when industry
salaries are significantly higher. Universities should explore means of
providing internal support for data science staff, perhaps sharing capacity
between academic groups or departments. Long-term vision and leadership in the
field are needed to recognize and measure relevant metrics and make them part
of advancement/career ladders.

REC-25 Recognize software as part of the career path.
Area: Workforce. Audience: Manager . Term: Short
Software should be recognized in hiring and career development as a core
product of modern astronomical research. Software outputs should be considered
in all aspects of academic performance appraisals, career applications, and
promotion and tenure review cases.

nrecommendations REC-8.5.1 Recognize software as part of the career path.
Area: Workforce. Audience: Manager . Term: Short
Software should be recognized in hiring and career development as a core
product of modern astronomical research. Software outputs should be considered
in all aspects of academic performance appraisals, career applications, and
promotion and tenure review cases. .
See page 8.5.1. REC-26 Partnerships to support data science staff.
Area: Workforce. Audience: University . Term: Medium
Universities should explore means of providing support for data-science
faculty and staff, perhaps sharing capacity between academic groups or
departments internally or partnerships outside the university.

nrecommendations REC-8.5.1 Partnerships to support data science staff.
Area: Workforce. Audience: University . Term: Medium
Universities should explore means of providing support for data-science
faculty and staff, perhaps sharing capacity between academic groups or
departments internally or partnerships outside the university..
See page 8.5.1. REC-27 Support long-term technical capacity.
Area: Workforce. Audience: Agency . Term: Medium
Funding agencies should explore longer-term grants aimed at building and
supporting professional (non-student) data science capacity.

nrecommendations REC-8.5.1 Support long-term technical capacity.
Area: Workforce. Audience: Agency . Term: Medium
Funding agencies should explore longer-term grants aimed at building and
supporting professional (non-student) data science capacity..
See page 8.5.1.

###  8.6 Elevating the role of software as a product of the research
enterprise

The ecosystem around the publishing of scholarly work has not been set up to
properly account for contributions to scientific discoveries made through
tools, services and other infrastructure. Changes for the modern way in which
science is done need to be made. Publications, like ApJ and AJ , are run by
the AAS , a professional society, and are answerable to their boards that are
elected by and comprise the membership of professional researchers, who also
publish in them. Therefore, it is important to educate the larger community on
changes that need to be made to support modern recognition standards for
software services and then advocate for these changes with professional
societies.

Social and cultural issues within the field also must be changed to normalize
the appropriate acknowledgment of those who write software and support other
science infrastructure tools. We need academics in positions of power (e.g. on
promotion and tenure review committees, recruitment teams, grant review
panels) to value software as an important product of research. Although change
takes time, it is important that we begin making those changes with concrete
and practical suggestions that can be incrementally introduced into accepted
procedures and communal norms. These suggestions include the identification of
metrics that support proper assessment of the impact of software on achieving
scientific results. In recent years, substantial improvements to enable the
citation of software and tracking of these citations has been made in
astronomy and astrophysics.

####  8.6.1 Measuring/citing the impact of software

One key factor for improving the recognition of software within academia is to
enable native software citation, that is, make it possible and required for
authors to cite the software packages they have used in the process of
carrying out their research, and to then count these citations in tools such
as the Astrophysics Data System ( ADS ). Enabling software citation is both a
technical challenge and a cultural one: recommendations for what software
should be cited and when to cite it have been explored in community-wide
effort at FORCE11 (Smith et al., 2016), and follow-on efforts are exploring
some of the more technical aspects of how to implement these recommendations
(11 (2019)).

Within astronomy and astrophysics, the Asclepias project
252525http://adsabs.github.io/blog/asclepias — a collaboration between AAS
publishing, ADS, and the Zenodo data archive (hen, 2017) — is working to
enable first-class support for software citation in AAS journals as well as
support for indexing (counting) these citations within ADS. While this project
is currently scoped to AAS journals only, the changes being made to support
the citation and indexing of software serve as an example for other journals
to follow suit.

####  8.6.2 Strategies for elevating the role of software

Part of the challenge of elevating the role of software within academia is to
find actionable changes that improve the career prospects of those individuals
writing research software. In this section, we outline a number of possible
approaches.

Software papers: One approach gaining traction across a number of research
disciplines is to allow papers about software to be published in
“conventional” journals alongside other research papers, thereby making
software more visible to the academic community, and giving software engineers
a citable “creditable” entity (a paper) to include on their resume. Examples
of journals within astronomy that demonstrate a willingness to follow this
approach include PASP 262626https://iopscience.iop.org/journal/1538-3873 and
AAS publishing, which recently changed its editorial policies to explicitly
allow software papers in their publications (AAS Publishing, 2015). More
recently AAS publishing has announced a partnership with another journal
specializing in software review (Vishniac & Lintott, 2018).

Enabling support for software citation and indexing: Another key factor in
raising the visibility of research software is to enable software citation,
count these citations, and then make these metrics visible to the world. As
part of the work of the Asclepias project, software citations are not only
being counted in the astronomical literature, they are also being made visible
on the ADS website next to the paper record on ADS .

Inform and educate the community about software contributions: Organizations
play a critical role in improving the career prospects of those writing
research software as they are responsible for hiring these individuals,
evaluating their performance, and making decisions about possible
promotions/career advancement. One immediately actionable approach is to
encourage prospective employees and current staff to list software they have
developed on their resumes and performance appraisals. This would allow review
committees to include software as part of their evaluations.

Community prizes: AAS has a collection of prizes for scientific merit,
instrumentation, education, and service to the field. As it is an important
part of scientific discovery, software contributions that have had a lasting
positive impact on the field, should also be recognized with a new dedicated
prize and/or as a recognized example of merit within these other prize
categories.

Grants: The amount of research funding secured is an established metric for
evaluating an individual. As recommended in software, allowing existing
funding streams to be utilized for software development provides a simple
mechanism for funding research software, but also signaling community
recognition for the impact and relevance of the individual writing this
software. Furthermore, widespread availability of grant funding in support of
software development would provide a strong incentive for universities to hire
technical astronomers into tenure track positions.

REC-28 Adopt best practices for software citation.
Area: Workforce. Audience: Astronomer . Term: Short
Journals and reviewers should adopt best practices for assuring that software
and other science support infrastructure is properly referenced and cited in
articles. Referees and other reviewers should be trained to recognize when
such acknowledgement is necessary and ask authors to provide that information.

nrecommendations REC-8.6.2 Adopt best practices for software citation.
Area: Workforce. Audience: Astronomer . Term: Short
Journals and reviewers should adopt best practices for assuring that software
and other science support infrastructure is properly referenced and cited in
articles. Referees and other reviewers should be trained to recognize when
such acknowledgement is necessary and ask authors to provide that
information..
See page 8.6.2. REC-29 Adopt promotion metrics that acknowledge software and
other science support.
Area: Workforce. Audience: Manager . Term: Long
Departments and other members of the community should adopt and use suggested
metrics for promotion and tenure reviews of those scientists whose work and
contributions involve software and science infrastructure.

nrecommendations REC-8.6.2 Adopt promotion metrics that acknowledge software
and other science support.
Area: Workforce. Audience: Manager . Term: Long
Departments and other members of the community should adopt and use suggested
metrics for promotion and tenure reviews of those scientists whose work and
contributions involve software and science infrastructure..
See page 8.6.2. REC-30 Community prizes for software contributions.
Area: Workforce. Audience: Agency . Term: Short
Professional astronomy societies should create dedicated prizes and allow for
software contributions to be recognized as a criteria of merit within existing
prizes.

nrecommendations REC-8.6.2 Community prizes for software contributions.
Area: Workforce. Audience: Agency . Term: Short
Professional astronomy societies should create dedicated prizes and allow for
software contributions to be recognized as a criteria of merit within existing
prizes. .
See page 8.6.2.

##  9 A need for dedicated education and outreach expertise

Contributors: Amanda E. Bauer<EMAIL_ADDRESS>Britt Lundgren, Meg Schwamb,
Brian Nord, Dara J Norman

Note: If you have come directly to this chapter we suggest you please read at
least the Introduction in Section 1 before delving further.

We need to capitalize on positive trends in digital literacy, the increasing
use of mobile devices, and a discovery space driven by social media, through
the progressive development of online resources in astronomy education and
public outreach (EPO). The goal for this chapter is to clarify and bolster the
multitude of opportunities that exist to develop newly accessible online tools
to engage fellow citizens in the era of petabyte-scale astronomy.

Maintaining support for astronomy research relies on our ability to
effectively communicate our science and cultivate public excitement and
engagement. Historically, strategic programming for astronomy EPO in science
projects has been an afterthought: the work has primarily been undertaken by
astronomers who are passionate about EPO but may lack the specific
professional skills required to do it effectively at scale. Moreover, most
astronomers are not compensated for their time or rewarded by their efforts in
EPO. To maximize the public impact of large projects in the petabyte era, we
must give professional credit to astronomers who do outreach work and also
dedicate resources to full-time personnel to develop, execute, and evaluate
modern EPO activities.

Traditional means of public engagement (e.g. classroom visits, online videos,
public lectures and panels, etc.) have demonstrated their importance and
value, and have carved a niche in the landscape of public engagement. However,
we have entered a new era of technology and social interaction, which
necessitates new modalities for innovative pedagogical techniques,
communication, and even scientific exploration. Taking advantage of
opportunities of modern technology requires putting in place the appropriate
professionals to create and develop the interfaces and connections to
curricula that maximize adaptability and use. For example, connecting non-
experts with ever larger datasets requires educators who have astronomy domain
expertise (to curate and work with datasets) as well as expertise in
innovative pedagogical practices.

In this new era of engagement, EPO teams who develop ground-breaking
activities and pedagogical frameworks will have started the design process as
early as possible (including during construction of new facilities) and will
have drawn on a number of areas of expertise: astronomical research methods,
educational theory and practice, web development and design, software
engineering, and multi-modal communication.

In this chapter, we discuss recommendations and effective practices for
advancing astronomy in society through data-driven education and outreach
activities for maximizing the impact large observing facilities and data
centers will provide. We begin by discussing the creation of accessible online
activities (Section 9.1), then identify a range of skills needed to create
such activities (Section 9.2), and finally, we establish the benefits of
resourcing dedicated EPO groups from the earliest stages of astronomy facility
planning and including EPO as part of the mission of projects (Section 9.3).

###  9.1 Create accessible online activities for the public

REC-31 Create accessible online Activities for the Public.
Area: EPO. Audience: Educator, Astronomer. Term: Short
To maximize the impact of astronomy in society in the rapidly approaching
petabyte and exabyte eras, we recommend that projects and data centers develop
accessible web interfaces and tools that enable the public to access, explore,
and analyze authentic astronomical data at unprecedented scale.

nrecommendations REC-9.1 Create accessible online Activities for the Public.
Area: EPO. Audience: Educator, Astronomer. Term: Short
To maximize the impact of astronomy in society in the rapidly approaching
petabyte and exabyte eras, we recommend that projects and data centers develop
accessible web interfaces and tools that enable the public to access, explore,
and analyze authentic astronomical data at unprecedented scale..
See page 9.1.

Many good arguments have been made for enabling non-professionals and students
to access and engage with authentic data and professional tools. However, in
practice, the increasing complexity of interfaces to large datasets can become
a barrier to access and use.

User interfaces need to be attractive and intuitive for non-specialists and
usable from mobile devices and platforms commonly used in schools (such as
chromebooks and tablets). Interfaces created for professionals do not
necessarily work for non-specialists, because they tend to have the following
characteristics: 1) offer too many options; 2) do not offer a clear path
toward a learning outcome; 3) too slow, unresponsive, or burdensome for the
internet connections. Effort should be spent on user interfaces for public
audiences, and ideally, on creating introductory activities as preparation for
more complicated tasks.

Surveying users to assess their needs and interests helps the content design
process and continues to improve the quality of an experience for users when a
program in running. User testing is a regular practice for many companies that
deliver a product to the public and is a process that should be adapted within
astronomy EPO programs to ensure activities remain relevant and useable.

####  9.1.1 Examples of online activities

Several examples of existing and planned infrastructures illuminate avenues
for online public engagement: below, we discuss Sloan Digital Sky Survey’s
(SDSS) SkyServer, Zooniverse’s Citizen Science, NASA’s Universe of Learning,
and the EPO program of LSST.

For over 15 years, the SDSS has made its vast database of imaging and
spectroscopic data ($\sim$200 TB) freely available to the world. The web-based
SDSS data browser, SkyServer272727http://skyserver.sdss.org, provides a public
entry point for navigating the data online. The numerous and diverse query and
analysis tools available through the SkyServer are designed to meet the needs
of astronomers and non-professionals alike. The benefit to this design is that
any interested student or member of the public has unrestricted access to
research-grade inquiries and applications of the data. However, the large
number of available features and the technical jargon that accompany them
often overwhelm non-experts, as well as professional astronomers who are
external to the SDSS collaboration.

In order to better support audiences who may be put off or overwhelmed by the
professional-grade access points to the SDSS database, the SDSS Education and
Public Outreach team developed activities with simplified query tools and
smaller, curated datasets to facilitate activities for pre-college educators
and students (e.g. SDSS Voyages282828http://voyages.sdss.org). For non-
specialist audiences, these activities lower the barrier to accessing the same
authentic data, while providing an introduction to concepts related to both
astronomy and data structures. For students and educators who may be
interested in using the data for more advanced explorations, SDSS Voyages
provides a helpful stepping stone. The next two sections of this chapter
suggest avenues to promote this transition in other ongoing and planned
astronomy projects and facilities.

Citizen science represents an example of successful use of the modern age of
web connectivity by directly engaging the public in scientific research.
Online citizen science enables scientists to work with the general public to
perform data-sorting and analysis tasks that are difficult or impossible to
automate, or that would be insurmountable for a single person or for small
groups of individuals to undertake (Marshall et al., 2015). Highly accessible
citizen science activities can advance both science and learning in the era of
large astronomical datasets. Moreover, most participants from the public claim
that the main reason they participate is the contribution they are making to
fundamental science research (Cox, 2017). Through online citizen science
portals such as the Zooniverse292929http://www.zooniverse.org (Lintott et al.,
2011), millions of volunteers have participated directly in this collaborative
research experience, contributing to over 70 astronomy-based research papers.
Another reason for the continued success of the Zooniverse platform in
particular, is that it looks good and feels modern, even after a decade of
activity. While professional astronomers are the PI’s of citizen science
projects, Zooniverse employs 13 developers, one designer, and two postdocs to
lead the infrastructure development of the platform between the Adler and
Oxford locations.

Members of the Zooniverse team have furthered the project’s educational impact
by developing a college-level data science curriculum around their crowd-
sourced data. The NSF-funded Improving Undergraduate STEM Education (Improving
Undergraduate STEM Education (IUSE)) Project: “Engaging Introductory Astronomy
Students in Authentic Research through Citizen Science" (PI: L. Trouille) is a
particularly successful example of scoping big-data astronomy for a college
non-major audience. This innovative curriculum equips students with the
essential tools to explore the intrinsic and environmental properties of
20,000 low-redshift SDSS galaxies that have morphological classifications from
Galaxy Zoo. This project utilizes a curated dataset in Google Sheets and a
simple, plug-in tool that enables intuitive data cropping and visualization.
Instead of learning about galaxies and cosmology through traditional readings
and lectures, students are challenged to discover key patterns and properties
of the universe themselves, through first-hand explorations of authentic
astronomical data. In the process, they gain skills in quantitative analysis,
improve their overall data literacy, and practice science communication. The
curriculum specifically provides an opportunity to discuss the complications
and limitations of authentic data, and the challenges of framing a question
that can be effectively tested with the data one has in hand. The project is a
great case study of delivering specific, high-impact learning outcomes through
an analysis of authentic data, without requiring students to navigate full-
scale datasets or jargon-rich professional tools for visualization and
analysis.

NASA’s Universe of Learning303030https://www.universe-of-learning.org/ offers
a variety of individual online web pages that are well presented. A potential
challenge for a typical user who finds one of these pages is knowing what to
do next. Beyond exploring the beautiful multi-wavelength images space
telescopes provide, there is not a clear path for a user to navigate toward
specific learning outcomes or experiences.

LSST’s EPO program313131https://www.lsst.org/about/epo is unique among ground-
based telescope projects: not only is it being constructed in tandem with the
physical observatory itself, but the outreach program is funded at 2% of the
project cost. EPO products will go live when the LSST Survey begins in 2022.
EPO products were included from the beginning as part of the construction
Project deliverables, because they faced similarly unique challenges as the
data resulting from the survey itself. During its design phase, the EPO team
selected specific audiences and invested in user needs assessments to examine
what these audiences want, and cannot find elsewhere. Some major findings
include the necessity for mobile-friendly interfaces, a clear path toward
learning goals, and educators needing no new software to download in order to
introduce classroom activities. This has shaped the overall strategy for LSST
EPO development and the skill sets needed on the EPO Team, which is a small,
interdisciplinary team of astronomers, writers, designers, educators, and
developers. The mission of LSST EPO is “to offer accessible and engaging
online experiences that provide non-specialists access to, and context for,
LSST data so anyone can explore the Universe and be part of the discovery
process.”

The operations website will feature news about LSST discoveries, profiles of
LSST scientists and engineers and their work, and will be optimized for use on
mobile devices. The EPO team is also developing online, data-driven classroom
investigation activities for students in advanced middle school through
college. The topics cover commonly-taught principles in astronomy and physics,
and each investigation is designed for use with Next Generation Science
Standards (NGSS) in the United States and the Curriculum Nacional in Chile.
All investigations come with support and assessment materials for instructors
and no special software is needed to access the investigations, which will be
available in English and Spanish. LSST EPO will maintain an easy-to-use
gallery of high-quality multimedia visualizations that can be downloaded and
integrated into exhibits and presentations. Finally, LSST EPO will provide
support to researchers who create Citizen Science projects using LSST data,
including a dedicated project-building tool on the Zooniverse platform. The
infrastructure to host these activities is being built during construction and
will take several years. Another critical task during construction is building
prototypes and performing user testing, which has continually proven to
improve the user experience and usability of interfaces.

A consistent theme that emerges when examining these examples is that well-
defined learning outcomes for activities, curated access to authentic data,
and simple, intuitive design are important to prepare for the EPO response to
the large data we will collect in the 2020s. The remaining sections identify
areas of expertise EPO teams can employ to achieve these outcomes.

###  9.2 Bring expertise to astronomy education and outreach teams

REC-32 Bring dedicated experts onto astronomy education and outreach teams.
Area: EPO. Audience: Manager, Educator. Term: Medium
To create the accessible online interfaces that maximize public impact in the
next decade, we recommend supporting dedicated education and outreach teams
that pair astronomers with technical and education specialists to increase
relevance, adoptability, and accessibility of activities.

nrecommendations REC-9.2 Bring dedicated experts onto astronomy education and
outreach teams.
Area: EPO. Audience: Manager, Educator. Term: Medium
To create the accessible online interfaces that maximize public impact in the
next decade, we recommend supporting dedicated education and outreach teams
that pair astronomers with technical and education specialists to increase
relevance, adoptability, and accessibility of activities..
See page 9.2.

The large-scale data challenges that face astronomy described in this paper
also represent challenges and opportunities for formal education, public
outreach, and science communication. A natural instinct for astronomers may be
to adapt their new computational experience to outreach efforts. This is a
noble goal, but astronomers are not be expected to know effective practices
around mobile-friendly development, intuitive user interfaces for the public,
marketing through social media, or how to connect astronomy activities to
formal education curriculum standards. A team of EPO experts can advise and
assist with these areas, which are essential to build successful activities
that are discoverable and adoptable.

We recommend astronomy organizations support creating EPO teams with expertise
in relevant areas. It is understood that to reach maximal impact of outreach
activities, these individuals work with astronomers to combine astronomy and
data science expertise with specific EPO expertise and experience. This
section describes options for areas of expertise and roles that can be brought
on to achieve specific goals.

Educators in the United States (US) are currently required to submit paperwork
to demonstrate that they are teaching specific topics related to curriculum
standards. An EPOeducation specialist or instructional designer brings
knowledge of relevant curriculum standards and rubrics (for example, the Next
Generation Science Standards323232NGSS: https://www.nextgenscience.org/) and
is able to connect astronomy activities to topics educators must cover. This
is the most relevant for K-12 formal education in a traditional setting or
homeschooling. An educational specialist can build professional development
programs to increase confidence for bringing such activities into their
classrooms if there is not an expert available to join in person.

An education specialist can also tap into educator networks to advertise
existing programs and perform professional development. An example is the
National Science Teachers Association (National Science Teachers Association
(NSTA)) annual meeting and AAS. An education specialist working with an
astronomer can create curate datasets to achieve specific learning outcomes
without overwhelming non-specialists (Rebull et al., 2018).

An Information Officer or Communications Manager can act as a primary contact
for a facility and also set overall communication strategies and
implementation plans. Topics covered in such strategies could include
audiences, content priorities, communication channels, messaging, procedures,
and more.

An Outreach Specialist could serve a range of purposes depending on the needs
of the group. This could be a science writer, someone who responds to and
directs questions received from audience groups, or contributes to social
media presence. or an astronomer trained in science communication. If this
person has astronomy training, he/she could work with astronomical datasets to
curate options for the public.

Social Media is becoming increasingly important as a source of news and
information in society. Dedicating a full-time equivalent (or more) to the
role of social media engagement specialist increases awareness of EPO
activities and engages various audiences to participate with activities that
exist.

Overall branding, the look and feel of online activities, and developing
interesting graphics and images to support press releases or other activities
are the role of a Graphic Designer.

An evaluation specialist informs methods for understanding the impact of
programs on specific audience groups. The most benefit occurs when the method
for evaluating the success of a program is built into the development of the
program itself. Metrics could include and are not limited to web analytics,
short or long surveys, interviews, login requests, focus groups, web
submission forms, and social media interactions.

A web developer considers the user interface and experience when visiting a
site. Mobile-friendly accessibility is a requirement for non-specialists since
most users of an online interface will discover the materials via social media
and will access them from a mobile device, not a desktop platform. In
addition, the most common machines used by schools are chromebooks and
potentially weak internet connections, which require lightweight design.
Development needs to satisfy these requirements are best implemented by
experts in the field.

A Software Architect designs, deploys, and maintains production services for
an online program. It is important to not overburden internet systems that can
be common in classroom settings or non-urban areas.

A Project Manager oversees the detailed budget, schedule, contracts,
documentation, and reporting. This role is important for programs being built
during the construction of an astronomical facility.

###  9.3 Fund dedicated astronomy education and outreach groups

REC-33 Fund dedicated or centralized astronomy education and outreach groups.
Area: EPO. Audience: Agency. Term: Long
We recommend that funding agencies supporting the development and operation of
large astronomical observing and data facilities fund professional education
and outreach groups who can provide strategy, oversight, and best practices to
maximize the impact of outreach efforts, and encourage EPO efforts to be part
of a project’s mission.

nrecommendations REC-9.3 Fund dedicated or centralized astronomy education and
outreach groups.
Area: EPO. Audience: Agency. Term: Long
We recommend that funding agencies supporting the development and operation of
large astronomical observing and data facilities fund professional education
and outreach groups who can provide strategy, oversight, and best practices to
maximize the impact of outreach efforts, and encourage EPO efforts to be part
of a project’s mission..
See page 9.3.

Having a dedicated individual or team to develop the EPO program for a
specific facility can improve efficiency, impact, and cost effectiveness.
Strategic planning provides an opportunity to emphasize the identity of a
particular large facility; to identify non-specialist audiences who could
benefit the most from dedicated engagement; put into place best practices in
outreach and communication programs; and complement the overall landscape of
astronomy EPO efforts. It is important that the EPO professionals are employed
directly at professional telescope facilities in order to emphasize the
uniqueness of the program, build and maintain relationships with those doing
the technical and scientific work, and help handle the astronomy-specific data
products that currently require a reasonable level of understanding to
interpret and use (see Section 4).

A dedicated EPO team also serves as a resource for enabling astronomers
working with large datasets and data facilities to do more impactful and wide-
reaching outreach. Groups that are specifically charged to do EPO can improve
the impact of the existing NSF Broader Impacts investment by supporting
astronomers to tap into existing programs. This improves discoverability of
the EPO work astronomers are doing increases the likelihood of achieving
Broader Impact goals at both the individual and NSF levels.

An EPO team could provide any of the following benefits:

  * •

Conducting science communication and media training sessions for astronomers
doing these activities.

  * •

Providing introductions to various social media platforms that can be used for
unique outreach experiences.

  * •

Marketing and promoting activities through established social media and common
online training resources (e.g. Code Academy).

  * •

Creating or tapping into a centralized repository for people looking for
resources

  * •

Creating opportunities for collaboration between astronomers and existing
outreach infrastructure that will promote success and provide wide-reaching
impact. Examples include Journey Through the Universe in Hawai’i or AstroDay
in Chile, both led by Gemini EPO.

  * •

Performing user needs assessments and user testing to improve the quality of
existing activities and to develop new programs that meet the needs of
specific audiences.

  * •

Evaluating and reporting on the impact of EPO activities. Evaluation methods
can and should be built into program design.

  * •

Providing guidance for astronomers when developing science drivers and use
cases for educational materials and public interfaces related to their
research expertise.

  * •

Broadening participation to non-traditional audience groups.

The timing for building expert EPO teams should occur during the construction
of a new facility and be included as part of the project’s mission. Starting
early affords time to implement appropriate strategy and infrastructure.
Educational materials, supplemental professional development materials,
striking visualizations and images and communications strategies should be
ready at the start of a project to maximize the public impact of the facility.

In this chapter, we discussed recommendations and effective practices that can
be employed to maximize the impact of large astronomy facilities and data
centers in the next decade. We prefaced the need for creating accessible
online astronomy activities for the public and identified a range of skills
needed to create such activities. Finally, we established the benefits of
resourcing dedicated EPO groups from the earliest stages of astronomy facility
planning and even including EPO as part of the mission of projects.

These recommendations are based on two main things: trends seen elsewhere on
the web that successfully respond to this new era of technology and social
interactions on the web, and case studies within astronomy that demonstrate
appropriate avenues for increasing engagement and accessibility through online
activities.

Note: This chapter is the basis for an Astro2020 Decadal Survey APC white
paper which will go into detail on resourcing and prioritizing
recommendations. If you are interested in commenting or constributing, please
contact Amanda E. Bauer<EMAIL_ADDRESS>
##  10 Conclusion

We had a good opportunity to think about a range of topics which have been
detailed in this document. Several APC white papers will be published using
this as a basis. If you are interested in contributing to or endorsing
https://tinyurl.com/y2ksemp2333333 https://tinyurl.com/y2ksemp2 or contact the
authors listed in this document. We do not intend this to be the solution to
all issues rather a discussion of potential ways forward for the next decade.
We shall host a third and final
workshop343434https://petabytestoscience.github.io/ in October 2019 which will
explore practical approaches to dealing with some of the recommendations
raised here.

### Acknowledgments

The workshops in this series were supported by The Kavli
Foundation353535https://www.kavlifoundation.org/. The Kavli Foundation is
dedicated to advancing science for the benefit of humanity, promoting public
understanding of scientific research, and supporting scientists and their
work.

## Glossary

1D

    One-dimensional
2D

    Two-dimensional
3D

    Three-dimensional
AAG

    Astronomy and Astrophysics Research Grants, an program
AAS

    American Astronomical Society
ADS

    Astrophysics Data System
AI

    Artificial Intelligence, used to cover many machine learning algorithms
APC

    activities, projects, or state of the profession considerations - wrt. the decadal survey
API

    Application Programming Interface. Usually this is either "web API" (meaning how you access/update data via http) or "software API" (meaning how you call the function/class/etc in a particular programming language or library), although often only "API" is used and the prefix is implicit.
APL

    Apache Public License
Archive

    The repository for documents required by the NSF to be kept. These include documents related to design and development, construction, integration, test, and operations of the LSST observatory system. The archive is maintained using the enterprise content management system DocuShare, which is accessible through a link on the project website www.project.lsst.org
astrometry

    In astronomy, the sub-discipline of astrometry concerns precision measurement of positions (at a reference epoch), and real and apparent motions of astrophysical objects. Real motion means 3-D motions of the object with respect to an inertial reference frame; apparent motions are an artifact of the motion of the Earth. Astrometry per se is sometimes confused with the act of determining a World Coordinate System (WCS), which is a functional characterization of the mapping from pixels in an image or spectrum to world coordinate such as (RA, Dec) or wavelength
ATLAS

    The Asteroid Terrestrial-impact Last Alert System
CADC

    Canadian Astronomy Data Centre
CAOM

    Common Archive Observation Model http://www.opencadc.org/caom2/
CI

Citizen Science

    \- Amanda, Arfon, Meg?,
cloud

    A visible mass of condensed water vapor floating in the atmosphere, typically high above the ground or in interstellar space acting as the birthplace for stars. Also a way of computing (on other peoples computers leveraging their services and availability).
CMB

    Cosmic Microwave Background
cold storage

    Data moved to cold storage means data moved to cheaper slower storage such as tape. The assumption is this is no longer accessed frequently.
community software

    Software developed for and shared among a large group of relatively like-minded users (e.g. astronomers). Typically, but not necessarily, and -based.
container

    A container is a software package that contains everything the software needs to run. This includes the executable program as well as system tools, libraries, and settings. … For example, a container that includes PHP and MySQL can run identically on both a Linux computer and a Windows machine.
CSSI

    Cyberinfrastructure for Sustained Scientific Innovation https://www.nsf.gov/pubs/2019/nsf19548/nsf19548.htm
cyberinfrastructure

    Sometimes denoted CI, A term first used by the US National Science Foundation (National Science Foundation (NSF)), and it typically is used to refer to information technology systems that provide particularly powerful and advanced capabilities.
Data Management

    The LSST Subsystem responsible for the Data Management System (DMS), which will capture, store, catalog, and serve the LSST dataset to the scientific community and public. The DM team is responsible for the DMS architecture, applications, middleware, infrastructure, algorithms, and Observatory Network Design. DM is a distributed team working at LSST and partner institutions, with the DM Subsystem Manager located at LSST headquarters in Tucson
Department of Energy

    cabinet department of the United States federal government; the DOE has assumed technical and financial responsibility for providing the LSST camera. The DOE’s responsibilities are executed by a collaboration led by SLAC National Accelerator Laboratory
DES

    Dark Energy Survey
DESI

    Dark Energy Spectroscopic Instrument
DIBBs

    Data Infrastructure Building Blocks
DKIST

    Daniel K. Inouye Solar Telescope (formerly the Advanced Technology Solar Telescope, ATST)
DM

Docker

    A popular implementation of technology.
DOE

DOI

    Digital Object Identifier https://www.doi.org/
DVCS

    Distributed Version Control System, a form of where the complete codebase - including its full history - is mirrored on every developer’s computer
ELT

    Extremely Large Telescope
EPO

    Education and Public Outreach
ESAC

    European Space Astronomy Centre
FITS

    Flexible Image Transport System, is an open standard defining a digital file format useful for storage, transmission and processing of data. Files are formatted as tables or 2D images
FPGA

    Field Programmable Gate Array, an integrated circuit which is fairly easily configurable.
git

    The most widely used DVCS software
GPL

    GNU Public License
GW

    Gravity Wave
HPC

    High Performance Computing
HTC

    High Throughput Computing
IAM

    Identity and Access Management
IDL

    Interactive Data Language, a programming language used for data analysis. Harris Geospatial 111https://www.harrisgeospatial.com/Software-Technology/IDL
interoperability

    the ability of systems or software to exchange and make use of information between them.
IPAC

    No longer an acronym; science and data center at Caltech
IRAF

    Image Reduction and Analysis Facility, a collection of software written at the National Optical Astronomy Observatory (NOAO) geared towards the reduction of astronomical images in pixel array form.
IRSA

    Infrared Science Archive
IUSE

    Improving Undergraduate STEM Education
IVOA

    International Virtual-Observatory Alliance
JWST

    James Webb Space Telescope (formerly known as NGST)
LIGO

    The Laser Interferometer Gravitational-Wave Observatory
LISA

    Laser Interferometer Space Antenna - ESA mission for 2030’s
LSST

    Large Synoptic Survey Telescope
LSSTC

    LSST Corporation, a not for profit organisation associated with LSST
MAST

    Mikulski Archive for Space Telescopes
MCMC

    Markov Chain Monte Carlo, a class of algorithms for sampling from a probability distribution.
metadata

    General term for data about data, e.g., attributes of astronomical objects (e.g. images, sources, astroObjects, etc.) that are characteristics of the objects themselves, and facilitate the organization, preservation, and query of data sets. (E.g., a FITS header contains metadata)
ML

    Machine Learning (see also AI)
MSE

    Maunakea Spectroscopic Explorer
NAS

    Neural Architecture Search
NASA ROSES

    Research Opportunities in Earth and Space Science
NASA

    National Aeronautics and Space Administration
NGSS

    Next Generation Science Standards https://www.nextgenscience.org/
NOAO

    National Optical Astronomy Observatories (USA)
NSF

    National Science Foundation
NSTA

    National Science Teachers Association
NYU

    New York University
ODBC

    Open DataBase Connectivity, a standard API for databases.
open development

    A process for developing software that emphasizes all code contribution and decision-making be done in the open, available to as wide a group as possible (This usually means anyone with internet access).
open source software

    Open source software is a type of software in which source code is released under a license in which the copyright holder grants users the rights to study, change, and distribute the software to anyone and for any purpose. Note that this is _not_ necessarily the same as open to contribution (see open development).
OpenEXR

    a high dynamic range raster file format, released as an open standard along with a set of software tools created by Industrial Light & Magic (ILM) http://www.openexr.com/index.html
OS

    Operating System
Pan-STARRS

    Panoramic Survey Telescope and Rapid Response System
parquet

    Parquet File Format Hadoop. Parquet, an open source file format for Hadoop. Parquet stores nested data structures in a flat columnar format. Compared to a traditional approach where data is stored in row-oriented approach, parquet is more efficient in terms of storage and performance.
PASP

    Publications of the Astronomical Society of the Pacific
PB

    PetaByte
PDF

    Probability Density Function
PI

    Principle Investigator
PLATO

    PLAnetary Transits and Oscillations of stars, the third medium-class mission in ESA’s Cosmic Vision programme.
PSF

    Point Spread Function, describes the response of an imaging system to a point source or point object.
reproducibility

    (this one should have many definitions and we have to say WHICH version we are talking about) The ability to combine the same code and data and get the same result, or the ability to use the same code with different data to enforce a result, or there may be others
S3

    Structured, imperative high level computer programming language, used as implementation language for the Virtual Machine Environment () operating system
SDSS

    Sloan Digital Sky Survey
SKA

    Square Kilometer Array
software

    The programs and other operating information used by a computer.
STEM

    Science, Technology, Engineering and Math
TAP

    Table Access Protocol
TB

    TeraByte
TESS

    Transiting Exoplanet Survey Satellite, a space telescope for NASA’s Explorer program
TPU

    Tensor Processing Unit , a proprietary type of processor designed by Google in 2016 for use with neural networks and in machine learning projects
US

    United States
version control

    The management of changes to documents, computer programs, and other collections of information. Changes are usually identified by a number or letter code. Each revision is associated with a timestamp and the person making the change. Revisions can be compared, restored, and merged
VO

    Virtual Observatory
WCS

WFIRST

    Wide Field Infrared Survey Telescope
World Coordinate System

    a mapping from image pixel coordinates to physical coordinates; in the case of images the mapping is to sky coordinates, generally in an equatorial (RA, Dec) system. The World Coordinate System (WCS) is expressed in FITS file extensions as a collection of header keyword=value pairs (basically, the values of parameters for a selected functional representation of the mapping) that are specified in the FITS Standard
ZTF

    Zwicky Transient Facility

## References

  * hen (2017) 2017, Asclepias - Capturing Software Citations in Astronomy (Zenodo)
  * 11 (2019) 11, F. 2019
  * AAS Publishing (2015) AAS Publishing. 2015
  * Andreessen (2011) Andreessen, M. 2011, online
  * Astronomy & Computing (2013) Astronomy & Computing. 2013, https://www.journals.elsevier.com/astronomy-and-computing
  * Astropy Collaboration (2019) Astropy Collaboration. 2019
  * Astropy Collaboration et al. (2013) Astropy Collaboration, Robitaille, T. P., Tollerud, E. J., et al. 2013, A&A, 558, A33
  * Astropy Collaboration et al. (2018) Astropy Collaboration, Price-Whelan, A. M., Sipőcz, B. M., et al. 2018, AJ, 156, 123
  * Baron & Poznanski (2017) Baron, D., & Poznanski, D. 2017, MNRAS, 465, 4530
  * Bechtol et al. (2019) Bechtol, K., et al. 2019, Astro2020 Science White Paper: Dark Matter Science in the Era of LSST
  * Behroozi et al. (2019) Behroozi, P., et al. 2019, Astro2020 Science White Paper: Empirically Constraining Galaxy Evolution
  * Bektesevic et al. (2019) Bektesevic, D., Mehta, P., Juric, M., et al. 2019, in American Astronomical Society Meeting Abstracts, Vol. 233, American Astronomical Society Meeting Abstracts #233, 245.05
  * Capak et al. (2019) Capak, P., et al. 2019, Astro2020 Science White Paper: Synergizing Deep Field Programs Across Multiple Surveys
  * Chang et al. (2019) Chang, P., et al. 2019, Astro2020 Science White Paper: Cyberinfrastructure Requirements to Enhance Multi-messenger Astrophysics
  * Chanover et al. (2019) Chanover, N., et al. 2019, Astro2020 Science White Paper: Triggered High-Priority Observations of Dynamic Solar System Phenomena
  * Chary et al. (2019) Chary, R., et al. 2019, Astro2020 Science White Paper: Cosmology in the 2020s Needs Precision and Accuracy: The Case for Euclid/LSST/WFIRST Joint Survey Processing
  * Collberg & Proebsting (2016) Collberg, C., & Proebsting, T. A. 2016, Commun. ACM, 59, 62
  * Cooray et al. (2019) Cooray, A., et al. 2019, Astro2020 Science White Paper: Cosmic Dawn and Reionization: Astrophysics in the Final Frontier
  * Cowperthwaite et al. (2019) Cowperthwaite, P., et al. 2019, Astro2020 Science White Paper: Joint Gravitational Wave and Electromagnetic Astronomy with LIGO and LSST in the 2020’s
  * Cox (2017) Cox, P. 2017, in American Astronomical Society Meeting Abstracts, Vol. 229, American Astronomical Society Meeting Abstracts #229, 411.07
  * Cuby et al. (2019) Cuby, J., et al. 2019, Astro2020 Science White Paper: Unveiling Cosmic Dawn: the synergetic role of space and ground-based telescopes
  * Dey et al. (2019) Dey, A., et al. 2019, Astro2020 Science White Paper: Mass Spectroscopy of the Milky Way
  * Dickinson et al. (2019) Dickinson, M., et al. 2019, Astro2020 Science White Paper: Observing Galaxy Evolution in the Context of Large-Scale Structure
  * Donoho et al. (2009) Donoho, D. L., Maleki, A., Rahman, I. U., Shahram, M., & Stodden, V. 2009, Computing in Science & Engineering, 11, 8
  * Dore et al. (2019) Dore, O., et al. 2019, Astro2020 Science White Paper: WFIRST: The Essential Cosmology Space Observatory for the Coming Decade
  * Dvorkin et al. (2019) Dvorkin, C., et al. 2019, Astro2020 Science White Paper: Neutrino Mass from Cosmology: Probing Physics Beyond the Standard Model
  * Eifler et al. (2019) Eifler, T., et al. 2019, Astro2020 Science White Paper: Partnering space and ground observatories - Synergies in cosmology from LSST and WFIRST
  * Eracleous et al. (2019) Eracleous, M., et al. 2019, Astro2020 Science White Paper: An Arena for Multi-Messenger Astrophysics: Inspiral and Tidal Disruption of White Dwarfs by Massive Black Holes
  * Fabbiano et al. (2019a) Fabbiano, G., et al. 2019a, Astro2020 Science White Paper: Increasing the Discovery Space in Astrophysics The Exploration Question for Compact Objects
  * Fabbiano et al. (2019b) —. 2019b, Astro2020 Science White Paper: Increasing the Discovery Space in Astrophysics The Exploration Question for Cosmology
  * Fabbiano et al. (2019c) —. 2019c, Astro2020 Science White Paper: Increasing the Discovery Space in Astrophysics The Exploration Question for Galaxy Evolution
  * Fabbiano et al. (2019d) —. 2019d, Astro2020 Science White Paper: Increasing the Discovery Space in Astrophysics The Exploration Question for Planetary Systems
  * Fabbiano et al. (2019e) —. 2019e, Astro2020 Science White Paper: Increasing the Discovery Space in Astrophysics: The Exploration Question for Resolved Stellar Populations
  * Fabbiano et al. (2019f) —. 2019f, Astro2020 Science White Paper: Increasing the Discovery Space in Astrophysics The Exploration Question for Stars and Stellar Evolution
  * Fan et al. (2019) Fan, X., et al. 2019, Astro2020 Science White Paper: The First Luminous Quasars and Their Host Galaxies
  * Ferraro et al. (2019) Ferraro, S., et al. 2019, Astro2020 Science White Paper: Inflation and Dark Energy from spectroscopy at z > 2
  * Ford et al. (2019) Ford, E., et al. 2019, Astro2020 Science White Paper: Advanced Statistical Modeling of Ground-Based RV Surveys as Critical Support for Future NASA Earth-Finding Missions
  * Furlanetto et al. (2019) Furlanetto, S., et al. 2019, Astro2020 Science White Paper: Synergies Between Galaxy Surveys and Reionization Measurements
  * Gaudi et al. (2019) Gaudi, B., et al. 2019, Astro2020 Science White Paper: “Auxiliary” Science with the WFIRST Microlensing Survey
  * Geach et al. (2019) Geach, J., et al. 2019, Astro2020 Science White Paper: The case for a ’sub-millimeter SDSS’: a 3D map of galaxy evolution to z 10
  * Geiger et al. (2018) Geiger, R. S., Cabasse, C., Cullens, C. Y., et al. 2018, Career Paths and Prospects in Academic Data Science: Report of the Moore-Sloan Data Science Environments Survey
  * GitHub (2016) GitHub. 2016, online
  * Gluscevic et al. (2019) Gluscevic, V., et al. 2019, Astro2020 Science White Paper: Cosmological Probes of Dark Matter Interactions: The Next Decade
  * Gnau (2017) Gnau, S. 2017, online
  * Graham et al. (2019) Graham, M., et al. 2019, Astro2020 Science White Paper: Discovery Frontiers of Explosive Transients: An ELT and LSST Perspective
  * Grin et al. (2019) Grin, D., et al. 2019, Astro2020 Science White Paper: Gravitational probes of ultra-light axions
  * Hannay et al. (2009) Hannay, J. E., MacLeod, C., Singer, J., et al. 2009, in 2009 ICSE Workshop on Software Engineering for Computational Science and Engineering (IEEE)
  * Hanselman (2006) Hanselman, S. 2006, Sandcastle - Microsoft CTP of a Help CHM file generator on the tails of the death of NDoc
  * Holler et al. (2019) Holler, B., et al. 2019, Astro2020 Science White Paper: “It’s full of asteroids!”: Solar system science with a large field of view
  * Hornik (2018) Hornik, K. 2018, R FAQ
  * Huber et al. (2019) Huber, D., et al. 2019, Astro2020 Science White Paper: Stellar Physics and Galactic Archeology using Asteroseismology in the 2020’s
  * Hunter (2007) Hunter, J. D. 2007, Computing In Science & Engineering, 9, 90
  * Ivezić et al. (2014) Ivezić, Ž., Connolly, A., Vanderplas, J., & Gray, A. 2014, Statistics, Data Mining and Machine Learning in Astronomy (Princeton University Press)
  * Jao et al. (2018) Jao, W.-C., Henry, T. J., Gies, D. R., & Hambly, N. C. 2018, The Astrophysical Journal, 861, L11
  * Jones et al. (2001–) Jones, E., Oliphant, T., Peterson, P., et al. 2001–, SciPy: Open source scientific tools for Python, [Online; accessed 2019-03-14]
  * Katz et al. (2018) Katz, D. S., Druskat, S., Haines, R., Jay, C., & Struck, A. 2018, arXiv e-prints, arXiv:1807.07387
  * Kirkpatrick et al. (2019) Kirkpatrick, J., et al. 2019, Astro2020 Science White Paper: Opportunities in Time-domain Stellar Astrophysics with the NASA Near-Earth Object Camera (NEOCam)
  * Kluyver et al. (2016) Kluyver, T., Ragan-Kelley, B., Pérez, F., et al. 2016, in Positioning and Power in Academic Publishing: Players, Agents and Agendas, ed. F. Loizides & B. Schmidt, IOS Press, 87
  * Kohl (2015) Kohl, M. 2015, Introduction to statistical data analysis with R (London: bookboon.com)
  * Kollmeier et al. (2019) Kollmeier, J., et al. 2019, Astro2020 Science White Paper: Precision Stellar Astrophysics and Galactic Archeology: 2020
  * Kupfer et al. (2019) Kupfer, T., et al. 2019, Astro2020 Science White Paper: A Summary of Multimessenger Science with Galactic Binaries
  * LeClair (2016) LeClair, H. 2016, online
  * Lehner et al. (2019) Lehner, N., et al. 2019, Astro2020 Science White Paper: Following the Metals in the Intergalactic and Circumgalactic Medium over Cosmic Time
  * Li et al. (2019) Li, T., et al. 2019, Astro2020 Science White Paper: Dark Matter Physics with Wide Field Spectroscopic Surveys
  * Limoncelli (2018) Limoncelli, T. A. 2018, Queue, 16, 20:13
  * Lintott et al. (2011) Lintott, C., Schawinski, K., Bamford, S., et al. 2011, MNRAS, 410, 166
  * Littenberg et al. (2019) Littenberg, T., et al. 2019, Astro2020 Science White Paper: GRAVITATIONAL WAVE SURVEY OF GALACTIC ULTRA COMPACT BINARIES
  * Maccarone et al. (2019) Maccarone, T., et al. 2019, Astro2020 Science White Paper: Populations of Black holes in Binaries
  * Mandelbaum et al. (2019) Mandelbaum, R., et al. 2019, Astro2020 Science White Paper: Wide-field Multi-object Spectroscopy to Enhance Dark Energy Science from LSST
  * Mantz et al. (2019) Mantz, A., et al. 2019, Astro2020 Science White Paper: The Future Landscape of High-Redshift Galaxy Cluster Science
  * Marshall et al. (2015) Marshall, P. J., Lintott, C. J., & Fletcher, L. N. 2015, ARA&A, 53, 247
  * McQueen et al. (2016) McQueen, J., Meila, M., VanderPlas, J., & Zhang, Z. 2016, arXiv e-prints, arXiv:1603.02763
  * Meeburg et al. (2019) Meeburg, P., et al. 2019, Astro2020 Science White Paper: Primordial Non-Gaussianity
  * Momcheva et al. (2019) Momcheva, I., Smith, A. M., & Fox, M. 2019, in American Astronomical Society Meeting Abstracts, Vol. 233, American Astronomical Society Meeting Abstracts #233, 457.06
  * Momcheva & Tollerud (2015) Momcheva, I., & Tollerud, E. 2015, arXiv e-prints, arXiv:1507.03989
  * Morris (2016) Morris, K. 2016, Infrastructure as Code: Managing Servers in the Cloud, Safari Books Online (O’Reilly Media)
  * Moustakas et al. (2019) Moustakas, L., et al. 2019, Astro2020 Science White Paper: Quasar microlensing: Revolutionizing our understanding of quasar structure and dynamics
  * National Academies of Sciences, Engineering, and Medicine (2018) National Academies of Sciences, Engineering, and Medicine. 2018, Open Source Software Policy Options for NASA Earth and Space Sciences (Washington, DC: The National Academies Press)
  * Newman et al. (2019) Newman, J., et al. 2019, Astro2020 Science White Paper: Deep Multi-object Spectroscopy to Enhance Dark Energy Science from LSST
  * Ntampaka et al. (2019) Ntampaka, M., et al. 2019, Astro2020 Science White Paper: The Role of Machine Learning in the Next Decade of Cosmology
  * Olsen et al. (2019) Olsen, K., et al. 2019, Astro2020 Science White Paper: Science Platforms for Resolved Stellar Populations in the Next Decade
  * Palmese et al. (2019) Palmese, A., et al. 2019, Astro2020 Science White Paper: Gravitational wave cosmology and astrophysics with large spectroscopic galaxy surveys
  * Pedregosa et al. (2011) Pedregosa, F., Varoquaux, G., Gramfort, A., et al. 2011, Journal of Machine Learning Research, 12, 2825
  * Pevtsov et al. (2019) Pevtsov, A., et al. 2019, Astro2020 Science White Paper: Historical astronomical data: urgent need for preservation, digitization enabling scientific exploration
  * Pisani et al. (2019) Pisani, A., et al. 2019, Astro2020 Science White Paper: Cosmic voids: a novel probe to shed light on our Universe
  * Pooley et al. (2019) Pooley, D., et al. 2019, Astro2020 Science White Paper: The Most Powerful Lenses in the Universe: Quasar Microlensing as a Probe of the Lensing Galaxy
  * Raymond (2001) Raymond, E. S. 2001, The Cathedral and the Bazaar: Musings on Linux and Open Source by an Accidental Revolutionary (Sebastopol, CA, USA: O’Reilly & Associates, Inc.)
  * Rebull et al. (2018) Rebull, L. M., French, D. A., Laurence, W., et al. 2018, Phys. Rev. Phys. Educ. Res., 14, 020102
  * Research Software Engineers International (2018) Research Software Engineers International. 2018
  * Rhodes et al. (2019a) Rhodes, J., et al. 2019a, Astro2020 Science White Paper: Cosmological Synergies Enabled by Joint Analysis of Multi-probe data from WFIRST, Euclid, and LSST
  * Rhodes et al. (2019b) —. 2019b, Astro2020 Science White Paper: The End of Galaxy Surveys
  * Rix et al. (2019) Rix, H., et al. 2019, Astro2020 Science White Paper: Binaries Matter Everywhere: from Precision Calibrations to Re-Ionization and Gravitational Waves
  * Sanderson et al. (2019) Sanderson, R., et al. 2019, Astro2020 Science White Paper: The Multidimensional Milky Way
  * Shen et al. (2019) Shen, Y., et al. 2019, Astro2020 Science White Paper: Mapping the Inner Structure of Quasars with Time-Domain Spectroscopy
  * Slosar et al. (2019a) Slosar, A., et al. 2019a, Astro2020 Science White Paper: Dark Energy and Modified Gravity
  * Slosar et al. (2019b) —. 2019b, Astro2020 Science White Paper: Scratches from the Past: Inflationary Archaeology through Features in the Power Spectrum of Primordial Fluctuations
  * Smith et al. (2016) Smith, A. M., Katz, D. S., & and, K. E. N. 2016, PeerJ Computer Science, 2, e86
  * Smith et al. (2018) Smith, A. M., Niemeyer, K. E., Katz, D. S., et al. 2018, PeerJ Computer Science, 4, e147
  * SunPy Community et al. (2015) SunPy Community, T., Mumford, S. J., Christe, S., et al. 2015, Computational Science and Discovery, 8, 014009
  * Suzuki & Fukugita (2018) Suzuki, N., & Fukugita, M. 2018, AJ, 156, 219
  * van der Walt et al. (2011) van der Walt, S., Colbert, S. C., & Varoquaux, G. 2011, Computing in Science & Engineering, 13, 22
  * Varoquaux (2013) Varoquaux, G. 2013, online
  * Vishniac & Lintott (2018) Vishniac, E. T., & Lintott, C. 2018, The Astrophysical Journal, 869, 156
  * von Hardenberg et al. (2019) von Hardenberg, A., Obeng, A., Pawlik, A., et al. 2019, Data Carpentry: R for data analysis and visualization of Ecological Data
  * Wang et al. (2019) Wang, Y., et al. 2019, Astro2020 Science White Paper: Illuminating the dark universe with a very high density galaxy redshift survey over a wide area
  * Weiner et al. (2009) Weiner, B., Blanton, M. R., Coil, A. L., et al. 2009, in astro2010: The Astronomy and Astrophysics Decadal Survey, Vol. 2010, P61
  * Williams et al. (2019) Williams, B., et al. 2019, Astro2020 Science White Paper: Far Reaching Science with Resolved Stellar Populations in the 2020s
  * Wilson (2013) Wilson, G. 2013, arXiv e-prints, arXiv:1307.5448
  * Wilson et al. (2014) Wilson, G., Aruliah, D. A., Brown, C. T., et al. 2014, PLOS Biology, 12, 1
  * Yale Roundtable Declaration (2010) Yale Roundtable Declaration. 2010, Computing in Science & Engineering, 12, 8
  * Yee et al. (2019) Yee, J., et al. 2019, Astro2020 Science White Paper: The Scientific Context of WFIRST, Microlensing in the 2020s