Paper

Walter et al.

[$\ast$]Corresponding author<EMAIL_ADDRESS>
0Year 0Year 0Year

# InfoSec.pptx: A Longitudinal Study of Speakers, Topics, and Sponsors at
Security Conferences in Academia and Industry

Lukas Walter  Clemens Sauerwein  Daniel W. Woods  Department of Computer
Science, University of Innsbruck, Austria  School of Informatics, University
of Edinburgh, UK

(Date; Date; Date)

###### Abstract

Security conferences are important venues at which academics and practitioners
share knowledge about new attacks and state-of-the-art defenses. Despite this,
researchers have not studied who shares information and about which security
topics. To address this, our study characterizes the speakers, sponsors, and
topics presented at the most prestigious academic and industry conferences. We
collect a longitudinal data set that contains 9,728 abstracts and 1,686
sponsors across 4 academic and 6 industry conferences. There is limited
knowledge sharing between industry and academia. Conferences vary
significantly in the equality of how talks/authorship is distributed across
individuals. The topics of academic and industry abstracts display consistent
coverage of techniques within the MITRE ATT&CK framework. Top tier academic
conferences, as well as DEFCON and Black Hat, inconsistently address the
governance, response and recovery functions of the NIST Cybersecurity
Framework. Commercial InfoSec and insurance conferences (RSA, Gartner, Advisen
and NetDillgience) cover the framework more consistently. Prevention and
detection remain the most common topic of talks, with no clear temporal trend.

###### keywords:

cybersecurity, science of security, sociology, information sharing,
quantitative social science

##  1 Introduction

Information security knowledge is produced by both academic and industry
researchers. Norms across both communities differ in terms of questions like
what “good” research looks like, and the role of peer review However, both
communities value gathering in dimly lit conference rooms to listen to
researchers explain findings with the support of a slide deck.

To attend conferences, attendees travel to foreign cities, sometimes across
continents, and spend thousands of dollars on registration. Although some of
these activities are not directly related to formal knowledge exchange (e.g.
hallway chats and other extracurricular activities), most participants spend
hours listening to talks providing knowledge about novel attacks, mitigation
measures, lessons learned, and security concepts. For some attendees in both
academia and industry, these conferences are akin to a pilgrimage.

Despite the attendees and their employers investing considerable time and
money, security conferences have not been studied as part of the security
information sharing ecosystem skopik2016problem ; laube2017strategic ;
ainslie2023cyber . We are not aware of research into the speakers or topics at
industry conferences. Studies of academic security conferences focus on
bibliometrics wendzel2020not ; balzarot2022  and which evaluation method was
used in a study carver2016establishing ; burcham2017characterizing .

Further investigation is needed because conferences are not a neutral forum to
exchange ideas. While conferences provide a platform for selected researchers
to present their work, the low acceptance rates mean other researchers are
silenced. In selecting the talks that thousands of attendees watch, program
committees control which security topics receive attention. This may influence
which solutions are built and implemented. Further, paid speaker slots at
industry conferences undermine the so-called market for ideas drezner2017ideas
.

These potential problems and the lack of prior research motivate a study that
characterizes prestigious security conferences. Our study asks: RQ1: Who
speaks at security conferences?; RQ2: About which topics?; and RQ3: How does
this vary by conference and over time? We collect a sample of titles,
abstracts and speakers/authors across $4$ academic and $6$ industry
conferences from 2014 to 2022. To analyze topics, we classified the abstracts
into offensive, defensive, or neutral (for theoretical papers). We then mapped
offensive talks to the MITRE ATT&CK Framework111https://attack.mitre.org/
(accessed 31 July 2023), and defensive talks to the NIST Cybersecurity
Framework222https://www.nist.gov/cyberframework (accessed 31 July 2023).

The rest of this paper is structured as follows. Section 2 discusses related
work in the field of information sharing. Section 3 describes our research
methodology. Section 4 presents the results of our study. Section 5 discusses
the limitations of the study. Section 6 discusses the implications of our
results and makes recommendations. Section 7 concludes the paper.

##  2 Related Work

Defenders use and share many forms of security information to better calibrate
their defensive posture to the threat landscape. Most research focuses on
sharing real-time technical indicators laube2017strategic , which we review in
Section 2.1. Other communication channels exist by which defenders can learn
about security, which are covered in Section 2.2.

###  2.1 Information Sharing

The two most common forms of security information sharing are threat
intelligence and vulnerability management. A range of schema and standards
have been introduced to facilitate sharing kampanakis2014security , which
includes MAEC, OVAL, XCCDF, CPE, CVE, CVSS and so on. Governments have passed
regulations to support information sharing yang2020impact , funded Computer
Emergency Response Teams to coordinate vulnerability disclosure
slayton2020trusting , and directly shared threat
intelligence333https://www.cisa.gov/known-exploited-vulnerabilities-catalog.
Commercial providers also offer threat intelligence (TI) and surrounding
services wagner2019cyber ; sauerwein2017threat ; brown2015cyber ;
sauerwein2019analysis . The associated platforms provide information
enrichment and analysis functions to obtain targeted and actionable threat
intelligence sauerwein2021threat ; dandurand2013towards ; brown2015cyber ;
chismon2015threat .

Recent empirical studies have questioned the quality of threat intelligence.
Li et al. li2019reading  find little overlap between different TI feeds and
cannot identify a way to distinguish quality, which confirms findings from
other studies thomas2016abuse ; bouwman2022helping . Expert interviews with TI
practitioners reveal processes to evaluate quality are informal
bouwman2020different , which is to be expected given this remains an open
research challenge zibak2022threat . Encouragingly, a study of threat
intelligence during COVID-19 found evidence that collaboration, in the form of
aggregating commercial TI feeds, improved coverage of threats
bouwman2022helping .

Turning to vulnerability management, bug bounties have proliferated, run both
independently by vendors zhao2017devising  and on platforms like HackerOne
sridhar2021hacking . Empirical evaluations support the efficacy of bug bounty
programs over internal testing finifter2013empirical ; zhao2015empirical ,
even though they follow similar processes votipka2018hackers . However, many
firms do not patch even when the vulnerability is known and the patch is
available frei2006large ; li2017large . This has motivated research into
understanding expert users who apply patches li2019keepers ; mathur2016they ,
and notifications that nudge firms towards patching stock2016hey ; li2016you ;
ccetin2019tell .

So much information is shared about security vulnerabilities that prioritizing
what to pay attention to has become a research problem jacobs2020improving ;
de2023no . Information overload is also a problem for threat intelligence
bouwman2020different . This motivates research exploring online advice and
knowledge sharing that may be easier for the recipients to digest than
technical indicators.

###  2.2 Advice and Knowledge Sharing

Online forums are often used to solve security and privacy problems
li2023itsup ; tahaei2020understanding , including by developers acar2016you .
In some cases, this involves copying insecure code snippets directly from the
forum post fischer2017stack . This has motivated research into how to nudge
developers towards better choices fischer2019stack .

Online security advice targeted at the general public is also widely available
redmiles2016think . An analysis of over a thousand sources of advice reveals
that the specific recommendations are actionable and comprehensible, albeit
with room for improvement redmiles2020comprehensive . However, the sources
recommended 374 unique behaviors, which leads to information overload
redmiles2020comprehensive . Some users address the quality issue by looking to
trusted sources redmiles2016think .

Conferences address this issue by curating the agenda, via peer review in
academia. Research into what information is shared at these conferences has
been small-scale. For example, Carver et al. carver2016establishing  identify
which evaluation method (e.g., proof or empirical) was used by each paper in
the 2015 IEEE Security and Privacy proceedings. This study was extended to
cover the 2016 ACM CCS proceedings burcham2017characterizing . A bibliometric
analysis was conducted of around 30,000 academic computer security papers
wendzel2020not , focusing on statistics like the number of words, authors, and
citations in each paper.

We are not aware of analysis of industry conferences, even though practitioner
conferences have tens of thousands of attendees (e.g. 45k at RSA). Across both
academia and industry, it remains unclear who speaks/publishes at security
conferences (RQ1), about which topics (RQ2), and how this changes over time
and conferences (RQ3).

##  3 Methodology

We focused on the most prestigious conferences because they have the most
influence. For each talk, one could analyze article PDFs, video recordings of
talks, or meta information like abstracts and titles. Although full academic
articles contain the most information, the industry equivalent (white papers)
are not available for all talks. Similarly, video recordings of talks are not
consistently available, and automated transcription is imperfect. For these
reasons, we analyzed abstracts, which are consistently available across
industry and academia.

Section 3.1 describes our corpus of conference abstracts. Section 3.2 explains
how we used natural language processing to classify what each talk was about,
and Section 3.3 describes how we validated this classification. Section 3.4
discusses ethical considerations and open science. Figure 1 provides a high-
level overview of our research design.

Figure 1: Our 2-stage classification uses OpenAI’s gpt model in the first step
and then both gpt and cosine similarity to map the abstracts to cybersecurity
frameworks. We also collect the sponsors of each conference.

###  3.1 Data Collection

##### Conferences

In academia, we focused on ‘Top Tier’ computer security conferences
soneji2022flawed , namely USENIX Security, IEEE S&P, ACM CCS, and NDSS. We did
not consider cryptography venues because they do not map onto either framework
(see Section 3.2). Journals were excluded because the lack of in-person
presentations undermines the comparison to industry conferences.

For industry, we conducted a web search for articles featuring the ‘Top
Security Conferences’. Only Black Hat, DEFCON and RSA were consistently
included in all of these articles. For example, the SANS Institute runs
influential topic-specific conferences (e.g. Threat Intelligence or Network
Security), but no specific event is consistently included as the flagship
event. We consider these seven conferences to be the most prestigious computer
security conferences.

To diversify our sample, we included Gartner Security because of its influence
over procurement decisions and also the two major cyber insurance conferences
(Advisen and NetDiligence). Insurance is increasingly used by firms to manage
cybersecurity risk dambra2020sok . This makes for an interesting comparison
because the insurance community’s roots are in finance rather than IT. Table 1
displays the conferences in our sample, along with some simple statistics.

Conference | Sample Window | Type | Talks per year | Characters in title | Words per abstract | Sponsors | Gini Index
---|---|---|---|---|---|---|---
$\mu$ | $\sigma$ | $\mu$ | $\sigma$ | $\mu$ | $\sigma$ | $\mu$ | $\sigma$ | 1 | 1/N
Black Hat | 2014 - 2022 | industry (InfoSec) | 252 | 53.83 | 54.82 | 26.43 | 170.98 | 99.16 | 179.0 | 54.29 | 0.234 | 0.381
RSA | 2017 - 2022 | industry (InfoSec) | 338.33 | 34.98 | 55.69 | 17.18 | 75.75 | 26.98 |  75.0a |  0.0a | 0.288 | 0.441
DEFCON | 2014 - 2022 | industry (InfoSec) | 92.22 | 32.19 | 56.10 | 25.75 | 176.72 | 77.72 | 0 | 0 | 0.179 | 0.382
Gartner Security | 2022 | industry (InfoSec) | 321.0 | 0.0 | 68.07 | 19.38 | 66.85 | 20.38 | N/A* | N/A* | 0.328 | 0.427
Advisen | 2014 - 2022 | industry (Insurance) | 10.0 | 4.92 | 38.81 | 17.62 | 58.67 | 22.25 |  18.13b |  5.95b | 0.244 | 0.383
NetDiligence | 2019 - 2022 | industry (Insurance) | 25.0 | 6.22 | 51.50 | 20.35 | 36.76 | 27.36 | 56.50 | 15.32 | 0.159 | 0.269
USENIX Security | 2014 - 2022 | academic | 132.78 | 68.33 | 72.30 | 22.40 | 184.12 | 62.10 | 15.22 | 3.33 | 0.284 | 0.417
IEEE S&P | 2014 - 2022 | academic | 82.22 | 34.08 | 69.77 | 21.42 | 222.52 | 63.53 | 14.67 | 6.22 | 0.280 | 0.380
ACM CCS | 2014 - 2022 | academic | 166.33 | 34.47 | 67.99 | 21.44 | 186.21 | 82.81 |  13.87c |  3.33c | 0.325 | 0.451
NDSS | 2014 - 2022 | academic | 73.0 | 15.80 | 72.11 | 19.79 | 186.54 | 82.22 |  10.14d |  3.27d | 0.209 | 0.376

Table 1: Descriptive statistic display the mean value across all years. The
Gini index is calculated with the total number of talks per individual across
all years. a contains data only from year 2022. b year 2020 is missing. c year
2018 is missing. d only years from 2016 - 2022. N/A* when sponsor data was not
available.

##### Data Extraction

The data was obtained from the conference websites or extracted directly from
the proceedings via web content mining. We used a semi-automated process to
extract information about speakers, affiliations, talk titles, abstracts, and
dates. The academic talks include keynotes and paper presentations for which
an abstract was available, but we did not include poster sessions. For
industry conferences, we considered all talks with an abstract listed in the
conference agenda and ignored sessions such as networking sessions, welcome
remarks, and opening speeches.

The data set comprises a total of 9,728 talks (5,639 industry talks and 4,089
academic talks). We also collected 1,686 entries for sponsors of information
security conferences, 451 for academic conferences, and 371 for cyber
insurance. Some firms sponsored conferences for multiple years.

###  3.2 Data Analysis

The analysis for RQ1 involved calculating descriptive statistics about the
distribution of talks, speakers, and sponsors. Answering RQ2 requires natural
language processing (NLP) techniques, as illustrated in Figure 1. We followed
a deductive approach by mapping abstracts to established security frameworks,
which describe different areas and aspects of defensive and offensive
security. The two main decisions were which _frameworks_ and _NLP techniques_
to use.

Choice of Framework The framework would ideally be: (i) comprehensive, (ii)
widely used in practice; and (iii) freely available. The first criterion
ensures we can meaningfully map more abstracts to the framework, which would
not be possible something like a digital forensics standard. The second
criterion makes our results easier to understand as some readers will be
familiar with the categories from the framework. The final criterion enables
others to replicate our work.

For offensive frameworks, MITRE ATT&CK and Cyber Kill Chain framework were
considered the most appropriate choices. The ATT&CK framework offers greater
depth, satisfying the first criterion. For defensive frameworks, we considered
the NIST Cybersecurity Framework, ISO 27001 and CIS Critical Security
Controls. We excluded ISO 27001 due to the cost of access. The CIS standard
was excluded because NIST CSF is broader, including governance. Although the
choice of framework is important, mappings between frameworks are widely
available444For example, CIS Controls v8 mapped to NIST CSF is available here:
https://www.cisecurity.org/insights/white-papers/cis-controls-v8-mapping-to-
nist-csf.

NLP Techniques For the first classification phase, we employed OpenAI’s
gpt-3.5-turbo model to classify abstracts into offensive, defensive, or
neutral via this prompt:

> “is this text cyber security offensive or defensive (just answer one word,
> namely: offensive, defensive or neutral)? $<$abstract$>$”

Neutral talks, which make up a minority (17%) of the corpus, are not further
classified.

For the second classification, we referred to the official documentation of
each standard to establish themes and sub-themes. For MITRE ATT&CK Framework,
we used the 14 tactics as themes and the 196 techniques as sub-themes. For
NIST Cybersecurity Framework, we used the 22 categories as themes and the 98
sub-categories as sub-themes. We estimated the similarity between the abstract
and the corresponding (sub) theme using two NLP techniques, namely cosine
similarity and a Large Language Model.

To calculate _cosine similarity_ , we converted abstracts and each (sub-)theme
to a bag of words. We removed stop words using the python nltk stopwords list
and then used the nltk wordnet lemmatizer to group the different word forms
together. We then compared the talk to each category from the framework using
cosine similarity. To compute this, we normalized embeddings with the
SentenceTransformer model ‘stsb-mpnet-base-v2’. This process was repeated for
the sub-categories of the NIST Cybersecurity Framework and also the techniques
from MITRE ATT&CK Framework.

For the _Large Language Model_ , we again used OpenAI’s gpt-3.5-turbo model
using the following prompt:

> “What are the MITRE ATT&CK tactics that can be identified in the following
> text. Only return the tactics which are identified: $<$abstract$>$”

We used a similar prompt for the categories of NIST CSF. Each theme was
individually submitted as a separate query to the model, which produced either
a 0 or 1 as output for each talk. To calculate aggregate similarity to each
theme, we took the average across all talks in a specific sub-sample.

###  3.3 Validation

The absence of an existing ground truth mapping for abstract talks to cyber
security frameworks made validation difficult. Further, frameworks are high-
level with blurry boundaries between categories. To address these issues, we
first evaluated the first classification using a hand-labeled data set.
Second, we conducted a comparative analysis of the agreement between the two
natural language processing techniques. Third, we measured the similarity of
offensive, neutral, and defensive talks against both frameworks.

First, we used a human-labeled data set comprising 50 abstracts to evaluate
the performance of OpenAI’s gpt-3.5-turbo model at classifying abstracts as
offensive, defensive, or neutral. The machine and human agreed on 98% of
abstracts.

We then compared the topic classifications across NLP techniques. Since GPT
outputs are either 0 or 1, we first apply a threshold to the cosine similarity
scores, which are floats between 0 and 1. We set the threshold such that both
GPT and cosine similarity would have the same number of 0s and 1s. Otherwise
the scores could be perfectly aligned, but appear to disagree purely due to
setting different thresholds. For the ATT&CK and NIST Cybersecurity framework
categories, the two techniques agreed for 67% and 81% of the abstracts
respectively. This ranged across individual categories, from a low of 52% up
to 95% (see Table 3). The categories with high accuracy are partly driven by
the class imbalance as so few defensive abstracts cover _Maintenance_ (from
NIST CSF) or _Resource Development_ (from MITRE ATT&CK).

Third, it might be expected that offensive or defensive talks display the
highest similarity to the offensive or defensive frameworks, while neutral
talks would exhibit low similarity to both frameworks. Table 2 shows that the
cosine similarity scores display the expected ranking. However, this is not
true for OpenAI’s gpt-3.5-turbo model, as offensive talks display higher
similarity to both frameworks, and defensive talks display the lowest
similarity to NIST CSF.

Talk | Cosine Similarity | gpt-3.5-turbo
---|---|---
Category | NIST CSF | ATT&CK | NIST CSF | ATT&CK
Offensive | 0.298 | 0.452 | 0.160 | 0.286
Defensive | 0.317 | 0.411 | 0.135 | 0.236
Neutral | 0.266 | 0.342 | 0.149 | 0.155
Table 2: Average similarity of each category of talks to each framework using
both NLP techniques.

Summary Overall, these three validation tests suggest our mapping captures
some signal, albeit with considerable noise. Some noise is to be expected
given the subjective nature of the task. The GPT method reliably classifies
talks as offensive, defensive or neutral. However, the agreement between NLP
techniques is disappointing.

Disagreement results from a mixture of errors/hallucinations and also
differences in approach. Cosine similarity focuses on keywords and phrases
used in the descriptions of each framework. This can lead to false positives
when these words are used in a different context, while also leading to false
negatives when abstracts use different words with the same meaning. On the
other hand, the GPT method captures how words and phrases are used in the data
set, which is mainly user-submitted data from the Internet, which is
vulnerable to context drift.

Going forward, we will report on aggregate results from OpenAI’s gpt-3.5-turbo
because it is a state-of-the-art LLM. Aggregating the classifications helps
mitigate random errors as these should cancel out. However, it does not
mitigate systemic bias associated with GPT, which we discuss in Section 5. In
the Appendix, we also display figures based on an ensemble model, which
outputs an equal weighting of the GPT and cosine similarity techniques.

ATT&CK Tactic | Agree | NIST CSF Function | Agree
---|---|---|---
Credential Access | 0.63 | Asset Management | 0.95
Execution | 0.52 | Business Environment | 0.95
Impact | 0.70 | Governance | 0.90
Persistence | 0.56 | Risk Assessment | 0.84
Privilege Escalation | 0.65 | Risk Management Strategy | 0.93
Lateral Movement | 0.79 | Supply Chain Risk Mgmt | 0.93
Defense Evasion | 0.57 | Identity Mgmt and Access Control | 0.86
Exfiltration | 0.71 | Awareness and Training | 0.90
Discovery | 0.52 | Data Security | 0.90
Collection | 0.54 | Info. Prot. Processes and Proc. | 0.94
Resource Dev. | 0.98 | Maintenance | 0.95
Reconnaissance | 0.90 | Protective Technology | 0.58
Command & Control | 0.73 | Anomalies and Events | 0.65
Initial Access | 0.56 | Security Continuous Monitoring | 0.84
|  | Detection Processes | 0.62
|  | Response Planning | 0.66
|  | Communications (Respond) | 0.78
|  | Analysis | 0.67
|  | Mitigation | 0.67
|  | Improvements (Respond) | 0.65
|  | Recovery Planning | 0.79
|  | Improvements (Recover) | 0.79
|  | Communications (Recover) | 0.81
Mean | 0.67 | Mean | 0.81
Table 3: Fraction of abstracts per category for which the GPT and cosine
similarity techniques agree.

###  3.4 Open Science & Ethics

We will release our corpus of titles, abstracts, speakers, and sponsors in an
open source repository, as well as our analysis scripts. We hope this allows
other researchers to extend this research project. We discuss these directions
in Section 6.

Our main ethical concern is that we process and then share personal data (e.g.
author names and affiliations). It would be impracticable to collect
affirmative consent from speakers. We believe the benefits in terms of open
science outweigh the potential privacy invasion, not least because presenting
at conferences is a public act.

##  4 Results

This section presents the results of our analysis on the composition of talks
and sponsors at each conference (Section 4.1), speakers (Section 4.2), and
abstract topics (Section 4.3).

###  4.1 Conference Statistics

RSA and Black Hat had the most talks across the sample period (see Figure 2).
Volatility in the size of Black Hat since 2019 is driven by the conference
introducing/discontinuing sessions like Day Zero, CISO Summit, and Sponsored
Workshops. The academic conferences display the highest rates of growth, as
noted by Balzarotti balzarot2022 . USENIX Security doubled in size from 2019
to 2021. The industry conferences do not display the same growth trend.
Insurance conferences had the fewest talks, with the absolute number remaining
stable over time. This is perhaps surprising given the market grew by over
360% between 2015 and 2021 naic2022report .

Figure 2: Number of talks at each conference over time. Figure 3: Top sponsors
of academic conferences. The number on top of each bar is the mean sponsorship
tier of that sponsor.

The academic conferences all had the longest abstracts in terms of word count,
which can be considered an imperfect proxy for information content. Table 1
shows that IEEE S&P displays the highest average at 223 words. Abstracts at
Black Hat and DEFCON had a similar word count to the other three academic
conferences. These five conferences have a mean word count of between 170 and
190, which makes sense given original research is often presented at Black Hat
and DEFCON. The conferences with the lowest word counts place less emphasis on
research. There was less variance in the length of titles, both within and
between conferences.

Black Hat has the most sponsors with an average of 197 per year (see Table 1).
It is followed by RSA (75), Netdiligence (57), and Advisen (18). We were
unable to find comprehensive sponsorship data for Gartner. The academic
conferences all have between 10 and 16 sponsors per year. DEFCON does not
accept sponsors. Conferences with fewer sponsors typically had longer
abstracts.

Figure 4: Speaker affiliation and sponsor correlation of Black Hat.

The academic conferences are sponsored by a mixture of public and private
sources (see Figure 3). Large global technology companies, along with the
National Science Foundation, support all four conferences in a similar
proportion. InfoSec vendors tend to support specific conferences. Non-profits
like the EFF, the FreeBSD Foundation, and DMTF only sponsor USENIX Security.

Although we could not collect information on the price of sponsorship, the
numbers on top of the bars in Figure 3 represent the average sponsor tier of
that company, where 1 is the highest. This shows that the non-profits like EFF
are consistently in lower tiers, while BigTech companies tend to be in the
higher tiers. However, the US-based National Science Foundation is most
consistently a tier one sponsor. No other country has an equivalent research
institution that consistently sponsors these conferences.

The industry conferences are mainly sponsored by InfoSec vendors, with large
technology companies sponsoring less frequently. This can be seen in Figure 19
in the Appendix. The analysis of sponsors of industry conferences is less
comprehensive because of gaps in the data. The insurance conferences contain a
wide range of sponsors (see Figure 19 in the Appendix). These include not just
insurance companies, but also law firms, digital forensics providers,
communications consultants, and data analytics firms. This is likely because
these firms receive work from insurers woods2023lessons .

To probe whether sponsorship influences the number of talks, we analyze the
only industry security conference for which we had longitudinal sponsor
information, namely Black Hat. Across all years, the conference had $592$
unique sponsors and $1493$ unique affiliation across speakers, which had an
intersection of just $219$. This means that 85% of the speakers’ affiliated
organizations never sponsored the conference, and 63% of sponsors did not have
an affiliated speaker at Black Hat. Figure 4 shows the relationship between
number of times an organization sponsored Black Hat and the number of
affiliated talks. The two outliers are Microsoft and Google, who respectively
had 61 and 55 affiliated talks.

Figure 5: Talks per individual for the top 30 most frequent speakers.

###  4.2 Speaker Statistics

This subsection examines the distribution of talks among specific individuals.
Throughout we assume that each unique name corresponds to a single unique
person. Figure 6 shows there is minimal overlap of speakers between industry,
academic, and insurance conferences. For this reason, we proceed by analyzing
each type of conference separately.

Figure 6: Number of unique speakers at each type of conference.

Figure 5 displays the individuals with the most associations with talks,
weighted by $\frac{1}{n}$ where $n$ is the number of authors on a paper or
individuals on a panel. Patrick Wardle is a dramatic outlier with 33 talks
including more than ten talks at each of Black Hat and DEFCON. The rest of the
top 30 industry information security speakers gave between 10 and 15 talks
across all four conferences. Some industry speakers presented exclusively at
one conference, such as Patrick Hevesi at Gartner. No other Top 30 speaker
gave a talk at Gartner. The top insurance speaker had seven talks in total and
just 19 speakers gave more than three talks.

Turning to the academic talks, the top authors have over fifty publications,
although rarely as a single author. Academics are more likely to publish at
multiple conferences than industry speakers. For example, all the top 30
authors have presented at IEEE Security and Privacy, USENIX Security and ACM
CCS, although some authors did not present at NDSS. This kind of analysis
raises the question of how speaking slots are distributed among participants.

Table 1 displays the Gini index for each conference. A score of 0 means all
speakers gave the same number of talks, while a score of 1 means one
individual gave all talks. Academic conferences are similarly equal to
industry conferences when publications count as $\frac{1}{n}$ where $n$ is the
number of co-authors/co-panelists, whereas academic conferences appear
considerably less equal when each publications counts as $1$ (see Table 1).
This corrects for academics who make small contributions to high $n$ papers—we
all have just 24 hours in the day. Both metrics agree that NDSS is the most
equal academic conference, while ACM CCS is the least.

Figure 7: Offensive, defensive, and neutral talks at each conference.

###  4.3 Content of Talks

We now explore the topics of abstracts. Figure 7 shows that defensive talks
are more common than offensive and neutral talks at all conferences apart from
DEFCON, which is explicitly set up for hackers to present their research. At
DEFCON, 64% of the talks were offensive and just 15% (20%) of the talks were
defensive (neutral). Across both cyber insurance conferences, just 1% of talks
were classified as offensive, whereas 55% of insurance talks were neutral and
44% were defensive. The academic conference all accepted more defensive work
than offensive, and some even accepted more neutral than offensive talks.

####  4.3.1 Defensive Talks

Figure 8 shows that the similarity of talks across areas of NIST CSF has
remained relatively stable over time. There is a slight trend of increasing
abstracts about risk assessment and governance, as well as decreasing
similarity to protective technology. This is generally true when considering
trends in topics at individual conferences, and we display some such figures
in the Appendix. For this reason, we proceed by analyzing the abstracts
aggregated across all years. The NIST Cybersecurity Framework is divided into
five core functions: Identify, Protect, Detect, Respond and Recover.

Figure 8: Similarity score of defensive talks mapped to NIST Cybersecurity
Framework over years

Before analyzing the aggregate data, there is one exception to the general
lack of a temporal trend in topics. Figure 9 shows how DEFCON displayed a
considerable temporal shift towards Detection and Response functions of NIST
CSF. This could be evidence of the conference’s commercialization, which is an
ongoing topic of discussion. The equivalent figures for other conferences can
be found in the Appendix.

Figure 9: Similarity score of DEFCON defensive talks mapped to NIST
Cybersecurity Framework over years

##### Identify

The identify function comprises the first six rows in Figure 10. Broadly
speaking, _Identify_ covers how organizations make security decisions. Both
industry InfoSec and insurance talks address these topics more than the
academic talks. The insurance conferences and Gartner display high similarity
to _Risk Assessment_ , which captures the likelihood and impact of different
security incidents, and _Governance_ , which cover the process of establishing
a security policy. It is also notable that the research-oriented industry
information security conferences (DEFCON and Black Hat) display less
similarity to _Identify_. Notably, there is low similarity to the final two
sections: _Risk Management Strategy_ involves understanding the firm’s meta-
reasoning like the assumptions and risk tolerance of the firm, meanwhile
_Supply Chain Risk Management_ involves assessing the risk of external firms.

Researchers should not dismiss the _Identify_ function as ‘non-technical’ and
outside the scope of computer security. Many of the _Identify_ tasks can be
solved using technical data. For example, a recent SoK on asset discovery
highlights how _Asset Management_ can be approached using network measurements
vermeer2021sok . Similarly, _Risk Assessment_ is analogous to cyber incident
prediction, which has been studied as a machine learning problem based on
technical indicators and published at top tier venues liu2015cloudy ;
bilge2017riskteller .

Figure 10: Similarity score of defensive talks mapped to NIST Cybersecurity
Framework.

##### Protect

The protect function covers the next six rows in Figure 10. Broadly speaking,
_Protect_ describes technical controls and organizational processes that
directly prevent incidents from occurring. Of these six categories, all
conferences had the most similarity to _Protective Technology_. Figure 11
displays cosine similarity to each sub-category of _Protective Technology_.
The two sub-categories with the highest similarity concern system hardening
(PR.PT-3) and network security (PR.PT-4). The next two subcategories concern
audits and logging (PR.PT-1), and removable media (PR.PT-2). Finally,
resilience mechanisms like fail-safe and load balancing (PR.PT-5) display the
lowest similarity.

Academic talks display some similarity to _Identity Management and Access
Control_. This include the evergreen research topic that is passwords
morris1979password ; bonneau2012quest . The _Awareness and Training_ category
received more coverage in the industry talks relative to the academic talks.
Academic research related to this may instead be presented at conferences
focusing on Human-Computer Interaction like SOUPS and Chi garfinkel2014usable
.

The remaining three categories had low similarity. _Maintenance_ had the
lowest similarity across all of the defensive categories. This finding
supports the observation that researchers often overlook maintenance even
though it is crucial to successful operations collier2020cybercrime . The
_Information Protection Processes and Procedures_ category consists of
procedures to ensure availability like back-ups, recovery plans and so on.
Industry talks more frequently cover these topics. Low similarity to _Data
Security_ was perplexing given this is so core to computer security. The first
two sub-categories of _Data Security_ , namely data at rest and in transit,
are core academic topics. However, the other sub-categories are organizational
procedures that tend not to be covered at academic conferences.

Figure 11: Cosine similarity score for the category within _Protective
Technology_.

##### Detect

The next three rows concern network monitoring and incident detection. Broadly
speaking, all of the industry conferences displayed moderate similarity to
_Security Continuous Monitoring_ , _Anomalies and Events_ , and _Detection
Processes_. These three categories respectively correspond to: what monitoring
technology is in place; how anomalies are identified and responded to; and the
surrounding organizational processes like testing, communication, and
improvement. NDSS displays higher similarity than the other academic
conferences to all three _Detect_ categories, which likely results from the
conference’s focus on network security. Across the five NIST Cybersecurity
framework functions, the abstracts were most consistently similar to _Detect_.

##### Respond

The respond function comprises the next five rows, which are broadly the steps
taken to investigate and contain an incident. Abstracts were most similar to
_Response Planning_ , which involves establishing incident response procedures
before the incident occurs. The talks also displayed reasonable similarity to
_Mitigation_ , which involves actions to contain and resolve an incident, and
also _Improvements_ , which is about extracting lessons learned from
incidents. This is perhaps surprising given the consensus that the security
community struggles to extract lessons from incidents knake2021learning .

There was low similarity with _Analysis_ category, which includes forensic
investigations and vulnerability disclosure policies, especially at academic
conferences. This could be because researchers present at specialist venues
like _Digital Investigation_ garfinkel2010digital . There was relatively low
similarity to _Communications_ , which involves coordinating with internal and
external stakeholders like law enforcement, especially at academic
conferences.

Much like with _Detect_ , NDSS displays higher similarity to than the other
academic conferences. The insurance conferences, especially NetDilligence,
display remarkably high similarity to all five categories. The high level of
similarity holds even for _Communications_ despite the other conferences
largely ignoring this topic, which supports the theme that insurers have a
different perspective on security.

##### Recover

The final function comprises three categories that all happen post-incident,
which all have equivalent categories in the _Response_ function. All
conferences show less similarity to the _Recover_ category. This suggests
conference talks are more focused on short-term tasks like containing the
incident than the long-term goal of restoring the victim to previous function,
which is the goal of _Recover_. The Advisen conference displays the most
similarity, which makes sense given recovery is the goal of insurance.

####  4.3.2 Offensive Talks

We mapped offensive talks to the MITRE ATT&CK framework. This consists of 14
tactics and each tactic has multiple techniques. Gartner, Advisen and
NetDilligence are not displayed due to a lack of offensive talks.

Figure 12: Similarity score of offensive talks mapped to MITRE over years

Much like with defensive talks, Figure 12 displays no clear temporal trend in
which offensive tactics are covered by conference abstracts. There is,
however, a subtle trend towards _Impact_. We group the 14 tactics into
temporal categories: Pre-compromise, First Compromise, Expansion, and Outcome.

##### Pre-Compromise

The _Reconnaissance_ and _Resource Development_ tactics involve collecting
information and other resources (e.g. acquiring exploits) before the attack is
launched. _Resource development_ is the tactic that the conferences are least
similar to. _Reconnaissance_ , which involves techniques like active scanning
and gathering information on the victim’s network, is also very rarely covered
by talks. These tactics may be infrequently discussed because they are non-
technical.

Figure 13: Similarity score of offensive talks mapped to MITRE ATT&CK

##### First Compromise

The next four tactics concern how the attacker compromises the first system on
a network. _Initial Access_ describes the entry vector into the network, which
includes different types of phishing and account compromises. The industry
talks conferences display moderate similarity to _Initial Access_. USENIX
Security abstracts are most similar to this among the academic conferences.
This breaks from the trend that NDSS talks were most similar to the NIST
Cybersecurity Framework out of the academic conferences.

_Execution_ describes tactics that attackers use to run malicious code on the
victim’s systems. The industry talks display the most similarity to this
category, in particular the Black Hat talks. The academic talks are highly
similar to this category, but it is not the highest. Figure 14 displays the
similarity of the tactics within _Execution_.

_Persistence_ describes how attackers maintain access across events like the
user rebooting, switching user or any other interruption. It is discussed most
often at DEFCON, with Black Hat talks the second most similar. The other
conferences display relatively low similarity. This suggests that academic
researchers are most concerned by the initial vulnerability, and not how the
adversary maintains access to the system.

_Privilege Escalation_ techniques allow an attacker to gain more permissions
to access files and other system resources. This technique shows a similar
pattern to _Persistence_ , in which academic talks display low similarity
relative to industry conferences. USENIX Security talks have the most
similarity out of the academic conferences.

##### Expansion

The next three steps concern how the attacker moves through a victim’s
network. _Defense Evasion_ concerns how the adversary avoids detection. The
talks display a high similarity to this and also _Discovery_ , which concerns
how the adversary collects information about the victim’s network. Unlike the
previous tactics, the academic talks display higher similarity to both than
the industry talks.

_Credential Access_ involves stealing account names and passwords, such as via
keylogging. The conference talks are only moderately similar to this, with
academic talks showing more similarity than industry. The conference talks
display low similarity to the third tactic in this category, namely _Lateral
Movement_. It covers how the adversary moves through the victim’s network.

Figure 14: Cosine similarity score for the techniques within _Execution_.

##### Outcome

These tactics concern what was achieved by the compromise, although these
techniques could also be used to expand access. _Collection_ concerns
identifying and gathering data on the victim network. It is the tactic in this
category with the highest similarity to the conference talks. Meanwhile,
_Exfiltration_ concerns how the data is stolen from the network, which has
reasonably low similarity to the conference talks.

The _Command and Control_ tactic involves establishing communication channels
to maintain control over the infected systems. This has moderate similarities
across both industry and academia. The NDSS conference displays the highest
similarity, again because this is fundamentally a problem of distributed
computing, albeit an illegal one. Finally, the _Impact_ technique concerns an
effort to “manipulate, interrupt, or destroy your systems and data”, which
would involve ransomware for example. The academic talks display higher
similarity to this category, although it is moderate-high for all conferences
apart from RSA.

##  5 Limitations

The four main limitations of our study are: missing data (e.g. sponsorship for
some conferences); processing speaker and company names; classifying abstracts
against frameworks; and incomplete coverage of security conferences.

Missing Data Our comparative longitudinal study was impacted by changes in
reporting and missing data. We could not collect some years of talks due to
data unavailability. For example, the RSA conference was created in 1991 but
we could only extract the program from 2017 onward. Similarly, there was a lot
of missing sponsorship information, which led to a long list of exceptions to
the sponsors column in Table 1.

Real Names Counting authors can be complex because two unique names may
correspond to the same individual (e.g. Emma Jones and Emma Q. Jones), and one
name may correspond to multiple individuals (e.g. Emma Jones from New York and
Emma Jones from California). Additionally, some authors may use pseudonyms
rather than their real names. For simplicity, we counted unique names as
unique people. We considered using affiliations to distinguish between two
unique individuals with the same name, but researchers switch institutions too
often for this to be reliable.

Similar issues are present for company names. Company names are inconsistently
abbreviated (Hewlett Packard vs HP), subsidiaries and even departments are
listed (e.g. both Microsoft Corporation and Microsoft Research were listed as
sponsors), and some companies change names (e.g. Facebook to Meta). We fixed
such issues manually where we could, but we likely missed instances for less
well known companies.

Classification Mapping abstracts to categories from security frameworks is
challenging because: (i) the task is inherently subjective; (ii) the
categories are high-level with fuzzy boundaries; (iii) abstracts typically
focus on specific technological systems; and (iv) even state-of-the-art LLMs
are known to have errors. These factors help to explain why the agreement
between the two NLP techniques (see Section 3.3) was disappointing.
Ultimately, we focused on the LLM-based mapping because it is the state-of-
the-art technique and it captures how language is used in real discussions in
the LLM’s training set. This can be contrasted against the cosine similarity
approach, which is trained on a bag-of-words extracted from official
documents. However, it is not unreasonable to prefer a different mapping, such
as using another NLP technique or another cyber security framework. To enable
this, we open-source our data and analysis scripts.

Conference Coverage Our comparative analysis (RQ3) would have been more
interesting if we collected data from more conferences and also went further
back in time. For example, the FIRST Conference on Computer Security Incident
Handling would likely have covered more topics related to _Respond_ , but from
a more technical perspective than the insurance conferences. The core
challenge to adding more data is that we could only semi-automated our
scrapers in order to ensure data quality, and had to create a new parser for
each conference. This made it costly to add additional conferences and also to
scrape further back in time. Again, we hope that open-sourcing our data and
analysis will allow future work to extend our work by adding new conferences.

##  6 Discussion

We discuss the results and how our framing assumptions shaped the analysis.

Speakers Academia displays more information sharing across conferences than
industry, given many industry speakers present at just one conference and
speakers rarely cross the insurance–InfoSec line (see Figure 6). It is unclear
whether industry conferences would benefit from more cross-pollination. The
call for broader information sharing is appealing, however it could be that
more efficient communication is enabled by the shared language and assumptions
within each community.

A different question is whether sponsorship buys influence drezner2017ideas .
Conferences with more sponsors have fewer words per abstract, which could be
interpreted as sponsored conferences involving less complex talks. Even though
Black Hat had the most sponsors, this appears to have little influence over
who speaks. For example, the majority of sponsors (63%) never had an
affiliated talk, and the majority (85%) of the speakers’ affiliated
organizations never sponsored Black Hat. The main concern for academic
conferences is that despite being privacy conferences, two of the five most
common sponsors are companies that directly profit from behavioral
advertising. The relationship between BigTech companies and researchers has
been called into question by other researchers abdalla2021grey .

A similar question is whether it is a problem that a small number of
speakers/authors hold the majority of influence. For illustrative purposes, we
can compare the Gini index for conferences (see Table 1) to the index of
income inequality in specific countries. The distribution of talks across
speakers is typically around 0.37–0.44, which is close to the post-tax
distribution of income in relatively unequal societies like the US (0.375) and
Mexico (0.42). All prestigious industry and academic talks have higher
inequality than Germany (0.296).555All figures sourced from the OECD using
post-tax income: https://stats.oecd.org Inequality in academic talks is not
necessarily a cause for concern. Scientific conferences should provide a
platform for the ‘best’ science and that likely means the ‘best’ scientists
will present more often. However, an unequal distribution can also result from
a narrow perspective of what constitutes ‘good’ research. It is worth noting
that ACM CCS was the most unequal academic conference across both approaches
to quantifying the Gini Index.

Topics One way to improve inequality metrics is to expand the topics that are
presented at top tier academic security conferences, allowing new researchers
to present ideas and findings. For example, there is comparatively little
research about the _Identify_ function that concerns security management. A
first step would be to understand how security is managed at present
hielscher2023employees , which can motivate technical solutions to management
problems. For example, _Asset Management_ can be inferred from network traffic
dekoven2019measuring ; _Risk Assessment_ can be framed as ML prediction
liu2015cloudy ; and _Governance_ can involve technical audits
rahaman2019security ; mahmud2020cardpliance .

Turning to offensive talks, the academic and industry talks are much more
evenly distributed across the techniques in MITRE ATT&CK. The one exception is
information gathering that occurs pre-compromise, which is rarely discussed at
conferences. Preventing resource development is perhaps better conducted by
governments, such as by disrupting criminal markets for exploits
allodi2015then . The lack of offensive talks at insurance conferences raises
the question of whether there is a sufficient understanding of attacker
operations. One potential reason this does not happen is insurers rely on
lawyers to manage incidents, and those lawyers do not report findings due to
confidentiality concerns woods2023lessons .

Variance The insurance conferences were the biggest outliers because they
focus on management and resilience more than technological controls and
attacker techniques. This confirms the finding that insurers are more focused
on post-incident response than pre-incident mitigation wolff2022cyberinsurance
. For the MITRE ATT&CK framework mapping, the only offensive conference
(DEFCON) is comparatively focused on how compromise was achieved (execution,
persistence and initial access), while the academic conferences focus on
activities that take place after compromise.

Turning to changes over time, there is an increasing focus on the _Identify_
function and perhaps a weakening focus on _Protective Technology_ and _Detect_
functions. This contradicts the prediction of Schneier that the 2010s would be
the decade of response schneier2014future . Instead, there appears to be a
subtle shift to security management. However, we should not overstate these
effects as the ‘declining’ categories remain the most frequently discussed in
absolute terms.

Assumptions In mapping abstracts to cybersecurity frameworks, we imply that
there should be equal coverage across topics of the framework. Our core
assumption is that more speakers/authors and different topics is better. We
justify this by pointing to the benefits researchers gain from learning about
areas outside of their specialism. On the other hand, focusing on ever more
specific areas of security facilitates deeper understanding and more progress.
This is an example of the explore vs exploit trade-off with an unknown
objective function, namely maximizing global security knowledge. We
acknowledge that reasonable readers may disagree with our emphasis on explore.

More fundamentally, it is unclear whether encouraging specific lines of work
is appropriate, especially given the sources of those frameworks. This has the
potential to impinge on academic freedom. Although we explained why ATT&CK and
NIST CSF were the best among the alternatives (see Section 3), it is entirely
unclear whether they are ‘good enough’ in terms of granularity, completeness
and so on. This may be actively contested by some.

Future Work We focused on speakers/authors but did not consider
listeners/readers. Doing so is crucial to understanding the impact of
conference presentations. Collecting this information risks introducing some
form of surveillance that undermines the privacy of attendees. One privacy-
preserving approach is to scrape the number of views that uploaded video
recordings received and to study the comments. Alternatively, an (imperfect)
academic proxy for impact on readers is citation count, but the industry
equivalent requires more sophisticated research designs.

To address the limitations described in Section 5, future work should add
extra conferences and map abstract to different security frameworks and
compare the results. Further, a qualitative study collecting data from
conference organizers could explore hidden dynamics, such as negotiations with
industry sponsors. This would provide insights into unseen details like what
sponsors demand from academic conferences. Gaining access to industry
conference organizers is considerably more difficult.

##  7 Conclusion

Security conferences are an important but understudied channel by which
knowledge and information is shared. Our paper built a longitudinal data set
of talks at 10 prestigious conferences across industry and academia. We
characterized who speaks at these conferences (RQ1), about which topics (RQ2),
and how this compares across conferences and over time (RQ3). To answer the
latter two research questions, we mapped conference abstracts to cybersecurity
frameworks using two NLP techniques, which show moderate agreement.

The results show academics moved freely between the big four venues, but just
5% of academic authors also presented at InfoSec or Insurance conferences. The
equivalent figure was 4% for insurance speakers, and 7% for InfoSec speakers,
which means there is scope for presenters to move between disciplines. We also
showed there is an unequal distribution of talks across speakers, with this
inequality varying across conferences. In terms of sponsors, academic
conference are dominated by BigTech firms, whereas InfoSec conference sponsors
are predominantly cybersecurity vendors.

All conferences contained more offensive than defensive talks, with the
exception of DEFCON. Offensive abstracts are distributed evenly across the
categories of MITRE ATT&CK, apart from _Reconnaissance_ and _Resource
Development_. Defensive abstracts focused on _Protective Technology_ and
categories related to monitoring. Academic conferences tend to overlook the
_Identify_ and _Recover_ functions from the NIST Cybersecurity Framework.

These findings support a more nuanced understanding of security information
sharing, in which success should be evaluated with regards to specific
channels and institutions. Researchers have been disappointed by the quality
of information shared via online sources acar2016you ; fischer2017stack  and
threat intelligence feeds li2019reading ; bouwman2020different . However,
there is more success when it comes to vulnerability management. Bug bounties
incentivize the discovery of new information finifter2013empirical ;
sridhar2021hacking . Further, information about fixes is widely available, the
problem lies in acting on the information by installing patches frei2006large
; li2017large . Our results suggest security conferences should be considered
a success story, closer to patch management than threat intelligence sharing.
This is perhaps not surprising given attendees, sponsors and research founders
invest considerable resources in attending and sponsoring these conferences.

Going forward, organizers of prestigious conferences should ensure there is a
range of speakers and topics. DEFCON is perhaps a good example in that it
refuses sponsorship, evenly covered all but two areas of the MITRE ATT&CK
framework, and was one of the most equal conference in terms of the
distribution of talks across speakers. An interesting follow-on study would
speak to conference organizers to enrich our quantitative analysis with
qualitative insights about the unobservable aspects of conference
organization.

## References

  * [1] Florian Skopik, Giuseppe Settanni, and Roman Fiedler.  A problem shared is a problem halved: A survey on the dimensions of collective cyber defense through security information sharing.  Computers & Security, 60:154–176, 2016.
  * [2] Stefan Laube and Rainer Böhme.  Strategic aspects of cyber risk information sharing.  ACM Computing Surveys, 50(5):1–36, 2017.
  * [3] Scott Ainslie, Dean Thompson, Sean Maynard, and Atif Ahmad.  Cyber-threat intelligence for security decision-making: A review and research agenda for practice.  Computers & Security, page 103352, 2023.
  * [4] Steffen Wendzel, Cédric Lévy-Bencheton, and Luca Caviglione.  Not all areas are equal: analysis of citations in information security research.  Scientometrics, 122(1):267–286, 2020.
  * [5] Davide Balzarotti.  System security circus 2022.  https://www.s3.eurecom.fr/~balzarot/security-circus/circus_stats.html, 2022\.  Accessed: 2023-10-10.
  * [6] Jeffrey C Carver, Morgan Burcham, Sedef Akinli Kocak, Ayse Bener, Michael Felderer, Matthias Gander, Jason King, Jouni Markkula, Markku Oivo, Clemens Sauerwein, et al.  Establishing a baseline for measuring advancement in the science of security: an analysis of the 2015 IEEE security & privacy proceedings.  In Proceedings of the Symposium and Bootcamp on the Science of Security, pages 38–51, 2016.
  * [7] Morgan Burcham, Mahran Al-Zyoud, Jeffrey C Carver, Mohammed Alsaleh, Hongying Du, Fida Gilani, Jun Jiang, Akond Rahman, Özgür Kafalı, Ehab Al-Shaer, et al.  Characterizing scientific reporting in security literature: An analysis of ACM CCS and IEEE S&P papers.  In Proceedings of the Hot Topics in Science of Security: Symposium and Bootcamp, pages 13–23, 2017.
  * [8] Daniel W Drezner.  The ideas industry.  Oxford University Press, 2017.
  * [9] Panos Kampanakis.  Security automation and threat information-sharing options.  IEEE Security & Privacy, 12(5):42–51, 2014.
  * [10] Agnes Yang, Young Jin Kwon, and Sang-Yong Tom Lee.  The impact of information sharing legislation on cybersecurity industry.  Industrial Management & Data Systems, 120(9):1777–1794, 2020.
  * [11] Rebecca Slayton and Brian Clarke.  Trusting infrastructure: The emergence of computer security incident response, 1989–2005.  Technology and Culture, 61(1):173–206, 2020.
  * [12] Thomas D Wagner, Khaled Mahbub, Esther Palomar, and Ali E Abdallah.  Cyber threat intelligence sharing: Survey and research directions.  Computers & Security, 87:101589, 2019.
  * [13] Clemens Sauerwein, Christian Sillaber, Andrea Mussmann, and Ruth Breu.  Threat intelligence sharing platforms: An exploratory study of software vendors and research perspectives.  In Wirtschaftsinformatik 2017, 2017.
  * [14] Sarah Brown, Joep Gommers, and Oscar Serrano.  From cyber security information sharing to threat management.  In Proceedings of the 2nd ACM workshop on information sharing and collaborative security, pages 43–49, 2015.
  * [15] Clemens Sauerwein, Irdin Pekaric, Michael Felderer, and Ruth Breu.  An analysis and classification of public information security data sources used in research and practice.  Computers & security, 82:140–155, 2019.
  * [16] Clemens Sauerwein, Daniel Fischer, Milena Rubsamen, Guido Rosenberger, Dirk Stelzer, and Ruth Breu.  From threat data to actionable intelligence: an exploratory analysis of the intelligence cycle implementation in cyber threat intelligence sharing platforms.  In Proceedings of the 16th International Conference on Availability, Reliability and Security, pages 1–9, 2021.
  * [17] Luc Dandurand and Oscar Serrano Serrano.  Towards improved cyber security information sharing.  In 2013 5th International Conference on Cyber Conflict (CYCON 2013), pages 1–16. IEEE, 2013.
  * [18] David Chismon and Martyn Ruks.  Threat intelligence: Collecting, analysing, evaluating.  MWR InfoSecurity Ltd, 3(2):36–42, 2015.
  * [19] Vector Guo Li, Matthew Dunn, Paul Pearce, Damon McCoy, Geoffrey M Voelker, and Stefan Savage.  Reading the tea leaves: A comparative analysis of threat intelligence.  In Proc. of the 28th USENIX Security Symposium, pages 851–867, 2019\.
  * [20] Kurt Thomas, Rony Amira, Adi Ben-Yoash, Ori Folger, Amir Hardon, Ari Berger, Elie Bursztein, and Michael Bailey.  The abuse sharing economy: Understanding the limits of threat exchanges.  In Research in Attacks, Intrusions, and Defenses: 19th International Symposium, RAID 2016, Paris, France, September 19-21, 2016, Proceedings 19, pages 143–164. Springer, 2016.
  * [21] Xander Bouwman, Victor Le Pochat, Pawel Foremski, Tom Van Goethem, Carlos H Gañán, Giovane Moura, Samaneh Tajalizadehkhoob, Wouter Joosen, and Michel van Eeten.  Helping hands: Measuring the impact of a large threat intelligence sharing community.  In Proc. of the 31st USENIX Security Symposium, pages 1149–1165, 2022.
  * [22] Xander Bouwman, Harm Griffioen, Jelle Egbers, Christian Doerr, Bram Klievink, and Michel van Eeten.  A different cup of TI? the added value of commercial threat intelligence.  In Proc. of the 30th USENIX Security Symposium, pages 433–450, 2020\.
  * [23] Adam Zibak, Clemens Sauerwein, and Andrew C Simpson.  Threat intelligence quality dimensions for research and practice.  Digital Threats: Research and Practice, 3(4):1–22, 2022.
  * [24] Mingyi Zhao, Aron Laszka, and Jens Grossklags.  Devising effective policies for bug-bounty platforms and security vulnerability discovery.  Journal of Information Policy, 7:372–418, 2017.
  * [25] Kiran Sridhar and Ming Ng.  Hacking for good: Leveraging HackerOne data to develop an economic model of bug bounties.  Journal of Cybersecurity, 7(1):tyab007, 2021.
  * [26] Matthew Finifter, Devdatta Akhawe, and David Wagner.  An empirical study of vulnerability rewards programs.  In Presented as part of the 22nd $\\{$USENIX$\\}$ Security Symp., pages 273–288. USENIX, 2013.
  * [27] Mingyi Zhao, Jens Grossklags, and Peng Liu.  An empirical study of web vulnerability discovery ecosystems.  In Proc. of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pages 1105–1117. ACM, 2015.
  * [28] Daniel Votipka, Rock Stevens, Elissa Redmiles, Jeremy Hu, and Michelle Mazurek.  Hackers vs. testers: A comparison of software vulnerability discovery processes.  In 2018 IEEE Symposium on Security and Privacy (SP), pages 374–391. IEEE, 2018.
  * [29] Stefan Frei, Martin May, Ulrich Fiedler, and Bernhard Plattner.  Large-scale vulnerability analysis.  In Proceedings of the SIGCOMM workshop on Large-scale attack defense, pages 131–138, 2006.
  * [30] Frank Li and Vern Paxson.  A large-scale empirical study of security patches.  In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pages 2201–2215, 2017.
  * [31] Frank Li, Lisa Rogers, Arunesh Mathur, Nathan Malkin, and Marshini Chetty.  Keepers of the machines: Examining how system administrators manage software updates for multiple machines.  In Fifteenth Symposium on Usable Privacy and Security (SOUPS 2019), pages 273–288, 2019.
  * [32] Arunesh Mathur, Josefine Engel, Sonam Sobti, Victoria Chang, and Marshini Chetty.  ”they keep coming back like zombies”: Improving software updating interfaces.  In Twelfth Symposium on Usable Privacy and Security (SOUPS 2016), pages 43–58, 2016.
  * [33] Ben Stock, Giancarlo Pellegrino, Christian Rossow, Martin Johns, and Michael Backes.  Hey, you have a problem: On the feasibility of large-scale web vulnerability notification.  In Proc. of the USENIX Security Symp., pages 1015–1032. USENIX, 2016.
  * [34] Frank Li, Zakir Durumeric, Jakub Czyz, Mohammad Karami, Michael Bailey, Damon McCoy, Stefan Savage, and Vern Paxson.  You’ve got vulnerability: Exploring effective vulnerability notifications.  In Proc. of the USENIX Security Symp., pages 1033–1050. USENIX, 2016.
  * [35] Orçun Çetin, Carlos Gañán, Lisette Altena, Samaneh Tajalizadehkhoob, and Michel Van Eeten.  Tell me you fixed it: Evaluating vulnerability notifications via quarantine networks.  In 2019 IEEE European Symposium on Security and Privacy (EuroS&P), pages 326–339. IEEE, 2019.
  * [36] Jay Jacobs, Sasha Romanosky, Idris Adjerid, and Wade Baker.  Improving vulnerability remediation through better exploit prediction.  Journal of Cybersecurity, 6(1):tyaa015, 2020.
  * [37] Stephanie de Smale, Rik van Dijk, Xander Bouwman, Jeroen van der Ham, and Michel van Eeten.  No one drinks from the firehose: How organizations filter and prioritize vulnerability information.  In 2023 IEEE Symposium on Security and Privacy (SP), 2023.
  * [38] Jingjie Li, Kaiwen Sun, Brittany Skye Huff, Anna Marie Bierley, Younghyun Kim, Florian Schaub, and Kassem Fawaz.  “it’s up to the consumer to be smart”: Understanding the security and privacy attitudes of smart home users on reddit.  In IEEE Symposium on Security and Privacy (SP)(SP), pages 380–396. IEEE Computer Society Los Alamitos, CA, 2023.
  * [39] Mohammad Tahaei, Kami Vaniea, and Naomi Saphra.  Understanding privacy-related questions on stack overflow.  In Proceedings of the 2020 CHI conference on human factors in computing systems, pages 1–14, 2020.
  * [40] Yasemin Acar, Michael Backes, Sascha Fahl, Doowon Kim, Michelle L Mazurek, and Christian Stransky.  You get where you’re looking for: The impact of information sources on code security.  In 2016 IEEE Symposium on Security and Privacy (SP), pages 289–305. IEEE, 2016.
  * [41] Felix Fischer, Konstantin Böttinger, Huang Xiao, Christian Stransky, Yasemin Acar, Michael Backes, and Sascha Fahl.  Stack overflow considered harmful? the impact of copy&paste on android application security.  In 2017 IEEE Symposium on Security and Privacy (SP), pages 121–136. IEEE, 2017.
  * [42] Felix Fischer, Huang Xiao, Ching-Yu Kao, Yannick Stachelscheid, Benjamin Johnson, Danial Razar, Paul Fawkesley, Nat Buckley, Konstantin Böttinger, Paul Muntean, and Jens Grossklags.  Stack overflow considered helpful! deep learning security nudges towards stronger cryptography.  In 28th USENIX Security Symposium (USENIX Security 19), pages 339–356, 2019.
  * [43] Elissa M Redmiles, Amelia R Malone, and Michelle L Mazurek.  I think they’re trying to tell me something: Advice sources and selection for digital security.  In 2016 IEEE Symposium on Security and Privacy (SP), pages 272–288. IEEE, 2016.
  * [44] Elissa M Redmiles, Noel Warford, Amritha Jayanti, Aravind Koneru, Sean Kross, Miraida Morales, Rock Stevens, and Michelle L Mazurek.  A comprehensive quality evaluation of security and privacy advice on the web.  In 29th USENIX Security Symposium (USENIX Security 20), pages 89–108, 2020.
  * [45] Ananta Soneji, Faris Bugra Kokulu, Carlos Rubio-Medrano, Tiffany Bao, Ruoyu Wang, Yan Shoshitaishvili, and Adam Doupé.  “flawed, but like democracy we don’t have a better system”: The experts’ insights on the peer review process of evaluating security papers.  In 2022 IEEE Symposium on Security and Privacy (SP), pages 1845–1862. IEEE, 2022.
  * [46] Savino Dambra, Leyla Bilge, and Davide Balzarotti.  SoK: Cyber insurance—Technical challenges and a system security roadmap.  In Proc. of the Symp. on Security and Privacy, pages 293–309. IEEE, 2020.
  * [47] National Association of Insurance Commissioners Staff.  Report on the cybersecurity insurance market.  https://content.naic.org/sites/default/files/cmte-c-cyber-supplement-report-2022-for-data-year-2021.pdf, 2021\.  Accessed: 2023-10-11.
  * [48] Daniel W Woods, Rainer Böhme, Josephine Wolff, and Daniel Schwarcz.  Lessons lost: Incident response in the age of cyber insurance and breach attorneys.  In Proceedings of the 32nd USENIX Security Symposium, 2023.
  * [49] Mathew Vermeer, Jonathan West, Alejandro Cuevas, Shuonan Niu, Nicolas Christin, Michel Van Eeten, Tobias Fiebig, Carlos Ganán, and Tyler Moore.  Sok: A framework for asset discovery: Systematizing advances in network measurements for protecting organizations.  In 2021 IEEE European Symposium on Security and Privacy (EuroS&P), pages 440–456. IEEE, 2021.
  * [50] Yang Liu, Armin Sarabi, Jing Zhang, Parinaz Naghizadeh, Manish Karir, Michael Bailey, and Mingyan Liu.  Cloudy with a chance of breach: Forecasting cyber security incidents.  In Proc. of the USENIX Sec. Symp., pages 1009–1024. USENIX, 2015\.
  * [51] Leyla Bilge, Yufei Han, and Matteo Dell’Amico.  Riskteller: Predicting the risk of cyber incidents.  In Proc. of the Conference on Computer and Communications Security, pages 1299–1311. ACM, 2017.
  * [52] Robert Morris and Ken Thompson.  Password security: A case history.  Communications of the ACM, 22(11):594–597, 1979.
  * [53] Joseph Bonneau, Cormac Herley, Paul C Van Oorschot, and Frank Stajano.  The quest to replace passwords: A framework for comparative evaluation of web authentication schemes.  In 2012 IEEE symposium on security and privacy, pages 553–567. IEEE, 2012.
  * [54] Simson Garfinkel and Heather Richter Lipford.  Usable security: History, themes, and challenges.  Synthesis Lectures on Information Security, Privacy, and Trust, 5(2):1–124, 2014.
  * [55] Benjamin Collier, Richard Clayton, Alice Hutchings, and Daniel Thomas.  Cybercrime is (often) boring: maintaining the infrastructure of cybercrime economies.  In Workshop on the Economics of Information Security, 2020.
  * [56] Rob Knake, Adam Shostack, and Tarah Wheeler.  Learning from cyber incidents: Adapting aviation safety models to cybersecurity.  Harvard Belfer Center, 2021.
  * [57] Simson L Garfinkel.  Digital forensics research: The next 10 years.  Digital Investigation, 7:S64–S73, 2010.
  * [58] Mohamed Abdalla and Moustafa Abdalla.  The grey hoodie project: Big tobacco, big tech, and the threat on academic integrity.  In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 287–297, 2021.
  * [59] Jonas Hielscher, Uta Menges, Simon Parkin, Annette Kluge, and M Angela Sasse.  “employees who don’t accept the time security takes are not aware enough”: The ciso view of human-centred security.  In 32st USENIX Security Symposium (USENIX Security 23), Boston, MA, 2023.
  * [60] Louis F DeKoven, Audrey Randall, Ariana Mirian, Gautam Akiwate, Ansel Blume, Lawrence K Saul, Aaron Schulman, Geoffrey M Voelker, and Stefan Savage.  Measuring security practices and how they impact security.  In Proc. of the Internet Measurement Conference, pages 36–49. ACM, 2019.
  * [61] Sazzadur Rahaman, Gang Wang, and Danfeng Yao.  Security certification in payment card industry: Testbeds, measurements, and recommendations.  In Proc. of the Conf. on Computer and Communications Security, pages 481–498. ACM, 2019.
  * [62] Samin Yaseer Mahmud, Akhil Acharya, Benjamin Andow, William Enck, and Bradley Reaves.  Cardpliance: PCI DSS compliance of android applications.  In 29th USENIX Security Symposium, pages 1517–1533, 2020.
  * [63] Luca Allodi, Marco Corradin, and Fabio Massacci.  Then and now: On the maturity of the cybercrime markets the lesson that black-hat marketeers learned.  IEEE Trans. on Emerging Topics in Computing, 4(1):35–46, 2015.
  * [64] Josephine Wolff.  Cyberinsurance Policy: Rethinking Risk in an Age of Ransomware, Computer Fraud, Data Breaches, and Cyberattacks.  MIT Press, 2022.
  * [65] Bruce Schneier.  The future of incident response.  IEEE Security & Privacy, 12(5):96–96, 2014.

Figure 15: Mean similarity score (across gpt and cosine similarity) of
defensive talks mapped to NIST Cybersecurity Framework. Figure 16: Mean
similarity score (across gpt and cosine similarity) of offensive talks mapped
to MITRE ATT&CK. Figure 17: Similarity score of Black Hat defensive talks
mapped to NIST Cybersecurity Framework over years Figure 18: Similarity score
of IEEE S&P defensive talks mapped to NIST Cybersecurity Framework over years

Figure 19: Top sponsors of InfoSec and cyber insurance conferences. The number
on top of each bar is the mean sponsorship tier of that sponsor. Note, there
was a lot of missing data when it came to the sponsors of InfoSec conferences.
Figure 20: Similarity score of USENIX Security defensive talks mapped to NIST
Cybersecurity Framework over years Figure 21: Similarity score of Black Hat
offensive talks mapped to MITRE over years Figure 22: Similarity score of
DEFCON offensive talks mapped to MITRE over years Figure 23: Similarity score
of IEEE S&P offensive talks mapped to MITRE over years Figure 24: Similarity
score of USENIX Security offensive talks mapped to MITRE over years