# Women, politics and Twitter:
Using machine learning to change the discourse

Lana Cuthberston
ParityYEG
<EMAIL_ADDRESS>
Alex Kearney
University of Alberta
<EMAIL_ADDRESS>
Riley Dawson
University of Alberta
<EMAIL_ADDRESS>
Ashia Zawaduk
Dev Edmonton Society
<EMAIL_ADDRESS>
Eve Cuthbertson
University of Toronto
<EMAIL_ADDRESS>
Ann Gordon-Tighe
University of Alberta
<EMAIL_ADDRESS>
Kory W Mathewson
University of Alberta
<EMAIL_ADDRESS>
Corresponding author

###### Abstract

Including diverse voices in political decision-making strengthens our
democratic institutions. Within the Canadian political system, there is gender
inequality across all levels of elected government. Online abuse, such as
hateful tweets, leveled at women engaged in politics contributes to this
inequity, particularly tweets focusing on their gender. In this paper, we
present ParityBOT: a Twitter bot which counters abusive tweets aimed at women
in politics by sending supportive tweets about influential female leaders and
facts about women in public life. ParityBOT is the first artificial
intelligence-based intervention aimed at affecting online discourse for women
in politics for the better. The goal of this project is to: $1$) raise
awareness of issues relating to gender inequity in politics, and $2$)
positively influence public discourse in politics. The main contribution of
this paper is a scalable model to classify and respond to hateful tweets with
quantitative and qualitative assessments. The ParityBOT abusive classification
system was validated on public online harassment datasets. We conclude with
analysis of the impact of ParityBOT, drawing from data gathered during
interventions in both the $2019$ Alberta provincial and $2019$ Canadian
federal elections.

##  1 Introduction

Our political systems are unequal, and we suffer for it. Diversity in
representation around decision-making tables is important for the health of
our democratic institutions [18]. One example of this inequity of
representation is the gender disparity in politics: there are fewer women in
politics than men, largely because women do not run for office at the same
rate as men. This is because women face systemic barriers in political systems
across the world [24]. One of these barriers is online harassment [21, 19].
Twitter is an important social media platform for politicians to share their
visions and engage with their constituents. Women are disproportionately
harassed on this platform because of their gender [12].

To raise awareness of online abuse and shift the discourse surrounding women
in politics, we designed, built, and deployed ParityBOT: a Twitter bot that
classifies hateful tweets directed at women in politics and then posts
“positivitweets”. This paper focuses on how ParityBOT improves discourse in
politics.

Previous work that addressed online harassment focused on collecting tweets
directed at women engaged in politics and journalism and determining if they
were problematic or abusive [3, 19, 7]. Inspired by these projects, we go one
step further and develop a tool that directly engages in the discourse on
Twitter in political communities. Our hypothesis is that by seeing
“positivitweets” from ParityBOT in their Twitter feeds, knowing that each
tweet is an anonymous response to a hateful tweet, women in politics will feel
encouraged and included in digital political communities[10]. This will reduce
the barrier to fair engagement on Twitter for women in politics. It will also
help achieve gender balance in Canadian politics and improve gender equality
in our society.

##  2 Methods

###  2.1 Technical Details for ParityBot

In this section, we outline the technical details of ParityBot. The system
consists of: $1$) a Twitter listener that collects and classifies tweets
directed at a known list of women candidates, and $2$) a responder that sends
out positivitweets when hateful tweets are detected.

We collect tweets from Twitter’s real-time streaming API. The stream listener
uses the open-source Python library Tweepy [22]. The listener analyses tweets
in real-time by firing an asynchronous tweet analysis and storage function for
each English tweet mentioning one or more candidate usernames of interest. We
limit the streaming to English as our text analysis models are trained on
English language corpora. We do not track or store retweets to avoid biasing
the analysis by counting the same content multiple times. Twitter data is
collected and used in accordance with the acceptable terms of use [23].

The tweet analysis and storage function acts as follows: $1$) parsing the
tweet information to clean and extract the tweet text, $2$) scoring the tweet
using multiple text analysis models, and $3$) storing the data in a database
table. We clean tweet text with a variety of rules to ensure that the tweets
are cleaned consistent with the expectations of the analysis models (see Appdx
1.1).

The text analysis models classify a tweet by using, as features, the outputs
from Perspective API from Jigsaw [17], HateSonar [8], and VADER sentiment
models [11]. Perspective API uses machine learning models to score the
perceived impact a tweet might have [17]. The outputs from these models (i.e.
$17$ from Perspective, $3$ from HateSonar, and $4$ from VADER) are combined
into a single feature vector for each tweet (see Appdx 1.2). No user features
are included in the tweet analysis models. While these features may improve
classification accuracy they can also lead to potential bias [25].

We measure the relative correlation of each feature with the hateful or not
hateful labels. We found that Perspective API’s TOXICITY probability was the
most consistently predictive feature for classifying hateful tweets. Fig. 1
shows the relative frequencies of hateful and non-hateful tweets over TOXICITY
scores. During both elections, we opted to use a single Perspective API
feature to trigger sending positivitweets. Using the single TOXICITY feature
is almost as predictive as using all features and a more complex model 1.3. It
was also simpler to implement and process tweets at scale. The TOXICITY
feature is the only output from the Perspective API with transparent
evaluation details summarized in a Model Card [15, 20].

###  2.2 Collecting Twitter handles, predicting candidate gender, curating
“positivitweets”

Deploying ParityBOT during the Alberta $2019$ election required volunteers to
use online resources to create a database of all the candidates running in the
Alberta provincial election. Volunteers recorded each candidate’s self-
identifying gender and Twitter handle in this database. For the $2019$ federal
Canadian election, we scraped a Wikipedia page that lists candidates [26]. We
used the Python library gender-guesser [5] to predict the gender of each
candidate based on their first names. As much as possible, we manually
validated these predictions with corroborating evidence found in candidates’
biographies on their party’s websites and in their online presence.

ParityBOT sent positivitweets composed by volunteers. These tweets expressed
encouragement, stated facts about women in politics, and aimed to inspire and
uplift the community. Volunteers submitted many of these positivitweets
through an online form.111The full list of positivitweets can be found at
https://paritybot.com Volunteers were not screened and anyone could access the
positivitweet submission form. However, we mitigate the impact of trolls
submitting hateful content, submitter bias, and ill-equipped submitters by
reviewing, copy editing, and fact checking each tweet. Asking for community
contribution in this way served to maximize limited copywriting resources and
engage the community in the project.

###  2.3 Qualitative Assessment

We evaluated the social impact of our system by interviewing individuals
involved in government ($n=5$). We designed a discussion guide based on user
experience research interview standards to speak with politicians in relevant
jurisdictions [14]. Participants had varying levels of prior awareness of the
ParityBOT project. Our participants included $3$ women candidates, each from a
different major political party in the $2019$ Alberta provincial election, and
$2$ men candidates at different levels of government representing Alberta
areas. The full discussion guide for qualitative assessment is included in
Appdx 3.1. All participants provided informed consent to their anonymous
feedback being included in this paper.

##  3 Results and Outcomes

We deployed ParityBOT during two elections: 1) the $2019$ Alberta provincial
election, and 2) the $2019$ Canadian federal election. For each tweet we
collected, we calculated the probability that the tweet was hateful or
abusive. If the probability was higher than our response decision threshold, a
positivitweet was posted. Comprehensive quantitative results are listed in
Appendix 2.

Figure 1: Visualizing the training data distribution. Relative frequency of
hateful versus not hateful tweets for varying levels of the Perspective API
[17] TOXICITY score. Normalized histograms are plotted underneath kernel
density estimation (KDE) plots.

During the Alberta election, we initially set the decision threshold to a
TOXICITY score above $0.5$ to capture the majority of hateful tweets, but we
were sending too many tweets given the number of positivitweets we had in our
library and the Twitter API daily limit [23]. Thus, after the first $24$ hours
that ParityBOT was live, we increased the decision threshold to $0.8$,
representing a significant inflection point for hatefulness in the training
data (Fig. 1). We further increased the decision threshold to $0.9$ for the
Canadian federal election given the increase in the number and rate of tweets
processed. For the Alberta provincial election, the model classified $1468$
tweets of the total $12726$ as hateful, and posted only $973$ positivitweets.
This means that we did not send out a positivitweet for every classified
hateful tweet, and reflects our decision rate-limit of ParityBOT. Similar
results were found for the $2019$ Canadian election.

###  3.1 Values and Limitations

We wrote guidelines and values for this to guide the ongoing development of
the ParityBOT project.222https://paritybot.com These values help us make
decision and maintain focus on the goal of this project.

While there is potential to misclassify tweets, the repercussions of doing so
are limited. With ParityBOT, false negatives, hateful tweets classified as
non-hateful, are not necessarily bad, since the bot is tweeting a positive
message. False positives, non-hateful tweets classified as hateful, may result
in tweeting too frequently, but this is mitigated by our choice of decision
threshold.

In developing ParityBOT, we discussed the risks of using bots on social media
and in politics. First, we included the word “bot” in the project title and
Twitter handle to be clear that the Twitter account was tweeting
automatically. We avoided automating any direct “at (@) mention” of Twitter
users, only identifying individuals’ Twitter handles manually when they had
requested credit for their submitted positivitweet. We also acknowledge that
we are limited in achieving certainty in assigning a gender to each candidate.

###  3.2 User experience research results

In our qualitative research, we discovered that ParityBOT played a role in
changing the discourse. One participant said, “it did send a message in this
election that there were people watching” (P2). We consistently heard that
negative online comments are a fact of public life, even to the point where
it’s a signal of growing influence. “When you’re being effective, a good
advocate, making good points, people are connecting with what you’re saying.
The downside is, it comes with a whole lot more negativity […] I can always
tell when a tweet has been effective because I notice I’m followed by trolls”
(P1).

We heard politicians say that the way they have coped with online abuse is to
ignore it. One participant explained, “I’ve tried to not read it because it’s
not fun to read horrible things about yourself” (P4). Others dismiss the idea
that social media is a useful space for constructive discourse: “Because of
the diminishing trust in social media, I’m stopping going there for more of my
intelligent discourse. I prefer to participate in group chats with people I
know and trust and listen to podcasts” (P3).

##  4 Future Work and Conclusions

We would like to run ParityBOT in more jurisdictions to expand the potential
impact and feedback possibilities. In future iterations, the system might
better match positive tweets to the specific type of negative tweet the bot is
responding to. Qualitative analysis helps to support the interventions we
explore in this paper. To that end, we plan to survey more women candidates to
better understand how a tool like this impacts them. Additionally, we look
forward to talking to more women interested in politics to better understand
whether a tool like this would impact their decision to run for office. We
would like to expand our hateful tweet classification validation study to
include larger, more recent abusive tweet datasets [1, 4]. We are also
exploring plans to extend ParityBOT to invite dialogue: for example, asking
people to actively engage with ParityBOT and analyse reply and comment tweet
text using natural language-based discourse analysis methods.

During the $2019$ Alberta provincial and $2019$ Canadian federal elections,
ParityBOT highlighted that hate speech is prevalent and difficult to combat on
our social media platforms as they currently exist, and it is impacting
democratic health and gender equality in our communities [2]. We strategically
designed ParityBOT to inject hope and positivity into politics, to encourage
more diverse candidates to participate. By using machine learning technology
to address these systemic issues, we can help change the discourse an link
progress in science to progress in humanity.

## References

  * [1] Valerio Basile et al.  “Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter”  In _Proceedings of the 13th International Workshop on Semantic Evaluation_ , 2019, pp. 54–63
  * [2] Danielle Keats Citron  “Cyber civil rights”  In _BUL Rev._ 89 HeinOnline, 2009, pp. 61
  * [3] Laure Delisle et al.  “A large-scale crowdsourced analysis of abuse against women journalists and politicians on Twitter”  In _arXiv preprint arXiv:1902.03093_ , 2019
  * [4] Antigoni Maria Founta et al.  “Large scale crowdsourcing and characterization of twitter abusive behavior”  In _Twelfth International AAAI Conference on Web and Social Media_ , 2018
  * [5] “Gender Guesser” Acc: 2019-09-06, https://pypi.org/project/gender-guesser/
  * [6] Jennifer Golbeck et al.  “A large labeled corpus for online harassment research”  In _Proceedings of the 2017 ACM on Web Science Conference_ , 2017, pp. 229–233  ACM
  * [7] Mark A Greenwood et al.  “Online Abuse of UK MPs from 2015 to 2019”  In _arXiv preprint arXiv:1904.11230_ , 2019
  * [8] “HateSonar” Acc: 2019-09-06, https://github.com/Hironsan/HateSonar
  * [9] Haibo He, Yang Bai, Edwardo A Garcia and Shutao Li  “ADASYN: Adaptive synthetic sampling approach for imbalanced learning”  In _2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence)_ , 2008, pp. 1322–1328  IEEE
  * [10] Frank La Rue Heiner Bielefeldt and Githu Muigai  “OHCHR expert workshops on the prohibition of incitement to national, racial or religious hatred”  In _Expert workshop on Africa_ , 2011
  * [11] Clayton J Hutto and Eric Gilbert  “Vader: A parsimonious rule-based model for sentiment analysis of social media text”  In _8th AAAI on weblogs and social media_ , 2014
  * [12] Amnesty International  “Toxic Twitter: Women’s experiences of violence and abuse on Twitter” http://bit.ly/2jZTb5w, 2018
  * [13] Guolin Ke et al.  “Lightgbm: A highly efficient gradient boosting decision tree”  In _Advances in Neural Information Processing Systems_ , 2017, pp. 3146–3154
  * [14] Steve Krug  “Rocket surgery made easy: The do-it-yourself guide to finding and fixing usability problems”  New Riders, 2009
  * [15] Margaret Mitchell et al.  “Model cards for model reporting”  In _Proceedings of the Conference on Fairness, Accountability, and Transparency_ , 2019, pp. 220–229  ACM
  * [16] Randal S Olson and Jason H Moore  “TPOT: A tree-based pipeline optimization tool for automating machine learning”  In _Automated Machine Learning_ Springer, 2019
  * [17] “Perspective API” Acc: 2019-09-06, https://www.perspectiveapi.com/
  * [18] Anne Phillips  “Democracy and representation: Or, why should it matter who our representatives are?”  In _Feminism and politics_ 224 Oxford University Press Oxford, 1998, pp. 240
  * [19] Ludovic Rheault, Erica Rayment and Andreea Musulan  “Politicians in the line of fire: Incivility and the treatment of women on social media”  In _Research & Politics_ 6.1 SAGE Publications Sage UK: London, England, 2019
  * [20] “Toxicity Model Card” Acc: 2019-11-14, https://github.com/conversationai/perspectiveapi/blob/master/2-api/model-cards/English/toxicity.md
  * [21] Linda Trimble  “Ms. Prime Minister: Gender, Media, and Leadership”  U. Toronto Press, 2018
  * [22] “Tweepy” Acc: 2019-09-06, http://www.tweepy.org/
  * [23] “Twitter Developer Agreement and Policy” Acc: 2019-09-06, https://developer.twitter.com/en/developer-terms/agreement-and-policy.html
  * [24] Inter-Parliamentary Union  “Sexism, harassment and violence against women parliamentarians” https://www.ipu.org/file/2425/download?token=0H5YdXVB, 2018
  * [25] Zeerak Waseem and Dirk Hovy  “Hateful symbols or hateful people? predictive features for hate speech detection on twitter”  In _Proceedings of the NAACL student research workshop_ , 2016, pp. 88–93
  * [26] “Wikipedia: List of candidates by riding for the 43rd Canadian federal election” Acc: 2019-09-06, https://w.wiki/7zT

##  1 Tweet Cleaning and Feature Details

###  1.1 Tweet Cleaning Methods

We use regular expression rules to clean tweets: convert the text to
lowercase, remove URLs, strip newlines, replace whitespace with a single
space, and replace mentions with the text tag ‘MENTION’. While these rules may
bias the classifiers, they allow for consistency and generalization between
training, validation, and testing datasets.

###  1.2 Tweet Featurization Details

Each tweet is processed by three models: Perspective API from Jigsaw [17],
HateSonar [8], and VADER sentiment models [11]. Each of these models outputs a
score between $[0,1]$ which correlates the text of the tweet with the specific
measure of the feature. The outputs from these models (i.e. $17$ from
Perspective, $3$ from HateSonar, and $4$ from VADER) are combined into a
single feature vector for each tweet. Below we list the outputs for each text
featurization model:

  * •

Perspective API: ’IDENTITY_ATTACK’, ’INCOHERENT’, ’TOXICITY_FAST’, ’THREAT’,
’INSULT’, ’LIKELY_TO_REJECT’, ’TOXICITY’, ’PROFANITY’, ’SEXUALLY_EXPLICIT’,
’ATTACK_ON_AUTHOR’, ’SPAM’, ’ATTACK_ON_COMMENTER’, ’OBSCENE’,
’SEVERE_TOXICITY’, ’INFLAMMATORY’

  * •

HateSonar: ’sonar_hate_speech’, ’sonar_offensive_language’, ’sonar_neither’

  * •

VADER: ’vader_neg’, ’vader_neu’, ’vader_pos’, ’vader_compound’

###  1.3 Validation and Ablation Experiments

For validation, we found the most relevant features and set an abusive
prediction threshold by using a dataset of $20194$ cleaned, unique tweets
identified as either hateful and not hateful from previous research [6]. Each
entry in our featurized dataset is composed of $24$ features and a class label
of hateful or not hateful. The dataset is shuffled and randomly split into
training (80%) and testing (20%) sets matching the class balance ($25.4\%$
hateful) of the full dataset. We use Adaptive Synthetic (ADASYN) sampling to
resample and balance class proportions in the dataset [9].

With the balanced training dataset, we found the best performing classifier to
be a gradient boosted decision tree [13] by sweeping over a set of possible
models and hyperparameters using TPOT [16]. For this sweep, we used $10$-fold
cross validation on the training data. We randomly partition this training
data $10$ times, fit a model on a training fraction, and validate on the held-
out set.

We performed an ablation experiment to test the relative impact of the
features derived from the various text classification models.

Figure 2: 10-fold cross validation ablation experiment showing the relative
impacts of including various feature sets (Perspective [17], Vader [11],
HateSonar [8], from left to right) in the feature vectors on performance.
Performance is measured using binary classification area under the receiver
operated characteristic curve (ROC AUC) and averaged over the 10-folds. These
methods are compared with the best performing classifier from the validation
study, a gradient-boosted decision tree (textscLGBM [13], left) and a
stratified RANDOM classifier (right).

##  2 Quantitative analysis of elections

This table includes quantitative results from the deployment of ParityBOT in
the Alberta 2019 provincial and Canadian 2019 federal elections.

|  Alberta 2019 provincial election Apr. 1-15 2019 |  Canada 2019 federal election Sep. 11-Oct. 26 2019
---|---|---
Total positivitweets sent | 973 | 2428
Total impressions | 84,961 | 304,600
Total retweets | 142 | 529
Total likes | 412 | 1500
Total replies | n/a | 30
Total tweets analysed | 12,726 | 228,255
Total tweets scored abusive | 1468 | 9987
Abusive rate | 7.65% | 4.38%
Total candidates tracked | 90 | 314
Decision threshold | 0.8 (80% likely to be abusive) | 0.9 (90% likely to be abusive)

##  3 ParityBOT Research Plan and Discussion Guide

Overview Interviews will be completed in three rounds with three different
target participant segments.

Research Objectives

  * •

Understand if and how the ParityBOT has impacted women in politics

  * •

Obtain feedback from Twitter users who’ve interacted with the bot

  * •

Explore potential opportunities to build on the existing idea and platform

  * •

Gain feedback and initial impressions from people who haven’t interacted with
the Bot, but are potential audience

Target Participants

  * •

Round 1: Women in politics who are familiar with the Bot

  * •

Round 2: Women who’ve interacted with the Bot (maybe those we don’t know)

  * •

Round 3: Some women who may be running in the federal election who haven’t
heard of the ParityBOT, but might benefit from following it

  * •

All participants: Must be involved in politics in Canada and must be engaged
on Twitter - i.e. have an account and follow political accounts and/or issues

Recruiting

  * •

Round 1: [Author] recruit from personal network via text

  * •

Round 2: Find people who’ve interacted with the bot on Twitter who we don’t
know, send them a DM, and ask if we can get their feedback over a 15- to
30-minute phone call

  * •

Round 3: Use contacts in Canadian politics to recruit participants who have no
prior awareness of ParityBOT

Method 15- to 30-minute interviews via telephone

Output Summary of findings in the form of a word document that can be put into
the paper

###  3.1 Discussion Guide

Introduction

[Author]: Hey! Thanks for doing this. This shouldn’t take longer than 20
minutes. [Author] is a UX researcher and is working with us. They’ll take it
from here and explain our process, get your consent and conduct the interview.
I’ll be taking notes. Over to [Author]!

[Author]: Hi, my name is [Author], I’m working with [Author] and [Author] to
get feedback on the ParityBOT; the Twitter Bot they created during the last
provincial election.

With your permission, we’d like to record our conversation. The recording will
only be used to help us capture notes from the session and figure out how to
improve the project, and it won’t be seen by anyone except the people working
on this project. We may use some quotes in an academic paper, You’ll be
anonymous and we won’t identify you personally by name.

If you have any concerns at time, we can stop the interview and the recording.
Do we have your permission to do this? (Wait for verbal “yes”).

Round 1 (Women in Politics familiar with ParityBOT)

Background and Warm Up

  * •

When you were thinking about running for politics what were your major
considerations? For example, barriers, concerns?

  * •

We know that online harassment is an issue for women in politics - have you
experienced this in your career? How do you deal with harassment? What are
your coping strategies?

  * •

What advice would you give to women in politics experiencing online
harassment?

Introduction to PartyBOT Thanks very much, now, more specifically about the
ParityBOT:

  * •

What do you know about the ParityBOT?

  * •

What do you think it’s purpose is?

  * •

Did you encounter it? Tell me about how you first encountered it? Did it
provide any value to you during your campaign? How? Do you think this is a
useful tool? Why or why not? Did it mitigate the barrier of online harassment
during your time as a politician?

  * •

Is there anything you don’t like about the Bot?

Next Steps If you could build on this idea of mitigating online harassment for
women in politics, what ideas or suggestions would you have?

Conclusion Any other thoughts or opinions about the ParityBOT you’d like to
share before we end our call?

Thank you very much for your time! If you have any questions, or further
comments, feel free to text or email [Author].


(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: Shanghai Jiao Tong University 22institutetext: Nanyang
Technological University

# Compact 3D Gaussian Splatting For Dense Visual SLAM

Tianchen Deng 11 Yaohui Chen 11 Leyan Zhang 11 Jianfei Yang 22 Shenghai Yuan
22 Danwei Wang 22 Weidong Chen 11

###### Abstract

Recent work has shown that 3D Gaussian-based SLAM enables high-quality
reconstruction, accurate pose estimation, and real-time rendering of scenes.
However, these approaches are built on a tremendous number of redundant 3D
Gaussian ellipsoids, leading to high memory and storage costs, and slow
training speed. To address the limitation, we propose a compact 3D Gaussian
Splatting SLAM system that reduces the number and the parameter size of
Gaussian ellipsoids. A sliding window-based masking strategy is first proposed
to reduce the redundant ellipsoids. Then we observe that the covariance matrix
(geometry) of most 3D Gaussian ellipsoids are extremely similar, which
motivates a novel geometry codebook to compress 3D Gaussian geometric
attributes, i.e., the parameters. Robust and accurate pose estimation is
achieved by a global bundle adjustment method with reprojection loss.
Extensive experiments demonstrate that our method achieves faster training and
rendering speed while maintaining the state-of-the-art (SOTA) quality of the
scene representation.

Figure 1: Our framework minimizes storage and accelerates rendering while
maintaining the SOTA image reconstruction performance. The proposed framework
eliminates unnecessary 3D Gaussian ellipsoids without affecting performance.
We highlight and enlarge some areas to show the significant reduction of 3D
Gaussian points.

##  1 Introduction

Simultaneous localization and mapping (SLAM) has been a fundamental computer
vision problem with wide applications such as autonomous driving, robotics,
and virtual/augmented reality [7, 28]. Several traditional methods, including
ORBSLAM [24, 25], VINS [27], etc. [6, 37, 38], have been introduced over the
years, representing scenes with sparse point cloud maps. However, due to the
sparse nature of the point cloud, it proves ineffective for navigation or
other purposes. Attention has turned to dense scene reconstruction,
exemplified by DTAM [26], Kintinuous [35], and ElasticFusion [36]. However,
their accuracy remains unsatisfactory due to high memory costs, slow
processing speeds, and other real-time running limitations.

Nowadays, with the proposal of Neural Radiance Fields (NeRF) [22], there are
many following works on different areas [4]. Many works focus on combining
implicit scene representation with SLAM systems. iMAP [32] is the first method
to use a single MLP to represent the scene. NICE-SLAM [45], ESLAM [11], Co-
SLAM [34], and PLGSLAM [5] further improve the scene representation with the
hybrid feature grids, axis-aligned feature planes, joint coordinate-parametric
encoding, and progressive scene representation. To further improve the
accuracy of rendering, recent methods have started to explore 3D Gaussian
Splatting(GS) [13] integration with SLAM, such as SplaTAM [12], GS-SLAM [39],
etc [42, 21]. GS-based SLAM methods leverage a point-based representation
associated with 3D Gaussian attributes and adopt the rasterization pipeline to
render the images, achieving fast rendering speed and promising image quality.
However, the original GS-based scene representation entails a substantial
number of 3D Gaussian ellipsoids to maintain high-fidelity reconstruction,
leading to high memory usage and storage requirements. GS-based SLAM systems
usually need more than 500MB to represent a small room-sized scene. Moreover,
the running speed of GS-based SLAM systems is significantly slower than NeRF-
based methods, which hinders practical deployment, especially on resource-
constrained devices.

To this end, we propose a compact 3D Gaussian scene representation method to
address the critical high memory demand and slow training speed issue in GS-
based SLAM systems. Our method notably enhances storage efficiency while
delivering high-quality reconstruction, fast training speed, and real-time
rendering capabilities. First, we design a novel sliding window-based online
masking method to remove the millions of redundant and unnecessary 3D Gaussian
ellipsoids created during the SLAM system operation. With the proposed masking
method, a compact 3D Gaussian scene representation is learned, achieving
faster rendering speed and efficient memory usage since the computational
complexity is linearly proportional to the number of 3D Gaussian points.

Second, we observe that the majority of Gaussian points exhibit similar
geometry information in scale and rotation attributes. To this end, a
codebook-based method is designed to compress the geometry of each Gaussian
point. It learns to find the similarities and geometry shared across the
scene. We only store the codebook index for each 3D Gaussian ellipsoid,
obtaining compact scene representation.

Third, the camera tracking accuracy of GS-based SLAM is relatively low
compared with other SLAM systems. A global BA method with reprojection loss is
proposed to achieve robust and accurate pose estimation. Our method maintains
a global keyframe database and performs bundle adjustment with all the
historical observations, which can effectively eliminate the cumulative error.
Overall, our contributions are shown as follows:

  * •

We propose a novel GS-based SLAM system with compact Gaussian scene
representation, achieving fast training and rendering speed, accurate pose
estimation, and significantly enhancing storage efficiency.

  * •

A novel sliding window-based online masking method is proposed to remove the
number of redundant Gaussian ellipsoids while achieving high-fidelity
performance during training.

  * •

We observe and analyze the geometry similarities of 3D Gaussian ellipsoids and
propose a codebook-based method to efficiently restore the geometry of each
Gaussian point during the SLAM system operation. A keyframe-based global BA
method with reprojection loss is proposed to improve the relative low
performance of camera tracking.

  * •

We conduct comprehensive experiments on different datasets and achieve nearly
176% increase in rendering speed and over 1.97$\times$ compression on memory
usage.

Figure 2: The pipeline of our GS-based SLAM system. The input of our system is
the current RGB-D frame. We start the SLAM system by initializing the 3D
Gaussian map construct. Then, we update our 3D Gaussian map by adding new
Gaussians and using the learnable mask to reduce the redundant 3D Gaussian
ellipsoids. We incorporate a codebook-based vector quantization method to
compress the scene representation. For camera tracking, we maintain a global
keyframe database for global BA and use reprojection loss for robust pose
estimation.

##  2 Related Work

Dense Visual SLAM and Localization. SLAM [2, 19] and localization [20] has
become an active field for the past two decades. DTAM [26] is the first method
to achieve dense scene reconstruction. Kinectfusion [10] uses projective
iterative-closet-point (ICP) for camera tracking. Some learning-based methods
integrate traditional geometry frameworks with deep learning networks for
accurate camera tracking and mapping, such as DROID-SLAM [33].

NeRF-based SLAM. With the proposal of Neural radiance fields (NeRF) [22], many
researchers explore taking advantage of the implicit method into SLAM systems.
iMAP [32] is the first method to use a single multi-layer perceptron (MLP) to
represent the scene, and NICE-SLAM [45]uses learnable hierarchical feature
grids. Vox-Fusion [40] employs octree architecture for dynamic map
scalability. ESLAM [11] and Co-SLAM [34] further improve the scene
representation with tri-planes and joint coordinate-parametric encoding. [44,
18, 17] use semantic feature embedding to improve scene representation. Point-
SLAM [29] uses neural point clouds for the scene representation. Instead of
representing maps with neural implicit features, our method utilizes the
explicit 3D Gaussian representation, which can significantly improve the
rendering speed using splatting-based rasterization.

GS-based SLAM. Recently, 3D Gaussian Splatting (3DGS) [13] using 3D Gaussians
as primitives for real-time neural rendering. 3DGS utilizes highly optimized
custom CUDA kernels and novel algorithmic approaches, which achieve
significant improvements in rendering speed without sacrificing image quality.
SplaTAM [12], GS-SLAM [39], Gaussian-SLAM [42], Gaussian Splatting SLAM [21]
are the pioneer works that successfully combine the advantages of 3D Gaussian
Splatting with SLAM. These methods achieve fast rendering speed and high-
fidelity reconstruction performance. However, the training speed is relatively
slow, which is crucial for SLAM as it is an online operation system. Memory
and storage usage are also heavy, which makes them difficult to use in real-
world scenarios and with handheld devices.

##  3 Method

The pipeline of our system is shown in Fig. 2. The input of our system is
sequential RGB-D frames $\\{I_{i},D_{i}\\}_{i=1}^{M}$ with known camera
intrinsic $K\in R_{3\times 3}$. Our system simultaneously reconstructs a dense
scene map and estimates camera poses $\\{R_{i}|t_{i}\\}^{M}_{i=1}$. For the
mapping thread, a compact 3D Gaussian scene representation (Sec. 3.1) is
designed to represent the environments with sliding window-based masks (Sec.
3.2) and geometry codebook (Sec. 3.3). For the camera tracking thread, a
global bundle adjustment method (Sec. 3.4) is designed for robust and accurate
pose estimation. The network is incrementally updated with the SLAM system
operation.

###  3.1 3D Gaussian Scene Representation

Inspired by [13], We represent the entire scene as a set of 3D Gaussian
ellipsoids. Each 3D Gaussian is associated with 3D attributes (positions,
opacity, scale, and rotation). Our Gaussian ellipsoids are defined by a full
3D covariance matrix $\boldsymbol{\Sigma}$ defined in world space
(normalized):

$f(\mathbf{x})=oe^{-\frac{1}{2}(x)^{T}\boldsymbol{\Sigma}^{-1}(x)},\quad\boldsymbol{\Sigma}=\boldsymbol{R}\boldsymbol{S}\boldsymbol{S^{T}}\boldsymbol{R^{T}}$
(1)

where $o\in[0,1]$ is the opacity value. $\boldsymbol{S}$ is the scaling
matrix, and $\boldsymbol{R}$ is the rotation matrix.

Then we use the 3D Gaussian ellipsoids to render 2D images with the technique
of splatting [14, 41]. Then we can formulate the covariance matrix
$\boldsymbol{\Sigma}^{\prime}$ in camera coordinates:

$\boldsymbol{\Sigma}^{\prime}=\boldsymbol{J}\boldsymbol{W}\boldsymbol{\Sigma}\boldsymbol{W}^{T}\boldsymbol{J}^{T}$
(2)

where $\boldsymbol{W}$ denotes the view direction, $\boldsymbol{J}$ denotes
the projection transformation matrix. For each pixel $p$, the color and
opacity of all Gaussian ellipsoids are computed and blended using this
formula:

$C(p)=\sum_{i\in
N}\boldsymbol{c}_{\boldsymbol{i}}f_{i}(p)\prod_{j=1}^{i-1}\left(1-f_{j}(p)\right)$
(3)

where $c_{i}$ denotes the color of Gaussian ellipsoids. We also propose a
similar depth rendering formula:

$D(p)=\sum_{i=1}^{n}d_{i}f_{i}(\mathbf{p})\prod_{j=1}^{i-1}\left(1-f_{j}(p)\right)$
(4)

We also render a silhouette image to determine visibility:

$S(p)=\sum_{i=1}^{n}f_{i}(p)\prod_{j=1}^{i-1}\left(1-f_{j}(p)\right)$ (5)

Figure 3: The left figure shows the learnable mask strategy. We perform
frustum selection and sliding widow reset to remove redundant Gaussian
ellipsoids while maintaining the reconstruction accuracy efficiently. The
dashed lines represent the removed 3D Gaussian ellipsoids. The right figure
shows the varying count of Gaussian ellipsoids during the SLAM system
operation. These two curves show the distinction between our system with and
without masks. Our mask strategy achieves  1.97 $\times$ compression on the
number of 3D Gaussians.

###  3.2 Sliding Window-based Mask

The existing GS-based SLAM systems, such as SplaTAM [12] and GS-SLAM [39],
directly use the original 3DGS for scene representation, achieving promising
image quality. However, we observe that the 3DGS creates a number of redundant
3D Gaussian ellipsoids with the SLAM system operation($\times$1.52 Gaussian
ellipsoids show similar performance in Fig. 3), while both of them fail to
discover this. This finally results in poor performance in training speed,
memory, and storage usage, which is crucial for online SLAM systems. Some
methods [16, 23, 9] propose novel Gaussian pruning and self-organizing methods
to compact the 3DGS attributes. However, all of these strategies are not
suitable for GS-based SLAM systems as they have to obtain all the images,
pose, and the corresponding point cloud at the beginning, while SLAM systems
are incrementally optimized.

To this end, we propose a learnable sliding window-based mask strategy to
remove the redundant 3D Gaussian ellipsoids with the SLAM system operation.
Compared to the original densification method, which only considers the
opacity, our method takes into account both the volume $V$ and opacity
$o\in[0,1]$ of Gaussian ellipsoids. The volume calculation is
$V=\frac{4}{3}\pi abc$, where $abc$ are the three dimensions of the scale
$\boldsymbol{S}$. We introduce a learnable mask parameter $m\in R^{N}$ and a
corresponding binary mask $M\in\\{0,1\\}^{N}$, $N$ is the number of Gaussian
ellipsoids.

$\displaystyle
M_{n}=\operatorname{sg}\left(\mathbb{I}\left[Sig\left(m_{n}\right)>\epsilon\right]-Sig\left(m_{n}\right)\right)+Sig\left(m_{n}\right)$
(6)
$\displaystyle\hat{\boldsymbol{S_{n}}}=M_{n}\boldsymbol{S_{n}},\qquad\hat{o_{n}}=M_{n}o_{n}$
(7)

where $n$ is the index of the Gaussian ellipsoids, $\epsilon$ denotes the mask
threshold. Inspired by [1], we employ the stop gradient operator $sg(\cdot)$
to calculate gradients from binary masks. $\mathbb{I}$ and $Sig(\cdot)$ denote
the indicator and sigmoid function. This formulation of mask strategy allows
us to effectively combine the influence of volume and opacity of Gaussian
ellipsoids. We formulate the loss function $L_{m}$ of our mask:

$L_{m}=\frac{1}{N}\sum_{n=1}^{N}Sig(m_{n})$ (8)

Figure 4: The R-VQ process to represent the scale and rotation of Gaussian
ellipsoids. In the first stage, we cluster the scale and rotation vectors and
randomly select codebook initialization with the closest code. In the
subsequent stage, the residual between the original vector and the result from
the first stage is stored in another codebook. This iterative process
continues through to the ultimate stage, at which point, the collectively
chosen indices and codebook from each stage provide a representation of the
original vector.

In order to better fit the online updating SLAM systems, we further improve
the masking strategy by adding frustum culling and sliding window-based reset
strategy, shown in Fig. 3. Our frustum culling strategy allows us to optimize
only the mask within the current viewing frustum while keeping the rest of the
3D Gaussian ellipsoids fixed. It will not only preserve the previously
reconstructed geometry but also significantly reduce the number of parameters
during optimization. Different from the original densification strategy
performed on every frame, we only perform mask on the keyframe (each $k^{th}$
frame) for efficiency and accuracy. We maintain a local sliding window and
perform sliding window reset to avoid the continuous optimization and
accumulated gradient of masks which will ultimately eliminate all Gaussian
ellipsoids. The sliding window consists of the current frame, the most
relevant keyframe, and $n-2$ previous keyframes, which have the highest
overlap with the current frame. Overlap is evaluated by analyzing the point
cloud of the current frame’s depth map and tallying points within the frustum
of each keyframe. This can also ensure the consistency of the mask within the
local sliding window. This approach allows us to continuously mask out
unnecessary Gaussians during online SLAM system operation, effectively
reducing computation overhead and ensuring efficient memory usage on GPU.

Figure 5: The KL divergence distribution of the Gaussian ellipsoids with the
online training of the SLAM system on different time steps (500, 1000, 1500,
2000). We can observe that the similarity in geometry consistently remains at
a high level of GS-based SLAM system.

###  3.3 Geometry Codebook

In this section, we analyze and observe the geometry similarities of the
Gaussian ellipsoids created by SLAM systems. Then, we propose a learnable
codebook and employ a residual vector quantization method to reduce
computational complexity and memory usage and further improve the training and
rendering speed.

For the GS-based SLAM system, a scene is composed of a number of small
Gaussian ellipsoids with 3D geometry attributes (scale and rotation matrix
$\boldsymbol{S},\boldsymbol{R}$). Consider that the 3D Gaussian ellipsoids
$G_{1}$,$G_{2}$ conform to an unbiased Gaussian distribution
$\mathcal{N}(0,\boldsymbol{\Sigma}_{1}),\mathcal{N}(0,\boldsymbol{\Sigma}_{2})$,
we adopt the Kullback-Leibler divergence [15] to analysis the geometry
similarities of 3D Gaussian ellipsoids:

$D_{KL}=\frac{1}{2p}tr(\boldsymbol{\Sigma}_{1}^{-1}\boldsymbol{\Sigma}_{2})-\frac{1}{2}+\frac{1}{2p}lndet(\boldsymbol{\Sigma}_{1}^{-1}\boldsymbol{\Sigma}_{2})$
(9)

where $p$ is the dimension of the covariance matrix. We conduct extensive
experiments and present our results on Tab. 1 and Fig. 5. We can see that the
percentage of 3D Gaussian ellipsoids is significantly elevated in a small
range of KL divergence, which demonstrates the similarities of the 3D Gaussian
ellipsoids shared across the scene. Our experiments also show that the
similarities of 3D Gaussians of the GS-based SLAM system are greater than the
original 3DGS. This is probably due to the online optimization strategy that
the SLAM system only uses the current frame and history keyframes to optimize
the 3D Gaussian attributes, which will exacerbate the geometry similarity.
Based on the similarity, we propose a learnable codebook to compress the
geometry attributes (scale and rotation), shown in Fig. 4. Inspired by
SoundStream [43] and Encodec [8], we incorporate the residual vector
quantization (R-VQ) to compress the scale and rotation. It cascades L stages
of VQ and is formulated as follows:

$\displaystyle\hat{S}_{n}^{l}$
$\displaystyle=\sum_{k=1}^{l}\mathcal{C}^{k}\left[i^{k}\right],\quad
l\in\\{1,\ldots,L\\},$ (10) $\displaystyle i_{n}^{l}$
$\displaystyle=\underset{k}{\operatorname{argmin}}\left\|\mathcal{C}^{l}[k]-\left(S_{n}-\hat{S}_{n}^{l-1}\right)\right\|_{2}^{2},\quad\hat{S}_{n}^{0}=\overrightarrow{0}$

where $S\in\mathcal{R}^{N\times 4}$ is the scale vector,
$\hat{S}^{l}\in\mathcal{R}^{N\times 4}$ is the output scale vector after $l$
stages quantization. $n$ denotes the index of the Gaussian ellipsoids.
$\mathcal{C}^{l}$ denotes the codebook at the stage l. $\mathcal{C}^{l}$
represents the vector at index i of the codebook $\mathcal{C}$. The
formulation of the rotation vector is the same. Then, the loss function is
defined as:

$L_{r}=\frac{1}{NP}\sum_{k=1}^{L}\sum_{n=1}^{N}\left\|\operatorname{sg}\left[S_{n}-\hat{S}_{n}^{k-1}\right]-\mathcal{C}^{k}\left[i_{n}^{k}\right]\right\|_{2}^{2}$
(11)

where $P$ is the size of codebook, $sg[\cdot]$ is the stop gradient operator.
After this, we can only store the codebook compressed scale and rotation
vector, which can significantly reduce storage and memory usage.

Table 1: The KL divergence analysis of GS-based SLAM and original 3DGS.

SplaTAM [12] | 3DGS [13]
---|---
Range | Percentage | Range | Percentage
(-2.5%,2.5%) | 87.67% | (-2.5%,2.5%) | 20.07%
(-5.0%,5.0%) | 97.86% | (-5.0%,5.0%) | 37.04%
(-7.5%,7.5%) | 99.16% | (-7.5%,7.5%) | 47.69%
(-10.0%,10.0%) | 99.52% | (-10.0%,10.0%) | 52.16%

###  3.4 Tracking and Global Bundle Adjustment

Our tracking and bundle adjustment are performed via minimizing our objective
functions. The camera pose is initialized for a new time step by a constant
velocity forward projection of the pose parameters. The color and depth loss
is defined as:

$\mathcal{L}_{c}=\frac{1}{N}\sum_{i=1}^{N}\left(\hat{\mathbf{C}}_{i}-\mathbf{C_{i}}\right)^{2},\quad\mathcal{L}_{d}=\frac{1}{\left|R_{i}\right|}\sum_{i\in
R_{i}}\left(\mathbf{\hat{D_{i}}}-\mathbf{D_{i}}\right)^{2}$ (12)

where $R_{i}$ is the set of rays that have a valid depth observation. The
reprojection error is common in traditional SLAM methods based on sparse point
clouds [25]. Since 3D Gaussian is also based on a point cloud representation,
we introduce this loss for the first time to improve the scene’s geometric
representation and consistency further. We formulate reprojection errors with
SIFT features:

$\vspace{-0.2cm}\mathcal{L}_{re}=\sum_{i=1}^{n}\left\|(u_{i^{\prime}},v_{i^{\prime}})-\Pi(R_{i\to
i^{\prime}}P_{i}+t_{i\to i^{\prime}})\right\|\vspace{-0.2cm}$ (13)

where $\Pi(R_{i\to i^{\prime}}P_{i}+t_{i\to i^{\prime}})$ represents the
reprojection of 3D point $P_{i}$ to the corresponding pixel
$(u_{i^{\prime}},v_{i^{\prime}})$ in image $i^{\prime}$. The tracking loss is
formulated as follows:

$L_{\mathrm{t}}=\sum_{\mathbf{p}}(S(\mathbf{p})>0.99)\left(\mathcal{L}_{c}+\lambda_{1}\mathcal{L}_{d}+\lambda_{2}\mathcal{L}_{re}\right)$
(14)

We use the rendered visibility silhouette to select the well-optimized pixels
for camera tracking, which can improve the tracking accuracy for the new
frames.

Global Bundle Adjustment. For global consistency and accuracy, our system
maintains a significantly larger global keyframe database than other GS-based
SLAM systems. We randomly sample a total number of N rays from our global
keyframe database to optimize our scene representation as well as camera
poses. This phase optimizes a loss similar to tracking loss, and we also add
an SSIM loss to RGB rendering. The global bundle adjustment is performed to
optimize the scene representation with the camera pose. Our global BA method
can effectively reduce cumulative errors and enhance the robustness of pose
estimation, especially for long sequences and large scenes.

Figure 6: The rendering visualization results on the Replica dataset [30] of
the proposed GS-based SLAM system compared with other SOTA methods. We present
the rendering PSNR and FPS on the image. Our method can achieve faster
rendering speed and high-quality image reconstruction performance compared
with other methods.

##  4 Experiments

Datasets and Evaluation Metrics. We evaluate our method on various datasets:
Replica [30], Scannet [3], and TUM-RGBD [31], following the settings form
SplaTAM [12]. The replica is a synthetic dataset, and we select eight small
room scene sequences for evaluation. We select six real-world scenes from the
ScanNet [3] dataset for relatively large-scale indoor scenes. We also select
five scenes from the TUM RGB-D [31] dataset as it is a common dataset for
evaluating camera tracking accuracy. We follow the evaluation metrics from
SplaTAM [12]. We use PSNR, SSIM, and LPIPS to measure rendering performance.
For camera pose estimation, we use the average absolute trajectory error (ATE
RMSE$[cm]$ $\downarrow$).

Table 2: Camera tracking results on Replica dataset [30]. We use the ATE RMSE
$\downarrow[cm]$ as the metric and compare it with other SOTA methods.

Methods | Avg. | R0 | R1 | R2 | Of0 | Of1 | Of2 | Of3 | Of4
---|---|---|---|---|---|---|---|---|---
Vox-Fusion [40] | 3.09 | 1.37 | 4.70 | 1.47 | 8.48 | 2.04 | 2.58 | 1.11 | 2.94
NICE-SLAM [45] | 1.06 | 0.97 | 1.31 | 1.07 | 0.88 | 1.00 | 1.06 | 1.10 | 1.13
ESLAM [11] | 0.63 | 0.71 | 0.70 | 0.52 | 0.57 | 0.55 | 0.58 | 0.72 | 0.63
Co-SLAM [34] | 0.78 | 0.61 | 0.79 | 0.96 | 0.54 | 0.52 | 1.13 | 0.97 | 0.74
Point-SLAM [29] | 0.52 | 0.61 | 0.41 | 0.37 | 0.38 | 0.48 | 0.54 | 0.69 | 0.72
SplaTAM [12] | 0.36 | 0.31 | 0.40 | 0.29 | 0.47 | 0.27 | 0.29 | 0.32 | 0.55
Ours | 0.33 | 0.27 | 0.38 | 0.25 | 0.44 | 0.27 | 0.27 | 0.29 | 0.50

Table 3: Camera tracking results on ScanNet datasets [3]. We use the ATE RMSE
$\downarrow[cm]$ as the metric and compare it with other SOTA methods.

Methods | Avg. | 0000 | 0059 | 0106 | 0169 | 0181 | 0207
---|---|---|---|---|---|---|---
Vox-Fusion [40] | 26.90 | 68.84 | 24.18 | 8.41 | 27.28 | 23.30 | 9.41
NICE-SLAM [45] | 10.88 | 12.00 | 14.00 | 7.90 | 10.90 | 13.40 | 6.20
Co-SLAM [34] | 8.72 | 7.35 | 11.45 | 9.68 | 6.03 | 11.93 | 8.86
ESLAM [11] | 7.82 | 7.84 | 9.24 | 7.82 | 6.78 | 9.35 | 5.88
Point-SLAM [29] | 12.19 | 10.24 | 7.81 | 8.65 | 22.16 | 14.77 | 9.54
SplaTAM [12] | 11.88 | 12.83 | 10.10 | 17.72 | 12.08 | 11.10 | 7.46
Ours | 10.84 | 11.28 | 9.24 | 16.49 | 11.09 | 10.88 | 6.58

Implementation Details. Our method is implemented in Python using the PyTorch
framework, incorporating CUDA for Gaussian Splatting, and trained on a desktop
PC with NVIDIA RTX 3090Ti GPU. We extended the existing differentiable
Gaussian Splatting rasterization code by adding functions to manage depth,
pose, and cumulative opacity during both forward and backward propagation.

Baselines. The main baseline method we compare to is SplaTAM [12] as the
representative of the GS-based SLAM systems because it is the only open-
sourced GS-based SLAM system so far. Our method is also a plug-and-play
method, which can improve the speed and memory usage for all GS-based methods.
We also compared to other NeRF-based SLAM methods, such as NICE-SLAM [45], Co-
SLAM [34], ESLAM [11], Vox-Fusion [40]. On the TUM-RGBD dataset, we also
compare against three traditional SLAM systems: Kintinuous [35], ElasticFusion
[36], and ORB-SLAM2 [25].

###  4.1 Experimental Results

In this section, we present our experimental results on three different
datasets. We evaluate the camera pose estimation, the 3D Gaussian
reconstruction, and the real-time performance and memory usage of different
SLAM systems.

Camera tracking results. In Tab. 2 3 4, we compare our method with other SOTA
methods on camera pose estimation on different datasets. Best results are
highlighted as first, second, and third. On the synthetic dataset Replica
[30], we can see that we successfully reduce the trajectory error over the GS-
based SLAM system (SplaTAM [12]) and achieve more accurate and robust pose
estimation, shown in Tab. 2. The ScanNet dataset is a real-world dataset that
has poor depth sensor information with high motion blur on RGB images. In Tab.
3, our method performs better than the GS-based SLAM system and similarly to
the previous NeRF-based SLAM systems. In Tab. 4, we present the experimental
results on TUM RGB-D datasets. We can see that our approach still outperforms
other GS-based or NeRF-based methods, and the trajectory error has nearly 10%
reduction. Our experiments demonstrate the effectiveness of our proposed
reprojection loss, which can still improve the pose estimation accuracy with a
number of Gaussian ellipsoids removed.

Table 4: Camera tracking results on TUM RGB-D dataset [31]. We use the ATE
RMSE $\downarrow[cm]$ as the metric and compare it with other SOTA methods.

Methods | Avg. | fr1/desk | fr1/desk2 | fr1/room | fr2/xyz | fr3/office
---|---|---|---|---|---|---
Kintinuous [35] | 4.84 | 3.70 | 7.10 | 7.50 | 2.90 | 3.00
ElasticFusion [36] | 6.91 | 2.53 | 6.83 | 21.49 | 1.17 | 2.52
ORB-SLAM2 [25] | 1.98 | 1.60 | 2.20 | 4.70 | 0.40 | 1.00
1-7 NICE-SLAM [45] | 15.87 | 4.26 | 4.99 | 34.49 | 31.73 | 3.87
Vox-Fusion [40] | 11.31 | 3.52 | 6.00 | 19.53 | 1.49 | 26.01
Point-SLAM [29] | 8.92 | 4.34 | 4.54 | 30.92 | 1.31 | 3.48
SplaTAM [12] | 5.48 | 3.35 | 6.54 | 11.13 | 1.24 | 5.16
Ours | 5.17 | 2.95 | 6.13 | 10.85 | 1.03 | 4.88

Figure 7: We visualized the learned codebook indices for scale and rotation
vector in the replica dataset. We can observe that as the stage progresses,
the distribution of the codebook becomes increasingly compact. Due to the
space limitation, we only visualize stages 1,3,6; the complete results are
provided in the supplementary materials.

Gaussian reconstruction results. In Tab. 5, we show the rendering quality of
the Replica dataset in 8 scenes. We also present the quantitative results in
Fig. 6. Our method achieves similar PSNR, SSIM, and LPIPS results as SplaTAM
[12]. Our approach achieves much better results than the NeRF-based baselines,
such as Vox-Fusion [40], NICE-SLAM [45], Co-SLAM [34], and ESLAM [11].
Overall, our method can achieve better performance than the SOTA methods,
while the speed and memory usage are significantly lower than other methods.

Table 5: Quantitative image reconstruction performance on Replica dataset
[30]. We use the PSNR $\uparrow$, SSIM $\uparrow$, and LPIPS $\downarrow$ as
the metrics and compare them with other SOTA methods.

Methods | Metrics | Avg. | R0 | R1 | R2 | Of0 | Of1 | Of2 | Of3 | Of4
---|---|---|---|---|---|---|---|---|---|---
Vox-Fusion [40] |  | PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 24.41
---
0.80
0.24
| 22.39
---
0.68
0.30
| 22.36
---
0.68
0.30
| 23.92
---
0.80
0.23
| 27.79
---
0.86
0.24
| 29.83
---
0.88
0.18
| 20.33
---
0.79
0.24
| 23.47
---
0.80
0.21
| 25.21
---
0.85
0.20
Co-SLAM [34] |  | PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 30.24
---
0.94
0.25
| 27.27
---
0.91
0.32
| 28.45
---
0.91
0.29
| 29.06
---
0.93
0.27
| 34.14
---
0.96
0.21
| 34.87
---
0.97
0.20
| 28.43
---
0.94
0.26
| 28.76
---
0.94
0.23
| 30.91
---
0.96
0.24
ESLAM [11] |  | PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 29.08
---
0.93
0.25
| 25.32
---
0.86
0.31
| 27.77
---
0.90
0.30
| 29.08
---
0.93
0.25
| 33.71
---
0.96
0.18
| 30.20
---
0.92
0.23
| 28.09
---
0.94
0.24
| 28.77
---
0.95
0.20
| 29.71
---
0.95
0.20
NICE-SLAM [45] |  | PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 24.42
---
0.81
0.23
| 22.12
---
0.69
0.33
| 22.47
---
0.76
0.27
| 24.52
---
0.81
0.21
| 29.07
---
0.87
0.23
| 30.34
---
0.89
0.18
| 19.66
---
0.80
0.24
| 22.23
---
0.80
0.21
| 24.49
---
0.86
0.20
Point-SLAM [29] |  | PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 35.17
---
0.98
0.12
| 32.40
---
0.97
0.11
| 34.08
---
0.98
0.12
| 35.50
---
0.98
0.11
| 38.26
---
0.98
0.10
| 39.16
---
0.99
0.12
| 33.99
---
0.96
0.16
| 33.48
---
0.96
0.13
| 33.49
---
0.98
0.14
SplaTAM [12] |  | PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 34.11
---
0.97
0.10
| 32.86
---
0.98
0.07
| 33.89
---
0.97
0.10
| 35.25
---
0.98
0.08
| 38.26
---
0.98
0.09
| 39.17
---
0.97
0.09
| 31.97
---
0.97
0.10
| 29.70
---
0.95
0.12
| 31.81
---
0.95
0.15
Ours |  | PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 34.44
---
0.98
0.09
| 32.98
---
0.99
0.07
| 34.08
---
0.98
0.10
| 35.35
---
0.98
0.08
| 38.16
---
0.98
0.09
| 39.07
---
0.98
0.09
| 32.37
---
0.97
0.10
| 31.08
---
0.96
0.12
| 32.31
---
0.96
0.15

Real-time performance and memory usage. Tab. 6 illustrates the runtime
performance and memory usage of our method and other GS-based and NeRF-based
SLAM systems on the Replica [30] room 0 scene and the ScanNet [3] 0000 scene.
We follow the settings of SplaTAM and use 60,90 iterations per frame for
tracking and mapping to get similar performance in [12]. We evaluate the time
consumption of tracking and mapping every iteration and every frame. Each
iteration of our system renders a full 1200$\times$980 pixel image (in Replica
dataset) and 640$\times$480 pixel image (in ScanNet dataset). Compared with
the GS-based SLAM system, we significantly improve the training speed (33%
faster), which is crucial for online SLAM systems, thanks to our compact 3D
Gaussian scene presentation method. We also evaluate the rendering speed, the
decoder parameters, and the memory of SOTA methods. Compared with the NeRF-
based SLAM system, we achieve 414.45 FPS rendering speed on the replica
dataset, which is 100$\times$ significantly faster than these methods. Note
that we do not use any neural network decoder in our system, which results in
zero learnable parameters of the decoder. Compared with the GS-based SLAM
system, our method also achieves 150% reduction on memory usage and 176%
faster on rendering speed because of the proposed compact 3D Gaussian scene
presentation.

Table 6: Runtime and memory performance evaluation on Replica [30] and ScanNet
[3] dataset. The De. Param denotes the parameter number of MLPs. The memory is
the memory usage of the checkpoint.

| Method | Track/It. | Map/It. | Track/Fr. | Map/Fr. | Render FPS | De. Param. | Memory
---|---|---|---|---|---|---|---|---
Replica | NICE-SLAM | 6.98ms | 28.88ms | 68.54ms | 1.23s | 0.30 | 0.06M | 48.48MB
ESLAM | 6.85ms | 19.98ms | 54.80ms | 0.29s | 2.82 | 0.003M | 27.12MB
Co-SLAM | 6.38ms | 14.25ms | 63.93ms | 0.15s | 3.68 | 0.013M | 24.85MB
Point-SLAM | 0.57ms | 34.85ms | 22.72ms | 10.47s | 1.33 | 0.127M | 55.42MB
2-9 | SplaTAM | 25.43ms | 23.8ms | 2.25s | 1.43s | 125.64 | 0M | 273.09MB
| Ours | 18.37ms | 17.48ms | 1.53s | 1.03s | 347.73 | 0M | 140.29MB
Scannet | NICE-SLAM | 12.3ms | 125.3ms | 0.62s | 7.52s | 0.27 | 0.07M | 84.25MB
ESLAM | 7.5ms | 23.4ms | 0.23s | 0.70s | 2.74 | 0.004M | 67.9MB
Co-SLAM | 7.8ms | 20.6ms | 78ms | 0.20s | 3.41 | 0.015M | 48.8MB
Point-SLAM | 0.63ms | 42.29ms | 64ms | 12.69s | 1.28 | 0.131M | 27.8MB
2-9 | SplaTAM | 29.22ms | 27.27ms | 2.63s | 1.64s | 115.48 | 0M | 178.09MB
| Ours | 21.91ms | 20.25ms | 1.99s | 1.23s | 328.93 | 0M | 98.88MB

Table 7: The ablation study on Replica [30] dataset. We conduct experiments to
verify the effectiveness of our method. Our full model achieves better pose
estimation performance as well as faster training and rendering speed and
lower memory usage.

Method | Accuracy | Real-time performance and Memory
---|---|---
PSNR | SSIM | LPIPS | ATE RMSE | Training Time | Render FPS | Memory
w/o mask | 32.72 | 0.96 | 0.11 | 0.28 | 2.01H | 245.28 | 243.09MB
w/o rvq | 32.87 | 0.97 | 0.10 | 0.27 | 1.58H | 272.29 | 178.88MB
w/o rep. loss | 32.80 | 0.97 | 0.10 | 0.30 | 1.42H | 347.73 | 140.29MB
Ours | 32.98 | 0.98 | 0.09 | 0.27 | 1.42H | 347.73 | 140.29MB

###  4.2 Ablation Study

In this section, we conduct various experiments to verify the effectiveness of
our method. Tab. 7 illustrates a quantitative evaluation with different
settings on Replica [30] room0 scene.

Sliding window mask As shown in Tab. 7, the proposed sliding window mask
effectively reduces the number of 3D Gaussian ellipsoids while retaining the
image reconstruction performance. This demonstrates that we successfully
remove the redundant and unessential 3D Gaussian ellipsoids created along with
the SLAM system operation. Our mask strategy shows 50% increase in the storage
efficiency and a 26% increase in rendering speed.

Residual vector codebook Our proposed geometry codebook method achieves a
reduction in memory usage while maintaining the image reconstruction
performance. We also present the scale and rotation codebook in different
stages in Fig. 7. As the stages progress, the magnitude of the codes
diminishes, demonstrating that the residuals for each stage are being
effectively trained to capture the geometry accurately.

Reprojection loss In Tab. 7, we can see that the proposed reprojection loss
effectively improves the accuracy of camera pose estimation. As the 3D
Gaussian represents the scene with a number of points, we use this
reprojection loss to build the bridge of 3D points with 2D features, which can
further improve the scene representation and the camera tracking accuracy.

##  5 Conclusion

In this paper, we propose a novel GS-based SLAM system with compact 3D
Gaussian scene representation, reducing the number of redundant Gaussian
ellipsoids without a decrease in performance. The proposed sliding window mask
method and the geometry codebook improve the compactness of the scene
representation, achieving faster training and rendering speed, and a
significant reduction in memory usage. The proposed global bundle adjustment
with reprojection loss further improves the camera tracking accuracy and scene
representation. The extensive experiments demonstrate that our work provides a
comprehensive dense visual SLAM system, achieving high-fidelity performance,
fast training, compactness, and real-time rendering.

## References

  * [1] Bengio, Y., Léonard, N., Courville, A.: Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 (2013)
  * [2] Cadena, C., Carlone, L., Carrillo, H., Latif, Y., Scaramuzza, D., Neira, J., Reid, I., Leonard, J.J.: Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age. IEEE Transactions on robotics 32(6), 1309–1332 (2016)
  * [3] Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T., Niessner, M.: Scannet: Richly-annotated 3d reconstructions of indoor scenes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (July 2017)
  * [4] Deng, T., Liu, S., Wang, X., Liu, Y., Wang, D., Chen, W.: Prosgnerf: Progressive dynamic neural scene graph with frequency modulated auto-encoder in urban scenes. arXiv preprint arXiv:2312.09076 (2023)
  * [5] Deng, T., Shen, G., Qin, T., Wang, J., Zhao, W., Wang, J., Wang, D., Chen, W.: Plgslam: Progressive neural scene represenation with local to global bundle adjustment. arXiv preprint arXiv:2312.09866 (2023)
  * [6] Deng, T., Xie, H., Wang, J., Chen, W.: Long-term visual simultaneous localization and mapping: Using a bayesian persistence filter-based global map prediction. IEEE Robotics & Automation Magazine 30(1), 36–49 (2023)
  * [7] Durrant-Whyte, H., Bailey, T.: Simultaneous localization and mapping: part i. IEEE robotics & automation magazine 13(2), 99–110 (2006)
  * [8] Défossez, A., Copet, J., Synnaeve, G., Adi, Y.: High fidelity neural audio compression. arXiv preprint arXiv:2210.13438 (2022)
  * [9] Fan, Z., Wang, K., Wen, K., Zhu, Z., Xu, D., Wang, Z.: Lightgaussian: Unbounded 3d gaussian compression with 15x reduction and 200+ fps. arXiv preprint arXiv:2311.17245 (2023)
  * [10] Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., Shotton, J., Hodges, S., Freeman, D., Davison, A., et al.: Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In: Proceedings of the 24th annual ACM symposium on User interface software and technology. pp. 559–568 (2011)
  * [11] Johari, M.M., Carta, C., Fleuret, F.: Eslam: Efficient dense slam system based on hybrid representation of signed distance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 17408–17419 (2023)
  * [12] Keetha, N., Karhade, J., Jatavallabhula, K.M., Yang, G., Scherer, S., Ramanan, D., Luiten, J.: Splatam: Splat, track & map 3d gaussians for dense rgb-d slam. arXiv preprint arXiv:2312.02126 (2023)
  * [13] Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics 42(4) (2023)
  * [14] Kopanas, G., Philip, J., Leimkühler, T., Drettakis, G.: Point-based neural rendering with per-view optimization. In: Computer Graphics Forum. vol. 40, pp. 29–43. Wiley Online Library (2021)
  * [15] Kullback, S., Leibler, R.A.: On information and sufficiency. The annals of mathematical statistics 22(1), 79–86 (1951)
  * [16] Lee, J.C., Rho, D., Sun, X., Ko, J.H., Park, E.: Compact 3d gaussian representation for radiance field. arXiv preprint arXiv:2311.13681 (2023)
  * [17] Li, M., He, J., Jiang, G., Wang, H.: Ddn-slam: Real-time dense dynamic neural implicit slam with joint semantic encoding. arXiv preprint arXiv:2401.01545 (2024)
  * [18] Li, M., Liu, S., Zhou, H.: Sgs-slam: Semantic gaussian splatting for neural dense slam. arXiv preprint arXiv:2402.03246 (2024)
  * [19] Liu, J., Wang, G., Jiang, C., Liu, Z., Wang, H.: Translo: A window-based masked point transformer framework for large-scale lidar odometry. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 1683–1691 (2023)
  * [20] Liu, J., Wang, G., Liu, Z., Jiang, C., Pollefeys, M., Wang, H.: Regformer: An efficient projection-aware transformer network for large-scale point cloud registration. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 8451–8460 (October 2023)
  * [21] Matsuki, H., Murai, R., Kelly, P.H., Davison, A.J.: Gaussian splatting slam. arXiv preprint arXiv:2312.06741 (2023)
  * [22] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. In: ECCV (2020)
  * [23] Morgenstern, W., Barthel, F., Hilsmann, A., Eisert, P.: Compact 3d scene representation via self-organizing gaussian grids. arXiv preprint arXiv:2312.13299 (2023)
  * [24] Mur-Artal, R., Montiel, J.M.M., Tardós, J.D.: Orb-slam: A versatile and accurate monocular slam system. IEEE Transactions on Robotics 31(5), 1147–1163 (2015). https://doi.org/10.1109/TRO.2015.2463671
  * [25] Mur-Artal, R., Tardós, J.D.: Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE Transactions on Robotics 33(5), 1255–1262 (2017). https://doi.org/10.1109/TRO.2017.2705103
  * [26] Newcombe, R.A., Lovegrove, S.J., Davison, A.J.: Dtam: Dense tracking and mapping in real-time. In: 2011 international conference on computer vision. pp. 2320–2327. IEEE (2011)
  * [27] Qin, T., Li, P., Shen, S.: Vins-mono: A robust and versatile monocular visual-inertial state estimator. IEEE Transactions on Robotics 34(4), 1004–1020 (2018). https://doi.org/10.1109/TRO.2018.2853729
  * [28] Reitmayr, G., Langlotz, T., Wagner, D., Mulloni, A., Schall, G., Schmalstieg, D., Pan, Q.: Simultaneous localization and mapping for augmented reality. In: 2010 International Symposium on Ubiquitous Virtual Reality. pp. 5–8. IEEE (2010)
  * [29] Sandström, E., Li, Y., Van Gool, L., Oswald, M.R.: Point-slam: Dense neural point cloud-based slam. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 18433–18444 (2023)
  * [30] Straub, J., Whelan, T., Ma, L., Chen, Y., Wijmans, E., Green, S., Engel, J.J., Mur-Artal, R., Ren, C., Verma, S., et al.: The replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05797 (2019)
  * [31] Sturm, J., Engelhard, N., Endres, F., Burgard, W., Cremers, D.: A benchmark for the evaluation of rgb-d slam systems. In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. pp. 573–580 (2012). https://doi.org/10.1109/IROS.2012.6385773
  * [32] Sucar, E., Liu, S., Ortiz, J., Davison, A.J.: imap: Implicit mapping and positioning in real-time. In: ICCV. pp. 6229–6238 (October 2021)
  * [33] Teed, Z., Deng, J.: Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in neural information processing systems 34, 16558–16569 (2021)
  * [34] Wang, H., Wang, J., Agapito, L.: Co-slam: Joint coordinate and sparse parametric encodings for neural real-time slam. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13293–13302 (2023)
  * [35] Whelan, T., Kaess, M., Johannsson, H., Fallon, M., Leonard, J.J., McDonald, J.: Real-time large-scale dense rgb-d slam with volumetric fusion. The International Journal of Robotics Research 34(4-5), 598–626 (2015)
  * [36] Whelan, T., Salas-Moreno, R.F., Glocker, B., Davison, A.J., Leutenegger, S.: Elasticfusion: Real-time dense slam and light source estimation. The International Journal of Robotics Research 35(14), 1697–1716 (2016)
  * [37] Xie, H., Deng, T., Wang, J., Chen, W.: Robust incremental long-term visual topological localization in changing environments. IEEE Transactions on Instrumentation and Measurement 72, 1–14 (2022)
  * [38] Xie, H., Deng, T., Wang, J., Chen, W.: Angular tracking consistency guided fast feature association for visual-inertial slam. IEEE Transactions on Instrumentation and Measurement (2024)
  * [39] Yan, C., Qu, D., Wang, D., Xu, D., Wang, Z., Zhao, B., Li, X.: Gs-slam: Dense visual slam with 3d gaussian splatting. arXiv preprint arXiv:2311.11700 (2023)
  * [40] Yang, X., Li, H., Zhai, H., Ming, Y., Liu, Y., Zhang, G.: Vox-fusion: Dense tracking and mapping with voxel-based neural implicit representation. In: 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). pp. 499–507 (2022). https://doi.org/10.1109/ISMAR55827.2022.00066
  * [41] Yifan, W., Serena, F., Wu, S., Öztireli, C., Sorkine-Hornung, O.: Differentiable surface splatting for point-based geometry processing. ACM Transactions on Graphics (TOG) 38(6), 1–14 (2019)
  * [42] Yugay, V., Li, Y., Gevers, T., Oswald, M.R.: Gaussian-slam: Photo-realistic dense slam with gaussian splatting. arXiv preprint arXiv:2312.10070 (2023)
  * [43] Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., Tagliasacchi, M.: Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing 30, 495–507 (2021)
  * [44] Zhu, S., Wang, G., Blum, H., Liu, J., Song, L., Pollefeys, M., Wang, H.: Sni-slam: Semantic neural implicit slam. arXiv preprint arXiv:2311.11016 (2023)
  * [45] Zhu, Z., Peng, S., Larsson, V., Xu, W., Bao, H., Cui, Z., Oswald, M.R., Pollefeys, M.: Nice-slam: Neural implicit scalable encoding for slam. In: CVPR. pp. 12786–12796 (June 2022)