# On the Hardy-Ramanujan Theorem

Benjamin Durkan

(Date: September 2023)

###### Abstract.

In 1917, G.H.Hardy and S.Ramanujan proved that the ‘typical’ number of prime
factors of a positive integer $n$ is approximately $\ln\ln n$. In this
technical paper we proffer a complete exposition of this proof, and further
provide novel approaches based on probabalistic techniques in order to extract
number theoretic results. Using elementary methods in probability theory, we
expand on current methods to prove a substantially sharper distributional
result than in current literature. We conclude the paper with an original
proof of the Hardy-Ramanujan theorem.

##  1\. Introduction and Preliminaries

###  1.1. Introduction

For millennia, mathematicians have been fascinated by the behaviour of the
distribution of prime numbers. Indeed, as the building blocks of all integers,
number theorists have devoted significant time in understanding them. A
natural question one may ask is

######  Problem 1.1.

‘How many prime factors does a ‘typical’ integer have?’

We will show that _generally_ , this quantity is roughly $\ln(\ln n))$, where
$\ln$ is the natural logarithm.

The cognizant reader will notice that $\ln\ln n$ is the result (up to some
error term) of summing over the reciprocals of the primes at most $n$. This is
a result due to Mertens [8], and in this project we will demonstrate the
relationship between these two, seemingly unrelated, number theoretic results.
Erdős and Kac [3] proved a stronger result than Hardy and Ramanujan in a
completely different way – they proved a distributional result via the use of
probabalistic tools. We will state their result in due course, and discuss
further probabalistic tools to prove the Hardy-Ramanujan Theorem, including
Diaconis’ [2] proof using Chebyshev’s inequality. We then combine these
results in a novel way to prove a refined bound which is not in publication.

As in standard notation (for example, see [11]), define by $\omega(n)$ the
number of _distinct_ prime divisors of a positive integer $n$, and $\Omega(n)$
the number of _total_ prime factors of $n$ (repetitions are permitted). It is
immediate that $\omega(n)\leq\Omega(n)$, and that equality holds precisely
when $n$ is squarefree. To see why, decompose $n$ into prime factors as
$n=p_{1}^{\alpha_{1}}p_{2}^{\alpha_{2}}\times\cdots\times p_{r}^{\alpha_{r}}$
where the $p_{i}$ are distinct primes with $p_{i}<p_{j}$ iff $i<j$. Then by
our definition we have $\omega(n)=r$, whereas
$\Omega(n)=\alpha_{1}+\cdots+\alpha_{r}$. These are equal iff $\alpha_{i}=1$
for each $1\leq i\leq r$, which is precisely the condition that $n$ is
squarefree.

The Hardy-Ramanujan theorem says that $\omega(n),\Omega(n)$ have _normal
order_ $\ln\ln n$. More concretely, our general goal throughout this project
is to demonstrate:

######  Theorem 1.2 (Hardy-Ramanujan).

Let $\kappa>0$ be arbitrary. Then for sufficiently large $N$ the following
inequality

$\displaystyle|f(n)-\ln\ln n|\geq\kappa\ln\ln n$

holds for at most $\kappa N$ integers $n\leq N$, where $f(n)$ is either
$\omega(n)$ or $\Omega(n)$.

A slightly weaker formulation of Theorem 1.2 is the following:

######  Theorem 1.3 (Hardy-Ramanujan (version 2)).

Let $\delta$ be some positive real number. Then the number of positive
integers $n\leq N$ which satisfy

$|\omega(n)-\ln\ln N|\geq(\ln\ln N)^{\frac{1}{2}+\delta}$

is $o(N)$.

We will prove stronger results in the following sections which will have
Theorem 1.2 as a corollary. There are many natural questions one might ask,
such as

######  Problem 1.4.

‘Is there a stronger result than Theorem 1.2 which gives a distribution for
$\omega(n)$ or $\Omega(n)$?’

M.Kac and P.Erdős [3] answered this question in the positive, asserting a
Central Limit Theorem for both $\omega(n)$ and $\Omega(n)$. Indeed, they
proved that for sufficiently large $n$, we have that

$\frac{f(n)-\ln\ln n}{\sqrt{\ln\ln n}}$

follows the standard normal distribution, where $f(n)=\omega(n)$ or
$\Omega(n)$. We will explore this result further and consider some examples of
this.

Finally, we will use elementary probabalistic methods to derive a novel result
not seen in current publications. More specifically, according to Diaconis [2]
we have that

$\mathbb{P}\left(\left|\frac{\omega(n)-\ln\ln n}{\sqrt{\ln\ln
n}}\right|>A\right)\leq\frac{1}{A^{2}}.$

We demonstrate that for large enough $A$, we have the refined upper bound

$\sqrt{2}\exp\left(-\frac{A^{2}}{2}\right),$

in place of the $\frac{1}{A^{2}}$ given by [2]. This is clearly significantly
stronger than the result in [2].

Finally, in Chapter 5 we straightforwardly prove Theorem 1.3 in a completely
new way not seen in current literature. The benefit of this proof is its
simplicity. The author of this thesis intends to publish this result.

###  1.2. Preliminary remarks

There are a few cases of interest with regards to our current problem - the
cases in which $n$ is squarefree or a prime power are particularly useful as
they both help us understand the general case better.

We begin with the case where $n$ is a prime power, that is, $n=p^{k}$ for
$k\in\mathbb{N}$ and some prime $p$. Then, by our definition we have
$\omega(n)=1$. We can see that

$\Omega(n)=k=\frac{k\ln p}{\ln p}=\frac{\ln p^{k}}{\ln p}=\frac{\ln n}{\ln
p}.$

The function $\frac{\ln n}{\ln p}$ is maximised when $\ln p$ is minimised, and
this occurs precisely when $p=2$. Hence an immediate corollary of this is that
for any $n\in\mathbb{N}$ we have $\Omega(n)\leq\frac{\ln n}{\ln 2}$. This
maximum occurs in the case when $n=2^{k}$. Note that here we are only
considering prime powers.

######  Proposition 1.5.

Let $m$ be a positive integer such that $m\leq 2^{n}$. Then $\omega(m)\leq n$.

###### Proof.

Suppose a positive integer has $n$ prime divisors. The minimum possible
product occurs when each of these prime divisors is equal to $2$. In this case
we obtain the product $2\times 2\times\cdots\times 2=2^{n}$. Therefore if
$m\leq 2^{n}$ we have $\omega(m)\leq n$. ∎

This shows intuitively that $\omega$ grows quite slowly.

In [5], it is proved that $\omega(n)=O\left(\frac{\ln n}{\ln\ln n}\right)$.
This result was proved by considering primorials (i.e. products of the first
$k$ primes). In fact, this bound is sharp, as for $n$ a primorial we have that
this bound is attained. We provide a novel proof of this, which uses the prime
number theorem.

######  Proposition 1.6.

The maximum order of $\omega(n)$ is at most $\frac{\ln n}{\ln\ln n}$, that is,

$\displaystyle\omega(n)=O\left(\frac{\ln n}{\ln\ln n}\right).$

We provide a novel proof of this result, which differs substantially from that
given in [5].

###### Proof.

Given a positive integer $n$, we have by definition that
$\omega(n)=\sum_{p|n}1$. Now, splitting this sum up depending on whether $p$
is larger than $r$ or otherwise (where $r$ is to be determined later during
the proof), we obtain the following:

$\displaystyle\omega(n)=\sum_{\begin{subarray}{c}p|n\\\ p\leq
r\end{subarray}}1+\sum_{\begin{subarray}{c}p|n\\\ p>r\end{subarray}}1.$

Now, appealing to the definition of the prime counting function, trivially we
have

$\sum_{\begin{subarray}{c}p|n\\\ p\leq r\end{subarray}}1\leq\pi(r),$

since $\pi(r)$ counts all the primes at most $r$, and imposing the condition
that $p|n$ gives us a smaller subset of values. Thus we have

$\displaystyle\omega(n)$
$\displaystyle\leq\pi(r)+\sum_{\begin{subarray}{c}p|n\\\ p>r\end{subarray}}1$
$\displaystyle\leq(1+\varepsilon)\frac{r}{\ln r}+\frac{\ln n}{\ln r}$

This bound holds since if all primes $p$ dividing $n$ are larger than $r$,
there cannot be more than $\frac{\ln n}{\ln r}$ of them because $r^{\frac{\ln
n}{\ln r}}=n$.

Now, upon writing $r=\frac{\ln n}{\ln\ln n}$ we have:

$\displaystyle\omega(n)$ $\displaystyle<(1+\varepsilon)\frac{r}{\ln
r}+\frac{\ln n}{\ln r}$ [using the previous inequality]
$\displaystyle=\frac{1}{\ln r}\left((1+\varepsilon)\frac{\ln n}{\ln\ln n}+\ln
n\right)$ [factoring out $\frac{1}{\ln r}$] $\displaystyle=\frac{\ln n}{\ln
r}\left(\frac{1+\varepsilon}{\ln\ln n}+1\right)$ [factoring out $\ln n$]
$\displaystyle<(1+\varepsilon)\frac{\ln n}{\ln r}$ [since $\ln\ln n-\ln\ln\ln
n=\ln\ln n\left(1-\frac{\ln\ln\ln n}{\ln\ln
n}\right)>\frac{1}{1+\varepsilon}\ln\ln n$]
$\displaystyle=(1+\varepsilon)\frac{\ln n}{\ln\ln n-\ln\ln\ln n}$ [since
$r=\frac{\ln n}{\ln\ln n}$] $\displaystyle<(1+\varepsilon)^{2}\frac{\ln
n}{\ln\ln n}$ $\displaystyle\text{\Big{[}since $\ln\ln n-\ln\ln\ln n=\ln\ln
n\left(1-\frac{\ln\ln\ln n}{\ln\ln n}\right)>\frac{1}{1+\varepsilon}\ln\ln
n$\Big{]}}.$

This completes the proof. ∎

##  2\. Hardy and Ramanujan’s proof

###  2.1. The case for squarefree $n\in\mathbb{N}$

####  2.1.1. Statement of results and overview of proof

Recall from Section 1.2 that if $n$ is squarefree, trivially we have
$\omega(n)=\Omega(n)$. As such, it would be judicious to study the squarefree
case initially in order to understand better the problem, and then use the
results later on via bootstrapping arguments for the general case.

######  Theorem 2.1.

Let $\phi:\mathbb{N}\to\mathbb{R}$ be _any_ function with
$\lim_{n\to\infty}\phi(n)=\infty$. Then almost all squarefree $n\in\mathbb{N}$
have a number of _distinct_ prime factors in the range

$\left(\ln\ln n-\phi(n)\sqrt{\ln\ln n},\ln\ln n+\phi(n)\sqrt{\ln\ln
n}\right).$

Let us pause to consider what this means. Morally what it says is that _most_
squarefree positive integers have approximately $\ln\ln n$ prime factors, and
that the frequency of integers which do not have this property is exceedingly
small, that is, as we take the limit $n\to\infty$ the probability that a
squarefree $n\in\mathbb{N}$ has a number of prime factors outside of this
range approaches zero. Theorem 2.1 implies Theorem 1.2 (in the squarefree
case) where we put $\phi(n)=\kappa\sqrt{\ln\ln n}$.

Recall in Section 1.2 our result Proposition 1.6 that

$\omega(n)=O\left(\frac{\ln n}{\ln\ln n}\right).$

What we are trying to show is that whilst most squarefree integers have
$\ln\ln n$ prime factors, not all of them do, and that _some_ of them have up
to $\frac{\ln n}{\ln\ln n}$ prime factors. This is clearly larger than $\ln\ln
n$ for sufficiently large $n$. We now delve into an overview of how we are
going to prove this result.

Following [5] we define $\pi_{\nu}(N)$ to be the number of positive integers
$1\leq n\leq N$ which are products of precisely $\nu$ prime factors. To build
our understanding we consider a numerical example. Suppose we wish to find
$\pi_{2}(20)$. This counts the amount of natural numbers at most $20$ which
are products of exactly $2$ prime factors. The integers of this form are
$2\times 3,2\times 5,2\times 7,3\times 5$. As there are $4$ integers in this
list we conclude that $\pi_{2}(20)=4$.

One may point out that in the special case $\nu=1$ we have
$\pi_{1}(N)=\pi(N)$, the prime counting function. This is because the positive
integers which are products of precisely one prime are themselves the primes.
Denote by $Q(N)$ the number of squarefree positive integers $n\leq N$. We note
that

$\sum_{\nu=1}^{\infty}\pi_{\nu}(N)=Q(N),$

since every squarefree integer not exceeding $N$ is counted precisely once on
the LHS.

We are interested in the distribution of $\pi_{\nu}(N)$. The main drive of our
proof is in showing that $\pi_{\nu}(N)$ is small – this will morally tell us
that the number of positive integers with many prime factors is small, and
this is what we want. We will do this by proving an upper bound for
$\pi_{\nu}(N)$. Landau [7] proved the asymptotic result

######  Theorem 2.2 (Landau).

$\pi_{\nu}(N)\sim\frac{N}{\ln N}\frac{(\ln\ln N)^{\nu-1}}{(\nu-1)!},$

as $N\to\infty$ for _fixed_ $\nu$.

However this isn’t quite strong enough for our purposes. We wish to find an
upper bound similar to the RHS of the above expression, but which holds as we
allow $\nu$ to vary. We introduce the following notation:

######  Notation 2.3.

$\displaystyle S_{1}$ $\displaystyle=\sum_{\nu<\ln\ln N-\phi(N)\sqrt{\ln\ln
N}}\pi_{\nu+1}(N)$ $\displaystyle S_{2}$ $\displaystyle=\sum_{\nu>\ln\ln
N+\phi(N)\sqrt{\ln\ln N}}\pi_{\nu+1}(N).$

Let us consider what $S_{1}$ and $S_{2}$ actually are. $S_{1}$ counts the
positive integers with ‘too few’ prime factors, and $S_{2}$ counts the
positive integers with ‘too many’ prime factors.

Our job is to show that $S_{1}$ and $S_{2}$ are sufficiently small, in fact,
that $S_{1},S_{2}=o(N)$. This is what will prove Theorem 2.1. We now elaborate
and explain why this is all we need to prove.

Denote by $R(N)$ the number of squarefree positive integers $n<N$ satisfying
the above inequalities (the summation conditions in Notation 2.3). Further
define $Q(N)$ to be the number of squarefree positive integers $n<N$. Then we
need to show that $\frac{R(N)}{Q(N)}\to 0$. Note that
$Q(N)\sim\frac{6N}{\pi^{2}}$ (see [6]) and so we need to show that
$R(N)=o(N)$. So if we show that $R(N)=S_{1}+S_{2}+o(N)$ then it suffices to
prove that $S_{1},S_{2}$ are $o(N)$. Very roughly, we will prove that
$S_{1},S_{2}$ are small by bounding the size of $\pi_{\nu}$, by overestimating
the number of terms it can have.

####  2.1.2. Technical Lemmas

We begin by demonstrating that $\pi_{\nu}(N)$ is sufficiently small.

######  Lemma 2.4.

There exist fixed constants $A_{1},A_{2}$ such that for any
$\nu\in\mathbb{Z}^{+}$ and $N>3$ we have:

(1) $\pi_{\nu+1}(N)<\frac{A_{1}N}{\ln N}\frac{(\ln\ln N+A_{2})^{\nu}}{\nu!}$

We remark that this is in fact stronger than Landau’s result (Theorem 2.2),
since the constants $A_{1},A_{2}$ are absolute for _any_
$\nu\in\mathbb{R}^{+}$, whereas in Landau’s asymptotic result, we are required
to fix $\nu$.

###### Proof.

Recall that Merten’s theorems [8] tell us that there exist constants
$B_{1},B_{2}$ such that:

$\displaystyle\sum_{p\leq N}\frac{1}{p}<\ln\ln N+B_{1}$
$\displaystyle\sum_{p\leq N}\frac{\ln p}{p}<B_{2}\ln N$

We shall prove the result for $\nu\geq 1$ by induction, in the case
$A_{2}>B_{1}+B_{2}$.

Following [5], consider the positive integers at most $N$ contained in the
table below, with each row of the following form:

$2\times p_{11}\times p_{12}\times\cdots\times p_{1\nu}$
---
$3\times p_{21}\times p_{22}\times\cdots\times p_{2\nu}$
⋮
$P\times p_{r1}\times p_{r2}\times\cdots\times p_{r\nu}$

(where $r=\pi(N)$) and so on up to $P\times p_{1}\times\cdots\times p_{\nu}$,
where in the $j$th row, $p_{jv}$ is at least the $j$th prime, i.e.
$p_{j\nu}\geq\pi(j)$.

To see what is going on here, consider the following example : $x=50,\nu=2$.
Then the table looks as follows:

$\displaystyle 12\ 20\ 28\ 30\ 42\ 44$ $\displaystyle 18\ 30\ 42\ 45$
$\displaystyle 50$

It is immediate that $P\leq\sqrt{N}$. To see why, suppose the contrary. Then
$P>\sqrt{N}$, and so our product will exceed $\sqrt{N}\sqrt{N}=N$, a
contradiction.

We claim the number of integers in our table is at most
$\sum_{p\leq\sqrt{N}}\pi_{\nu}\left(\frac{N}{p}\right)$.

Again, this is simple to see - if we have a number $p\times
p_{1}\times\cdots\times p_{\nu}$ where the $p_{i}$ are distinct primes at
least as large as $p$, and this number is at most $N$, then $p^{2}\leq N$.

For any $p^{2}\leq N$, the number of such $p\times p_{1}\times\cdots\times
p_{\nu}$ which are at most $N$ cannot exceed
$\pi_{\nu}\left(\frac{N}{p}\right)$, since $\pi_{\nu}\left(\frac{N}{p}\right)$
counts the number of products of $\nu$ primes at most $\frac{N}{p}$. The
result follows.

Now let $\omega_{1}<\omega_{2}<\cdots<\omega_{\nu+1}$ be primes such that
$\displaystyle\prod_{j=1}^{\nu+1}\omega_{j}\leq N$. Consequently the number
$\omega_{1}\omega_{2}\cdots\omega_{\nu+1}$ will appear a minimum of $\nu$
times in our table; once in which the first figure is $\omega_{1}$, once in
that which it is $\omega_{2}$, and so on. Thus we have

(2)
$\nu\pi_{\nu+1}(x)\leq\sum_{p\leq\sqrt{N}}\pi_{\nu}\left(\frac{N}{p}\right).$

Equivalently, upon dividing by the nonzero quantity $\nu$ we obtain the
following inequality which will act as the pivot of our induction argument:

(3)
$\pi_{\nu+1}(N)\leq\frac{1}{\nu}\sum_{p\leq\sqrt{N}}\pi_{\nu}\left(\frac{N}{p}\right).$

We now begin the main part of the induction argument. Assume that the equation
(1) holds when $\nu$ is replaced by $\nu-1$. As such, our inductive hypothesis
is that

$\pi_{\nu}(N)<\frac{A_{1}N}{\ln N}\frac{(\ln\ln N+A_{2})^{\nu-1}}{(\nu-1)!}.$

Noting that $\ln\ln 2+C$ is strictly positive, we obtain the following from
the induction hypothesis and our inequality (3):

$\displaystyle\pi_{\nu+1(N)}$ $\displaystyle<\sum_{p^{2}\leq
N}\frac{1}{\nu}\frac{A_{1}\left(\frac{N}{p}\right)}{\ln\left(\frac{N}{p}\right)}\frac{\left(\ln\ln\frac{N}{p}+A_{2}\right)^{\nu-1}}{(\nu-1)!}$
[due to (3)] $\displaystyle=\sum_{p^{2}\leq
N}\frac{1}{\nu!}\frac{A_{1}N}{p\ln\frac{N}{p}}\left(\ln\ln\frac{N}{p}+A_{2}\right)^{\nu-1}$
[since $\nu!=(\nu-1)!\nu$] $\displaystyle=\frac{A_{1}N}{\nu!}\sum_{p^{2}\leq
N}\frac{\left(\ln\ln\frac{N}{p}+A_{2}\right)^{\nu-1}}{p\ln\frac{N}{p}}$
[factoring out the constant $\frac{A_{1}N}{\nu!}$]
$\displaystyle<\frac{A_{1}N}{\nu!}\sum_{p^{2}\leq N}\frac{(\ln\ln
N)^{\nu-1}}{p\ln\frac{N}{p}}$ [since $\ln\ln$ is monotone increasing]
$\displaystyle=\frac{A_{1}N(\ln\ln N+A_{2})^{\nu-1}}{\nu!}\sum_{p^{2}\leq
N}\frac{1}{p\ln\frac{N}{p}}$ [factoring out a constant]

Moreover, by Taylor’s theorem for $\left(1-\frac{\ln p}{\ln N}\right)$ we
obtain the following (noting that this expansion is valid since
$\displaystyle\frac{\ln p}{\ln N}<1$):

$\displaystyle\frac{1}{\ln N-\ln p}$ $\displaystyle=\frac{1}{\ln
N\left(1-\frac{\log p}{\log N}\right)}$ [factorising]
$\displaystyle=\sum_{j=1}^{\infty}\frac{(\ln p)^{j-1}}{(\ln N)^{j}}$ [using
the Taylor expansion] $\displaystyle=\frac{1}{\ln N}+\frac{\ln p}{(\ln
N)^{2}}\sum_{j=0}^{\infty}\left(\frac{\ln p}{\ln N}\right)^{j}$ [taking some
terms out of the sum] $\displaystyle\leq\frac{1}{\ln N}+\frac{\ln p}{(\ln
N)^{2}}\sum_{j=0}^{\infty}\left(\frac{1}{2}\right)^{j}$ [since $p\leq\sqrt{N}$
we have $\ln p\leq 2\ln N$] $\displaystyle=\frac{1}{\ln N}+\frac{2\ln p}{(\ln
N)^{2}}$ [using the formula for infinite geometric series]

Using this fact, we automatically get the following string of inequalities:

(4) $\displaystyle\sum_{p^{2}\leq N}\frac{1}{p\ln\frac{N}{p}}$
$\displaystyle\leq\frac{1}{\ln N}\sum_{p^{2}\leq N}\frac{1}{p}+\frac{2}{(\ln
N)^{2}}\sum_{p^{2}\leq N}\frac{\ln p}{p}$ (5) $\displaystyle<\frac{\ln\ln
N+B_{1}}{\ln N}+\frac{B_{2}}{\ln N}$ (6) $\displaystyle<\frac{\ln\ln
N+A_{1}}{\ln N},$

using Merten’s theorems as stated above. Substituting in our result that
$\pi_{\nu+1}(N)<\frac{A_{1}N(\ln\ln N+A_{2})^{\nu-1}}{\nu!}\sum_{p^{2}\leq
N}\frac{1}{p\ln\frac{N}{p}}$ gives the result we sought, and so we are done. ∎

As discussed in the previous section, we need to prove that $S_{1},S_{2}$ are
small. In order to prove that $S_{1},S_{2}=o(N)$, the argument is almost
identical for each case. Consequently we give only the proof for $S_{2}$ for
the sake of brevity.

######  Lemma 2.5.

We have that

$S_{2}=o(N).$

The proof of this Lemma relies on some rather technical details so we will
defer the proof until later. Firstly, observe that Lemma 2.4 has the immediate
corollary that

$S_{2}<\frac{A_{1}N}{\ln N}\sum_{\nu>\ln\ln N+\phi(N)\sqrt{\ln\ln
N}}\frac{(\ln\ln N+A_{2})^{\nu}}{\nu!}.$

We now consider the series

$\frac{\delta}{2\times 3}+\frac{\delta^{2}}{3\times
4}+\frac{\delta^{3}}{4\times 5}+\cdots,$

which will be helpful in proving an upper bound in our next result,
Proposition 2.7.

One can easily check that this series has positive radius of convergence and
therefore by continuity we can find a $\delta>0$ such that this series has
value less than and number less than $\frac{1}{2}$, for instance we can choose
a $\delta$ such that the series has value less than $\frac{1}{4}$.
(Alternatively, one could note that the series is bounded above by
$\delta+\delta^{2}+\cdots=\frac{\delta}{1-\delta}$, and choosing for example
$\delta=1/5$ will give that the series is bounded above by $1/4$).

In our above corollary of Lemma 2.4, note that if we write $\xi=\ln\ln
x+A_{2}$ we have that our summation condition is equivalent to
$\nu>\xi+\psi(\xi)\sqrt{\xi}$. We introduce the following notation:

######  Notation 2.6.

$S=\sum_{\nu>\xi+\psi(\xi)\sqrt{\xi}}\frac{\xi^{\nu}}{\nu!}$

We need only prove $S=o(e^{\xi})$. Splitting our sum into two sums which we
will treat separately, this is equivalent to

$S=\sum_{\xi+\psi\sqrt{\xi}<\nu\leq(1+\delta)\xi}\frac{\xi^{\nu}}{\nu!}+\sum_{\nu>(1+\delta)\xi}\frac{\xi^{\nu}}{\nu!}:=S^{\prime}+S^{\prime\prime}$

We now pause to explain why we split the sum up in this way. Bearing in mind
we chose $\delta>0$ to be small, $S^{\prime}$ counts the contribution to $S$
where $\nu$ is _at most_ a small distance away from $\xi$. Similarly
$S^{\prime\prime}$ counts the contribution to $S$ where $\nu$ is only a small
amount larger than $\xi$.

In light of this, we need only prove that $S^{\prime},S^{\prime\prime}$ are
both $o(N)$. Defining $\nu_{1}=\left\lceil(1+\delta)\xi\right\rceil$, we will
show that $S^{\prime\prime}=o(N)$, and a similar argument will show that
$S^{\prime}=o(N)$ so we leave out the proof for the sake of space. We now
prove an upper bound for $S^{\prime\prime}$:

######  Proposition 2.7.

$S^{\prime\prime}<\frac{1+\delta}{\delta}\frac{\xi^{\nu}}{\nu_{1}!}$

###### Proof.

Firstly we prove that

$\sum_{\nu>(1+\delta)\xi}\frac{\xi^{\nu}}{\nu!}<\frac{\xi^{\nu_{1}}}{\nu_{1}!}\left(\sum_{j=0}^{\infty}\frac{\xi^{j}}{(\nu_{1}+1)\cdots(\nu_{1}+j)}\right).$

To do so, note the following:

$\sum_{\nu>(1+\delta)\xi}\frac{\xi^{\nu}}{\nu!}=\frac{\xi^{\nu_{1}}}{\nu_{1}!}+\sum_{\nu>(1+\delta)\xi}\frac{\xi^{\nu}}{\nu!}.$

This is in turn equal to the following:

$\displaystyle S^{\prime\prime}$
$\displaystyle=\sum_{j=1}^{\infty}\frac{\xi^{j}}{\nu_{j}!}$ [by definition of
$S^{\prime\prime}$]
$\displaystyle<\sum_{j=1}^{\infty}\frac{\xi^{\nu_{1}+j-1}}{\nu_{j}!}$ [since
$\nu_{1}+j-1>\nu_{j}$]
$\displaystyle<\sum_{j=1}^{\infty}\frac{\xi^{\nu_{1}+j-1}}{(\nu_{1}+j-1)!}$
[since $\nu_{1}+j-1>\nu_{j}$]
$\displaystyle=\frac{\xi^{\nu_{1}}}{\nu_{1}!}\sum_{j=1}^{\infty}\frac{\xi^{\nu_{1}+j-1}}{(\nu_{1}+1)(\nu_{1}+2)\cdots(\nu_{1}+j-1)}$
[trivially]
$\displaystyle<\frac{\xi^{\nu_{1}}}{\nu_{1}!}\sum_{j=1}^{\infty}\left(\frac{\xi}{\nu_{1}+1}\right)^{j-1}$
[trivially]
$\displaystyle<\frac{\xi^{\nu_{1}}}{\nu_{1}!}\sum_{j=1}^{\infty}\left(\frac{1}{1+\delta}\right)^{j}$
[using the fact that $\frac{\xi}{\nu_{1}+1}<\frac{1}{1+\delta}$]
$\displaystyle=\frac{\xi^{\nu_{1}}}{\nu_{1}!}\frac{1+\delta}{\delta}$ [using
the formula for infinite geometric series]

This completes the proof.
∎

######  Proposition 2.8.

We have that

$\displaystyle S^{\prime\prime}$
$\displaystyle<\frac{K}{\delta\sqrt{\nu_{1}}}\exp(\nu_{1}(\ln\xi-\ln\nu_{1}+1))$
$\displaystyle<\frac{K}{\delta\sqrt{\xi}}\exp((1-\Delta)\xi),$

where for brevity we introduce the notation
$\Delta=(1+\delta)\ln(1+\delta)-\delta$.

###### Proof.

The first inequality is a simple application of Stirling’s formula for
$\displaystyle\frac{1}{\nu_{1}!}$. We prove that for any positive integer $n$
we have $\left(\frac{n}{e}\right)^{n}<n!$. To see why, recall the Maclaurin
series $e^{x}=\sum_{n=0}^{\infty}\frac{x^{n}}{n!}>\frac{x^{n}}{n!}$. If we
write $x=n$ we get the required result. The proof of our proposition follows.
∎

We now prove a trivial result about the size of $\Delta$, in order for our
bound for $S^{\prime\prime}$ to be meaningful.

######  Proposition 2.9.

We have

$\Delta>\frac{1}{2}\delta^{2}(1-\delta).$

###### Proof.

From our definition we have $\Delta=(1+\delta)\ln(1+\delta)-\delta$. Now,
using the Maclaurin series of $\ln(1+\delta)$:

$\displaystyle\ln(1+\delta)$ $\displaystyle=\sum_{n\geq
0}\frac{(-1)^{n}\delta^{n+1}}{n+1}$ [Taylor-Maclaurin expansion of
$\ln(1+\delta)$]
$\displaystyle=\delta-\frac{\delta^{2}}{2}+\frac{\delta^{3}}{3}-\frac{\delta^{4}}{4}+\cdots$
[expanding term-by-term] $\displaystyle<\delta-\frac{1}{2}\delta^{2}$
[trivially]

This yields
$(1+\delta)\ln(1+\delta)-\delta>(1+\delta)\left(\delta-\frac{1}{2}\delta^{2}\right)=\frac{1}{2}\delta^{2}(1-\delta)$,
as claimed. ∎

We are now in a position to prove the following:

######  Proposition 2.10.

$S^{\prime\prime}=o(e^{\xi}),$

i.e. $S^{\prime\prime}=o(N)$.

###### Proof.

From prior results in this section have that
$S^{\prime\prime}<\frac{K}{\delta\sqrt{\xi}}\exp((1-\Delta)\xi)$. Using the
fact that $\Delta>\nu$ we trivially have that
$\frac{K}{\delta\sqrt{\xi}}\exp((1-\Delta)\xi)<\frac{K}{\delta\sqrt{\xi}}\exp((1-\nu)\xi)$.
The result that $S^{\prime\prime}=o(e^{\xi})$ then follows from definition of
$o-$notation. ∎

Analogously we have

######  Proposition 2.11.

$S^{\prime}=o(e^{\xi}).$

Now, since both $S^{\prime},S^{\prime\prime}$ are $o(N)$ their sum is $o(N)$.
But their sum is $S$, and therefore $S=o(N)$. We elaborate slightly on this.
Since $\xi=\ln\ln N$ we have $e^{\xi}=\ln N$, and so $S=o(\ln N)$. Thus
$S_{2}=\frac{KN}{\ln N}\cdot S=o\left(\frac{N}{\ln N}\cdot\ln N\right)=o(N)$.

###### Proof of Lemma 3.4.

The fact that $S=o(N)$ automatically implies that $S_{2}=o(N)$. ∎

What this tells us is precisely what we wanted, that the number of squarefree
integers with normal order _not_ equal to $\ln\ln n$ is exceptionally small,
as we wanted. The proof of our theorem follows.

###### Proof of Theorem 2.1.

This follows immediately from our proof of Lemma 2.4 in conjunction with our
bound for $S$. ∎

###  2.2. The case for general $n\in\mathbb{N}$

####  2.2.1. Statement of results and overview of proof

In Section 1.1 we showed that $\omega(n)=\Omega(n)$ precisely when $n$ is
squarefree. If we now relax the squarefree constraint, we do not have
$\omega(n)=\Omega(n)$ and so we must study these functions separately.
Fortunately, examining the squarefree case was useful as we now need only
minimally alter our arguments to obtain the general case.

In this section we prove the following:

######  Theorem 2.12.

Let $\phi:\mathbb{N}\to\mathbb{R}$ with $\lim_{n\to\infty}\phi(n)=\infty$.
Then almost all $n\in\mathbb{N}$ have a number of prime factors in the range

$\left(\ln\ln n-\phi(n)\sqrt{\ln\ln n},\ln\ln n+\phi(n)\sqrt{\ln\ln
n}\right).$

To do so, we will employ largely the same techniques as in the squarefree
case.

We remark that as in the previous section, this result will imply Theorem 1.3

Following [5] we denote by $\overline{\omega_{\nu}}(N)$ the number of positive
integers $1\leq\nu\leq N$ for which $\omega(n)=\nu$. We begin with a simple
observation:

######  Remark 2.13.

For any pair $N,\nu$ we have that

$\overline{\omega_{\nu}}(N)\geq\pi_{\nu}(N).$

To see why this is the case, notice that $\pi_{\nu}(N)$ is, by definition, the
number of natural numbers no larger than $N$ which are products of _precisely_
$\nu$ distinct prime factors. On the other hand, $\overline{\omega_{\nu}}(N)$
counts these numbers, and also counts those numbers with _repeated_ prime
factors. The result follows, and equality holds if and only if $N$ is
squarefree.

We prove a result specificially for the case $\nu=1$:

######  Proposition 2.14.

We have that

$\overline{\omega_{1}}(N)=\sum_{k=1}^{\infty}\pi\left(N^{\frac{1}{k}}\right)=O\left(\frac{N}{\ln
N}\right).$

###### Proof.

According to Chebyshev’s theorem, $\pi(N)=O\left(\frac{N}{\ln N}\right)$, and
since we trivially have $\pi(N^{\frac{1}{k}})<\pi(N)$ for any $k$, we have
that our sum for
$\displaystyle\overline{\omega_{1}}(N)=O(\pi(N))=O\left(\frac{N}{\ln
N}\right)$. ∎

The way we will prove our result begins by finding a bound for
$\overline{\omega_{\nu}}(N)$ analogous to as in Lemma 2.4, and proceeding in a
similar way to in the squarefree case.

####  2.2.2. Technical Lemmas

We start by proving an upper bound for $\overline{\omega_{\nu+1}}(N)$, much in
the same way as in Lemma 2.4.

######  Lemma 2.15.

There exist fixed constants $C_{1},C_{2}$ such that

$\overline{\omega_{\nu+1}}(N)<\frac{C_{1}N}{\ln N}\frac{(\ln\ln
N+C_{2})^{\nu}}{\nu!}$

provided that $\nu\in\mathbb{Z}^{+}$ and $N>3$.

One should note the striking similarity between the statement of this Lemma
and its equivalent in the previous section - indeed, the proofs in this
section are merely modifications of those in the last section.

###### Proof.

We proceed by induction. Firstly consider the case $\nu=0$. Our claim in this
case is equivalent to

$\overline{\omega_{1}}(N)<\frac{C_{1}N}{\ln N}\frac{(\ln\ln
N+C_{2})^{0}}{0!},$

which is the same as saying

$\overline{\omega_{1}}(N)<\frac{C_{1}N}{\ln N}$

for some constant $C_{1}$. But the existence of such $C_{1}$ is guaranteed by
Chebyshev’s theorem. Let us pause to discuss why this is the case. We have
that $\overline{\omega_{1}}(N)$ counts the number of $p<N$, the number of
$p^{2}<N$ and so on. This sum can be estimated using the Prime Number Theorem.

We now show, in a similar way to the earlier lemmas, that the inequality we
claim holds whenever $C_{2}>B_{1}+B_{2}+B_{3}$ and

$B_{3}:=\sum_{s\geq
2}(s+1)\left(\frac{1}{2^{s}}+\frac{1}{3^{s}}+\frac{1}{5^{s}}+\cdots\right).$

Following [5], consider the numbers not exceeding $N$ in the following table,
with each row containing numbers of the following form:

$\displaystyle 2^{\alpha_{1}}\times p_{11}^{\alpha_{11}}\times
p_{12}^{\alpha_{12}}\times\cdots\times p_{1\nu}^{\alpha_{1\nu}}$
$\displaystyle 3^{\alpha_{2}}\times p_{21}^{\alpha_{21}}\times
p_{22}^{\alpha_{22}}\times\cdots\times p_{2\nu}^{\alpha_{2\nu}}$
$\displaystyle\hskip 56.9055pt\vdots$ $\displaystyle P^{\alpha_{r}}\times
p_{r1}^{\alpha_{r1}}\times p_{r2}^{\alpha_{r2}}\times\cdots\times
p_{r\nu}^{\alpha_{r\nu}},$

where the $p_{i}$ are distinct primes.

We show that we have the upper bound

$P\leq N^{\frac{1}{n+1}},$

when $a=n$. This results in the total number of numbers in our table being
bounded above by

$\sum_{p\leq N^{\frac{1}{a+1}}}\omega_{\nu}\left(\frac{N}{p^{\alpha}}\right).$

To see why this is the case, if we have $a=n$ and $P>N^{\frac{1}{n+1}}$ then
the last row of our table has product exceeding $N^{\frac{1}{n+1}}\times
p_{1}\times p_{2}\times\cdots\times p_{\nu}\geq N$, a contradiction.

Now let $\omega_{1}<\omega_{2}<\cdots,\omega_{\nu+1}$ be primes such that
$\displaystyle\prod_{j=1}^{\nu+1}\omega_{j}^{\alpha_{j}}\leq N$. Then the
number
$\omega_{1}^{\alpha_{1}}\times\cdots\times\omega_{\nu+1}^{\alpha_{\nu+1}}$
will appear a minimum of $\nu$ times in our table. Consequently we obtain the
following upper bound, analogous to that of (3):

(7) $\nu\overline{\omega_{\nu+1}}(N)\leq\sum_{k=2}^{\infty}\sum_{p^{k}\leq
N}\overline{\omega_{\nu}}\left(\frac{N}{p^{k-1}}\right)$

We divide by $\nu\neq 0$ to obtain the following inequality which we will
utilise in our induction argument:

(8)
$\overline{\omega_{\nu+1}}(N)\leq\frac{1}{\nu}\sum_{k=2}^{\infty}\sum_{p^{k}\leq
N}\overline{\omega_{\nu}}\left(\frac{N}{p^{k-1}}\right)$

Now begins the inductive step. Assume that the inequality in Lemma 2.15 holds
for $\nu$, that is,

$\overline{\omega_{\nu}}(N)<\frac{C_{1}N}{\ln N}\frac{(\ln\ln
N+C_{2})^{\nu-1}}{(\nu-1)!},$

for some constants $C_{1},C_{2}$ and $N$ sufficiently large. Applying our
result (8) we obtain the following string of inequalities:

$\displaystyle\overline{\omega_{\nu+1}}(N)$
$\displaystyle\leq\frac{1}{\nu}\sum_{k=2}^{\infty}\sum_{p^{k}\leq
N}\overline{\omega_{\nu}}\left(\frac{N}{p^{k-1}}\right)$ [immediately from
(8)] $\displaystyle<\frac{1}{\nu}\frac{(\ln\ln
N+C_{2})^{\nu}}{(\nu-1)!}\sum_{k=2}^{\infty}\sum_{p^{k}\leq
N}\frac{1}{p^{k-1}\ln\left(\frac{N}{p^{k-1}}\right)}$ [using the inductive
step] $\displaystyle=\frac{(\ln\ln
N+C_{2})^{\nu}}{\nu!}\sum_{k=2}^{\infty}\sum_{p^{k}\leq
N}\frac{1}{p^{k-1}\ln\left(\frac{N}{p^{k-1}}\right)}$
$\displaystyle\text{\Big{[}as $\nu!=(\nu-1)!\nu$\Big{]}}.$

From (6) in the proof of Lemma 2.4 we know that

$\sum_{p^{2}\leq N}\frac{1}{p\ln\frac{N}{p}}<\frac{\ln\ln N+B_{1}+B_{2}}{\ln
N.}$

Now we prove a trivial result. Suppose $p^{s+1}\leq N$. Then we have

$\ln\frac{N}{p^{s}}\geq\frac{\ln N}{s+1}.$

This is straightforward - since $p^{s+1}\leq N$, the least possible value that
$N$ can take is $p^{s+1}$. Therefore

$\frac{N}{p^{s}}\geq p^{s+1}p^{-s}=p.$

But $p\leq N^{\frac{1}{s+1}}$, so $\frac{N}{p^{s}}\geq N^{\frac{1}{s+1}}$.
Taking logarithms of both sides and noticing that $\ln$ is monotone increasing
yields the result.

For $s\geq 2$ observe that the preceding result implies that

$\frac{1}{\ln\frac{N}{p^{s}}}\leq\frac{s+1}{\ln N},$

and so it follows that

(9) $\frac{1}{p^{s}\ln\frac{N}{p^{s}}}\leq\frac{s+1}{p^{s}\ln N}.$

This gives us the following inequalities:

$\displaystyle\sum_{p^{s+1}\leq N}\frac{1}{p^{s}\ln\frac{N}{p^{s}}}$
$\displaystyle\leq\sum_{p^{s+1}\leq N}\frac{s+1}{p^{s}\ln N}$ [summing over
(9)] $\displaystyle=\sum_{p^{s+1}\leq N}\frac{s+1}{\ln N}\sum_{p^{s+1}\leq
N}\frac{1}{p^{s}}$ [breaking up the sum into two products]
$\displaystyle=\frac{s+1}{\ln N}\sum_{p^{s+1}\leq N}\frac{1}{p^{s}}$
$\displaystyle\text{\Big{[}factoring out $\frac{s+1}{\ln N}$\Big{]}}.$

Upon taking the sum over $s$ we have:

$\displaystyle\sum_{s}\sum_{p^{s+1}\leq N}\frac{1}{p^{s}\ln\frac{N}{p^{s}}}$
$\displaystyle<\sum_{s}\frac{s+1}{\ln
N}\left(\frac{1}{2^{s}}+\frac{1}{3^{s}}+\frac{1}{5^{s}}+\cdots\right)$
$\displaystyle<\frac{\ln\ln N+B_{1}+B_{2}+B_{3}}{\ln N}$
$\displaystyle<\frac{\ln\ln N+C_{2}}{\ln N},$

since $B_{1}+B_{2}+B_{3}<C_{2}$. This proves the result. ∎

In a similar way to in Lemma 2.4, what this tells us is that the number of
positive integers with many prime factors is bounded quite sharply, and we
shall use this result to extract a more meaningful bound.

The proof of the theorem we derived in the previous section follows
immediately from this observation - there are no changes in the proof.

###### Proof of Theorem 2.12.

See proof of Theorem 2.1 \- the proofs are identical. ∎

To conclude this section, we have shown that for almost all positive integers
$n$, the number of distinct prime factors is $\ln\ln n$, or at least very
close to this.

######  Theorem 2.16.

The result of Theorem 2.12 holds when $\omega$ is replaced by $\Omega$.

The proof of this result follows closely the methods used in the previous two
sections - we shall omit it to save space for the probabalistic approaches in
the next chapters.

##  3\. Probabalistic arguments

###  3.1. General remarks

We wish to use probabalistic arguments to obtain number theoretic results. To
do so, we will compute the mean and variance of $\Omega(n)$ considered as a
function of $n$. To do so, we induce a random variable. Fix $N\in\mathbb{N}$.
Then for $n$ uniformly distributed in $[1,N]$ we have $\omega(n)$ and
$\Omega(n)$ are induced random variables, which we intend to study now.

######  Lemma 3.1.

For $n$ chosen uniformly in $[1,N]$ we have the following:

$\mathbb{E}[\omega(n)]=O(\ln\ln N).$

This is easy to see; the probability of a prime $p$ dividing $n$ is
$\frac{1}{p}$, so the mean of $\omega(n)$ will be the result of $\sum_{p\leq
n}\frac{1}{p}$, which is $O(\ln\ln N)$ by [8].

######  Theorem 3.2 (Turan).

Taking the limit $N\to\infty$ we have that for $n\in[1,N]$ we have

$\mathrm{Var}(\Omega(n))=O(\ln\ln N),$

The proof of this result is due to Turan [12]. Observe that Theorem 3.2
actually implies that of Hardy-Ramanujan. This is explained further in [4].

###  3.2. The Erdős-Kac Theorem

Over two decades after Hardy and Ramanujan published [5], Erdős and Kac [3]
provided a probabalistic proof which substantially strengthens our original
result:

######  Theorem 3.3 (Erdős-Kac).

Given $k\in\mathbb{R}$ we have the following:

$\frac{1}{N}\sum_{\begin{subarray}{c}n\leq N\\\ \omega(n)-\ln\ln N\leq
k\sqrt{\ln\ln
N}\end{subarray}}1\longrightarrow\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{k}\exp\left(-\frac{t^{2}}{2}\right),$

taking the limit as $N\to\infty$.

Let us consider what this actually means. The RHS is clearly the density
function for the standard normal distribution; in other words, we have the
more compact version of the Erdős-Kac theorem:

######  Theorem 3.4 (Erdős-Kac (version 2)).

Again taking the limit $N\to\infty$ and taking $n\in[1,N]$ uniformly we have

$\frac{\omega(n)-\ln\ln n}{\sqrt{\ln\ln n}}\longrightarrow N(0,1),$

where the LHS converges in distribution to the standard normal variable.

One can see that this clearly implies the Hardy-Ramanujan theorem; in fact, it
is a stronger result. This is because it gives us more information - it gives
us a distributional result which Hardy-Ramanujan do not. For the sake of
completeness we shall provide a proof that Erdős-Kac’s result implies the
Hardy-Ramanujan theorem.

######  Proposition 3.5.

The Erdős-Kac Theorem implies Theorem 1.2.

###### Proof.

We need to prove that

$\lim_{N\to\infty}\mathbb{P}(|\omega(n)-\ln\ln n|\geq\kappa\ln\ln n)=0.$

If we prove this, we will automatically recover Theorem 1.2, as if the limit
of the probability is zero, then for sufficiently large $N$, the probability
will be less than $\varepsilon$, so at most $\varepsilon N$ integers will
satisfy it. In what follows, we let $Z$ be standard normally distributed, that
is, $Z\sim N(0,1)$.

We have

$\displaystyle\lim_{N\to\infty}\mathbb{P}(|\omega(n)-\ln\ln n|\geq\kappa\ln\ln
n)$
$\displaystyle=\lim_{N\to\infty}\mathbb{P}\left(\left|\frac{\omega(n)-\ln\ln
n}{\sqrt{\ln\ln n}}\right|\geq\kappa\sqrt{\ln\ln n}\right)$ [dividing by
$\sqrt{\ln\ln n}\neq 0$]
$\displaystyle=\lim_{N\to\infty}\mathbb{P}(|Z|\geq\kappa\sqrt{\ln\ln n})$
[since $\frac{\omega(n)-\ln\ln n}{\sqrt{\ln\ln n}}\sim N(0,1)$]
$\displaystyle=1-\lim_{N\to\infty}\mathbb{P}(|Z|<\kappa\sqrt{\ln\ln n})$
[complementary probabilities sum to $1$]
$\displaystyle=1-\lim_{N\to\infty}\mathbb{P}(-\kappa\sqrt{\ln\ln
n}<Z<\kappa\sqrt{\ln\ln n})$ [using the fact that $|N|<a$ is the same as
$-a<N<a$] $\displaystyle=1-1$ [taking $n$ large will cover all probabilities]
$\displaystyle=0,$

This completes the proof.
∎

We now sketch very vaguely a proof of the Erdős-Kac theorem, following
Granville and Soundarajan [4].

###### Sketch proof of Erdős-Kac.

Recall that (in probabalistic terms) the $k$th moment is given by
$\mathbb{E}[X^{k}]=\frac{1}{n}\sum_{i=1}^{n}X_{i}^{k}$, where the $X_{i}$ are
realisations of the random variable $X$.

The Normal distribution is characterised by its moments (see [1]). This means
that if a distribution has identical moments to a $N(\mu,\sigma^{2})$, then
our distribution must follow a $N(\mu,\sigma^{2})$ distribution. Note that not
all distributions are characterised by their moments. For example, a Lognormal
random variable is not characterised by its moments.

Therefore, our goal is to prove that for each $k$, upon defining

(10) $X:=\frac{\omega(n)-\ln\ln n}{\sqrt{\ln\ln n}},$

if we can demonstrate that $\mathbb{E}[X^{k}]$ is the same as that for a
$N(0,1)$ random variable, we will be done. Note that we are not actually
proving each moment matches; instead, we are showing that _asymptotically_
they match - equally, up to some small error term, the moments match.

We have, according to Lemma 3.1 and Theorem 3.2, that the first two moments
match. However, we need to determine each moment as opposed to just two. By
symmetry of the probability density function, the odd moments of the standard
normal variable are all zero. For even $k$ we have

$\mathbb{E}[Z^{k}]=\frac{k!}{2^{k/2}(k/2)!},$

where $Z\sim N(0,1)$. If we can demonstrate that these moments match that of
(10), the Erdős-Kac theorem will follow, using the characterisation of
$N(0,1)$ by its moments.

According to Granville & Soundarajan’s paper [4], we have the following:

######  Lemma 3.6.

Given any fixed odd $k\in\mathbb{N}$ we have the following:

$\frac{1}{N}\sum_{n\leq N}(\omega(n)-\ln\ln N)^{k}=O(\ln\ln
N)^{k/2}\frac{1}{\sqrt{\ln\ln N}}.$

######  Lemma 3.7.

Given any fixed even $k\in\mathbb{N}$ we have the following:

$\frac{1}{N}\sum_{n\leq N}(\omega(n)-\ln\ln
N)^{k}=\frac{k!}{2^{k/2}(k/2)!}(\ln\ln
N)^{k/2}\left(1+O\left(\frac{1}{\sqrt{\ln\ln N}}\right)\right).$

According to Lemma 3.6, upon taking the limit $N\to\infty$ we have that

$\frac{1}{N}\sum_{n\leq N}(\omega(n)-\ln\ln N)^{k}\sim(\ln\ln N)^{-k/2},$

which in turn means that asymptotically the odd moments tend to zero.
Therefore we have that the odd moments match those of the Standard Normal
Distribution, as we wanted.

It now remains to demonstrate that the even moments match the standard normal
distribution too, and this is roughly the same approach as for the odd
moments.

In Lemma 3.7, take the limit as $N\to\infty$. Then we have

(11) $\lim_{N\to\infty}\left(\frac{1}{N}\sum_{n\leq N}(\omega(n)-\ln\ln
N)\right)^{k}=\frac{k!}{2^{k/2}(k/2)!},$

which is precisely the $k$th moment for even $k$ of a standard normal
distribution.

Since the authors of [4] have proved all moments of $\frac{\omega(n)-\ln\ln
n}{\sqrt{\ln\ln n}}$ match those of a standard normal distribution, the Erdős-
Kac theorem follows as a result. ∎

The proof by Granville and Soundarajan was entirely different to that of Hardy
and Ramanujan, and also shorter. However, Diaconis [2] proved the Hardy-
Ramanujan theorem using Chebyshev’s inequality. This is a remarkable result -
in Erdős-Kac, we needed to determine each moment of our random variable $X$ as
we defined in (10). However, Chebyshev’s inequality requires only the first
two moments.

We will now demonstrate the heuristic argument using Chebyshev to show what is
going on here - this will be crucial in our refined proof in the final
chapter.

###  3.3. Diaconis’ proof using Chebyshev’s inequality

In this section we will provide a brief exposition of Diaconis’ argument [2].
We begin by recalling Chebyshev’s inequality:

######  Theorem 3.8 (Chebyshev).

Let $Y$ be a random variable with $\mathbb{E}[Y]=\mu$ and
$\mathrm{Var}(Y)=\sigma^{2}$. Then, given any $k\geq 0$ we have the following:

$\mathbb{P}(|Y-\mu|\geq k\sigma)\leq\frac{1}{k^{2}}.$

Now, we have that $\omega(n)$, treated as a random variable, has
$\mathbb{E}[\omega(n)]=\ln\ln n=\mathrm{Var}(\omega(n))$. Therefore, applying
Chebyshev’s inequality with $Y=\omega(n)$ with $n$ chosen uniformly between
$1$ and $N$, we have

$\mathbb{P}(|\omega(n)-\ln\ln n|\geq\kappa\sqrt{\ln\ln
n})\leq\frac{1}{\kappa^{2}}.$

Dividing through by the nonzero quantity $\sqrt{\ln\ln n}$ we obtain

$\mathbb{P}\left(\left|\frac{\omega(n)-\ln\ln n}{\sqrt{\ln\ln
n}}\right|\geq\kappa\right)\leq\frac{1}{\kappa^{2}}.$

Note that this does not give us any distributional results - Chebyshev’s
inequality applies to any random variable with known mean and variance.
However, _morally_ what is going on here is that the probability that

$\left|\frac{\omega(n)-\ln\ln n}{\sqrt{\ln\ln n}}\right|$

is large is exceedingly small.

##  4\. A novel approach

###  4.1. Background

Whilst reading various literature on the probabalistic proof of Hardy-
Ramanujan, the one that stood out was undoubtedly Diaconis’ proof using
Chebyshev’s inequality. During one of my meetings with my supervisor, whilst
we were discussing this, I wondered what would happen if we had a function
with more than two moments. After all, Chebyshev _always_ holds and all we
need is two moments. Moreover, Chebyshev’s inequality is _sharp_ ; that is, we
cannot improve the bound of $\frac{1}{k^{2}}$ with just the knowledge of mean
and variance of our random variable.

This raised a natural question - Chebyshev doesn’t really require any results
about the distribution we are working with; would we gain any further
information if we knew more about our distribution? For example, can we
squeeze more information out from Chebyshev’s inequality if we had, say, three
or more moments?

This led to our novel approach, whereby we use information about $\omega(n)$
which we can compute from first principles (i.e. the first few moments, which
doesn’t require us to know Granville and Soundarajan’s result), in order to
extract a refined bound on the Chebyshev inequality.

###  4.2. Outline of Proof

In Section 3.3, we saw the basic argument for the use of Chebyshev’s
inequality in proving Hardy-Ramanujan. We will use a similar but modified
result where we _know_ higher moments.

We have the following corollary of Chebyshev’s inequality (see [9]):

######  Lemma 4.1.

$\mathbb{P}\left(|X-\mathbb{E}[X]|>\lambda\right)\leq\frac{\mathbb{E}(|X-\mathbb{E}[X]|^{k})}{\lambda^{k}}$

We now do a sanity check, to demonstrate that the result holds in the case
$k=2$, as in the original Chebyshev inequality. We have the following chain of
inequalities:

$\displaystyle\mathbb{P}(|X-\mu|>k\sigma)$
$\displaystyle\leq\frac{\mathbb{E}[|X-\mu|^{2}]}{(k\sigma)^{2}}$ [due to
(4.1)]
$\displaystyle=\frac{\mathbb{E}[X^{2}]-2\mu\mathbb{E}[X]+\mu^{2}}{k^{2}\sigma^{2}}$
[expanding the brackets]
$\displaystyle=\frac{\mathbb{E}[X^{2}]-\mu^{2}}{k^{2}\sigma^{2}}$ [since
$\mu=\mathbb{E}[X]$]
$\displaystyle=\frac{\mathbb{E}[X^{2}]-\mathbb{E}[X]^{2}}{k^{2}\sigma^{2}}$
[since $\mu=\mathbb{E}[X]$] $\displaystyle=\frac{\sigma^{2}}{k^{2}\sigma^{2}}$
[since $\mathrm{Var}(X)=\mathbb{E}[X^{2}]-(\mathbb{E}[X])^{2}$]
$\displaystyle=\frac{1}{k^{2}}$ [dividing by $\sigma^{2}\neq 0$]

which is what we would expect as per Lemma 4.1.

Now, using Lemma 3.7, we obtain an analogous result in the case we have a
$r$th moment for even $r$.

We start with the inequality

$\mathbb{P}(|X-\mu|>\kappa\sigma)\leq\frac{\mathbb{E}[|X-\mu|^{r}]}{(\kappa\sigma)^{r}},$

and manipulate the RHS of this expression. Observe that for even $r$ we have
the following:

$\mathbb{E}[|\omega(n)-\ln\ln n|^{r}]=\frac{r!}{2^{r/2}(r/2)!}(\ln\ln
N)^{r/2},$

by applying Lemma 3.7. Then, given any $\varepsilon>0$, we obtain the
following:

$\displaystyle\mathbb{P}(|\omega(n)-\ln\ln n|>\kappa\sqrt{\ln\ln n})$
$\displaystyle\leq(1+\varepsilon)\frac{r!(\ln\ln
N)^{r/2}}{2^{r/2}(r/2)!\kappa^{r}\sigma^{r}}$
$\displaystyle=(1+\varepsilon)\frac{r!}{2^{r/2}(r/2)!\kappa^{r}},$

since the $(\ln\ln N)^{r/2}$ cancels with the $\sigma^{r}$ term; this happens
because $\sigma^{2}=\ln\ln N$.

This gives us the following Lemma which we use later on in a bounding
argument:

######  Lemma 4.2.

For sufficiently large $N$ and even $r$ we have, for any $\varepsilon>0$

(12) $\mathbb{P}\left(\left|\frac{\omega(n)-\ln\ln N}{\sqrt{\ln\ln
N}}\right|>A\right)<(1+\varepsilon)\frac{r!}{2^{r/2}(r/2)!A^{r}}.$

This follows from a slightly more rigorous treatment of the above results, but
nonetheless does not require further exposition to see why this must be the
case - simply substitute in the result Lemma 3.7.

Our goal is to demonstrate that _more moments are better_. In other words, we
aim to show that the bound provided by the inequality in Lemma 4.1 is tighter
when we have more knowledge of the moments.

Now, the RHS will initially decrease as we increase $r$, but eventually the
$r!$ term will dominate. As such, we can optimise our bound by choosing the
_correct_ number of moments. Therefore we will need a better method to
guarantee an optimal bound.

We demonstrate the power of our result in Lemma 4.2. Consider the probability

$\mathbb{P}(|\omega(n)-\log\log n|>\kappa\sqrt{\log\log n}).$

Let $r$ denote the number of moments we know, which we will utilise in Lemma
4.1. The following table comprises numerical results which demonstrate how
significantly better the bound in Lemma 4.2 is than the bound given by
Chebyshev’s inequality:

Comparison of bounds given by Chebyshev and our bound
---
Value of $r$ |  Value of $\kappa$ |  Chebyshev bound |  Bound from 4.1
$4$ |  $2$ |  $1/4$ |  $3/16$
$4$ |  $10$ |  $1/100$ |  $3/10000$
$6$ |  $2$ |  $1/4$ |  $15/64$
$6$ |  $10$ |  $1/100$ |  $3/200000$
$8$ |  $10$ |  $1/100$ |  $315/781253$
$8$ |  $20$ |  $1/400$ |  $41/1000000000$

What we can see is that the bound given by Theorem 4.2 is most useful when we
consider large values of $\kappa$, which is the probability at a _tail_. That
is, the bound we obtain is best when considering extreme probabilities.

Furthermore, given a value of $A$, there will be some $r$ for which we attain
a best bound. To think about why this must be the case, for large $r$ we have
the RHS of this expression approaching $\infty$ (since the factorial in the
numerator will dominate the expression).

Now we claim that in order to attain the best possible bound, we should choose
$r$ to be as close as possible to $A^{2}$. To see why this must be the case,
consider the ratio of the bounds in the cases given by $r$ moments and $r-2$
moments:

$\displaystyle\frac{r!}{2^{r/2}(r/2)!A^{r}}\bigg{/}\frac{(r-2)!}{(2^{(r-2)/2})(\frac{r-2}{2})!A^{r-2}}$
$\displaystyle=\frac{r!}{(r-2)!}\frac{2^{(r-2)/2}}{2^{r/2}}\frac{((r-2)/2)!}{(r/2)!}\frac{A^{r-2}}{A^{r}}$
$\displaystyle=\frac{r(r-1)}{2}\frac{A^{r-2}}{A^{r}}\frac{((r-2)/2)!}{(r/2)!}$
$\displaystyle=\frac{r(r-1)}{2A^{2}(r/2)}$
$\displaystyle=\frac{r(r-1)}{rA^{2}}$ $\displaystyle=\frac{r-1}{A^{2}}.$

So, by following some simple algebraic manipulation, we have that the ratio is
$\frac{r-1}{A^{2}}$. This is less than $1$ if, and only if, $r<1+A^{2}$.
Therefore if we choose $r$ to be the largest even integer such that
$r<1+A^{2}$ we will have the best bound.

Now assume $A$ is large. Upon applying Stirling’s approximation, we have:

$\displaystyle\frac{r!}{2^{r/2}(r/2)!}$
$\displaystyle\approx\frac{\sqrt{r}r^{r}e^{-r}}{2^{r/2}\sqrt{r/2}(r/2)^{r/2}e^{-r/2}}$
$\displaystyle=\sqrt{2}r^{r/2}e^{-r/2}.$

If we now set $r=A^{2}$ (we do this as for large $A$ we have $A^{2}\approx
A^{2}+1$) we obtain

$\displaystyle\frac{r!}{(r/2)!2^{r/2}A^{r}}$
$\displaystyle=\frac{\sqrt{2}A^{A^{2}}e^{-A^{2}/2}}{A^{A^{2}}}$
$\displaystyle=\sqrt{2}e^{-\frac{A^{2}}{2}}.$

Notice that this is substantially stronger than the result of [2].

##  5\. An original proof of the Hardy-Ramanujan theorem

###  5.1. A discussion of the proof

Recall the Lemmas of Granville and Soundarajan [4] (Lemma 3.6 and Lemma 3.7)
whereby the authors determine an order relation for the moments. The crux of
their proof of Erdős-Kac’s theorem was in proving each moment matched each
moment of the standard normal distribution. Suppose we only had one moment,
say, a fourth moment. We demonstrate that this is in fact sufficient to prove
Theorem 1.2, thus bypassing the requirement of knowing each moment. This
concise and straightforward proof will be submitted to a research journal in
the near future.

Note that our result requires an even moment - we will explain throughout the
course of the proof, but simply having an odd moment will not suffice with
this method. The statement of our result is as follows:

######  Theorem 5.1.

Let $k\in\mathbb{N}$ be even. Then the assertion as in Lemma 3.6, that

$\sum_{n\leq N}(\omega(n)-\ln\ln N)^{k}=O(x(\ln\ln N)^{k/2}),$

automatically implies Theorem 1.2.

###  5.2. The proof

We will prove our theorem in the form of Theorem 1.3. Firstly recall Lemma 3.7
for the case $k$ even:

(13) $\sum_{n\leq N}(\omega(n)-\ln\ln N)^{k}=O(N(\ln\ln N)^{k/2}).$

Suppose for the sake of contradiction that the number of integers $n\leq N$
not satisfying Theorem 1.3 is _not_ $o(N)$. On this assumption, there exists
some $M\in\mathbb{R}$ such that more than $MN$ of the possible integers $n\leq
N$ (for arbitrarily large $N$) satisfy the following inequality:

(14) $|\omega(n)-\ln\ln N|\geq(\ln\ln N)^{\frac{1}{2}+\delta}.$

Then, in the result of (13), we obtain the following string of inequalities:

$\displaystyle\sum_{n\leq N}(\omega(n)-\ln\ln N)^{k}$ $\displaystyle\geq
MN\left((\ln\ln N)^{\frac{1}{2}+\delta}\right)^{k}$ [using (13) and (14)]
$\displaystyle=MN(\ln\ln N)^{\frac{k+2k\delta}{2}}$ [using laws of indices]
$\displaystyle>MN(\ln\ln N)^{k/2}$ [for sufficiently large $N$]

Consequently (13) is not satisfied, which is a contradiction. Theorem 1.3
follows. Note that such an approach will not work in the case where $k$ is
odd. This is because the moments as in Lemma 3.7 could have large magnitude
for many $n$, but the sum could be small since some terms could be negative
and some positive. Of course, there is no danger of this in the case that $k$
is even since then we have a sum of nonnegative terms.

## Acknowledgements

I wish to extend my gratitude to Chris Hughes for his support and guidance
throughout the course of my undergraduate dissertation. I was always inspired
by my mathematical discussions with him, and this work would not have been
possible without his guidance.

## References

  * [1] Patrick Billingsley  “Probability and measure” Anniversary edition [of MR1324786], With a foreword by Steve Lalley and a brief biography of Billingsley by Steve Koppes, Wiley Series in Probability and Statistics  John Wiley & Sons, Inc., Hoboken, NJ, 2012, pp. xviii+624
  * [2] Persi Diaconis  “G. H. Hardy and probability???”  In _Bull. London Math. Soc._ 34.4, 2002, pp. 385–402  DOI: 10.1112/S002460930200111X
  * [3] P. Erdös and M. Kac  “The Gaussian law of errors in the theory of additive number theoretic functions”  In _Amer. J. Math._ 62, 1940, pp. 738–742  DOI: 10.2307/2371483
  * [4] Andrew Granville and K. Soundararajan  “Sieving and the Erdős-Kac theorem”  In _Equidistribution in number theory, an introduction_ 237, NATO Sci. Ser. II Math. Phys. Chem.  Springer, Dordrecht, 2007, pp. 15–27  DOI: 10.1007/978-1-4020-5404-4“˙2
  * [5] G.. Hardy and S. Ramanujan  “The normal number of prime factors of a number $n$ [Quart. J. Math. 48 (1917), 76–92]”  In _Collected papers of Srinivasa Ramanujan_ AMS Chelsea Publ., Providence, RI, 2000, pp. 262–275
  * [6] G.. Hardy and E.. Wright  “An introduction to the theory of numbers” Revised by D. R. Heath-Brown and J. H. Silverman, With a foreword by Andrew Wiles  Oxford University Press, Oxford, 2008, pp. xxii+621
  * [7] Edmund Landau  “Handbuch der Lehre von der Verteilung der Primzahlen. 2 Bände” 2d ed, With an appendix by Paul T. Bateman  Chelsea Publishing Co., New York, 1953, pp. xviii+pp. 1–564ix+pp. 565–1001
  * [8] Franz Mertens  “Ein Beitrag zur analytischen Zahlentheorie”  In _J. Reine Angew. Math._ 78, 1874, pp. 46–62  DOI: 10.1515/crll.1874.78.46
  * [9] Michael Mitzenmacher and Eli Upfal  “Probability and computing” Randomized algorithms and probabilistic analysis  Cambridge University Press, Cambridge, 2005, pp. xvi+352  DOI: 10.1017/CBO9780511813603
  * [10] M. Murty and V. Murty  “A variant of the Hardy-Ramanujan theorem”  In _Hardy-Ramanujan J._ 44, 2021, pp. 32–40  DOI: 10.3836/tjm/1502179309
  * [11] József Sándor, Dragoslav S. Mitrinović and Borislav Crstici  “Handbook of number theory. I” Second printing of the 1996 original  Springer, Dordrecht, 2006, pp. xxvi+622
  * [12] Paul Turán  “On a Theorem of Hardy and Ramanujan”  In _J. London Math. Soc._ 9.4, 1934, pp. 274–276  DOI: 10.1112/jlms/s1-9.4.274
  * [13] G. & A. Weingartner  “An Erdős-Kac theorem for integers with dense divisors”  Arxiv, 2022


11institutetext: Institute of Computing, Federal University of Bahia (UFBA),
Salvador - BA, Brazil 22institutetext: State University of Feira de Santana
(UEFS), Feira de Santana - BA, Brazil
22email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
# Developers’ perception on the severity of test smells: an empirical study

Denivan Campos 11 Larissa Rocha 1122 Ivan Machado 11

###### Abstract

Unit testing is an essential component of the software development life-cycle.
A developer could easily and quickly catch and fix software faults introduced
in the source code by creating and running unit tests. Despite their
importance, unit tests are subject to bad design or implementation decisions,
the so-called test smells. These might decrease software systems’ quality from
various aspects, making it harder to understand, more complex to maintain, and
more prone to errors and bugs. Many studies discuss the likely effects of test
smells on test code. However, there is a lack of studies that capture
developers’ perceptions of such issues. This study empirically analyzes how
developers perceive the severity of test smells in the test code they develop.
Severity refers to the degree to how a test smell may negatively impact the
test code. We selected six open-source software projects from GitHub and
interviewed their developers to understand whether and how the test smells
affected the test code. Although most of the interviewed developers considered
the test smells as having a low severity to their code, they indicated that
test smells might negatively impact the project, particularly in test code
maintainability and evolution. Also, detecting and removing test smells from
the test code may be positive for the project.

###### Keywords:

Software Testing Test Smells Empirical Software Engineering

##  1 Introduction

Software testing is an essential strategy to ensure the quality of software
systems. Creating and executing test code encompass activities that require a
lot of effort and cost, especially when manually developed [13]. Nonetheless,
the need to find defects early in the development cycle has increased the
interest in employing unit testing practices [3].

In recent years, test code has become persistent and needs to evolve along
with the production code to be more effective [3]. For example, at Google, the
amount of code and the size of the test pool grows linearly to maintain the
software quality. Google runs over 2 Million Lines of Code (LOC) and 150
million tests per day [8]. Keeping up this whole process of evolving and
maintaining these test cases might be a time-consuming task. In addition, as
the amount of tests grows, the effort invested in maintaining the tests can
make it a costly activity. On the other side, creating and running unit tests
can quickly capture and support fixing software failures introduced in the
source code [13].

Despite their importance, unit tests might be subject to inappropriate design
or poor implementation decisions. Such issues may lead to the insertion of
smells in test code, the so-called test smells [2]. The presence of test
smells makes test code more difficult to comprehend, more complex to maintain,
and prone to errors [13, 18]. Software developers and testers are usually
unaware of the concept of test smells [17].

Software projects commonly develop extensive sets of test cases to assure the
quality of developed systems [16]. To this effect, automated support for
creating test cases has gained momentum, and the software testing community
has released many support tools lately [6, 11]. In addition, many tools are
providing automated strategies to prevent, detect, and refactor test smells,
as discussed by Aljedaani et al. [1].

Aligned with the need to understand whether and how harmful test smells are to
a software project, many researchers have empirically evaluated the likely
effects of test smells [7, 9, 17, 19]. In a recent study, Garousi et al. [7]
enlisted the main negative consequences of the presence of test smell in test
code: smells reduce tests changeability, stability, readability, and
maintainability. Silva-Junior et al. [17] surveyed software testing
professionals to understand and analyze how often they insert test smells into
the test code and the reasons behind it. The authors showed that,
independently of the professional experience, they are prone to insert test
smells in their test code, even when using standard practices.

Although some studies claim that test smells might hinder code maintenance,
there is still a lack of studies that capture the developers’ perception of
that issues. In this context, this study investigates how developers recognize
test smells. We aim to empirically analyze how developers perceive the
severity of test smells in the test code they develop. We refer severity to
the degree to which a given test smell can negatively impact the test code. In
addition to severity, we were also interested in investigating how the test
code would behave after removing the test smells.

We conducted an empirical study split into three phases. In the first, we
selected six open-source software projects from GitHub. The second phase
consisted of interviews with the developers of the selected projects to
investigate their perception of test smells. During the interviews, we
introduced the definitions of each test smell and asked the developers whether
they could refactor them. The third step was the analysis of the gathered
data. To support this empirical study, we used the JNose Test [20] tool to
identify the test smells in the selected projects.

In summary, our study brings the following contributions:

  * •

an empirical study showing that most developers perceive test smells as low or
middle in severity for their software projects;

  * •

initial insights of the consequences of the test smells after refactoring.

##  2 Test Smells

Test smells are sub-optimal designs in the unit test code that the developer
chose when implementing test cases [1, 7, 12, 15]. Deursen et al. [5] presents
a catalog of 11 test smells and proposed refactoring strategies to remove them
from the test code. Peruma et al. [15] extended that catalog and included
another 12 new test smells inspired by bad practices in unit test-based
programming. Other researchers identified other test smells types and analyzed
the effects of various types on production and test code [9, 10, 13, 18, 22].

In this study, we analyzed eight types of test smells. We selected the most
frequent ones in the test suites of the projects used in the study, which we
briefly introduce next:

  * •

Assertion Roulette (AR): It occurs when a test method contains more than one
assertion statement without an explanation or parameter in the assertion
method [5];

  * •

Eager Test (ET): When a test method contains multiple calls to multiple
production methods [5];

  * •

Empty Test (EpT): It occurs when a test method does not contain executable
instructions [15];

  * •

Lazy test (LT): It occurs when multiple test methods call the same production
method [5];

  * •

Redundant Assertion (RA): When the test methods contain assertion statements
that are always true or always false [15];

  * •

Constructor Initialization (CI): It occurs when a test method implements a
constructor. It is recommended that all the fields be initialized inside the
setUp() method [15];

  * •

Unknown Test (UT): When a test method does not contain assertions. An
assertion statement describes an expected condition for a test method [15];

  * •

Sensitive Equality (SE): It occurs when the test method makes an equality
checking using the toString method [5].

###  2.1 JNose Test Tool

Aljedaani et al. [1] identified 22 tools for test smell detection. Most of the
tools support the JUnit framework and Java programming language. For our
study, we applied the JNose Test [20], as it elaborates on the state-of-the-
art tool by providing users with an easy-to-use graphical interface that
facilitates the detection of test smells. It analyzes Java projects that use
Maven and JUnit Framework111https://junit.org/.

The JNose Test222https://github.com/arieslab/jnose is a web-based application
that analyzes the quality of the test code by detecting test smells in the
Java test code [20]. JNose Test detects 21 types of test smells by following
the detection rules from the TsDetect tool [14]. The tool encompasses a set of
rules to identify and quantify the types of test smells in each test class and
supports the analysis of test smells through several project versions [20,
21]. Once started, the tool requires the user to configure the data entry to
enable and specify one of the four types of analysis mode, TestClass,
TestSmell, TestFile, and Evolution. JNose shows, as a result, the amount of
LOC, methods, types, and amount of test smells in each test class of the
project. With such a tool, it is possible to verify the test quality from an
evolutionary perspective. It enables capturing metrics, and the occurrence of
test smells through several project versions.

##  3 Research Methodology

We defined two research questions in this study:

  1. RQ1.

How do software developers consider the severity of test smells and their
effects on test code quality? We aim to analyze how the software developers
perceive the impact of the presence of test smells in test code, in terms of
quality attributes such as comprehensibility and maintainability.

  2. RQ2.

What is the behavior of test code after refactoring test smells? We aim to
analyze the changes in the test code after refactoring test smells.

###  3.1 Study Design

The design employed in this study consists of three phases: Repository
Creation, Interview, and Data Analysis. Each phase comprises a set of steps,
as Figure 1 shows.

Figure 1: Empirical Study Design

Phase 1 - Repository Creation. This phase comprised the following steps:

  * •

Step 1. Filtering by JNose Test limitations. In this step, we used a dataset
composed of 21,482 projects available in public GitHub repositories. First, we
manually filtered each of the repositories with the following criteria:
projects written in Java, composed of at least two stars, with Issues, and
test cases created with the JUnit framework (versions 4 or 5). The choice for
projects written in Java and tests with JUnit was due to the JNose Test
limitations. As a result, we obtained a sample with 4,452 public repositories.

  * •

Step 2. Filtering repositories by ownership. In this step, we collected the
URLs of the 4,452 projects and manually opened each one. We filtered the
projects that contained e-mails with Brazilian developers. Then, we ran each
of these projects in the JNose Test to validate them. As a result, we obtained
a sample with 90 public repositories.

  * •

Step 3. Filtering repositories by responses. In this phase, we sent out
e-mails to the 90 selected projects. We received six positive replies, which
composed our dataset.

Phase 2 - Interview. The interview considered each of the developers
individually. It comprised the following steps:

  * •

Step 4. Executing the JNose Test. The first step was to execute the selected
projects in the JNose Test [20] to gather evidence about the test smells
identified in each project. Based on the yielded results (e.g., number of test
smells present in each project), we created the protocol for the interviews.

  * •

Step 4.1. Carrying our the interviews. Next, we proceeded with the actual
interviews. First, we carried out the characterization of the participants,
which included gathering information on their knowledge about creating and
maintaining test cases. We created an online questionnaire using Google Forms
and sent it to the interviewees. The interviews were held virtually, through
Google Meet, and guided by the protocol created after detecting test smells in
each of the projects. We presented the JNose Test to the participants during
the interviews, explained that we ran the tool in their project, and presented
the test smells we detected in each project. We also showed the most frequent
test smells.

  * •

Step 4.2. Refactoring test code. After analyzing the presence of test smells
in the test code, they had to discuss (i) whether the test smells were harmful
to their projects and (ii) whether they should refactor them. In this step,
the developers had to refactor at least one test smell. We did not impose
which test smell they must refactor. They could select any test smell they
wanted.

  * •

Step 5. Commiting test code changes. After refactoring, we asked the
developers to commit the changes to their GitHub repositories. This step was
essential to allow us to rerun the JNose Test on the refactored project.

  * •

Step 6. Rerunning the JNose Test. In this final step, we reran the project in
the JNose Test to compare the test classes before and after refactorings.

Phase 3 - Data Analysis. This phase comprises the analysis of gathered data,
as follows:

  * •

Step 7. Results. In this step, we present the results of our empirical study.
This research has an exploratory character with a qualitative analysis. For
open-ended questions that could combine multiple answers, the sum of
percentages could be greater than 100%.

##  4 Results

###  4.1 Developers’ Profiles

In this study, we interviewed the developers of each selected project. We next
describe their background.

They all hold a Bachelor’s degree (two in Computer Engineering, two in
Electrical Engineering, and two in Business Information Systems), and two out
of them have a Master’s degree. They work in different Brazilian states: three
of them work in São Paulo, and the others work in the states of Rio Grande do
Sul, Bahia, and Santa Catarina.

Regarding their professional experience with software development, three had
over ten years of professional experience, two had less than ten years, and
one developer had less than five years. Among the three developers with more
than ten years of experience, two of them also had more than ten years of
experience in software testing. Among the two developers with up to 10 years
of development experience, one also had a strong background in software
testing. The others have between one and five years of experience in software
testing.

###  4.2 Characterization of the selected projects

The six projects we analyzed vary in size - considering the number of LOC and
the number of test methods, as Table 1 shows. The largest project is the
Fatiador (the project aims to convert flat strings to Java objects and vice
versa) with 162 test methods. For the csv2Bib (the project aims to convert
.csv files into .bib or .ris files) and Our Digital Bank (the project provides
an API to support the operation of a our digital bank) projects, we could only
select two test smell types each.

Table 1: Selected projects Project | LOC* |  Test Methods |  Test Classes |  Project URL
---|---|---|---|---
Our Digital Bank | 84 | 7 | 4 | https://bit.ly/3ABZui2
csv2bib | 98 | 5 | 2 | https://bit.ly/3xlHyXf
CursoMassa | 261 | 32 | 4 | https://bit.ly/3dQEwSV
Dependency injection | 390 | 23 | 5 | https://bit.ly/36h3ff3
l2jserver | 538 | 31 | 5 | https://bit.ly/3hInK9I
Fatiador | 2,362 | 162 | 17 | https://bit.ly/36gtS3F
(*) Number of LOC in the test classes.

###  4.3 How do software developers consider the severity of test smells and
their effects on test code quality? (RQ1)

We made an attempt to analyze how the developers would perceive the impact of
the presence of test smells in test code. To accomplish that, we first
considered the degree of severity of each test smell. Table 2 summarizes the
data about the severity of the test smells identified in each project.

The AR test smell was the one which varied the most in terms of severity.
Three projects presented variations for this test smell. The project Curso
Massa (this project aims to support programming classes) presented 2 degrees
of severity for the AR test smell: low and middle; the project l2jserver (this
is a rewrite of the l2jserver project333https://bitbucket.org/l2jserver/) also
presented 2 degrees of severity: low and high; and the project Dependency
Injection (this project implements a dependency injection framework in Java
with annotations) presented 3 degrees of severity: low, middle, and high
severity.

In addition, the UT and ET test smells also presented differences in their
perceived severity, ranging from low to middle and low to high, respectively.
The remaining projects and test smells only presented a low severity degree.

Although most of the interviewees classified AR test smell as low severity, we
observed small variations. For example, the developer of the project
Dependency Injection analyzed 3 test smells of the AR type and gave a
different classification for each. For this developer, AR could impact
different her project in different ways.

Table 2: Degree of severity of test smells (TS) per project Curso Massa | l2jserver | Dep. Injection | Fatiador | csv2bib | Our Digital Bank
---|---|---|---|---|---
TS |  Severity | TS |  Severity | TS |  Severity | TS |  Severity | TS |  Severity | TS |  Severity
AR | Low | AR | Low | AR | Middle | SE | Low | AR | Low | AR | Low
AR | Low | AR | Low | AR | High | SE | Low | AR | Low | AR | Low
AR | Middle | AR | High | AR | Low | SE | Low | RA | Low | EpT | Low
ET | Low | ET | Low | ET | Low | AR | Low | RA | Low | EpT | Low
ET | Low | ET | High | ET | Low | AR | Low | CI | Low | UT | Middle
ET | Low | ET | Low | ET | Low | AR | Low | CI | Low | UT | Low
LT | Low | LT | Low | LT | Low | LT | Low | - | - | - | -
LT | Low | LT | Low | LT | Low | LT | Low | - | - | - | -
LT | Low | LT | Low | LT | Low | LT | Low | - | - | - | -
Legend: Assertion Roulette (AR), Eager Test (ET), Empty Test (EpT), Lazy test
(LT), Redundant Assertion (RA), Constructor Initialization (CI), Unknown Test
(UT), Sensitive Equality (SE).

We asked the developers about the test smells introduced in their respective
projects. We wanted to know whether that test smells affect the
maintainability of the test code. We next present the main findings.

Two projects (Fatiador and csv2bib) reported that no test smells found in the
projects would affect the maintainability or cause any impact on the test
code. Table 3 refers to the other four projects, and shows the that test
smells AR is the test smell that has the most negative impact, followed by the
test smell ET and UT.

For the Curso Massa project, the developer reported that the AR test smells
would affect maintainability over time. Conversely, for ET and LT test smells,
the developer reported that they do not affect the maintainability of the test
code. Thus, for the Curso Massa, l2jserver, Dependency Injection projects, the
developers reported that the AR test smells have a negative impact on the test
code. For the l2jserver and Dependency Injection projects, the developers
reported that the ET test smells have a negative impact on the test code.
Regarding the Our Digital Bank project, the developer considered that only the
UT negatively impacts the test code.

Although they have informed that few test smells might negatively impact the
code, 5 out of 6 developers reported that performing test smells maintenance
in the test code would improve the comprehension and evolution of the system.

Table 3: Maintainability and impact of test smells (TS) per project CursoMassa | l2jserver | Dep. Injection | Our Digital Bank
---|---|---|---
TS | M | I | TS | M | I | TS | M | I | TS | M | I
AR | Yes | Negative | AR | Yes | Negative | AR | Yes | Negative | AR | No |  -
AR | Yes | Negative | AR | Yes | Negative | AR | Yes | Negative | AR | No |  -
AR | Yes | Negative | AR | Yes | Negative | AR | Yes | Negative | EpT | No |  -
ET | No | - | ET | No | - | ET | No | - | EpT | No |  -
ET | No | - | ET | Yes | Negative | ET | Yes | Negative | UT | Yes |  Negative
ET | No | - | ET | No | - | ET | No | - | UT | Yes |  Negative
LT | No | - | LT | No | - | LT | No | - | - | - |  -
LT | No | - | LT | No | - | LT | No | - | - | - |  -
LT | No | - | LT | No | - | LT | No | - | - | - |  -
Legend: Maintainability (M), Impact (I).

###  4.4 What is the behavior of test code after refactoring test smells?
(RQ2)

Before the interviews, we ran the JNose Test for each project. The tool
generated a csv file containing the test smells identified in the projects.
During the interviews, we asked the developers to refactor some test smells
and next commit the changed test files to their GitHub repositories. Then, we
introduced each test smell and asked if they could refactor them. After the
modifications, we run the JNose Test for each project again, obtaining a new
csv file with the latest results. We next present the results for each
project.

Dependency Injection. We selected 9 test smells: 3 AR, 3 LT, and 3 ET to
present to the developer. From the three AR test smells, he decided to
refactor two out of them into two distinct test classes. He successfully
removed the test smells from the test code. From the three LT test smells, he
refactored one smell, but he did not remove it. Particularly, for the LT test
smell, the developer should place the creation of an object in the setUp()
method, but he linked the LT test smell to the use of external components, and
to refactor, he made use of mocks. Hence, he performed incorrect refactoring.
We also observed that the LOC and the number of test methods did not change
after refactoring this project. The other test classes, methods, and smells
remained without changes.

Curso Massa. We selected 9 test smells: 3 AR, 3 ET and 3 LT. The developer
decided to refactor the AR ones. From the AR test smells refactored by the
developer, only one was removed. The remaining AR were refactored but not
removed. To refactor AR smells, it is necessary to include an explanation
(message) in the first parameter of the assertion (e.g., assertTrue([message,]
boolean condition)). The developer placed the message as the last parameter
instead.

Csv2bib. We selected 6 test smells: 2 AR, 2 RA, and 2 CI. The developer
decided to refactor 1 CI test smell. After refactoring, he removed the test
smell. The CI test smell occurs when a test method implements a constructor.
It is necessary to initialize all the fields within the setUp() method.
Therefore, for that test smell, a possible refactoring would be to remove the
constructor method since it had no instructions. However, the developer
attempted to refactor the constructor method by including instructions for
setting variables used in other test methods. The JNose Test continued to
consider it a CI test smell. Since the developer modified the test class, the
LOC and the number of test methods changed. For the Run.java test class, its
LOC increased from 44 to 57, and the number of test methods risen from 2 to 5.

Fatiador. We selected 9 test smells: 3 SE, 3 AR, and 3 LT. The developer
decided to refactor 2 AR smells from 2 different test classes:
DecimalWriter.java and IntegerWriter.java. After refactoring, the developer
successfully removed the two smells. For each test class, he added 1 LOC each
(from 47 to 48 LOC and from 40 to 41 LOC), respectively. The other test smells
did not present any change.

l2jserver2. We selected 9 test smells: 3 AR, 3 ET, and 3 LT. The developer
refactored 2 AR and 1 ET. In addition, he also refactored other AR test smells
not previously selected. For example, the BitSetIDAllocator.java test class
contained 7 AR before refactoring. Although we asked him to refactor just 1 of
them, he refactored 3 occurrences of this test smell. In total, he refactored
4 AR test smells for 2 different classes (BitSetIDAllocator.java and
CharacterIDProvider.java), in which all of them were successfully removed
after refactoring.

For the ET test smell, the developer chose to refactor the smell inside the
CharacterIDProvider.java test class. Therefore, it was necessary to remove the
multiple calls to the multiple production methods. However, the developer did
not perform the necessary removals. Moreover, during the incorrect refactoring
of the ET test smell, he introduced a new type of smell in the project, the
LT, which occurs when multiple test methods call the same production method.
Thus, 2 test classes (CharacterIDProvider.java and WorldServiceImpl.java) have
changed. The first increased in 1 LOC, while the latter decreased by 1. For
the number of test methods and other test smells, there was no change in
behavior.

Our Digital Bank. We selected 6 test smells: 2 AR, 2 Ept, and 2 UT. The
developer decided to refactor 1 AR, 2 Ept, and 2 UT test smells. This project
contained 4 test classes. During the refactoring, the developer removed 2 of
them. In addition, the Validator.java test class contained 35 LOC, and 90 LOC
after refactoring. Also, the amount of test methods increased from 4 to 8. All
the 3 types of test smells, AR, UT, and Ept presented different behaviors
after refactoring: the amount of AR test smells increased from 2 to 9; the 2
UT refactored were successfully removed; and 2 Ept smells were refactored and
removed. The project had 2 Ept, 1 was removed after refactoring and the other
after removing one of the test classes. Additionally, after refactoring, the
developer added other 9 test smells, 7 IT and 2 LT. From the analyzed
projects, Our Digital Bank was the one that had the most changes in the test
code, which had decreased the number of test classes, increased the number of
test smells, and the number of LOC and test methods. After refactoring, two
test smells that did not exist before in the project were added to the test
code.

##  5 Discussion

In this study, we considered three degrees of severity (low, middle, and high)
for the test smells. We used this approach to evaluate eight test smells (AR,
Ept, UT, ET, LT, CI, SE, and RA). Five test smells were considered as low
severity for all projects: LT, SE, EpT, RA, and CI. From the set of test
smells the interviewees analyzed (48 test smells identified in their
projects), 42 were considered as low severity ($\approx$87,5%). Only 6 out 48
test smells varied in degrees of severity ($\approx$12,5%), 4 AR, 1 ET, and 1
UT, spread over 4 projects.

The AR was the one that varied the most. It was perceived by the developers in
the three severity degrees, while the ET and UT presented two different
degrees. According to the developers, the AR test smell might harm test code
maintainability. For example, one interviewee claimed that “if someone else is
going to test the system, they will not know why the test failed”. Another
interviewee also commented about the UT test smells, as follows: “UT harms my
code because it shows that it has unnecessary code, and triggers a function in
my system that is not validated as it should”.

Although most test smells received a low severity degree, the developers
reported that, in general, test smells might be indicative of problems and
harmful to the test code. On the other hand, they claimed that their systems
were relatively small, containing simple test cases, and therefore, the test
smells may not have as much impact. For larger projects, on the other hand,
they may affect more. Therefore, there is a need to conduct further studies to
know if those and other types of test smells have a degree of severity that
would be different.

According to the developers, fixing test smells may positively impact the test
code by improving the quality of the test code, making it more comprehensible,
and easing the system’s evolution. Two out of six developers also believe that
refactoring test smells can result in detecting more bugs. Although four
developers claim that refactored test smells do not contribute to the
detection of bugs.

Moreover, for one developer, the LT should not be considered a test smell at
all. They argue that it might be necessary to have more than 1 test method
testing the same production method. In this case, each test method has a
different purpose concerning the production class method. For example, for a
given test method, the focus may be to exclude something, while in another, it
may be to change something else; and the developer cannot see how to perform
the test methods in a single method.

##  6 Threats to Validity

Internal validity. We selected projects from 84 to 2,362 lines of code. This
subset of projects may not represent industrial software systems, and
therefore replications in this context are desirable. Although Deursen et al.
[5] address 21 test smells, this study considered only 8. However, we selected
the most frequent test smells in the projects, gathered from the JNose Test
results.

External validity. The number of participants we found willing to participate
in the interview represents a threat to the results. Nonetheless, we sent
emails to 90 GitHub developers, and six replied. This sample is not
representative, and we expect to replicate this study with a more significant
number of participants. Although the results cannot be generalized, they
represent a partial view of developers’ practices. We believe that this
preliminary study is significant and indicative of a trend in the area. In
addition, we intend to investigate this topic further with a more considerable
number of test smells and projects. We also present the step-by-step
methodology of this work that may allow further replications of this study.
Although most of the smells investigated in this study show low severity, we
cannot generalize to other projects and other test smells.

Construct validity. The interpretation might influence the research results.
Regarding refactoring, we did not provide information on how one developer
should refactor test smells. We only provided them with definitions and how to
detect each of them. The number of refactored test smells varied in each
project because we asked the developers if they could refactor and then which
one(s) they would like to refactor. The number of smells refactored depended
on the developer’s choice. As each project presented different test smells, it
was impossible to compare the same test smell in all projects. Since the
projects have not been in development for more than two years, this can
threaten validity. However, after refactoring, all projects were committed to
Github without any problems.

##  7 Related Work

Silva-Junior et al. [17] presented a study to understand whether test
professionals insert unintentional test smells. They surveyed experts to
analyze the frequency of use of a set of test smells during the creation and
execution of the test cases. The survey was conducted with 60 professionals
from different companies and approached 14 test smells widely studied in the
literature. According to the authors, experienced professionals introduce test
smells during their daily programming activities; they also can provide
insights for a better comprehension of how and what practices can lead to the
insertion of test smells in the test code. Our study extends such study by
providing further information about the degree of severity of test smells and
the consequence of refactorings from a developers’ perspective.

Spadini et al. [19] present an investigation into the severity classification
concerning four types of test smells and their impact on maintaining the set
of tests implemented by the developers. The authors analyzed about 1,500 open
source projects to obtain thresholds for the test smells’ severity. The
authors also integrated a tool for detecting test smells (tsDetect) in a
prototype extension (back-end) of the BetterCodeHub (BCH), a web-based code
quality analysis tool. In this study, 31 developers of the BCH project
interacted with the prototype. They had to comment on the instances of test
smells in the code base of 47 projects. As a result, the developers detected
301 test smells. In the developers’ perception, EpT, Sleepy Test, and Mystery
Guest present the highest priority as candidates for refactorings, while Empty
Test, Ignored Test, and Conditional Test Logic were considered the smells with
the most significant impact on the code maintenance. Unlike such study [19],
in this work, we investigated the severity of 8 test smells through three
degrees of severity and refactored the test smells from the developers’
perspective. To support the detection of the test smells, we used the JNose
Test [20].

Bleser et al. [4] conducted two empirical studies. The first observed the
propagation of test smells on 164 open source SCALA projects available on
GitHub. The authors implemented a tool called SOCRATES for automated test
smells detection. In the second study, they analyzed the perception and
capability of 14 developers in SCALA projects to identify test smells. As a
result, test smells had a low propagation among test classes. The most
frequently test smells were Lazy Test, Eager Test, and Assertion Roulette. Our
work differs from such a study as we analyzed developers’ perceptions
regarding the degree of severity and the consequences of refactoring smells
tests.

##  8 Concluding Remarks

This study aimed to analyze how developers perceive the severity of test
smells on test code quality. To accomplish this goal, we selected a set of
open-source software projects from Github and interviewed their developers. We
also asked them to refactor their test code to remove test smells. Hence, we
could observe, from their standpoint, how the developers could consider the
effects of such removal.

The results indicated that test smell severity is highly dependent on the
context. For example, for the same project, a given test smell may present
different degrees of severity. Regarding test smell refactoring, our initial
results pointed out that testers may not know how to refactor a test smell to
remove it. In addition, we also identified that it is likely that the removal
process could induce the inclusion of new test smells.

As future work, we plan to replicate this study with larger projects,
including a more extensive set of test smells. We also plan to investigate the
consequences of refactoring test smells further.

#### Acknowledgments.

This research was partially funded by INES 2.0; CNPq grants 465614/2014-0 and
408356/2018-9 and FAPESB grants BOL0599/2019 and JCB0060/2016.

## References

  * [1] Aljedaani, W., Peruma, A., Aljohani, A., Alotaibi, M., Mkaouer, M.W., Ouni, A., Newman, C.D., Ghallab, A., Ludi, S.: Test smell detection tools: A systematic mapping study. In: Evaluation and Assessment in Software Engineering (EASE). p. 170–180. ACM (2021)
  * [2] Bavota, G., Qusef, A., Oliveto, R., De Lucia, A., Binkley, D.: An empirical analysis of the distribution of unit test smells and their impact on software maintenance. In: 28th IEEE International Conference on Software Maintenance (ICSM) (2012)
  * [3] Breugelmans, M., Van Rompaey, B.: Testq: Exploring structural and maintenance characteristics of unit test suites. In: WASDeTT-1: 1st International Workshop on Advanced Software Development Tools and Techniques (2008)
  * [4] De Bleser, J., Di Nucci, D., De Roover, C.: Assessing diffusion and perception of test smells in scala projects. In: Proceedings of the 16th International Conference on Mining Software Repositories. p. 457–467. MSR ’19, IEEE Press (2019)
  * [5] Deursen, A., Moonen, L.M., Bergh, A., Kok, G.: Refactoring test code. Tech. rep. (2001), CWI (Centre for Mathematics and Computer Science), Amsterdam, Netherlands
  * [6] Fraser, G., Arcuri, A.: Evosuite: Automatic test suite generation for object-oriented software. In: Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering. p. 416–419. ESEC/FSE ’11, ACM, New York, NY, USA (2011)
  * [7] Garousi, V., Küçük, B.: Smells in software test code: A survey of knowledge in industry and academia. Journal of Systems and Software 138, 52–81 (2018)
  * [8] Memon, A., Gao, Z., Nguyen, B., Dhanda, S., Nickell, E., Siemborski, R., Micco, J.: Taming google-scale continuous testing. In: IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP). pp. 233–242 (2017)
  * [9] Meszaros, G.: xUnit test patterns: Refactoring test code. Addison-Wesley Professional (may 2007)
  * [10] Meszaros, G., Smith, S.M., Andrea, J.: The test automation manifesto. In: Maurer, F., Wells, D. (eds.) Extreme Programming and Agile Methods - XP/Agile Universe 2003. pp. 73–81. Springer Berlin Heidelberg, Berlin, Heidelberg (2003)
  * [11] Pacheco, C., Ernst, M.D.: Randoop: Feedback-directed random testing for java. In: 22nd ACM SIGPLAN Conference on Object-Oriented Programming Systems and Applications Companion (OOPSLA). ACM (2007)
  * [12] Palomba, F., Di Nucci, D., Panichella, A., Oliveto, R., De Lucia, A.: On the diffusion of test smells in automatically generated test code: An empirical study. In: 9th International Workshop on Search-Based Software Testing (SBST). ACM (2016)
  * [13] Peruma, A., Almalki, K., Newman, C.D., Mkaouer, M.W., Ouni, A., Palomba, F.: On the distribution of test smells in open source android applications: An exploratory study. In: Proceedings of the 29th Annual International Conference on Computer Science and Software Engineering. p. 193–202. CASCON ’19, IBM Corp., USA (2019)
  * [14] Peruma, A., Almalki, K., Newman, C.D., Mkaouer, M.W., Ouni, A., Palomba, F.: Tsdetect: An open source test smells detection tool. In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. p. 1650–1654. ESEC/FSE 2020, ACM, New York, NY, USA (2020)
  * [15] Peruma, A.S.A.: What the smell? an empirical investigation on the distribution and severity of test smells in open source android applications. Master’s thesis (2018), Rochester Institute of Technology.
  * [16] Reichhart, S., Gîrba, T., Ducasse, S.: Rule-based assessment of test quality. Journal of Object Technology 6(9), 231–251 (2007)
  * [17] Silva-Junior, N., Rocha, L., Martins, L.A., Machado, I.: A survey on test practitioners’ awareness of test smells. In: Proceedings of the XXIII Iberoamerican Conference on Software Engineering (CIbSE). pp. 462–475. Curran Associates (2020)
  * [18] Spadini, D., Palomba, F., Zaidman, A., Bruntink, M., Bacchelli, A.: On the relation of test smells to software code quality. In: IEEE International Conference on Software Maintenance and Evolution, ICSME. IEEE (2018)
  * [19] Spadini, D., Schvarcbacher, M., Oprescu, A.M., Bruntink, M., Bacchelli, A.: Investigating severity thresholds for test smells. In: 17th International Conference on Mining Software Repositories. ACM (2020)
  * [20] Virgínio, T., Martins, L., Rocha, L., Santana, R., Cruz, A., Costa, H., Machado, I.: Jnose: Java test smell detector. In: 34th Brazilian Symposium on Software Engineering (SBES). ACM (2020)
  * [21] Virgínio, T., Martins, L.A., Soares, L.R., Santana, R., Costa, H., Machado, I.: An empirical study of automatically-generated tests from the perspective of test smells. In: 34th Brazilian Symposium on Software Engineering (SBES). ACM (2020)
  * [22] Virgínio, T., Santana, R., Martins, L.A., Soares, L.R., Costa, H., Machado, I.: On the influence of test smells on test coverage. In: XXXIII Brazilian Symposium on Software Engineering (SBES). ACM (2019)