# Spectrum Monitoring and Analysis in Urban and Rural Environments at
Different Altitudes ††thanks: This research is supported in part by the NSF
award CNS-1939334 and its supplement for studying NRDZs.

Amir Hossein Fahim Raouf1, Sung Joon Maeng1, Ismail Guvenc1, Özgür Özdemir1,
and Mihail Sichitiu1  1Department of Electrical and Computer Engineering,
North Carolina State University, Raleigh, NC<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Due to the scarcity of spectrum resources, the emergence of new technologies
and ever-increasing number of wireless devices operating in the radio
frequency spectrum lead to data congestion and interference. In this work, we
study the effect of altitude on sub-6 GHz spectrum measurement results
obtained at a Helikite flying over two distinct scenarios; i.e., urban and
rural environments. Specifically, we aim at investigating the spectrum
occupancy of various long-term evolution (LTE), $5^{\text{th}}$ generation
(5G) and citizens broadband radio service (CBRS) bands utilized in the United
States for both uplink and downlink at altitudes up to 180 meters. Our results
reveal that generally the mean value of the measured power increases as the
altitude increases where the line-of-sight links with nearby base stations is
more available. SigMF-compliant spectrum measurement datasets used in this
paper covering all the bands between 100 MHz to 6 GHz are also provided.

###### Index Terms:

5G, C-Band, CBRS, helikite, LTE, spectrum monitoring, unmanned aerial vehicles
(UAV).

##  I Introduction

Wireless communication services and the emergence of new technologies have
created a huge demand for radio frequency spectrum [1]. One prominent problem
is the availability of the spectrum and the increase in interference in the
current wireless networks [2]. In addition, more aggressive frequency reuse is
gaining interest recently for achieving higher link capacity in networks
without introducing additional spectrum [3]. It is necessary to conduct
occupancy studies using spectrum sensing techniques to understand and
characterize interference problems and identify spectrum sharing
opportunities.

There are various recent examples that highlight the importance of
understanding spectrum occupancy characteristics, including non-terrestrial
scenarios, for developing effective spectrum sharing mechanisms. The launch of
$5^{\text{th}}$ generation (5G) cellular service in the United States was a
concern for the commercial airline and private aircraft communities who used
the radar altimeters of the aircraft industry. Although the assigned spectrum
band for the altimeters is between 4.2-4.4 GHz, due to their poor design the
current versions suffer from out-of-band leakage problem; i.e., they ignore
their assigned spectrum boundaries [4]. More specifically, Verizon and AT&T
have recently begun operating in the 3.7 GHz to 3.8 GHz spectrum range which
is 400 MHz away from the altimeter band. However, this gap may not be
sufficient for some aircraft to land safely. Moreover, while both Verizon and
AT&T have been delaying switching on portions of their respective 5G C-band
wireless networks until July 2023, it is expected after that day that the
whole 3.7-3.98 GHz C-band may be used for 5G transmissions [5], introducing
additional concerns. There is a similar coexistence concern for spectrum
sharing between the 5G networks to be deployed in the 3.1-3.55 GHz band in the
future and the existing airborne radars using the same spectrum. In another
recent debate, there is a concern in using terrestrial nationwide network in
the L-Band (i.e., 1-2 GHz) and its potential interference with GPS [6].

Some existing academic studies on spectrum occupancy are summarized in [7]. In
more recent works, [8] presents a framework that captures and models the
short-time spectrum occupancy to determine the existing interference for
Internet-of-things (IoT) applications. In another study [9], current state-of-
the-art artificial intelligence techniques are reviewed for channel
forecasting, spectrum sensing, signal detection, network optimization, and
security in mega-satellite networks. In [10], authors investigate and
characterize the performance of coexisting aerial radar and communication
networks for spectrum overlay and time-division multiple access by utilizing
stochastic geometry. In [11], the effect of interference coming from
coexisting ground networks on the aerial link is studied, which could be the
uplink (UL) of an aerial cell served by a drone base station. By considering a
Poisson field of ground interferers, they characterize aggregate interference
experienced by the drone.

In this paper, by post-processing the measurements from the experiments
conducted by the NSF AERPAW platform in Raleigh, NC [12] at urban and rural
environments, we analyze the spectrum occupancy in different U.S. cellular
network bands as well as the citizens broadband radio service (CBRS) band. In
addition, we study the effect of Helikite altitude on the signal strength
pattern. In Section II, we describe the data structure and the overall
information of the measurement campaign. Section III and Section IV present
the spectrum monitoring results for various sub-6 Ghz bands in the urban and
rural environments, respectively. Section V studies the time dependency of the
spectrum occupancy for the frequency bands under consideration. Finally,
Section VI highlights the conclusions of this work.

##  II Data Structure

The experiment for the urban environment was conducted by a Helikite flying up
to 140 m on August 27, 2022. For the rural environment, the Helikite flew up
to 180 m altitude on May 5, 2022. An NI USRP B205mini SDR was mounted on the
Helikite which enables executing a Python script to collect samples at the
desired center frequency with the desired sampling rate. The datasets are
SigMF compliant and include information on spectrum usage in frequency bands
ranging from 89 MHz up to 6 GHz for different altitudes [13, 14]. The data
consist of time, altitude, power and Helikite location. A detailed description
of the measurement setups can be found in [15]. Fig. 1 illustrates the height
of the Helikite during the operation time.

(a) Experiment scenario in NC State Main Campus (urban).

(b) Experiment scenario in NC State Lake Wheeler Field (rural).

Figure 1: Helikite altitude and experiment scenario for: (a) urban
environment, and (b) rural environment.

##  III Urban Spectrum Occupancy Results

In this section, we present the spectrum occupancy results for several LTE, 5G
and CBRS bands. Table I summarizes the spectrum allocations for some major
cellular providers based on the technology exploited in the United States
[16]. In this work, we investigate the aggregate in-band power for UL and
downlink (DL) spectrum of various bands.

TABLE I: Summary of LTE and 5G bands in United States.

Technology |  | Band
---
No
| Duplex
---
Mode
| Uplink Band
---
(MHz)
| DL Band
---
(MHz)
Operators
LTE | 12 | FDD | 698 - 716 | 728 - 746 |  | AT&T, Verizon,
---
T-Mobile
13 | FDD | 777 - 787 | 746 - 756 | Verizon
14 | FDD | 788 - 798 | 758 - 768 | AT&T, FirstNet
41111It is worth mentioning that T-Mobile 5G n41 also uses the same spectrum. | TDD | 2496 - 2690 | 2496 - 2690 | T-Mobile
5G | n5 | FDD | 824 - 849 | 869 - 894 | AT&T, Verizon
n71 | FDD | 663 - 698 | 617 - 652 | T-Mobile
n77 | TDD | 3700 - 3980 | 3700 - 3980 |  | AT&T, Verizon,
---
T-Mobile
CBRS | n48 | TDD | 3550 - 3700 | 3550 - 3700 | North America

###  III-A LTE Bands - Uplink

Fig. 2 presents the measured power for LTE bands 13, 14, 15 and 41 considering
the UL frequency spectrum ranges. As it can be seen, the spectrum of LTE 12
and LTE 41 bands are more crowded compared with LTE 13 and LTE 14 bands. It is
worth mentioning that, unlike other LTE bands under consideration, LTE 41
works in time-division duplexing (TDD) mode and includes both UL and DL
transmissions. The mean and variance of the measured power for various LTE
bands are presented in Fig. 3. As it can be observed from Fig. 3(a), generally
the mean value of the measured power increases as the altitude increases. The
mean power value for LTE bands 12 and 41 are almost identical and much higher
than the other two bands under consideration. Note that band 41 has
significantly larger bandwidth than band 12 and it includes both UL and DL
transmission. From Fig. 3(b), it can be observed that the fluctuation of
variance for LTE band 13 is much lower than the other ones. Although the mean
value of LTE 12 and 41 show similar behaviour, the variance of LTE 41 is lower
than LTE band 12.

(a) LTE band 12 (UL).

(b) LTE band 13 (UL).

(c) LTE band 14 (UL).

(d) LTE band 41 (TDD UL/DL).

Figure 2: Measured LTE UL power for urban environment.

(a) Mean.

(b) Variance.

Figure 3: Spectrum occupancy versus altitude in LTE bands 12, 13, 14 and 41
(UL) for urban environment.

###  III-B LTE Bands - Downlink

Considering the DL frequency range for different LTE bands, Fig. 4 illustrates
the measured power for the bands under consideration. It can be readily
checked that the spectrum of DL frequency ranges are more crowded compared
with the UL ones. Although the occupied spectrum for LTE 13 and 14 expand the
whole range, the main frequency usage of LTE 12 is between 735 - 745 MHz.

(a) LTE band 12 (DL).

(b) LTE band 13 (DL).

(c) LTE band 14 (DL).

(d) LTE band 41 (TDD UL/DL).

Figure 4: Measured LTE DL power for urban environment.

(a) Mean.

(b) Variance.

Figure 5: Spectrum occupancy versus altitude in LTE bands 12, 13, 14 and 41
(DL) for urban environment.

Fig. 5 shows the mean and variance of the measured power versus altitude. As
it can be observed from Fig. 5(a), the mean value of the measured power
increases as the altitude increases up to almost 80 m. This is due to the fact
that at high altitudes the probability of receiving signal from neighbor cells
increases as the obstacles decrease, which results in the availability of the
line of sight (LoS). For higher altitudes (i.e., higher than 80 m), the mean
values for LTE bands under consideration remain almost constant. As it is
shown in Fig. 5(b), the variance of the measured power for LTE bands 13, 14
and 41 show relatively smaller variation over different altitudes compared to
LTE band 12. The main reason for this behavior can be found by observing the
measured power for LTE band 12 shown in Fig. 4(a). It seems that some portion
of the LTE band 12 is not fully utilized.

###  III-C 5G Bands - Uplink

Fig. 6 presents the measured power for 5G bands n5, n71 and n77 considering
the UL frequency spectrum ranges. This result reveals that the spectrum of n77
is mainly occupied between 3700-3800 MHz. One should also note that 5G band n5
and n71 utilize the frequency-division duplexing (FDD), while 5G band n77
exploit TDD mode. The performance of mean and variance of the measured power
for 5G bands (uplink) are presented in Fig. 7. As it can be observed from Fig.
7(a), the mean value of the measured power increases as the altitude increases
up to almost 80 m due to the same argument mentioned earlier. The mean value
of 5G band n5 shows higher value compared with n71 and n77. As it is shown in
Fig. 7(b), the variance of the measured power for 5G bands n5 and n77
intersect with each other around the altitude of 60 m. The variance of n77
band keeps increasing as the altitude increases.

(a) 5G band n5 (UL).

(b) 5G band n71 (UL)

(c) 5G band n77 (TDD UL/DL).

Figure 6: Measured 5G UL power for urban environment.

(a) Mean.

(b) Variance.

Figure 7: Spectrum occupancy versus altitude in 5G n5, n71 and n77 bands (UL)
for urban environment.

(a) 5G band n5 (DL).

(b) 5G band n71 (DL).

Figure 8: Measured 5G DL power for urban environment.

(a) Mean.

(b) Variance.

Figure 9: Spectrum occupancy versus altitude in 5G bands n5 and n77 (DL) for
urban environment.

###  III-D 5G Bands - Downlink

Fig. 8 illustrates the measured power for 5G n5 and n71 bands by considering
the DL frequency range. It can be seen that the measured power for 870 - 880
MHz and 885-894 MHz are higher than the rest of spectrum. Fig. 9 shows the
mean and variance of the measured power versus altitude. As it can be observed
from Fig. 9(a), the mean value of the measured power for n5 and n71 are
similar and significantly higher than n77. For the bands under consideration,
the mean value increases as the altitude increases up to almost 80 m. As it is
shown in Fig. 9(b), the variance of the measured power for n77 starts with a
small value, while it climes up to near those of n5 values as the altitude
increases. The variance of n71 band depicts a higher value for all the
measured altitudes compared with those others 5G bands.

###  III-E CBRS Band

(a)

(b)

Figure 10:  (a) CBRS spectrum and tiers; and (b) Measured CBRS band n48 power
for urban environment (TDD UL/DL).

(a) Mean.

(b) Variance.

Figure 11: Spectrum occupancy versus altitude in CBRS band for urban
environment.

Fig. 10(a) illustrates the CBRS spectrum which it lays out three tiers of
users. Fig. 10(b) presents the measured power for CBRS n48 band. Similar to
LTE 41 and 5G n77 bands, n48 also exploits TDD mode. As it can be seen, the
spectrum is mainly occupied within the range of 3610-3690 MHz. In Fig. 11, we
study the mean and variance of the measured power versus altitude whereas the
CBRS band is divided into three equal portions. As it can be observed, the
mean and variance of the measured power for the first portion (i.e., 3550-3600
MHz) are lower than the other parts. The mean value of the third portion
(i.e., 3650-3700 MHz) increases as the altitude increases up to 60 m and then
it drops afterwards. However, the man value of the second part (i.e.,
3600-3650 MHz) keeps increasing as the altitude increases.

##  IV Rural Spectrum Occupancy Results

In this section, we study the spectrum occupancy and its characteristic for
the similar bands as previous section by considering the experimental results
for the rural environment.

(a) LTE band 12 (UL).

(b) LTE band 13 (UL).

(c) LTE band 14 (UL).

(d) LTE band 41 (TDD UL/DL).

Figure 12: Measured LTE UL power for rural environment.

###  IV-A LTE Bands - Uplink

Fig. 12 illustrates the measured power for for LTE bands 13, 14, 15 and 41
considering the UL frequency spectrum. As it can be seen, LTE bands 12 and 41
show more crowded spectrum compared with LTE bands 13 and 14. The mean and
variance of the measured power for various LTE bands are presented in Fig. 13.
As opposed to the urban environment (cf. Fig. 3(a)), the mean value for LTE
bands 13 and 14 are much higher than the other two bands under consideration.

(a) Mean.

(b) Variance.

Figure 13: Spectrum occupancy versus altitude in LTE bands 12, 13, 14 and 41
(UL) for rural environment.

(a) LTE band 12 (DL).

(b) LTE band 13 (DL).

(c) LTE band 14 (DL).

(d) LTE band 41 (TDD UL/DL).

Figure 14: Measured LTE DL power for rural environment.

###  IV-B LTE Bands - Downlink

Considering the DL frequency range for different LTE bands, Fig. 14
illustrates the measured power for the bands under consideration. Same as the
urban results, the spectrum of DL frequency range are more crowded compared
with the UL ones in the rural environment. Fig. 15 shows the mean and variance
of the measured power versus altitude. As it can be observed from Fig. 15(a),
the mean value of the measured power increases as the altitude increases up to
80 m and it remains almost constant for the higher altitudes. The variance of
LTE bands 13, 14, and 41 show similar behaviour, while the corresponded plot
for LTE band 12 starts with increasing for the altitude up to 40 m and then it
drops afterwards.

(a) Mean.

(b) Variance.

Figure 15: Spectrum occupancy versus altitude in LTE bands 12, 13, 14 and 41
(DL) for rural environment.

###  IV-C 5G Bands - Uplink

Fig. 16 illustrates the measured power for 5G bands n5, n71 and n77
considering the UL frequency spectrum ranges. This result reveals that the
spectrum of n77 is less crowded than those of n5 and n71. The performance of
mean and variance of the measured power for 5G bands (uplink) are presented in
Fig. 17. As it can be observed from Fig. 17(a), while the mean value of the
measured power for n77 is almost independent of the altitude, it increases for
n5 and n71 bands as the altitude increases. As it is shown in Fig. 17(b), the
variance of the measured power for n71 depicts higher value compared with the
other 5G bands.

(a) 5G band n5 (UL).

(b) 5G band n71 (UL).

(c) 5G band n77 (TDD UL/DL).

Figure 16: Measured 5G UL power for rural environment.

(a) Mean.

(b) Variance.

Figure 17: Spectrum occupancy versus altitude in 5G n5 and n77 bands (UL) for
rural environment.

###  IV-D 5G Bands - Downlink

Fig. 18 illustrates the measured power for 5G n5 and n71 bands by considering
the DL frequency range. Similar to the urban case, it can be seen that the
measured power for 870 - 880 MHz and 885-894 MHz are higher than the rest of
spectrum in the rural environment. Fig. 19 depicts the mean and variance of
the measured power versus altitude. As it can be observed from Fig. 19(a), the
mean value of the measured power for n77 band remains almost constant for
different altitudes, while it increases as the altitude increases up to almost
80 m for n5 and n71 bands. As it is shown in Fig. 19(b), the variance of the
measured power for 5G band n71 shows higher values compared with n5 and n77.

(a) 5G band n5 (DL).

(b) 5G band n71 (DL).

Figure 18: Measured 5G DL power for rural environment.

(a) Mean.

(b) Variance.

Figure 19: Spectrum occupancy versus altitude in 5G bands n5 and n77 (DL) for
rural environment.

###  IV-E CBRS Band

Figure 20: Measured power during Helikite operation over rural environment for
CBRS band n48 (TDD UL/DL).

(a) Mean.

(b) Variance.

Figure 21: Spectrum occupancy versus altitude in CBRS band for rural
environment.

Fig. 20 present the measured power for CBRS n48 band for rural environment. As
it can be seen, the spectrum is less crowded compared with the rural
environment. In Fig. 21, we study the mean and variance of the measured power
versus altitude. As it can be observed, the mean value of the measured power
for all three considered portions are almost similar and remain constant as
the altitude increases. In addition, the variance also shows slight
fluctuations compared to the other bands under consideration.

##  V Time Domain Analysis of Spectrum Occupency

(a)

(b)

(c)

Figure 22: Spectrum monitoring during the measurement time. We observe
different LTE and NR bands’ occupancy and the received signal strength is
strong when the Helikite floats at a high altitude.

(a)

(b)

(c)

Figure 23: Received power of different LTE and NR bands during the measurement
time. The solid lines represent the mean value of signal power and shaded
areas indicate half of the standard deviation (std) of signal strength, which
shows the variation of signal strength inside the specific bands.

In this section, we focus on the spectrum occupancy of LTE and NR signals in
time, while we describe the altitude dependency of the spectrum in the
previous section. For around 8 hours of measurement duration by the Helikite
in the urban environment, we observe signal strength changes. This section
focuses exclusively on those urban environment measurements.

Fig. 22 shows the spectrum monitoring results by the Helikite. The x-axis is
the monitored spectrum range and the y-axis is the measured time stamp, which
is indicated by hours and minutes. In Fig. 22(a), we capture the frequency
range from 700 MHz to 800 MHz, which contains LTE FDD bands 12, 13, 14 (see
Table I). First of all, we can clearly observe a series of occupied 10 MHz
bandwidth 12, 13, and, 14 DL bands. On the other hand, the signal strength of
UL bands is lower than DL bands, and UL bands 13 and 14 are scarcely occupied.
We also observe that there are time periods when signal strength becomes low
for the whole observed frequency range, which coincides with the periods where
the altitude of the Helikite stays low in Fig. 1. It implies that received
signal strength is abruptly reduced by the blockage when the altitude of the
Helikite is lower than a certain height. In addition, this tendency is
observed in other frequency bands as well in Fig. 22(b) and Fig. 22(c). In Fig
22(b), we capture the frequency range 2500 MHz - 2700 MHz, which contains LTE
TDD 41 band. Since carrier frequency is higher than Fig. 22(a), we observe
that this LTE band covers wider bandwidth: 20 MHz, 40 MHz, and 100 MHz. It is
also observed that the received signal strength is lower than the frequency
range in Fig. 22(a). This is due to the fact that as carrier frequency
increases a received signal suffers higher path loss, which is also observed
in a much higher carrier frequency range in Fig. 22(c). In particular, Fig.
22(c) shows spectrum occupancy of NR TDD n77 band, 3700 MHz - 3800 MHz. We can
observe 40 MHz and 60 MHz bandwidth signals.

Fig. 23 shows the received signal strength changes during the measurement time
for the captured LTE and NR bands. In Fig. 23(a), we observe the LTE FDD UL/DL
12 band shown in Fig. 22(a). Mean value of the received signal strength across
the frequency band is represented by lines and half of the standard deviation
(std) of signal strength is described by the shaded area around lines. It is
observed that the signal strength of UL is lower than DL, while the variation
of the signal strength of UL inside the band is higher than DL, which can be
observed from higher std values. Fig. 23(b) and Fig. 23(b) show the received
signal strength changes of LTE TDD 41 and NR TDD 77 bands which can be shown
in Fig. 22(b) and Fig. 22(c). It is observed that the signal strength
fluctuation of NR TDD 77 band is higher than other bands such as LTE 12 and 41
bands.

##  VI conclusion

Using the data measured by a Helikite flying over an urban and rural
environments, in this paper we studied spectrum measurements in various sub-6
GHz 4G, 5G and CBRS bands. Both UL and DL spectrum occupancy has been
investigated. Our results revealed that generally the mean value of measured
power tends to increase as the altitude increases due to higher probability of
line-of-sight, at least for the considered maximum altitude range. Further,
the spectrum of DL frequency ranges showed to be more crowded compared with
the uplink ones for both environments. It has been also seen that for the
rural environment the mean value for LTE bands 13 and 14 are much higher than
the other two bands under consideration, as opposed to the urban environment.
Furthermore, the performance of CBRS band for urban environment indicates more
activity compared with the rural condition.

## References

  * [1] M. H. Islam, C. L. Koh, S. W. Oh, X. Qing, Y. Y. Lai, C. Wang, Y.-C. Liang, B. E. Toh, F. Chin, G. L. Tan _et al._ , “Spectrum survey in Singapore: Occupancy measurements and analyses,” in _Proc. IEEE Int. conf. Cognitive Radio Oriented Wireless Networks and Communications_ , 2008, pp. 1–7.
  * [2] F. Adelantado, X. Vilajosana, P. Tuset-Peiro, B. Martinez, J. Melia-Segui, and T. Watteyne, “Understanding the limits of LoRaWAN,” _IEEE Commun. Mag._ , vol. 55, no. 9, pp. 34–40, 2017.
  * [3] Ericsson, “Frequency reuse in limited spectrum networks,” Oct. 2022, accessed: 2023-01-04. [Online]. Available: https://www.ericsson.com/en/reports-and-papers/microwave-outlook/articles/maximizing-capacity-in-spectrum-limited-networks
  * [4] D. Roberson, “The 5G/aviation crisis that never should have happened,” https://www.linkedin.com/pulse/5g-aviation-crisis-never-should-have-happened-dennis-roberson/, 2022, accessed: 2023-01-04.
  * [5] W. Bellamy, “US Airlines Begin Installing 5G C-Band Filter for Radio Altimeters on Airbus A320s,” Sep. 2022, accessed: 2023-01-04. [Online]. Available: https://www.aviationtoday.com/2022/09/14/us-airlines-begin-installing-5g-c-band-filter-radio-altimeters-airbus-a320s/
  * [6] K. Dennehy, “Ligado scraps 5G network launch amid interference flap,” 2022, accessed: 2023-01-04. [Online]. Available: https://locationbusinessnews.substack.com/p/ligado-scraps-5g-network-launch-amid
  * [7] Y. Chen and H.-S. Oh, “A survey of measurement-based spectrum occupancy modeling for cognitive radios,” _IEEE Commun. Surv. Tuts._ , vol. 18, no. 1, pp. 848–859, 2014.
  * [8] B. Al Homssi, A. Al-Hourani, Z. Krusevac, and W. S. Rowe, “Machine learning framework for sensing and modeling interference in IoT frequency bands,” _IEEE Internet Things J._ , vol. 8, no. 6, pp. 4461–4471, 2020.
  * [9] B. A. Homssi, K. Dakic, K. Wang, T. Alpcan, B. Allen, S. Kandeepan, A. Al-Hourani, and W. Saad, “Artificial intelligence techniques for next-generation mega satellite networks,” _arXiv preprint arXiv:2207.00414_ , 2022.
  * [10] S. J. Maeng, J. Park, and I. Guvenc, “Analysis of UAV radar and communication network coexistence with different multiple access protocols,” _arXiv preprint arXiv:2211.16614_ , 2022.
  * [11] M. M. Azari, F. Rosas, A. Chiumento, A. Ligata, and S. Pollin, “Uplink performance analysis of a drone cell in a random field of ground interferers,” in _Proc. IEEE Wireless Commun. and Netw. Conf. (WCNC)_ , 2018, pp. 1–6.
  * [12] V. Marojevic, I. Guvenc, R. Dutta, M. L. Sichitiu, and B. A. Floyd, “Advanced wireless for unmanned aerial systems: 5G standardization, research challenges, and AERPAW architecture,” _IEEE Vehicular Technology Magazine_ , vol. 15, no. 2, pp. 22–30, 2020.
  * [13] AERPAW, “Helikite spectrum measurements (packapalooza),” Aug. 2022, accessed: 2023-01-04. [Online]. Available: https://sites.google.com/ncsu.edu/aerpaw-wiki/aerpaw-user-manual/4-sample-experiments-repository/4-4-data-repository/aerpaw-nrdz-research/august-2022-helikite-spectrum-measurements-packapalooza
  * [14] ——, “Helikite spectrum measurements,” May. 2022, accessed: 2023-01-04. [Online]. Available: https://sites.google.com/ncsu.edu/aerpaw-wiki/aerpaw-user-manual/4-sample-experiments-repository/4-4-data-repository/aerpaw-nrdz-research/may-2022-helikite-spectrum-measurements
  * [15] S. J. Maeng, O. Ozdemir, H. Nandakumar, I. Guvenc, M. Sichitiu, R. Dutta, and M. Mushi, “Spectrum Activity Monitoring and Analysis for Sub-6 GHz Bands Using a Helikite,” in _Proc Int. Conf. Commun. Syst. Netw. (COMSNETS)_ , Bengaluru, India, Jan. 2023.
  * [16] S. J. Maeng, I. Güvenç, M. Sichitiu, B. A. Floyd, R. Dutta, T. Zajkowski, Ö. Özdemir, and M. J. Mushi, “National radio dynamic zone concept with autonomous aerial and ground spectrum sensors,” in _IEEE Int. Conf. Communications Workshops (ICC Workshops)_ , 2022, pp. 687–692.


# Efficient pre-training objectives for Transformers

Luca Di Liello
DISI Department
University of Trento
Trento 38121, Italy
<EMAIL_ADDRESS>
&Matteo Gabburo
DISI Department
University of Trento
Trento 38121, Italy
<EMAIL_ADDRESS>
&Alessandro Moschitti
Amazon Alexa
Manhattan Beach, CA, USA
<EMAIL_ADDRESS>

###### Abstract

Transformer-based neural networks have heavily impacted the field of natural
language processing, outperforming most previous state-of-the-art models.
However, well-known models such as BERT, RoBERTa, and GPT-2 require a huge
compute budget to create a high quality contextualised representations. In
this paper, we study several efficient pre-training objectives for
Transformers-based models. By testing these objectives on different tasks, we
determine which of the ELECTRA model’s new features is the most relevant: (i)
Transformers pre-training can be improved when the input is not altered with
artificial symbols, e.g., masked tokens; and (ii) loss functions computed
using the whole output reduce training time. (iii) Additionally, we study
efficient models composed of two blocks: a discriminator and a simple
generator (inspired by the ELECTRA architecture). Our generator is based on a
much simpler statistical approach, which minimally increases the computational
cost. Our experiments show that it is possible to efficiently train BERT-like
models using a discriminative approach as in ELECTRA but without a complex
generator. Finally, we show that ELECTRA largely benefits from a deep hyper-
parameter search.

##  1 Introduction

Based on the architecture presented in [1], transformer-based models have
quickly become the standard in NLP. However, these models require expensive
hardware to be pre-trained [2]. Recently, there has been many attempts to
reduce the computational costs [3, 4, 5, 6]. In particular, we are interested
in ELECTRA. This model overcomes the most important BERT’s problem of being
frequently fed with the MASK token. This causes what is called a pre-
training/fine-tuning discrepancy because the MASK token is not seen in fine-
tuning. They solve this problem by replacing the Masked Language Model (MLM)
objective used by BERT [7] with a simpler objective, which allows the network
to be trained as a discriminator instead of a generator. So, instead of
predicting original masked tokens, in ELECTRA, the discriminator classifies if
tokens are original or fake. Moreover, ELECTRA computes the loss over the
entire discriminator output: this can reduce training time because of more
content is used from each training example.

We propose several efficient pre-training strategies, and we compare them on
well known NLP benchmarks. Specifically, we provide improved versions of MLM
and token detection (TD) tasks. For example, we require to predict the token
that was replaced instead of simply classifying original and substituted
tokens in a binary way. Furthermore, we make the token selection of ELECTRA
more efficient by substituting the generator with a simpler generative
approaches based on either random selection or history-based statistical
model. The latter simply counts the number of time each token has been
misclassified by the model. We cluster tokens together to reduce token/count
sparsity.

We pre-trained several language models along with state-of-the-art baselines
with the same hyper-parameters and datasets. Our models are obtained by
combining the above techniques using RoBERTa[8] pre-trained transformer, and
we test them on four natural language inference benchmarks: GLUE, SQUAD,
ASNQ-R and WikiQA. Moreover, we test the performance on transfer learning, by
transferring models on ASNQ-R and by testing them on WikiQA.

Our approaches exhibit a general superiority over MLM. For example, our
RoBERTa model that uses a history-based selection of replacement outperforms a
similar model based on MLM in most tasks, especially when doing transfer
learning, requiring also smaller computational budget. Moreover, our model
that only receives replaced tokens and predicts their original value
outperforms RoBERTa-MLM in every task by a margin of $1\%$, sometimes matching
the performance of ELECTRA, which is a more expensive and refined approach.

Most importantly, (i) in terms of computation time, our generators are much
more efficient than the original ones, and (ii) our pre-training objectives
require smaller classification heads, thus using less memory and computation
resources. We release the source code to reproduce our experiments.
111https://github.com/iKernels/efficient-pre-training-objectives-for-
transformers..

##  2 Related Work

Transformer-based models represent a new milestone for the AI community and,
in particular, for NLP applications. These models proved to outperform the
state of the art in many Natural Language Understanding tasks. The original
Transformer architecture was presented in [1], where the authors proposed a
model based solely on the attention mechanism [9], without using recurrence.
This approach had the major benefit of considering very longer sequences at a
time, as opposed to RNN. Moreover, Transformers-based models are well suited
for transfer learning. In fact, many language models are pre-trained on huge
amounts of unlabelled data with different techniques and finally are fine-
tuned on the target task. Pre-training is the phase that requires most
computational effort, but then the languages models can be shared and fine-
tuned on many different tasks in only a few training steps.

In this paper, we focus on more efficient pre-training objectives. The latter
are techniques that enable the training of language models using unlabelled
text. An example is represented by the Causal Language Modeling, exploited by
[10, 11, 12] for the GPT class of models. CLM consists of providing a model
with a truncated sentence and asking the model to predict the next token.

Another significant language modelling objective was proposed in BERT
(Bidirectional Encoder Representation for Transformers) [7]. The Masked
Language Modeling (MLM) objective consists of masking some tokens of the input
sentence and ask the model to predict their original value. By combining the
MLM objective with the Next Sentence Prediction loss, the authors of BERT
created a state-of-the-art Transformers-based model which, differently from
GPT, considers the input in a bidirectional way and is particularly effective
for NLU tasks.

Moreover, a relevant language modelling objective was introduced in ELECTRA
[6]. ELECTRA is composed of a generator and a discriminator network, which are
both BERT models. The generator is trained with MLM to find suitable
candidates to replace a special MASK token. The discriminator recognizes the
tokens replaced by the generator in the original text. After pre-training, the
generator is removed, and the discriminator model is used as the pre-trained
language model. ELECTRA is able to (i) outperform BERT when trained with the
same compute budget or (ii) reach the same performance in much less training
time. ELECTRA introduces many innovations, but the most effective is the use
of the whole output of the discriminator for computing the loss function. This
allows the model to receive a better signal. Additionally, the discriminator
input is not affected by the presence of a spurious token such as the MASK
token. Indeed, the latter, creating a discrepancy between pre-training and
fine-tuning, is one of the main drawbacks of the original BERT.

In the reminder of this section, we present other pre-training techniques that
can be used to create lighter and faster models.

In [4], the authors exploit distillation to create a smaller BERT model.
Distillation consists of a transfer learning technique, where the knowledge of
a large already-trained model is transferred to a smaller model. DistilBERT
can achieve $97\%$ of BERT performance by using only about half the
computation. A similar technique was exploited in [5]. In this work, the
authors train many small models by distillation, studying which of BERT hyper-
parameters (e.g., the number of layers, hidden size, embedding size) is the
most untactful on the final performance. Even thought the resulting models are
very fast in inference because of the reduced number of internal parameters,
they are not trained efficiently. Distillation requires a very larger model
from which transferring the knowledge to the smaller one.

Finally, ALBERT proposes another relevant methodology that aims at improving
pre-training efficiency. The authors of ALBERT [3] propose to share the
weights of every Transformer layer to save GPU memory and thus be able to use
bigger batch sizes. The train a model able to reach state-of-the-art
performance in some NLU tasks. Moreover, they show that the contextualized
representations created by every Transformer layer are not that different. A
very detailed overview of all the recent techniques proposed to improve the
efficiency and effectiveness of the Transformer is given in [13].

In conclusion, many works are addressing the problem of efficient training
models. Some apply enhancement at the architectural level, while others design
more refined pre-training tasks to train models faster.

##  3 Methodology

This section presents alternative pre-training objectives that can be applied
to a wide range of Transformer-based models. Our focus is on efficiency, thus
we propose new techniques and/or lighter language models, which can replace
inefficient solutions such as, the use of ELECTRA generator, or MLM.

###  3.1 Random Token Substitution (RTS)

Our Random Token Substitution approach carries out pre-training technique by
learning to discriminate the original tokens from substituted tokens,
similarly to the ELECTRA discriminator. The main difference is that RTS
chooses the tokens to be replaced randomly, thus avoiding the use of
computational resources to train a separate and expensive generator network.
Specifically, the model selects and replaces $15\%$ of the input tokens with
others from the vocabulary. Then the modified sentence is provided to the
model, which classifies whether tokens are original or not. We apply this
technique to a RoBERTa model, and we call it RoBERTa-RTS.

####  3.1.1 RTS using aggregated probabilities of misclassifying tokens
(C-RTS)

We replace a token $\alpha$ with a token $\beta$ using the posterior
misclassification probability, $P(\beta|\alpha\rightarrow\beta)$, where $P$ is
the probability that the discriminator is incorrect when classifying $\beta$,
given that $\alpha$ was replaced by $\beta$. $P$ can be derived by counting
the number of failures/successes for each pair ($\alpha,\beta$) in the
previous iterations. Thus, the main difference with the ELECTRA generator is
in the context: ELECTRA exploits the whole input sentence to create
challenging replacements, while our algorithm only uses the prediction history
of single tokens, and only depends on the past predictions of the model over
it.

Modelling exactly $P$ is unfeasible given that RoBERTa’s vocabulary size of
about $50K$ tokens would generate a huge amount of $(\alpha,\beta)$ entries,
being also extremely sparse.

Thus, we partition tokens into $n$ different clusters by measuring the
Euclidean distance between the corresponding embedding vectors obtained by
training a word2vec model [14] on the same data used for pre-training. After
that, we use the K-Means [15] algorithm to group the tokens into the clusters
$C_{1},\dots,C_{n}$.

Our implementation only uses a matrix ${\bm{F}}\in{\mathbb{Z}}^{n\times n}$,
which counts the difference between failures and successes of the
discriminator for each cluster pairs. ${\bm{F}}$ is initialised with zeros,
then, while training, some tokens are changed into others and the model should
discover them. For each pair of tokens $(\alpha\in C_{i},\beta\in C_{j})$,
such that $\alpha$ is replaced with $\beta$, we decrease ${\bm{F}}_{i,j}$ by
$1$ if the model correctly predicts $\alpha\neq\beta$, otherwise we increment
it.

For each training step, $P$ is estimated as follows:

$P(\beta|\alpha\rightarrow\beta)=P(C_{j}|C_{i}\rightarrow C_{j})\times
P(\beta|C_{j})$ (1)

assuming that the sampling on the target cluster $C_{j}$ is done with uniform
probability, $P(\beta|C_{j})=\frac{1}{|C_{j}|}$. Then the previous equation
becomes:

$P(\beta|\alpha\rightarrow\beta)=P(C_{j}|C_{i}\rightarrow
C_{j})\times\frac{1}{|C_{j}|}$ (2)

Thus, we only need to compute $P(C_{j}|C_{i})$ from the counters matrix
${\bm{F}}$. For each token $\alpha\in C_{i}$, we define a multinomial
distribution over the target clusters by indexing the ${\bm{F}}_{i}$ row. To
obtain a vector of values that can be interpreted as probabilities, we first
apply the min-max normalisation:

$\overline{{\bm{F}}_{i}}=\frac{{\bm{F}}_{i}-\min({\bm{F}}_{i})}{\max({\bm{F}}_{i})-\min({\bm{F}}_{i})}$

and then a $\gamma$-softmax:

$P(C_{j}|C_{i})=\frac{e^{\gamma*\overline{{\bm{F}}_{i,j}}}}{\sum_{k}e^{\gamma*\overline{{\bm{F}}_{i,k}}}}\\\
$

The $\gamma$ coefficient is used to control how the probability mass is
concentrated or relaxed on the most probable cluster. After sampling the
target cluster from this multinomial distribution, the model selects the
target token randomly with uniform probability.

To summarize, we randomly select some tokens of the input sentence and, given
their clusters, we define a multinomial distribution over the target clusters.
We finally select tokens with uniform probability from the target clusters and
put them in the original sentence. We looked for the best number of clusters
$n$ among $\\{30,100,300,1000\\}$ and for the best $\gamma$ in
$\\{1,2,5,10\\}$. After preliminary experiments we found that the best
combination is $n=100,\gamma=2$. We named the RoBERTa model exploiting this
pre-training technique as RoBERTa-C-RTS.

###  3.2 Swapped Language Model (SLM)

Swapped Language Model is a pre-training technique similar to the Masked
Language Modeling introduced by BERT[7]. In this case, tokens are only
randomly replaced with others and never with the special MASK token. Then,
differently from RTS, the model is trained to predict the original token and
not to discriminate between fakes and originals. This model is still
generative but without the need to exploit the MASK token. We apply this
technique to RoBERTa and we call it RoBERTa-SLM.

####  3.2.1 SLM-all

The SLM loss cannot be directly applied to the ELECTRA model. The reason is
that SLM (like MLM) does not discriminate between original and replaced tokens
since it is only applied to the output position corresponding to tampered
tokens. Then, we propose an alternative objective called SLM-all. In this
case, the discriminator has to predict the whole input sentence, estimating
which tokens were changed and predicting their original values. At the same
time, it should only reproduce the input in output for unchanged inputs. We
call this model ELECTRA-SLM-all.

##  4 Experiments

This Section describes the datasets, the hyper-parameters and the metrics used
both in pre-training and fine-tuning.

###  4.1 Models

To make a fair comparison, we always use the same architecture (corresponding
to a RoBERTa-base model): $12$ Transformer layers, a hidden size of $768$,
$12$ attention heads and an intermediate size equal to $3072$. For ELECTRA, we
used a generator with reduced width as described in the original work. Then,
it features an intermediate size of $1024$, a hidden-size equal to $256$ and
$4$ attention heads and $12$ layers. The discriminator is instead equal to a
RoBERTa-base. The RoBERTa models contain about 125M parameters, while on the
other hand ELECTRA, which is composed of both a generator and a discriminator,
has 142M parameters. Please notice that these numbers do not consider the
various classification heads used in pre-training and fine-tuning.

Since pre-training time is not proportional to the number of parameters222For
example, in every training step only a very small portion of the embedding
layer is updated., we measure the number of FLOPs (floating-point operations)
required to do pre-training, as in [6]. FLOPs measure the number of
mathematical operations performed on the hardware during the whole pre-
training phase, thus this number is independent of the used accelerators (GPUs
or TPUs) and of the model size.

###  4.2 Pre-training

We consider the same pre-training setting as BERT [7] for the base
architectures. More precisely, we pre-train each model on the English
Wikipedia and the BookCorpus dataset for $900K$ steps with a maximum sequence
length of $128$ tokens and the last $100K$ steps at $512$. This setting allows
for saving a lot of training time because the computational complexity of the
attention layer is quadratic with the maximum sequence length. However,
performance is slightly affected, but since we use the same pre-training
setting (we also pre-train the baselines) for every model, this is a fair
comparison.

We train every model with a learning rate equal to $1*10^{-4}$ and the Adam
optimizer with the following parameters:
$\epsilon=1*10^{-8},\beta_{1}=0.9,\beta_{2}=0.999$. The learning rate
scheduler is designed to warm-up for $10K$ steps and then to decreases
linearly. We used a batch size of $256$, and we apply a weight decay rate of
$0.01$ over the network parameters to stabilize training.

Since ELECTRA models require more FLOPs because of the generator, we reduce
the number of steps proportionally to the presence of the additional generator
as in [6]. Then, we train for a total of $766K$ steps, of which $689K$ with a
maximum sequence length of 128 tokens and the remaining $77K$ at 512.

###  4.3 Fine-tuning

This section provides the details about all the datasets and the experimental
settings for fine-tuning.

##### GLUE

The General Language Understanding Evaluation (GLUE) [16] is a well-known
benchmark suite to test models in 9 different NLU tasks. This collection
includes: (i) two datasets to test performance in paraphrasing capabilities,
one composed of questions (QQP) pairs and the other of the sentence pairs
(MRPC); (ii) a dataset for question-answer entailment (QNLI) derived from the
SQUAD dataset [17]; (iii) three datasets for textual entailment (RTE, MNLI and
WNLI); (iv) a single dataset (STS-B) to test the model on textual similarity;
(v) a dataset (SST-2) to evaluate performance on sentiment analysis and
finally (vi) a dataset to check linguistic acceptability (CoLA).

Results on the development set are computed by training with a batch size of
$32$ for $3$ epochs, a learning rate of $2*10^{-5}$, a max sequence length of
$128$ and by taking the average over the $5$ runs with different seeds. For
every model, we take the best checkpoint on the development set and we
evaluate it on the GLUE Leader-board.

##### SQUAD

The Stanford Question Answering Dataset [17] is a collection of questions
created by crowd-workers and the relative article passage taken from Wikipedia
in which there may be the answer. The dataset features $100K$ training
examples and slightly more than $10K$ validation examples. Since no test set
is publicly available, we compare the models by training all of them for a
fixed amount of steps with the same hyper-parameters. In particular, we train
for $3$ epochs with a batch size of $32$ and a learning rate of $3*10^{-5}$.
Moreover, we truncate input longer than $384$ tokens, and we repeat the
experiment over each model for $3$ times with different seeds.

##### ASNQ-Reduced

Answer-Sentence Natural Questions (ASNQ) [18], is a dataset built for the
Answer Sentence Selection (AS2) tasks. It was built using the Natural
Questions (NQ) corpus from [19], which was originally created for Machine
Reading. Natural Questions contains questions asked to the Google search
engine and corresponding Wikipedia pages that almost always contains the
answer. Crowd-workers extract long and short sentences from the articles. A
long sentence is typically a paragraph or an HTML bounding box, while a short
answer, which should be contained in the long, is a concise exact answer to
the question. The TandA [18] identified four different type of labels as shown
in Table 1

Label | $S\in LA$ | $SA\in S$ | # Train | # Dev
---|---|---|---|---
1 | No | No | 19,446,120 | 870,404
2 | No | Yes | 428,122 | 25,814
3 | Yes | No | 442,140 | 29,558
4 | Yes | Yes | 61,186 | 4,286
Table 1: $S$ is a sentence in the Wikipedia article, $SA$ is the short answer
and $LA$ is the long answer.

The resulting dataset contains more than $20M$ entries, which is great for
transfer learning. However, we remove all the sentences with the label $1$
because (i) they are easily recognizable since they are entirely off-topic;
(ii) [18] shows that transfer performance is not affected by this choice. By
considering only sentences with labels equal to 2, 3 or 4, we built a dataset
called ASNQ-Reduced, composed of slightly less than $1M$ training examples and
about $60K$ validation samples. In order to have also a test set, we take the
train, validation, and test splits from [20], obtaining a validation and a
test set with about $30K$ entries each.

We measure the performance using Mean Average Precision (MAP), Mean Reciprocal
Rank (MRR) and Precision@1. For fine-tuning, we used a batch size of $128$, a
learning rate of $2*10^{-5}$ and a maximum sequence length of $128$ tokens. We
fine-tune $10$ times with different seeds.

##### WikiQA

WikiQA [21] is a dataset for Answer Sentence Selection built from questions
asked to the Microsoft Bing search engine. Questions have been manually paired
with answers taken from Wikipedia articles and labelled as relevant or not.
The original WikiQA dataset is provided in different versions based on how
questions with all negative or all positive answers are filtered. We follow
the most common approach, excluding the questions with only negative answers
for training and both questions with all negative or all positive answers for
the development and test sets. By applying those filters, we obtain $2,118$
questions, and $20,360$ answers for training, $121$ questions and $1,126$
answers for the development set and $237$ questions and $2,341$ answers in the
test set. We train using batches with $32$ examples, a learning rate of
$1*10^{-6}$ and for $10$ epochs. As with ASNQ-R, we measure performance with
Mean Average Precision (MAP), Mean Reciprocal Rank (MRR) and Precision@1. For
each metric, we take the average of over $10$ runs with different seeds to
show reliable results.

##### ASNQ-R $\xrightarrow{}$ WikiQA

We also test our models on transfer learning for the AS2 task. We adopt the
same setting as TandA [18] and we exploit the ASNQ-R dataset to do the
transfer step, after which we fine-tune and test on WikiQA. Hyper-parameters
for transfer and fine-tuning are the same used for fine-tuning ASNQ-R and
WikiQA respectively.

Models | RoBERTa-MLM | RoBERTa-RTS | RoBERTa-C-RTS | RoBERTa-SLM | ELECTRA | ELECTRA-SLM-all
---|---|---|---|---|---|---
FLOPS | $1.64*10^{19}$ | $1.54*10^{19}$ | $1.54*10^{19}$ | $1.64*10^{19}$ | $1.98*10^{19}$ | $2.55*10^{19}$
Table 2: FLOPS used to pre-train each model. The huge gap between ELECTRA and ELECTRA-SLM-all is due to the ELECTRA model using only a small binary classification head on the discriminator as opposed to ELECTRA enhanced with SLM, which does token prediction for every input. Moreover, even by reducing the number of training steps of the ELECTRA models by about the $25\%$ (to balance the presence of a discriminator with size $1/3$) it uses more FLOPs than RoBERTa models. Finally, notice that RoBERTa-RTS and -C-RTS use less memory thanks to having only a binary classification head. Model | CoLA | MNLI | MRPC | QNLI | QQP | RTE | SST-2 | STS-B | AVG |
---|---|---|---|---|---|---|---|---|---|---
matt | acc | acc | acc | acc | acc | acc | spear | % |
BERT-MLM+NSP | 54.0 | 83.3 | 82.9 | 90.6 | 89.8 | 63.6 | 91.3 | 87.8 | 80.4 |
RoBERTa-MLM | 55.1 | 83.1 | 84.7 | 89.7 | 90.2 | 57.8 | 91.1 | 87.0 | 79.8 |
RoBERTa-RTS | 55.7 | 82.0 | 85.2 | 89.1 | 89.7 | 62.6 | 90.1 | 85.5 | 80.0 |
RoBERTa-C-RTS | 57.3 | 81.4 | 83.7 | 89.2 | 89.5 | 62.2 | 89.7 | 85.8 | 79.9 |
RoBERTa-SLM | 56.0 | 82.5 | 85.8 | 89.0 | 89.8 | 65.0 | 91.7 | 87.5 | 80.9 |
ELECTRA | 60.9 | 83.3 | 86.0 | 90.7 | 90.8 | 69.5 | 90.8 | 88.2 | 82.5 |
ELECTRA-SLM-all | 51.4 | 82.5 | 84.2 | 89.3 | 90.4 | 61.4 | 90.8 | 87.1 | 79.6 |
Table 3: Results on GLUE development set as the average over $5$ different runs. We measure performance on STS-B and CoLA with Matthews and Spearman correlation coefficient respectively and on the other GLUE tasks with accuracy. The results average show a standard deviation that reaches up to $0.9\%$ because of the high standard deviation of results on small datasets like CoLA, RTE, SST-2 and STS-B. As in [6, 7], we do not show results on WNLI because it is even hard to beat a trivial majority classifier. We include also the BERT-base model trained by [7] using the same compute as our RoBERTa-MLM to show that our models are aligned with the state-of-the-art. The NSP loss, as reported in many works, does not provide a noticeable improvement [8]. Model | CoLA | MNLI | MRPC | QNLI | QQP | RTE | SST-2 | STS-B | WNLI | AVG
---|---|---|---|---|---|---|---|---|---|---
matt | acc | acc | acc | acc | acc | acc | spear | acc | %
BERT-MLM+NSP | 50.0 | 82.9 | 82.7 | 90.4 | 88.3 | 65.2 | 90.6 | 82.8 | 65.8 | 77.6
RoBERTa-MLM | 54.7 | 82.6 | 84.4 | 90.2 | 88.7 | 55.6 | 91.2 | 82.5 | 60.3 | 76.7
RoBERTa-RTS | 56.2 | 81.6 | 84.2 | 89.6 | 87.7 | 61.0 | 90.3 | 79.5 | 65.1 | 77.2
RoBERTa-C-RTS | 52.7 | 81.6 | 84.0 | 89.5 | 88.7 | 61.4 | 89.5 | 79.6 | 65.8 | 77.0
RoBERTa-SLM | 54.8 | 82.7 | 81.9 | 89.5 | 88.7 | 61.3 | 91.0 | 83.6 | 65.1 | 77.6
ELECTRA | 59.6 | 82.9 | 85.3 | 91.0 | 89.3 | 66.1 | 91.7 | 84.8 | 65.1 | 79.5
ELECTRA-SLM-all | 51.1 | 82.6 | 84.9 | 89.5 | 89.1 | 62.9 | 92.6 | 82.3 | 65.1 | 77.8
Table 4: Results on GLUE test set. We evaluate with the same standard metrics
used on the development set. For each task we fine-tune $5$ times and take the
best model on the development set. Again, we include results on BERT-base of
[7] as described in Table 3.

##  5 Results

As said in the previous section, we evaluated our models on GLUE, SQUAD,
WikiQA and ASNQ-R. We also include some baselines of RoBERTa-base and ELECTRA-
base. To fairly compare the various objectives, we trained all baselines using
the setting described in Section 4.2. For this reason, these baselines cannot
be compared directly with the models released by the RoBERTa or ELECTRA teams.
Moreover, we do not use tricks to improve results on GLUE as many recent works
do [3, 6, 8, 22]. Finally, it is worth mentioning that we reproduced the
performance of ELECTRA in a standard pre-training setting, without using their
custom (i) optimizer333See https://github.com/google-
research/electra/blob/8a46635f32083ada044d7e9ad09604742600ee7b/model/optimization.py#L70.,
(ii) very high and unstable learning rate, and (iii) the particular layer-wise
learning rate decay [6]. This way, we have highly improved reproducibility.
Baselines are reported as RoBERTa-MLM and ELECTRA in the results tables.

Before discussing results, we show the number of FLOPs used to pre-train each
model in Table 2. We measure FLOPs using the same procedure explained in the
[6]. These numbers are significant indicators but may reflect in different
practical performance if the underlying hardware implements special
acceleration for some type of matrix operations.

Model | SQUAD V1.1 | WikiQA | ASNQ-R |  ASNQ-R $\xrightarrow{}$ WikiQA
---|---|---|---|---
EM | F1 | MAP | MRR | P@1 | MAP | MRR | P@1 | MAP | MRR | P@1
RoBERTa-MLM | 80.8 | 88.1 | 73.3 | 74.0 | 62.2 | 79.1 | 84.0 | 74.4 | 83.4 | 84.1 | 74.4
RoBERTa-RTS | 78.7 | 86.2 | 74.3 | 75.1 | 64.4 | 78.4 | 83.3 | 73.6 | 83.2 | 83.8 | 72.6
RoBERTa-C-RTS | 79.2 | 86.7 | 74.3 | 75.2 | 64.8 | 78.6 | 83.4 | 73.9 | 83.3 | 84.2 | 73.9
RoBERTa-SLM | 81.2 | 88.7 | 75.5 | 76.0 | 65.1 | 79.4 | 84.1 | 74.8 | 82.5 | 83.0 | 72.5
ELECTRA | 81.3 | 88.7 | 75.6 | 75.9 | 63.7 | 80.2 | 84.9 | 76.0 | 84.5 | 85.2 | 75.0
ELECTRA-SLM-all | 80.5 | 88.1 | 75.2 | 75.6 | 64.6 | 79.1 | 83.9 | 74.2 | 83.5 | 83.9 | 73.7
Table 5: Results on development sets for SQUAD v1.1, WikiQA, ASNQ-R and WikiQA after the transfer step on ASNQ-R. Standard deviations for SQUAD results are always below $0.3\%$. The standard deviations for ASNQ are instead always smaller than $0.5\%$ for MAP and MRR. WikiQA, being a smaller dataset, provides less uniform results and the standard deviation reaches $1\%$ for MAP and MRR on all models. In ASNQ-R $\xrightarrow{}$ WikiQA, the standard deviation is even higher because in transfer learning one combines two training phases. Thus, considering again MAP and MRR, the standard deviation increases up to $1.7\%$ for RoBERTa-RTS and RoBERTa-SLM, while it is slightly over $1.0\%$ for the other models. Model | WikiQA | ASNQ-R |  ASNQ-R $\xrightarrow{}$ WikiQA
---|---|---|---
MAP | MRR | P@1 | MAP | MRR | P@1 | MAP | MRR | P@1
RoBERTa-MLM | 70.8 | 72.1 | 58.6 | 78.7 | 83.3 | 73.0 | 82.9 | 84.2 | 74.3
RoBERTa-RTS | 71.0 | 72.0 | 57.7 | 78.2 | 82.9 | 72.5 | 83.0 | 84.5 | 74.6
RoBERTa-C-RTS | 71.0 | 72.1 | 57.6 | 78.4 | 83.1 | 72.8 | 84.3 | 85.8 | 77.1
RoBERTa-SLM | 72.1 | 73.2 | 59.6 | 78.8 | 83.3 | 73.3 | 84.2 | 85.4 | 75.4
ELECTRA | 75.3 | 76.7 | 62.8 | 79.3 | 83.8 | 73.9 | 86.9 | 88.0 | 80.4
ELECTRA-SLM-all | 72.1 | 73.4 | 59.8 | 79.0 | 83.5 | 73.7 | 87.4 | 88.9 | 82.5
Table 6: Results on the test set for WikiQA, ASNQ-R and WikiQA after the
transfer step on ASNQ-R. Standard deviations of MAP and MRR for WikiQA are
always below $0.8\%$ apart from RoBERTa-C-RTS and ELECTRA that reach $1.4\%$.
In the case of ASNQ-R, standard deviations are smaller for most models
$<0.3\%$ apart from ELECTRA, which reaches $1.5\%$ for both MAP and MRR.
Finally, standard deviations for ASNQ $\xrightarrow{}$ WikiQA are small in
average and are always below $0.5\%$. This proves that transfer learning is
also a way to stabilize results on the target task.

Tables 3 and 4 show the main results on the development and test sets of GLUE.
They show that all the considered approaches, with the exception of ELECTRA
base, obtain comparable performance. In particular, the RoBERTa models based
on RTS and C-RTS obtain a general improvement over MLM on the test set while
requiring a lower amount of computational effort to be pre-trained. Besides,
is also noticeable that the difference in performance between ELECTRA and our
techniques is not significant on the most stable tasks of GLUE (QQP and MNLI).
Indeed, ELECTRA widely outperforms our models only on CoLA, RTE and STS-B.

It is also worth mentioning that our RoBERTa-SLM model achieves better
performance than MLM on both GLUE development and test sets by simply removing
the MASK token from the pre-training. On the other hand ELECTRA-SLM-all
struggles in every GLUE test, where the data availability is scarce, thus
showing low generalization capabilities. Moreover, it requires more
computation resources due to the large classification head on the main
network444The equivalent of the discriminator in ELECTRA, but is not a
discriminator.. This confirms that the ELECTRA discriminator works well even
if it is not trained with a generative approach.

Table 5 and Table 6 show the results on the different QA and AS2 benchmarks
described in Section 4. The results on these datasets are aligned with those
on the GLUE benchmark.

In particular, considering the RoBERTa models, SLM is always superior to MLM
for every considered dataset and it even matches ELECTRA on SQUAD. It obtains
a significant margin over MLM especially in low-resource tasks like WikiQA,
confirming better generalization capabilities. Simultaneously, RTS and C-RTS
objectives lead to similar performances but require a lower amount of
resources to be pre-trained. Moreover, RoBERTa-C-RTS shows that a harder pre-
training using challenging replacements provides better performances in
transfer learning, where RoBERTa-RTS is very similar to RoBERTa-MLM instead.

Besides studying the results of the ELECTRA models, we noticed wider
differences between the two approaches. In particular, considering the results
on the test sets, the SLM-all are comparable with the ELECTRA performance on
the ASNQ-R benchmark ($79.0$ of ELECTRA-SLM-all against $79.3$ considering
MAP), worse on the WikiQA ($75.3$ and $72.1$) but better after the transfer
learning step from ASNQ-R to WikiQA ($87.4$ of the SLM-all approach compared
with the $86.9$ of ELECTRA). These results point out the quality of SLM-all
applied to an ELECTRA model, but also its limits. Indeed, ELECTRA-SLM-all
demonstrates that the advantages of transforming the ELECTRA discriminator
into a generative network is expensive and does not ensure better performance
for the majority of the tasks, but could be still a valuable choice for
transfer learning.

##  6 Conclusion and Future Works

In this work, we studied several methods to efficiently pre-train Transformer
models. Our approaches aim to match the results of well known pre-training
objectives such as MLM but consuming less computational resources. These
research directions have several benefits, since a lower computational effort
leads to a shorter training and to lower memory usage. We evaluate our
approach using several benchmark dataset such as GLUE, SQUAD, WikiQA and
ASNQ-R to have a clear understanding of the behaviours of our models.

The results show that RoBERTa-RTS and RoBERTa-C-RTS can match the performance
of the vanilla RoBERTa-MLM in every task while requiring less training time.
Moreover, RoBERTa-C-RTS shows better performance also with respect to RTS and
MLM in transfer learning. RoBERTa-SLM is another valid alternative to MLM:
while using exactly the same compute, it outperforms MLM on every task by a
consistent margin, thus confirming that the removal of the MASK token is
essential. Furthermore, RoBERTa-SLM is able to match ELECTRA in some tasks and
confirms that the original ELECTRA benefits also from a fine-grained hyper-
parameters search, other than using the whole output to compute the loss and
no MASK token. Finally, our ELECTRA-SLM-all model shows that the discriminator
of ELECTRA is not affected by doing only binary classification, but has
interesting performance in transfer-learning, reaching the highest performance
in this work.

We plan to combine our new techniques with efficient architectures like
DistilBERT and ALBERT in the near future, to take advantage of both a lighter
structure and a more effective pre-training objective.

## References

  * [1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need, 2017.
  * [2] Emma Strubell, Ananya Ganesh, and Andrew McCallum.  Energy and policy considerations for deep learning in NLP.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645–3650, Florence, Italy, July 2019. Association for Computational Linguistics.
  * [3] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  Albert: A lite bert for self-supervised learning of language representations, 2020.
  * [4] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020.
  * [5] Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Well-read students learn better: On the importance of pre-training compact models, 2019.
  * [6] Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning.  Electra: Pre-training text encoders as discriminators rather than generators, 2020.
  * [7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * [8] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach, 2019.
  * [9] Minh-Thang Luong, Hieu Pham, and Christopher D. Manning.  Effective approaches to attention-based neural machine translation, 2015\.
  * [10] A. Radford.  Improving language understanding by generative pre-training.  2018\.
  * [11] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019\.
  * [12] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners, 2020.
  * [13] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler.  Efficient transformers: A survey, 2020.
  * [14] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  Distributed representations of words and phrases and their compositionality.  In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26, pages 3111–3119. Curran Associates, Inc., 2013.
  * [15] Stuart P. Lloyd.  Least squares quantization in pcm.  IEEE Transactions on Information Theory, 28:129–137, 1982.
  * [16] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman.  Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019.
  * [17] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  Squad: 100,000+ questions for machine comprehension of text, 2016.
  * [18] Siddhant Garg, Thuy Vu, and Alessandro Moschitti.  Tanda: Transfer and adapt pre-trained transformer models for answer sentence selection, 2019.
  * [19] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov.  Natural questions: A benchmark for question answering research.  Transactions of the Association for Computational Linguistics, 7:452–466, March 2019.
  * [20] Luca Soldaini and Alessandro Moschitti.  The cascade transformer: an application for efficient answer sentence selection.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5697–5708, Online, July 2020. Association for Computational Linguistics.
  * [21] Yi Yang, Scott Wen-tau Yih, and Chris Meek.  Wikiqa: A challenge dataset for open-domain question answering.  In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. ACL - Association for Computational Linguistics, September 2015.
  * [22] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le.  Xlnet: Generalized autoregressive pretraining for language understanding, 2020.