# Text Difficulty Study: Do machines behave the same as humans regarding text
difficulty?

Bowen Chen, Xiao Ding, Li Du, Qin Bing, Ting Liu
Harbin Institute of Technology
{xding, bwchen, ldu, qbing<EMAIL_ADDRESS>
###### Abstract

Given a task, human learns from easy to hard, whereas the model learns
randomly. Undeniably, difficulty insensitive learning leads to great success
in NLP, but little attention has been paid to the effect of text difficulty in
NLP. In this research, we propose the Human Learning Matching Index (HLM
Index) to investigate the effect of text difficulty. Experiment results show:
(1) LSTM has more human-like learning behavior than BERT. (2) UID-SuperLinear
gives the best evaluation of text difficulty among four text difficulty
criteria. (3) Among nine tasks, some tasks’ performance is related to text
difficulty, whereas some are not. (4) Model trained on easy data performs best
in easy and medium data, whereas trains on a hard level only perform well on
hard data. (5) Training the model from easy to hard leads to fast
convergence.111Our code is available at
https://github.com/X3F4697J/ExamReadingBehavior

##  1 Introduction

Previous research in psycholinguistic shows _language learner perform better
on language tests when they start from easy sentences_(Amendum et al.,
2018;Faulkner and Levy, 1994;Yang and McNamara, 2014) , which leads to a
natural question that _does machine behaves like a human concerning the text
difficulty?_ In this paper, we investigate how text difficulty affects models.
Specifically, we aim to answer the following questions:

(1)In which criteria of text difficulty, neural-based or feature-based or
information-theory-based, the model behave like a human?

(2)In which model type, transformer(Vaswani et al., 2017)-based or
RNN(Rumelhart et al., 1986)-based model, the model behave like a human?

(3)In which kind of task, classification or regression task, the model behave
like a human?

##  2 Text Difficulty

(1) Flesch-Kincaid Score(Kincaid et al., 1975): Given a document $d$, the
number of sentences , word and syllables are $d_{s}$, $d_{w}$ and $d_{l}$,
then equation of Flesch-Kincaid Score is:

$\begin{split}Flesch(d)=&206.835-\dfrac{1.015d_{w}}{d_{s}}-\dfrac{84.6d_{l}}{d_{w}}\end{split}$
(1)

$206.835$, $1.015$ and $84.6$ are empirical values.

(2)Neural Evaluation: Training model on datasets like Weebit(V. and Meurers,
2012), One-stop Corpus(Vajjala and Lučić, 2018), in which human experts judge
the text difficulty. We fine-tune BERT(Devlin et al., 2019) on the One-stop
corpus to rank the text difficulty.

(3)Uniform Information Density (UID; Fenk and Fenk-Oczlon, 1980;Genzel and
Charniak, 2002;Aylett and Turk, 2004;Jaeger and Levy, 2007) hypothesis: UID is
based on the information theory(Shannon, 1948), which the cognitive processing
load of words is ratio to its log-probability and the ideal distribution of
information should be uniform. A sentence that follows UID will not be
cognitively taxing. We test two UID hypothesis operationalizations: UID Super-
Linear(UID-SL) and UID Variance(UID-Var).

(1) UID Super-Linear:

$UID(\textbf{u})^{-1}=\dfrac{1}{N}\sum_{n=1}^{n}s(u_{n})^{k}$ (2)

$k$ controls the strength of super-linearity. UID-SL suggests the text
difficulty increases regarding the exponential sum of sentence surprisal.

(2) UID Language-Variance:

$UID(\textbf{u})^{-1}=\dfrac{1}{N}\sum_{n=1}^{n}(s(u_{n})-\mu_{lang})^{2}$ (3)

$s(u_{n})\overset{def}{=}-log\ p(u_{n}|u_{<n})$ means the log-probability
conditioned on its prior context in both equations.UID-Var suggests the text
difficulty is decided by variance between the sentence surprisal and the mean
language-level surprisal $u_{lang}$.We follow the implementation of Meister et
al. (2021), which the $k$ is $1.25$ and $u_{lang}$ is $3.8845$.

##  3 Human Learning Matching Index

We propose Human Learning Matching Index (HLM Index) to answer above
questions, which has 3 sub-indexes $I_{task}$, $I_{model}$ and
$I_{criteria}$.222For details about split of dataset please refer to Appendix.

We split each dataset into 3 text difficulty level computed by different
criteria, which corresponds to easy, medium and hard level. Given tasks
$T=\\{t_{1}\dots t_{j}\\}$, models $M=\\{m_{1}\dots m_{k}\\}$ and criteria
$C=\\{c_{1}\dots c_{l}\\}$. Under task $t_{o}$ , criterion $c_{p}$ and model
$m_{q}$, a model trained on each difficulty level has test performance on each
difficulty level of test set
$P(t_{o},c_{p},m_{q})=\\{p_{e}^{opq},p_{m}^{opq},p_{h}^{opq}\\}$. Then, we
define a logical score function $s$:

$s(p_{e},p_{i},p_{s})=\begin{cases}0.75,&p_{e}\geq p_{m}\geq p_{h},\\\
0.375,&p_{e}\geq p_{h}\geq p_{m}\\\ 0,&p_{m}\geq p_{h}\geq p_{e}\\\
0,&p_{m}\geq p_{e}\geq p_{h}\\\ -0.375,&p_{h}\geq p_{e}\geq p_{m}\\\
-0.75,&p_{h}\geq p_{m}\geq p_{e}\end{cases}$ (4)

Then we compute the $I_{model}$ index as following:

$\begin{split}I_{model}(m_{k})=\dfrac{1}{JL}\sum_{j=1}^{j}\sum_{l=1}^{l}(s(P(t_{j},c_{l},m_{k}))+\\\
0.25sgn(s(P(t_{j},c_{l},m_{k}))f(STD(P(t_{j},c_{l},m_{k}))))\end{split}$ (5)

$STD$ means standard deviation, $f$ is a Sigmoid function.Input of both $STD$
and $s$ is the performance triplet in easy, medium and hard difficulty level
$p_{e}$, $p_{m}$ and $p_{h}$, and $sgn$ is the Sign function that is to decide
the sign of STD. The reason to include STD is to consider dispersion of
performance. As $s$ only considers logical relation, the performance gap
between different difficulty levels is ignored. However, the $f$ is to prevent
$STD$ from dominating the HLM Index.By replacing $m_{k}$ to $t_{j}$ or
$c_{l}$, we have $I_{task}$ and $I_{criteria}$. The reason to choose $0.75$,
$0.375$, and $0.25$ is to make the HLM index has a maximum and minimum value
of $\pm 1$, which is straightforward to illustrate the results. Changing these
parameters only affects the maximum value and does not affect the results.

##  4 Experiments

###  4.1 Datasets and Models

To cover the spectrum of NLP tasks as much as possible, we select SST2, MRPC
(Single/Pair-wise Sentence Binary Classification), QNLI, RTE(Multiple
Classification), and STS-B(Regression) from GLUE Benchmark(Wang et al., 2019).
We also include ROC Story (Multiple Choice) (Mostafazadeh et al.,
2017),WIKITEXT2(Language Modelling) (Merity et al., 2016),SQUAD 2.0(QA)
(Rajpurkar et al., 2018),CoNLL2003 NER(Tagging) (Tjong Kim Sang and De
Meulder, 2003).333We use WT2, SQUAD, CoNLL2003, ROC to denote WIKITEXT2, SQUAD
2.0, CoNLL2003 NER and ROC Story.

Models in this paper are BERT and LSTM (Hochreiter and Schmidhuber, 1997),
which represent parallel and recurrent NLP models.

###  4.2 Main Results

Figure 1: HLM Index heatmap for models, criteria, and tasks. SL and Var
represent super-linear and variance.

####  4.2.1 Models HLM Index

Figure 1 (a) shows the $I_{m}$ of LSTM and BERT on different tasks divided by
different criteria.

From the result, the LSTM has a higher average $I_{m}$ than BERT, which means
the LSTM has a more human-like learning behavior. This contradicts Merkx and
Frank (2021), where they find the Transformer is more human-like with regard
to learning human eye-tracking data, whereas we investigate it from learning
behavior on multiple tasks. The results also contradict the idea that a model
with higher task performance is more intelligent as the BERT model is known
for the competitive performance with humans on several NLP tasks. The results
suggest that a higher model performance does not guarantee that the model
could also behave like a human.

####  4.2.2 Tasks HLM Index

Figure 1 (b) shows $I_{t}$ of different tasks on different splits. From the
results, SQUAD and WT2 have the highest average, which means the performance
of the LSTM and BERT on these tasks are highly related to the text difficulty
even under different criteria. Especially in WT2, the $I_{t}$ reaches maximum
$1$ in both LSTM and BERT in UID-SL, UID-Var and Neural criteria. However, the
model gives a contrary score in the Flesch-Kincaid criterion, which means the
sentence length and syllables are not a good criterion to split WT2 due to a
strong tendency to give low difficulty to short sentences, which does not
guarantee easiness. A high score in SQUAD indicates that the model’s
performance in QA is highly related to the difficulty of the context and
question.

Moreover, CoNLL2003 and QNLI have a close zero score. For CoNLL2003, text
difficulty is not an obstacle for NER as entities are mainly represented as
uppercase in English, so the text difficulty does not influence the model or
human to tag the entity. For QNLI, the model infers answers from context and
question, which the difficulty is the inference rather than the text.

For other tasks, the $I_{t}$ are positive, which means the model performance
on these tasks is also related to text difficulty but also intertwines with
difficulty at a higher level like inference difficulty.

####  4.2.3 Criteria HLM Index

Figure 1 (c) shows the $I_{c}$ of different criteria.

From the result, the UID-SL gives the highest match, whereas the Neural-based
method gives the lowest match. The UID-SL gives the highest score, indicating
that the text difficulty is better evaluated by the super-linear function,
which is close to linear at the beginning and increases exponentially with the
sentence surprisal.

Surprisingly, the Neural criteria yield the lowest match, and we expect the
model to learn more sophisticated criteria from human experts as they could
judge text difficulty from a higher level. This may be due to insufficient
training data, and the model is not effectively learning human judgment from
human experts and only learns surface features that cannot generalize well on
unseen texts. Additionally, the UID-Var and Flesch-Kincaid give similar
results, showing that these two criteria are relatively less expressive than
UID-SL.

Criterion | Model | Train | SST2 | MRPC | RTE | QNLI | WT2 | STS-B | CoNLL2003 | SQUAD | ROC
---|---|---|---|---|---|---|---|---|---|---|---
Flesch | BERT | Hard | 92.17 | 84.24 | 59.56 | 89.62 | 68.83 | 83.61 | 89.92 | 81.53 | 85.27
Medium | 92.66 | 83.69 | 63.18 | 89.54 | 83.15 | 85.69 | 90.03 | 81.57 | 84.55
Easy | 91.63 | 84.01 | 61.15 | 89.89 | 89.44 | 85.96 | 89.89 | 82.05 | 86.08
LSTM | Hard | 84.69 | 81.23 | 54.24 | 60.25 | 156.24 | 76.79 | 87.78 | 53.52 | 67.22
Medium | 85.48 | 81.22 | 55.30 | 60.59 | 206.19 | 78.31 | 87.91 | 53.69 | 66.90
Easy | 86.32 | 81.37 | 54.94 | 60.77 | 475.91 | 75.43 | 87.48 | 54.12 | 65.36
UID-SL | BERT | Hard | 91.17 | 84.26 | 60.57 | 89.96 | 92.88 | 85.98 | 90.15 | 81.49 | 84.24
Medium | 84.39 | 84.24 | 59.35 | 89.62 | 86.36 | 86.86 | 90.21 | 81.66 | 84.80
Easy | 92.57 | 83.46 | 62.38 | 89.70 | 67.90 | 85.87 | 90.32 | 81.69 | 83.60
LSTM | Hard | 85.18 | 81.34 | 54.31 | 60.42 | 522.19 | 73.86 | 87.48 | 53.20 | 66.96
Medium | 85.66 | 81.54 | 55.65 | 60.67 | 199.31 | 76.15 | 88.42 | 53.60 | 65.98
Easy | 85.85 | 82.02 | 56.48 | 60.44 | 166.33 | 74.99 | 88.79 | 54.36 | 67.27
UID-Var | BERT | Hard | 91.76 | 84.24 | 60.28 | 89.95 | 93.78 | 87.72 | 89.98 | 81.83 | 85.24
Medium | 92.52 | 84.92 | 58.92 | 89.99 | 86.19 | 86.95 | 89.75 | 81.55 | 84.76
Easy | 92.20 | 83.37 | 63.39 | 89.53 | 74.84 | 86.18 | 89.91 | 81.95 | 86.24
LSTM | Hard | 85.25 | 81.58 | 54.08 | 61.72 | 407.09 | 75.51 | 88.21 | 53.34 | 65.30
Medium | 85.66 | 81.71 | 55.16 | 62.06 | 196.23 | 76.39 | 87.71 | 53.76 | 67.82
Easy | 85.85 | 81.28 | 55.67 | 61.69 | 166.41 | 74.04 | 87.91 | 54.43 | 65.22
Neural | BERT | Hard | 92.36 | 86.54 | 61.22 | 89.76 | 90.79 | 85.72 | 90.32 | 81.62 | 87.90
Medium | 92.50 | 85.62 | 60.94 | 89.84 | 87.38 | 85.75 | 90.11 | 82.10 | 86.94
Easy | 92.80 | 84.74 | 64.18 | 86.53 | 78.63 | 85.12 | 90.07 | 81.81 | 84.77
LSTM | Hard | 85.36 | 81.27 | 55.38 | 60.94 | 304.66 | 75.40 | 87.62 | 53.14 | 69.18
Medium | 85.82 | 81.73 | 56.34 | 60.72 | 200.26 | 78.13 | 88.79 | 54.98 | 70.88
Easy | 85.52 | 81.61 | 56.91 | 60.61 | 198.69 | 76.43 | 88.41 | 54.43 | 62.76
Table 1: Results of models on different difficulties evaluated by four
criteria. Accuracy for SST2, RTE, QNLI, and ROC. F1 for MRPC,SQUAD and
CoNLL2003. Perplexity for WT2. Pearson correlation for STS-B. The HLM Index is
computed based on this table.

###  4.3 Effect of Training Order

In this part, we train the model in a human-like schedule that trains from
easy to hard and another schedule that trains reversely, then report the
convergence step divided by the total steps and the performance on the test
set. We select RTE, SQUAD, MRPC, and SST2 to perform the experiments. The
results are in Table 2.444Default criterion to split the data is UID-SL.

Task | LSTM | BERT
---|---|---
E $\rightarrow$ H | H $\rightarrow$ E | Rand | E $\rightarrow$ H | H $\rightarrow$ E | Rand
P | C(%) | P | C(%) | P | C(%) | P | C(%) | P | C(%) | P | C(%)
SST2 | $81.65$ | 15.07 | $81.67$ | $36.29$ | 81.77 | $18.50$ | 93.00 | 22.49 | $92.88$ | $44.23$ | $92.78$ | $28.44$
MRPC | $82.06$ | 20.04 | $81.91$ | $53.22$ | 82.26 | $40.16$ | 87.73 | 19.04 | $87.43$ | $43.24$ | $86.59$ | $31.09$
RTE | 58.18 | 16.75 | $58.19$ | $75.38$ | $58.12$ | $42.33$ | $64.62$ | 31.22 | $63.89$ | $54.23$ | 65.34 | $44.17$
SQUAD | $56.32$ | 39.22 | $56.21$ | $68.63$ | 56.43 | $47.21$ | $82.34$ | 30.21 | $82.03$ | $53.21$ | 83.21 | $43.46$
Table 2: Performance and convergence of different tasks. Rand means a randomly
shuffled train set. P means performance. C means the convergent step divided
by total steps, which is the lower, the better. Figure 2: LSTM learning curve
in SST2 task. Transition points are annotated using arrow symbols.

From the results, training the model in a human-like schedule leads to a
quicker convergence, whereas a reverse schedule shows the slowest convergence.
The random schedule is in between. This means we could train the model more
efficiently in a human-like schedule. Additionally, different training
schedules give a close best performance, indicating the final performance is
not sensitive to the training schedule, but the convergence is sensitive to
it, which helps explain why a difficulty insensitive schedule is successful in
NLP. However, even the best performance is insensitive to difficulty, but we
could still reach comparable performance with fewer data in a human-like
training schedule. A case on the SST2 task is in Figure 2. We could see a
human-like training order leads to an evident quicker convergence, whereas the
reverse schedule leads to the slowest convergence. The LSTM uses the easy part
and a subset of the medium part of train data in the human-like training
schedule to achieve the best performance. In contrast, the reverse schedule
needs to use whole data to achieve the same performance, implying that the
model did not effectively learn the hard data.

###  4.4 Transfer Between Text Difficulty

This part investigates the difficulty transfer on the test set. For example,
suppose a model trained on easy data performs better. In that case, we ask _in
which difficulty level of the test set, the performance improves or decreases
?_ We split the test set of SST2, WT2, MRPC, RTE, QNLI, and SQUAD into 3
difficulty levels using all criteria then collects the model performances on
each difficulty level of the test set. We sort the results and give $3,2$ and
$1$ scores to the best, mediocre and lowest performances. The average score is
in Table 3.

Train | LSTM | BERT
---|---|---
Evaluation Set | Evaluation Set
Set | Easy | Med | Hard | Easy | Med | Hard
Easy | 2.54 | 2.33 | 1.29 | 2.67 | 2.37 | 1.67
Med | 2.16 | 2.21 | 2.17 | 1.95 | 2.24 | 1.83
Hard | 1.29 | 1.46 | 2.54 | 1.38 | 1.46 | 2.50
Table 3: Transfer scores between different text difficulties. Med means medium
difficulty level.

From the results, the model trained on the easy level has the best performance
in both easy and medium levels, whereas trained on the hard level only
performs well on the hard level and fails in other levels. Additionally, the
model trained on a medium level gives a stable performance in all difficulty
levels of the test set. The reason may be hard text difficulty dataset follows
a different distribution from the easy and medium levels. In the hard dataset,
as pointed out by results in Sec 4.2.2, the difficulty not only relates to the
text level but also relates to a higher concept like the difficulty of
inference or understanding, and a hard text is a feature of such difficulty.
This suggests the distribution of the hard level might be different from easy
and medium level, so the model trained on hard data does not perform well on
other levels.

##  5 Conclusion

In this work, we investigated how and in what way the text difficulty affects
and exists in NLP tasks and models. We analyzed experiments on nine tasks
using HLM Index. Results show that LSTM gives more human-like behavior. UID-SL
gives the best text difficulty evaluation. Some tasks are related to text
difficulty, whereas some are not. Moreover, transfer experiment shows that the
training begins with the easy data leads to a more general and better
performance than hard data. Additionally, training with a human-like schedule
is more efficient than other schedule and leads to a quicker convergence.

## References

  * Amendum et al. (2018) Steve Amendum, Kristin Conradi, and Elfrieda Hiebert. 2018.  Does text complexity matter in the elementary grades? a research synthesis of text difficulty and elementary students’ reading fluency and comprehension.  _Educational Psychology Review_ , 30.
  * Aylett and Turk (2004) Matthew Aylett and Alice Turk. 2004.  The smooth signal redundancy hypothesis: A functional explanation for relationships between redundancy, prosodic prominence, and duration in spontaneous speech.  _Language and Speech_ , 47(1):31–56.  PMID: 15298329.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.
  * Faulkner and Levy (1994) Heather J. Faulkner and Betty Ann Levy. 1994.  How text difficulty and reader skill interact to produce differential reliance on word and content overlap in reading transfer.  58(1):1–24.
  * Fenk and Fenk-Oczlon (1980) August Fenk and Gertraud Fenk-Oczlon. 1980.  Konstanz im kurzzeitgedächtnis - konstanz im sprachlichen informationsfluß?  _Zeitschrift für experimentelle und angewandte Psychologie_ , 27:400–414.
  * Genzel and Charniak (2002) Dmitriy Genzel and Eugene Charniak. 2002.  Entropy rate constancy in text.  In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_ , ACL ’02, page 199–206, USA. Association for Computational Linguistics.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9:1735–80.
  * Jaeger and Levy (2007) T. Jaeger and Roger Levy. 2007.  Speakers optimize information density through syntactic reduction.  In _Advances in Neural Information Processing Systems_ , volume 19. MIT Press.
  * Kincaid et al. (1975) J. Peter Kincaid, Robert P. Fishburne, Richard Lawrence Rogers, and Brad S. Chissom. 1975.  Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel.
  * Meister et al. (2021) Clara Meister, Tiago Pimentel, Patrick Haller, Lena Jäger, Ryan Cotterell, and Roger Levy. 2021.  Revisiting the uniform information density hypothesis.
  * Merity et al. (2016) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016.  Pointer sentinel mixture models.
  * Merkx and Frank (2021) Danny Merkx and Stefan L. Frank. 2021.  Human sentence processing: Recurrence or attention? In _Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics_ , pages 12–22, Online. Association for Computational Linguistics.
  * Mostafazadeh et al. (2017) Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. 2017.  LSDSem 2017 shared task: The story cloze test.  In _Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics_ , pages 46–51, Valencia, Spain. Association for Computational Linguistics.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for squad.
  * Rumelhart et al. (1986) David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. 1986.  Learning representations by back-propagating errors.  _Nature_ , 323:533–536.
  * Shannon (1948) C. E. Shannon. 1948.  A mathematical theory of communication.  _The Bell System Technical Journal_ , 27(3):379–423.
  * Tjong Kim Sang and De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_ , pages 142–147.
  * V. and Meurers (2012) Sowmya V. and Detmar Meurers. 2012.  On improving the accuracy of readability classification using insights from second language acquisition.  pages 163––173.
  * Vajjala and Lučić (2018) Sowmya Vajjala and Ivana Lučić. 2018.  OneStopEnglish corpus: A new corpus for automatic readability assessment and text simplification.  In _Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 297–304, New Orleans, Louisiana. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.
  * Wang et al. (2019) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In the Proceedings of ICLR.
  * Yang and McNamara (2014) H.S. Yang and Danielle McNamara. 2014.  What’s so simple about simplified texts? a computational and psycholinguistic investigation of text comprehension and text processing.  _Reading in a Foreign Language_ , 26:92–113.


# A Two-Dimensional Deep Network for RF-based Drone Detection and
Identification Towards Secure Coverage Extension
††thanks: This work reported in this paper is supported in part by the
National Key R&D Program of China under Grant No. 2020YFB1806905, the National
Natural Science Foundation of China under the Grant No. 62071368, and the Key
Research and Development Program of Shaanxi Province under the Grant No.
2023-GHZD-05.

Zixiao Zhao1,2, Qinghe Du1,2, Xiang Yao1,2, Lei Lu1,2, and Shijiao Zhang1,2
1 School of Information and Communications Engineering, Xi’an Jiaotong
University, China
2 National Simulation Education Center for Communications and Information
Systems, Xi’an Jiaotong University, China
Emails<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

As drones become increasingly prevalent in human life, they also raises
security concerns such as unauthorized access and control, as well as
collisions and interference with manned aircraft. Therefore, ensuring the
ability to accurately detect and identify between different drones holds
significant implications for coverage extension. Assisted by machine learning,
radio frequency (RF) detection can recognize the type and flight mode of
drones based on the sampled drone signals. In this paper, we first utilize
Short-Time Fourier Transform (STFT) to extract two-dimensional features from
the raw signals, which contain both time-domain and frequency-domain
information. Then, we employ a Convolutional Neural Network (CNN) built with
ResNet structure to achieve multi-class classifications. Our experimental
results show that the proposed ResNet-STFT can achieve higher accuracy and
faster convergence on the extended dataset. Additionally, it exhibits balanced
performance compared to other baselines on the raw dataset.

###### Index Terms:

radio frequency (RF) detection, short-time fourier transform (STFT),
convolutional neural network (CNN), drone detection and identification.

##  I Introduction

Nowadays, the applications of drones, which are also known as unmanned aerial
vehicles (UAVs), have been penetrated into every aspects of human life,
including aerial photography, plant protection, military, etc. The global
civil drone industry is expected to reach about 21.6 billion U.S. dollars by
2027. Military use has previously accounted for much of drone use, but the
industry is increasingly entering commercial, scientific, and agricultural
usage.

While drones offer numerous benefits and opportunities, they also present
several security concerns that need to be addressed [1], raising challenges
for coverage extension. One of the primary concerns is the potential for
unauthorized access and control of drones. Hackers or malicious individuals
could attempt to gain control over a drone by exploiting vulnerabilities in
its communication systems or flight controls. This can lead to misuse of the
drone for illegal activities or sabotage. Another concern arises from the
growing number of drones in the airspace, which increases the risk of
collisions and interference with manned aircraft. Unauthorized or unregulated
drone flights can pose risks to aviation safety, especially near airports or
in restricted airspace. An example occurred in Oct. 2017 that a civil aircraft
collided with a drone as the former was approaching the airport near Quebec
City, Canada. Lastly, data breaches and privacy issues are also hidden
troubles. Drones often capture and transmit data during their operations,
including images, videos, and other sensor readings. If proper security
measures are not in place, there is a risk of data breaches, where the
captured information can be intercepted or accessed by unauthorized parties.
This raises concerns about privacy violations and the potential misuse of
sensitive data [2].

Thus as drones become increasingly prevalent, ensuring the ability to
accurately detect and identify between different drones holds significant
implications for secure coverage extension. In order to reduce or eliminate
the threats posed by illegal drone flights, there are four main detection
methods: optical detection, acoustic detection, radar detection, and radio
frequency (RF) detection [3]. Compared with the other three detection methods,
the advantages of RF detection which will be achieved based on the captured
communication signals include: it can detect drones of any size and distance,
i.e., within line of sight or beyond line of sight, and can also be used to
identify the flight mode of drones, such as flying, hovering, recording, etc
[4].

In this paper, we design a deep network to accurately detect the presence of
drone signals and identify various drone states, taking into full
consideration the characteristics of sampled radio signals. The contributions
of this work include:

  1. 1)

We extend the raw dataset considering the situation that several different
types of drones are coexisting and simultaneously transmitting signals.
Correspondingly, we design extra classification tasks which have not been
discussed in previous research.

  2. 2)

We employ short-time fourier transform (STFT) algorithm to extract two-
dimensional features, i.e., time-domain and frequency-domain. They can provide
more information of hidden correlations for classification by feeding into
two-dimensional convolutional neural network (2D CNN).

  3. 3)

Our experiments show that the proposed ResNet-STFT algorithm is able to
achieve 98.7% accuracy in seven-class classification based on the extended
dataset. Moreover, it can also achieve faster convergence compared with the
one-dimensional baseline method.

The rest of the paper is organized as follows. Section II introduces the
related works. Section III describes the dataset and extended version. Section
IV proposes our methodology including feature extraction and classification
network. Section V presents the experimental results. Finally, Section VI
concludes the work.

##  II Related Works

Machine learning has been widely used in drone detection. Nie. et al. in [3]
extracted fractal dimension, axially integrated bispectra, and square
integrated bispectra as UAV radio frequency (RF) fingerprint. The principal
component analysis (PCA) algorithm was applied to reduce the dimensionality of
features, then machine learning classifier achieved UAV identification.
Medaiyese. et al. in [4] proposed a three-level hierarchical framework to
detect UAV signals in the presence of other wireless signals such as Bluetooth
and WiFi, which utilized a semi-supervised learning approach.

DroneRF is a common drone dataset proposed by Allahham. et al. in [5]. Some
research have been done based on this dataset. In [6] authors proposed a fully
connected neural network with three hidden layers to classify drone signals.
Allahham. et al. in [7] further proposed a multi-channel 1D CNN achieving
higher accuracy performances. In [8] Raina. et al. proposed ConvLGBM model
which combined the feature extraction capability of a CNN network with the
high classification accuracy of the Light Gradient Boosting Machine
(LightGBM). Considering feature engineering, Inani. et al. in [9]
synthetically discussed several features in time and frequency-domain,
including root mean square energy (RMS), discrete-fourier transform (DFT),
power spectral density (DFT), etc. They further proposed a 1D CNN to identify
the target drone signals with these extracted features. In this paper, we will
extend the raw DroneRF dataset, and introduce feature extraction methods and
classification networks working in two-dimensional space.

##  III Dataset and Extended Version

###  III-A DroneRF dataset

M.S.Allahham et al. in [5] provide the DroneRF dataset which they collected
from three types of drones and five types of function modes. Specifically,
Parrot Bebop and Parrot AR Drone were both tested in Off, On and connected,
Hovering, Flying, and Video recording modes. Another DJI phantom was merely
tested in two modes, i.e., Off and On and connected. The RF receiver can
capture the transmission signals between the drone and controller, which
sampling bandwidth is equal to 40MHz. Thus for scanning 80MHz spectrum, the
authors adopted two RF receivers, sampling for the lower 0$\sim$40MHz
frequency band and higher 40$\sim$80MHz frequency band separately. Each sample
of this dataset, which is also denoted as a segment, is composed of $10^{7}$
time-domain points. The dataset has 227 segments in total, and the proportions
of them can be listed in Table I.

Moreover, the authors design binary unique identifier (BUI) rule to
effectively name and distinguish the segments. The BUI number of each segment
is composed of five binary digits. The first number indicates the presence of
drone activities, the second and third numbers are used to characterize the
three drone types, and the last two digits are corresponding to the four
function modes. This scientific naming rule will also be practiced in the
following dataset augmentation. In Table I the samples with BUI belonging to
0xxxx or 1xxxx come from the raw dataset. Others in Table I are extended
samples, and we will introduce them below.

###  III-B Data augmentation

We further consider the extended condition that there are two types of drones
coexisting and working in the same mode in a segment. Due to the records of
Phantom merely contain On and connected mode in the raw DroneRF dataset, we
mainly discuss this mode. Specifically, we add the time-domain points of Bebop
& AR, Bebop & Phantom, and AR & Phantom respectively, indicating the two
drones transmitting signals simultaneously. We show the detailed information
in Table I.

Moreover, we utilize BUI 2xxxx to name the extended data. The first number
indicates there two types of drones are coexisting. The second and third
numbers, i.e., 00, 01, and 10, are related to Bebop & AR, Bebop & Phantom, and
AR & Phantom, respectively. The last two numbers indicates the function mode,
which are merely set as 00 because of On and connected mode.

TABLE I: Summary of DroneRF Dataset and its Extended Version BUI | Types | Function modes |  Numbers of samples | No.
---|---|---|---|---
0xxxx | No drone | No drone | 41 | 00000
1xxxx | Bebop | On and connected | 21 | 10000
Hovering | 21 | 10001
Flying | 21 | 10010
Video recording | 21 | 10011
AR | On and connected | 21 | 10100
Hovering | 21 | 10101
Flying | 21 | 10110
Video recording | 18 | 10111
Phantom | On and connected | 21 | 11000
2xxxx | Bebop & AR | On and connected | 441 | 20000
Bebop & Phantom | On and connected | 441 | 20100
AR & Phantom | On and connected | 441 | 21000

###  III-C Classification cases

According to the data types of dataset, we design five kinds of classification
cases described as follows:

  * •

Case I: Binary classification. The classifier needs to detect whether a piece
of given data contains drone signals.

  * •

Case II-A: Four-class classification. The classifier needs to identify none or
which type of drone signal a piece of given data contains, including Bebop,
AR, or Phantom.

  * •

Case II-B: Three-class classification. The classifier needs to identify which
two types of drone signals a piece of given data contains, including Bebop &
AR, Bebop & Phantom, or AR & Phantom.

  * •

Case II-C: Seven-class classification. The integrated case of Case II-A and
Case II-B.

  * •

Case III: Ten-class classification. The classifier needs to identify none or
which type of drone signal and its function mode a piece of given data
contains.

Note that Case I, Case II-A, and Case III are based on the raw DroneRF dataset
which have been discussed by other papers [6, 7, 8, 9], while Case II-B and
Case II-C are based on our extended dataset. The following experiments will
compare our detection algorithm with other outstanding baselines and prove
that ours can achieve balanced performances on general cases and better
performances on extended cases.

##  IV methodology

###  IV-A Feature extraction with Short-time fourier transform

In [6] a segment was divided into several continuous but non-overlapping time-
domain parts, and we denote this as simple-cutting (SCU) method. Compared with
SCU, short-time fourier transform (STFT) algorithm compensates the information
loss by introducing window function. Specifically, the overlapping part
between two windows contributes to catch hidden but continuous time-domain
information. The formulation of STFT can be depicted as follows:

$STFT(\tau,f)=\int_{-\infty}^{+\infty}x(t)h(t-\tau)e^{-j2\pi ft}d\tau,$ (1)

where $x(t)$ denotes the target signal, and $h(t-\tau)$ denotes a window
function which is used for intercepting a frame from $x(t)$. Shifting $\tau$
along timeline, STFT can get the fourier transform result of each intercepted
frame, and finally, help to analyze the whole time-domain and frequency-domain
information of $x(t)$.

The function spectrogram(x,window,noverlap,nfft,fs) in MATLAB can fulfill the
calculation of STFT. We set the parameter nfft as 128, indicating 128-point
FFT for each frame. The type and length of window function both can be
altered. To ensure that the numbers of time-domain and frequency-domain points
are both equal to 128, which will be convenient for the following CNN to
recognize and classify, we set the length of Hamming window as $8.8\times
10^{4}$ and overlapping parts between two windows as $10^{4}$ accordingly. The
overlapping ratio is equal to 11.4%.

We further compare the STFT algorithm with SCU method in visualization form.
The upper-half spectrogram is generated with lower sampling band, while the
lower-half spectrogram is generated with higher sampling band that were
sampled simultaneously. Hereinafter, we will use this concatenation way to
generate feature patterns from dataset. For instance, we choose a segment
coming from BUI = 20000 which indicates that Bebop drone and AR drone are
coexisting. As shown in Fig. 1, compared with SCU style (Fig. 1), the
frequency points with high energy in STFT style (Fig. 1) tend to be more
concentrative, and some hidden features are also enhanced. We believe that
STFT is able to provide additional and valuable information for
identification.

Moreover, it is evident that high energy points mainly appear in the upper-
half spectrogram. Authors in [10] discussed the performances with lower band,
higher band, and both two bands, respectively. They believed that the lower
sampling band has carried enough features for detecting and identification
drones. Our spectrogram results confirm this conclusion well.

Figure 1: These figures are spectrogram of signals when Bebop drone and AR
drone are coexisting. The size is equal to 128*128. The horizontal axis
indicates time-domain containing 128 intercepted parts, and the vertical axis
indicates frequency-domain containing 128-points FFT. (a) is generated with
STFT algorithm, and (b) is generated with SCU method. Compared with (b), some
hidden features are exhibited in (a).

###  IV-B ResNet structure

Convolutional neural network (CNN) is one of the most typical algorithm in
deep learning, which contains convolutional computation and deep structure.
The applications of CNN are mainly related to computer vision area, such as
image classification, semantic segmentation, pose estimation, etc. Especially
when the dimensions of input data are quite large, CNN is able to avoid
explosive scale of network parameters by local and distributed convolutional
computing.

One-dimensional (1D) CNN is typically used for signal processing, in which the
input data contains correlation characteristic on timeline and needs to be
predicted or classified. Two-dimensional (2D) CNN has border application
prospect, in which the input data is shaped into matrix form. The typical 2D
CNN structures include AlexNet [11], VGG [12], GoogleNet [13], ResNet [14],
etc. In this paper we employ 2D CNN with ResNet structure to solve the
problems.

In computer vision, the depth of the network will increase with the larger
scale of input features. Typically deeper network is able to achieve better
performances, however, the new problem, i.e., gradient explosion and gradient
vanishing, which will result in the failure of network convergence, becomes an
obstacle to training such a network. ResNet structure is proposed to counter
this problem. Firstly, the shortcut connection of ResNet structure can
accelerate the information propagation in the whole network. Secondly, the
batch normalization (BN) is utilized to ensure that the input feature map of
every convolutional layer follows the normal distribution with mean 0 and
variance 1. The formulations of BN can be depicted as follows [15]:

$\left\\{\begin{split}&y_{i}=\gamma\hat{x_{i}}+\beta;\\\
&\hat{x_{i}}=\frac{x_{i}-\mu_{\mathcal{B}}}{\sqrt{\sigma_{\mathcal{B}}^{2}+\epsilon}};\\\
&\sigma_{\mathcal{B}}^{2}=\frac{1}{m}\sum_{i=1}^{m}(x_{i}-\mu_{\mathcal{B}})^{2};\\\
&\mu_{\mathcal{B}}=\frac{1}{m}\sum_{i=1}^{m}x_{i}.\end{split}\right.$ (2)

The input of BN is a mini-batch $\mathcal{B}=\\{x_{1\dots m}\\}$, and for each
value $x_{i}$ of $\mathcal{B}$, the output is denoted as
$y_{i}=\text{BN}_{\gamma,\beta}(x_{i})$. The $\mu_{\mathcal{B}}$ and
$\sigma_{\mathcal{B}}^{2}$ represent the mean and variance value of mini-
batch, respectively. The $\hat{x_{i}}$ is the result of normalization.
Moreover, the parameter $\gamma$ and $\beta$ need to be learned in the back
propagation process, which reflect the mean and variance of the whole training
dataset and can help to scale and shift the normalized $\hat{x_{i}}$.

###  IV-C ResNet-CNN classifier for STFT feature map identification

In this paper we develop a 72-layers ResNet, as shown in Fig. 2. The input of
network is 128*128 STFT feature map which has been introduced in Section IV-A.
For each orange rectangle, which represents the convolutional layer, the first
number denotes the size of filter, the second denotes the number of filters,
and the last number denotes stride. The internal structure of Resnet block is
shown in Fig. 2. For Resnet block I, II, and III, the numbers of filters,
i.e., the values of parameter $c\\_num$, are equal to 128, 256, and 512,
respectively. Note that the shortcut connection of the upper-half part
includes dimension reduction implemented by a convolutional layer (stride=2).
Lastly, the traditional fully connected layer is replaced with the global
average pooling layer, thus the feature map is directly fed into the softmax
layer, assisting to reduce network parameters.

Figure 2: The network structure of ResNet-CNN. (a) shows the complete
structure, where the input is STFT feature map and the output is
classification result. (b) shows the internal structure of Resnet block which
decreases the height and width dimensions of input but increases the depth
dimension.

##  V Experiment Results

###  V-A Experimental setup

####  V-A1 Baselines

We employ the newly proposed work in [9] as baseline which has been verified
to achieve good accuracy performances on the raw DroneRF dataset. It developed
a 1D-CNN network and recommended to extract power spectral density (PSD)
features for the raw signals. For brief descriptions, we simply denote this
baseline as 1D-PSD, and denote our recommended algorithm as ResNet-STFT below.
The formulation of PSD is as follows, where $x_{N}$ denotes $N$ sampling
points:

$P(w)=\frac{1}{N}\left\|\sum_{n=0}^{N-1}x_{N}(n)e^{-jwn}\right\|^{2}.$ (3)

####  V-A2 Metrics

We use the common metrics, i.e., accuracy, precision, recall, and F1-score, to
evaluate the classification performances. Especially, for the multi-class
classification, we calculate ‘one vs rest’ separately for each class and
choose the average value to denote the overall performance. Moreover, we will
also show the confusion matrix of each classifier. The formulations can be
depicted as follows:

$\begin{split}\text{Accuracy}&=\frac{TP+TN}{TP+TN+FP+FN};\\\
\text{Precision}&=\frac{TP}{TP+FP};\\\ \text{Recall}&=\frac{TP}{P};\\\
\text{F1-score}&=\frac{2*\text{Precision}*\text{Recall}}{\text{Precision}+\text{Recall}}\end{split}$
(4)

####  V-A3 Experiment settings

The experiments have been carried out on MATLAB. STFT feature maps are
extracted from the raw data by spectrogram function. We use
deepNetworkDesigner toolbox to design and analyze the deep networks. The
network training function is trainNetwork, where we set: optimizer$->$‘adam’,
initial learn rate$->$0.0001, and L2 regularization factor$->$0.0001.
Considering different scales of cases which have been defined in Section
III-C, for the experiments based on the raw dataset, we set ‘MiniBatchSize’=8
and ‘MaxEpochs’=50, while for those based on the extended dataset, we set
‘MiniBatchSize’=32 and ‘MaxEpochs’=5. The loss function is cross entropy.

We repetitively carry out each experiment for five times, and use the average
as the final results. In each training-testing process, we divide the whole
dataset into three sections, i.e., 80% training data, 10% validation data, and
10% testing data.

###  V-B Experiment I: Classifications on raw dataset

This experiment compares the performances of our proposed ResNet-STFT with
baseline 1D-PSD on Case I, Case II-A, and Case III. Specifically, the cases
are related to two, four, and ten-class classifications, respectively. The
results can be found in Table II.

In binary classification, two models both can achieve no error, i.e., always
accurately detect whether the sample contains drone signals. In complex multi-
class classifications, there comes some decline on accuracy, especially in
ten-class classification. The factors that contribute to this situation
include: 1) the small number of samples in each class is not sufficient for
training; 2) correspondingly, the overfitting problem appears in deep network
which has significant affects on the testing dataset. In the future work, we
will further develop DroneRF dataset, such as adding noise, to compensate this
problem.

TABLE II: Accuracy and F1-score of Two, Four, and Ten-Class classifications | Case I | Case II-A | Case III
---|---|---|---
| Accu | F1-score | Accu | F1-score | Accu
ResNet-STFT | 1 | 1 | 0.985 | 1 | 0.682
1D-PSD | 1 | 1 | 0.955 | 0.98 | 0.697

###  V-C Experiment II: classifications on extended dataset

This experiment compares the performances of our proposed ResNet-STFT with
baseline 1D-PSD on Case II-B and Case II-C. Specifically, the cases are
related to three and seven-class classifications, respectively. The results
can be found in Table III.

In Case II-B, our network can precisely identify which two types of drones are
coexisting. The accuracy and f-score performances of ResNet-STFT are a bit
better than those of 1D-PSD. In Case II-C, the detection algorithm needs to
identify which type(s) of drones it contains, i.e., none, single three types,
or coexisting in pairs. The accuracy of ResNet-STFT can achieve 98.7%, which
also takes less time to converge. As shown in Fig. 3, the accuracy and loss
curves of ResNet-STFT quickly converge into relatively ideal levels in the
first epoch. Comparatively, after 5 epochs 1D-CNN still cannot well converge
and merely reaches 81.1% accuracy.

Moreover, we also show the confusion matrix figures in Fig. 4. ResNet-STFT
mistook two samples of class 2 into class 5, which represent single Bebop and
Bebop & AR respectively. However, 1D-PSD seriously confused class 5 and class
6, which represent Bebop & AR and Bebop & Phantom respectively. The precision
of class 5 and class 6 are equal to 74% and 82.1%.

Thus ResNet-STFT has potential of identifying complex energy signals. On the
one hand, STFT features catch more time-domain information which can perform
as a good supplement to frequency-domain. Especially when several signals
overlap, excavating the correlations between time-domain and frequency-domain
will be significant. On the other hand, ResNet-STFT is much deeper than
1D-PSD, but its scale of parameters does not explosively increase. On the
contrary, it can achieve faster convergence compared with baseline.

TABLE III: Accuracy and F1-score of Three and Seven-Class classifications | Case II-B | Case II-C
---|---|---
| Accuracy | F1-score | Accuracy | F1-score
ResNet-STFT | 1 | 1 | 0.987 | 1
1D-PSD | 0.947 | 0.98 | 0.811 | 0.924

Figure 3: The loss and accuracy curves of training and validation process. (a)
shows performances of ResNet -STFT, and (b) shows performances of 1D-PSD.
‘Training with MA’ denotes the moving average (MA) results of every five
Training points. There are 190 iterations contributed to 5 epochs in total.

Figure 4: The confusion matrix of ResNet-STFT (a) and 1D-PSD (b). The column
on the far right of the figure shows the percentages of all the examples
predicted to belong to each class that are correctly classified, i.e.,
precision. The row at the bottom of the figure shows the percentages of all
the examples belonging to each class that are correctly classified, i.e.,
recall. The cell in the bottom right of the figure shows the overall accuracy.

##  VI Conclusions

As drones become increasingly prevalent in human life, ensuring the ability to
accurately detect and identify between different drones holds significant
implications for public safety. We selected a common dataset DroneRF to verify
our algorithm. Moreover, based on the raw dataset, we also considered the
extended condition that there were two types of drones coexisting. We first
utilized Short-Time Fourier Transform (STFT) to extract two-dimensional
features from the raw signals, which contained both time-domain and frequency-
domain information. Then, we employed a Convolutional Neural Network (CNN)
built with ResNet structure to achieve multi-class classifications. Our
experimental results showed that the proposed ResNet-STFT could achieve higher
accuracy and faster convergence on the extended dataset, especially 98.7%
accuracy in seven-class classification. Additionally, it exhibited balanced
performance compared to other baselines on the raw dataset.

In the future, we will further develop DroneRF dataset, such as adding noise,
to compensate the problem that too small numbers of samples can be trained in
ten-class classification. Moreover, we also consider employ and cascade other
efficient machine learning classifiers after extracting abstract features by
our ResNet-STFT.

## References

  * [1] Zhang, H., Xiao, Y., Tang, W. et al, “Directional modulation with distributed receiver selection for secure wireless communications, ” _Science China Information Sciences_ , vol. 64, no. 12, pp. 222303, Nov. 2021.
  * [2] Chen, J., Lei, X., Xiao, Y. et al, “Polarized spatial and directional modulation toward secure wireless transmission,” _Science China Information Sciences_ , vol. 65, no. 7, pp. 179304, June. 2022.
  * [3] W. Nie, Z. -C. Han, M. Zhou, L. -B. Xie and Q. Jiang, “UAV Detection and Identification Based on WiFi Signal and RF Fingerprint,” _IEEE Sensors Journal_ , vol. 21, no. 12, pp. 13540-13550, June. 2021.
  * [4] O. O. Medaiyese, M. Ezuma, A. P. Lauf and A. A. Adeniran, “Hierarchical Learning Framework for UAV Detection and Identification,” _IEEE Journal of Radio Frequency Identification_ , vol. 6, pp. 176-188, Mar. 2022.
  * [5] Allahham, MHD Saria, et al, “DroneRF dataset: A dataset of drones for RF-based detection, classification and identification,” _Data in brief_ , vol. 26, pp. 104313, 2019.
  * [6] Al-Sa’d, Mohammad F., et al, “RF-based drone detection and identification using deep learning approaches: An initiative towards a large open source drone database,” _Future Generation Computer Systems_ , vol. 100, pp. 86-97, 2019.
  * [7] M. S. Allahham, T. Khattab and A. Mohamed, “Deep Learning for RF-Based Drone Detection and Identification: A Multi-Channel 1-D Convolutional Neural Networks Approach,” in _Proc. 2020 IEEE International Conference on Informatics, IoT, and Enabling Technologies (ICIoT)_ , Doha, Qatar, pp. 112-117, Feb. 2020.
  * [8] K. Raina, T. Alladi, V. Chamola and F. R. Yu, “Detecting UAV Presence Using Convolution Feature Vectors in Light Gradient Boosting Machine,” _IEEE Transactions on Vehicular Technology_ , vol. 72, no. 4, pp. 4332-4341, Apr. 2023.
  * [9] K. N. Inani, K. S. Sangwan and Dhiraj, “Machine Learning based framework for Drone Detection and Identification using RF signals,” in _Proc. 2023 4th International Conference on Innovative Trends in Information Technology (ICITIIT)_ , Kottayam, India, pp. 1-8, Feb. 2023.
  * [10] O. O. Medaiyese, A. Syed and A. P. Lauf, “Machine Learning Framework for RF-Based Drone Detection and Identification System,” in _Proc. 2021 2nd International Conference On Smart Cities, Automation & Intelligent Computing Systems (ICON-SONICS)_, Tangerang, Indonesia, pp. 58-64, Oct. 2021.
  * [11] A. Krizhevsky, I. Sutskever and G. E. Hinton, “Imagenet classification with deep convolutional neural networks”, _Advances in neural information processing systems_ , vol.25, 2012.
  * [12] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition”, _arXiv:1409.1556_ , 2014.
  * [13] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, et al., “Going deeper with convolutions”, in _Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 1-9, 2015.
  * [14] He K, Zhang X, Ren S, et al, “Deep residual learning for image recognition,” in _Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 770-778, 2016.
  * [15] Ioffe S, Szegedy C. “Batch normalization: Accelerating deep network training by reducing internal covariate shift.” in _Proc. International conference on machine learning. pmlr_ , pp. 448-456, 2015.