∎

11institutetext: Gaoshuai WANG, Amir HAJJAM EL HASSANI 22institutetext:
Nanomédicine, Imagerie, Therapeutique - EA4662, Université Bourgogne Franche-
Comté, UTBM, F-90010 Belfort, France
22email<EMAIL_ADDRESS><EMAIL_ADDRESS>33institutetext: Fabrice Lauri 44institutetext: Connaissance et Intelligence
Artificielle Distribuées (CIAD) - UMR 7533, Université Bourgogne Franche-
Comté, UTBM, F-90010 Belfort, France
44email<EMAIL_ADDRESS>
# A GA-like Dynamic Probability Method With Mutual Information for Feature
Selection

Gaoshuai WANG  Fabrice Lauri  Amir HAJJAM EL HASSANI

(Received: date / Accepted: date)

###### Abstract

Feature selection plays a vital role in promoting the classifier’s
performance. However, current methods ineffectively distinguish the complex
interaction in the selected features. To further remove these hidden negative
interactions, we propose a GA-like dynamic probability (GADP) method with
mutual information which has a two-layer structure. The first layer applies
the mutual information method to obtain a primary feature subset. The GA-like
dynamic probability algorithm, as the second layer, mines more supportive
features based on the former candidate features. Essentially, the GA-like
method is one of the population-based algorithms so its work mechanism is
similar to the GA. Different from the popular works which frequently focus on
improving GA’s operators for enhancing the search ability and lowering the
converge time, we boldly abandon GA’s operators and employ the dynamic
probability that relies on the performance of each chromosome to determine
feature selection in the new generation. The dynamic probability mechanism
significantly reduces the parameter number in GA that making it easy to use.
As each gene’s probability is independent, the chromosome variety in GADP is
more notable than in traditional GA, which ensures GADP has a wider search
space and selects relevant features more effectively and accurately. To verify
our method’s superiority, we evaluate our method under multiple conditions on
15 datasets. The results demonstrate the outperformance of the proposed
method. Generally, it has the best accuracy. Further, we also compare the
proposed model to the popular heuristic methods like POS, FPA, and WOA. Our
model still owns advantages over them.

###### Keywords:

Genetic Algorithm Dynamic Probability Feature Selection Mutual Information

††journal: Applied Intelligence

##  1 Introduction

Feature selection is a non-negligible pre-processing step when dealing with
high-volume or high-dimensional datasets Sorzano et al. [2014]. It has been
extensively used in various domains, such as image process Too and Mirjalili
[2021], Sun et al. [2020], data clustering Purushothaman et al. [2020], Abasi
et al. [2021], Zhong and Pun [2020], data classification Peng et al. [2010],
Tang et al. [2014]. By eliminating the irrelevant features and reducing the
dataset’s feature scale, it can promote the classifier’s performance and
convergence speed Lin et al. [2015]. In total, we categorize feature selection
methods into three parts, typically, they are filter methods, wrapper methods,
and embedded methods Li et al. [2017]. Filter methods choose a certain number
of features based on criteria like correlation, similarity, information gain,
etc; Wrapper methods attempt to evaluate all feature combinations with the
classifier which needs lots of computing resources; The embedded methods
integrate feature selection and classifier learning into a single process.
Filter methods usually are not as computationally expensive as wrapper
methods. While filters lack the evaluation of the selected features’
performance on a classifier, wrappers contain a classifier as a part of it,
which leads to a better result by wrappers than filters Xue et al. [2015].

The appearance of Shannon entropy theory Shannon [1948] provides a series of
powerful methodologies. Methods like the decision tree and mutual information
are designed based on Shannon’s theory. As noted above, the mutual information
method should belong to the filter method, which assesses a feature’s
importance by information gain. If the existence of one feature can
tremendously alleviate the uncertainty of the class label, it represents the
feature is in favor of the classifier to discriminate the classification. In
recent years, it has gained much attention, and lots of works have
demonstrated mutual information’s effectiveness Vergara and Estévez [2014].
Initially, researchers directly rank the candidate features and adopt the
features with top-ranked mutual information as the input of the classifier.
However, this method ignored the redundancy existing in features, which will
harm the former mutual information selection. Fig. 1 depicts the accuracy
growth tendency to the increment of subset features. The accuracy curve’s
fluctuation means some features are redundant. Hence, MRMR (minimum redundancy
maximum relevance) is proposed by Ding et al. Ding and Peng [2005] which
considers both relevancy and redundancy and achieved huge success. It also
inspired researchers to develop more similar methods.

Nevertheless, after summarizing mutual information methods, we found that they
only locally release some interactions, for example, these methods often
employ the sum of mutual information between the candidate feature and
individual feature to replace the mutual information between the candidate
feature and the whole of the selected subset. Namely, it’s not a perfect
replacement and remains some redundancies.

(a) KNN

(b) GB

(c) SVM

Figure 1: Variation trend of accuracy with features.

Except for the mutual information methods, researchers also attempt to
introduce the heuristic methods such as genetic algorithm (GA) Holland [1973],
particle swarm optimization (PSO) Eberhart and Kennedy [1995] to select
optimal features Xue et al. [2015], Kar [2016]. Heuristic methods are a series
of algorithms inspired by creatures and often used in NP-hard problems.
Commonly, the heuristic method owns a strong global search ability which
assures they can perform well Chak and Feng [1995]. GA is one of the high
frequently applied heuristic methods, which is from the biological evolution
process Michalewicz and Schoenauer [1996]. A classical GA comprises several
elements, such as chromosome, gene, fitness function, crossover, and mutation.
The representative flow generally goes through: initialize the chromosome;
select the chromosome by a certain fitness function, generate the new
chromosome, crossover, and mutation, repeatedly. The whole process will end
until the result converges or reaches the threshold. In most cases, GA method
is a feature selector with some classifiers for searching in a global space.
However, directly employing GA to choose features still has some defects: it
costs too much computation time and hardly makes a balance between local
search and global search El-Mihoub et al. [2006]. Besides, there are plenty of
parameters lying in the traditional GA methods which adversely increase the
usage difficulty Hansen et al. [2015].

Considering the characteristics of filter and wrapper methods, the hybrid
algorithm has increasingly gained attraction from researchers and various
combined methods are constantly emerging. Admiring the merits of the hybrid
algorithm, we propose a novel GA-like dynamic probability algorithm with
mutual information for feature selection. In our method, the mutual
information method is employed to rank all features and then selects top-$N$
features. It can be treated as a global feature selector. Then, the GA-like
method exploits further search locally and offers the selected features to a
classifier. In terms of the GA-like method, it abandons the crossover and
mutation operators and generates the new chromosome depending on each gene’s
probability. It’s easier for setting parameters and strengthen our proposed
method’s local search ability.

In this research, our contributions are:

  1. 1)

We propose a two-layer feature selection model which both considers the global
search and local search.

  2. 2)

To accelerate the feature selection, we employ the MRMR method as the global
selector.

  3. 3)

We propose a new GA-like dynamic probability method that is simpler compared
to GA for searching features locally.

  4. 4)

Dynamic probability update mechanism replaces the complex operators in GA.

  5. 5)

Lots of experiments are exploited to demonstrate the characteristics and
performance of the proposed method.

The rest of this paper is arranged as follows. In Section 2, we present some
related works on mutual information and heuristic algorithm in feature
selection. Section 3 presents the main concept of the GA method and other
methods. Section 4 detailedly introduces our proposal GA-like method and the
whole process of feature selection. In Section 5, the datasets and processing
method are described. After, we compare our method to the classical GA method
with various conditions. Finally, Section 6 concludes this paper and refers to
the possible further work.

##  2 Related works

Numerous works have been done since the effectiveness of genetic algorithms
and mutual information algorithms. This section will present researchers’
efforts at improving and applying them in various domains.

As for the genetic algorithm, researchers often improve its crossover and
mutation operators which have a profound effect on its performance. Xue et al.
Xue et al. [2021] proposed an adaptive crossover operator based on a genetic
algorithm for feature selection. In their method, they adopted five crossover
operators with proper probability and each of them will be selected in
different process stages because they consider the single crossover operator
is problem-dependent. Their method was compared with five well-known
evolutionary multi-objective algorithms on ten datasets. The results
demonstrated that the proposed algorithm can remove redundant features and
keep low classification error and is superior to other methods. Manzoni et al.
Manzoni et al. [2020] used three different balanced crossover operators in
cryptography and design theory for the sake of their balancedness constraint.
They found that the balanced crossover operators perform better than the one-
point crossover. Sun and Liu Sun and Lu [2019] considered that premature
convergence often happened on classical genetic algorithms due to the rapid
multiplication of outstanding individuals. They put up a method to qualify the
diversity and similarity between adjacent generations and adjust the crossover
and mutation probability dynamically. Their method is proven that greatly
improved the convergence speed and global search ability. In Amini and Hu
[2021], GA was used to select features directly for the second elastic net
which was flexible in adjusting the penalty terms in the regularization
process and time efficiency. Moreover, researchers also explored the
initialization Mustafi and Sahoo [2019], Gupta and Ghafir [2012], encoding/
decoding schemes Kumar [2013], Costa et al. [2010] and other aspects.

About mutual information, reducing irrelevant features, and keeping supportive
features are the core of researchers’ work. Maes et al. Bahl et al. [1986]
applied the MIM method for estimating the parameters of the hidden Markov
model in 1986. It promoted the probability largely when parameters were
estimated by maximum likelihood estimation. However, the feature relevancy
ignored by MIM makes mutual information still has a huge improvement space.
Battiti Battiti [1994] proposed the mutual information feature selection(MIFS)
which considers both the relevancy and redundancy. $\beta$ represents the
parameter of the feature redundancy. But, it’s difficult to choose a proper
value of $\beta$. The same problem also happens on its variants, MIFS-ND and
MIFS-U Hoque et al. [2014], Kwak and Choi [2002]. To balance the relationship
between feature relevancy and feature redundancy, Pent et al. Peng et al.
[2005] set the $\beta$ equal to the inverse of the selected feature scale in
MRMR. After that, other methods such as conditional information feature
extraction (CIFE) Lin and Tang [2006], double input symmetrical relevance
(DISR) Meyer et al. [2008], and Joint mutual information (JMI) Yang and Moody
[1999] are brought up which emphasised different aspects.

Except for the single genetic algorithm and the single mutual information
method, the combined methods of genetic algorithm and mutual information
attracted many researchers’ attention. Jadhav et al. Jadhav et al. [2018]
carried out the information gain and genetic algorithm in credit rating. The
gained information availed reducing the computing complexity and the genetic
algorithm further promoted the classifiers’ performance. Salesi et al. Salesi
et al. [2021] used a two-layer framework that combined the MRMR method and
tabu asexual genetic algorithm to enhance the generalization power. The
results show that their algorithm was superior to other conventional and
state-of-art algorithms. Huang et al. Huang et al. [2007] made the genetic
algorithm as the first layer and mutual information as the second layer. Their
experimental results show that the hybrid method achieved excellent
classification accuracy.

##  3 Preliminaries

Darwin’s theory of evolution by natural selection inspires the appearance of
GA that mimics nearly all the stages of gene heredity. Generally, gene
heredity is a complex process in biological view, which mainly takes account
of several vital parts: such as chromosome representation, crossover,
mutation, and fitness selection. Chromosome representation transfers the
specific problem into chromosome format; Crossover indicates the parent
chromosomes change their part gene in the same position and produce new
chromosomes; Mutation represents some positions of the chromosomes are
randomly flipped on a certain probability; Fitness selection will discard or
change gene’s existence probability in the next off-springs, which assures the
gene’s environmental adaptability.

Fig. 2 depicts several relative elements and processes in GA. The color bar
means a chromosome where several genes exist.

  1. 1)

Chromosome representation: In the beginning, we need to transform our specific
problem into a chromosome which is an essential step because it determines the
successful application of GA. Different from the creature which usually
contains dozens of chromosomes, in GA, one chromosome represents the whole
resolution space, numerous chromosomes will form a population. In terms of
feature selection, we can regard each feature as a chromosome gene located in
a fixed position. A feature only has two states: be chosen or not, in Fig. 2,
color block means the gene is selected. There are lots of chromosomes in each
process which assures the global search ability of GA.

  2. 2)

Crossover: When the new generation chromosomes are produced, the crossover
will happen among the chromosomes, for example, in Fig. 2, Step1 to Step2, two
parents’ chromosomes generate two children. G1, G4, and G6 are exchanged
between the parents. Crossover increasingly enhances solution space’s variety
and avoids the local optimum. But, it also adds more parameters in the setting
process, the crossover proportion, and position.

  3. 3)

Mutation: Mutation means a novel gene appearing or some changes beyond the
crossover process. In Fig. 2, the mutation process exists in Step2 to Step3.
G7 disappears in one chromosome and a new gene appears in the G8 position. It
must be emphasized that there is no new gene’s appearance in the feature
selection problem.

  4. 4)

Fitness selection: After the previous processes, fitness selection evaluates
each individual’s performance. In the natural environment, an excellent gene
will strengthen the individual’s survival ability, then, its heredity
probability increases, correspondingly, and the population gradually owns the
adaptability. In GA, when dealing with the fitness result, people usually take
two kinds of actions: One is discarding the bad performance individuals
directly; Another is reducing their chances of reproduction.

Loop these four steps repeatedly until trigger the terminal conditions.

Figure 2:  Chromosome, gene and operators.

PSO was brought up by Kennedy and Eberhart. The basic elements named particles
are placed in the search space and the fitness function evaluates its current
location. Then, each particle decides its velocity and direction in the space
relying on its current location and the best location with some members of the
swarm. All the particles that have been moved are a complete iteration.
Eventually, all the particles like a flock of birds seek food until they find
the fitness function‘s optimum.

Flower Pollination Algorithm (FPA) Yang [2012] was inspired by the natural
pollination process. FPA incorporates biotic cross-pollination and abiotic
pollination as the global search and local search processes. Pollen-carrying
pollinators such as insects or the wind move in a way that obeys Levy flights.
Besides, the switching of local pollination and global pollination is
controlled by the probability $p\in[0,1]$. The whole process intimates the
plants’ pollination.

The whale optimization algorithm (WOA) Mirjalili and Lewis [2016] mimics the
social behavior of humpback whales. The way that whales hunt their prey is
named the bubble-net feeding method. In WOA, each solution is regarded as a
whale. The whale moves its place in the search space considered as a reference
to the best element of the group. Two search mechanisms are used by the whales
to hunt. In the first one, the preys are encircled and the second is the
bubble net.

##  4 Proposed method

Although GA has a puissant ability for global search, it’s exhaustive when the
scale of the dataset’s dimensionality increases to several thousand. It’s not
easy for GA to obtain a pretty good result. Considering the high effectiveness
of filter methods such as mutual information, we combine these two methods and
form a two-layer hybrid method. In mutual information methods proposed with
different considerations, we found that MRMR often is one of the baselines and
its performance is well. Thus, in the first layer, we adopt the MRMR method
for ranking all the candidate features and select top-$N$ features as the
second layer’s input. Next, the proposed GA-like method with a specific
classifier to search for optimal combinations until it triggers some
termination conditions.

Let $X=\\{x_{1},x_{2},...,x_{n}\\}$ be a feature vector and $p(x_{i})$ is the
probability of $x_{i}$, let $Y=\\{y_{1},y_{2},...,y_{n}\\}$ as the class
vector. The mutual information between $X$ and $Y$ is calculated in Eq. 1:

$I(X,Y)=H(X)-H(X|Y)$ (1)

input : The traning dataset $D$ with original feature set
$F=\\{f_{1},f_{2}...f_{n}\\}$, class label $Y$ and the required feature number
$T=70$

output : The selected subset feature $S_{T}$

1

2 _$S_{T}\leftarrow$ $\varnothing$_;

3 foreach  _ $f_{i}$ in $F$_ do $MI_{i}=I(f_{i};y)$;

4 _$f\leftarrow\ max(\ MI)$_ ;

5 _$S_{T}\cup f$ _;

6 _$F=F-f$_ ;

7 for  _$i\leftarrow 2$ to $T$_ do

8

9 for  _each $f_{i}\ in\ F$_ do

10 $J(f_{i})$ = $MI_{i}$ \- $\frac{1}{|S_{T}|}\sum_{f_{j}\in
S_{T}}I(f_{j};f_{i})$;

11

12  _select $f_{s}$ from $J(f_{i})\ with\ the\ largest\ value$_;

13 _$S_{T}=S_{T}\cup f_{s}$_ ;

14 _$F=F-f_{s}$_ ;

return $S_{T}$

Algorithm 1 MRMR Ding and Peng [2005]

###  4.1 GA-like dynamic probability method

GA has already been applied in many areas because it can obtain pretty results
in a short time even with a complex task. In GA, the typical processes are
crossover, mutation, selection, and reproduction. The crossover and mutation
make the off-springs inherit merits from their parent chromosomes and surpass
them in variety. Meanwhile, there is an inevitable issue that we should have a
proper change rate and position during the crossover and mutation processes
which easily lead to results unpredictable. Hence, to discover better
operators, researchers designed lots of rules in crossover and mutation
processes, such as the single point crossover, uniform crossover, cycle
crossover; Inversion mutation, the bit flipping mutation, etc. What’s more,
the selection method is also mandatory when the algorithm produces a new
generation. As mentioned above, the GA parameters setting is so complex that
researchers need to spend much attention on this part.

Furthermore, we think that GA could not balance the trade-off well locally and
globally as there is no feature pre-screening, which will hinder its
application. For simplifying the traditional GA, we propose a GA-like method
based on a dynamic probability mechanism. In the whole process, the birth of
each new generation is according to their probability.

  1. 1.

Encode and Decode. The feature selection problem is simple to encode, we just
employ the binary encode to process the feature subset. Its merit is that each
feature is independent which assures chromosome variety. In the binary encode,
‘1’ represents the feature participating to classify the label; ‘0’ indicates
the feature is not selected. Decode is transforming the binary chromosome to
the feature subset. We define the $i$ th chromosome in population as
$\theta_{i}$, the $g$ th gene or feature in chromosome $\theta_{i}$ as
$\theta_{i}^{g}$. $P_{t}(\theta_{i}^{g})$ is the probability of $i$ th
chromosome’s $g$ th gene at $t$ th generation.

  2. 2.

Initialization. Initialization demonstrates whether a feature should be
selected or not in the beginning. In most cases, GA adopts the random
initialization method. If the scale of the population is large enough,
chromosome variety enhances its global search ability. In our proposed method,
whether a feature appears in the chromosome depends on its probability. All
the features have the same probability. The too-large probability will degrade
the selection function and the too-small probability will decrease the variety
of chromosomes.

$P_{0}(\theta_{i}^{g})=0.65\qquad 0\leq i<n,0\leq g<m$ (2)

where $n$ is the number of population, $m$ is the amount of gene or feature in
one chromosome. $P_{0}$ is the probability at $0$ th generation.

  3. 3.

Fitness selection. In our method, we introduce the Gaussian bayesian (GB),
Support vector machine (SVM), and K-nearest neighbors (KNN) as the evaluation
function. The higher accuracy means the chromosome has a better performance.
In each Fitness selection, 1/3 population will be discarded. The rest of the
population will take part in the probability update. Among these populations,
The best performance subset will be kept. If a chromosome in the next
generation is better than the current chromosome, it will be replaced.

  4. 4.

Probability update. After the discarding process, relying on remained
chromosomes, each gene’s probability will be updated. If the existence of one
feature promotes accuracy, which indicates the feature is positive in
classification, its probability will increase in the next generation. The
probability $P_{t}(\theta_{i}^{g})$ is calculated as:

$E_{t-1}({\theta_{i}^{g}})=\begin{cases}1&\text{gene selected}\\\ 0&\text{gene
not selected}\end{cases}\\\ $ (3)

$P_{t}(\theta_{i}^{g})=(\sum_{i=1}^{\frac{2n}{3}}E_{t-1}({\theta_{i}^{g}}))/\frac{2n}{3}$
(4)

  5. 5.

Chromosome replication. Different from the popular chromosome replication
method where the new chromosome is produced from a pair of parents. In our
proposed method, there is no parent chromosome, all the genes or features only
rely on each probability.

To have a better understanding of the variety, we apply the information
entropy to calculate the information volume of each gene. At generation $t$,
the information entropy of gene $g$ and a chromosome are computed by Eq. 5 and
Eq. 6. In Fig. 3, we present one gene’s information entropy variation with its
probability.

$H_{t}(\theta^{g})=-P_{t}(\theta^{g})\ln(P_{t}(\theta^{g}))-(1-P_{t}(\theta^{g}))\ln(1-P_{t}(\theta^{g}))$
(5)

$H_{t}(\theta)=\sum H_{t}(\theta^{g})$ (6)

Figure 3:  The information entropy of gene $g$. Point 1 is the initial
information entropy. With the growth of generations, each gene’s probability
tends to be stable, these supportive genes’ information entropy will be close
to Point 2 and the redundant genes’ information entropy will reach Point 3.

In contrast to the classical GA, seemingly, their selection method, such as
the elite selection or probabilistic selection is similar to our proposed
method. Information entropy also can estimate GA’s information volume.
However, each gene’s probability is not independent on account of the
crossover mechanism that decreases off-springs uncertainty and the mutation
only increases the uncertainty limitedly. What’s more, the proposed method
needn’t keep the aforementioned elite generation chromosomes for obtaining
good results, all training process influences are in the probability. Thus, we
deem that our proposed method has a more wide search area. We display the
Pseudo-code of GADP in Algorithm 2.

1

input : Feature set $F=\\{f_{1},f_{2}...f_{n}\\}$, class label $Y$, population
set $N$, initial probability set $P$, iteration number $I$, threshold $t$,
discard rate $l$

output : The selected subset feature $S_{F}$

2

3$N.size\leftarrow 50$, $P.size\leftarrow len(F)$, $I\leftarrow 50$,
$t\leftarrow 10$ ;

4 Push $N.size$ empty chromosomes into population set $N$ ;

5 $P\leftarrow\\{0.65,0.65...0.65\\}$ ;

6 $i\leftarrow 0$, $Best_{set}\leftarrow\varnothing$ ;

7 while  _$i <I$_ do

8 foreach  _chromosome $\theta$ in population set $N$_ do

9 foreach  _gene $g$ in chromosome $\theta$_ do

$g\leftarrow 1$ at the probability of $P(g)$ // Produce new chromosome

10

11

12 foreach  _chromosome $\theta$ in population set $N$_ do

13 $F_{position}\leftarrow Decode(\theta)$;

14 $x_{train}\leftarrow F[:,F_{position}]$;

15 $E(\theta)\leftarrow EvulationFunction(x_{train},Y)$ ;

16

17 $E(\theta)\leftarrow Sort(E(\theta))$;

18 Discard $N/3$ chromosomes based on $E(\theta)$;

19 $Best_{set}\leftarrow$ best performance chromosome ;

20 Break the condition If $Best_{set}$ doesn’t update more than $t$ times
continuously;

21 foreach  _gene $g$ in one chromosome_ do

$P(g)\leftarrow Eq.~{}\ref{lable5}$ // Probability update

22

23 Remove all chromosomes and push $N.size$ empty chromosomes into population
set $N$ ;

24

return $Best_{set}$

Algorithm 2 Pseudo-code for GADP

Finally, we present the flowchart of our proposed method and classical GA with
mutual information process in Fig. 4. It could be brutally divided into two
search parts: global search and local search. MRMR works for the global search
part, which reduces the scale of search space largely; GADP is responsible for
further removing redundant features. Compared to the classical GA, its
structure is more compact.

Figure 4:  The flowchart of proposed method and classical GA

##  5 Experimental results and analysis

###  5.1 Metrics

We use the accuracy, F1 and AUC as our evaluation indicators, their
calculation formulas are as follows:

$Accuracy=\frac{TP+TN}{TP+TN+FP+FN}$ (7)

$Precision=\frac{TP}{TP+FP},Recall=\frac{TP}{TP+FN}$ (8)

$F1=\frac{2*Precision*Recall}{Precision+Recall}$ (9)

$AUC=\int_{0}^{1}t_{pr}(f_{pr})df_{pr}=P(X1>X0)$ (10)

where $TP,TN,FP$ and $FN$ are the true positive, true negative, false positive
and false negative, respectively. The $t_{pr}$ is the true positive rate,
$f_{pr}$ is the false positive rate, and $P(X1>X0)$ are the probability of the
confidence scores $X1>X0$ for a negative and positive instance. The AUC value
describes the probability that a classifier ranks a randomly chosen positive
instance higher than a randomly chosen negative one.

###  5.2 Dataset and pre-processing

To verify the generalization of our proposed method, we employ 15 datasets to
assess its performance. We summarize their characteristics of instance,
feature, and classes in Table 1. Three datasets are the binary classification
task, and the rest of them are the multi-classification task. Some datasets
are continuous types, which will hinder the accuracy of mutual information
selection because continuous types could paralyze mutual information
calculation. Therefore, we need to redivide all feature attributes into 10
intervals. The first step is normalizing the attribute as Eq. 11. Afterward,
multiply $x^{*}$ by 10, so all the attributes will fall into the 10 regions.
It’s a necessary process when using mutual information in a continuous
dataset. K-fold cross-validation is used in this research, $K=5$.

Table 1: The summary of benchmark datasets

Number | Name | Instances | Features | Classes | Notes
---|---|---|---|---|---
1 | PCMAC | 1943 | 3289 | 2 | {1: 982, 2: 961}
2 | Isolet | 1560 | 617 | 26 | balanced
3 | GLIOMA | 50 | 4434 | 4 | {4: 15, 1: 14, 3: 14, 2: 7}
4 | Prostate_GE | 102 | 5966 | 2 | {2: 52, 1: 50}
5 | RELATHE | 1427 | 4322 | 2 | {1: 779, 2: 648}
6 | TOX_171 | 171 | 5748 | 4 | {1: 45, 2: 45, 4: 42, 3: 39}
7 | USPS | 9298 | 256 | 10 | {1: 1553, 2: 1269, 3: 929, 5: 852, 7: 834, 4: 824, 10: 821, 8: 792, 6: 716, 9: 708}
8 | CNAE-9 | 1079 | 855 | 9 | {7: 120, 6: 120, 5: 120, 4: 120, 8: 120, 3: 120, 2: 120, 9: 120, 1: 119}
9 | lung_discrete | 73 | 325 | 7 | {7: 21, 4: 16, 6: 13, 5: 7, 1: 6, 3: 5, 2: 5}
10 | warpPIE10P | 210 | 2420 | 10 | balanced
11 | lung | 203 | 3312 | 5 | {1: 139, 3: 21, 4: 20, 2: 17, 5: 6}
12 | COIL20 | 1440 | 1024 | 20 | balanced
13 | warpAR10P | 130 | 2400 | 10 | balanced
14 | Yale | 165 | 1024 | 15 | balanced
15 | Brain_Tumor_1 | 90 | 5920 | 5 | {5: 60, 1: 10, 2: 10, 4: 6, 3: 4}

$x^{*}=\frac{x-x_{min}}{x_{max}-x_{min}}$ (11)

###  5.3 Experimental setting

Some of our datasets are unbalanced, thus, we use the macro averaging metrics
to evaluate the algorithm’s performance. In the mutual information process, we
only select the first 70 features provided to GADP and other compared
algorithms. Essentially, GA belongs to the wrapper method, the classifier type
has a certain influence on the algorithm’s performance, we adopt GB, KNN
(k=3), and SVM algorithms as classifiers. What’s more, we compare GADP with GA
and other heuristic methods, such as PSO, FPA, and WOA. The hyperparameters
and initial settings are described in Table 2. We apply the average value and
standard deviation to implement a paired two-tailed t-test with other methods
or settings. ”+”, ”=” and ”-” mean that the objective algorithm performs
”better than ”, ”equal to”, and ”worse than” the corresponding method When the
P-Value is less than 5%. W/T/L indicates the win/tie/loss statistics of one
method relative to another method.

Table 2: The parameter setting of heuristic algorithm

Name | Parameter Setting
---|---
GADP | max_feature = 50, initial probability = 0.65, population = 50, max_iteration = 50
GA | max_feature = 50, population = 50, max_iteration = 50
PSO | particles = 50, max_iteration= 50
FPA | search_agents = 50, max_iteration = 50
WOA | search_agents = 50, max_iteration = 50

###  5.4 The effect of MRMR selection

To evaluate the effect of MRMR, we compare the prediction results of MRMR and
the full features based on GADP. After primary selection, only the top 70
features remain. In Table 3, we can see that the accuracies of MRMR are higher
than the full features on three classifiers except for a few cases. It denotes
that eliminating redundant features is effective and MRMR can select more
supportive features to promote the algorithm’s performance. Besides,
generally, we regard that a smaller feature scale should have a smaller time
cost. In most cases, MRMR indeed spends less time because the search space of
the full features is wider which makes the algorithm converge slower. As for
the full features cases with higher accuracies, we can notice that their time
cost usually is higher. This is because searching for better results needs
more time. The phenomenon also appears on the MRMR feature subset. For
example, COIL20 in GB and KNN.

Table 3: The influence of MRMR selection

Name |  GB(%) |  KNN(%) |  SVM(%)
---|---|---|---
GADP |  | GADP |  | GADP |  |
| MRMR | full features | $T_{MRMR}(s)$ | $T_{full}(s)$ | MRMR | full features | $T_{MRMR}(s)$ | $T_{full}(s)$ | MRMR | full features | $T_{MRMR}(s)$ | $T_{full}(s)$
PCMAC | 83.53±4.21 | 61.40±1.60 (+) | 7 | 39 | 87.65±1.31 | 61.40±1.60 (+) | 234 | 243 | 89.19±0.95 | 63.97±1.06 (+) | 74 | 168
Isolet | 70.96±1.17 | 56.67±1.07 (+) | 23 | 26 | 72.18±3.65 | 56.67±1.07 (+) | 173 | 274 | 76.47±1.23 | 62.24±1.63 (+) | 166 | 612
GLIOMA | 91.99±6.81 | 83.97±6.84 (+) | 2 | 13 | 87.98±4.66 | 83.97±6.84 (+) | 2 | 12 | 83.97±6.32 | 78.21±6.70 (+) | 35 | 176
Prostate_GE | 94.19±6.65 | 65.81±10.77 (+) | 7 | 12 | 96.15±5.44 | 65.81±10.77 (+) | 7 | 29 | 93.23±6.55 | 87.42±10.55 (+) | 32 | 132
RELATHE | 82.06±3.04 | 67.34±1.32 (+) | 9 | 10 | 81.08±1.25 | 67.34±1.32 (+) | 212 | 219 | 83.39±1.65 | 68.68±0.58 (+) | 54 | 203
TOX_171 | 78.99±4.87 |  79.58±6.83 (=) | 22 | 74 | 73.09±4.10 |  79.58±6.83 (-) | 33 | 118 | 70.17±12.67 |  83.67±6.77 (-) | 48 | 243
USPS | 76.61±0.48 |  86.57±0.72 (-) | 57 | 65 | 88.95±0.41 | 86.57±0.72 (+) | 427 | 131 | 95.92±0.24 | 90.93±0.61 (+) | 452 | 1653
CNAE-9 | 88.23±2.42 | 34.75±1.28 (+) | 9 | 13 | 86.93±0.54 | 34.75±1.28 (+) | 150 | 104 | 87.67±1.26 | 43.56±0.42 (+) | 65 | 92
lung_discrete | 83.48±8.00 | 80.85±7.04 (+) | 1 | 2 | 97.22±3.21 | 80.85±7.04 (+) | 1 | 2 | 94.59±4.30 | 85.01±6.62 (+) | 65 | 106
warpPIE10P | 92.88±5.40 | 81.40±5.15 (+) | 1 | 6 | 96.65±2.42 | 81.40±5.15 (+) | 26 | 74 | 98.57±1.84 | 95.71±1.83(+) | 40 | 42
lung | 97.55±1.88 | 94.61±3.71 (+) | 8 | 16 | 97.06±1.96 | 94.61±3.71 (+) | 53 | 105 | 96.08±3.20 | 90.65±1.84 (+) | 75 | 103
COIL20 | 90.97±1.85 | 71.74±2.06 (+) | 59 | 37 | 98.54±0.83 | 71.74±2.06 (+) | 305 | 276 | 98.96±0.57 | 96.25±1.12 (+) | 252 | 278
warpAR10P | 79.97±11.54 | 43.09±7.46 (+) | 1 | 5 | 83.83±6.94 | 43.09±7.46 (+) | 1 | 8 | 83.81±4.09 | 75.36±6.86 (+) | 60 | 72
Yale | 67.33±7.88 | 52.15±11.49 (+) | 1 | 2 | 66.71±7.74 | 52.15±11.49 (+) | 1 | 4 | 73.39±8.59 | 70.95±7.45 (+) | 94 | 105
Brain_Tumor_1 | 87.85±6.39 | 85.62±8.12 (+) | 4 | 15 | 86.66±3.57 | 85.62±8.12 (=) | 2 | 10 | 84.44±5.65 | 83.35±7.39 (=) | 28 | 132
W/T/L |  | 13/1/1 |  |  |  | 13/1/1 |  |  |  | 13/1/1 |  |

###  5.5 The influence of probability setting

Gene probability is the core of GADP, which determines the production of each
new chromosome. A proper probability setting is vital for GADP to search the
feature space and 3 pairs of probability are studied based on GADP and MRMR in
this part. We depict the tendency of partial chromosomes’ probability in Fig.
5. There are three pairs of probability thresholds, 0-1, 0.3-0.8, and 0.4-0.7.
We can note that, in the first subfigure, each gene’s probability reaches its
final value (0-1) after approximately 20 generations. When the selection
process is terminated, the results demonstrate that 2 of 7 genes are selected.
The earlier convergence of gene probability will decrease the search space, in
some way, which could lead to the local optimum.

Compared with the aforementioned figure, the probability fluctuation is
obvious in Fig. 5b, even lasting to the end. The probability could increase or
decrease from opposite values. Finally, we set the gene probability threshold
as the 0.4-0.7 pair in Fig. 5c, which makes converge slower and has a more
broad search space, thus, it will cost lots of time. In Fig. 5c, nearly all
the features are under an unstable situation, it seems that the fitness
function loses its effect.

Moreover, we also make a comparison of different probability thresholds in
Table 4. The accuracies without the probability limitation are better than
those with limitations. They achieve 15, 14 and 10 of 15 with the highest
accuracies on GB, KNN, and SVM. It seems that the advantages of 0-1 pair on
SVM are not as apparent as on GB and KNN. We think that the SVM has a strong
information mining capability which partly decimates the probability
threshold’s influence. About the time cost, 0-1 pair spends less average time
on GB and SVM. However, on KNN, the situation is different and this should be
caused by the influence of KNN’s characteristics. On the whole, we consider
that the 0-1 pair’s performance is better and adopt it in the rest of the
experiments.

(a) 0-1

(b) 0.3-0.8

(c) 0.4-0.7

Figure 5: Gene probability with the increment of generation Table 4: The
influence of gene probability

Name | GB | KNN | SVM
---|---|---|---
MRMR + GADP | MRMR + GADP | MRMR + GADP
0-1 | 0.3-0.8 | 0.4-0.7 | 0-1 | 0.3-0.8 | 0.4-0.7 | 0-1 | 0.3-0.8 | 0.4-0.7
PCMAC | 83.53±4.21 | 82.66±4.09 (+) | 81.42±4.79 (+) | 87.65±1.31 | 85.38±1.39 (+) | 86.67±1.58 (+) | 89.19±0.95 | 88.57±0.92 (+) | 88.73±0.71 (+)
Isolet | 70.96±1.17 | 68.65±1.92 (+) | 67.56±1.35 (+) | 72.18±3.65 | 70.71±3.27 (+) | 70.71±4.23 (+) | 78.01±1.83 | 76.92±0.89 (+) | 76.47±1.23 (+)
GLIOMA | 91.99±6.81 | 88.14±7.93 (+) | 88.14±7.93 (+) | 87.98±4.66 | 86.06±3.79 (+) | 87.98±10.03 (=) | 83.97±6.32 | 80.98±4.66 (+) | 82.97±6.32 (=)
Prostate_GE | 94.19±6.65 |  94.19±6.65 (=) |  94.19±6.65 (=) | 96.15±5.44 | 95.15±4.83 (+) | 94.19±6.65 (+) | 93.23±6.55 |  95.19±7.28 (-) | 94.15±6.63 (-)
RELATHE | 82.06±3.04 | 81.71±2.29 (=) | 81.29±2.32 (=) | 81.08±1.25 |  81.36±0.73 (=) | 80.87±0.93 (=) | 83.39±1.65 | 83.53±1.59 (=) |  83.81±2.84 (=)
TOX_171 | 78.99±4.87 | 71.95±4.86 (+) | 70.76±2.24 (+) | 73.09±4.10 | 63.73±1.64 (+) | 62.54±5.67 (+) | 73.68±4.39 | 70.17±12.67 (+) | 67.88±11.93 (+)
USPS | 76.61±0.48 | 74.10±0.71 (+) | 73.17±0.52 (+) | 88.95±0.41 | 88.62±0.41 (=) | 88.72±0.38 (=) | 90.93±0.61 | 90.73±0.53 (=) |  90.99±0.53 (=)
CNAE-9 | 88.23±2.42 | 86.75±1.49 (+) | 85.45±0.97 (+) | 86.93±0.54 | 85.08±1.32 (+) | 83.04±1.14 (+) | 87.67±1.26 | 86.95±1.08 (+) | 86.93±1.06 (+)
lung_discrete | 83.48±8.00 | 83.48±10.25 (=) | 80.70±9.79 (+) | 97.22±3.21 | 94.44±6.42 (+) | 97.22±3.21 (=) | 85.01±6.62 |  91.81±3.05 (-) | 90.74±3.29 (-)
warpPIE10P | 92.88±5.40 | 91.46±7.12 (=) | 91.43±3.93 (=) | 96.65±2.42 |  96.65±2.42 (=) | 96.18±2.69 (=) | 95.71±1.83 | 94.75±2.43 (+) | 95.01±1.83 (=)
lung | 97.55±1.88 |  97.55±1.88 (=) | 96.08±3.20 (+) | 97.06±1.96 | 96.56±0.96 (+) | 96.57±2.47 (+) | 96.08±3.20 | 95.09±2.52 (=) | 94.59±0.94 (+)
COIL20 | 90.97±1.85 | 89.24±1.19 (+) | 88.61±1.65 (+) | 98.54±0.83 | 98.13±0.95 (=) | 97.64±1.03 (+) | 98.96±0.57 | 97.68±0.76 (+) | 96.68±0.47 (+)
warpAR10P | 79.97±11.54 | 79.21±8.60 (=) | 77.63±13.41 (+) | 83.83±6.94 | 76.92±5.35 (+) | 79.14±8.01 (+) | 83.81±4.09 | 80.75±3.04 (+) | 82.36±5.63 (=)
Yale | 67.33±7.88 | 66.72±6.49 (=) | 64.33±9.62 (+) | 66.71±7.74 | 63.69±8.28 (+) | 61.85±10.81 (+) | 73.39±8.59 | 74.64±10.52 (=) |  77.05±10.65 (-)
Brain_Tumor_1 | 87.85±6.39 | 85.62±8.12 (+) | 84.49±5.57 (+) | 86.66±3.57 | 85.62±8.12 (=) | 86.66±3.57 (=) | 83.35±7.39 | 80.09±7.29 (+) | 79.00±7.11 (+)
Average Time | 16.6 | 20.3 | 17.4 | 122 | 80.13 | 70 | 52 | 431.93 | 377.5
W/T/L |  | 8/7/0 | 12/3/0 |  | 10/5/0 | 9/6/0 |  | 9/4/2 | 7/5/3

###  5.6 Comparison of GADP with GA and MRMR

We note that features selected by the MRMR method have a certain redundancy,
thus, further selection tries to eliminate the irrelevant features. In this
section, we compare the results of GADP algorithm with GA and no process based
on MRMR classified by GB, KNN, and SVM. We depict the average accuracies and
standard deviations in Tables 5-7. From accuracy and F1 aspects on three
classifiers, the GADP outperforms GA. Both GADP and GA can further promote
classifier’s performance. Therefore, further feature selection based on MRMR
is necessary. About the AUC, MRMR performs better than the results on accuracy
and F1. By comparison, we conclude that GADP is superior to the traditional
genetic algorithm for feature selection.

Table 5: Comparsion of GADP with MRMR and MRMR-GA on GB

Name | Accuracy | F1 | AUC
---|---|---|---
MRMR | MRMR | MRMR
GADP | GA |  | GADP | GA |  | GADP | GA |
PCMAC | 83.53±4.21 | 82.66+4.09 (=) | 77.82±2.22(+) | 87.65+1.31 | 85.38+1.39 (+) | 76.85±2.45(+) | 89.19+0.95 | 88.57+0.92 (+) | 89.91±6.37(=)
Isolet | 70.96+1.17 | 68.65+1.92 (+) | 63.33±1.51(+) | 72.18+3.65 | 70.71+3.27 (+) | 61.32±1.51(+) | 76.47+1.23 | 76.92+0.89 (=) |  97.20±0.28(-)
GLIOMA | 91.99+6.81 | 88.14+7.93 (+) | 75.96±6.38(+) | 87.98+4.66 | 86.06+3.79 (+) | 75.36±5.90(+) | 83.97+6.32 | 80.98+4.66 (+) |  94.62±2.88(-)
Prostate_GE | 94.19+6.65 | 94.19+6.65 (=) | 92.23±5.37(+) | 96.15+5.44 | 95.15+4.83 (=) | 92.20±5.41(+) | 93.23+6.55 |  95.19+7.28 (-) | 93.18±7.36(=)
RELATHE | 82.06+3.04 | 81.71+2.29 (=) | 71.41±0.85(+) | 81.08+1.25 |  81.36+0.73 (=) | 68.43±1.21(+) | 83.39+1.65 | 83.53+1.59 (=) |  89.52±0.91(-)
TOX_171 | 78.99+4.87 | 71.95+4.86 (+) | 60.78±4.01(+) | 73.09+4.10 | 63.73+1.64 (+) | 59.81±3.29(+) | 70.17+12.67 |  73.68+4.39 (=) |  89.25±2.42(-)
USPS | 76.61+0.48 | 74.10+0.71 (+) | 64.40±1.05(+) | 88.95+0.41 | 88.62+0.41 (=) | 64.17±0.85(+) | 90.93+0.61 | 90.73+0.53 (=) |  92.47±0.47(-)
CNAE-9 | 88.23+2.42 | 86.75+1.49 (+) | 85.91±2.61(+) | 86.93+0.54 | 85.08+1.32 (-) | 86.02±2.46(=) | 87.67+1.26 | 86.95+1.08 (+) | 98.23±0.29
lung_discrete | 83.48+8.00 | 83.48+10.25 (=) | 60.31±9.21(+) | 97.22+3.21 | 94.44+6.42 (+) | 51.64±13.23(+) | 85.01+6.62 |  91.81+3.05 (-) | 68.49±7.17(+)
warpPIE10P | 92.88+5.40 | 91.46+7.12 (=) | 89.08±6.56(=) | 96.65+2.42 |  96.65+2.42 (=) | 88.99±6.94(+) | 95.71+1.83 | 94.75+2.43 (=) |  97.58±1.62(-)
lung | 97.55+1.88 | 97.55+1.88 (=) | 93.61±2.92(+) | 97.06+1.96 | 96.56+0.96 (=) | 93.63±2.72(+) | 96.08+3.20 | 95.09+2.52 (=) |  97.91±1.95(=)
COIL20 | 90.97+1.85 | 89.24+1.19 (+) | 85.21±1.90(+) | 98.54+0.83 | 98.13+0.95 (=) | 84.29±2.15(+) | 98.96+0.57 | 97.68+0.76 (+) | 98.69±0.25(=)
warpAR10P | 79.97+11.54 | 79.21+8.60 (=) | 72.16±12.47(+) | 83.83+6.94 | 76.92+5.35 (+) | 71.79±12.25(+) | 83.81+4.09 | 80.75+3.04 (+) |  94.73±3.12(-)
Yale | 67.33+7.88 | 66.72+6.49 (=) | 58.25±8.45(+) | 66.71+7.74 | 63.69+8.28 (+) | 55.25±8.01(+) | 73.39+8.59 | 74.64+10.52 (=) |  86.65±5.34(-)
Brain_Tumor_1 | 87.85+6.39 | 85.62+8.12 (+) | 76.78±8.72(+) | 86.66+3.57 | 85.62+8.12 (=) | 74.03±12.10(+) | 83.35+7.39 | 80.09+7.29 (+) | 81.76±11.02(=)
W/T/L |  | 7/8/0 | 14/1/0 |  | 7/7/1 | 14/1/0 |  | 6/7/2 | 1/5/9

Table 6: Comparsion of GADP with MRMR-GA and MRMR on KNN

Name | Accuracy | F1 | AUC
---|---|---|---
MRMR | MRMR | MRMR
GADP | GA |  | GADP | GA |  | GADP | GA |
PCMAC | 87.65±1.31 |  87.91±0.99 (=) | 79.62±1.12(+) | 87.54±1.32 |  87.76±1.02 (=) | 79.61±1.12(+) | 93.42±1.34 | 88.27±0.91 (+) | 86.22±1.63(+)
Isolet | 72.18±3.65 | 68.85±2.32 (+) | 68.08±3.11(+) | 71.50±3.65 | 68.70±2.35 (+) | 67.57±3.04(+) | 93.53±3.92 | 92.70±0.88 (=) | 92.81±0.80(+)
GLIOMA | 87.98±4.66 | 81.89±7.80 (+) | 78.04±13.08(+) | 87.57±4.93 | 81.68±7.69 (+) | 77.59±8.45(+) | 92.61±4.56 |  95.31±1.96 (-) | 92.32±4.24(=)
Prostate_GE | 96.15±5.44 | 93.19±4.86 (+) | 90.27±4.87(+) | 96.15±5.45 | 93.16±4.88 (+) | 90.22±4.90(+) | 95.78±5.44 | 95.74±3.82 (=) | 94.72±5.30(=)
RELATHE | 81.08±1.25 | 74.63±2.43 (+) | 79.19±0.91(+) | 81.11±1.26 | 73.86±2.42 (+) | 79.23±0.91(+) | 84.35±1.25 | 76.62±0.82 (+) |  84.59±1.45(=)
TOX_171 | 73.09±4.10 | 70.78±6.04 (+) | 45.60±5.35(+) | 73.05±4.68 | 70.57±5.29 (+) | 45.60±5.48(+) | 87.81±4.49 |  88.04±1.93 (=) | 71.76±5.28
USPS | 88.95±0.41 | 88.52±0.99 (=) | 87.97±0.42(+) | 88.87±0.42 | 88.46±1.00 (=) | 87.86±0.40(+) | 96.79±0.43 | 96.74±0.59 (=) | 96.68±0.22(=)
CNAE-9 | 86.93±0.54 | 84.62±1.41 (+) | 80.72±2.75(+) | 87.38±0.44 | 84.80±1.41 (+) | 81.12±2.90(+) | 95.27±0.51 | 95.19±1.03 (=) | 94.21±1.72(=)
lung_discrete | 97.22±3.21 | 93.20±2.49 (+) | 90.35±8.38(+) | 97.14±3.30 | 92.99±3.75 (+) | 88.71±11.85(+) | 99.25±3.21 | 98.03±2.02 (=) | 98.36±1.61(=)
warpPIE10P | 96.65±2.42 | 91.42±2.50 (+) | 94.76±2.39(+) | 96.70±2.38 | 90.85±2.58 (+) | 94.81±2.32(+) | 98.44±2.41 | 98.21±0.64 (=) | 98.33±1.34(=)
lung | 97.06±1.96 | 96.06±2.77 (=) | 94.61±4.35(+) | 96.59±2.28 | 95.90±2.92 (=) | 93.69±5.18(=) | 96.23±2.08 | 96.03±3.61 (=) | 95.98±3.88(=)
COIL20 | 98.54±0.83 | 97.22±0.51 (+) | 97.01±0.97(+) | 98.53±0.84 | 97.20±0.51 (+) | 96.99±0.98(+) | 99.70±0.83 | 99.58±0.18 (=) | 99.62±0.28(=)
warpAR10P | 83.83±6.94 | 72.28±4.53 (+) | 70.74±7.47(+) | 82.97±7.29 | 72.09±4.80 (+) | 70.25±8.01(+) | 95.92±6.71 | 94.40±2.76 (=) | 92.90±3.56(+)
Yale | 66.71±7.74 | 56.94±9.57 (+) | 55.17±7.42(+) | 65.69±8.36 | 55.07±8.84 (+) | 52.52±7.99(+) | 84.19±8.18 | 81.68±4.71 (+) | 82.08±5.15(+)
Brain_Tumor_1 | 86.66±3.57 | 85.57±2.00 (=) | 76.63±6.87(+) | 83.15±3.96 | 81.17±2.73 (+) | 70.71±6.76(+) | 78.52±2.06 |  83.03±4.27 (-) | 78.25±4.07(+)
W/T/L |  | 11/4/0 | 15/0/0 |  | 12/3/0 | 14/1/0 |  | 3/10/2 | 5/10/0

Table 7: Comparsion of GADP with MRMR-GA and MRMR on SVM

Name | Accuracy | F1 | AUC
---|---|---|---
MRMR | MRMR | MRMR
GADP | GA |  | GADP | GA |  | GADP | GA |
PCMAC | 89.19±0.95 | 87.37±0.43 (+) | 83.78±0.49(+) | 90.03±1.14 | 88.48±0.39 (+) | 84.41±0.77(+) | 95.91±0.95 | 94.79±0.93(+) |  96.19±0.76(=)
Isolet | 76.47±1.23 |  76.79±3.26 (=) | 72.05±0.88(+) | 76.70±1.23 |  77.21±3.05 (=) | 72.60±1.32(+) | 98.74±1.51 | 98.60±0.30(=) |  98.75±0.12(=)
GLIOMA | 85.97±6.32 | 83.97±4.29 (+) | 75.13±3.70(+) | 86.19±5.77 | 84.67±5.03 (+) | 78.74±5.03(+) | 93.61±6.17 |  95.50±2.60(=) | 93.93±3.18(=)
Prostate_GE | 93.23±6.55 | 93.15±6.56 (=) | 87.23±5.37(+) | 93.77±6.18 | 93.73±6.19 (=) | 87.85±5.12(+) | 96.52±6.55 | 96.60±6.80(=) |  97.03±5.52(=)
RELATHE | 83.39±1.65 | 80.11±2.84 (+) | 77.97±1.95(+) | 84.24±1.75 | 82.34±2.72 (+) | 78.93±2.34(+) | 92.19±1.65 | 90.46±1.24(+) |  92.55±1.02(=)
TOX_171 | 75.17±6.67 | 64.37±5.53 (+) | 67.52±6.90(+) | 75.05±6.09 | 66.15±4.87 (+) | 67.90±7.57(+) | 90.34±2.94 | 88.75±2.85(+) |  90.63±2.24(=)
USPS | 90.93±0.61 | 88.85±0.57 (+) | 85.66±0.58(+) | 90.97±0.59 | 88.88±0.58 (+) | 85.69±0.59(+) | 99.32±0.62 | 99.25±0.19(=) | 99.26±0.10(=)
CNAE-9 | 87.67±1.26 |  88.42±2.77 (=) | 82.95±1.67(+) | 89.37±1.21 |  89.78±2.02 (=) | 84.19±1.72(+) | 98.74±1.35 | 98.41±0.50(=) |  98.84±0.26(=)
lung_discrete | 85.01±6.62 | 83.63±4.07 (+) |  85.35±8.38(=) | 83.89±8.37 | 79.83±8.37 (+) | 82.59±13.01(=) | 98.33±7.62 | 98.63±1.39(=) |  98.68±1.85(=)
warpPIE10P | 95.71±1.83 | 94.29±1.54 (+) | 88.32±3.30(+) | 96.32±1.64 | 94.42±1.32 (+) | 89.73±2.47(+) | 99.60±1.83 | 98.90±0.71(=) | 99.48±0.55(=)
lung | 96.08±3.20 | 95.56±3.37 (=) | 89.11±2.75(+) | 96.17±3.28 | 95.79±3.09 (=) | 89.39±2.66(+) | 99.45±3.22 | 98.80±1.20(=) | 99.24±1.18(=)
COIL20 | 98.96±0.57 | 98.61±0.39 (=) | 93.82±0.14(+) | 99.04±0.52 | 98.71±0.34 (=) | 93.89±0.14(+) | 99.99±0.57 | 99.98±0.03(=) | 99.99±0.01(=)
warpAR10P | 83.81±4.09 | 82.29±9.91 (=) | 74.26±3.65(+) | 88.01±2.69 | 85.08±4.81 (+) | 76.62±6.13(+) | 97.15±3.68 | 96.27±3.65(=) |  98.01±1.33(=)
Yale | 73.39±8.59 | 70.28±7.90 (+) | 66.57±9.92(+) | 76.01±8.29 |  76.13±5.61 (=) | 70.22±12.79(=) | 90.41±9.07 |  92.22±2.01(=) | 91.79±3.80(=)
Brain_Tumor_1 | 83.35±7.39 | 75.59±1.94 (+) | 71.73±5.16(+) | 77.09±6.52 | 65.53±9.03 (+) | 61.20±10.95(+) | 89.28±6.09 | 87.28±9.22(=) | 88.95±4.81(=)
W/T/L |  | 9/6/0 | 14/1/0 |  | 9/6/0 | 13/2/0 |  | 3/12/0 | 0/15/0

###  5.7 Comparsion of GADP, PSO, FPA and WOA

In Section 5.4, we explored the effect of MRMR, the results directly applying
GADP on full features are not good. Thus, we compare our method to other
heuristic algorithms, such as PSO, FPA, and WOA based on MRMR in Table 8. On
GB and KNN, our proposed method obtains the highest accuracies nearly on all
the datasets. On SVM, GADP achieves 11 of 15 the highest accuracies.
Otherwise, it seems that the PSO is a little better than FPA and WOA. What’s
more, compared with MRMR’s results in Table 5 \- 7, PSO, FPA and WOA also
promote their performances as GADP and GA. Finally, for the feature selection
task, we believe that GADP is a considerable method.

Table 8: Comparsion of GADP, PSO, FPA and WOA

Name | GB | KNN | SVM
---|---|---|---
MRMR | MRMR | MRMR
GADP | PSO | FPA | WOA | GADP | PSO | FPA | WOA | GADP | PSO | FPA | WOA
PCMAC | 83.53±4.21 | 81.16±4.23 (+) | 79.41±3.22 (+) | 80.24±4.24 (+) | 87.65±1.31 | 86.36±0.85 (+) | 81.42±2.62 (+) | 80.96±1.44 (+) | 89.19±0.95 | 88.42±0.67 (+) | 85.95±0.21 (=) | 87.08±0.30 (=)
Isolet | 70.96±1.17 | 69.04±2.11 (+) | 64.87±2.00 (+) | 65.58±0.97 (+) | 72.18±2.65 | 70.90±2.74 (+) | 68.65±3.24 (+) | 68.40±2.99 (+) | 76.47±0.43 | 73.28±1.71 (+) | 72.90±0.55 (+) | 71.38±1.05 (=)
GLIOMA | 91.99±6.81 | 90.06±7.72 (=) | 90.06±7.72 (=) | 86.06±3.79 (+) | 87.98±4.66 | 86.06±7.33 (=) | 83.81±2.09 (+) | 85.90±1.32 (+) | 83.97±3.32 |  87.98±4.66 (-) | 80.13±3.70 (+) | 81.89±2.63 (+)
Prostate_GE | 94.19±4.65 |  94.19±4.65 (=) | 93.23±3.55 (+) | 93.23±3.55 (+) | 96.15±2.44 | 95.15±5.44 (=) | 94.12±2.27 (+) | 94.15±2.01 (+) | 93.23±6.55 | 93.23±6.55 (=) | 94.19±6.65 (=) |  94.23±7.36 (=)
RELATHE | 82.06±3.04 | 81.50±0.81 (+) | 77.22±2.43 (+) | 79.40±2.03 (+) | 81.08±1.25 | 80.01±3.70 (=) | 78.77±1.17 (+) | 77.65±2.13 (+) | 83.39±1.65 | 80.18±2.20 (+) | 80.25±1.79 (+) | 81.57±1.26 (+)
TOX_171 | 78.99±4.87 | 75.44±2.25 (+) | 68.41±1.60 (+) | 69.02±3.32 (+) | 73.09±4.10 | 61.97±2.50 (+) | 58.51±4.43 (+) | 51.44±3.36 (+) | 70.17±2.67 | 66.16±2.57 (+) | 67.59±3.31 (+) | 65.45±6.54 (+)
USPS | 76.61±0.48 | 75.31±0.53 (+) | 70.81±1.04 (+) | 69.51±1.07 (+) | 88.95±0.41 | 88.44±0.19 (+) | 88.01±0.68 (+) | 88.41±0.61 (=) | 90.93±0.61 | 89.58±0.71 (+) | 88.62±0.76 (+) | 88.35±0.51 (+)
CNAE-9 | 88.23±2.42 | 84.25±2.17 (+) | 79.15±3.64 (+) | 77.20±2.22 (+) | 86.93±0.54 | 84.24±1.07 (+) | 79.43±1.91 (+) | 78.50±2.65 (+) | 87.67±1.26 | 83.22±1.09 (+) | 81.09±1.30 (+) | 84.43±2.62 (+)
lung_discrete | 83.48±8.00 |  84.94±9.46 (=) | 73.76±19.07 (+) | 75.22±9.88 (+) | 97.22±3.21 | 94.52±4.54 (+) | 93.20±2.49 (+) | 91.89±5.12 (+) | 87.01±2.62 |  90.42±3.46 (-) | 82.09±4.57 (+) | 88.96±4.66 (=)
warpPIE10P | 92.88±5.40 | 91.94±7.26 (=) | 89.07±5.37 (+) | 88.61±7.50 (+) | 96.65±2.42 | 96.18±2.69 (=) | 96.18±2.69 (=) | 95.71±3.24 (=) | 95.71±1.83 | 95.71±3.92 (=) | 93.79±3.31 (+) | 94.76±1.39 (+)
lung | 97.55±1.88 | 95.59±2.35 (+) | 95.58±1.86 (+) | 96.08±2.77 (=) | 97.06±1.96 | 96.57±2.47 (=) | 96.57±3.35 (=) | 96.08±3.20 (=) | 96.08±3.20 | 93.61±2.44 (=) | 95.09±2.52 (=) | 95.09±2.52 (=)
COIL20 | 90.97±1.85 | 89.03±2.00 (+) | 87.08±1.29 (+) | 87.01±0.62 (+) | 98.54±0.83 | 97.92±0.70 (+) | 97.36±0.86 (+) | 97.01±1.29 (=) | 98.96±0.57 | 97.89±0.51 (+) | 97.75±0.66 (+) | 97.75±0.48 (+)
warpAR10P | 79.97±4.54 | 77.63±3.41 (+) | 76.07±4.18 (+) | 76.87±5.38 (+) | 83.83±6.94 | 76.16±1.36 (+) | 73.82±3.40 (+) | 73.01±9.18 (+) | 83.81±2.09 | 81.32±2.87 (+) | 81.53±1.68 (+) | 81.29±2.08 (+)
Yale | 67.33±4.88 | 65.51±3.56 (=) | 62.49±3.02 (+) | 63.73±4.85 (+) | 66.71±3.74 | 61.22±3.25 (+) | 61.85±5.44 (+) | 58.19±2.11 (+) | 73.39±8.59 |  73.98±7.80 (=) | 73.37±6.91 (=) | 71.62±12.93 (=)
Brain_Tumor_1 | 87.85±2.39 | 83.35±4.39 (+) | 82.31±3.28 (+) | 83.35±2.39 (+) | 86.66±3.57 | 85.62±8.12 (=) | 84.54±7.36 (=) | 82.31±4.68 (+) | 83.35±1.39 | 81.35±2.39 (+) | 78.95±3.27 (+) | 82.26±9.31 (=)
W/T/L |  | 10/5/0 | 14/1/0 | 14/1/0 |  | 9/6/0 | 12/3/0 | 11/4/0 |  | 9/4/2 | 11/4/0 | 8/7/0

##  6 Conclusion

Selecting the optimal subset of features is a vital job for promoting the
classifier’s performance. Combining the advantages of different methods is a
reasonable approach. In this article, we propose a novel population-based
algorithm inspired by the genetic algorithm. Different from the traditional
genetic algorithm, the crossover and mutation operators are replaced by the
dynamic probability. The value of probability assures the existence of
heredity and mutation. For evaluating the performance of our proposed method,
we explore the influence of probability limitation. Results show that no
limation makes the convergence faster and accuracy higher. After, we test the
effect of MRMR by comparing GADP’s results on the selected features and full
features. The results of MRMR are better than the full features’ because there
exist lots of irrelevant features which let classifiers trap into local
optimum and increase their time consumption. What’s more, we also compare the
GADP with GA and no process. Further feature selection can promote the
classifier’s results. GADP surpasses GA which indicates our proposed method is
accessible. Most of cases, the time cost of MRMR is lower than full features’,
at the same time, with higher accuracy. Sometimes, the results of MRMR have a
higher time cost and lower accuracy. This may concern the search space. Larger
search space and less supportive features make the classifier’s performance
bad and cost more time. MRMR filter may lose some key features. Finally, other
famous heuristic algorithms like PSO, FPA, and WOA are compared with our
method. Still, our algorithm shows its advantages in feature selection. In
these heuristic algorithms, PSO performs better than FPA and WOA. In the
future, we plan to extend our method to other domains.

## References

  * Sorzano et al. [2014] Carlos Oscar Sánchez Sorzano, Javier Vargas, and A Pascual Montano.  A survey of dimensionality reduction techniques.  _arXiv preprint arXiv:1403.2877_ , 2014.
  * Too and Mirjalili [2021] Jingwei Too and Seyedali Mirjalili.  A hyper learning binary dragonfly algorithm for feature selection: A covid-19 case study.  _Knowledge-Based Systems_ , 212:106553, 2021.
  * Sun et al. [2020] Liang Sun, Zhanhao Mo, Fuhua Yan, Liming Xia, Fei Shan, Zhongxiang Ding, Bin Song, Wanchun Gao, Wei Shao, Feng Shi, et al.  Adaptive feature selection guided deep forest for covid-19 classification with chest ct.  _IEEE Journal of Biomedical and Health Informatics_ , 24(10):2798–2805, 2020.
  * Purushothaman et al. [2020] R Purushothaman, SP Rajagopalan, and Gopinath Dhandapani.  Hybridizing gray wolf optimization (gwo) with grasshopper optimization algorithm (goa) for text feature selection and clustering.  _Applied Soft Computing_ , 96:106651, 2020.
  * Abasi et al. [2021] Ammar Kamal Abasi, Ahamad Tajudin Khader, Mohammed Azmi Al-Betar, Syibrah Naim, Sharif Naser Makhadmeh, and Zaid Abdi Alkareem Alyasseri.  An improved text feature selection for clustering using binary grey wolf optimizer.  In _Proceedings of the 11th national technical seminar on unmanned system technology 2019_ , pages 503–516. Springer, 2021.
  * Zhong and Pun [2020] Guo Zhong and Chi-Man Pun.  Subspace clustering by simultaneously feature selection and similarity learning.  _Knowledge-Based Systems_ , 193:105512, 2020.
  * Peng et al. [2010] Yonghong Peng, Zhiqing Wu, and Jianmin Jiang.  A novel feature selection approach for biomedical data classification.  _Journal of Biomedical Informatics_ , 43(1):15–23, 2010.
  * Tang et al. [2014] Jiliang Tang, Salem Alelyani, and Huan Liu.  Feature selection for classification: A review.  _Data classification: Algorithms and applications_ , page 37, 2014\.
  * Lin et al. [2015] Kuan-Cheng Lin, Yi-Hung Huang, Jason C Hung, and Yung-Tso Lin.  Feature selection and parameter optimization of support vector machines based on modified cat swarm optimization.  _International Journal of Distributed Sensor Networks_ , 11(7):365869, 2015.
  * Li et al. [2017] Jundong Li, Kewei Cheng, Suhang Wang, Fred Morstatter, Robert P Trevino, Jiliang Tang, and Huan Liu.  Feature selection: A data perspective.  _ACM computing surveys (CSUR)_ , 50(6):1–45, 2017\.
  * Xue et al. [2015] Bing Xue, Mengjie Zhang, Will N Browne, and Xin Yao.  A survey on evolutionary computation approaches to feature selection.  _IEEE Transactions on Evolutionary Computation_ , 20(4):606–626, 2015.
  * Shannon [1948] Claude Elwood Shannon.  A mathematical theory of communication.  _The Bell system technical journal_ , 27(3):379–423, 1948.
  * Vergara and Estévez [2014] Jorge R Vergara and Pablo A Estévez.  A review of feature selection methods based on mutual information.  _Neural computing and applications_ , 24(1):175–186, 2014.
  * Ding and Peng [2005] Chris Ding and Hanchuan Peng.  Minimum redundancy feature selection from microarray gene expression data.  _Journal of bioinformatics and computational biology_ , 3(02):185–205, 2005.
  * Holland [1973] John H Holland.  Genetic algorithms and the optimal allocation of trials.  _SIAM journal on computing_ , 2(2):88–105, 1973\.
  * Eberhart and Kennedy [1995] Russell Eberhart and James Kennedy.  Particle swarm optimization.  In _Proceedings of the IEEE international conference on neural networks_ , volume 4, pages 1942–1948. Citeseer, 1995.
  * Kar [2016] Arpan Kumar Kar.  Bio inspired computing–a review of algorithms and scope of applications.  _Expert Systems with Applications_ , 59:20–32, 2016.
  * Chak and Feng [1995] Chu Kwong Chak and Gang Feng.  Accelerated genetic algorithms: combined with local search techniques for fast and accurate global search.  In _Proceedings of the 1995 IEEE International Conference on Evolutionary Computation. Part 1 (of 2)_ , pages 378–383, 1995.
  * Michalewicz and Schoenauer [1996] Zbigniew Michalewicz and Marc Schoenauer.  Evolutionary algorithms for constrained parameter optimization problems.  _Evolutionary computation_ , 4(1):1–32, 1996\.
  * El-Mihoub et al. [2006] Tarek A El-Mihoub, Adrian A Hopgood, Lars Nolle, and Alan Battersby.  Hybrid genetic algorithms: A review.  _Eng. Lett._ , 13(2):124–137, 2006.
  * Hansen et al. [2015] Nikolaus Hansen, Dirk V Arnold, and Anne Auger.  Evolution strategies.  In _Springer handbook of computational intelligence_ , pages 871–898. Springer, 2015.
  * Xue et al. [2021] Yu Xue, Haokai Zhu, Jiayu Liang, and Adam Słowik.  Adaptive crossover operator based multi-objective binary genetic algorithm for feature selection in classification.  _Knowledge-Based Systems_ , 227:107218, 2021.
  * Manzoni et al. [2020] Luca Manzoni, Luca Mariot, and Eva Tuba.  Balanced crossover operators in genetic algorithms.  _Swarm and Evolutionary Computation_ , 54:100646, 2020.
  * Sun and Lu [2019] Na Sun and Yong Lu.  A self-adaptive genetic algorithm with improved mutation mode based on measurement of population diversity.  _Neural Computing and Applications_ , 31(5):1435–1443, 2019.
  * Amini and Hu [2021] Fatemeh Amini and Guiping Hu.  A two-layer feature selection method using genetic algorithm and elastic net.  _Expert Systems with Applications_ , 166:114072, 2021.
  * Mustafi and Sahoo [2019] Debjani Mustafi and Gadadhar Sahoo.  A hybrid approach using genetic algorithm and the differential evolution heuristic for enhanced initialization of the k-means algorithm with applications in text clustering.  _Soft Computing_ , 23(15):6361–6378, 2019.
  * Gupta and Ghafir [2012] Deepti Gupta and Shabina Ghafir.  An overview of methods maintaining diversity in genetic algorithms.  _International journal of emerging technology and advanced engineering_ , 2(5):56–60, 2012.
  * Kumar [2013] Anit Kumar.  Encoding schemes in genetic algorithm.  _International Journal of Advanced Research in IT and Engineering_ , 2(3):1–7, 2013.
  * Costa et al. [2010] Antonio Costa, Giovanni Celano, Sergio Fichera, and Enrico Trovato.  A new efficient encoding/decoding procedure for the design of a supply chain network with genetic algorithms.  _Computers & Industrial Engineering_, 59(4):986–999, 2010.
  * Bahl et al. [1986] Lalit Bahl, Peter Brown, Peter De Souza, and Robert Mercer.  Maximum mutual information estimation of hidden markov model parameters for speech recognition.  In _ICASSP’86. IEEE International Conference on Acoustics, Speech, and Signal Processing_ , volume 11, pages 49–52. IEEE, 1986.
  * Battiti [1994] Roberto Battiti.  Using mutual information for selecting features in supervised neural net learning.  _IEEE Transactions on neural networks_ , 5(4):537–550, 1994.
  * Hoque et al. [2014] Nazrul Hoque, Dhruba K Bhattacharyya, and Jugal K Kalita.  Mifs-nd: A mutual information-based feature selection method.  _Expert Systems with Applications_ , 41(14):6371–6385, 2014.
  * Kwak and Choi [2002] Nojun Kwak and Chong-Ho Choi.  Input feature selection for classification problems.  _IEEE transactions on neural networks_ , 13(1):143–159, 2002.
  * Peng et al. [2005] Hanchuan Peng, Fuhui Long, and Chris Ding.  Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy.  _IEEE Transactions on pattern analysis and machine intelligence_ , 27(8):1226–1238, 2005.
  * Lin and Tang [2006] Dahua Lin and Xiaoou Tang.  Conditional infomax learning: An integrated framework for feature extraction and fusion.  In _European conference on computer vision_ , pages 68–82. Springer, 2006.
  * Meyer et al. [2008] Patrick Emmanuel Meyer, Colas Schretter, and Gianluca Bontempi.  Information-theoretic feature selection in microarray data using variable complementarity.  _IEEE Journal of Selected Topics in Signal Processing_ , 2(3):261–274, 2008.
  * Yang and Moody [1999] Howard Yang and John Moody.  Data visualization and feature selection: New algorithms for nongaussian data.  _Advances in neural information processing systems_ , 12, 1999.
  * Jadhav et al. [2018] Swati Jadhav, Hongmei He, and Karl Jenkins.  Information gain directed genetic algorithm wrapper feature selection for credit rating.  _Applied Soft Computing_ , 69:541–553, 2018.
  * Salesi et al. [2021] Sadegh Salesi, Georgina Cosma, and Michalis Mavrovouniotis.  Taga: Tabu asexual genetic algorithm embedded in a filter/filter feature selection approach for high-dimensional data.  _Information Sciences_ , 565:105–127, 2021.
  * Huang et al. [2007] Jinjie Huang, Yunze Cai, and Xiaoming Xu.  A hybrid genetic algorithm for feature selection wrapper based on mutual information.  _Pattern recognition letters_ , 28(13):1825–1844, 2007.
  * Yang [2012] Xin-She Yang.  Flower pollination algorithm for global optimization.  In _International conference on unconventional computing and natural computation_ , pages 240–249. Springer, 2012.
  * Mirjalili and Lewis [2016] Seyedali Mirjalili and Andrew Lewis.  The whale optimization algorithm.  _Advances in engineering software_ , 95:51–67, 2016.