**4. Biological basis of pesticides resistance**

transferase, and carboxylesterase [16, 17]; all these enzymes hydrolyze or sequester different kinds of pesticides. Exposure to a pesticide can exert enough pressure to select an enzymatic system or a specific isozyme within each family. Esterase isozyme overexpression is generally accepted as a mechanism involved in OP resistance. However, in the Coatzacoalcos laboratory strain of *R*. *microplus*, designated as such to reflect the name of the village in Mexico where the original tick population sample was obtained, metabolic detoxication has been identified by its efficient esterase activity resulting from enzyme overexpression as a resistance mechanism for permethrin that belongs to the pyrethroid chemical class of pesticides. This strain has been toxicologically characterized using the larval packet test (LPT) [18], which helped to elucidate the esterase-based mechanism of resistance to permethrin. The *R*. *microplus* Coatzacoalcos strain exhibits a significant enhanced capacity to hydrolyze permethrin as well as an increased esterase activity. This suggests an esterase based metabolic mechanism as a main component of permethrin resistance [19]. The esterase gene responsible for permethrin resistance was identified and named *CzEST9*. It is also known that the overexpression mechanism of this isozyme is the result of *CzEST9* duplication in the Coatzacoalcos strain that leads to metabolic detoxication through the overexpression of esterase 9 activity in *R. microplus* [20]. The sequence of *CzEST9* gene has been determined and the recombinant product yielded a 62.8 kDa protein [19]. Since the Coatzacoalcos strain does not include the *Kdr* variation in the sodium channel gene found in other Mexican strains of *R*. *microplus*, it is suggested that there are two inde‐ pendent mechanisms of acaricide resistance to pyrethroids. However, common mechanisms of acaricide resistance to pyrethroids in Mexico apparently involve the presence of sequence

The sodium channel is the known target site of pyrethroids. Sequence variation in the sodium channel prevents pyrethroids from attaching to the target site due to an alteration in the sodium channel stereochemical structure. For this reason, the process is described as target site modification mechanism, or *Kdr*-type resistance (Knock down resistance). This is one of the

Two important allele variants occurring in the sodium channel gene associated with pyreth‐ roid resistance in the cattle tick *R. microplus* are the variation occurring in domain III segment 6 (III-S6) [22] and the variation occurring at the bridge joining segments 4–5 in domain II (II-S4-5) [23]. The former is a Phe-Ile substitution produced by a nucleotide variation at domain III-S6 that was first reported in Mexican tick strains. Its role and contribution to pyrethroid resistance has been confirmed [21]. The other is a Le-Ile substitution found thus far only in Australian tick strains; this variation is very similar to a variation found in the crop insect pest

Findings on the diversity of allele variants occurring in the sodium channel gene associated with pyrethroid resistance in the cattle tick *R. microplus* have been confirmed by experiments based on differences in melting temperatures (Tm) of sodium channel allele specific gene fragments obtained with single larvae DNA from México and Australia. These experiments revealed that substitution III-S6 (Phe-Ile) only occurs in Mexican tick populations whereas substitution II-S4-5 (Le-Ile) only occurs in Australian tick strains [24]. The information available suggests that there are at least two different and independent mechanisms involved based on

mechanisms of pesticide resistance in insects that is better understood.

variation in the sodium channel gene [21].

*Bemicia tabaci* [23].

338 Insecticides Resistance

Allelochemical interactions are defensive processes or are involved in food competition mechanisms that different species employ to inhibit the action of natural enemies. Plant– arthropod coevolution is a natural selection mechanism driven by allelochemical interactions between plants and arthropods over millions of years [7]. As a result of the reciprocal inter‐ actions between these two groups of organisms [8], arthropods have conserved within their genomes all those traits conferring them the ability to inhibit or avoid toxicants produced by plants that function as defensive mechanisms against herbivorous insects [9, 10]. Sophisticated metabolic detoxication mechanisms have been developed by herbivorous arthropods in order to survive the exposure to toxic plant metabolites [11], which represents a natural process of resistance to plant toxicants. The preservation of these components in the genome of arthro‐ pods provides the foundation of molecular systems that allow them to get adapted and become resistant against pesticides currently used.

Insects affect the survival and reproduction of plants. Secondary metabolites like phenolic acids, flavonoids, terpenoids, steroids, alkaloids, and organic cyanides are produced by plants as part of defensive mechanisms. A coevolutionary race is established through these interac‐ tions where mutual selection pressure has led to the process of speciation to preserve natural equilibrium.

Adaptation to new environmental conditions requires the development of defensive processes through natural selection. Plants are biological engines producing a wide variety of natural defensive chemicals including repellents, antifeeding molecules, and poisons, some of which have been used as natural strategies to protect cultivars from plagues or to control vector-borne diseases. These mechanisms evolved separately in different herbivorous insects driven by diverse insect–plant interactions. In general, herbivorous insects feed on few plant species and plant species are attacked by pests specialized to overcome natural defensive substances. Herbivorous insects have developed the molecular machinery to metabolize most of the toxic material produced by plants, but not all toxicants are metabolized by all pest species.

Specialized insects have an adaptive advantage because their biochemical systems evolved to detoxify one or few potentially harmful substrates. The metabolic system of a polyphagous species reflects more diverse detoxication mechanisms against a wide variety of chemically defined plants. Thus, polyphagous insects have a "higher metabolic load". The activity of mixed function oxidases in the intestines of moths and butterfly larvae is higher in polypha‐ gous species than in species restricted to a single family of plants. Pyrethrins are part of the wide variety of allelochemicals metabolized by this family of enzymes [25].


*2.3. Sequestering of Iron*

Some PGPM, like *Pseudomonas fluorescens* and *Rhizobia meliloti*, sequester iron through the production of siderophores, which can be grouped into four, namely: hydroxamates, catecholates, carboxylates and pyoverdines [50]. Currently, about 500 siderophores have been reported by researchers. Although plants cannot absorb Fe3+, siderophores have a high affinity for Fe3+, which results in an iron-siderophore complex that is then absorbed by plants [51], into their tissues, hence, aiding plants in meeting their iron requirements [14,17,43,52]. In 2013, study findings of Radzki et al. [53] showed an increase in iron content at 12 weeks for iron deficient tomato plants, following inoculation of siderophore producing bacteria, evidence that microbial siderophores can be a source of iron for plants. A study by Sharma and Johri [54] also showed an increase in maize plant growth following inoculation with siderophore producing PGPR. The uptake of Fe-microbial siderophore complexes by strategy II plants, via ligand exchange, between ferrated microbes and a phyto siderophore, was also reported by Yehuda et al. [55] It should also be noted that some plant species can also produce siderophores which bind Fe3+, to form a complex that can be taken up by the plant with the aid of ligands. Production of siderophores is also a benefit in the context of biocontrol in a sense that potential plant pathogens, especially fungal pathogens, are outcompeted for iron sources, which may lead to their death, or ineffectiveness.

#### *2.4. Potassium Solubilisation*

Microbes such as: *Arthrobacter* sp. *Bacillus edaphicus*, *Bacillus circulans* and *Bacillus mucilaginosus* convert sparingly soluble and mineral potassium to soluble forms available for plant use [56]. Through the release of H+ and organic anions, such as citrate, malate and oxalate, arbuscular mycorrhiza can also increase the solubility of mineral K [57], thereby increasing the availability of potassium anions for plant uptake, although the increase in K+ availability is sometimes related to the increase in phosphorus availability [22,58].

Some PGPR can also directly influence plant growth through the production of phytohormones such as auxins and gibberellins, which enhance plant growth when plant phytohormones are at suboptimal concentrations [37]. They may also produce enzymes which regulate hormone concentration in plant tissue. For instance, some plant growth promoting microorganisms can produce an enzyme, ACC deaminase, which breaks down ACC, a precursor of ethylene, into an alpha keto butyrate and ammonium, hence lowering ethylene concentration in plant tissues [56,59–62]. With more research and proper manipulation, PGPM, with the ability to enhance plant growth, may not necessarily fully replace chemical fertilizer, but lower their use, directly and indirectly, through increasing the plants' nutrient uptake efficiency from applied chemical fertilizers [63]. Manipulations such as developing an effective consortium of microbes that are able to make available key elements in the soil would greatly reduce the need to use chemical fertilizers. For instance, rhizobial species require iron for good growth, and in their nitrogenase complex, hence co-inoculation of rhizobia and siderophore producing PGPM could enhance nodulation and nitrogen fixation [64]. Use of biofertilizers can lower the need to burn fossil fuels for fertilizer production, and the associated contribution to greenhouse gas emissions.

#### **3. PGPM and Control of Plant Pests and Diseases**

With global warming comes new species of pests, weeds and pathogens currently prevalent in warmer environments. The use of chemicals to suppress such plant growth inhibitors is effective but with negative outcomes related to improper use, cost, and increasing evolution of tolerance to the chemical. The antagonist properties of biocontrols against such plant growth suppressors have been reported by many researchers, and the results are promising. A diversity of PGPM with biocontrol properties has been identified by researchers, conferring benefits to a variety of crop species [65–73]. Berendsen [74] showed that plants, when exposed to pathogen attack, can recruit specific plant growth promoting microbes with biocontrol activity, against the pathogen in question. It is believed that manipulating plant recruited PGPM for inoculant production could be more effective in controlling targeted pathogens, than PGPM isolated from places with no pathogen attack. Biocontrols have the potential to minimise the use of industrially manufactured chemicals in agricultural production. This would mean a decline in burning of fossil fuels, and hence a reduction in greenhouse gas emissions. This is because some pesticides are synthesized in laboratories using hydrocarbons like petroleum, which is a fossil fuel. Reduction in their use can mean a reduction in burning fossil fuels, hence less CO2 emission to the atmosphere. It would also reduce effects on non-targeted members in the ecosystems, which are sometimes affected by chemical use.

Biocontrols may act directly to inhibit growth of biotic agents through hyper parasitism and production of bioactive substances, such as: antibiotics, hydrogen cyanide and phenazines [73], or indirectly through competition for nutrients and active sites on plants, as well as inducing the plant's systemic resistance against the harmful biotic factor [22,38]. Siderophore producing PGPM tend to outcompete other microorganisms for iron sources, which causes inefficiencies in terms of pathogen activities, especially for fungal pathogens, which eventually leads to their death [64]. Induced systemic resistance is triggered by microbe associated molecular patterns (MAMPS), such as lipopolysaccharides, that plants recognise and respond to by turning on their defence systems [75]. Since MAMPS differ among PGPM, it is believed that microbial consortia made up of more than one microbe may induce stronger systemic resistance than single strains [72], although further research needs to be done for a clearer understanding of this potential. PGPM can not only mitigate crop, but also suppress crop pests, such as spidermites [76], moths [77], aphids [78], nematodes [67,79], leaffolder pest [80] and cutworms [81], which greatly contribute to losses incurred in crop production, right from planting to harvesting and storage, if not managed well. PGPR control pests through mechanisms such as production of volatile compounds, such as compounds such as β-ocimene and β-caryophyllene [76], that attract natural enemies of the pest in question. For example, a study by Pangesti et al. [77] showed an increase in the concentration of parasitoid *Microplitis mediator*, a natural predator of *Mamestra brassicae* following the inoculation of *Arabidopsis thaliana* roots with the rhizobacterium *Pseudomonas fluorescens* WCS417r. Other mechanisms through which PGPM mitigate the effects of pests include, increased activity of antioxidant enzymes and increased content of proteins and phenolics in plants, etc. In other cases, the biocontrol agent may not have an effect on the biotic antagonist, but will enhance plant yield in the presence of the antagonist [78]. This particular strategy seems very useful, especially in cases where biotic stress factors such as weeds and pests become resistant or unresponsive to other control strategies. It may also enhance/preserve species diversity, hence maintaining ecosystem functionality.

Some PGPM are efficient against pathogens as single strains, while others perform better as a consortium. Details of single strains vs. consortia are discussed later in this review. Table 1 lists PGPM with potential biocontrol activities against the pathogens of various crop species. With the increasing campaign against the use of chemicals, as a means of combating climate change, such strains are a promising substitute for chemicals that are currently prevalent in agricultural production. Currently, the global biocontrol market is approximately 2 billion USD [79], and is expected to grow further. More research on existing microbial species or microbe-produced compounds with biocontrol properties is still desirable, as is the identification of new ones.

**Table 1.** Biocontrol species against biotic stressors of different crop species.

**Table 1.** *Cont.*

#### **4. PGPM and Abiotic Stress**

With climate change, the occurrence of extreme abiotic stresses, such as floods, salinity, high temperature and drought are expected to increase [3,8–10,105]. In fact, much of this is already being experienced in some parts of the world. Winters are becoming warmer in some regions; rainfall is becoming scarcer and more erratic, causing droughts and desertification [1,2,104] in other regions. With less rainfall, salinity is more likely to occur, either through irrigation or natural causes [106–111]. All these factors affect crop production, and their management inputs are sufficiently costly that many farmers may not be able to afford them. Factors such as high temperatures can generally not be managed under field conditions. Therefore, there is the need for a strategy that is ecofriendly and manageable by the majority of crop producers. PGPM have been reported to mitigate effects of abiotic stress on plants, hence, allowing the plant to grow and yield relatively well under stress conditions [112–115]. Various researchers have reported the ability of a wide range of PGPM to enhance plant growth, in the presence of abiotic stressors, such as salinity [116,117], drought [114,118,119], heavy metals and acidity. In fact, the ability of some PGPMs to enhance plant growth is only triggered in the presence of stress [118]. They employ mechanisms such as the production of ROS scavenging compounds, possession of ACC deaminase (an enzyme that lowers ethylene concentration in plants exposed to stress), and the production of exopolysaccharides and osmolytes. For example, Akhtar et al. [120] observed an increase in the antioxidant activity of catalase (CAT) in the roots of drought stressed maize plants treated with *Bacillus licheniformis* (FMCH001). Treated plants also exhibited a higher dry weight and higher water use efficiency. Yang et al. [121] also reported the increased activity of catalase and dehydroascobate reductase enzymes in salinity stressed Quinoa plants treated with an endophytic bacterium known as *Burkholderia Phytofirmans* PsJN, compared to the untreated plants. The former also exhibited a higher shoot biomass, grain weight and grain yield compared to the latter. Some rhizobia spp. produce the compound rhizobitoxine, which inhibits the activity of ACC synthetase, hence lowering ethylene activity that would otherwise inhibit nitrogen fixation. A PGPM may possess one or more of these mechanisms, all of which act to help a plant thrive under stress conditions. Like plants, PGPM can also be affected by abiotic stress, such as salinity, high temperature and drought, which can lower their efficacy in promoting plant growth, or even death of the microbe, in cases of prolonged exposure to extremes of such conditions [122]. Therefore, it is essential that the strains chosen for use are tolerant to the abiotic stress, whose effect in plants they tend to mitigate. Strains isolated from areas affected by abiotic stress may have an edge over those isolated under normal conditions, although this may not always be the case. The use of microbial consortia may be helpful, especially in areas where more than one factor inhibits crop growth (which is almost always the case under field conditions). However, more research needs to be conducted, for the better deployment of PGPM technology. The exploitation of such microbes has a definite potential to maintain crop production amidst increasing abiotic stresses that are rendering some currently arable land unfit for crop production. Table 2, below, shows some PGPM strains that have been discovered and characterized by researchers, with the potential to mitigate the effects of abiotic stress on a range of plant species.

**Table 2.** *Cont.*

#### **5. Commercialisation of Microbial Inoculants**

Making PGPM technology available for farmers is key to ensuring their adaptation as agricultural inputs. Commercialisation of promising strains is one way of making promising strains accessible by farmers. Although various strains that possess desirable properties under laboratory and greenhouse conditions may be isolated, developing a commercial product, effective under field conditions, is not an easy task, especially because numerous factors determine the efficiency of introduced species. Characteristics such as: possession of multiple mechanisms of enhancing plant growth, ability to compete favorably and establish populations in the rhizosphere, persistence in the rhizosphere over seasons, and ability to be cultured in artificial environments [15,61,89] are desired for potential PGPM strains. However, many plant and soil factors, such as plant species, soil temperature, composition and prevalence of native microbes, soil pH, etc., may work together against a strain which is otherwise excellent under controlled environment conditions. Even before introduction into the field, factors such as formulation play a major part concerning a product's efficacy. For instance, solid inoculant formulations are desired for their longer shelf life, however, the process of drying microbes often results in lower microbial cell counts, hence lowering their competitiveness, since number contributes greatly to their ability to compete with native microbes [134]. Exposing a potential PGPM to some level of stress before formulation may increase its survival rates during formulation and after field application [134]. Before introducing a potential PGPM inoculant into the market, a series of events, such as greenhouse and field trials, characterization, toxicology profiling, etc. occur, most of which are intended to increase strain survival and efficacy in the field.

#### *Formulation of Microbial Inoculants for Commercial Purposes and Their Mode of Application*

Microbial inoculants are usually a combination of microbial cells and/or their parts/compounds and a nonliving carrier that may be in form of a liquid or solid material [15,24,38]. Microbial cells may be either active or dormant; in the latter case, they have to be activated before or after inoculation [15]. They may also be pure cultures (single strains) or a combination of microbial strains (microbial consortia) [15,38]. Formulation is a major contributor to the variation in performance of inoculants observed in farmers' fields and at research stations. Formulation can shield the microbe from adverse environmental conditions, increase their shelf life and also supply their nutritional requirements, hence enhancing their chances of survival in the field [23,134]. Normally, a group of microbes are isolated from their natural habitat (soil or plant tissue), tested for their ability to promote plant growth under a range of conditions, and the superior strains are selected for commercialisation purposes. The strains are multiplied and formulated under controlled environment conditions, after which the efficiency of the inoculant is evaluated under field conditions [23]. The method of formulation ought to consider the target crop, target market and mode of application, the latter because the type of formulation often dictates the mode of application of the inoculant. For instance, solid formulations are mainly applied through seed dressing, or broadcasting onto the field, while liquid formulations have a wide range of application methods [15,24,38]. Liquid carriers are mostly water and/or organic solvents (other than microbial media), such as glycerol and carboxymethyl cellulose that are added to increase properties such as stickiness and dispersal abilities [23]. There are several types of solid carriers, such as clay, vermiculite, peat and charcoal [15]. Care should be taken, when selecting microbial carriers, to ensure they have no negative impact on the environment or the microbe itself [15,38]. Although they are easy to handle and work with, liquid carriers may require specialised storage conditions (cool conditions that necessitate a cooling mechanism) for a long shelf life [23], which makes their marketing and use in developing countries difficult, due to limited and unstable power supply on most farms. Solid formulations, on the other hand, are bulky and may require larger storage facilities, when compared to liquids. However, materials such as peat have an outstanding reputation as inoculant carriers, and are successfully used in both North and South America [23]. The formulation method opted for should ensure the affordability of the final product by the target market, since a very expensive product is likely to meaningfully increase production costs, which is undesirable. For instance, sterile carriers are preferred over nonsterile carriers [23], however the former are costlier than the latter, which may make them unaffordable to many farmers across the globe. The formulation method should also ensure the compatibility of the inoculant with agronomic practices, such as weed control methods, irrigation, etc.

Once a formulated product exhibits positive responses, in field and greenhouse trials, it is put on the market for accessibility by farmers. While the isolation and characterisation of microbial strains from their natural habitats is largely done by academic research institutions, the production of microbial inoculants for commercial purposes is dominated by registered companies, which obtain patents and rights over specific inoculants. Table 3 below shows such microbial based products on the market as plant growth stimulants.

#### **Table 3.** Examples of microbial inoculants currently available on the market, and their producing companies.

#### **6. Limitations to Global Use of Microbial Inoculants**

Although microbial inoculants are viewed as the most viable hope, with regard to sustainable agriculture in the face of climate change, their use and adoption globally are still wanting, due to a range of reasons, that vary between developed and developing countries. Adaptation to use of microbial inoculants is developing at a relatively faster pace [24] in the developed world than in developing areas, such as Africa, where their use is restricted by limited availability of resources and knowledge, among other factors. In the developed world, microbial use is slowed largely by inconsistencies in enhancing plant growth, in which case crop producers opt for chemicals, which generally provide stable results. There are many cases where the excellent performance of an inoculant observed during pre-commercialisation trials does not translate to efficiency on farmers' fields. Even when it does, sometimes the results are not consistent, which frustrates the farmers. Some of these inconsistences may be attributed to biotic and abiotic soil factors and plant factors which directly or indirectly affect the introduced microorganism(s) [23]. For instance, some inoculants are cultivar and species specific, in that applying them outside the target species will yield no results. Soil factors such as salinity and temperature are dynamic and affect the survival and effectiveness of the applied microbial strains. This implies that soil conditions should always be favorable for the introduced microbe, otherwise inconsistencies are bound to prevail. Therefore, there is a need to sensitise farmers regarding the proper use of microbial products to minimise such inconsistencies. Unless sensitisation is properly conducted, we cannot rule out inappropriate practices such as farmers applying rhizobial inoculants together with high doses of nitrogen fertilizer, expecting better results than the inoculant or fertilizer used alone. In fact, nitrogen fertilizer will inhibit biological nitrogen fixation. Similarly, applying a biocontrol to a soil or plant that lacks the pathogen it can antagonise/suppress may not yield results. It is also important to understand the status of the soil/plant as the application of microbial inoculants may inhibit plant growth where the soil/plant already contains optimal concentrations of the compound that the microbe produces to enhance plant growth. For instance, application of IAA producing PGPM on plants with an already optimal concentration of IAA may yield negative effects on the plant, due to excess IAA [43]. Understanding soil conditions will also guide the farmer regarding how often to apply the inoculant. Some require seasonal, annual or even twice in a season application, while after some time, application may not be necessary, especially where the microbe establishes reasonable populations in the soil. Successful microbial inoculants employ mechanisms that give them a competitive advantage over the native strains. For instance, rhizobia and mycorrhizal fungi have a signaling system with their host plants, which gives them an advantage over their competitors. Introduced microbes may also outcompete native microbes through the production of antimicrobials, which may kill or deter other microbes, as well as the production of siderophores that give them a competitive advantage over other microbes for iron resources in the soil, hence proliferating better, especially in iron limited soils [14]. Nevertheless, it is important to increase the competitive advantage of introduced microbes, by ensuring high microbial concentrations in the inoculant and use of adequate formulations [18]. With approaches such as metagenomics, the microbial population of the target environment can be studied, and potential PGPM studied for their ability to out compete the latter in field, greenhouse and laboratory conditions. However, this may not be an easy task, given that microbial populations in crop production fields may differ meaningfully due to a wide range of factors. Location and plant specific nature of some phytomicrobiome elements for inoculant production should also be prioritised, since such microbes, to a great extent, are more adapted to the environment and/or plant conditions, which may increase their chances of survival and persistence in the soil. The idea of using microbial consortia may also work to our benefit, as will be discussed below. This does not, however, disqualify single strain inoculants; their advantages are also discussed below.

In less developed countries, especially in sub-Saharan Africa, reasons for low adoption also vary between large- and small-scale farmers. For large-scale farmers, such as those in Zimbabwe, South Africa and Kenya, the ineffectiveness of many microbe-based products in the field contributes meaningfully to the low adoption of microbial inoculants [137,138]. For small scale farmers, costs and

inadequate knowledge of such products are the major drivers. These two factors, especially costs, also limit the use of other agricultural inputs, such as high-quality seed. Exceptions can be made for a smaller group of small scale farmers, whose farms' researchers run experiments/field trials, because then, they can obtain access to the inputs from researchers largely free of charge, otherwise, they mostly depend on crop rotations (which are sometimes not properly done) and animal manures, while others just grow their preferred crops year in year out. The lack of knowledge can be attributed to the large gap between research and extension. Researchers achieve good findings, but due to poor funding and poor dissemination techniques, this knowledge never reaches the farmer [138]. Publications do not help much, because many small-scale farmers are illiterate, and even those who can read have limited access to technologies such as smart phones, computers and the internet. It should be noted that many small-scale farmers are also low-income earners, who struggle to meet their basic needs. In countries where governments are not directly involved in the distribution of agricultural inputs, dealers may not be willing to extend products to people who they well know cannot afford them, which leads to unavailability of and/or inaccessibility of the products by the farmers. In such cases, intervention strategies should definitely be at least a bit different and more vigorous. First and foremost, the knowledge of existence of PGPM technology needs to be spread to these largely small scale farmers. Projects like N2 Africa have done a good job in trying to spread the BNF technology, although more effort is still needed. Extension officers should be updated on new findings and products, and be properly facilitated to extend this knowledge to the farmers. Governments may consider subsidizing products and getting directly involved in their distribution to the farmers. Promiscuous soybean varieties are already a good strategy of eliminating the need for inoculation. It would be better to develop strategies that enable the use of farm-based PGPM inoculants, as many farmers have limited access to agro-input markets, in part due to poor transport networks. Locally made cooling facilities such as charcoal based refrigerators and unglazed clay pots may also be helpful. However, the former would be a contradictory measure, given that it would encourage deforestation. The whole sensitisation process should involve all stakeholders, such as governments, extension officers, agricultural schools, and private companies that contract small scale farmers to grow crops for them for use as raw materials. The latter, especially, provide the farmer with chemicals such as pesticides and fertilizers; therefore, their involvement cannot be ignored.

#### **7. Microbial Consortia**

In order to address issues associated with the use of single strains as inoculants, microbial consortia have gained popularity. This may be relevant, especially now that the prevalence of both biotic and abiotic stresses due to climate change are likely to increase. Microbial consortia technology involves the use of more than one microbial species in a single inoculant product. The microbes may have the same or different modes of action [18,70,89], and may be from different phyla, genera or even other groupings, for example, a combination of bacterial and fungal strains. Microbial consortia may have an advantage over single strains when the species synergistically interact and confer benefits to each other [70,71,89,117]. For instance, one strain may breakdown a substrate, unavailable to other species, converting it into forms that the other members of the consortium can utilise as a source of nutrients [14], or produce exopolysaccharides which offer protection against stress to all members of the consortium [134], produce compounds which are signals that activate plant growth promotion capability of other members of the consortium, through the production of plant growth stimulating compounds, that they would otherwise not produce, for instance, in pure culture. In cases where microbes with the same mode of action are used, members may have varying tolerance to different biotic and abiotic stresses, which enhances survival of at least a member that will confer intended benefits to the plant. In the case of different modes of action, these complement each other and confer a more effective benefit to the plant. It could also be that some members of the consortium are simply helpers of the strains meant to benefit the plant. Such helper strains, for instance mycorrhiza helper bacteria, should facilitate the target strain in plant colonisation, conferring benefits to the plant. Researchers have reported inefficient strains that became efficient in a consortium. For example, Santhanam et al. [89] observed that the inclusion of two bacterial strains with insignificant effects on mortality of sudden wilt pathogens in tobacco, in a consortium with three other bacteria improved resistance of plants to the same pathogen, in comparison to the consortium of 3 used alone. Mycorrhizal fungi, in association with a helper bacterium, may have better established mycelia and plant root colonisation, if the bacterium produces substances that directly enhance the germination of fungal spores, or indirectly enhance the establishment of mycorrhiza through the production of antimicrobials that reduce competition from other microbes or minor pathogens [14].

Because of the interaction advantage, some microbes perform better in microbial consortia than when applied individually [70,89]. However, the reverse is true for some PGPM species, as reported by other researchers [70,92,100]. Therefore, the role that single strain inoculants play cannot be written off easily, especially because microbial consortia also have their shortcomings. Coming up with effective compatible combinations in which all members actively benefit the plant can be challenging, practically given that some members of the consortium may produce compounds lethal to other members [133]. Even if the produced compounds do not go to the extreme of killing other members, they may cause a shutdown of their plant growth promoting system, or interfere with their growth, as de Vrieze et al. [70] observed in a consortium of five *Pseudomonas* strains. In such cases, it is probable that only a subset of the consortium members will actively benefit the plant, the rest being "dormant" or dead. Difficulties concerning the formulation of microbial consortia may also be associated with the variations in optimal growth conditions. For more than one species, or even genus, creating conditions that will favour all members while retaining their ability to promote plant growth may not be easy. Finally, manufacturing consortia can be challenging, as very small changes at the outset can result in very different levels of consortium members in the final product, resulting in product inconsistencies.

#### **8. Microbial Compounds as "Inoculants"**

The use of microbial compounds as "inoculants" is slowly gaining popularity after successful trials [139–145]. To be a true inoculant, the material must contain living cells that colonize the plant. In this case, the technology may be the product of microbial growth and may be more valuable as a result of climate change were biotic and abiotic factors may lower or completely halt the effectiveness of microbial cell based inoculants. This practice involves the separation of cell free supernatant from microbial cells, and the subsequent separation and purification of the compound from the cell-free supernatant, mainly through high pressure liquid chromatography (HPLC). The pure compound is then tested for its ability to promote plant growth under greenhouse and field conditions, prior to commercialisation. Before commercialisation, other tests, such as the effect of the compound on non-target organisms and humans, as well as checks regarding legal regulations, are usually carried out. The effect of the compound on non-target organisms such as plants, humans and animals ought to be substantially understood too, as with studying the residual effects of the compound (how much of it remains in the edible parts of the plant, and in the soil, following application). Therefore, before any compound can be commercialised, its ability to be purified, and produced on a large scale, should be verified [143]. The compound should be identified and characterised based on its physiological and biological properties.

The efficacy and type of microbial bioactive compounds produced are influenced by microbial species and conditions to which the PGPM is exposed. Slight alterations in growth conditions may result in different compounds, produced at different levels, and with varying degrees of efficacy. For instance, varying the pH, a *Pseudomonas* species culture caused it to produce different phenazine compounds with varying efficacy against *Fusarium oxysporum* f. sp. radicis-lycopersici [143]. Sometimes, the PGPM has to be exposed to stressful conditions before it will produce bioactive compounds, as such compounds may only be produced to enhance the survival of the microorganism under stressful conditions. Therefore, it is important to have an adequate understanding of the conditions under which a certain PGPM will produce plant growth stimulating compounds.

So far, not many (compared to microbial strains) bioactive compounds have been identified for use in crop production. The Smith laboratory at McGill University has thuricin17 and lipochitooligosaccharide (LCO). Thuricin 17 is a bacteriocin secreted by *Bacillus thuringiensis*, a non-symbiotic endophytic bacterium. The compound is known to have anti-microbial properties, which gives *Bacillus thuringiensis* a competitive advantage over other bacteria of the same grouping [140]. After a series of experiments, thuricin 17 was discovered to have growth promoting properties for tomato, soybean, canola, arabidopsis, and rapeseed and switch grass [117,140–142,144,145]. More trials are on-going, and the technology has yet to be commercialised. Lipo-chitooligosaccharide, on the other hand, is produced by rhizobia, as a signal to its host plants [139]. Formerly extensively studied for its role in the nodulation process, the compound is currently patented and being marketed by Novozymes as a plant growth stimulant, where its effects are greatest under abiotic stress conditions. Other compounds such as phenazine-1-carboxylic acid (PCA) have also been commercialised [143,146–153]. Table 4 shows the various compounds with potential use as agro-inputs. Some of them are already commercialised.

**Table 4.** *Cont.*

#### **9. Microbial Cells or Microbial Compounds?**

Given the current understanding, a question would be, what should a crop producer adopt, given a choice between the microbial cells and microbial compound based products. The answer to such a question cannot be as definite as that specific factors may call for either of the two, or even the use of both simultaneously. Before one reaches the level of farmer preferences, soil and environmental factors as well as economic implications, intended use and handling may be major considerations. For instance, in the reclamation of areas heavily affected by abiotic stress, use of microbial cells may not be a good idea, if they are not able to survive some harsh conditions. Even if they did, the efficacy of their plant growth promotion capacity may be greatly affected. Compounds, on the other hand, are less affected by such abiotic stresses, and hence have a greater chance of being successful under such conditions. The use of compounds or both compound(s) and microbial cells may be desirable, especially when an abiotic stress such as drought interrupts signaling between plant and PGPM. In such a case, external application of the signal may rectify the disruption. Prudent et al. [142] observed a 17% increase in soybean biomass under drought conditions following co-inoculation with *Bradyrhizobium japonicum* and thuricin17, compared to inoculation with the rhizobial cells alone. The use of microbial compounds may also be a better choice in cases where the microbe is a facultative pathogen, such as *Agrobacterium* spp. [92]. In such cases, the pathogen effect of the microbe on plants is minimised. Application of microbial compounds may also benefit a wider range of crop species compared to microbial cells, given that many microbes can be at least somewhat species specific. A case would be that of lipochitooligosaccharides (LCOs), which can be utilised to enhance growth of legumes and non-leguminous crops [148], under stressed and non-stressed conditions [115,137], but to a greater extent, under stress conditions. For instance, LCOs enhanced fruit and flower production in tomato (*Lycopersicon esculentum*) plants [148], and stimulated the growth of soybean and corn plants [139]. The compound was also reported to enhance the germination of soybean seeds subjected to high NaCl concentrations [117] and canola [145]. Such benefits from LCO would not be provided to these crops had *Bradyrhizobium japonicum* been applied. Compounds are also less bulky and less costly, in most cases requiring small doses to be efficient. This relieves crop producers of storage and transportation concerns.

However, there are scenarios where the use of microbial cells is inevitable. For instance, the role that rhizobia play in nitrogen fixation, or mycorrhizae in P mobilisation and acquisition by plant roots could not be fulfilled by microbial compounds. Nitrogen fixing bacteria cannot be substituted by compounds in areas were N is limiting. Microbial cells have the potential to establish microbial populations in the rhizosphere, which may eliminate the need for further inoculation, a characteristic most farmers would desire, since it not only has positive financial implications, but also saves labour. Based on this, it is safe to assume that marketing companies would opt for compounds, since they guarantee continuous sales. However, the long and laborious process of isolating and purifying microbial compounds may also contribute to their scarcity and willingness of some researchers and companies to take that route.

#### **10. Way Forward and Recommendations**

With climate change conditions increasing, and the desperate need to come up with sustainable approaches of enhancing crop productivity to meet the food demand of the growing population, microbes are a prominent source of hope. However, a great deal still needs to be done to bridge the gap between their use in developed and developing countries. More research should be done to address issues of inconsistencies observed on crop producers' fields, following the use of microbial inoculants. It is obvious that single strains and consortia, or microbial cells and microbial compounds are issues that need to be evaluated on a case-by-case basis. Therefore, a better suggestion would be that more research be done to provide consumers with options that can address their unique needs, while being economically viable.


**1. Introduction**

Accurately estimating gaze direction plays a major role in applications, such as the analysis of visual attention, research on consumer behavior, augmented reality (AR), and virtual reality (VR). Because inference results are more improved by using deep-learning models than other approaches, they can be applied to advanced technologies, such as autonomous driving [1] and smart glasses [2], and can overcome challenges in the medical field [3]. Using these deep-learning models is quite helpful, but it is difficult to train them due to lighting conditions and insufficient and poor-quality datasets. Moreover, the value of gaze datasets is very expensive and complicated to process. To alleviate this problem, we propose a model that extracts eye features using UnityEyes [4], high-quality synthetic data. An exact position of a feature is obtained from the enhanced model by using a self-attention module. Subsequently, gaze estimation is performed through using high-level eye features, which is less restrictive as it does not utilize complex information, such as full-face information and head poses used for gaze estimation [5–7].

Recently, deep-learning-based eye-tracking technology has been developed mainly through appearance-based methods [5–11] that use eye images or face images. These appearance-based models currently perform particularly well in a controlled, environment

**Citation:** Oh, J.; Lee, Y.; Yoo, J.; Kwon, S. Improved Feature-Based Gaze Estimation Using Self-Attention Module and Synthetic Eye Images. *Sensors* **2022**, *22*, 4026. https:// doi.org/10.3390/s22114026

Academic Editor: Jing Tian

Received: 21 April 2022 Accepted: 22 May 2022 Published: 26 May 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

in which there are no disturbances, such as noise in an input frame. However, these models have some drawbacks. First, the cost of datasets is very high, and the quality of data has a significant impact on the training of the model. Second, most models are black-box solutions, which pose the challenge of locating and understanding points for improvement. This study reduces the dependency of the feature map, which is difficult to interpret and approaches a feature-based method that can estimate a gaze vector through accurate feature points after acquiring landmarks obtained from an image. Refs. [12,13] used a stacked hourglass model [14] to extract a few eyelid and iris points.

In this study, we reinforced and used an advanced model, called HRNet [15], which shows state-of-the-art performance in the pose estimation task to extract high-quality landmarks. In pixel-wise fields, such as pose estimation and landmark detection, the resolutions and sizes of images have huge impacts on performance. Therefore, we extracted landmarks with a high accuracy by remodeling the model using a self-attention module [16–18]. CBAM [18], a self-attention technology, helps to generate a refined feature map that better encodes positional information using channel and spatial attention.

Because we aimed to estimate a gaze vector centered on a landmark extraction, a labeled gaze vector and eye-landmark dataset were essential. However, because gaze data are very expensive and difficult to generate, it is more difficult to obtain a dataset that provides both high-resolution images and landmarks simultaneously. Therefore, UnityEyes, a synthetic eye-image dataset with eye landmarks, was adopted as a training dataset through high-resolution images and an automatic labeling system. The model was trained by processing 32 iris and 16 eyelid points from the eye image obtained by fixing the head pose. Figure 1 shows the predicted heatmaps during the training We evaluated landmark and gaze performance by composing a test set for UnityEyes and performed a gaze performance evaluation using MPIIGaze [11], which has real environment settings.

Our paper is organized as follows. We first summarize related work in Section 2. In Section 3, the proposed gaze estimation method is explained. Section 4 describes the datasets used in the experiments. The experiment results are provided in Section 5. Finally, Section 6 presents our discussion on this study, and Section 7 presents the conclusion.

**Figure 1.** From left to right, the predicted heatmaps are shown as the training epoch increases. The heatmaps have high confidence scores where the landmarks are most likely to be located, and the right bar represents the color space corresponding to the confidence score.

#### **2. Related Work**

The gaze estimation method is a research topic of great interest as it has excellent applicabilities to real environments. As it can be applied to various fields, obtaining and creating accurate gaze values and gaze estimations with less constraints are challenging tasks. In this section, we provide a brief overview of the research related to our method. The studies in each subsection are summarized in Tables 1–3, respectively.

#### *2.1. Feature-Based Method*

Feature-based estimation [13,19–22], a method of gaze estimation, mainly uses unique features that have geometric relationships with the eye. Existing research studies have focused on objects that have a strong visual influences on images.

Roberto et al. [19] used the saliency method to estimate visual gaze behavior and used gaze estimation devices to compensate for errors caused by incorrect calibrations, thereby reducing restrictions caused by user movements. However, the use of these devices increases the error rate as the head moves and interferes with gaze detection.

Some researchers added multiple cameras to compute and focus head movements in multiple directions, extending the influence of eye information and head posture information [20,21]. Head pose has a significant effect on the gaze and requires many restrictions.

To avoid this problem, studies dealing with the static head-pose problem were conducted [13,22] using the convolutional neural network (CNN) model, in which images from a single camera are used to perform gaze estimations based on landmarks as they are less restrictive features. This makes it less difficult to build an experimental environment because there is no need for separate camera calibrations. As only eye images are used for gaze estimations, the dependence of the eye landmark feature vector is increased. After acquiring eye landmarks using a CNN model, a gaze is inferred using support-vector regression (SVR) [23].

Bernard et al. [24] used two gaze maps; one represented the eyeball region and the other represented the iris region. A gaze vector was regressed through the positional relationship between the two gaze maps.

#### *2.2. Landmark Detection*

We used a deep-learning-based pose-estimation model as a tool to acquire eye region features. The landmark-detection task includes a key-point detection to detect a skeleton representing the body structure and a facial-landmark detection to extract landmarks in the face; this is a field that requires a large number of datasets depending on the domain. Some models have a direct regression structure based on the deep neural network and have predicted key points [25]. The predicted key-point positions are progressively improved using feedback on the error prediction.

Some researchers proposed a heatmap generation method through using soft-max in a manner that can be fully differentiated [26]. The convolutional pose machine [27] predicts a heatmap with intermediate supervision to prevent vanishing gradients, which are detrimental to deep-learning models. A new network architecture called the stacked hourglass [14] proved that repeated bottom-up and top-down processing with intermediate supervision is an important process for improving performance. A network structure that used high to low sub-networks in parallel is one of the networks that are currently showing the best performance [15]. For a spatially accurate heatmap estimation, highresolution learning is maintained throughout the entire process; unlike the stack hourglass, it does not use intermediate heatmap supervision, which makes it efficient in terms of complexity and parameters and can generate highly information-rich feature outputs through a multi-scale feature fusion process. Yang et al. [28] introduced a transformer for key-point detections using HRNet as a backbone that extracts a feature map. It causes a performance improvement over existing performance through multi-head self-attention but is computationally demanding.

**Table 2.** A summary table of the landmark detection.

#### *2.3. Attention Mechanism*

Attention mechanisms in computer vision aim to selectively focus on the prominent parts of an image to better capture the human visual structure. Several attempts have been made to improve the performance of CNN models in large-scale classification tasks. Residual attention networks improve feature maps through encoder–decoder style attention modules and are very robust to noisy inputs. The attention module reduces the complexity and parameters by dividing calculations into channels and spaces instead of performing calculations in the typical three-dimensional space manner in addition to achieving a significant effect.

The squeeze-and-excitation module [16] proposes an attention module to exploit the relationship between channels. Channel weights are generated through average pooling or max pooling to apply attention to each channel. BAM [17] adds a spatial attention module in addition to the above channel method and places it in the bottleneck to create richer features. CBAM [18] is not only located in each bottleneck of a network but also forms a convolution block to configure the network. In addition, the performance is increased empirically by using the sequential processing method for channel and spatial attention; this method has an empirically better performance than using only the channel unit and has achieved state-of-the-art performance in the classification field.

The self-attention module with such a flexible structure has been applied to many tasks, such as image captioning and visual question answering [29]. The self-attention module is widely used in detection and key-point detection in which spatial information is important [30,31].

**Table 3.** A summary table of the attention mechanism.

#### **3. Proposed Method**

#### *3.1. Overview of Gaze Estimation Based on Landmark Features*

In this section, we introduce a network structure and a process for extracting a rich and accurate landmark feature vector from an eye image and then estimating a gaze based on it. A series of procedures for estimating a proposed gaze is shown in Figure 2. Eye images can simply be acquired from a single camera. If a frame contains a full-face image, the frame must be cropped to a 160 × 96 sized image centered on the eye area using the face detection algorithm [32]. The image is converted into black and white image for simple processing. This can enhance the performance output of the infrared camera.

**Figure 2.** Overall flowchart of our feature-based gaze-estimation system.

To obtain eye-feature vectors from processed images, we selected HRNet as a baseline model that can generate feature maps containing rich information through fusions with various feature maps while maintaining a high image resolution. HRNet showed the best performance in the key-point detection task, proving its utility. We modified HRNet by additionally using the self-attention module CBAM. Channel-wise and spatial-wise weights were applied to infer the most important channels in the 3D feature map and most important spatial points in the channels. Section 5.1 shows that the proposed model achieved higher landmark accuracy than models in previous studies.

The EAR [33] threshold (*T*), which can be different for each individual, was set using the initial 30 input frames. The EAR ratio value (*E*) was calculated for each frame; if the calculated EAR ratio value was less than the threshold value, it was judged that there was no need to estimate gaze because the eyes were closed. By reducing false-positive errors, it was possible to proceed with a gaze estimation that had a computational advantage. In some cases, 3D gaze regressions use SVR, but we proceeded by constructing an optimal MLP. The architecture configurations of these models have the advantage of being able to proceed one step when learning.

The most important task of our proposed method was to acquire a high-level landmark feature that affects the EAR ratio and gaze. Before training the model, we were faced with the problem of a lack of a dataset, which is a chronic problem of deep-learning models, adversely affects their training, and can result in over-fitting. Landmark datasets are especially expensive, and only a few datasets include both a gaze and a landmark. To avoid this problem, we used a large set of UnityEyes synthetic data for training the dataset. UnityEyes synthetic data is a dataset that includes annotations, such as rich eye landmarks and gazes, by modeling a 3D eyeball based on an actual eye shape created by using the game engine Unity. The models [4] trained with this synthetic dataset showed good performances and had a lot of information and high resolutions; therefore, they are very suitable for processing and applications.

#### *3.2. Architecture of Proposed Landmark-Detection Model*

We used a feature vector for gazes with large amounts of eye landmarks obtained through the model from the input frame that contains eye information from a single camera. To increase the gaze accuracy, it was important to generate a high-level feature, and we set the advancement of the model that extracted a heatmap output most similar to the correct answer as the main goal of this study. Previous studies [12,13] that used eye landmarks as features mainly adopted [14] the production of feature outputs. However, because the feature map is restored through decoding after passing forward from high resolution to low resolution, it is weak in expression learning at a high resolution. Because our model requires extracting more eye landmarks from small-sized eye images than previous

studies [13], feature learning at high resolution, which has high sensitivity to positional information in an image space, was necessary. Therefore, we adopted HRNet, which maintains multi-resolution learning (including a high-resolution), as a baseline model.

The basic structure of HRNet consists of 4 steps and 4 stages. Each step creates a feature that doubles the number of channels with half the resolution of the previous step. Each stage consists of a residual block and an exchange unit, and the feature map of each step is processed in parallel. The exchange unit is an information exchange process through fusion between feature maps of each step through fusion, and the second, third, and fourth stages have one, four, and three units, respectively. At the end of each stage, there are feature fusion and transition processes that increase a step by generating a feature map that is half the previous size. Fusion between multi-scale features includes an up-sampling process that uses 1 × 1 convolution, a nearest-neighbor interpolation in the bottom-up path, and a down-sampling process that uses several 3 × 3 convolution blocks with strides of 2 in the top-down path.

$$input = \{X\_1, X\_2, \ldots X\_r\} \qquad output = \{\mathbf{Y}\_1, \mathbf{Y}\_2, \ldots \mathbf{Y}\_r\}$$

$$\mathbf{Y}\_k = \sum\_{i=1}^r F(\mathbf{X}\_i, k) \quad , \quad F(\mathbf{X}\_i, k) = \begin{cases} \text{identity connection}, & \text{if } i = k \\ \text{up sampling}, & \text{if } i < k \\ \text{down sampling}, & \text{if } i > k \end{cases} \tag{1}$$

Equation (1) describes the feature fusion process. For input {*X*1, *X*2, ... *Xr*} of different resolutions, output features {*Y*1,*Y*2, ... *Yr*} are generated through an element-wise summation of features after down-sampling and up-sampling. *r* represents resolution numbers; if *r* is the same, the widths and resolutions of the input and output are the same.

To solve the problem of the typically acquired eye image having a small resolution, we introduced an additional residual block layer composed of a 3 × 3 convolution to the model to create feature (*Fo*) of the origin resolution that stores the information of the largest resolution. Through the summation of *Fo* and up-sampled *Fb*, more spatially accurate features are created.

Because the heat map, which is the final result of the network, requires accurate spatial information for each channel, we applied CBAM, a self-attention technique, to the normal residual and convolution blocks of each stage. Architecture of the modified network is illustrated in Figure 3. These techniques (adding the residual CBAM layer and applying CBAM to all stages of the residual block) improved the landmark-detection performance, which is described in Section 5.1.

**Figure 3.** Our landmark-detection network architecture used to extract feature map.

#### *3.3. Network Engineering with the Self-Attention Module CBAM*

Attention mechanisms have been widely used for feature selection using multi-modal relationships. Refining the feature maps using attention module helps the network and causes it to perform well and become robust to noisy inputs. Based on empirical results, such as those in [16,17], the CBAM self-attention module has developed rapidly and showed higher accuracy than existing modules in the image classification task through various structure and processing experiments. We judged that the positional information of the refined feature would improve the performance; therefore, we applied the residual block of the network by replacing the CBAM block. The architecture of CBAM is illustrated in Figure 4.

**Figure 4.** Convolutional block attention module (CBAM) architecture in residual block.

CBAM adds two sub-networks that consist of channel attention and spatial attention networks to the basic residual block. Feature *F* ∈ R *C*×*H*×*W* is generated through a 3 × 3 convolution of the residual, which is the output of the previous block. *F* goes through the channel attention and spatial attention networks sequentially. First, in the case of channel attention, the two types of channel-wise pooling, that is, max pooling and average pooling, are performed to obtain weight parameters for channels. Feature vectors *Fmax* ∈ R *C*×1×1 and *Favg* ∈ R *C*×1×1, generated through pixel-wise pooling, share an MLP that has a bottleneck structure with the advantages of parameter reduction and generalization and are merged using element-wise summation. Finally, the product is normalized using sigmoid function to obtain the meaningful weights *Mc*(*F*) ∈ R *C*×1×1 and generate *Fc* ∈ R *C*×*H*×*W* by multiplying *Mc*(*F*) and *F*. The above process is described by using Equation (2).

$$\begin{aligned} M\_{\mathfrak{c}}(F) &= F\_{\text{sigmoid}}(MLP(F\_{\text{max}}) + MLP(F\_{\text{avg}})), \\ F^{\mathbb{C}} &= M\_{\mathfrak{c}}(F) \bigotimes F \end{aligned} \tag{2}$$

Subsequently, using the channel-refined feature (*Fc*) as an input, *Ms*(*Fc*) is generated through the spatial attention module.

$$M\_{\mathfrak{s}}(F^{\mathfrak{c}}) = F\_{\text{sigmoid}}(\mathsf{Conv}\_{\mathcal{T}\times\mathsf{T}}([F\_{\text{max}}; F\_{\text{avg}}])),$$

$$F^{\mathfrak{s}\mathfrak{c}} = M\_{\mathfrak{c}}(F^{\mathfrak{c}}) \bigotimes F^{\mathfrak{c}},\tag{3}$$

$$\text{output} = \text{Residual} \bigoplus F$$

In Equation (3), spatial weight feature *Ms*(*F*) ∈ R 1×*H*×*W* is made by using sequential process pooling, concatenation, 7 × 7 convolution and normalizing using with sigmoid function, then *Fsc* ∈ R *C*×*H*×*W* is merged by multiplying *Ms*(*Fc*) ∈ R 1×*H*×*W* and *Fc*. The output of blocks that are merged by using the element-summation residual and *Fsc* is refined with a focus on 'what' and 'where', respectively. Because we applied this module to the residual block of the processing stage in parallel, the output at each stage contains very rich information and encodes channel information at each pixel over all spatial locations due to attention and fusion.

We applied the CBAM module of the additional residual layer and additionally applied the CBAM module to all steps of the stage. We showed performance improvement through the normalized mean error (NME) value, which is a key-point-detection performance value. Detailed outcome indicators are described in Section 5.

#### *3.4. Gaze-Estimation-Based Eye Landmarks with EAR*

We estimated gaze vectors based on an eye feature that consists of a total of 50 eye landmarks (1 from an eye center, 1 from an iris center, 16 from an eyelid, and 32 from an iris). We extracted high-accuracy eye landmark localization while optimizing and improving the network. As the quality of the landmark extracted by the network improved, the gaze regression performance also improved empirically. Existing feature-based studies [13,34] mainly used the SVR for gaze regressions. We empirically confirmed that the difference between the SVR and multi-layer perceptron (MLP) performance is very small and that the MLP performance is relatively good. The MLP simply contains two hidden layers and uses Leaky ReLU [35] as an activation function. In addition, when the MLP is used, there is the advantage that landmark detection and gaze estimation are possible in one-stage training. The MLP contains two hidden layers and uses Leaky ReLU as an activation function. The co-ordinates used as the inputs are normalized to the distance between the eye endpoints, and all eye points are translated with respect to the eye center coordinates.

To reduce false positives and increase efficiency, we utilized an EAR value. The EAR value was calculated to decide whether an eye was closed or not using 16 eyelid points. We introduced a new EAR metric based on a method that uses 6 points because we could obtain richer, high-quality eyelid points. Figure 5 shows the measured lengths of an eye using images that include a closed eye. We measured the horizontal length through the *p*1 and *p*9 points out of a total of 16 points {*p*1, *p*2, ...*p*16}, and the average value of the remaining seven pairs of points {(*p*2, *p*16),(*p*3, *p*15)...(*p*8, *p*10)} was defined as the vertical length. The EAR was calculated using Equation (4).

$$EAR = \frac{\sum\_{n=1}^{7} \parallel p\_{n+1} - p\_{17-n} \parallel\_1}{\sum \parallel p\_1 - p\_9 \parallel\_1} \tag{4}$$

Because the EAR varies considerably from user to user, we set the EAR threshold ( *T*) to half the median value after receiving the EARs for the initial 30 frames' inputs. Then, if the measured EAR was smaller than *T* , the network did not estimate gaze.

**Figure 5.** EAR was calculated through the displayed landmark coordinates (from *P*1 to *P*16). The blue dots are used to represent the width of the eye and the red the height. The eye on the right is in a state at which there is no need to estimate the gaze.


**1. Introduction**

In the past decades, myeloma research has been focalized on the malignant cell leading to the identification of various genes (i.e., oncogenes and tumor suppressor genes) and of signaling pathways by which the identified genes themselves control survival and proliferation of cancer cells [1–4]. More recently, newly developed technologies have enabled us to investigate cancer cells at the genomic level. Such gene profiling studies are providing insight into the pathogenesis and risk stratification of plasma cell diseases, and help to predict both prognosis and treatment response [3, 4].

junctions, and a discontinuous basement membrane [16]. These alterations are consequent to the rapid neovascularization induced by tumor plasma cells by mean of three different pro-

Angiogenesis and Antiangiogenesis in Multiple Myeloma

http://dx.doi.org/10.5772/intechopen.76931

99

In 1994, Vacca and colleagues [16] demonstrated for the first time that BM microvascular density was significantly increased in MM compared to monoclonal gammopathy of undetermined significance (MGUS) and moreover in active (diagnosis, relapse, and leukemic phase) versus non-active (complete/objective response and plateau) MM. The authors first hypothesized that progression from MGUS to MM is accompanied by an increase in BM microvascular density. Subsequent studies by other groups confirmed the observation of increased angiogenesis in active MM compared to healthy individuals or MGUS

Angiogenesis is the sprouting of new blood vessels from pre-existing ones and is finely regulated [17, 18]. Angiogenesis is essential for tumor growth, invasion, and metastasis starting from the balanced early avascular phase of cancer up to being uncontrolled and unlimited in time during the vascular phase [6, 17, 20]. The angiogenic switch from the avascular to the vascular phase is controlled by the many oncogenes, among which c-myc, c-fos, c-jun, and ets-1 have been recognized [20, 21]. They are activated in tumor plasma cells as a consequence of immunoglobulin translocations and genetic instability [20, 21], and induce the angiogenic phenotype in MM plasma cells [21]. MM plasma cells become CD45-negative and begin to produce VEGF [22]. The same angiogenic switch represents a crucial event for the progression from asymptomatic to symptomatic MM [23]. So, angiogenesis represents an important process in MM progression as well as an important

Vasculogenesis is responsible for the primary development of the vascular system during embryogenesis and is fundamental for the formation of the yolk sac vasculature, of the heart, and of the dorsal aortae [24]. It derives from the differentiation of endothelial progenitors, namely angioblasts, deriving from mesoderm and aggregate into a primitive capillary plexus [24]. Important evidence suggests that vasculogenesis contributes to neovascularization in the bone marrow of MM patients [25–27]. In fact, putative endothelial progenitor cells have been isolated from peripheral blood and several studies have suggested that angioblasts contribute to the formation of tumor neovessels [25, 26]. It has been demonstrated that when CD34+ VEGFR-2+ cells isolated from peripheral blood of MM patients were cultured on fibronectincoated plates and exposed to angiogenic cytokines, they acquire a typical spindle-shaped morphology and express endothelial cell markers (CD34, CD31, Flk-1, Tie-2, and E-selectins) [26]. Moreover, in the BM of MM patients, but not of MGUS patients, some endothelial cells of neovessel wall express on their surface the typical endothelial cell markers: factor VIII-related antigen (FVIII-RA), vascular endothelial-cadherin (VE-cadherin), VEGFR-2, and TIE/Tek, as well as the CD133 staminal antigen whose expression was found in the microvascular wall

together with FVIII-RA or VE-cadherin in some active MM patients [26].

cesses: (i) angiogenesis, (ii) vasculogenesis, and (iii) vasculogenic mimicry [17].

**2.1. Angiogenesis**

patients [17–20].

prognostic factor [17, 19, 20].

**2.2. Vasculogenesis**

Cancer cells interact with all cells composing the microenvironment and with components of extracellular matrix (ECM) [5, 6]. These interactions play the most important role in the epigenetic control of the malignant phenotype, as in primary sites as in the metastatic ones [6, 7]. Moreover, interactions between host cells in the niche microenvironment and ECM represent an intense area of research [5–9]. The aim of these studies is the better understanding of the pathophysiological events in the tumor process, including malignant cells, surrounding cells, and ECM components [5–9].

Multiple myeloma (MM) is a malignancy of plasma cells that home to and expand in the bone marrow (BM) [9]. MM is characterized by a high genomic heterogeneity but, generally, it shows the same histological features, [8–10]. The interactions between MM plasma cells and BM microenvironment (stromal cells, hematopoietic cells, ahnd ECM) represent near genetic modifications an important factor for disease progression [11–14]. Pathophysiological interactions of myeloma cells with the components of BM microenvironment are pivotal during the progression-associated bone disease and neovascularization [13]. These interactions are mediated by autocrine and paracrine loops that regulate multiple signaling pathways and influence many fundamental biological aspects of the malignant phenotype (i.e., apoptosis, survival, proliferation, invasion, bone damage, and angiogenesis) [12–14].

Neovascularization is the formation of new vessels from existing ones (angiogenesis) or from endothelial precursors (vasculogenesis) and represents one of the principal biological process controlled by the interactions between plasma cells and BM microenvironment. It is a constant hallmark of disease progression [11–15]. Angiogenesis is controlled by several angiogenic cytokines [14, 15]. The major of these are vascular endothelial growth factor (VEGF), fibroblast growth factor-2 (FGF-2), and hepatocyte growth factor (HGF) directly secreted not only by the tumor plasma cells but also by stromal cells [14, 15].

The observation of an increased BM angiogenesis in MM, an overexpression of angiogenic cytokines, and their correlation with disease activity, overall survival and the development of new antiangiogenic compounds, led to consider angiogenesis as a new target in the treatment of MM [11–15].


**8. Conclusions**

Problems associated with study design, diagnostic criteria, end-point choice and evaluation, and use of single or several pharmacological agent, make it difficult to evaluate and compare the outcomes of different drug trials in hepatic thrombotic disorders.

Despite the absence of studies against placebo/no treatment to establish absolute efficacy, comparative studies versus a LMWH suggest that danaparoid is effective and safe for the treatment of PVT and not only prevents SOS and TA-TMA but reduces TRM and increases OS after HSCT. It is unclear if the efficacy of danaparoid is improved with the addition of AT or UDCA.

Danaparoid has a high cost of treatment but a cost-efficacy analysis has not been performed. Its safety, particularly the low haemorrhagic risk, may offset some of these costs.

*Danaparoid Sodium: A Review of Its Use in Hepatic Thrombotic Disorders DOI: http://dx.doi.org/10.5772/intechopen.103851*

Given the potentially fatal outcomes of hepatic thrombotic disorders, the ease of administration of danaparoid, its apparent efficacy and safety in patients with a high bleeding risk, danaparoid merits further investigation in the management of hepatic thrombotic disorders. However the dose of danaparoid needs to be optimised.

There remains a great need for sufficiently powered, double-blind, randomised controlled trials, with well defined end-points relevant for clinical practise, to evaluate the short and long-term effects of currently available antithrombotics used in the management of hepatic thrombotic disorders.