**3. Results**

Table 2 summarizes the chemical compositions of the Cr-W-N coatings synthesized using different sputtering powers. The Cr content varied from 13.9 ± 0.7 at % to 35.9 ± 2.3 at %, the W content varied from 21.8 ± 5.0 at % to 42.1 ± 6.9 at %, and the N content varied from 40.2 ± 4.8 at % to 43.1 ± 4.4 at %. The oxygen content fluctuated between 0.9 ± 0.4 and 2.1 ± 0.6 at %.

**Table 2.** Chemical compositions of as-deposited coatings.

Figure 1 shows the XRD patterns of the as-deposited coatings. The main diffraction peaks were observed at diffraction angles of 37.21◦, 43.27◦, 62.85◦, 75.64◦ and 78.73◦, corresponding to the (111), (200), (220), (311) and (222) planes of both the c-CrN and c-W2N phases [22]. The value of these diffraction peaks slightly shifted to lower angles with the increase of W content. This change likely originated from lattice expansion due to the solid solution of W atoms [20,23,24]. In addition, the coatings exhibited a (111) preferential orientation, although the intensity of the strong (111) peak was considerably lower as the W content is increased from 21.8 ± 5.0 at.% to 42.1 ± 6.9 at %, The results are in accordance with those of previous studies, indicating that the doping of a small amount of W to CrN can induce an evolution of the texture from a preferential orientation of (111) to one of (200) [20]. Moreover, the weak (222) peak is very wide and close to the diffraction angle of (311) peak, leading to the asymmetry of the (311) peak.

**Figure 1.** XRD patterns of as-deposited coatings.

Figure 2 shows the XPS spectra of Cr2p, W4f, N1s and O1s in sample C2. As shown in Figure 2a, the binding energies of the Cr2p3/2 and Cr2p1/2 peaks centered at 574.7 eV and 583.4 eV, which correspond to the Cr-N state [25,26]. Meanwhile, the binding energies of Cr2p3/2 and Cr2p1/2 peaks at 576.6 and 586.3 eV correspond to the Cr–O states [27]. The binding energies of W4f7/2 and W4f5/2 peaks located at 31.8 eV and 33.9 eV (Figure 2b), which are associated with W-N binding state [28,29]. The binding energies of W4f7/2 and W4f5/2 peaks at 35.6 eV and 37.8 eV are assigned to the W–O state [30]. Figure 2c displays that the binding energies of N1s peaked at 396.9 eV, 397.7 eV and 399.8 eV, which can be assign to Cr-N and W-N binding states, respectively [25,28]. The O1s spectrum in Figure 2d shows that the binding energies of the O1s peaked at 530.1, 530.7 and 531.8 eV, which can be assign to Cr–O, W–O and H–O states, respectively [31–33].

**Figure 2.** XPS spectra of sample C2: (**a**) Cr2p, (**b**) W4f, (**c**) N1s, (**d**) O1s.

Figure 3 shows the surface images of the as-deposited coatings. Sample C1 exhibited a granular morphology, as shown in Figure 3a, plenty of micropores were distributed around grain boundaries, indicating a loose and coarse growth structure. The surface features changed considerably with an increase of the W content. As shown in Figure 3b, sample C2 exhibited a dense surface morphology, plenty of nm-sized grains were closely packed in large cluster particles. Sample C3 also showed a dense surface (Figure 3c). There were plenty of cluster particles consisting of numerous fine grains, but the sizes of such cluster particles decreased considerably in comparison to sample C2.

**Figure 3.** Surface SEM images of as-deposited coatings: (**a**) C1, (**b**) C2, (**c**) C3.

Figure 4 shows the cross-section TEM images of sample C2. The dark field TEM image in Figure 4a displayed a distinct columnar grain morphology. Moreover, a polycrystalline cubic structure with (111), (200), (220), and (311) reflections was identified according to the selected area diffraction pattern (SADP). The HAADF STEM image in Figure 4b showed an obvious two-layered structure with alternate bright (W2N) and dark (CrN) contrast according to the element mapping image. These multilayer structure

mainly originated from the rotation of sample holder during the coating deposition. Meanwhile, these nano-multilayers exhibited a typical coherent epitaxial growth mode (Figure 4c). The excellent coherence relations were clearly kept between W2N and CrN phases, as shown in Figure 4d, the d-spacing values were 0.145 nm for the CrN phase and 0.146 nm for the W2N phase. These similar d-spacing values were assign to the (220) planes. Additionally, plenty of W atoms were dissolved in the CrN matrix, and some Cr atoms were also identified in the W2N matrix. Both of which indicated that a significant solid solution effect occurred during the coating deposition.

**Figure 4.** Cross-section TEM images of sample C2: (**a**) TEM image with corresponding SADP, (**b**) HAADF STEM image and element mapping, (**c**,**d**) HRTEM images.

Figure 5 shows the AFM images of the as-deposited coatings. Here sample C1 showed a relatively rough surface with a high Ra value of 10.841 nm (Figure 5a). The roughness value decreased considerably with the increase in W content. Figure 5b shows a smooth surface for sample C2, having a low Ra value of 2.417 nm. Figure 5c indicates a similar Ra value for sample C3 (2.698 nm). Based on the XRD, SEM, TEM, and AFM results, we can infer that the CrWN coating has an improved surface quality, although it undergoes aggravated lattice expansion under increasing of W content.

**Figure 5.** AFM images of as-deposited coatings: (**a**) C1, (**b**) C2, (**c**) C3.

Figure 6 shows the XRD patterns of the vacuum annealed coatings. The annealed samples C1 and C2 exhibited similar phase structures to those of the as-deposited coatings. Strong diffraction peaks of mixed c-CrN and c-W2N phases can be seen in the XRD patterns. Simultaneously, weak peaks of h-WN phases can be seen in these annealed coatings, implying a slight phase decomposition. However, the phase decomposition is aggravated significantly with the increase of W content. Apart from the original diffraction peaks of the mixed c-CrN and c-W2N phases, the diffraction intensity of the h-WN peak was found to increase significantly in the annealed sample C3, indicating the formation of a large volume fraction of h-WN phases.

**Figure 6.** XRD patterns of vacuum annealed coatings.

Figure 7 shows the XRD patterns and cross-section TEM images of as-deposited and annealed sample C2. Compared with the as-deposited coating, the annealed coating remained stable structure of CrN and W2N phases, but their diffraction peaks obviously shifted to higher angles, indicating a mitigating lattice expansion. Meanwhile, the as-deposited coating showed clearly two-layered structure with alternate bright (W2N) and dark (CrN) contrast (Figure 7a). By contrast, the annealed coating exhibited obviously three-layered structure with bright (W2N), gray (mixed CrN-W2N), and dark (CrN) contrast, as shown in Figure 7b, a distinct gray diffusion layer forms between CrN and W2N sublayers. The partial enlarged image in Figure 7b reveals that the excellent coherence relations were kept in the diffusion layer. Apparently, the vacuum annealing effectively drove the decomposition of the supersaturated solid solution resulting in the formation of a diffused solid solution matrix.

**Figure 7.** XRD patterns and cross-section TEM images of as-deposited (**a**) and annealed (**b**) sample C2.

Figure 8 shows the surface images and EDS spectra of the vacuum annealed coatings. As shown in Figure 8a, the annealed sample C1 exhibited a granular surface morphology with plenty of newly grown nm particles. The O content in this coating was of 9.6 ± 1.7 at %. Meanwhile, the surface

features of the annealed samples C2 and C3 were slightly different from those of the as-deposited coating (Figure 8b,c). These nm-sized grains in fact grew, leading to a visible coarsening of the cluster particles. The O contents were of 7.4 ± 1.3 at % and 12.1 ± 1.7 at % for annealed samples C2 and C3, respectively. The SEM characterizations clearly confirmed that these coatings suffered slight oxidation damages during the vacuum annealing.

**Figure 8.** Surface images and EDS spectra of vacuum annealed coatings: (**a**) C1, (**b**) C2, (**c**) C3.

Figure 9 shows the roughness values and AFM images of the as-deposited and annealed coatings. The Ra values of samples C1 and C2 increased from 10.841 to 11.332 nm and from 2.417 to 3.204 nm, respectively; by contrast, the Ra value of sample C3 increased abruptly from 2.698 to 4.945 nm, indicating a severe surface coarsening.

Based on the SEM and AFM characterizations, we found that the annealed coatings underwent various degrees of surface coarsening, which was likely triggered by surface oxidation. The EDS results further showed the occurrence of slight oxidation erosion during vacuum annealing, which consequently resulted in the formation of Cr–W oxides [20]. Although these oxides are too small to be detected by XRD in Figure 6, but they led to severe surface coarsening because of their loose structure [20,34]. The significant differences in Ra value between the annealed coatings having different W content can be likely attributed to the varying composition of the oxide layers. According to the EDS results (Figure 8), the volume fraction of Cr oxides decrease considerably and was replaced by a rising volume fraction of W oxides under increasing W content. Previously, it has been reported that dense Cr oxides can act as protective layers and inhibit O diffusion into the coating [35], whereas the W oxides usually exhibit a more porous structure [34]. Therefore, an increase in W oxides can eventually lead to a constant increase of the Ra values.

**Figure 9.** Roughness values and AFM images of as-deposited and annealed coatings: C1, C2, C3.

Figure 10 shows the hardness of the as-deposited and annealed coatings. The as-deposited coatings exhibited a constant hardness enhancement under increasing W content. The hardness was 12.9 GPa for sample C1, and slightly increased to 13.7 GPa for sample C2. By contrast, sample C3 showed a higher hardness of 15.5 GPa. The vacuum annealing induced a remarkable age-hardening in sample C1 and C2. The annealed sample C1 showed a slight increase in hardness (from 12.9 to 14.8 GPa), whereas that of the annealed sample C2 increased rapidly (from 13.7 to 21.6 GPa). Meanwhile, the annealed sample C3 underwent a serious hardness degradation (from 15.5 to 10.1 GPa).

**Figure 10.** Hardness of as-deposited and annealed coatings.

Based on the XRD and TEM characterizations, it can be inferred that the sputtered atoms or ions triggered an obvious injection effect in the as-deposited coatings, as shown in Figure 4d, a significant solid solution effect appeared in both W2N and CrN sublayers. According to the previous results [20,23,24], these solid solution atoms induced serious lattice expansion resulting in a visible peak shift as determined from the XRD test (Figure 1). Meanwhile, this lattice expansion was significantly aggravated following an increase in the W content, which provided an obvious strengthening effect and eventually led to a constant hardness enhancement. Moreover, the addition of W to CrN led to an obvious reduction of the particle (i.e., grains or clusters) sizes (Figure 3) and refined their structure, causing a hardness improvement. The annealed coatings with different W contents exhibited significant differences in phase composition, which consequently had a strong effect on their mechanical properties. As determined from the nanoindentation test, the annealing sample C1 and C2 showed a prominent age-hardening, which mainly originated from their microstructure evolution. As identified in Figures 4d and 7a, these doping atoms induced the formation of supersaturated solid solution in the as-deposited coating, but the supersaturated matrix exhibited a non-uniform distribution resulting in limit strengthening effect. By contrast, vacuum annealing induced spinodal decomposition of supersaturated solid solution to form nm-sized c-CrN, c-W2N, and h-WN domains (see Figure 6). Although a small amount of h-WN phases was formed, but the annealed coating showed similar face-centred cubic (f.c.c) structure to that of as-deposited coating. According to the previous age-hardening theories [36–39], strain fields, originating from the lattice mismatch, acted as obstacles for the dislocation movement and caused hardness enhancement. Additionally, thermal activation greatly driven the homogenization of the supersaturated matrix, as evidenced in Figure 7b, a distinct diffusion layer formed between CrN and W2N sublayers. In comparison to the limited strengthening effect in as-deposited coating, the coherent solid solution diffusion matrix provided strong strengthening effect due to the larger mismatch, and consequently resulted in a profound age-hardening in sample C1 and C2. Nevertheless, the annealed sample C3 underwent serious spinodal decomposition accompanied by the formation of a mixed f.c.c and hexagonal close-packed (h.c.p) matrix structure. The large volume fraction of h-WN phases weakened considerably strengthening effect of the coherent interface and eventually led to a drop of the hardness value.


*3.3. Coke Oven Gas Injection*

In order to decrease the CO2 emissions of the blast furnace, coke oven gas was mixed with hot blast and injected from tuyeres. Table 5 shows the composition of the coke oven gas in Bayi Steel. The total amount of coke oven gas and blast gas was kept constant when injecting the coke oven gas. The proportion of coke oven gas in the gas mixture was from 3% to 15% at a step of 3%.

**Table 5.** Composition of coke oven gas.

The CH4 and O2 in coke oven gas were converted to CO and H2 for simplicity. The replacement ratio of coke oven gas to coke was 0.4 kg·Nm−3. Therefore, for each 3% increase of coke oven gas, the oxygen enrichment rate increased by 0.55%. The production increased by 246 ton HM and the coke ratio decreases by 16kg·ton HM−1. The temperature profiles at different coke oven gas injection ratios are shown in Figure 6. The carbon solution loss ratio under different coke oven gas injection ratios was also calculated after simulation, as shown in Figure 7.

**Figure 6.** Temperature profiles at different proportions of coke oven gas injected.

It can be seen from Figure 6 that the temperature decreases with the increase of the proportion of coke oven gas injected, and the temperature in the middle zone increase relatively more. When the proportion of coke oven gas injected is 15%, the facet average temperature at 14 m height decreases by 210 ◦C than the base model. This suggests that the cohesive zone widens with the increase of proportion of coke oven gas injected.

**Figure 7.** Carbon solution loss ratio at different amount of coke oven gas injected.

It can be seen from Figure 7 that the carbon solution loss ratio of the coke increases gradually as the coke oven gas injection increases. For each 3% increase in the coke oven gas injection ratio, the carbon solution loss ratio of coke increases by an average of 0.83%. The carbon solution loss ratio of the coke increases by 4.15% when the coke oven gas injection is 15%.

The reason may be that the replacement of hot blast with coke oven gas leads to an increase of H2 in the gas mixture due to the high H2 content of the coke oven gas. This results in a higher H2O content in the blast furnace due to the reduction reaction of the H2 and iron ore. The higher H2O content may be part in a water gas reaction, which increases the reaction of the carbon solution loss. The temperature of the strong reaction between the coke and the H2O is from 800 to 1300 ◦C. The average H2O content in the gas in the blast furnace increases from 2.45% to 8.67% when the coke oven gas ratio increases from 0% to 15%. Moreover, the high temperature zone shrinks after the injection of the coke oven gas, which leads to the expansion of the zone for the reaction of the carbon solution loss.

Therefore, on the one hand, coke oven gas can reduce the coke rate and CO2 emission of the blast furnace as a hydrogen-rich gas. On the other hand, the injection of coke oven gas causes the temperature in the lower part of the blast furnace to decrease and the carbon solution loss to increase, which impairs the operation of blast furnace. Therefore, coke oven gas should not be injected into the blast furnace in Bayi Steel in its present condition. Coke oven gas should only be injected only if the carbon solution loss is low enough, and it should be done a higher gas temperature and with better coke quality of a high M40 and a high CSR (coke strength after reaction).

#### *3.4. Steel Scrap Charging*

Since the steel market has been strong in China recently, steel scraps are charged so as to increase production. The metallic iron (MFe) of steel scrap is 52.29% and the FeO is 9.07%. The steel scrap is added when the total amount of iron-bearing materials is kept constant. The proportion of steel scrap is from 0% to 10% at a step of 2%, as shown in Table 6, where the absolute amount is also illustrated. For each 2% increase of steel scrap, the oxygen enrichment ratio increases 0.06%, the coke rate decreases 4 kg·ton HM−1, and production increases 70 ton HM while the coal rate and the amount of hot blast remain unchanged. The temperature profiles at different proportions of steel scrap added are shown in Figure 8. The carbon solution loss ratio is calculated after simulation, as shown in Figure 9.

**Table 6.** Amount of steel scrap added.

**Figure 8.** Temperature profiles at different proportions of steel scrap added.

It can be seen from Figure 8 that the temperature increases a little in the lower zone of the blast furnace with the increase of the proportion of steel scrap added, but the degree is very small. When the proportion of steel scrap added is 10%, the facet average temperature at 4 m height increases by 22 ◦C than the base model. Hardly any difference of temperature in the upper zone of the blast furnace can be found. This indicates that the cohesive zone only narrows a little with the increase of the proportion of steel scrap added.

**Figure 9.** Carbon solution loss ratio at different amounts of steel scrap added.

The carbon solution loss ratio decreases gradually as the proportion of steel scraps added increases; when 10% steel scrap is added, it decreases by 2.6%. The reason may be that the amount of iron ore that needs to be reduced decreases when steel scraps are added in the top of blast furnace since steel scrap provides iron content that does not need reduction. However, this leads the utilization ratio of the gas to decrease. The ratio decreases by 1.32% when the proportion of steel scraps increases to 10%. As a result, the CO content in the gas increases while the CO2 content decreases. This also leads to a decrease of the carbon solution loss ratio.

It should be noticed that the addition of steel scrap reduces the gas utilization ratio, and also the temperature in the upper part of the blast furnace. But the reduction in the gas utilization ratio and the temperature of the upper part of the blast furnace is small and within acceptable limits. Therefore, the addition of steel scrap is beneficial for the operation of blast furnace in the aspect of carbon solution loss. Steel scrap can be added to the blast furnace in appropriate amounts.

#### **4. Conclusions**

A three dimensional model was established to analyze the carbon solution loss of coke based on the practical operational parameters of blast furnace B in Bayi Steel. The model was then used to investigate the effects of the oxygen enrichment ratio, the injection of coke oven gas, and the addition of steel scraps on the carbon solution loss ratio of the coke. The conclusions are as follows.

(1) The carbon solution loss ratio of the blast furnace studied is 18.61% when the coke reaches 1300 ◦C.

(2) The carbon solution loss ratio decreases with the increase of the oxygen enrichment ratio or when a certain proportion of steel scraps is added, while it increases with an increase in the amount of coke oven gas injected.

(3) The oxygen enrichment ratio and the proportion of steel scraps added can be increased to 5% and 10%, respectively in order to reduce the carbon solution loss without affecting the operation of the blast furnace.

(4) The injection of coke oven gas is not recommended given the current condition of blast furnace B of Bayi Steel.

**Author Contributions:** Conceptualization, M.K. and S.W.; methodology, H.Z. and Z.H.; Software, M.K and H.Z.; validation, S.Y. and H.X.; formal analysis, L.P.W. and H.X.; investigation, M.K. and H.Z.; resources, H.X. and S.W.; data curation, Z.H. and S.Y.; writing, M.K and Z.H.; supervision, M.K and L.P.W.; funding acquisition, M.K., L.P.W. and H.X.

**Funding:** The authors would like to acknowledge the financial support by National Key R&D Program of China (grant number 2017YFB0603800, 2017YFB0603803), the National Natural Science Foundation of China (grant number 51804027) and USTB-NTUT Joint Research Program (grant number TW201909, NTUT-USTB-108-06).

**Acknowledgments:** The authors would like to appreciate much for the anonymous reviewers and editors for the improvement of this work, and Prof. Mark Buck for correcting language.

**Conflicts of Interest:** The authors declare no conflict of interest.


*2.2. Tissue Morphology*

H&E and trichrome-stained gular tissue cross-sections are shown in Figure 3. Gular tissue dissected from frogs belonging to the genus *Xenopus* are substantially thicker and contain a higher density of muscle bundles and mucous glands. Observations of the collagen structure between the families showed marked contrasts. The collagen structure in the male *EC* had a continuous and crimp structure, in stark contrast to the female *EC*, *XL*, and *XM*, which have more spread out crimp structures. Additionally, the tissue sample of the genus *Xenopus* has a nearly linear collagen structure. Interestingly, *XL* and *XM* have distinct regions where the collagen is perpendicularly aligned. These areas seemingly discretize the tissue into collagen segments.

**Figure 3.** Tissue morphology by histology of gular tissue dissected from various species of frogs. Scale bar 100 μm. Muscle bundles (black arrows), mucous glands (red arrows), collagen structure (green arrows), perpendicularly aligned collagen (yellow arrows), urothelium (orange), lamina propria (purple), and detrusor muscle (grey).

Histological cross-sections of leg tissue show similar trends with the collagen structures between the frog species. *EC* tissue has a more pronounced crimp structure compared to *XL* and *XM*. Once again, the muscle bundles and mucous glands constitute a more substantial portion of the tissue in *Xenopus* samples. Collagen in the male *EC* leg has a less compact crimp structure than that found in the gular tissue. Similar to the gular tissue of the *Xenopus* samples, the collagen structure is more linear and has minimal crimping. Leg tissues do not exhibit the same inflatable function as the gular tissue; therefore, it does not require the crimped collagen structure. Lastly, sectioning of fresh rat bladder showed an abundance of muscle bundles with intermittent collagen fibers, dissimilar to the frog tissues. Collagen ultrastructures found in gular tissues were not visually identified in the rat bladder seen in Figures 3 and 4.

**Figure 4.** Hematoxylin and eosin (H&E) staining of leg tissue dissected from various species of frogs. Scale bar 100 μm. Muscle bundles (black arrows), mucous glands (red arrows), collagen structure (green arrows), perpendicularly aligned collagen (yellow arrows), urothelium (orange), lamina propria (purple), and detrusor muscle (grey).


**Part 5**

**The Interaction of the Solar Wind with the Magnetosphere**

314 Exploring the Solar Wind

Valdés-Galicia, J.F. Energetic particle transport coefficients in the heliosphere, Space Science

Valdés-Galicia, J.F., Low energy galactic cosmic rays in the heliosphere, Adv. Space Res., 35,

Van Allen, James A., Editor. Cosmic Rays, the Sun and Geomagnetism: The works of Scott E.

Vashenyuk, E.V., Yu. V. Balabin, and B. B. Gvozdevsky. Features of relativistic solar proton

Venkatesan, D. and Badruddin. Cosmic ray intensity variations in the 3-dimensional

Vilmer, N. A. L. MacKinnon, and G. J. Hurford. Properties of Energetic Ions in the Solar

Vivek Gupta and Badruddin, Solar magnetic cycle dependence in corotating modulation of

Wang, F. and Agrawal, V.D. Soft Error Considerations for Computer Web Servers, Proc.

Webber, J.W. and J.A. Lockwood. Characteristics of the 22-year modulation of cosmic rays

Zank, G.P. W.H. Matthaeus, J.W. Bieber and H. Moraal, The radial and latitudinal

Zarrouk, N. and R. Bennaceur, Extrapolating cosmic ray variations and impacts on life:

Morlet wavelet analysis, Acta Astronautica, 65, 1-2, 262-272, 2009.

dependence of the cosmic ray diffusion tensor in the heliosphere, J. Geophys. Res.,

galactic cosmic rays, Astrophys Space Sci., 321: 185–195, 2009.

42nd Southeastern Symposium on System Theory (SSST), 2010.

as seen by neutron monitors. JGR, 93, 8735-8740, 1988.

spectra derived from ground level enhancement events (GLE) modeling,

Atmosphere from -Ray and Neutron Observations, Space Sci. Rev., 159:167–224,

Reviews, 62, no. 1-2, 67-93, 1993.

Forbush, AGU, pp. 471, 1993.

103, A2, 2085-2097, 1998.

Astrophys. Space Sci. Trans., 7, 459–463, 2011.

heliosphere, Space Sci. Rev., 52, 121-194, 1990.

755–767, 2005.

2011.

**0**

**14**

*USA*

**Impact of Solar Wind on the Earth**

**of Ring Current and Radiation Belts**

Natalia Buzulukova1, Mei-Ching Fok2 and Alex Glocer2

2*NASA Goddard Space Flight Center*

**Magnetosphere: Recent Progress in the Modeling**

When solar wind interacts with the Earth's magnetosphere it causes disturbances in the near-Earth plasma environment. Large disturbances result in geomagnetic storms, and affect not only Earth magnetosphere but also space-borne and ground-based technological systems. The systematic studies of cause-effect relations between solar wind variations and resulting disturbances in near-Earth plasma environment as well as the construction of relevant numerical models are important subjects of *Space Weather*, which is currently a very

A most common reaction of the magnetosphere to prolonged solar wind disturbances are geomagnetic storms. Understanding geomagnetic storms and their solar wind drivers is one of the most important problems in geophysics and space weather. A geomagnetic storms is defined by a deviation of the H component of the Earth's magnetic field at low latitudes, i.e., the Dst index. It is believed that the main source of this deviation is the so called ring current which is comprised of plasma with energies 1–200 keV. Hence, to understand geomagnetic storms, we need to understand how the ring current is formed. Many previous studies have addressed the question of solar wind drivers for geomagnetic storms. Most of these were statistical in nature (Borovsky & Denton, 2006; Denton et al., 2006; Turner et al., 2009; Weigel, 2010; Yermolaev et al., 2010; Zhang, Richardson, Webb, Gopalswamy, Huttunen, Kasper, Nitta, Poomvises, Thompson, Wu, Yashiro & Zhukov, 2007). They focused upon the effectiveness of different types of solar wind structures in producing geomagnetic disturbances (e.g., a combinations of global activity indices and/or various

In addition to statistical analysis, a great deal of effort has been put into the development of ring current models. Typically a kinetic approach is used in which modeling ring current plasma is described with a number of species. Each population has its own energy and drift velocity which is a sum of the gradient/curvature drift and E×B drift. Plasma in this representation is generally anisotropic and is not in a thermodynamic equilibrium. The corresponding transport equations are usually written in terms of bounce-averaged quantities. This approach is sometimes referred to as 'drift physics' approach and has proven to be very useful in describing the plasma population of inner magnetosphere with energies

**1. Introduction**

active topic of research.

coupling functions) and different types of storms.

1*NASA Goddard Space Flight Center/CRESST/University of Maryland College Park*


*3.3. Translation and Environmental Stress Responses*

Translation is the final step in chloroplast gene expression. Chloroplast gene translation regulates protein accumulation to optimize photosynthetic performance and to attenuate photooxidative damage. Thus, the regulation of chloroplast gene translation represents a unique component of plant responses to internal and external stimuli.

Most plants growing in direct sunlight routinely encounter high-light stress; the resulting high photon flux exceeds the photosynthetic capacity, thereby damaging the chloroplast. To explore the regulation of chloroplast gene translation during the rapid adaptation of plants to high light, a systematic ribosome profiling study was performed to detect changes in chloroplast gene translation efficiency in tobacco seedlings following transfer from moderate light to high light. The ribosome

occupancy on *psbA* transcripts (encoding PSII reaction center protein D1) increased in response to high-light treatment [115]. Given that D1 protein is the main site prone to photodamage by high light, the upregulated *psbA* translation should substantially facilitate the repair of PSII under high-light stress. However, the molecular mechanisms underlying the translational activation of *psbA* under these conditions remain to be further explored.

Studies on the functions of chloroplast ribosome proteins have revealed that maintaining sufficiently high chloroplast gene translation efficiency is important for proper chloroplast development at low temperature. Maize (*Zea mays*) mutants with a loss of ribosomal protein RPS17 were pale green when grown at moderate temperature (27 ◦C) but appeared albino under cool conditions (17 ◦C) [83]. Tobacco mutants with a loss of the ribosomal protein Rpl33 showed no visible phenotypes at any stage of development under standard conditions, with similar development, growth rates, and onset of flowering to wild-type plants. However, the *Rpl33* knockout mutants were sensitive to cold stress, although not to heat or to low or high light levels. When *Rpl33* knockout mutants were transferred to cold-stress conditions (4 ◦C), they exhibited strong photooxidative damage symptoms and recovered much more slowly from low-temperature stress than wild-type plants [85]. As with Rpl33, the loss of the ribosomal protein Rps15 in tobacco resulted in a growth phenotype almost identical growth to that of wild-type plants, although young plants grew slightly more slowly and the onset of flowering was slightly delayed. However, the *Rps15* knockout mutants were cold sensitive, with more severe pigment loss and worse photosynthetic performance than wild-type plants [84]. Together, these findings suggest that the maintenance of plastid translational capacity is important in enabling plant tolerance to chilling stress.

In Arabidopsis, the expression of the chloroplast ribosome protein gene *RPS1* was considerably induced by heat stress (2 h at 38 ◦C).  When *rps1* seedlings were exposed to transient high-temperature conditions (3 h at 45 ◦C), they were much more heat sensitive than wild-type seedlings, as almost no mutants survived after a 7-d recovery, whereas more than 90% of wild-type seedlings did. However, there were no significant differences between *rps1* and wild-type plants under osmotic and salinity stress. These results suggest that decreased *RPS1* expression alters cellular heat stress responses by disrupting chloroplast gene translation rather than through general physiological defects. RPS1 is required to activate the expression of *HsfA2* (*HEAT STRESS TRANSCRIPTION FACTOR A-2*), a highly heat-shock-inducible gene encoding a transcription factor that is crucial for triggering cellular responses to heat stress. The constitutive expression of *HsfA2* was sufficient to rescue the heat-sensitive phenotype of *rps1* mutants, suggesting that the defective expression of *HsfA2* is responsible for the heat-sensitive phenotype of *rps1* mutants. Like the *rps1* mutant phenotype, treatment with lincomycin, an inhibitor of chloroplast gene translation, also led to an obvious reduction in the expression of *HsfA2* in response to heat stress [82]. These findings reveal a plastid-to-nucleus retrograde signaling pathway that regulates chloroplast gene translational capacity to transcriptionally activate cellular heat stress responses, especially the HsfA2-dependent heat tolerance pathway.

#### **4. Conclusions and Future Perspectives**

Photosynthesis, one of the most important physiological processes in plants, is highly sensitive to environmental stresses. These stresses often inhibit photosynthesis considerably [116–124]. Many studies have reported that chloroplasts can act as sensors of the external environment [7,19]. Thus, in addition to hosting photosynthesis, chloroplasts play important roles in plant responses to various environmental stresses. Likewise, the proper expression of chloroplast genes is crucial for chloroplast development, photosynthesis, and plant development. However, as summarized in this review, many studies indicate that chloroplast gene expression is also important for plant stress responses.

Further elucidating the roles of chloroplast gene expression in plant responses to various environmental stresses would lay the foundation for genetically improving plant tolerance to the environment. However, the underlying molecular mechanisms remains largely unknown. Most studies have focused on the roles of nucleus-encoded proteins in regulating chloroplast gene expression and plant responses to environmental stress, while the direct connection between chloroplast gene expression and environmental stress responses has been largely ignored. Future studies should therefore address two major issues: How environmental stress triggers chloroplast gene expression, and what roles chloroplast gene expression plays in plant responses to environmental stress. Several specific issues need to be investigated. For example, D1 protein synthesis is important for PSII repair under high-light conditions [71,115,125–128]. Thus, enhanced *psbA* transcription and translation would help increase the tolerance of PSII to high light; however, it is still unclear to what extent high light triggers the transcription and translation of *psbA*. In addition, RNA editing, particularly C-to-U base conversion, is a widespread phenomenon in chloroplasts across nearly all plant species [48]. As discussed above, it is clear that RNA editing is important for plant tolerance of low temperature [74–76], yet the exact editing sites, and how RNA editing at these sites improves plant tolerance to low temperatures, remain to be explored. Finally, translation and splicing of several chloroplast genes trigger plastid-to-nucleus retrograde signaling and ABA signaling [73,77,79,80,114], but it remains unclear how the signals generated from these processes are transferred out of chloroplasts and integrated into these signaling pathways.

New technology is needed to better investigate the molecular mechanism of chloroplast gene expression in response to the environment. Chloroplast transformation is an extremely time-consuming and difficult process that has only been achieved in a few plant species, such as lettuce, poplar, and Arabidopsis. This make it extremely difficult to control chloroplast gene transcription using traditional genetic engineering approaches. To further explore the connection between chloroplast gene transcription and environmental stress responses, novel tools must be designed that are similar to CRISPR/Cas9 and RNA interference to knock out and knock down chloroplast genes. It is also critical to design an artificial RNA editing system to carry out RNA editing of specific sites in chloroplasts. This system would be useful for investigating the specific roles of different RNA editing sites in plant responses to environmental stresses and even creating new RNA editing sites that could improve stress tolerance in plants.

**Author Contributions:** Y.Z. and C.L. conceived and planned this review paper designed the project. Y.Z., C.L., A.Z., and X.L. prepared and drafted the manuscript. Y.Z. and C.L. revised the manuscript. All the authors read and approved the final manuscript. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was supported by the National Natural Science Foundation of China (reference numbers 31730102 and 31670238).

**Acknowledgments:** This work was supported by the National Natural Science Foundation of China (reference numbers 31730102 and 31670238). The authors are grateful to the authors of the excellent papers discussed. The authors also apologize to the authors of the other excellent papers in this area that could not be discussed because of lack of space.

**Conflicts of Interest:** The authors declare no conflict of interest.


*Article* **Efficiently Mastering the Game of NoGo with Deep Reinforcement Learning Supported by Domain Knowledge**

**Yifan Gao 1,\*,† and Lezhou Wu 2,†**

**Abstract:** Computer games have been regarded as an important field of artificial intelligence (AI) for a long time. The AlphaZero structure has been successful in the game of Go, beating the top professional human players and becoming the baseline method in computer games. However, the AlphaZero training process requires tremendous computing resources, imposing additional difficulties for the AlphaZero-based AI. In this paper, we propose NoGoZero+ to improve the AlphaZero process and apply it to a game similar to Go, NoGo. NoGoZero+ employs several innovative features to improve training speed and performance, and most improvement strategies can be transferred to other nonspecific areas. This paper compares it with the original AlphaZero process, and results show that NoGoZero+ increases the training speed to about six times that of the original AlphaZero process. Moreover, in the experiment, our agent beat the original AlphaZero agent with a score of 81:19 after only being trained by 20,000 self-play games' data (small in quantity compared with 120,000 self-play games' data consumed by the original AlphaZero). The NoGo game program based on NoGoZero+ was the runner-up in the 2020 China Computer Game Championship (CCGC) with limited resources, defeating many AlphaZero-based programs. Our code, pretrained models, and self-play datasets are publicly available. The ultimate goal of this paper is to provide exploratory insights and mature auxiliary tools to enable AI researchers and computer-game communities to study, test, and improve these promising state-of-the-art methods at a much lower cost of computing resources.

**Keywords:** artificial intelligence; deep learning; AlphaZero; NoGo games; reinforcement learning

#### **1. Introduction**

The successive appearance of AlphaGo [1], AlphaGo Zero [2], and AlphaZero [3], achieving remarkable performance in one of the most complex games, Go, demonstrate the capabilities of deep reinforcement learning.

In 2017, Deepmind's AlphaGo Zero showed the possibility for computers to achieve superhuman performance in Go without relying on human knowledge or pre-existing data. Subsequently, AlphaZero made outstanding achievements in chess and shogi. However, a large amount of computational resources was required. Deepmind ran the training progress for Go for several days with 5000 TPUs, while Facebook's ELF OpenGo used 2000 V100 GPUs to achieve the top level of performance [4]. Therefore, it is an important and meaningful research direction to improve AlphaZero with limited computational resources.

This paper introduces several methods to speed up the training process and improve the final performance of the original AlphaGo Zero model (pipeline shown in Figure 1). The reinforced model is called NoGoZero+. Although NoGoZero+ uses some domainspecific features and optimization methods, it still starts from a random policy without using external strategic knowledge or existing data. Additionally, techniques used in

**Citation:** Gao, Y.; Wu, L. Efficiently Mastering the Game of NoGo with Deep Reinforcement Learning Supported by Domain Knowledge. *Electronics* **2021**, *10*, 1533. https://doi.org/10.3390/ electronics10131533

Academic Editor: Amir Mosavi

Received: 31 May 2021 Accepted: 23 June 2021 Published: 24 June 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

NoGoZero+ can be transferred to other nonspecific domains. By comparing the training process under the same condition in NoGo, the training efficiency of NoGoZero+ was at least six times that of the original AlphaZero.

**Figure 1.** AlphaZero's pipeline. Self-play games' data are continuously generated and collected to train deep neural networks. After each round of training, the new model is compared with the previous model. If the new model defeats the previous model, the training process continues. Otherwise, the previous training result is discarded and the previous step is restarted.

NoGo is a new kind of board game that originated in 2005 [5]. Different than traditional Chinese board game Go, it forbids players from capturing stones. Once a player has no choice but to capture the counterpart's stones, the player loses. The formal rules of NoGo are shown below [6]:

Because of NoGo's novel rules and extremely limited background studies, NoGo does not have a mature strategy. This paper creates a precedent of successfully applying reinforcement learning to NoGo.

The main contributions are summarized as follows:

First, the paper proposes methods that can speed up the training progress and improve the final performance. These methods can be either directly or indirectly applied to similar AlphaZero training processes or general reinforcement-learning processes.

Second, the authors applied NoGoZero+ to the NoGo game and obtained better results than those achieved by the original AlphaZero under the condition of limited resources. This result shows the efficiency gap between AlphaZero's general methods and indicates the existence of better methods under specific conditions.

Third, to help research in this field, we provide the source code used to train the model, some pretrained high-level models, and a comprehensive self-play dataset that contains about 160,000 self-play games (available online: https://github.com/yifangao18/ NoGoZero (accessed on 20 June 2021)).

The rest of this paper is organized as follows. In Section 2, we present some closely related works. The paper summarizes the basic architecture in Section 3 and describes the proposed techniques. In Section 4, we introduce experimental settings, criteria, and details. Section 5 reports the result and reviews the performance of NoGoZero+ in the competition. Section 6 discusses the results obtained with our approach. Lastly, we conclude our work and plan future works in Section 7.

#### **2. Related Work**

In this section, we first go over the AlphaGo family, which first introduced deep reinforcement learning (DRL) to board games with a large searching space, and some famous board game AI based on AlphaGo-like methods. Then, we introduce early work on the game of NoGo. Most of them achieved great success in the early years.

#### *2.1. State of the Art in Board Game AI*

The AlphaGo family, including AlphaGo, AlphaGo Zero, and AlphaZero [1–3], have had great success in many complex board games such as Go and chess. Unprecedentedly, AlphaGo family introduced DRL to board games, and DRL-based methods attracted many researchers' attention when AlphaGo beat top human player Lee Sedol. In particular, AlphaGo Zero successfully trains a Go AI from scratch using only the rules of the game because of the successful combination between Monte Carlo tree search (MCTS) [7,8] and deep neural network. Furthermore, the so-called zero-learning method used by AlphaZero is a more general method that can be used both in board games and other, more practical areas.

However, the large amount of computational resources consumed by the training process of AlphaZero and the relatively immature network structures encouraged many researchers to look into improved methods based on AlphaZero. An open-source project called ELF OpenGo [4] reached superhuman level in the game of Go after two weeks of training on 2000 GPUs (a relatively small number compared with the 5000 TPUs used by AlphaZero). KataGo [9], a reimplementation of AlphaGo Zero, improved the learning process in many ways, including using different optimization approaches to have more data about the value in a shorter period of time and using additional training targets to speed up the training process. Leela Zero [10], which is an open-source program trained with GPUs donated by a community of contributors, mastered Go and chess. Morandin et al. [11] proposed a sensible artificial intelligence (SAI) that plays Go to overcome the problem that AIs cannot target their margin of victory, and this is the common problem shared by most of the famous early-game AIs based on AlphaZero. Thus, the SAI successfully overcame the negative consequences: AIs often win by a small margin, cannot be used with komi 6.5, and show lousy play in handicap games.

#### *2.2. NoGo AI Research*

NoGo, as opposed to the ancient game of Go, is becoming the new favorite in the game AI community because of its relatively easy rules and lower computational resource requirements. Some early studies on NoGo AI achieved great results. Lee et al. [12] proposed an approach using ontologies, evolutionary computation, fuzzy logic, and fuzzy markup language combined with a genetic-algorithm-based system. Their NoGo AI could analyze the situation of the current board and play the next move to an inferred good-move position. Sun et al. [13] put forward a static-evaluation method to accurately estimate the value of each state of NoGo. Sun et al. [14] successfully used an improved pattern-matching algorithm to find out the best move in the game of NoGo.

As far as we know, there are few studies about the combination of DRL or zerolearning and the game of NoGo. Although former NoGo AI combined with traditional methods achieved relatively good performance, the AlphaGo family showed that there is a great performance gap between traditional methods and the DRL method. As a result, the implementation of DRL and zero-learning method in the game of NoGo is necessary.

#### **3. Methods**

In this section, we introduce methods that we used to build up NoGoZero+. We first go over the basic architecture of NoGoZero+. Then, we introduce the main novel techniques that we used to improve the training process and the final performance of NoGoZero+.

#### *3.1. Basic Architecture*

While NoGoZero+ has various novel details and improvements, it has a similar basic architecture to that of AlphaZero. We basically followed the parameters in [9] because this study provides complete parameter information compared to other AlphaZero implementations.

MCTS is the core searching method of AlphaZero, which is guided by a neural network. NoGoZero+ uses it to play the game against itself to generate training data. It also uses a variant of PUCT [15] to balance exploration and exploitation, and *cpuct* is a constant that determines the importance of both. When *cpuct* is small, MCTS tends to exploit game states with high value. Conversely, when *cpuct* is large, MCTS tends to explore unknown move locations, and the exploration is guided by the policy prior outputs by the neural network. With playouts repeating, the searching process continues, and the search tree grows. The process of playouts starts from the root and goes down the tree, and each node n selects child c with the largest PUCT(c) value:

$$PLICT(c) = V(c) + c\_{punt} P(c) \frac{\sqrt{\sum\_{c'} N(c')}}{1 + N(c)} \tag{1}$$

where *V(c)* represents the average predicted score of all nodes in *c*'s subtree, *P(c)* represents the policy prior of *c* from the neural network, *N(c)* represents the number of playouts that are used to go through child *c*, and we set *cpuct* = 1.1. The Iteration of MCTS is terminated after a certain amount, and generates new policies on the basis of the visit frequency of their child nodes in the tree.

NoGoZero+ adds noise to the policy prior at the root to encourage exploration:

$$P(c) = 0.75P\_{\text{rms}}(c) + 0.25\eta \tag{2}$$

where *η* is a draw from Dirichlet distribution on legal moves with parameter *α* = 0.03 ∗ 92/*N*, where *N* is the number of legal moves, and *Praw*(*c*) is the policy prior at the root.

The board of NoGo, unlike the traditional 19 × 19 Go board, has a size of 9 × 9, so the 92 part of parameter *α* corresponds to the NoGo board size. NoGoZero+ also applies a softmax temperature at the root of 1.03 to improve policy convergence stability according to [11].

A convolutional residual network [16] with preactivation is used to guide the search. The residual network has a trunk of *b* residual blocks with *c* channels. However, after various improvements, our network structure was very different compared to that of AlphaZero; see Figure 2. Furthermore, in the primary stage of training, inspired by curriculum learning [17], we designed a method called network curriculum learning to speed up the training progress. NoGoZero+ began with a relatively small residual network and gradually improved the size of the network when it converged under the condition of using the earlier residual network.

#### *3.2. Techniques in NoGoZero+*

#### 3.2.1. Global Attention Residual Block

In general, the AlphaZero-based model has a strong ability to capture the local information of the board. However, due to the limited perceptual radius of the classical convolution layer, global strategy prediction remains challenging. Some findings in previous studies on board games, such as the fatal 'ladder failure' that commonly occurs in Go, support this view [4]. Therefore, we began exploring the attention-based structure. As a popular design of computer vision, the attention mechanism can help models to globally pay more attention to important information.

**Figure 2.** Comparison of network architecture of AlphaZero and NoGoZero+ (5 residual blocks).

In this section, we propose the global attention residual block (GARB), a novel attention-based structure based on the global pooling layer, as shown in Figure 3. The global pooling layer was proposed in previous research on the game of Go [9]. The layer enables the convolutional layers to condition in a global context, which can be hard or impossible for convolutional layers with limited perceptual radius. NoGoZero+ replaces parts of the ordinary convolutional layers with the global pooling structure (GPS) to improve the neural network's ability of synthesizing the global context.

**Figure 3.** Proposed GARB. The structure globally aggregates values of one set of channels to bias another set of channels, potentially providing the final output with information on the global context in NoGoZero+.

Given a set of c channels, the global pooling layer computes the mean of each channel and the maximum of each channel. The whole process outputs a total of 2*c* values. The global pooling layer is a part of the GPS. Given input *X* ∈ *RC*1×*H*×*W* and *G* ∈ *RC*2×*H*×*W* (*C*1 and *C*2 represent the channel, *H* and *W* represent the height and width respectively), the GPS includes the following components:

Previous studies showed that avoiding dimensionality reduction in the FC layer is crucial for learning channel attention [18]. Therefore, we set the value of *C*1 to twice that of *C*2 to avoid performance degradation caused by dimensional changes. GPS was inserted into a standard residual block and eventually constituted GARB. Given input *Xin* ∈ *RC*×*H*×*W* of GARB, two independent convolutional layers decomposed the input into *X* with channel dimension number 2*C* 3 , and *G* with channel dimension number *C* 3 before sending it to GPS.

Moreover, GARB is more focused on extracting long-range dependencies, and it is insufficient for local-information extraction compared to the standard residual block. In order to supplement local information, GARB alternates between two configurations in consecutive residual blocks, as illustrated in Figure 2b.

#### 3.2.2. Multitask Learning

AlphaZero has both a policy head and a value head at the end of its deep neural network. They contribute to the policy network and value network, respectively. The policy head predicts potential good moves, and the value head predicts the final result of the game.The deep neural network outputs policy and value on the basis of the current state of the board. Two parallel networks can be regarded as two training tasks for the neural network.

Such a multitask learning method [19] has had great success in training AlphaZero. NoGoZero+ follows and expands the idea. The paper adds two other output heads, dominion head and score head, which contribute to the two extra tasks (shown in Figure 4a). In NoGo, dominion head predicts the ownership of each location on the board. The ownership indicates the key locations that heavily impact the final result. In both computer games and real world, the final result is suggested both by outputs of a prediction network or noisy 1 and −1 (win and loss), and by more details observed during the game. The neural network can have more insight into the cause of the final result, including the true gap between the winner and loser in a playout round.

Our proposed dominion head architecture is shown in Figure 4b. It contains a convolutional block, a global pooling convolutional block, and an output layer. The convolutional block includes a batch-normalization layer, a ReLU activation layer, and a 3 × 3 convolutional layer. The global pooling convolutional block includes the GPS, a batch-normalization layer, a ReLU activation layer, and a 1 × 1 convolutional layer with 2 filters.

We assumed input feature map *x* ∈ *RC*×*H*×*W*, where C, H, and W are channel, height, and width. First, the convolutional block is applied to *x*, output with the same size as that of *x*. Then, the output of the convolutional block is passed through the global pooling convolutional block, and the number of channel dimensions of *x* is reduced to 2. Lastly, we apply a FC layer with sigmoid activation in the output layer that provides dominion output *yd* ∈ *RH*×*W*.

Our proposed score-head architecture is shown in Figure 4c. The network structure of the score head is similar to that of the dominion head, with two differences. The first is that the 1 × 1 convolutional layer of the global pooling convolutional block has only one filter, so the output feature map is 1 in the channel dimension. The second is that the output layer of the score head contains a FC layer with tanh activation, outputting a scalar in the range of [−1, 1]. Lastly, we multiply it by the score factor to reflect the score gap between the two players. In NoGo, since most score gaps are within 5 points, the score factor was set to 5, and a larger score gap is regarded as 5 points once appearing. Therefore, output ys is a scalar in the range of [−5, 5].

**Figure 4.** (**a**) Multitask learning network structure. (**b**) Illustration of dominion head. (**c**) Illustration of the score head. Except for the policy head and value head, which are also used in AlphaZero, we added score head and dominion head to evaluate the gap of the performance of two players and the key winning position, respectively. The two additional heads can effectively help the NoGoZero+ agent have insight into the game state and make full use of self-play data because of the more detailed information.

#### 3.2.3. Network Curriculum Learning

Curriculum learning is an important method in reinforcement learning [20,21]. To prevent the agent from being stuck at the early stage of training because of the hard initial task or having dissatisfactory performance at the end of training because of oversimplified tasks, we carried out curriculum learning by imitating the learning process of humans and animals and gradually improve the model trough progressive samples and knowledge.

In this paper, the idea of curriculum learning was extended to the neural network and is called 'network curriculum learning'. A ahallower and narrower structure leads to a network with less complexity, but it converges more quickly. While the deeper and wider network with more complexity is usually harder to be trained, network curriculum learning tries to balance complexity and convergence rate. During NoGoZero+ training, the process starts with a simpler network with fewer parameters. Self-play and training data are generated by the shallow network, and used to train a deeper and more complex network (and the simpler network itself), while the deeper network itself does not generate new data. As soon as the simpler network meets its bottleneck, which means that this network has converged, the whole training process is transferred to a larger network. The process is repeated until the size and the performance of the network satisfy our needs.

Figure 5 shows the course of network curriculum learning. The whole process of training the NoGoZero+ agent can be separated into three stages, and the number of residual blocks (b in Figure 5) grew from 5 to 20, while the number of channels (c in Figure 5) of each residual block grew from 32 to 128. With the help of network curriculum learning, the agent avoids the stagnation of training in the early stage and saves much training time.

**Figure 5.** Process of network curriculum learning. Data generated by the shallower network are used to train both the shallow network itself and a deeper network. The whole process contains three stages, and the network with b = 20, c = 128 was extracted as the final network that is used in the agent.

#### 3.2.4. Advanced Specific Features

AlphaZero provides a universal method to train game agents. The method performs perfectly on chess, shogi, and Go. However, a sea of computational resources that make the universal method work. With limited resources, it is important for us to add some advanced specific features to speed up the training process and make full use of the obtained data from every game. Like other machine-learning methods [22], reinforcement learning also attaches much importance to well-designed specific features, which can make a great difference to the speed of training process, as well as the final performance of agents.

Similar to imitation learning, some domain-specific professional knowledge guides the deep model to converge faster during the initial stage. Besides current and historical steps, which are also the features used by AlphaZero, NoGoZero+ adds another kind of advanced feature, liberty, which is also called 'qi'. Four input layers are added on the basis of the input layers of the network to show the features of liberty:

The four above features are (1) 'half-dead', (2) 'near-dead', (3) 'half-kill', and (4) 'near-kill', respectively. One-hot encoding is used to represent the four advanced features. The corresponding location in the input layer is set to 1 once a specific feature appears. The corresponding features are schematically shown in Figure 6.

**Figure 6.** Demonstration of advanced features. Location of existing features is marked with one-hot encoding in the corresponding input layer. Four additional layers are added to the input layer set of the original AlphaZero and provide the NoGoZero+ agent with additional advanced information about the board state. Results in the next section show that such modification can obviously improve training speed and final performance.

#### **4. Experiments**

In the two following sections, we use some control tests to verify the improvements our techniques bring to the basic AlphaZero method, and use ablation experiments to examine the contributions of each approach used in NoGoZero+. In this section, we introduce the experimental settings, evaluation criteria, and training parameters to detail the background and process of our experiments.

#### *4.1. Experimental Settings*

To look into the behavior of NoGoZero+ and demonstrate the effects of the unique techniques introduced in the paper, two experiments were conducted.

First, we designed an experiment to show the significant speedups attributing to adding extra training methods to the original AlphaZero theory by comparing three models' training processes on the NoGo:

Second, to explore the effects of every single technique, an ablation experiment was conducted. To more clearly demonstrate the contributions of each technique to the final result, all agents in the ablation experiment did not use the network-curriculum-learning method.

#### *4.2. Evaluation Criteria*

The Elo model [23] is an effective way to measure the level of game agents, widely used to evaluate the performance of agents in, for example, chess, Go, basketball, and football. Assuming that the Elo ratings of Agents *A* and *B* are *RA* and *RB*, respectively, the expected winning probability of *A* versus *B* is:

$$E\_A = \frac{1}{1 + 10^{\frac{R\_R - R\_A}{400}}} \tag{3}$$

When Player *A*'s true score *SA* is different from its expected winning probability (1 for win, 0 for loss, 0.5 for tie. There is no tie in NoGo.), its Elo rating should be adjusted as

$$R\_A^{'} = R\_A + a(S\_A - E\_A) \tag{4}$$

where *R A* is the Elo rating of Player *A* after adjustment, and *α* is a weight that is often set to 16 in master tournaments (*α* = 16 is also used in our experiment).

#### *4.3. Training Parameters and Details*

We set stochastic gradient descent (SGD) as the optimizer with a learning rate equal to 0.001 and momentum equal to 0.8. Our experiments were performed on an NVIDIA Tesla V100 GPU (with 32 GB GPU memory). The deep-learning models were implemented using Pytorch. In training, the data-augmentation technique that we used includes all eight reflections and rotations for each position. The entire training process took about 100 h, and most of the resources were consumed in self-play. Multithreading self-play is recommended to make full use of GPU and CPU resources.

Considering the four output heads of the improved network structure, the loss function is the sum of the five following kinds of losses.

• Policy Loss

$$-\sum\_{m} \pi(m) \log\left(\hbar \mathfrak{r}(m)\right) \tag{5}$$

where *m* is the set of the rest legal moves, *π* is the target policy derived from the playouts of the MCTS search, and *π*ˆ is the neural network's prediction of policy *π*.

• Value Loss

$$-c\_{\mathcal{S}} \sum\_{r} z(r) \log \left( \mathcal{E}(r) \right) \tag{6}$$

where *r* is the final result for the current player ((*r* ∈ {*win*, *lost*}), *z* is a one-hot encoding function of *r*, *z*ˆ is the neural network's prediction of the final result. *cg* = 1.5 is a scaling constant.

• Dominion Loss

$$-c\_d \sum\_m \gamma(m) \log\left(\hat{\gamma}(m)\right) + (1 - \gamma(m)) \log\left(1 - \hat{\gamma}(m)\right) \tag{7}$$

where *m* is the set of the moves in the board, *γ* is the final result for the current player (*r* ∈ {*dominion*, *not dominion*}), *z*ˆ is the neural network's prediction of *γ*. *cd* = 0.5 is a scaling constant.

• Score Loss

$$c\_s(\vartheta - v)^2\tag{8}$$

where *v* is the scalar of score difference, *v*ˆ is the prediction of the final result. *cs* = 0.25 is a scaling constant.

• L2 Penalty

$$c\_{L2} \|\theta\|^2 \tag{9}$$

where *cL*2 = 10−5, so as to prevent the network from overfitting due to the relatively deep neural network structure.

#### **5. Results**

In this section, the experiment results are shown and discussed. Then, some interesting playouts are displayed and explained to demonstrate the intelligence of NoGoZero+. Lastly, we introduce the performance of NoGoZero+ in the competition and summarize the possible reasons for not winning first place.

#### *5.1. Experiment Results*

The results of the first experiment are shown in Table 1. AlphaZero's Elo rating reached 2500 after 120,000 self-play games, while it only took NoGoZero+ 20,000 self-play games to reach an Elo rating of 2750, and NoGoZero+ defeated original AlphaZero with 81 wins and 19 losses in 100 playouts. The AlphaZero agent, with the help of the network-curriculumlearning method, had a similar learning curve to that of NoGoZero+. However, an obvious gap of Elo rating between the AlphaZero agent with network curriculum learning and complete NoGoZero+ indicated that other techniques that we use make a great difference in the behavior of the agent.

**Table 1.** Comparison of Elo ratings among original AlphaZero, AlphaZero with only network curriculum learning, and complete NoGoZero+. NCL, network curriculum learning. NoGoZero+ had advantages over the original AlphaZero in final performance (Elo rating) and training speed. Moreover, the obvious gap between complete NoGoZero+ and AlphaZero with only network curriculum learning indicated the supplementary power of other techniques. Blocks represent the number of residual blocks in the network.

The results of the second experiment (ablation experiments) are shown in Table 2. Comparing the final behavior of the agents with different parts weeded out, GARB had a relatively larger influence on the final performance (the Elo rating of which dropped from 2300 to 2045 without it), while multitask learning and advanced features closely impacted the performance of the agent. Adding advanced features was less influential because the main aim of adding advanced features is to help the agent make full use of detailed information obtained from one single playout to speed up the training process with relatively limited resources. Overall, every technique used in the training process of NoGoZero+ had a positive effect, which can effectively help to improve the Elo rating of the agent.

#### *5.2. Strategies Learnt by the Agent*

Unlike Go, which originated thousands of years ago and has been researched for quite a long time, NoGo is a new board game with few mature strategies. However, with the help of techniques shown in the previous sections, the NoGoZero+ agent developed a series of inspirational strategies. The paper demonstrates and discusses some distinct, impressive strategies shown by the NoGoZero+ agent in self-play games.

**Table 2.** Comparison of Elo ratings among original AlphaZero, AlphaZero with only network curriculum learning, AlphaZero using some of the techniques in this paper, and complete NoGoZero+. ASF, advanced specific features; MTL, multitask learning network structure; 5b, number of residual blocks in the network.

#### 5.2.1. Triangle Strategy

A triangle layout and a prismatic layout are two kinds of favorable layouts to players because the central location of such layouts cannot be occupied by a counterpart's stones. As a result, such layouts can greatly help at the end of the game when there are few locations to place a stone. At the beginning of a game, the agent tries to build as many triangle and prismatic layouts as possible, if not disturbed by the counterpart player (see Figure 7). Moreover, the agent sometimes manages to prevent its counterpart from successfully building a triangle or a prismatic layout after weighing advantages and disadvantages. Such behavior is called the 'triangle strategy'.

**Figure 7.** Demonstration of 'triangle strategy'. Black successfully built a triangle structure in the location marked by , and white successfully prevented black from building triangle structure in Step 4 because neither black nor white could occupy the location marked by ×. Moreover, the white stone placed in Step 8 also tried to destroy a potential triangle structure that may have contained the black stones placed in Steps 5 and 7.

#### 5.2.2. Predictive Strategy

With the progress of the NoGoZero+ agent, it becomes harder for white to win (the average win rate is 4.5) because black learns to make full use of the sente advantage.

A typical situation is: white tries hard to prevent black from using the triangle strategy and fails because black takes the lead in the order (Figure 8).

**Figure 8.** A typical situation where the white falls behind. Starting from Step 4, white tried hard to prevent black from using the triangle strategy, but black always took the lead. Although white made a wise choice in Step 14 and terminated the 'chasing game', black successfully built a triangle structure in Step 7. The whole process indicates that black has an obvious advantage over white when the NoGoZero+ agent is well-trained.

Since NoGo is considered to be a game where the black player is more likely to win, the white player needs to adopt a more aggressive strategy to win the game. We looked into games in which white won, and found that white seemed to gain foresight under certain conditions.

Figure 9 shows a part of a game where the white player won. Instead of chasing black's steps and trying hard to stop black in a relatively small area, the white player controlled the overall situation and successfully disturbed black's deployment on a larger scale.

Such a decision shows a sense of prediction, so the behavior is called 'predictive strategy'. The sense of the overall situation and the prediction should be attributed to the GARB, which gives the agent a high level of understanding of the whole situation, and multitask learning makes the agent capable of predicting the behavior of its counterpart.

Although the white player did not fully understand the predictive strategy in the experiment, it is impressive that the agent developed such a high-level tactic. Hopefully, after more self-play games, the agent can master predictive strategy or even other high-level strategies.

**Figure 9.** Demonstration of predictive strategy. When black tried to build a triangle structure in Step 3, instead of chasing black's step like the behavior shown in Figure 8, white took Step 4 and built a wider encirclement, thus rendering the triangle structure built by black (marked by ×) useless.

#### *5.3. NoGoZero+ in CCGC*

The China Computer Game Championship (CCGC) is the largest computer-game competition in China, held annually since 2006. In 2020, a total of 18 teams entered the national finals. Our team competed for the first time and unfortunately lost in the finals to the KnighTeam-NG (KNG) program developed by the Chongqing University of Technology. KNG has won consecutive championships in NoGo since 2014. As the code is not open-source, we cannot know what kind of technology KNG uses.

One lesson is that we unreasonably allocated the game time. NoGoZero+ spent much time in the first half of the game, and when the second half of the time was exhausted, some strange and unreasonable moves appeared before it lastly lost the game. In addition, our equipment for participating in the competition was not good enough. For each move, NoGoZero+ could only perform 1400–1800 MCTS on average, which led to worse model performance.

#### **6. Discussion**

Although the main battlefield of NoGoZero+ is still the game of NoGo, the techniques used by NoGoZero+ can be extended to other nonspecific areas to reduce resource consumption caused by large search spaces. First, GARB is an efficient attention mechanism that helps improve the network's ability to perceive global information. Second, the success of the multitask learning network structure proved that adding more output heads to neural networks can efficiently help the training process. Third, the network-curriculumlearning technique can help to balance the complexity and the rate of convergence of a deep neural network. Fourth, adding moderate domain-specific knowledge to the training process of neural networks can make full use of training data and reduce the training period, especially when computational resources are limited.

The game of NoGo is becoming a useful tool for many researchers who are interested in game AI but do not want to spend too much time learning the complex game rules to carry out studies. On the basis of the NoGoZero+ result, they can quickly develop a basic structure of NoGo game AI and save time for further study. As a result, with the help of techniques used by NoGoZero+, the game AI community benefits from the lower research thresholds.

Moreover, AlphaZero-based methods mainly use convolutional neural networks (CNN) to capture board information, just like CNN captures information from pictures in computer-vision tasks. This means that many tricks used in computer vision can be transformed into AI training processes. The attention model, which is commonly used to select the most crucial information from pictures during computer-vision tasks, can be combined with the original CNN structure to improve efficiency in the usage of computational resources, and further enhance the final performance of the AI. This method is reasonable and needs further research because some positions are also more important than others are in board games.

#### **7. Conclusions and Future Work**

This paper presented NoGoZero+, which efficiently masters the game of NoGo on the basis of the improved AlphaZero algorithm. By using several techniques, the improvement is nearly cost-free. The methods enable a unified reinforcement-learning-based system to be trained from scratch to being a powerful agent in a few days. The techniques used in NoGoZero+ can easily be extended to other areas to reduce computational-resource consumption and improve training efficiency. Experiments showed that NoGoZero+ had six times better training speed and better performance than those of the original AlphaZero. This study highlights the tremendous potential of reducing computing resources in board-game AI.

Although the experiment results of this study are encouraging, there are still some limitations. For example, we were only awarded second place in the CCGC. The future of this study includes exploring more powerful neural-network architectures and more efficient sample-utilization methods to further improve performance. Our ultimate goal is to provide insights and additional tools for the community to explore large-scale deeplearning methods of computer games. Our code and data are public to help researchers in the game AI community.

**Author Contributions:** Conceptualization, Y.G. and L.W.; methodology, L.W. and Y.G.; validation, L.W. and Y.G.; resources, Y.G. and L.W.; data curation, Y.G. and L.W.; writing—original-draft preparation, Y.G. and L.W.; writing—review and editing, L.W. and Y.G.; supervision—L.W. and Y.G. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Acknowledgments:** The authors thank Jiao Wang from the College of Information Science and Engineering, Northeastern University, China for their support in preparing this study.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel. +41 61 683 77 34 Fax +41 61 302 89 18 www.mdpi.com

*Electronics* Editorial Office E-mail<EMAIL_ADDRESS>www.mdpi.com/journal/electronics

MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel: +41 61 683 77 34

www.mdpi.com ISBN 978-3-0365-7039-6