**Optimizing Manufacturing and Osseointegration of Ti6Al4V Implants through Precision Casting and Calcium and Phosphorus Ion Implantation? In Vivo Results of a Large-Scale Animal Trial**

**Wölfle-Roos JV 1,\*, Katmer Amet B 2, Fiedler J 2 , Michels H 3, Kappelt G 4, Ignatius A 5 , Dürselen L 5 , Reichel H 1 and Brenner RE 2**

Received: 25 February 2020; Accepted: 26 March 2020; Published: 3 April 2020

**Abstract:** Background: Uncemented implants are still associated with several major challenges, especially with regard to their manufacturing and their osseointegration. In this study, a novel manufacturing technique—an optimized form of precision casting—and a novel surface modification to promote osseointegration—calcium and phosphorus ion implantation into the implant surface—were tested in vivo. Methods: Cylindrical Ti6Al4V implants were inserted bilaterally into the tibia of 110 rats. We compared two generations of cast Ti6Al4V implants (CAST 1st GEN, n = 22, and CAST 2nd GEN, n = 22) as well as cast 2nd GEN Ti6Al4V implants with calcium (CAST + CA, n = 22) and phosphorus (CAST + P, n = 22) ion implantation to standard machined Ti6Al4V implants (control, n = 22). After 4 and 12 weeks, maximal pull-out force and bone-to-implant contact rate (BIC) were measured and compared between all five groups. Results: There was no significant difference between all five groups after 4 weeks or 12 weeks with regard to pull-out force (*p* > 0.05, Kruskal Wallis test). Histomorphometric analysis showed no significant difference of BIC after 4 weeks (*p* > 0.05, Kruskal–Wallis test), whereas there was a trend towards a higher BIC in the CAST + P group (54.8% ± 15.2%), especially compared to the control group (38.6% ± 12.8%) after 12 weeks (*p* = 0.053, Kruskal–Wallis test). Conclusion: In this study, we found no indication of inferiority of Ti6Al4V implants cast with the optimized centrifugal precision casting technique of the second generation compared to standard Ti6Al4V implants. As the employed manufacturing process holds considerable economic potential, mainly due to a significantly decreased material demand per implant by casting near net-shape instead of milling away most of the starting ingot, its application in manufacturing uncemented implants seems promising. However, no significant advantages of calcium or phosphorus ion implantation could be observed in this study. Due to the promising results of ion implantation in previous in vitro and in vivo studies, further in vivo studies with different ion implantation conditions should be considered.

**Keywords:** ion implantation; precision casting; Ti6Al4V; calcium; phosphorus; centrifugal casting

#### **1. Introduction**

Uncemented implants are widely used in arthroplasty, in primary total hip arthroplasty (THA), up to 80% of implants are uncemented [1]. As the long-term stability of uncemented implants can only be achieved by osseointegration—i.e., a stable connection between the implant surface and the adjacent bone—titanium-based implants are preferred for uncemented implantation due to their high biocompatibility [2,3].

However, the manufacturing process of titanium and its alloys is much more complex than that of cobalt–chromium-based implants [4]. Due to the high melting point of titanium of >1700 ◦C, its high reactivity to oxygen [5] and its unfavorable fluidity properties [6], precision casting of titanium implants is challenging and currently not routinely used [7]. At present, standard manufacturing of titanium implants involves machining blanks on a milling machine, which results in a high waste of Ti6Al4V material and fabrication costs, as well as a limitation of titanium implants to comparatively simple implant designs [8]. Recent innovations have led to the optimization of centrifugal casting units, see Figure 1. By introducing a cold wall induction crucible and by improving casting conditions, mould material and subsequent heat treatment, centrifugal precision casting has become feasible for the manufacturing of titanium-based implants [9].

**Figure 1.** Schematic illustration of a centrifugal precision casting unit: The ingot-casting material is melted (**A**) and the liquid melt is then pressed into the casting mould by rotating the casting arm around a vertical shaft (**B**).

The technically demanding manufacturing process of titanium-based implants is not the only challenge of uncemented implantation. Osseointegration of the uncemented implant still remains a crucial issue, as the stability of the implantation depends on a strong connection between implant surface and adjacent bone [10]. To enhance osseointegration, countless different methods of surface modification have been developed. One of the most prominent surface modifications is hydroxyapatite coating (HA-coating) [11], which has been widely used in uncemented THA. As, however, delamination of the HA-coating and three-body-wear due to HA-molecules have been increasingly reported in the literature [11–13], attempts have been made to implant calcium and phosphorus ions—the chemical components of HA—directly into the implant surface [14], encouraging the formation of calcium phosphate precipitates [15,16]. For complex implant shapes, the so-called plasma immersion ion implantation is currently used [17], see Figure 2.

**Figure 2.** Schematic illustration of plasma immersion ion implantation. In a vacuum chamber, calcium ions are extracted from the plasma—i.e., ionized gas—and accelerated towards the negatively charged implant surface by applying a high voltage direct current.

To improve both the manufacturing and osseointegration of uncemented titanium implants, a novel manufacturing technique—an optimized form of centrifugal precision casting—and a novel surface modification to promote osseointegration—calcium and phosphorus ion implantation into the implant surface—were tested in vivo in this study.

#### **2. Methods**

#### *2.1. Animals*

110 adult female WISTAR rats were supplied by Charles River Laboratories (Kißlegg, Germany). They were kept under climate-controlled conditions (21 ± 1.5 ◦C, 47.5% ± 7.5% humidity, light–dark cycle 12/12 h). Access to food and tap water was ad libitum. Age at the time of surgery was ten weeks; mean body weight was 277 ± 15 g. The animal trial was conducted according to relevant national and international guidelines, such as the ARRIVE guidelines; the study was approved by the Regional Administrative Council (registration number 1246).

#### *2.2. Implants*

The standard implants consisted of machined, aluminium-oxide-blasted Ti6AlV4 cylindrical rods of 5.2 mm length with a diameter of 1.6 mm (provided by Peter Brehm GmbH, Weisendorf, Germany). One end of the rod was threaded (length 1.2 mm) to enable biomechanical pull-out testing.

All other implants had identical dimensions compared to the standard implants and were manufactured with the optimized centrifugal precision-casting technique, which had been developed in cooperation between the Technical University Aachen (RWTH, Germany), the associated research center Access technology e. V. (Aachen, Germany), and the implant manufacturer Peter Brehm GmbH (Weisendorf, Germany). Implants were cast in a combined vacuum-induction centrifugal casting device Leicomelt 5 TP (ALD Vacuum Technologie GmbH, Hanau, Germany). As the optimized centrifugal precision casting technique is currently undergoing a patent grant procedure, only a general overview of the manufacturing process is given here.

Casting moulds were made of multilayer ceramics with the lost wax technique, with the help of 3D-simulation programs for the casting and solidification process. The ingot-casting material Ti6Al4V was melted in a cold wall induction crucible in a vacuum system with inert gas flushing. The liquid melt was then pressed into the casting mould by rotating the casting arm around a vertical shaft, see Figure 1.

In contrast to other metals' higher centrifugal forces and higher speed are required for titanium implants to completely fill out the mould due to its low specific weight, high melting point and high solidification speed. After solidification of the melt, the casting was revealed under the destruction of the ceramic form, see Figure 3. The cast implants were separated from the feed channel with a cutting disc. The implants were then submitted to hot isostatic pressing with 920 ± 10 ◦C at a pressure of 1000 ± 50 bar for 120 ± 30 min to reduce porosity, and subsequent heat treatment.

**Figure 3.** Cast cylindrical implants (→) before separation from the feed channel.

In cast Ti6Al4V implants of the first generation (CAST 1st GEN), the so-called alpha case layer—a hardened brittle layer at the implant surface due to the reaction between the oxides of the mould and the molten titanium, see Figure 4—had to be removed by acid etching at room temperature with Ceramex® (Renfert GmbH, Hilzingen, Germany), i.e., a mixture of 3% hydrofluoric acid and10% sulphuric acid (MediMet Precision Casting and Implants Technology GmbH, Stade, Germany).

**Figure 4.** Alpha case layer on the implant surface of cast Ti6Al4V implants of the first generation.

By further optimizing the conditions of the casting process, the alpha case layer of cast Ti6Al4V implants of the second generation (CAST 2nd GEN) was negligible, so acid etching could be dispensed with. All cast implants were then submitted to identical aluminium-oxide-blasting to the standard implants. Material analysis showed that the cast material met with the standard grade for Ti6Al4V alloys [4].

Cast Ti6Al4V implants of the second generation were then submitted to plasma immersion ion implantation (PIII) with calcium (CAST + CA group) or phosphorus (CAST + P group) in cooperation with Helmholtz Center Dresden-Rossendorf in a 5-MV-Tandem EGP-10-1 (Efremov-Institut NIIEFA Leningrad, St. Petersburg, Russia). As an ion source, H3P, and a calcium arc source, respectively, were used. PIII was conducted at room temperature, ion energy was 30 keV, and ion current 2.4 μA (calcium) and 3–500 μA (phosphorus), respectively, at a dose of 1 × 1016/cm2.

#### *2.3. Experimental Design*

The animals were randomly assigned 22 animals each to one of the following groups: (1) control group, (2) CAST 1st GEN, (3) CAST 2nd GEN, (4) CAST + CA, or (5) CAST + P group. According to which group they belonged to, one of the following implants was inserted:

Eleven animals of each group were sacrificed after 4 weeks, and the remaining animals 12 weeks after implant insertion. The right tibia was prepared for biomechanical testing and the left tibia for histomorphometric analysis.

#### *2.4. Surgical Procedure*

The above-mentioned implants were inserted into the proximal tibia on both sides. Anaesthesia was administered by means of an inhalation device (isoflurane 2%) and the subcutaneous injection of analgetics (tramadolor 20 mg/kg) and buprenorphin (0.03 mg/kg). The surgical technique was identical to previous studies conducted in our study group [18]: a 10 mm incision was made at the medial aspect of the proximal tibia, and the periosteum was incised ventrally to the medial collateral ligament. A 1.7 mm drill hole was made level with the insertion of the patella tendon ventrally to the medial collateral ligament using hand-held drills held strictly perpendicular to the longitudinal axis of the tibia. The implant was then inserted into the bone. The threaded part remained outside and was covered by a 2 mm tube cut off from a venous catheter (fluorinated ethylene propylene, Vasofix® Braunüle® 18 G, Braun B., Melsungen, Germany) in order to prevent osseous overgrowth. Postoperative analgesia was ensured by adding tramadolor to the drinking water (25 mg per litre). Antibiotics (clindamycine 45 mg/kg) were administered subcutaneously daily on the first three postoperative days.

#### *2.5. Biomechanical Testing*

For biomechanical testing, a specifically designed cylindrical device with a matching internal thread was screwed onto the threaded part of the implant of the right tibia. This device was passed through a perforated plate, which served as a mobile fixation of the implant, and then attached to a 200 N load cell (HBM, Darmstadt) of a standard testing machine (Z010, Zwick, Ulm, Germany; see Figure 1A). The surrounding soft tissue around the implant had been purposely left in place to ensure an even contact between the specimen and the perforated plate. The implant was then aligned straight—i.e., longitudinally to the tensile axis—by a low preload of 0.5 N. A force-displacement diagram (test speed 10 mm/s, preload 0.5 N) was recorded by a testing software (testXpert II, Zwick, Ulm, Germany) and the load occurring before the first sudden drop in the tensile force was defined as

maximum pull-out force. To calculate the pull-out force per area, a Micro-CT scanning of the former implant bed of the first 44 animals was conducted with a μCT system (Skyscan 1172, Kontich, Belgium). As the correlation between pull-out force and pull-out force per area was found to be highly significant and very strong (Pearson's correlation coefficient R = 0.970, *p* < 0.001), pull-out force was used as the only biomechanical parameter for the remaining animals.

#### *2.6. Histomorphometric Analysis*

The left tibia including the inserted implant was embedded in Technovit VLC7200 (Kulzer, Germany) and ground down to sections of 100 μm along the longitudinal axis of the tibia. Masson–Goldner staining of the sections was used to visualize the connective tissue surrounding the implant. The sections were inspected and scanned with a fully automated inverted light microscope (Leica DMI6000B, Wetzlar, Germany). To quantify the amount of bone surrounding the implant, the following parameters were determined semi-automatically with the aid of an imaging analysis software (MetaMorph®, Leica, Wetzlar, Germany). The bone-to-implant contact rate (BIC) was calculated by dividing the total length of bone-to-implant contact by the total length around the implant within the tibia. Histomorphometric analysis was conducted by two independent observers blinded to the implant material used.

#### *2.7. Statistical Analysis*

Statistical advice, including an estimation of required sample size, was gained before the planning of the study based on the only comparable study with a similar animal model and an identical location of the implant insertion [19,20]. For statistical analysis, the Statistical Package for Social Sciences (SPSS® Inc., IBM, version 24) was used. Continuous variables were summarized as mean ± standard deviation. To compare the results of the five groups after 4 and 12 weeks, Kruskal–Wallis test was used. Pearson's correlation coefficient was used to calculate interobserver reliability and correlation between pull-out force and pull-out force per area. A probability value of less than 0.05 was considered to indicate statistical significance.

#### **3. Results**

With regard to maximum pull-out force, there was no significant difference between all five groups after 4 weeks and after 12 weeks (*p* = 0.596 and *p* = 0.127 respectively, Kruskal–Wallis test). After 12 weeks, cast Ti6Al4V implants of the first generation (CAST 1st GEN) showed a slightly lower maximum pull-out force (78.4 ± 16.9 N) compared to the control group (99.8 ± 25.0 N), though this difference did not reach statistical significance. However, no disadvantage of the cast Ti6Al4V implants of the second generation could be observed (95.4 ± 23.0 N) after 12 weeks, see Figure 5.

Analysis of the histological sections showed a thin layer of osseous tissue covering a large part of the implant surface on the section of all five groups. Examples of histologic sections of both cast and machined Ti6Al4V implants are depicted in Figure 6.

Histomorphometric measurement revealed no significant difference in BIC after 4 weeks between all five groups (*p* > 0.05, Kruskal–Wallis test), whereas there was a trend towards higher BIC in the CAST + P group (54.8% ± 15.2%), especially when compared to the control group (38.6% ± 12.8%) after 12 weeks (*p* = 0.053, Kruskal–Wallis test), see Figure 7.

**Figure 5.** Boxplot of maximum pull-out force after 4 and 12 weeks. The 25th and the 75th percentile as well as the median are indicated by the box, while the whiskers depict maximum and minimum values. No significant difference between all 5 groups was observed (*p* > 0.05, Kruskal–Wallis test).

**Figure 6.** Exemplary Masson–Goldner stains of cast (**A**) and standard (**B**) Ti6l4V implants after 12 weeks. Mature osseous tissue is stained blue, while newly formed bone as well as connective tissue appears red. In both sections, a thin layer of mature bone covers the implant surface.

**Figure 7.** Boxplot of bone-to-implant contact rate (BIC) after 4 and 12 weeks. The 25th and the 75th percentile as well as the median are indicated by the box while the whiskers depict maximum and minimum values. Outlies are plotted as individual dots. There was no significant difference between all five groups after 4 weeks (*p* > 0.05), whereas there was a trend towards higher BIC in the CAST + P group after 12 weeks (*p* = 0.053, Kruskal–Wallis test).

#### **4. Discussion**

The most important findings of this study were that no indication of the inferiority of the cast Ti6Al4V implants of the second generation could be observed with regard to osseointegration when compared to standard machined implants, and that no significant advantages after calcium and phosphorus ion implantation could be seen.

#### *4.1. Centrifugal Precision Casting*

Precision casting of titanium-based implants is technically challenging due to the high melting point of titanium of >1700 ◦C, its high reactivity to oxygen and its unfavorable fluidity properties. Only in highly specialized centrifugal precision casting units are sufficient centrifugal forces and speed reached to completely fill out the mould, due to the low specific weight and high solidification speed. However, precision-casting of titanium implants holds vast potential economic advantages over the standard milling process [21]. On the one hand, the process offers a significantly decreased material demand per implant by directly casting near net-shape instead of milling away most of the starting ingot, thus greatly lowering the base material demand and cost. Adding to this lower base demand is the inherent improvement in sustainability. Milling waste must be recycled by complex reclaiming

processes before it can be reused, while the direct reclamation of casting scrap is regular practice in casting processes, given that constant cast part quality is proven. On the other hand, the centrifugal casting process offers a great freedom of design, in particular by using custom-printed wax patterns in the future, which will enable us to produce implants of complex geometrical shapes (especially those involving undercuts) in large numbers.

Mechanical properties of cast and wrought titanium—the latter being the raw material from which current standard titanium implants are machined—need not necessarily be identical, as the solidification process and the exposure of molten titanium to oxygen varies between the two manufacturing techniques. However, Nastac et al. conducted a review on investment casting of Ti6Al4V alloys and found that almost all the static and dynamic mechanical properties of cast Ti6Al4V alloys are similar to wrought Ti6Al4V. Only the fatigue strength—especially high-cycle fatigue—of cast Ti6Al4V alloys was found to be inferior, which, however, can be improved by subsequent heat treatment [6]. To achieve optimal mechanical properties of cast Ti6Al4V alloys, countless improvements in centrifugal precision casting have been introduced in recent years, e.g., a vacuum system and inert gas flushing due to the high reactivity of titanium with oxygen, high centrifugal forces to ensure complete filling of the mould, or hot isostatic pressing to reduce porosity [6,21]. The alpha case layer—a hardened brittle layer at the implant surface due to the reaction between the oxides of the mould and the molten titanium—still remains a challenge for precision casting of Ti6Al4V [22]. This layer is most commonly removed by acid etching, a technique which in itself bears the risk of hydrogen embrittlement of the implant surface through hydrogen absorption and hydride formation [23]. In our study, we found slightly inferior maximum pull-out force after 12 weeks in cast Ti6Al4V of the first generation, in which the alpha case layer had been removed by acid etching (CAST 1st GEN). By further optimizing the conditions of the casting process, the thickness of the alpha case layer was further reduced (CAST 2nd GEN), so that acid etching could be dispensed with. In these cast Ti6Al4V implants of the second generation, we did not observe any indication of inferiority in comparison to standard Ti6Al4V either on biomechanical testing or on histomorphometric analysis. Standard material testing carried out by our project partners before delivering the implants showed no deviation from standard implant requirements, but further biomechanical studies will have to follow to prove the equivalence of the main mechanical properties such as modulus of elasticity, hardness, toughness, fatigue strength, or tensile strength.

There are only a few in vivo studies comparing cast and machined implants with regard to biocompatibility and osseointegration. Mohammadi et al. implanted cast and machined titanium implants into the abdominal wall of rats so that part of the implant was located in the abdominal wall and part in the peritoneal cavity. On light and electron microscopy they found no difference in peri-implant tissue in the abdominal wall between the two groups. For intraperitoneal implants, significantly more fibroblasts and macrophages were observed on the implant surface of cast implants (in 6/8 implants) when compared to machined implants (in 1/9 implants) [24]. However, as there was a significant difference between tissue response in muscular and peritoneal tissue, it seems doubtful that these findings can predict tissue response in osseous tissue. Moreover, cast titanium implants in their study—as opposed to implants cast with modern centrifugal precision casting units—received hardly any post-processing treatment and showed a comparatively rough implant surface [24]. As this rough implant surface is probably the main reason for the wash-out of titanium particles and the subsequent heightened response of macrophages, their results cannot be compared with modern cast titanium implants. In a more recent study, Mohammadi et al. implanted machined cast titanium implants and standard machined titanium implants into the tibia of rabbits and found no significant difference in bone–implant contact after 3 months (19% BIC in cast implants vs. 25% in machined implants) and 6 months (45% in cast implants vs. 37% in machined implants) between both groups [7]. As in their study cast titanium implants were machined after casting and 0.25 mm of the implant surface were removed, again, their results cannot be compared to ours. There is only one study reporting the application of cast titanium implants in humans: 15 patients received individualized cast titanium implants for the reconstruction of bony skull defects using data from 3D computer tomography scans. There was no control group and mean follow-up was 16.6 months. With the exception of one case of early infection which resulted in the removal of the implant, no complications were noted and osseointegration was successful on clinical and radiological examination [25]. These findings agree with our study, which showed no indication of the inferiority of cast Ti6Al4V implants of the second generation compared to standard Ti6Al4V implants. However, given the small number of patients, the short follow-up, the lack of a control group, and the location of the implants in a non-weight-bearing part of the skeleton [23], evidence regarding the long-term equivalence of cast orthopedic titanium implants when compared to today's standard implants has yet to be provided. Thus, further in vivo trials involving a large animal model in a biomechanically loaded position might be considered before establishing centrifugal precision casting as a manufacturing technique of titanium orthopedic implants.

#### *4.2. Calcium and Phosphorus Ion Implantation*

The second part of this study was dedicated to calcium and phosphorus ion implantation into the surface of cast Ti6Al4V implants of the second generation and its effects on osseointegration. Ion implantation represents an ultra-clean process in which the concentration and depth distribution of ions can be controlled with high accuracy [17]. Several in vitro studies have shown the advantageous effects of calcium and phosphorus ion implantation on the mechanical properties of titanium. Corrosion resistance, as measured by electrochemical methods in a simulated body fluid, was significantly increased after calcium and phosphorus ion implantation (ion dose 1 × 1017/cm2) which is a very important aspect of biocompatibility [26,27]. X-ray photoelectron spectroscopy of calcium and phosphorus implanted titanium (ion dose 1.8 × 1017/cm2 and 9 × 1016/cm2 respectively) was conducted to analyze the resulting chemical composition of the implant surface. It was shown that under hydrothermal oxidation a three-step chemical reaction takes place with (1) oxidation of phosphorus ions to P2O5, (2) hydrolysis resulting in Ca2+, PO4 3− und H+, followed by (3) the appearance of needle-like crystallites of calcium phosphates such as hydroxyapatite on the implant surface [16].

The influence of calcium and phosphorus ion implantation on osteogenesis has been thoroughly investigated in vitro by Nayab et al. [28–31] and Krupa et al. [14,26,27]. Nayab et al. seeded radioisotopically tagged alveolar bone cells on calcium-implanted titanium and found that, although cell adhesion was reduced on the calcium ion-implanted surface, cell spreading and subsequent cell growth of the alveolar bone cells was significantly enhanced [28]. This effect seemed to be dependent on the ion dose: in implants with a high calcium ion dose (1 × 1017/cm2), the adhesion of MG-63-cells, though initially reduced, was substantially increased in time, and cell spreading was significantly enhanced. In contrast, no marked differences were observed with regard to the adhesion and spreading of MG-63-cells on titanium implanted with low (1 × 1015/cm2) or medium (1 × 1016/cm2) ion doses [29]. Gene expression analysis showed the up-regulation of key proteins of osteogenesis such as bone sialoprotein, bone morphogenetic protein receptor-1B and osteopontin in MG-63 cells [30] as well as increased markers of proliferation [31] on calcium-implanted titanium (ion dose 1 × 1017/cm2). Krupa et al. seeded human mesenchymal stem cells on titanium implanted with calcium ions (ion dose 1 × 1017/cm2) [26], phosphorus ions (ion dose 1 × 1017/cm2) [27] and calcium/phosphorus ions combined (ion dose 1 × 1017/cm2) [14]. In contrast to Nayab et al., they found no positive effects of calcium and/or phosphorus ion implantation on osteogenesis; there was no difference with regard to the vitality of mesenchymal stem cells and the expression of alkaline phosphatase when compared to the control group [14,26,27]. These findings agree with our study, which did not show significant positive effects of either calcium or phosphorus ion implantation on osseointegration.

The effects of ion implantation on osseointegration has only been studied in vivo with calcium-implanted titanium to date. Hanawa et al. investigated bone formation in rats around a titanium implant which had been implanted with calcium ions (ion dose 1 × 1017/cm2) on one side only; tetracycline and calcein were used as hard-tissue labels. They found more bone formation on the calcium-implanted side of the implant compared to the other side after 2 and 8 days; 18 weeks after surgery, no difference between both sides could be seen [32]. Jinno et al. performed THA in dogs with

Ti6Al4V implants with and without calcium ion implantation; doxycycline and fluorescein were used as labels to investigate new bone formation. Histomorphometric analysis showed significantly greater new bone apposition in calcium-implanted stems compared to the control group; 6 and 12 months after implantation no significant difference could be seen between both groups [33]. Cheng et al. inserted cylindrical titanium implants with and without calcium ion implantation into the femur of rats. After 4, 8, and 12 weeks they found significant advantages of calcium-implanted titanium implants on histomorphometric analysis, computer tomographic assessment of the implant bed, and analysis of new bone formation, as well as the biomechanical testing of the push-force when compared to the control group [34]. The findings of these three in vivo studies did not concur with our study, in which no positive effects of calcium ion implantation on osseointegration could be observed. Possibly, the ion density used in our study (1 × 1016/cm2) was too low, or the ion implantation energy too high (30 keV) to reach enough ion deposition close to the implant surface to allow for any noticeable effects on osseointegration. Furthermore, the properties of the implant surface of the studies mentioned above were not identical when compared to our study, as in some cases polished implants [32] and/or pure titanium implants [32,34] were used. Additionally, the casting process itself may have influenced the metallurgic properties of the implant surface, and thus the depth of ion implantation. As we did not expect a relevant release of implanted ions, serum calcium or phosphorus levels were not determined. This might be included in future studies on the effects of different dosages of ion implantation.

#### **5. Conclusions**

In this study, we found no indication of inferiority of Ti6Al4V implants cast with the optimized centrifugal precision-casting technique of the second generation compared to standard Ti6Al4V implants. As the employed manufacturing process holds considerable economic potential, mainly due to a significantly decreased material demand per implant by casting near net-shape instead of milling away most of the starting ingot, its application in manufacturing uncemented implants seems promising.

However, no significant advantages of calcium or phosphorus ion implantation could be observed in this study with the doses applied. Due to the promising results of ion implantation in previous in vivo studies and its positive effect on the mechanical properties of titanium implants, further in vivo studies with different ion implantation conditions (regarding ion density and implantation energy) should be considered.

**Author Contributions:** Conceptualization, W.-R.J.V., F.J., M.H., K.G., I.A., D.L., R.H. and B.R.E.; Data curation, W.-R.J.V.; Funding acquisition, B.R.E.; Investigation, W.-R.J.V. and K.A.B.; Methodology, W.-R.J.V.; Project administration, B.R.E.; Supervision, B.R.E.; Writing—original draft, W.-R.J.V.; Writing—review and editing, K.A.B., F.J., M.H., K.G., I.A., D.L., R.H. and B.R.E. All authors have read and agreed to the published version of the manuscript.

**Funding:** This study has been funded by a grant from the German Federal Ministry of Education and Research (BMBF, No. 13 GW0020E).

**Acknowledgments:** We wish to acknowledge the support of Andreas Seitz, Annette Palmer, Patrizia Horny, Giovanni Ravalli, and Christiane Schulz in conducting the experimental work.

**Conflicts of Interest:** The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results. The authors declare no conflict of interest.

#### **References**

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

*Article*


**2. Integration of STEM education in a school: A phased approach**

Introduction of STEM education requires a phased approach to give time to the school administration and faculty to fully understand and prepare for a full integration of STEM methodology in their regular classroom sessions. Starting with a STEM club, then introduction in a couple of classes and finally expanding gradually to cover more classes allows the system to ease in into the integration of STEM throughout the school.

#### **2.1 STEM lab design and setup**

STEM lab provides an environment for students to foster learning in a fun and exciting way. It provides space, tools and means for students to practice their theoretical knowledge by creating solutions for real-world problems that are both authentic and relevant to them. In a STEM lab, students not only engage in learning of corecurriculum concepts and nurture core competencies but also acquire new psychomotor skills.

A STEM Lab must be designed to facilitate hands-on experiential learning in an effective manner utilizing the space optimally and efficiently within the confines of specified constraints and limitations. This is a balancing act. The lab design process begins by understanding the purpose and basic requirements of the lab. These requirements emanate from the planned activities and the number of students that will be accommodated for these activities at any given time. Unlike a typical classroom where the setup is static, a STEM lab is designed on the principles of freedom of movement and freedom of choice. Dynamic spaces are created through modular

and mobile furniture, which offers the flexibility for the students to move around and work in groups. The furniture is designed to promote active lean-forward learning instead of passive lean-back information transfer.

Modular and mobile furniture provides opportunities to create open spaces when needed so that the students can work on the floor or gather around for some activities. Sufficient storage spaces are provided in the form of shelves and cabinets to stow away tools, equipment and materials used in STEM activities and also showcase student projects. The material used in furniture needs to be lightweight, durable and water proof.

Continuous electricity supply, especially in the rural areas is one of the many challenges in developing countries. Backup power becomes a necessity. Similarly, Internet connection in remote areas is mostly available through wireless carriers and as such needs to be arranged for the STEM lab. A view of such a model STEM lab is shown in **Figure 1**.


*1.2.3. Air drier*

The air drier (Fig. 4) is a proprietary designed manufactured in Australia by B.C. Johnson Ltd. It is designed for inlet air conditions of 1.5 ib/sec at a pressure of 115 psig, a maximum temperature of 105 0 F and an ambient relative humidity at compressor inlet of 100%. The dryer is required to provide an outlet humidity equivalent to -50 0 F at atmospheric pressure after a two hour drying cycle.

**Figure 4.** Air Drier System

The air dryer is of the single tower at a larger stage. Regeneration is accomplished by air pre-heated in an 80 kW electrical heater and forced through the dryer stack in a counterflow direction by a centrifugal blower. The desiccant employed is 800 lb of activated alumi‐ na in a 2 ft diameter by 4.5 ft high bed, preceded by 80lb of buffer desiccant whose purpose is to prevent damage to the main desiccant by liquid carry over from filtration equipment. The dryer is fitted with a water coil for cooling the desiccant bed after regeneration and a felt pad and fibreglass after-filter is installed to prevent any carryover of desiccant dust into the storage vessels.

In operation, dew points of as low as -80 0 F have been obtained after regeneration. With one compressor in operation, drying times of up to six hours have been achieved, although a higher final dew points than -50 0 F regeneration time is about four hours for heating and four hours for cooling.

#### *1.2.4. Air storage vessels*

*1.2.2. Filtration system*

78 Wind Tunnel Designs and Their Diverse Engineering Applications

automatic water drains.

temperature of 105 0

**Figure 4.** Air Drier System

after a two hour drying cycle.

*1.2.3. Air drier*

After compression, air from each compressor is passed through individual shell and tube af‐ ter coolers, centrifugal action water separators, pre-filters and oil-mist filters. The pre-filters consist of a centrifugal action separator combined with a 70 micron porosity, sintered bronze filter element. The oil mist filters are a proprietary design of Daltech Engineering In‐ corporated, USA, and consist of a stainless wire wool pre-filter and a chemical absorption type main element. These filter units are claimed to have a 99.4% filtration efficiency for all oil particulates down to 0.5 microns. Both pre-filters and oil-mist filters operate with a 'col‐ our change' principle in that, as the element becomes saturated with oil, its colour changes from light pink to bright red. All filters, after coolers and water separators are fitted with

The air drier (Fig. 4) is a proprietary designed manufactured in Australia by B.C. Johnson Ltd. It is designed for inlet air conditions of 1.5 ib/sec at a pressure of 115 psig, a maximum

dryer is required to provide an outlet humidity equivalent to -50 0 F at atmospheric pressure

F and an ambient relative humidity at compressor inlet of 100%. The

Four storage vessels (Fig.5 ), each 5 ft diameter by 11 ft 6 ins overall length and designed for a working pressure of 130 psig have been installed with a total storage capacity o f800 cubic feet as mentioned earlier were placed in an overhead structural steel support in the Aerody‐ namics laboratory near which a supersonic wind tunnel was built.

### *1.2.5. Air conditions 105 0 F at atmospheric pressure*

Air conditions in the storage vessels are a volume of 800 cubic feet at ambient temperature and dried to a dew point the equivalent of at least -50 0 F at atmospheric pressure. For inter‐ mittent operation, maximum pressure is 125 psia. Maximum intermittent mass flow rates are in the region of 10 to 20 lb/sec to limit the system pressure loss to approximately 5 psi. Maximum continuous mass flow is 1.5 lb/sec with a pressure not exceeding 110 psia.

Because the pressure available for supersonic tunnel injection is comparatively low, care was taken in the design and piping and filtration equipment to reduce pressure losses. The overall pressure drop between compressor outlet and storage vessels has been kept between 3 and 5 psi depending upon filter condition.


**Learning Attention Representation with a Multi-Scale CNN for Gear Fault Diagnosis under Di**ff**erent Working Conditions**

#### **Yong Yao 1,\*, Sen Zhang 2,3, Suixian Yang 1,\* and Gui Gui 4**

Received: 30 January 2020; Accepted: 21 February 2020; Published: 24 February 2020

**Abstract:** The gear fault signal under different working conditions is non-linear and non-stationary, which makes it difficult to distinguish faulty signals from normal signals. Currently, gear fault diagnosis under different working conditions is mainly based on vibration signals. However, vibration signal acquisition is limited by its requirement for contact measurement, while vibration signal analysis methods relies heavily on diagnostic expertise and prior knowledge of signal processing technology. To solve this problem, a novel acoustic-based diagnosis (ABD) method for gear fault diagnosis under different working conditions based on a multi-scale convolutional learning structure and attention mechanism is proposed in this paper. The multi-scale convolutional learning structure was designed to automatically mine multiple scale features using different filter banks from raw acoustic signals. Subsequently, the novel attention mechanism, which was based on a multi-scale convolutional learning structure, was established to adaptively allow the multi-scale network to focus on relevant fault pattern information under different working conditions. Finally, a stacked convolutional neural network (CNN) model was proposed to detect the fault mode of gears. The experimental results show that our method achieved much better performance in acoustic based gear fault diagnosis under different working conditions compared with a standard CNN model (without an attention mechanism), an end-to-end CNN model based on time and frequency domain signals, and other traditional fault diagnosis methods involving feature engineering.

**Keywords:** acoustic-based diagnosis; gear fault diagnosis; attention mechanism; convolutional neural network

#### **1. Introduction**

As a one of the most important components in transmission systems, gears are widely used in many types of machinery, such as wind turbines, construction machinery, automobiles, and other fields [1], thanks to their unique merits, such as large transmission ratio, high efficiency, and heavy load capacity [2,3]. The working performance of gears directly influences the operational reliability of the whole machinery [4]. However, due to poor environmental conditions and the intensive impact load operational condition of transmission systems, gears are vulnerable to display some faults and cause the machine to break down. This may lead to significant economic losses [5]. Therefore, research on fault diagnosis for gears can effectively avoid catastrophic failure and reduce economic loss.

Recently, the fault diagnosis of gears has been extensively studied by researchers. However, most current studies focus on mainly stable working conditions. In the real world, gears usually work under variable and fluctuant operation conditions [6]. As such, the nonlinear and non-stationary characteristics of signals under variable conditions exhibit many unique characteristics, such as strong nonstationary, frequency mixing, and modulated phenomena [7]. Traditional fault diagnosis methods and technologies, which are only applicable to gears under stationary conditions [1], are incapable of detecting and identifying gear fault patterns under variable conditions.

To solve this issue, gear fault diagnosis under variable conditions has become the subject of extensive research and has aroused researchers' great concern in the past few years. Liu et al. [8] proposed a method for gear fault diagnosis under slight variations in working conditions via empirical mode decomposition (EMD) and multi-fractal detrended cross-correlation analysis (MFDCCA). By using EMD and MFDCCA methods, the multi-fractal fault features can effectively extract and distinguish fault modes. In order to avoid mode mixing, Chen et al. [9] proposed to use complementary ensemble empirical mode decomposition (CEEMD) technology to decompose the raw vibration signals and select the intrinsic mode functions (IMFs) using a correlation analysis algorithm (CorAA) for a probabilistic neural network to classify the gear fault patterns under different working conditions. Xing et al. [10] adopted the intrinsic time-scale decomposition (ITD) and singular value decomposition methods to improve the robustness of gear fault feature extraction under variable conditions. Zhang et al. [11] proposed a method for gear fault diagnosis under different working conditions based on local characteristic-scale decomposition (LCD) denoising and the vector mutual information method. Chen et al. [12] performed gearbox fault diagnosis under variable speed conditions via analysis of the torsional vibration signals in the time-frequency domain. Though these studies, rich methods, and technologies for gear fault diagnosis have been accumulated and provide a pivotal function under variable conditions, most of the methods typically use vibration signals as the main measurement values to diagnose gear faults in variable working conditions for use in vibration analysis [13]. In many practical conditions, the installation of vibration sensors is constrained by some working conditions and the complex structure of the equipment themselves, which makes the signal acquisition inconvenient. Moreover, vibration signals are easily masked in some special environments, such as high humidity, high temperature, and high corrosion; therefore, the application of vibration signal analysis methods for gear fault diagnosis under variable conditions is limited due to the requirement of contacted measuring. Meanwhile, those studies that adopt vibration analysis methods, usually rely on signal processing technology to decompose raw vibration signals into several proper signal components to extract valuable features for distinguishing gear fault patterns under different working conditions. Although all these vibration signal analysis methods can work well in fault mode detection tasks, they rely heavily on diagnostic expertise and prior knowledge of signal processing technology [14], which may lead to tedious and inefficient procedures in practical diagnosis tasks. Considering the existing issues, the effective methods and technologies of gear fault diagnosis under variable conditions still needs to be further developed.

As a typical non-contact measurement, acoustic-based diagnosis (ABD) methods, which have the capability to overcome the limitation of vibration measurement, are widely used in the fault diagnosis field. Lu et al. [15–17] proposed an acoustic-based fault diagnosis method based on near-field acoustic holography for detecting gear fault patterns under stationary working conditions. Glowacz [18,19] design several acoustic-based diagnosis methods with novelty acoustic features to detect the fault of commutator motors, electric impact drills, and coffee grinders. By combing time-frequency data fusion technology and the Doppler feature matching search (DFMS) algorithm, Zhang et al. [20] proposed a train bearings fault diagnosis method, which is based on wayside acoustic signals. Inspired by their study, Zhang et al. [21] designed an improved singular value decomposition with a resonance-based wayside acoustic signal sparse decomposition technique as an adaptive form of train bearings fault feature extraction. However, like the vibration-based diagnosis method, all the existing acoustic-based methods are also heavily rely on prior knowledge of signal processing technology rather than utilizing intelligent fault diagnosis techniques. This is because the fault data distribution that we obtain in one working condition are not consistent in another different working condition in real applications [22], which means the distribution difference between training data and test data changes as the working condition varies, which can lead to a dramatic drop in performance.

To manage the obstacles, we considered the role of the attention mechanism. As a novel intelligent method, the attention mechanism, which has the capability to adaptively capture temporal correlations between different sequences [23] and allows for feature extraction networks to focus on the relevant characteristics without signal processing technology and feature engineering, are commonly explored in various structural prediction tasks, such as document classification [24], speech recognition [25–27], and environmental classification [28,29]. Therefore, in this paper, we propose a novel ABD method for gear fault diagnosis under different working conditions based on a multi-scale convolutional learning structure and attention mechanism. In our methods, a multi-scale convolutional learning structure was designed to automatically mine multi-scale features using different filter banks from raw acoustic signals. Then, a novel attention mechanism, which was based on a multi-scale convolutional learning structure, was established to adaptively allow the multi-scale network to focus on relevant fault pattern information under different working conditions. Finally, a stacked convolutional neural network (CNN) model was proposed to detect the fault mode of the gears.

The main contributions of this paper are as follows:

#### **2. Model Building**

In this section, we briefly introduce the mathematical model of our acoustic-based gear fault diagnosis method, which can be roughly divided into three parts. In the first part, a multi-scale convolutional layer operates directly on raw acoustic signals and automatically mines fault features using different filter sizes and strides to construct feature vectors. In the second part, an attention mechanism is adopted to obtain reasonable attention weight vectors from the convolutional layer, which are multiplied with each feature vector of the pooling layer. In the last part, the multi-dimensional attention output matrix, which is concatenated with the multi-scale attention structure, is constructed as a stacked CNN input to train the network. The block diagram of the proposed method is shown in Figure 1.

**Figure 1.** Block diagram of the proposed method.

#### *2.1. Multi-Scale Convolution Operation*

A commonly used approach for an end-to-end neural network is to pass the raw acoustic signal through a 1D convolutional layer, which has a fixed filter size and stride length to create invariance to phase shifts and further down-sample the signals. However, those methods are still constrained in various prediction tasks for two reasons: (1) There is always a trade-off when choosing the filter size. A high-scale filter size may have a good frequency resolution but does not have a sufficient filter for location in the high frequency area. A low-scale filter size, on the contrary, focuses on more frequency bands but has a low resolution [30,31]. (2) Features extracted using a fixed filter size cannot make full use of the raw signal information to build a discriminative representation for different patterns. Considering this, a multi-scale convolutional function, which has the capability to learn discrepant features, has been applied to address the obstacles. By extracting features with multiple different scale filter banks and splitting responsibilities based on what filter banks can efficiently represent, multi-scale convolutions have already been successfully used in various recognition fields, such as image classification [32], environmental sound classification [30], and speech recognition [33].

Inspired by their work, we designed a multi-scale convolutional learning structure to extract multi-scale fault features from raw acoustic signals. The structure is shown in Figure 2.

**Figure 2.**

Three scale convolution kernels were used to operate the input signal vector to extract different features and increase the bias to achieve a result. The function is defined as followed:

$$\mathbf{x}\_{j}^{l} = f(\mathbf{x}\_{k} \cdot \boldsymbol{\omega}\_{j}^{l} + \boldsymbol{b}\_{j}^{l}),\tag{1}$$

where *xk* represents the input raw signal vector; *j* represents the three different convolutional scales (*j* = 1, 2, 3), corresponding to low, mid, and high filter sizes; ω*l j* represents the convolutional operation between the input vector and output feature map *l* at different scales; *bl j* is a bias that corresponds to the output vector *xl j* , which represent the convolution operation result of the feature map *l* in scale *j*; and *f* is an activation function.

Then, each of the three different output vectors were subsampled by the max pooling layer in turn such that vectors of different sizes were rescaled to the same size.

#### *2.2. Temporal Attention Mechanism*

The acoustic signals obtained in one working condition may not follow the same temporal structure in another working condition and those signals are often masked by noises that are generated from the gearbox parts and transmitted via an elastic medium, i.e., through the air. We designed a novel temporal attention mechanism that puts more attention on the relevant information frames and suppresses noise ones to provide acoustic-based fault diagnosis under different working conditions to overcome those limitations.

In order to reduce the impact of channel information, we first used a 1 × 1 kernel size with one channel to aggregate the feature maps along the channel dimension to produce a multi-scale convolutional learning structure. Then, we adopted different convolutional operations for multi-scale vectors to transform the features map into the same scale and generate a temporal attention map through the softmax activation function. Finally, we multiplied the attention map with the feature vector of the pooling layer to obtain the attention output matrix. The detailed structure of the temporal attention mechanism is shown in Figure 3 and detailed information about the operation process are given below.

**Figure 3.** Temporal attention mechanism.

Let *Convi*\_1(*i* = 1, 2, 3) denote the feature vector of the multi-scale convolution learning structure. We first operate a 1 × 1 kernel size over *Convi*\_1 ∈ *RT*×1×*C* to generate one channel feature map *Att*\_*Convi*\_1 ∈ *RT*×1×1. Then, multi-scale 2-D convolution with a different kernel size was adopted to learn the hidden representation and compress the features map into the same scale using different strides. The softmax activation function was applied to normalize the attention weight of *Convi*\_1 and produce the temporal attention map. The mathematical equations are expressed as:

$$Att\\_Convi\\_1 = Conv^{1 \times 1} (Convi\\_1) \; i = 1, 2, 3,\tag{2}$$

$$Att\\_Convi\\_2 = Att\\_Convi\\_1 \cdot a\iota\_i + b\_i \text{ } i = 1, 2, 3\,, \tag{3}$$

$$A\_{\rm tm} = \text{softmax}(\text{Att\\_Convi\\_2}) \text{ i} = 1,2,3. \tag{4}$$

Finally, by multiplying the attention map *Atm* with the feature vector of the pooling layer, we obtained the attention output matrix *Ato*. The equation is defined as followed:

$$A^i\_{\rm at} = \left(\mathbf{x}^l\_j\right)\_{p\rm pol} \times A^i\_{tm'} \tag{5}$$

where *xl j pool* represents the *l*th feature map of the *j*th pooling layer (*j* = 1, 2, 3). The *j* value refers to what is described in Section 2.1.

#### *2.3. Fault Pattern Recognition Based on a CNN*

We adopted a stacked convolutional neural network as a base structure for the recognition of gear fault patterns under different working conditions. The network consisted of four functional layers: the convolutional layer, the batch normalization layer (BN), the pooling layer, and the fully connected layer.

The attention output matrix from three different scales were concatenated along the channel dimension as a multi-dimensional matrix input into the convolutional layer. The stacked convolutional layer can be viewed as a fault pattern recognition module in an attention-based multi-scale CNN model. Through repetitive convolution operations, the network has the ability to learn high-level representation from the inputted multi-dimensional matrix. The process can be expressed as follows:

$$y\_j^l = \sum\_{i=t}^T y\_i^{l-1} \cdot a\_{ij}^l + b\_{j'}^l \tag{6}$$

where *yl*−1 *i* represents the *i*th output feature map of the former convolutional layer, and ω*l ij* represents the convolutional kernel, which is used to operate between the *i*th feature map of the former layer and the *j*th feature map of layer *l*. *T* represents the feature atlas of the former layer and *bl j* represents the bias of layer *l* corresponding to the output matrix *yl i* , which represents the convolution operation result of the *j*th feature map in layer *l*.

The output matrix of the convolutional layer is normalized by the BN layer such that the mean and variance of the feature become 0 and 1, respectively. Then, we used functions to transform and reconstruct a certain level of features to maintain the data distribution. Those equations can be expressed as:

$$y\_2 = \frac{y\_1 - \mu}{\sqrt{\sigma^2 + \varepsilon}},\tag{7}$$

$$
\Delta y\_3 = f(\gamma y\_2 + \beta),
\tag{8}
$$

where μ and σ2 represent the mean and variance of the mini-batch in Equation (7), and γ and β are the two parameters in Equation (8), which can be learned by the training network. *f* represents the activation function, which is used to analyze nonlinear information for the output features of the BN layer.

Then, the max pooling layer was applied to the subsample feature information of the BN layer to prevent overfitting.

Finally, through a combination of these high-level representations in a nonlinear way, a fully connected layer that recognizes gear fault patterns under different working condition was produced. The mathematical equations of the fully connected layer can be expressed as:

$$h\left(y^{l}\right) = f\left(wy^{l-1} + b\right),\tag{9}$$

where *yl*−1 represents high-level information of the former layer, *h yl* represents the output nonlinear information from the fully connected layer *l*, and ω and *b* represent the weight and bias, respectively.

#### *2.4. Architecture and Parameters of the Attention-Based Multi-Scale CNN Model*

The detailed information of the architecture and parameters of the attention-based multi-scale CNN model are shown in Figure 4 and Table 1, respectively. The architecture of the attention-based multi-scale CNN model consists of two parts: (a) the attention-based multi-scale feature extraction module and (b) the fault pattern recognition module.

**Figure 4.** Architecture of the attention-based multi-scale convolutional neural network (CNN) model. It contains (**a**) an attention-based multi-scale feature extraction module and (**b**) a fault pattern recognition module.

These two modules can be divided from the concatenate layer. The attention-based multi-scale feature extraction module contains a multi-scale convolutional learning structure with an attention mechanism. The network operates on an input acoustic signal, which consists of 16,000 sampling points using three different scale convolutional neural networks of different scales. The three chosen scales are low-scale (size 11, stride1), mid-scale (size 51, stride 4), and high-scale (size 101, stride 8). Each scale has 32 filters in their convolutional layer. As an independent part in the network, the

attention mechanism contains two convolutional layers to adaptively pay more attention to the relevant information frames from the output of the three convolutional layers. The kernel size of the two convolutional layers were set as (1, 1) and (64, 1) in the low scale, (1, 1) and (32, 1) in the middle scale, and (1, 1) and (16, 1) in the high scale. Then, the attention output matrix is concatenated along the channel dimension as a multi-dimensional matrix 500 × 1 × 96 (two-dimensional feature matrix × channels). Finally, in order to convolve the feature matrix from time and frequency in module (b), the attention output matrix is reshaped from 500 × 1 × 96 to 500 × 96 × 1 before being used as an input to "conv2" for further processing.

The fault pattern recognition module contains three convolutional layers, two pooling layers, and one fully connected layer. The first convolutional layer uses 32 filters with a (5,5) kernel size and stride of (1,1). The second and third convolutional layers repeatedly use 64 filters with a (3,3) kernel size and the strides were set to (1,1) and (2,2), respectively. The pooling layer is applied to subsample feature information after the first and second convolutional layers. All the kernel sizes and strides for the pooling layer were set to (2,2). In addition, the BN layer is used as a function layer, which can alleviate the problem of exploding and vanishing gradients following each convolutional layer (before the pooling layer). Finally, the output feature of the BN layer is flattened and passed to the fully connected layer with 256 nodes and the softmax activated output layer for final recognition.

**Table 1.** Parameters of the attention-based multi-scale CNN model.

#### **3. Experimental Setup and Datasets**

#### *3.1. Experimental System*

Our study was a preliminary attempt at going from the theory and simulation experiment to the practical engineering application. In order to obtain pure acoustic signals that were not disturbed by environmental noise, the gear fault diagnosis experiments were conducted in a semi-anechoic chamber. The experimental system that we designed can be divided into three parts: the experiment table, the measuring system, and the data recoding software. The experiment table, as shown in the top-left corner of Figure 5 [34], consisted of the following equipment: a variable frequency motor, a two-stage gearbox, a tension controller, a frequency converter, and a magnetic brake. By adjusting the frequency converter and the tension controller, we could control the speed of the motor and simulate the load condition of the two-stage gearbox. The measuring system, as shown in the right picture of Figure 5, consisted of four free-field 4189-A-021 model microphones from Brüel & Kjær (Copenhagen, Denmark) and a data acquisition instrument from HEAD Acoustics (Herzogenrath, Germany). In our study, the four free-field microphones were arranged to provide a four-channel microphone array, where they were arranged symmetrically with a hemispherical enveloping surface and the coordinates were set according to the ISO 3745:2003 standard to collect all the gears' acoustic signals. Then, the microphone array and data acquisition instrument were connected using a Bayonet Nut Connector (BNC) interface for data transmission. In addition, the data was recoded using Artemis 6.0 software, which is shown in bottom-left of Figure 5.

In our experiments, we chose the low-speed shaft of the two-stage gearbox as the object for detecting the gears' fault patterns under different working conditions. The gears' fault patterns, as shown in Figure 6 [34], consisted of a tooth fracture, pitting, and wear. We set the motor at three speeds—900 rev/min, 1800 rev/min, and 2700 rev/min—by controlling the frequency and adjusting the magnetic brake using two load conditions—0 Nm and 13.5 Nm—via tension control to simulate different working conditions. Regarding those conditions, we believe that the acoustic signal that we obtained can be viewed as only containing the gears due to the general assumption that the interference of other parts of the gearbox, such as the bearing and shaft via vibration, was minor. All the acoustic signals of the gears were recorded as an audio file that was 60 s long for further analysis.

**Figure 5.** Experimental system in a semi-anechoic chamber.

**Figure 6.** Fault pattern of the gears.

#### *3.2. Dataset*

In order to verify that the method we proposed is effective and feasible under different working conditions, we built two different datasets, A and B, that represented the two load conditions of 0 Nm and 13.5 Nm, respectively. Each dataset contained an audio file of four types of gears (one normal type and three fault types) at three different speeds, and each type was recorded to produce a four-channel audio file that was 60 s long. Each file was divided into 1-s samples with no overlap because 1-s samples are an optimal size for analysis based on empirical experiments in audio processing tasks. Then, each dataset contained 21,600 samples with 18,000 used as training samples and 3600 used as testing samples.

#### *3.3. Implementation Detail*

We used the cross-entropy as a loss function to train the attention-based multi-scale CNN model for multi-fault type classification under different working conditions. We applied the Adam algorithm in the training step to optimize the model, where learning rate was set to 0.003. In addition, we used rectified linear units as activation functions for each layer. When training, the dropout layer, which was followed by the fully connected layer, was employed to prevent overfitting with a 0.5 dropout rate. Finally, the early stopping approach and the no-improvement-in-10-epochs strategy was adopted to identify the number of epochs via the testing set.

#### **4. Experimental Result and Analysis**

#### *4.1. Time and Frequency Analysis in Di*ff*erent Working Conditions*

The time and frequency domain information of the acoustic signals that we obtained from four types of gears at three different speeds under two load conditions are shown in Figure 7. The subplots (a), (b), and (c) represent the time and frequency domain signal of gears under two load conditions at 900 rev/min, 1800 rev/min, and 2700 rev/min, respectively. The left panel of each subplot shows the four types of gears in a no-load condition and the right panel shows the same type in a 13.5-Nm-load condition.

Comparing the time and frequency domain signals from the subplots, we can see that the signal amplitudes of the gears were different in the time and frequency domains under variable speed conditions. To be specific, with the increasing of the operation speed, the maximum amplitude of the majority types of gears in the time domain also increased under the same load condition. Meanwhile, from the frequency domain signal, we found that the magnitude of the frequency amplitude for the same types of fault signals under variable speed conditions was different, but the distribution of the amplitude was not affected by the varying speed. For example, in the frequency domain, the normal gears had a higher amplitude at 2700 rev/min speed condition than the same type of gears at 900 rev/min and 1800 rev/min under the no-load condition, but the distribution of the amplitude, which was concentrated in the range of 0–500 Hz and 1000–1200 Hz, was consistent in the three different speed conditions. Based on the description above, we could infer that the variable speed caused the amplitude modulation phenomenon of the acoustic signal, but did not influence the frequency modulation.

Furthermore, comparing the time and frequency domain signals under the no-load condition and load condition at 900 rev/min, 1800 rev/min, and 2700 rev/min, we observed that the acoustic signals for four types of gears seemed to be obviously different in the two load conditions. From the time-domain signal, we saw that the gears with a wear type and the gears with pitting under the no-load conditions had a higher amplitude range than the same fault type under the 13.5-Nm-load condition at 900 rev/min, but it was the opposite at 1800 rev/min. Moreover, the waveform of each type of gear in the time domain were different, which means the temporal structure and energy modulation patterns of the signal under the two load conditions were diverse. Meanwhile, according to the magnitude and distribution of the frequency amplitude in the ranges of 0–500 Hz and 1000–1200 Hz, we found that the frequency signal of the four types of gears under the two loads were also different, especially for the normal and pitting types of gears. As for the normal gears at 900 rev/min, the frequency component under the no-load condition was around 0–500 Hz and 1000–1200 Hz, while the frequency component under the load condition was around 1000–1200 Hz. Equally, the gears with the pitting type had the same phenomenon at 1800 rev/min. This indicates that the acoustic signals of gears under different load conditions were not only affected by the amplitude modulation, but also by the frequency modulation. In addition, another interesting phenomenon we observed was that the type of gear signal that we obtained under the no-load condition at one speed may follow the same magnitude and distribution of frequency amplitude under the load condition at another speed. For example, the normal gears under the no-load condition at 900 rev/min had a similar amplitude and distribution of frequency to the same gear type under the 13.5-Nm-load condition at 1800 rev/min. This means that the gear fault diagnosis under variable load conditions is more complex and difficult than that of a variable speed condition. Therefore, we proposed an attention-based multi-scale CNN model for gear fault diagnosis under variable load conditions.

**Figure 7.** *Cont.*

**Figure 7.** Four different types of gear signals in the time and frequency domains under two load conditions: (**a**) signal at 900 rev/min, (**b**) signal at 1800 rev/min, and (**c**) signal at 2700 rev/min.

#### *4.2. E*ff*ectiveness of the Multi-Scale Convolution Operation*

In order to verify the hypothesis that the multi-scale convolutional learning structure is superior to the single-scale convolutional structure in for gear fault diagnosis tasks, we first compared the performance of our multi-scale convolutional neural network with that of a low-scale convolutional neural network, mid-scale convolutional neural network, and high-scale convolutional neural network with no attention mechanism. These three models remained at only one scale each, which is shown in Figure 2. The input and the rest of the network structure is the same as in Figure 4 for fair comparison. The two evaluation methods were designed to evaluate the performance of models under different load conditions:

In evaluation A, we used the training samples of dataset A to train these CNN models and test it on the test samples of dataset B.

In evaluation B, we used the training samples of dataset B to train these CNN models and test it on the test samples of dataset A.

The classification accuracy was used as the evaluation criterion. The equation is defined as:

$$\text{ACC} = \frac{N\_1}{N\_2} \,\text{s} \tag{10}$$

where *N*1 represents the number of test samples that were predicted properly and *N*2 represents the total number of test samples.

The test results of the multi-scale network and single-scale network in the two evaluation methods are shown in Table 2. From the result, we saw that the recognition accuracy of the multi-convolutional neural network reached 81.1% and 71.0% for the two evaluation methods, respectively. This was an improvement of 2.5%, 2.3%, and 1.6% compared with the low-scale, mid-scale, and high-scale networks for evaluation A, respectively. Also, the multi-scale convolutional neural network achieved the best accuracy for evaluation B. The improvement from the single-scale model to the multi-scale model proved that the model, which contained different scales, was capable of learning more discriminative features from the waveforms.

**Table 2.** Prediction accuracy of multi-scale CNN and single CNN model.

To further understand how the multi-scale convolution operations help to improve the performance of fault pattern recognition, we visualized the frequency magnitude of the response of the multi-scale feature maps Conv1 of the model in Figure 8. As indicated in this figure, the 32 filters were viewed as band-pass filters to learn a particular frequency area, and each filter was sorted based on their center frequencies. From the left picture of Figure 8, we observed that the curve of the center frequency almost matched the sound feature of the human auditory system. This means that the low-scale structure was able to extract features from all frequency areas. Conversely, the high-scale structure, which is shown in the right of Figure 8, was located in the low-frequency area with fine-grained filters. It shows that the high-scale structure tended to concentrate on low-frequency components and ignore high-frequency information. Moreover, the mid-scale performed between the low-scale and high-scale networks. In general, a model with a narrow kernel size can cover all frequency areas but obtains a low-frequency resolution, and a model with a wide kernel size does not have sufficient filters in the high-frequency range but gives good frequency resolution. It indicates that learning structures of different scales can extract discrepant features based on what they can efficiently represent. This may explain the result that we present in Table 2 where the multi-scale models obtain a better performance than the single-scale models.

**Figure 8.** Frequency magnitude response of the multi-scale convolutional filters of the first layer. The filters are sorted by their center frequency. Left shows the frequency response of the low-scale network, middle shows the frequency response of the mid-scale network, and right shows the frequency response of the high-scale network.

#### *4.3. Comparison of a Standard CNN Model and Attention Models*

To verify the effectiveness of the attention mechanism, we applied the attention mechanism to a multi-scale convolutional neural network and each single-scale network for a fair comparison. The performances of the models are summarized in Table 3.

**Table 3.** Prediction accuracy when applying an attention mechanism.

In Table 3, it is shown that the performances of each model presented a significant improvement when using an attention mechanism compared with the standard model, which did not use an attention mechanism, especially in the case of the multi-scale convolutional model. The attention-based multi-scale CNN model achieved a 93.3% accuracy in evaluation A and 82.8% in evaluation B. This was 12.2% and 11.8% higher than the accuracy of the standard multi-scale CNN model. Meanwhile, by using an attention mechanism, the improvement range of recognition performance of the single-scale model was from 6.8% to 8.8% for evaluation A and from 5.4% to 14.1% for evaluation B. The test results indicate that the attention mechanism was effective at gear fault diagnosis under different load conditions.

To provide a better understanding how a temporal attention mechanism helped to improve the performance of the multi-scale CNN model in the gear fault diagnosis task, we visualized the results of randomly selected filters from the multi-scale pool layer and temporal attention output for four different types of gear input signals under the two load conditions. Figure 9 shows the visualization result of the attention-based multi-scale CNN model for evaluation A. The first and third rows represent the waveform under the no-load condition and 13.5-Nm-load conditions, respectively. The second and fourth rows represent the attention output corresponding to the waveform of the two load conditions, respectively. From this figure, we found that the temporal attention mechanism, which was based on the multi-scale CNN model, was able to adaptively focus on the relevant temporal information from the different waveforms of the two load conditions while reducing the impact of the data distribution variation. Furthermore, from the attention output, we found that the attention weights of the four gear types were different for different time stamps. For example, the tooth fracture condition in gears had three high-weighted areas at time stamps in the ranges of #30–#50, #100–#200, and #430–#500 frames, while the high-weighted areas of the gears with pitting were in the ranges of #180–#230, #300–#320, and #450–#480. This may indicate that the learned temporal attention was able to detect the essential feature information required to distinguish these four types of gears under the two load conditions.

Figure 10 shows the visualization results of the attention based multi-scale CNN model for evaluation B. From this figure, we found two phenomena. First, the learned temporal attention was also able to locate the meaningful temporal parts from the different waveforms of the two load conditions for evaluation B. Meanwhile, the four types of gears could be easily distinguished according to the distribution of the time stamps. This visualization result was similar to that obtained for evaluation A. Second, the high-attention-weights area of the four gear types for evaluation B were not consistent with the high-attention-weights area for evaluation A. This may be due to the fact that the temporal attention mechanism, which was trained on different datasets, could generate different attention weights to focus on different temporal parts.

The above visualization results may explain why the temporal attention mechanism was effective at gear fault diagnosis under different load conditions.

**Figure 9.** Visualization results of randomly selected filters from the multi-scale pool layer and temporal attention output for four different types of gear input signals under the two load conditions corresponding to the attention based multi-scale CNN model for evaluation A.

**Figure 10.** Visualization results of randomly selected filters from the multi-scale pool layer and temporal attention output for four different types of raw gear signals under the two load conditions corresponding to the attention based multi-scale CNN model for evaluation B.

#### *4.4. Comparison between the Attention-Based Multi-Scale CNN Model and Other Methods*

In this section, we compare our model with traditional methods that combine one of the most powerful acoustic features, namely MFCC (Mel-frequency cepstral coefficients), with a convolutional neural network and several classic fault diagnosis methods, which have been successfully used in fault pattern recognition tasks based on time, frequency, manual features, and machine learning algorithms, to analyze the performance of our model. Meanwhile, the end-to-end model, which is based on time and frequency domain signals that we proposed in a previous work, was also used for further comparison.

We adopted the commonly used parameter [35,36] to construct the MFCC feature, the first derivative of the MFCC feature (MFCC-delta) and the second derivative of the MFCC feature (MFCC-delta-delta) as matrix features. Then, we fed it into the convolutional neural network, whose structure is the same as module (b) of the multi-scale CNN model in Figure 3 to provide a fair comparison. The procedure of the classic fault diagnosis methods can be divided into two parts: manual feature extraction and fault identification. The manual features that we extracted include the root-mean-square error (RMS) [37], spectral centroid [38], and Mel spectrogram in the log domain [39], which represent the popularity acoustic features in the time, frequency, and time–frequency domains, respectively. We fused these manual features at the feature level for improving the representation of the fault information. Then, we fed it into several classic machine learning classifiers [40–45] that are widely used in fault diagnosis tasks for comparison. The end-to-end model that we proposed in a previous work [34] used time and frequency signals as raw input signals to detect the gear fault patterns. The test results of those methods on two datasets are shown in Table 4.

From the results, we found that the accuracy of the MFCC-delta CNN and the MFCC-delta-delta CNN had similar performances, which were better than the MFCC CNN, but the accuracy of the best traditional method, namely the MFCC-delta CNN, was 10% lower than our attention-based multi-scale CNN for evaluation A. Moreover, the prediction accuracy of all methods declined for evaluation B, but those traditional methods gave worse performances compared with our model. The above conclusions indicate that our attention-based multi-scale CNN structure could learn more discriminative features than traditional manual features by combining attention-based multi-scale information.

Furthermore, the recognition accuracy of our attention multi-scale CNN model reached 93.3% for evaluation A and 82.8% for evaluation B. This was 5.4 % and 6.0% higher than the accuracy of the best classic diagnosis method, namely manual features + k-nearest neighbor (KNN).

In addition, our attention multi-scale CNN model showed an improvement of 3.6% when compared with the end-to-end CNN model for evaluation A. Furthermore, our model achieved at least a 1.7% improvement over the end-to-end CNN model for evaluation B. The improvement proved that the multi-scale CNN model based on the attention mechanism was able to adaptively learn more efficient frequency representations using attention based multi-scale band-pass filters from raw waveforms without frequency input signals.

In Figure 11, we provide the confusion matrix in order to further analyze the performance of our proposed method regarding the two evaluation methods. From the confusion matrix for evaluation A, we observed that most fault patterns under different working conditions could obtain a high classification accuracy, except for gears with pitting types at 1800 rev/min and 2700 rev/min. The two categories appeared to be easily misclassified due to the signal of pitting types at 1800 rev/min under the no load condition being similar to the same type at 2700 rev/min under the 13.5-Nm-load condition, as caused by the amplitude modulation and frequency modulation, the phenomena of which is discussed in Section 4.1. Furthermore, from the confusion matrix for evaluation B, we noticed that the recognition accuracy declined for some classes, especially for gears with pitting types at 1800 rev/min. According to Figure 10 in Section 4.3, we found that the temporal parts, where the attention mechanism was located, were totally different for pitting-type gears, which means that the discriminative feature of pitting-type gears that the model learned were different at 1800 rev/min under the two load conditions. This may explain why the pitting-type gears displayed a lower performance for evaluation B. As for

the rest of the classes, we suspect that the easy misclassification was due to some extra information between the acoustic signals and fault types existing in the load working condition but not in the no-load working condition such that the training the model in the load condition and testing it in no-load condition could be viewed as diagnosis fault patterns with noise, which led to some degree of overfitting in our model. In general, the attention-based multi-scale CNN model that we propose still had better generalization capabilities under variable load conditions compared with the other methods.

**Table 4.** Prediction accuracy comparison of our attention-based multi-scale CNN model and other methods. GBDT: Gradient Boosting Decision Tree, KNN: k-nearest neighbor, SVM: support vector machine.

**Figure 11.** Confusion matrixes for the proposed attention-based multi-scale CNN model. The left matrix shows the statistics for evaluation (**A**), while the right matrix shows the statistics for evaluation (**B**).

Finally, to better show the performance of the attention-based multi-scale CNN model for the two evaluation methods, we visualized the prediction result of the model by using a t-SNE (t-distributed stochastic neighbor embedding) algorithm (Figure 12). The t-SNE algorithm was operated on the output matrix of the last fully connected layer to reduce the dimensionality to conveniently show the classification result in three-dimensional space. From these visual results of three-dimensional space, we observed that most features clustered successfully around the two evaluation methods, which also proved that our attention-based multi-scale model achieved a better performance at acoustic signal-based gear fault diagnosis under different working conditions.

**Figure 12.** Feature visualization via t-SNE (t-distributed stochastic neighbor embedding). Left shows the feature representations for the last fully connected layer of attention-based multi-scale CNN model for evaluation (**A**). Right shows the feature representations for evaluation (**B**).

#### **5. Conclusions**

In this paper, a novel ABD method was proposed for gear fault diagnosis under different working conditions based on a multi-scale convolutional learning structure and attention mechanism. By using a multi-scale convolutional learning structure, our model was able to automatically mine more efficient feature representations from raw acoustic signals. It achieved better performance than the single-scale models. Based on the multi-scale convolutional learning structure, a novel attention mechanism, which operated on the convolutional layer, was applied to adaptively extract relevant fault information and reduce the data distribution variation under different working conditions. The experimental result for the two evaluation methods showed that the accuracy of our model reached 93.3% and 82.8%, respectively. All the performance metrics were higher than those of the standard CNN model, end-to-end CNN model based on time and frequency domain signals, and other traditional fault diagnosis methods with manual features. This indicates that our model was more effective at acoustic-based gear fault diagnosis under different working conditions. Furthermore, we analyzed the discrimination of different scale convolutional learning structures using feature representations and visualized the attention output to provide insight into the reason for the performance improvement in the gear fault diagnosis task. In future, the attention-based mechanism method can be further developed in the ABD field. We will continue to explore the effectiveness of the attention-based method for bearing fault diagnosis, the multi-fault diagnosis of gears, and the coupling fault diagnosis of gearboxes. Meanwhile, we will extend our attention-based method into other fault diagnosis fields that are out of the controlled environment, such as fault diagnosis in normal environmental conditions and strong environmental noise conditions.

**Author Contributions:** Y.Y. contributed to formal analysis, methodology, software and writing-original draft. S.Z. contributed to validation and writing-review&editing; S.Y. contributed to funding acquisition, supervision and writing—review&editing; G.G. contributed to data curation and formal analysis. All authors have read and agreed to the published version of the manuscript.

**Funding:** We gratefully acknowledge the financial support of National Natural Science Foundation of China (NSFC) under grant no. 51275325.

**Acknowledgments:** We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan X Pascal GPU used for this research and the support of National Institute of Measurement and Testing Technology with the provision of the semi-anechoic laboratory.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel. +41 61 683 77 34 Fax +41 61 302 89 18 www.mdpi.com

*Sensors* Editorial Office E-mail<EMAIL_ADDRESS>www.mdpi.com/journal/sensors

MDPI St. Alban-Anlage 66 4052 Basel Switzerland

Tel: +41 61 683 77 34 Fax: +41 61 302 89 18


2.4 The training method of DBN

As same as the training process proposed in [10], the training process of DBN is performed by two steps. The first one, pretraining, utilizes the learning rules of RBM, i.e., Eqs. (4–6), for each RBM independently. The second step is a fine-tuning process using the pretrained parameters of RBMs and BP algorithm. These processes are shown in Figure 4 and Eqs. (11)–(13).

$$
\Delta w\_{ji}^{L} = -e \left( \sum\_{i} \frac{\partial E}{\partial w\_{ji}^{L+1}} w\_{ji}^{L+1} \right) \left( \mathbf{1} - h\_j^{L} \right) \nu\_i^{L} \tag{11}
$$

$$
\Delta b\_j^L = -\varepsilon \left( \sum\_i \frac{\partial E}{\partial w\_{ji}^{L+1}} w\_{ji}^{L+1} \right) \left( \mathbf{1} - h\_j^L \right) \tag{12}
$$

$$E = \frac{1}{2} \sum\_{t=1}^{T} \left( \mathfrak{y}\_t - \mathfrak{y}\_t \right) \tag{13}$$

In the case of reinforcement learning (RL), the output is decided by a probability distribution, e.g., the Gaussian distribution y � π μ; σ2 ð Þ. So the output units are the mean μ and the variance σ instead of one unit y.

$$
\mu = \sum\_{j} w\_{\mu j} x\_j \tag{14}
$$

$$\sigma = \frac{1}{1 + \exp\left(-\sum\_{j} w\_{\sigma j} z\_{j}\right)}\tag{15}$$

$$\pi(\mu, \sigma^2) = \frac{1}{\sqrt{2\pi}\sigma} \exp\left(-\frac{(y-\mu)^2}{2\sigma^2}\right) \tag{16}$$

The learning algorithm of stochastic gradient ascent (SGA) [7] is as follows.

Step 1. Observe an input xt ¼ xt ð Þ ; xt�1; …; xt�nþ1 .

Step 2. Predict a future data yt ¼ xtþ1 according to a probability yt � π xt ð Þ ; w with ANN models which are constructed by parameters w wμj; wσj; wij; vji � �.

Step 3. Receive a scalar reward/punishment rt by calculating the prediction error:

Figure 4. The training of DBN by BP method.

Time Series Analysis - Data, Methods, and Applications

$$
\sigma\_t = \begin{cases}
\mathbf{1} & \text{if } \left(\mathbf{y}\_t - \mathbf{\tilde{y}}\_t\right)^2 \le \zeta \\
\end{cases}
\tag{17}
$$

evijðÞ¼ t ewμj

DOI: http://dx.doi.org/10.5772/intechopen.85457

e L wij

> e L bj

is proposed as in Eq. (27):

shown as the following.

method is as follows:

45

• The number of RBMs: [0–3]

• The number of units in each layer of DBN: [2–20]

• Learning rate of each RBM in Eqs. (4)–(6): [10�5

• Fixed learning rate of SGA in Eq. (21): [10�5

or else if the error is not changed, stop the exploration,

• Discount factor in Eq. (19): [10�5

• Coefficient in Eq. (27) [0.5–2.0]

limitations.

connections. Step 3. If the error between yt

parameters,

else return to step 1.

where is 0≤ β a constant.

2.5 Optimization of meta-parameters

ðÞ¼ t ∑ j e

ðÞ¼ t ∑ k e

ð Þt wμj þ ewσj

Training Deep Neural Networks with Reinforcement Learning for Time Series Forecasting

The eið Þt of the RBM of Lth layer in the case of the DBN is given as follows:

ð Þt wσj

Lþ1 wij ð Þt wLþ1 wij ! 1 � hL

Lþ1 wij ð Þt wLþ1 wij � � 1 � hL

The learning rate ε in Eq. (21) affects the learning performance of fine-tuning of DBN. Different values to result different training error (mean squared error (MSE)) as shown in Figure 5. An adaptive learning rate as a linear function of learning error

The number of RBM that constitute the DBN and the number of neurons of

optimization (PSO) method is used to decide the structure of DBN, and in [13] it is suggested that random search method [16] is more efficient. In the experiment of time series forecasting by DBN and SGA shown in this chapter, these metaparameters were decided by the random search, and the exploration limits are

> –10�1 ]

The optimization algorithm of these meta-parameters by the random search

Step 2. Predict a future data yt ≈xtþ1 by MLP or DBN using the current weighted

Step 1. Set random values of meta-parameters beyond the exploration

each layer affects prediction performance seriously. In [9], particle swarm

� � 1 � zjð Þt � �xið Þt (24)

j � �vL

j

ε ¼ βMSEð Þ t � 1 (27)

–10�1 ]

, xtþ1 is reduced enough, store the values of meta-

–10�1 ] i (25)

� � (26)

where ζ is an evaluation constant greater than or equal to zero. Step 4. Calculate characteristic eligibility eið Þt and eligibility trace Dið Þt :

$$
\sigma\_i(t) = \frac{\partial}{\partial w\_i} \ln \left\{ \pi(\mathbf{x}\_t, \mathbf{w}) \right\} \tag{18}
$$

$$
\overline{D}\_i(t) = e\_i(t) + \chi \overline{D}\_i(t-1) \tag{19}
$$

where 0 ≤γ <1 is a discount factor and wi denotes ith internal variable of DBN.

Step 5. Calculate the modification Δwið Þt :

$$
\Delta w\_i(t) = (r\_t - b)\overline{D}\_i(t) \tag{20}
$$

where b≥ 0 denotes the reinforcement baseline (it can be set as zero). Step 6. Improve the policy Eq. (16) by renewing its internal variable wi by Eq. (21):

$$
\omega w\_i \leftarrow w\_i + \varepsilon \Delta w\_i \tag{21}
$$

where 0≤ε ≤1 is a learning rate.

Step 7. For the next time step t þ 1, return to step 1.

Characteristic eligibility eið Þt , shown in Eq. (18), means that the change of the policy function concerns with the change of system internal variable vector. In fact, the algorithm combines reward/punishment to modify the stochastic policy with its internal variable renewing by Step 4 and Step 5.

The calculation of ewμj ð Þt , ewσj ð Þt , evijð Þt in MLP part of DBN is induced as follows;

$$\mathbf{e}\_{w\_{\neq}}(\mathbf{t}) = \frac{\mathbf{y}\_t - \mu\_{\mathbf{f}}}{\sigma\_{\mathbf{f}}^2} \mathbf{z}\_{\mathbf{j}}(\mathbf{t}) \tag{22}$$

$$\varepsilon\_{w\_{\eta}}(t) = \frac{\left(\mathcal{y}\_t - \mu\_t\right)^2 - \sigma^2}{\sigma\_t^2} (\mathbf{1} - \sigma\_t) \mathbf{z}\_{\dot{\jmath}}(t) \tag{23}$$

Figure 5. The learning errors given by different learning rates.

Training Deep Neural Networks with Reinforcement Learning for Time Series Forecasting DOI: http://dx.doi.org/10.5772/intechopen.85457

$$\boldsymbol{e}\_{v\_{\overline{\boldsymbol{\eta}}}}(t) = \left(\boldsymbol{e}\_{w\_{\overline{\boldsymbol{\eta}}}}(t)\boldsymbol{w}\_{\boldsymbol{\mu}\overline{\boldsymbol{\eta}}} + \boldsymbol{e}\_{w\_{\overline{\boldsymbol{\eta}}}}(t)\boldsymbol{w}\_{\boldsymbol{\sigma}\overline{\boldsymbol{\eta}}}\right) \left(\mathbbm{1} - \boldsymbol{z}\_{\overline{\boldsymbol{\eta}}}(t)\right) \boldsymbol{\varkappa}\_{i}(t) \tag{24}$$

The eið Þt of the RBM of Lth layer in the case of the DBN is given as follows:

$$\boldsymbol{e}\_{w\_{\vec{\eta}}}^{L}(t) = \left(\sum\_{j} \boldsymbol{e}\_{w\_{\vec{\eta}}}^{L+1}(t) \boldsymbol{w}\_{w\_{\vec{\eta}}}^{L+1}\right) \left(\mathbf{1} - \boldsymbol{h}\_{j}^{L}\right) \boldsymbol{v}\_{i}^{L} \tag{25}$$

$$\boldsymbol{e}\_{b\_{j}}^{L}(t) = \left(\sum\_{k} \boldsymbol{e}\_{w\_{\vec{\imath}}}^{L+1}(t) \boldsymbol{w}\_{w\_{\vec{\imath}}}^{L+1}\right) \left(1 - h\_{j}^{L}\right) \tag{26}$$

The learning rate ε in Eq. (21) affects the learning performance of fine-tuning of DBN. Different values to result different training error (mean squared error (MSE)) as shown in Figure 5. An adaptive learning rate as a linear function of learning error is proposed as in Eq. (27):

$$
\varepsilon = \beta \text{MSE}(t-1) \tag{27}
$$

where is 0≤ β a constant.

rt ¼ 1 if yt � ~yt

(

eiðÞ¼ t

Step 5. Calculate the modification Δwið Þt :

Time Series Analysis - Data, Methods, and Applications

where 0≤ε ≤1 is a learning rate. Step 7. For the next time step t þ 1, return to step 1.

ð Þt , ewσj

ewμj

ðÞ¼ t yt � μt

internal variable renewing by Step 4 and Step 5.

ewσj

The calculation of ewμj

DBN.

Eq. (21):

Figure 5.

44

The learning errors given by different learning rates.

�1 else

Step 4. Calculate characteristic eligibility eið Þt and eligibility trace Dið Þt :

∂ ∂wi

where ζ is an evaluation constant greater than or equal to zero.

where 0 ≤γ <1 is a discount factor and wi denotes ith internal variable of

Step 6. Improve the policy Eq. (16) by renewing its internal variable wi by

Characteristic eligibility eið Þt , shown in Eq. (18), means that the change of the policy function concerns with the change of system internal variable vector. In fact, the algorithm combines reward/punishment to modify the stochastic policy with its

> ðÞ¼ t yt � μt σ2 t

� �2 � σ2 σ2 t

where b≥ 0 denotes the reinforcement baseline (it can be set as zero).

� �2

≤ζ

DiðÞ¼ t eiðÞþt γDið Þ t � 1 (19)

ΔwiðÞ¼ t ð Þ rt � b Dið Þt (20)

wi wi þ εΔwi (21)

ð Þt , evijð Þt in MLP part of DBN is induced as follows;

zjð Þt (22)

ð Þ 1 � σt zjð Þt (23)

ln π xt f g ð Þ ; w (18)

(17)

#### 2.5 Optimization of meta-parameters

The number of RBM that constitute the DBN and the number of neurons of each layer affects prediction performance seriously. In [9], particle swarm optimization (PSO) method is used to decide the structure of DBN, and in [13] it is suggested that random search method [16] is more efficient. In the experiment of time series forecasting by DBN and SGA shown in this chapter, these metaparameters were decided by the random search, and the exploration limits are shown as the following.

The optimization algorithm of these meta-parameters by the random search method is as follows:

or else if the error is not changed,

stop the exploration,

else return to step 1.