Ref. [75] is a typical example of papers concerning methodical maintenance of NIRS experiments. Research focuses on developing a method for choosing effective training data sets for BCI training. In particular, BCI was considered for definition of concentration of examinees during experiment. This research also was devoted to integration of EEG and NIRS for their joint usage in "cognitive level" BCI applications. This term designates level of the interest of the examinee in experiment process. For construction of such interface, the paradigm based on P300 evoked potential was used. In this work, two experiments are presented: first—the mathematical task with NIRS-only measurements took place. Hybrid EEG + NIRS was used under EP paradigm (P300), and BCI system of type "ON-OFF machine" was constructed on its basis. Experiments had shown that with the use of NIRS, it is possible to differentiate concentration conditions (mental activity) and to

distinguish them from level of mental rest. The second experiment, however, has

As for the struggle against artifacts, Ref. [76] brings it on a new level. The work sets as its purpose working out methods of artifacts cutout and implementation of those methods. Not only artifacts of true or "internal" physiological nature were considered (such as relatively stable heart beating waves, appearing on HbR and HbO concentrations' changes curves), but also artifacts of physiological nature, caused by external factors. For example, there were considered changes in signal, conditioned by sudden attraction of attention to external objects or conditions. A number of external distracting factors, in particular, sharp sounds, distracting noise, are considered. The whole purpose of this work was to develop compensation for distracting factors, for NIRS-BCI to satisfy practical conditions. In the article, the system of filtration of the mentioned types of the artifacts based on hidden Markov chains (HMM) is considered. In [72], struggle against artifacts was carried out by medicamentous methods—by the local application of vasodilator drug.

One of the techniques directed to improvement of classifiers functioning within BCI is the utilization of so-called hybrid BCI in which signals about a condition of the examinee were received by means of at least two devices of various modalities [51, 66, 76–81]. In [81], the concept of hybrid BCI in general—not only in sense of various modalities, but also in sense of various paradigms of record (ERD/EP), was introduced. The basic criteria of such device with reference to practice were given. The necessary quality standards were discussed. Various types of BCI, for example "the brain switch," were considered, basically focusing on an artificial hand limb,

One can consider [82] as a good "head first" text about nervous processes studying techniques. Here, it is noticed that NIRS possesses the low time resolution and can even interfere with transitions between conditions. To cut out this lack, it was recommended to use it together with EEG. In general, the article affirms that the BCI based on NIRS are inexpensive, but does not show high-quality results. Various methods of preprocessing (CAR, SL, ICA, CSP, PCA, SVD, CSSP, Freq-Norm, LAT, LKF, and CSSD) are considered and estimated. Most often used are ICA, CAR, LS, PCA, CSP, and adaptive filtering. In this paper, the hybrid BCI on the basis of NIRS and MEG were also mentioned and also stated that they are

FOS responses [31–34] do not directly relate to BCI, but nevertheless were considered for a possibility to build optical EP BCI based on such phenomena. FOS usage can increase "reaction time" of hemodynamic responses and hence can improve NIRS-FOS BCI performance. FOS registration demands extreme temporal resolution on NIRS equipment. Analogically, research of correlation with fMRI also difficult in that sense that fMRI-BOLD signal with "reception" time about 2.5 s has

disproportionately expensive, considering achievable results.

no sufficient time resolution to obtain information on "fast" reactions.

revealed essential and statistically significant result only in EEG.

New Frontiers in Brain-Computer Interfaces

which operates by means of EP.

20

neurobiology as a whole. In the context of problem at hand, it makes sense to concentrate on papers, which aims to use some kind of NIRS or optical equipment. optical properties of a living tissue and physical features of distribution of nearinfrared range radiation in biological substances possess complex structure. Hybrid interfaces are focused mainly on conducting NIRS cognitive experiments (in which some cognitive effects or conditions exploited and responses were acquired with NIRS) in synchrony or in parallel with registration of motor activity with the EEG usage only. There are exceptionally few publications, concerning the technical details and improvement of characteristics of the spectrometer equipment. Technical features are not addressed in detail even in papers that were using devices of "own manufacture." The field of NIRS devices with technical features, obviously, is assigned exclusively to manufacturers of the equipment. The insignificant part of papers, reviewed here, was devoted to the usage of NIRS/EEG BCI in one setup with assistive robotic devices, artificial limbs, and exoskeletons. Due to pressing nature of this matter and under a condition of extraordinary results, occurrences in the near future in this field, obviously the share of such publications, will only grow, probably, involving in the pool of papers on the technical details of such installa-

Near-Infrared Optical Technologies in Brain-Computer Interface Systems

DOI: http://dx.doi.org/10.5772/intechopen.83345

tions as a whole, including NIRS and robotics.

Author details

Russian Federation

23

Korshakov Alexei Vyacheslavovich1,2,3

(IHNA&NPh RAS), Moscow, Russian Federation

provided the original work is properly cited.

\*Address all correspondence to<EMAIL_ADDRESS>
1 Pirogov Russian National Research Medical University (RNRMU), Moscow,

2 National Research Center "Kurchatov Institute", Moscow, Russian Federation

© 2019 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/ by/3.0), which permits unrestricted use, distribution, and reproduction in any medium,

3 Institute of Higher Nervous Activity and Neurophysiology of RAS

In this way, Refs. [91, 92] describe methods, which utilize NIR radiation in confocal microscope in order of correction and control of neural cell growth. Further developments of this direction can be found in [93], where laser microsurgery of cellular membrane was described.

Ref. [47] was devoted to optogenetics methods of neural cells activation with high spatial precision by making those cells produce reaction when irradiated with light. This is achievable by the means of transfection of cell genome with gene constructs, which corresponds to channelrhodopsin 2 protein. Activation of transfected cells develops upon exposure to light of NIR range (860–1028 nm). Such manipulations may provide a researcher with method of external activation of nerve cells', deeply situated in experimental animals' brain and with high spatial resolution. Experiments in the field also features ability to "read" neurons' activity patterns, induced on purpose. This can be achieved by variation of wavelength and intensity of excitatory radiation. This direction of thought is interesting because it represents fully optical method for monitoring, excitation, and detection of neural activity on the cellular level. One may hypothesize that in distant enough perspective, such a research may lead to obtaining new and more precise data on brain structure, functioning, and connections. Also, new generation of genetically coded voltage sensitive dyes can yield state-of-the-art methods and techniques of acquiring new data on functioning of neuron cells' population level [94]. Alas, in their present state, optogenetics methods are very suitable and useful for fundamental research (usually not even in vivo), but they have not yet leveled up with practical demands of problem discussed here.

At last, it is necessary to notice that despite essential successes in the field of construction of BCI and machine interfaces (these are researches that generate about thousands of publications every year), progress in the field of creation of devices would allow patients with a full paralysis and "locked-in" syndrome to interact with an external world, which is, for now, possible to characterize only as moderate [95].

## 6. Conclusion

Objectively, the quantity of works in dynamics for more than 10 years grows, and the considered area of researches extends. Also, the subjects of papers in the field extend in more details, growing with new data.

The fact that the considerable part of papers in the field is reviews attracts attention. It can testify to the big number of scientific personnel, which began or begin at the moment the work on a considered problem. Also from this number, it is possible to draw indirect conclusions about quantity of scientific personnel.

The offered system of classification of publications was given here in hope that it will be useful for researchers' choice of a direction in considered area and in order to aid more effective development of this direction as a whole.

There are a big number of articles in which only the near-infrared range spectrometry is used (without EEG and other means of neuroimaging) and also the ERD/ERS experimental paradigm most frequent and limited enough circle of cognitive effects was measured.

Researches, in which hybrid interfaces were used in comparison with aforementioned, are scarce. Even less papers concerning fast optical signals, fundamental physiological questions with the use of NIRS or connection of nervous activity, and

## Near-Infrared Optical Technologies in Brain-Computer Interface Systems DOI: http://dx.doi.org/10.5772/intechopen.83345

optical properties of a living tissue and physical features of distribution of nearinfrared range radiation in biological substances possess complex structure. Hybrid interfaces are focused mainly on conducting NIRS cognitive experiments (in which some cognitive effects or conditions exploited and responses were acquired with NIRS) in synchrony or in parallel with registration of motor activity with the EEG usage only. There are exceptionally few publications, concerning the technical details and improvement of characteristics of the spectrometer equipment. Technical features are not addressed in detail even in papers that were using devices of "own manufacture." The field of NIRS devices with technical features, obviously, is assigned exclusively to manufacturers of the equipment. The insignificant part of papers, reviewed here, was devoted to the usage of NIRS/EEG BCI in one setup with assistive robotic devices, artificial limbs, and exoskeletons. Due to pressing nature of this matter and under a condition of extraordinary results, occurrences in the near future in this field, obviously the share of such publications, will only grow, probably, involving in the pool of papers on the technical details of such installations as a whole, including NIRS and robotics.

## Author details

neurobiology as a whole. In the context of problem at hand, it makes sense to concentrate on papers, which aims to use some kind of NIRS or optical equipment. In this way, Refs. [91, 92] describe methods, which utilize NIR radiation in confocal microscope in order of correction and control of neural cell growth. Further developments of this direction can be found in [93], where laser microsurgery

Ref. [47] was devoted to optogenetics methods of neural cells activation with high spatial precision by making those cells produce reaction when irradiated with light. This is achievable by the means of transfection of cell genome with gene constructs, which corresponds to channelrhodopsin 2 protein. Activation of transfected cells develops upon exposure to light of NIR range (860–1028 nm). Such manipulations may provide a researcher with method of external activation of nerve cells', deeply situated in experimental animals' brain and with high spatial resolution. Experiments in the field also features ability to "read" neurons' activity patterns, induced on purpose. This can be achieved by variation of wavelength and intensity of excitatory radiation. This direction of thought is interesting because it represents fully optical method for monitoring, excitation, and detection of neural activity on the cellular level. One may hypothesize that in distant enough perspective, such a research may lead to obtaining new and more precise data on brain structure, functioning, and connections. Also, new generation of genetically coded voltage sensitive dyes can yield state-of-the-art methods and techniques of acquiring new data on functioning of neuron cells' population level [94]. Alas, in their present state, optogenetics methods are very suitable and useful for fundamental research (usually not even in vivo), but they have not yet leveled up with practical

At last, it is necessary to notice that despite essential successes in the field of construction of BCI and machine interfaces (these are researches that generate about thousands of publications every year), progress in the field of creation of devices would allow patients with a full paralysis and "locked-in" syndrome to interact with an external world, which is, for now, possible to characterize only as

Objectively, the quantity of works in dynamics for more than 10 years grows, and the considered area of researches extends. Also, the subjects of papers in the

The fact that the considerable part of papers in the field is reviews attracts attention. It can testify to the big number of scientific personnel, which began or begin at the moment the work on a considered problem. Also from this number, it is

The offered system of classification of publications was given here in hope that it will be useful for researchers' choice of a direction in considered area and in order to

There are a big number of articles in which only the near-infrared range spectrometry is used (without EEG and other means of neuroimaging) and also the ERD/ERS experimental paradigm most frequent and limited enough circle of cog-

Researches, in which hybrid interfaces were used in comparison with aforementioned, are scarce. Even less papers concerning fast optical signals, fundamental physiological questions with the use of NIRS or connection of nervous activity, and

possible to draw indirect conclusions about quantity of scientific personnel.

of cellular membrane was described.

New Frontiers in Brain-Computer Interfaces

demands of problem discussed here.

field extend in more details, growing with new data.

aid more effective development of this direction as a whole.

moderate [95].

6. Conclusion

nitive effects was measured.

22

Korshakov Alexei Vyacheslavovich1,2,3

1 Pirogov Russian National Research Medical University (RNRMU), Moscow, Russian Federation

2 National Research Center "Kurchatov Institute", Moscow, Russian Federation

3 Institute of Higher Nervous Activity and Neurophysiology of RAS (IHNA&NPh RAS), Moscow, Russian Federation

\*Address all correspondence to<EMAIL_ADDRESS>
© 2019 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/ by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

## References

[1] Lal TL, Schröder M, Hill NJ, Preissl H, Hinterberger T, Mellinger J, et al. A brain computer interface with online feedback based on magnetoencephalography. In: Proceedings of the 22nd International Conference on Machine Learning (ICML 2005); 07-11 August 2005; Bonn, Germany. New York: ACM; 2005. p. 465–472. DOI: 10.1145/1102351.1102410

[2] Tamada JA, Lesho M, Tierney MJ. Keeping watch on glucose. IEEE Spectrum. 2002;39(4):52-57. DOI: 10.1109/6.993789

[3] Ishizawa H, Muro A, Takano T, Honda K, Kanai H. Non-invasive blood glucose measurement based on ATR infrared spectroscopy. In: SICE Annual Conference (SICE 2008); 20–22 August 2008; Tokyo, Japan. Tokyo: IEEE; 2008. p. 321-324. DOI: 10.1109/ SICE.2008.4654672

[4] Smith JL. The Pursuit of Noninvasive Glucose. 5th ed. 2017 [Internet]. 2017. Available from: https://www. researchgate.net/publication/ 317267760\_The\_Pursuit\_of\_ Noninvasive\_Glucose\_5th\_Edition.pdf [Accessed: 2018-12-29]

[5] Khalil OS. Spectroscopic and clinical aspects of noninvasive glucose measurements. Clinical Chemistry. 1999;45:165-177

[6] León-Carrión J, León-Domínguez U. Functional Near-Infrared Spectroscopy (fNIRS): Principles and Neuroscientific Applications. In: Bright P, editor. Neuroimaging. Rijeka: IntechOpen; 2012. p. 47-74. DOI: 10.5772/23146

[7] Song S, Kobayashi Y, Fujie MG. Monte-Carlo simulation of light propagation considering characteristic of near-infrared LED and evaluation on tissue phantom. In: 1st CIRP Conference on BioManufacturing (BioM 2013); 3–5 March 2013; Tokyo, Japan: Elsevier;

2013. p. 25-30. DOI: 10.1016/j. procir.2013.01.005

[8] Delpy DT, Cope M, van der Zee P, Arridge SR, Wray S, Wyatt JS. Estimation of optical pathlength through tissue from direct time of flight measurement. Physics in Medicine and Biology. 1988;33(12):1433-1442. DOI: 10.1088/0031-9155/33/12/008

[14] Tuchin VV. Handbook of Optical Biomedical Diagnostics. Chichester, Bellingham: SPIE Press; 2002. PM107

DOI: http://dx.doi.org/10.5772/intechopen.83345

Near-Infrared Optical Technologies in Brain-Computer Interface Systems

Technical and Theoretical Literature Publishing House; 1951. 288 p

[23] IUPAC. Compendium of Chemical Terminology. 2nd ed. (the "Gold Book"); 1997. Online corrected version: (2006–) "Beer–Lambert law". Available from: http://goldbook.iupac.org/

[24] Takatani S, Graham MD. Theoretical analysis of diffuse reflectance from a two-layer tissue model. IEEE Transactions on Biomedical Engineering. 1979;26(12):656-664

[25] Kim JG, Liu H. Variation of

haemoglobin extinction coefficients can cause errors in the determination of haemoglobin concentration measured by near-infrared spectroscopy. Physics in Medicine and Biology. 2007;52: 6295-6322. DOI: 10.1088/0031-9155/52/

[26] Jonson C, Gay A. The influence of non-ionizing electro-magnitic radiation on biological media and systems. TIIER.

[27] Harry LG, Xu Y, Pei Y, Barbour RL. Spatial deconvolution technique to improve the accuracy of reconstructed three-dimensional diffuse optical tomographic images. Applied Optics. 2005;44(6):941-953. DOI: 10.1364/

[28] Schmitz CH, Klemer DP, Hardin R, Katz MS, Pei Y, Graber HL, et al. Design and implementation of dynamic nearinfrared optical tomographic imaging instrumentation for simultaneous dualbreast measurements. Applied Optics. 2005;44(11):2140-2153. DOI: 10.1364/

[29] Xu Y, Graber HL, Barbour RL. Image correction algorithm for functional three-dimensional diffuse optical tomography brain imaging. Applied Optics. 2007;46(10):1693-1704.

DOI: 10.1364/AO.46.001693

B00626.html

20/014

1972;T60(6):49-79

AO.44.000941

AO.44.002140

[15] Torricelli A, Contini D, Pifferi A, Caffini M, Re R, Zucchelli L, et al. Time domain functional NIRS imaging for human brain mapping. NeuroImage. 2014;85:28-50. DOI: 10.1016./j. neuroimage.2013.05.105

[16] Naulaers G, Morren G, Van Huffel S, Casaer P, Devlieger H. Cerebral tissue oxygenation index in very premature infants. Archives of Disease in

Childhood. Fetal and Neonatal Edition. 2002;87:F189-F192. DOI: 10.1136/

[17] Cui X, Bray S, Bryant DM, Glover

comparison of NIRS and fMRI across multiple cognitive tasks. NeuroImage. 2011;54(4):2808-2821. DOI: 10.1016/j.

[18] Ward TE. Hybrid optical–electrical brain computer interfaces, practices and possibilities. In: Allison B, Dunne S, Leeb R, Del R, Millan J, Nijholt A, editors. Towards Practical Brain-Computer Interfaces. Biological and

Springer; 2012. pp. 17-40. DOI: 10.1007/

[19] Kolosovskii BN. Circulation of blood in brain. In: Kolosovskii BN, editor. Moscow: Medgiz; 1951. 371 p

[20] Schmidt RF, Thews G, editors. Human Physiology. Berlin: Springer-

[21] Chandrasekhar S. Radiative

Transfer. New York: Dover Publications

[22] Shifrin KS. Scattering of Light in Turbid Media. Leningrad: State

GH, Reiss AL. A quantitative

neuroimage.2010.10.069

Medical Physics. Biomedical Engineering. Berlin, Heidelberg:

978-3-642-29746-5\_2

Verlag; 1989

Inc; 1960. 393 p

25

1110 p

fn.87.3.F189

[9] Bakker A, Smith B, Ainslie P, Smith K. Near-infrared spectroscopy. In: Ainslie P, editor. Applied Aspects of Ultrasonography in Humans. Rijeka: IntechOpen; 2012. p. 65-88. DOI: 10.5772/324939

[10] Gagnon L, Yucel MA, Dehaes M, Cooper RJ, Perdue KL, Selb J, et al. Quantification of the cortical contribution to the NIRS signal over the motor cortex using concurrent NIRSfMRI measurements. NeuroImage. 2012; 59(4):3933-3940. DOI: 10.1016/j. neuroimage.2011.10.054

[11] Cope M, Delpy D. System for longterm measurement of cerebral blood and tissue oxygenation on newborn infants by near infra-red transillumination. Medical and Biological Engineering and Computing. 1988;26(3):289-294

[12] Durduran T, Choe R, Baker WB, Yodh AG. Diffuse optics for tissue monitoring and tomography. Reports on Progress in Physics. 2010;73(076701): 43. DOI: 10.1088/0034-4885/73/7/ 076701

[13] Arifler D, Zhu T, Madaan S, Tachtsidis I. Optimal wavelength combinations for near-infrared spectroscopic monitoring of changes in brain tissue hemoglobin and cytochrome c oxidase concentrations. Biomedical Optics Express. 2015;6(3): 933-947. DOI: 10.1364/BOE.6.000933

Near-Infrared Optical Technologies in Brain-Computer Interface Systems DOI: http://dx.doi.org/10.5772/intechopen.83345

[14] Tuchin VV. Handbook of Optical Biomedical Diagnostics. Chichester, Bellingham: SPIE Press; 2002. PM107 1110 p

References

[1] Lal TL, Schröder M, Hill NJ, Preissl H, Hinterberger T, Mellinger J, et al. A brain computer interface with

Proceedings of the 22nd International Conference on Machine Learning (ICML 2005); 07-11 August 2005; Bonn, Germany. New York: ACM; 2005. p. 465–472. DOI: 10.1145/1102351.1102410

New Frontiers in Brain-Computer Interfaces

2013. p. 25-30. DOI: 10.1016/j.

10.1088/0031-9155/33/12/008

[8] Delpy DT, Cope M, van der Zee P, Arridge SR, Wray S, Wyatt JS. Estimation of optical pathlength

through tissue from direct time of flight measurement. Physics in Medicine and Biology. 1988;33(12):1433-1442. DOI:

[9] Bakker A, Smith B, Ainslie P, Smith K. Near-infrared spectroscopy. In: Ainslie P, editor. Applied Aspects of Ultrasonography in Humans. Rijeka: IntechOpen; 2012. p. 65-88. DOI:

[10] Gagnon L, Yucel MA, Dehaes M, Cooper RJ, Perdue KL, Selb J, et al. Quantification of the cortical

contribution to the NIRS signal over the motor cortex using concurrent NIRSfMRI measurements. NeuroImage. 2012; 59(4):3933-3940. DOI: 10.1016/j.

[11] Cope M, Delpy D. System for longterm measurement of cerebral blood and tissue oxygenation on newborn

Biological Engineering and Computing.

[12] Durduran T, Choe R, Baker WB, Yodh AG. Diffuse optics for tissue monitoring and tomography. Reports on Progress in Physics. 2010;73(076701): 43. DOI: 10.1088/0034-4885/73/7/

[13] Arifler D, Zhu T, Madaan S, Tachtsidis I. Optimal wavelength combinations for near-infrared

brain tissue hemoglobin and

spectroscopic monitoring of changes in

cytochrome c oxidase concentrations. Biomedical Optics Express. 2015;6(3): 933-947. DOI: 10.1364/BOE.6.000933

procir.2013.01.005

10.5772/324939

neuroimage.2011.10.054

infants by near infra-red transillumination. Medical and

1988;26(3):289-294

076701

[2] Tamada JA, Lesho M, Tierney MJ. Keeping watch on glucose. IEEE Spectrum. 2002;39(4):52-57. DOI:

[3] Ishizawa H, Muro A, Takano T, Honda K, Kanai H. Non-invasive blood glucose measurement based on ATR infrared spectroscopy. In: SICE Annual Conference (SICE 2008); 20–22 August 2008; Tokyo, Japan. Tokyo: IEEE; 2008.

[4] Smith JL. The Pursuit of Noninvasive Glucose. 5th ed. 2017 [Internet]. 2017.

Noninvasive\_Glucose\_5th\_Edition.pdf

[5] Khalil OS. Spectroscopic and clinical

[6] León-Carrión J, León-Domínguez U. Functional Near-Infrared Spectroscopy (fNIRS): Principles and Neuroscientific Applications. In: Bright P, editor. Neuroimaging. Rijeka: IntechOpen; 2012. p. 47-74. DOI: 10.5772/23146

[7] Song S, Kobayashi Y, Fujie MG. Monte-Carlo simulation of light propagation considering characteristic of near-infrared LED and evaluation on tissue phantom. In: 1st CIRP Conference on BioManufacturing (BioM 2013); 3–5 March 2013; Tokyo, Japan: Elsevier;

p. 321-324. DOI: 10.1109/ SICE.2008.4654672

Available from: https://www. researchgate.net/publication/ 317267760\_The\_Pursuit\_of\_

aspects of noninvasive glucose measurements. Clinical Chemistry.

[Accessed: 2018-12-29]

1999;45:165-177

24

online feedback based on magnetoencephalography. In:

10.1109/6.993789

[15] Torricelli A, Contini D, Pifferi A, Caffini M, Re R, Zucchelli L, et al. Time domain functional NIRS imaging for human brain mapping. NeuroImage. 2014;85:28-50. DOI: 10.1016./j. neuroimage.2013.05.105

[16] Naulaers G, Morren G, Van Huffel S, Casaer P, Devlieger H. Cerebral tissue oxygenation index in very premature infants. Archives of Disease in Childhood. Fetal and Neonatal Edition. 2002;87:F189-F192. DOI: 10.1136/ fn.87.3.F189

[17] Cui X, Bray S, Bryant DM, Glover GH, Reiss AL. A quantitative comparison of NIRS and fMRI across multiple cognitive tasks. NeuroImage. 2011;54(4):2808-2821. DOI: 10.1016/j. neuroimage.2010.10.069

[18] Ward TE. Hybrid optical–electrical brain computer interfaces, practices and possibilities. In: Allison B, Dunne S, Leeb R, Del R, Millan J, Nijholt A, editors. Towards Practical Brain-Computer Interfaces. Biological and Medical Physics. Biomedical Engineering. Berlin, Heidelberg: Springer; 2012. pp. 17-40. DOI: 10.1007/ 978-3-642-29746-5\_2

[19] Kolosovskii BN. Circulation of blood in brain. In: Kolosovskii BN, editor. Moscow: Medgiz; 1951. 371 p

[20] Schmidt RF, Thews G, editors. Human Physiology. Berlin: Springer-Verlag; 1989

[21] Chandrasekhar S. Radiative Transfer. New York: Dover Publications Inc; 1960. 393 p

[22] Shifrin KS. Scattering of Light in Turbid Media. Leningrad: State

Technical and Theoretical Literature Publishing House; 1951. 288 p

[23] IUPAC. Compendium of Chemical Terminology. 2nd ed. (the "Gold Book"); 1997. Online corrected version: (2006–) "Beer–Lambert law". Available from: http://goldbook.iupac.org/ B00626.html

[24] Takatani S, Graham MD. Theoretical analysis of diffuse reflectance from a two-layer tissue model. IEEE Transactions on Biomedical Engineering. 1979;26(12):656-664

[25] Kim JG, Liu H. Variation of haemoglobin extinction coefficients can cause errors in the determination of haemoglobin concentration measured by near-infrared spectroscopy. Physics in Medicine and Biology. 2007;52: 6295-6322. DOI: 10.1088/0031-9155/52/ 20/014

[26] Jonson C, Gay A. The influence of non-ionizing electro-magnitic radiation on biological media and systems. TIIER. 1972;T60(6):49-79

[27] Harry LG, Xu Y, Pei Y, Barbour RL. Spatial deconvolution technique to improve the accuracy of reconstructed three-dimensional diffuse optical tomographic images. Applied Optics. 2005;44(6):941-953. DOI: 10.1364/ AO.44.000941

[28] Schmitz CH, Klemer DP, Hardin R, Katz MS, Pei Y, Graber HL, et al. Design and implementation of dynamic nearinfrared optical tomographic imaging instrumentation for simultaneous dualbreast measurements. Applied Optics. 2005;44(11):2140-2153. DOI: 10.1364/ AO.44.002140

[29] Xu Y, Graber HL, Barbour RL. Image correction algorithm for functional three-dimensional diffuse optical tomography brain imaging. Applied Optics. 2007;46(10):1693-1704. DOI: 10.1364/AO.46.001693

[30] Habermehl C, Schmitz CH, Steinbrink J. Contrast enhanced highresolution diffuse optical tomography of the human brain using ICG. Optics Express. 2011;19(19):18636-18644. DOI: 10.1364/OE.19.018636

[31] Medvedev AV, Borisov SV, Gandjbakhche AH, VanMeter J, Kainerstorfer JM. Seeing electroencephalogram through the skull: Imaging prefrontal cortex with fast optical signal. Biomedical Optics. 2010; 15(6):061702. DOI: 10.1117/1.3505007

[32] Huang J, Wang S, Jia S, Mo D, Chen H-C. Cortical dynamics of semantic processing during sentence comprehension: Evidence from eventrelated optical signals. PLoS One. 2013;8 (8):e70671. DOI: 10.1371/journal. pone.0070671

[33] Medvedev AV, Kainerstorfer JM, Borisov SV, VanMetera J. Functional connectivity in the prefrontal cortex measured by near-infrared spectroscopy during ultrarapid object recognition. Journal of Biomedical Optics. 2011;16 (1):016008. DOI: 10.1117/1.3533266

[34] Chiarelli AM, Romani GL, Merla A. Fast optical signals in the sensorimotor cortex: General linear convolution model applied to multiple source– detector distance-based data. NeuroImage. 2014;85:245-254. DOI: 10.1016/j.neuroimage.2013.07.021

[35] Medvedev AV, Kainerstorfer J, Borisov SV, Barbour RL, VanMeter J. Event-related fast optical signal in a rapid object recognition task: Improving detection by the independent component analysis. Brain Research. 2008;1236:145-158. DOI: 10.1016/j. brainres.2008.07.122

[36] Wang J, Dong Q, Niu H. The minimum resting-state fNIRS imaging duration for accurate and stable mapping of brain connectivity network in children. Scientific Reports. 2017;

7(1):6461. DOI: 10.1038/s41598-017- 06340-7

Bioinformatics. 2014;9(1):296-308.

DOI: http://dx.doi.org/10.5772/intechopen.83345

Near-Infrared Optical Technologies in Brain-Computer Interface Systems

[48] Okamoto M, Dan H, Shimizu K, Takeo K, Amita T, Oda I, et al. Multimodal assessment of cortical activation during apple peeling by NIRS and fMRI. NeuroImage. 2004;21: 1275-1288. DOI: 10.1016/j. neuroimage.2003.12.003

[49] Herff C, Heger D, Putze F, Guan C, Schultz T. Self-paced BCI with NIRS

Proceedings of the Fifth International Brain-Computer Interface Meeting 2013 (International BCI Meeting 2013); 3–7 June 2013; Asilomar Conference Center, Pacific Grove, California. Graz: Graz University of Technology Publishing House; 2013. Article ID: 111 DOI: 10.3217/978-3-85125-260-6-111

[50] Xu G, Xu LX, Li D, Liu X.  Computational and

Mathematical Methods in Medicine. 2014;2014:107320. 9 p. DOI: 10.1155/

[51] Almajidy RK, Le KS, Hofmanna UG. Novel near infrared sensors for hybrid

America, 2015. p. 95361H. DOI: 10.1364/

[52] Matthews F, Soraghan C, Ward T, Markham C, Pearlmutter B. Software platform for rapid prototyping of NIRS brain computer interfacing techniques. In: Annual International Conference of the IEEE Engineering in Medicine and Biology Society (IEEE Engineering in Medicine and Biology Society.

Conference 2008); 20–24 August 2008; Vancouver, Canada: IEEE; 2008. p.

4840-4843. DOI: 10.1109/ IEMBS.2008.4650297

BCI applications. In: European Conference on Biomedical Optics (ECBO 2015). Advanced Microscopy Techniques IV; and Neurophotonics II; 21–25 June 2015; Munich Germany. Washington: Optical Society of

ECBO.2015.95361H

2014/107320

based on speech activity. In:

[43] Huang NE, Shen Z, Long SR, Wu MC, Shih HH, Zheng Q, Yen NC, Tung CC, Liu HH. The Empirical Mode Decomposition and the Hilbert Spectrum for Nonlinear and

Nonstationary Time Series Analysis. Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences. 1998;454(1971):903–995. DOI: 10.1098/

[44] Korshakov AV. NIRS signal processing by the means of modified empirical mode decomposition method

for the purpose of separating

[45] Kozel FA, Tian F, Dhamne S, Croarkin PE, McClintock SM, Elliott A, et al. Using simultaneous repetitive transcranial magnetic stimulation/ functional near infrared spectroscopy (rTMS/fNIRS) to measure brain activation and connectivity.

NeuroImage. 2009;47:1177-1184. DOI: 10.1016/j.neuroimage.2009.05.016

Adamantidis A, de Lecea L, Deisseroth K. Circuit-breakers: Optical technologies for probing neural signals and systems. Nature Reviews Neuroscience. 2007;8:

[47] Mohanty SK, Reinscheid RK, Liu X, Okamura N, Krasieva TB, Berns MW.

channelrhodopsin-2 sensitized excitable cells with high spatial resolution using two-photon excitation with a nearinfrared laser microbeam. Biophysical Journal. 2008;95:11. DOI: 10.1529/

[46] Zhang F, Aravanis AM,

577-581. DOI: 10.1038/nrn2192

In-depth activation of

biophysj.108.130187

27

hemodynamic responses for BCI target mental states with specific buildup times. In: International Congress "Neuroscience for Medicine and Psychology"; 1–11 June 2016; Sudak, Crimea, Russia. Moscow: MAX Press;

DOI: 10.17537/2014.9.296

rspa.1998.0193

2016. p. 224-225

[37] Racz FS, Mukli P, Nagy Z, Eke A. Increased prefrontal cortex connectivity during cognitive challenge assessed by fNIRS imaging. Biomedical Optics Express. 2017;8(8):3842-3855. DOI: 10.1364/BOE.8.003842

[38] Bobrov PD, Isaev MH, Korshakov AV, Oganesyan VV, Kerechanin Y, Popodko A, et al. Sources of electrophysiological and foci of hemodynamic brain activity most relevant for controlling a hybrid brain–computer interface based on classification of EEG patterns and nearinfrared spectrography signals during motor imagery. Human Physiology. 2016;42(3):241-251. DOI: 10.1134/ S036211971603004X

[39] Cichocki A, Amari SI. Adaptive Blind Signal and Image Processing, Learning Algorithms and Application. Chichester: John Wiley & Sons, Ltd.; 2002

[40] Beckmann CF, DeLuca M, Devlin JT, Smith SM. Investigations into resting-state connectivity using independent component analysis. Philosophical Transactions of the Royal Society B. 2005;360:1001-1013. DOI: 10.1098/rstb.2005.1634

[41] Frolov A, Bobrov P, Mokienko O, Húsek D, Chernikova L, Konovalov R, et al. Sources of EEG activity most relevant to performance of braincomputer interface based on motor imagery. Neural Network World. 2012; 22(1):21-37. DOI: 10.14311/ NNW.2012.22.002

[42] Korshakov AV, Polikarpov MA, Ustinin MN, Sychev VV, Rykunov SD, Naurzakov SP, et al. Registration and analysis of precise frequency EEG/MEG responses of human brain auditory cortex to monaural sound stimulation with fixed frequency components. Mathematical Biology and

Near-Infrared Optical Technologies in Brain-Computer Interface Systems DOI: http://dx.doi.org/10.5772/intechopen.83345

Bioinformatics. 2014;9(1):296-308. DOI: 10.17537/2014.9.296

[30] Habermehl C, Schmitz CH, Steinbrink J. Contrast enhanced highresolution diffuse optical tomography of the human brain using ICG. Optics Express. 2011;19(19):18636-18644. DOI:

New Frontiers in Brain-Computer Interfaces

7(1):6461. DOI: 10.1038/s41598-017-

[37] Racz FS, Mukli P, Nagy Z, Eke A. Increased prefrontal cortex connectivity during cognitive challenge assessed by fNIRS imaging. Biomedical Optics Express. 2017;8(8):3842-3855. DOI:

[38] Bobrov PD, Isaev MH, Korshakov AV, Oganesyan VV, Kerechanin Y, Popodko A, et al. Sources of electrophysiological and foci of hemodynamic brain activity most relevant for controlling a hybrid brain–computer interface based on classification of EEG patterns and nearinfrared spectrography signals during motor imagery. Human Physiology. 2016;42(3):241-251. DOI: 10.1134/

[39] Cichocki A, Amari SI. Adaptive Blind Signal and Image Processing, Learning Algorithms and Application. Chichester:

[40] Beckmann CF, DeLuca M, Devlin JT, Smith SM. Investigations into resting-state connectivity using independent component analysis. Philosophical Transactions of the Royal Society B. 2005;360:1001-1013. DOI:

[41] Frolov A, Bobrov P, Mokienko O, Húsek D, Chernikova L, Konovalov R, et al. Sources of EEG activity most relevant to performance of braincomputer interface based on motor imagery. Neural Network World. 2012;

[42] Korshakov AV, Polikarpov MA, Ustinin MN, Sychev VV, Rykunov SD, Naurzakov SP, et al. Registration and analysis of precise frequency EEG/MEG responses of human brain auditory cortex to monaural sound stimulation with fixed frequency components.

John Wiley & Sons, Ltd.; 2002

10.1098/rstb.2005.1634

22(1):21-37. DOI: 10.14311/

Mathematical Biology and

NNW.2012.22.002

10.1364/BOE.8.003842

S036211971603004X

06340-7

[31] Medvedev AV, Borisov SV, Gandjbakhche AH, VanMeter J, Kainerstorfer JM. Seeing

electroencephalogram through the skull: Imaging prefrontal cortex with fast optical signal. Biomedical Optics. 2010; 15(6):061702. DOI: 10.1117/1.3505007

[32] Huang J, Wang S, Jia S, Mo D, Chen H-C. Cortical dynamics of semantic

comprehension: Evidence from eventrelated optical signals. PLoS One. 2013;8 (8):e70671. DOI: 10.1371/journal.

[33] Medvedev AV, Kainerstorfer JM, Borisov SV, VanMetera J. Functional connectivity in the prefrontal cortex measured by near-infrared spectroscopy during ultrarapid object recognition. Journal of Biomedical Optics. 2011;16 (1):016008. DOI: 10.1117/1.3533266

[34] Chiarelli AM, Romani GL, Merla A. Fast optical signals in the sensorimotor cortex: General linear convolution model applied to multiple source– detector distance-based data. NeuroImage. 2014;85:245-254. DOI: 10.1016/j.neuroimage.2013.07.021

[35] Medvedev AV, Kainerstorfer J, Borisov SV, Barbour RL, VanMeter J. Event-related fast optical signal in a rapid object recognition task: Improving

detection by the independent component analysis. Brain Research. 2008;1236:145-158. DOI: 10.1016/j.

[36] Wang J, Dong Q, Niu H. The minimum resting-state fNIRS imaging duration for accurate and stable

mapping of brain connectivity network in children. Scientific Reports. 2017;

brainres.2008.07.122

26

processing during sentence

pone.0070671

10.1364/OE.19.018636

[43] Huang NE, Shen Z, Long SR, Wu MC, Shih HH, Zheng Q, Yen NC, Tung CC, Liu HH. The Empirical Mode Decomposition and the Hilbert Spectrum for Nonlinear and Nonstationary Time Series Analysis. Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences. 1998;454(1971):903–995. DOI: 10.1098/ rspa.1998.0193

[44] Korshakov AV. NIRS signal processing by the means of modified empirical mode decomposition method for the purpose of separating hemodynamic responses for BCI target mental states with specific buildup times. In: International Congress "Neuroscience for Medicine and Psychology"; 1–11 June 2016; Sudak, Crimea, Russia. Moscow: MAX Press; 2016. p. 224-225

[45] Kozel FA, Tian F, Dhamne S, Croarkin PE, McClintock SM, Elliott A, et al. Using simultaneous repetitive transcranial magnetic stimulation/ functional near infrared spectroscopy (rTMS/fNIRS) to measure brain activation and connectivity. NeuroImage. 2009;47:1177-1184. DOI: 10.1016/j.neuroimage.2009.05.016

[46] Zhang F, Aravanis AM, Adamantidis A, de Lecea L, Deisseroth K. Circuit-breakers: Optical technologies for probing neural signals and systems. Nature Reviews Neuroscience. 2007;8: 577-581. DOI: 10.1038/nrn2192

[47] Mohanty SK, Reinscheid RK, Liu X, Okamura N, Krasieva TB, Berns MW. In-depth activation of channelrhodopsin-2 sensitized excitable cells with high spatial resolution using two-photon excitation with a nearinfrared laser microbeam. Biophysical Journal. 2008;95:11. DOI: 10.1529/ biophysj.108.130187

[48] Okamoto M, Dan H, Shimizu K, Takeo K, Amita T, Oda I, et al. Multimodal assessment of cortical activation during apple peeling by NIRS and fMRI. NeuroImage. 2004;21: 1275-1288. DOI: 10.1016/j. neuroimage.2003.12.003

[49] Herff C, Heger D, Putze F, Guan C, Schultz T. Self-paced BCI with NIRS based on speech activity. In: Proceedings of the Fifth International Brain-Computer Interface Meeting 2013 (International BCI Meeting 2013); 3–7 June 2013; Asilomar Conference Center, Pacific Grove, California. Graz: Graz University of Technology Publishing House; 2013. Article ID: 111 DOI: 10.3217/978-3-85125-260-6-111

[50] Xu G, Xu LX, Li D, Liu X.  Computational and Mathematical Methods in Medicine. 2014;2014:107320. 9 p. DOI: 10.1155/ 2014/107320

[51] Almajidy RK, Le KS, Hofmanna UG. Novel near infrared sensors for hybrid BCI applications. In: European Conference on Biomedical Optics (ECBO 2015). Advanced Microscopy Techniques IV; and Neurophotonics II; 21–25 June 2015; Munich Germany. Washington: Optical Society of America, 2015. p. 95361H. DOI: 10.1364/ ECBO.2015.95361H

[52] Matthews F, Soraghan C, Ward T, Markham C, Pearlmutter B. Software platform for rapid prototyping of NIRS brain computer interfacing techniques. In: Annual International Conference of the IEEE Engineering in Medicine and Biology Society (IEEE Engineering in Medicine and Biology Society. Conference 2008); 20–24 August 2008; Vancouver, Canada: IEEE; 2008. p. 4840-4843. DOI: 10.1109/ IEMBS.2008.4650297

[53] Van Erp J, Lotte BF, Tangermann M. Brain-computer interfaces: Beyond medical applications. Computer. 2012; 45(4):26-34. DOI: 10.1109/MC.2012.107

[54] Ang KK, Guan C. Brain-computer interface in stroke rehabilitation. Computing Science and Engineering. 2013;7(2):139-146. DOI: 10.5626/ JCSE.2<IP_ADDRESS>

[55] Li WS. FOMs of consciousness measurement. In: International Conference on Artificial Intelligence and Industrial Engineering (AIIE 2015); 26–27 July 2015; Phuket. Thailand. Paris: Atlantis Press; p. 121-125. DOI: 10.2991/ aiie-15.2015.34

[56] Hwang HJ, Lim JH, Kim DW, Ima CH. Evaluation of various mental task combinations for near-infrared spectroscopy-based brain-computer interfaces. Biomedical Optics. 2014;19 (7). 07 Enhanced performance by a hybrid NIRS–EEG 7005 (July 2014). DOI: 10.1117/1.JBO.19.7.077005

[57] Naseer N, Hong KS. Classification of functional near-infrared spectroscopy signals corresponding to the right- and left-wrist motor imagery for development of a brain–computer interface. Neuroscience Letters. 2013; 553:84-89. DOI: 10.1016/j. neulet.2013.08.021

[58] Sitaram R, Zhang H, Guan C, Thulasidas M, Hoshi Y, Ishikawa A, et al. Temporal classification of multichannel near-infrared spectroscopy signals of motor imagery for developing a brain–computer interface. NeuroImage. 2007;34: 1416-1427. DOI: 10.1016/j. neuroimage.2006.11.00

[59] Tomita Y, Vialatte F-B, Dreyfus G, Mitsukura Y, Bakardjian H, Cichocki A. Bimodal BCI using simultaneously NIRS and EEG. IEEE Transactions on Biomedical Engineering. 2014;61(4):

1274-1284. DOI: 10.1109/ TBME.2014.2300492

[60] Tam ND, Zouridakis G. Optical imaging of motor cortical hemodynamic response to directional arm movements using near-infrared spectroscopy. International Journal of Biological Engineering. 2013;3(2):11-17. DOI: 10.5923/j.ijbe.20130302.01

functional near infrared spectroscopic measurement of brain activity. International Journal of Biological and Life Sciences. 2008;4:34-43. DOI:

DOI: http://dx.doi.org/10.5772/intechopen.83345

Near-Infrared Optical Technologies in Brain-Computer Interface Systems

study hemodynamic changes during activation of brain function in human adults. Neuroscience Letters. 1993;154: 101-104. DOI: 10.1016/0304-3940(93)

[73] Matsuda G, Hiraki K. Sustained decrease in oxygenated hemoglobin during video games in the dorsal prefrontal cortex: A NIRS study of children. NeuroImage. 2006;29:706-711. DOI: 10.1016/j.gturoimagt.2005.08.019

[74] Guirgis M, Falk TH, Power S, Blain S, Chau T. Harnessing physiological responses to improve NIRS-based braincomputer interface performance. In: ISSNIP Biosignals Biorobotics; 4–6 January 2010; Vitoria, Brazil, New York:

[75] Gupta CN, Palaniappan R. Using EEG and NIRS for rain-computer interface and cognitive performance measures: A pilot study. International Journal of Cognitive Performance Support. 2013;1(1):69. ISSN 1742-7207. DOI: 10.1504/ IJCPS.2013.053576

[76] Falk TH, Guirgis M, Power S, Chau TT. Taking NIRS-BCIs outside the lab: Towards achieving robustness against environment noise. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2011;19(2):136-146. DOI:

10.1109/TNSRE.2010.2078516

[Accessed: 2018-12-29]

[78] Khan MJ, Hong MJ, Hong KS. Decoding of four movement directions using hybrid NIRS-EEG brain-computer

[77] Khan MJ, Hong KS, Naseer N, Bhutta MR, Yoon SH. Hybrid EEG-NIRS BCI for rehabilitation using differentsource brain signals [Internet] 2014. Available from: https://www. researchgate.net/profile/Noman\_ Naseer/publication/271508614\_Hybrid\_ EEG-NIRS\_BCI\_for\_rehabilitation\_ using\_different\_brain\_signals/links/ 55876a8008aeb0cdade0bbe5.pdf

IEEE; 2010. p. 59-62

90181-J

[66] Fazli S, Mehnert J, Steinbrink J, Curio G, Villringer A, Muller K-R, et al. Enhanced performance by a hybrid NIRS–EEG brain computer interface. NeuroImage. 2012;59:519-529. DOI: 10.1016/j.neuroimage.2011.07.084

[67] Cui X, Bray S, Reiss AL. Speeded near infrared spectroscopy (NIRS) response detection. PLoS One. 2010; 5(11):e15474. DOI: 10.1371/ journal.

[68] Ward LM, Aitchison RT, Tawse M, Simmers AJ, Shahani U. Reduced haemodynamic response in the ageing visual cortex measured by absolute fNIRS. PLoS One. 2015;10(4):e0125012. DOI: 10.1371/journal.pone.01250122015

[69] Limongi T, Di Sante G, Ferrari M, Quaresima V. Detecting mental calculation related frontal cortex

oxygenation changes for brain computer interface using multi-channel functional near infrared topography. International Journal of Bioelectromagnetism. 2009;

[70] Ang KK, Yu J, Guan C. Extracting and selecting discriminative features from high density NIRS-based BCI for numerical cognition. In: International Joint Conference on Neural Networks (WCCI 2012); 10–15 June 2012;

Brisbane, Australia. IEEE; 2012. p. 1716- 1721. DOI: 10.1109/IJCNN.2012.6252604

[71] Fishburn FA, Norr ME, Medvedev AB, Vaidya CJ. Sensitivity off NIRS to cognitive state and load. Frontiers in Human Neuroscience. 2014;8:76. DOI:

10.3389/fnhum.2014.00076

29

[72] Villringer A, Planck J, Hock C, Schlenkofer L, Dirnagl U. Near infrared spectroscopy (NIRS): A new tool to

10.5281/zenodo.1084415

pone.0015474

11(2):86-90

[61] Volpe BT, Krebs HI, Hogan EL, Diels CM, Aisen ML. Robot training enhanced motor outcome in patients with stroke maintained over 3 years. Neurology. 1999;53(8):1874-1876

[62] Ferraro M, Palazzolo JJ, Krol J, Krebs HI, Hogan N, Volpe BT. Robotaided sensorimotor arm training improves outcome in patients with chronic stroke. Neurology. 2003;61(11): 1604-1607. DOI: 10.1212/01. WNL.0000095963.00970.68

[63] Colombo R, Pisano F, Micera S, Mazzone A, Delconte C, Carrozza MC, et al. Robotic techniques for upper limb evaluation and rehabilitation of stroke patients. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2005;13(3):311-324. DOI: 10.1109/ NNSRE.2005.848352

[64] Strait M, Scheutz M. Building a literal bridge between robotics and neuroscience using functional near infrared spectroscopy (NIRS) [Internet]. 1999. Available from: https:// www.researchgate.net/profile/ Matthias\_Scheutz/publication/ 262774032\_Building\_a\_literal\_bridge\_ between\_robotics\_and\_neuroscience\_ using\_functional\_near\_infrared\_ spectroscopy/links/ 55e436c908ae2fac47215330/Building-aliteral-bridge-between-robotics-andneuroscience-using-functional-nearinfrared-spectroscopy.pdf [Accessed: 2019-01-02]

[65] Soe NN, Nakagawa M. Chaotic properties of hemodynamic response in Near-Infrared Optical Technologies in Brain-Computer Interface Systems DOI: http://dx.doi.org/10.5772/intechopen.83345

functional near infrared spectroscopic measurement of brain activity. International Journal of Biological and Life Sciences. 2008;4:34-43. DOI: 10.5281/zenodo.1084415

[53] Van Erp J, Lotte BF, Tangermann M. Brain-computer interfaces: Beyond medical applications. Computer. 2012; 45(4):26-34. DOI: 10.1109/MC.2012.107

New Frontiers in Brain-Computer Interfaces

1274-1284. DOI: 10.1109/ TBME.2014.2300492

10.5923/j.ijbe.20130302.01

[60] Tam ND, Zouridakis G. Optical imaging of motor cortical hemodynamic response to directional arm movements using near-infrared spectroscopy. International Journal of Biological Engineering. 2013;3(2):11-17. DOI:

[61] Volpe BT, Krebs HI, Hogan EL, Diels CM, Aisen ML. Robot training enhanced motor outcome in patients with stroke maintained over 3 years. Neurology. 1999;53(8):1874-1876

[62] Ferraro M, Palazzolo JJ, Krol J, Krebs HI, Hogan N, Volpe BT. Robotaided sensorimotor arm training improves outcome in patients with chronic stroke. Neurology. 2003;61(11):

[63] Colombo R, Pisano F, Micera S, Mazzone A, Delconte C, Carrozza MC, et al. Robotic techniques for upper limb evaluation and rehabilitation of stroke patients. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2005;13(3):311-324. DOI: 10.1109/

[64] Strait M, Scheutz M. Building a literal bridge between robotics and neuroscience using functional near infrared spectroscopy (NIRS)

www.researchgate.net/profile/ Matthias\_Scheutz/publication/ 262774032\_Building\_a\_literal\_bridge\_ between\_robotics\_and\_neuroscience\_ using\_functional\_near\_infrared\_

[Internet]. 1999. Available from: https://

55e436c908ae2fac47215330/Building-aliteral-bridge-between-robotics-andneuroscience-using-functional-nearinfrared-spectroscopy.pdf [Accessed:

[65] Soe NN, Nakagawa M. Chaotic properties of hemodynamic response in

1604-1607. DOI: 10.1212/01. WNL.0000095963.00970.68

NNSRE.2005.848352

spectroscopy/links/

2019-01-02]

[54] Ang KK, Guan C. Brain-computer interface in stroke rehabilitation. Computing Science and Engineering. 2013;7(2):139-146. DOI: 10.5626/

[55] Li WS. FOMs of consciousness measurement. In: International Conference on Artificial Intelligence and Industrial Engineering (AIIE 2015); 26–27 July 2015; Phuket. Thailand. Paris: Atlantis Press; p. 121-125. DOI: 10.2991/

[56] Hwang HJ, Lim JH, Kim DW, Ima CH. Evaluation of various mental task combinations for near-infrared spectroscopy-based brain-computer interfaces. Biomedical Optics. 2014;19 (7). 07 Enhanced performance by a hybrid NIRS–EEG 7005 (July 2014). DOI: 10.1117/1.JBO.19.7.077005

[57] Naseer N, Hong KS. Classification of functional near-infrared spectroscopy signals corresponding to the right- and

left-wrist motor imagery for development of a brain–computer interface. Neuroscience Letters. 2013;

553:84-89. DOI: 10.1016/j. neulet.2013.08.021

[58] Sitaram R, Zhang H, Guan C, Thulasidas M, Hoshi Y, Ishikawa A, et al. Temporal classification of multichannel near-infrared

spectroscopy signals of motor imagery for developing a brain–computer interface. NeuroImage. 2007;34: 1416-1427. DOI: 10.1016/j. neuroimage.2006.11.00

[59] Tomita Y, Vialatte F-B, Dreyfus G, Mitsukura Y, Bakardjian H, Cichocki A. Bimodal BCI using simultaneously NIRS

and EEG.  NeuroImage. 2012;59:519-529. DOI: 10.1016/j.neuroimage.2011.07.084

[67] Cui X, Bray S, Reiss AL. Speeded near infrared spectroscopy (NIRS) response detection. PLoS One. 2010; 5(11):e15474. DOI: 10.1371/ journal. pone.0015474

[68] Ward LM, Aitchison RT, Tawse M, Simmers AJ, Shahani U. Reduced haemodynamic response in the ageing visual cortex measured by absolute fNIRS. PLoS One. 2015;10(4):e0125012. DOI: 10.1371/journal.pone.01250122015

[69] Limongi T, Di Sante G, Ferrari M, Quaresima V. Detecting mental calculation related frontal cortex oxygenation changes for brain computer interface using multi-channel functional near infrared topography. International Journal of Bioelectromagnetism. 2009; 11(2):86-90

[70] Ang KK, Yu J, Guan C. Extracting and selecting discriminative features from high density NIRS-based BCI for numerical cognition. In: International Joint Conference on Neural Networks (WCCI 2012); 10–15 June 2012; Brisbane, Australia. IEEE; 2012. p. 1716- 1721. DOI: 10.1109/IJCNN.2012.6252604

[71] Fishburn FA, Norr ME, Medvedev AB, Vaidya CJ. Sensitivity off NIRS to cognitive state and load. Frontiers in Human Neuroscience. 2014;8:76. DOI: 10.3389/fnhum.2014.00076

[72] Villringer A, Planck J, Hock C, Schlenkofer L, Dirnagl U. Near infrared spectroscopy (NIRS): A new tool to

study hemodynamic changes during activation of brain function in human adults. Neuroscience Letters. 1993;154: 101-104. DOI: 10.1016/0304-3940(93) 90181-J

[73] Matsuda G, Hiraki K. Sustained decrease in oxygenated hemoglobin during video games in the dorsal prefrontal cortex: A NIRS study of children. NeuroImage. 2006;29:706-711. DOI: 10.1016/j.gturoimagt.2005.08.019

[74] Guirgis M, Falk TH, Power S, Blain S, Chau T. Harnessing physiological responses to improve NIRS-based braincomputer interface performance. In: ISSNIP Biosignals Biorobotics; 4–6 January 2010; Vitoria, Brazil, New York: IEEE; 2010. p. 59-62

[75] Gupta CN, Palaniappan R. Using EEG and NIRS for rain-computer interface and cognitive performance measures: A pilot study. International Journal of Cognitive Performance Support. 2013;1(1):69. ISSN 1742-7207. DOI: 10.1504/ IJCPS.2013.053576

[76] Falk TH, Guirgis M, Power S, Chau TT. Taking NIRS-BCIs outside the lab: Towards achieving robustness against environment noise. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2011;19(2):136-146. DOI: 10.1109/TNSRE.2010.2078516

[77] Khan MJ, Hong KS, Naseer N, Bhutta MR, Yoon SH. Hybrid EEG-NIRS BCI for rehabilitation using differentsource brain signals [Internet] 2014. Available from: https://www. researchgate.net/profile/Noman\_ Naseer/publication/271508614\_Hybrid\_ EEG-NIRS\_BCI\_for\_rehabilitation\_ using\_different\_brain\_signals/links/ 55876a8008aeb0cdade0bbe5.pdf [Accessed: 2018-12-29]

[78] Khan MJ, Hong MJ, Hong KS. Decoding of four movement directions using hybrid NIRS-EEG brain-computer interface. Frontiers in Human Neuroscience. 2014;8:244. DOI: 10.3389/fnhum.2014.00244

[79] Khan MJ, Hong KH, Naseer N, Bhutta MR. Multi-decision detection using EEG-NIRS based hybrid braincomputer interface (BCI). In: 20th Annual Meeting of the Organization for Human Brain Mapping (OHMB 2014); 8–13 June 2014; Hamburg, Germany. AI Attendee Interactive; 2014. p. 1-5

[80] Koo B, Lee HG, Nam Y, Kang H, Kohd CS, Shin HC, et al. A hybrid NIRS-EEG system for self-paced brain computer interface with online motor imagery. Journal of Neuroscience Methods. 2015;244:26-32. DOI: 10.1016/ j.jneumeth.2014.04.016

[81] Pfurtscheller G, Allison BZ, Brunner C, Bauernfeind G, Solis-Escalante T, Scherer R, et al. The hybrid BCI. Frontiers in Neuroscience. 2010;4:30. DOI: 10.3389/fnpro.2010.00003

[82] Lakshmi R, Prasad TV, Prakash VC. Survey on EEG signal processing methods. International Journal of Advanced Research in Computer Science and Software Engineering. 2014;4(1):84-91

[83] Tam ND, Zouridakis G. Temporal decoupling of oxy- and deoxyhemoglobin hemodynamic responses detected by functional near-infrared spectroscopy (fNIRS). Journal of Biomedical Engineering and Medical Imaging. 2014;1(2):18-28. DOI: 10.14738/jbemi.12.146

[84] Biswal BB, VanKylen J, Hyde JS. Simultaneous assessment of flow and BOLD signals in resting-statefunctional connectivity maps. NMR in Biomedicine. 1997;10(4–5):165-170.  PLoS One. 2012;7(9): e45771. DOI: 10.1371/journal. pone.0045771

Neuroscience Methods. 2012;209:

[92] Ehrlicher AJ, Betz T, Stuhrmann B, Koch D, Milner V, Raizen MG, et al. Guiding neuronal growth with light. PNAS. 2002;99(25):16024-16028. DOI:

DOI: http://dx.doi.org/10.5772/intechopen.83345

Near-Infrared Optical Technologies in Brain-Computer Interface Systems

[93] Ilina IV, Ovchinnikov AV, Sitnikov DS, Chefonov OV, Agranat MB, Khramova YV, et al. Microsurgery of cell membrane with femtosecond laser pulses for cell fusion and optical injection. In: International Conference Advanced Laser Technologies (ALT '12); 2–6 September 2012. Vol. 1.  Improvements in the optical recording of neuron activity using voltagedependent dyes. Neuroscience and Behavioral Physiology. 2015;45(2): 132-139. DOI: 10.1007/s11055-015-

[95] Birbaumer N, Gallegos-Ayala G, Wildgruber M, Silvoni S, Soekadar SR. Direct brain control and communication in paralysis. Brain Topography. 2014; 27(1):4-11. DOI: 10.1007/s10548-013-

168-177. DOI: 10.1016/j. jneumeth.2012.02.006

10.1073/pnas.252631899

2012. DOI: 10.12684/alt.1.61

0050-7

0282-1

31

[86] Barbour RL, Graber HL, Xu Y, Pei Y, Schmitz CH, Pfeil DS, et al. A programmable laboratory testbed in support of evaluation of functional brain activation and connectivity. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2012;20(2): 170-183. DOI: 10.1109/ TNSRE.2012.2185514

[87] Pei Y, Wang Z, Barbour RL. NAVI: A problem solving environment (PSE) for NIRS data analysis [Internet]. 2006. Available from: https://www. researchgate.net/profile/Yaling\_Pei/ publication/228412067\_NAVI\_A\_ problem\_solving\_environment\_PSE\_ for\_NIRS\_data\_analysis/links/<PHONE_NUMBER>ae7921246e787b/NAVI-Aproblem-solving-environment-PSE-for-NIRS-data-analysis.pdf [Accessed: 2018-12-29]

[88] NIRx fNIRS Analysis Environment user's Guide. Available from: http://otg. downstate.edu/Publication/NIRxPac kage\_02.pdf

[89] Li Z, Liu H, Liao X, Xu J, Liu W, Tian F, et al. Dynamic functional connectivity revealed by resting-state functional near-infrared spectroscopy. Biomedical Optics Express. 2015;6(7): 2337-2352. DOI: 10.1364/BOE.6.002337

[90] Mehnert J, Akhrif A, Telkemeyer S, Rossi S, Schmitz CH, Steinbrink J, et al. Developmental changes in brain activation and functional connectivity during response inhibition in the early childhood brain. Brain and Development. 2013;35:894-904. DOI: 10.1016/j.braindev.2012.11.006

[91] Ebbesen CL, Bruus H. Analysis of laser-induced heating in optical neuronal guidance. Journal of

Near-Infrared Optical Technologies in Brain-Computer Interface Systems DOI: http://dx.doi.org/10.5772/intechopen.83345

Neuroscience Methods. 2012;209: 168-177. DOI: 10.1016/j. jneumeth.2012.02.006

interface. Frontiers in Human Neuroscience. 2014;8:244. DOI: 10.3389/fnhum.2014.00244

[79] Khan MJ, Hong KH, Naseer N, Bhutta MR. Multi-decision detection using EEG-NIRS based hybrid braincomputer interface (BCI). In: 20th Annual Meeting of the Organization for Human Brain Mapping (OHMB 2014); 8–13 June 2014; Hamburg, Germany. AI Attendee Interactive; 2014. p. 1-5

New Frontiers in Brain-Computer Interfaces

human brain functional networks with resting-state functional near infrared spectroscopy. PLoS One. 2012;7(9): e45771. DOI: 10.1371/journal.

[86] Barbour RL, Graber HL, Xu Y, Pei Y, Schmitz CH, Pfeil DS, et al. A programmable laboratory testbed in support of evaluation of functional brain activation and connectivity. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2012;20(2):

[87] Pei Y, Wang Z, Barbour RL. NAVI: A problem solving environment (PSE) for NIRS data analysis [Internet]. 2006.

5592819608ae7921246e787b/NAVI-Aproblem-solving-environment-PSE-for-NIRS-data-analysis.pdf [Accessed:

[88] NIRx fNIRS Analysis Environment user's Guide. Available from: http://otg. downstate.edu/Publication/NIRxPac

[89] Li Z, Liu H, Liao X, Xu J, Liu W, Tian F, et al. Dynamic functional connectivity revealed by resting-state functional near-infrared spectroscopy. Biomedical Optics Express. 2015;6(7): 2337-2352. DOI: 10.1364/BOE.6.002337

[90] Mehnert J, Akhrif A, Telkemeyer S, Rossi S, Schmitz CH, Steinbrink J, et al. Developmental changes in brain activation and functional connectivity during response inhibition in the early

Development. 2013;35:894-904. DOI: 10.1016/j.braindev.2012.11.006

[91] Ebbesen CL, Bruus H. Analysis of laser-induced heating in optical neuronal guidance. Journal of

childhood brain. Brain and

pone.0045771

170-183. DOI: 10.1109/ TNSRE.2012.2185514

Available from: https://www. researchgate.net/profile/Yaling\_Pei/ publication/228412067\_NAVI\_A\_ problem\_solving\_environment\_PSE\_ for\_NIRS\_data\_analysis/links/

2018-12-29]

kage\_02.pdf

[80] Koo B, Lee HG, Nam Y, Kang H, Kohd CS, Shin HC, et al. A hybrid NIRS-

[81] Pfurtscheller G, Allison BZ, Brunner C, Bauernfeind G, Solis-Escalante T, Scherer R, et al. The hybrid BCI. Frontiers in Neuroscience. 2010;4:30. DOI: 10.3389/fnpro.2010.00003

[82] Lakshmi R, Prasad TV, Prakash VC. Survey on EEG signal processing methods. International Journal of Advanced Research in Computer Science and Software Engineering.

[83] Tam ND, Zouridakis G. Temporal decoupling of oxy- and deoxyhemoglobin hemodynamic responses detected by functional near-infrared spectroscopy (fNIRS). Journal of Biomedical Engineering and Medical Imaging. 2014;1(2):18-28. DOI:

[84] Biswal BB, VanKylen J, Hyde JS. Simultaneous assessment of flow and BOLD signals in resting-statefunctional

Biomedicine. 1997;10(4–5):165-170. DOI: 10.1002/(sici)1099-1492(199706/ 08)10:4/5<165::aidnbm454>3.0.co;2-7

[85] Niu H, Wang J, Zhao T, Shu N, He Y. Revealing topological organization of

EEG system for self-paced brain computer interface with online motor imagery. Journal of Neuroscience Methods. 2015;244:26-32. DOI: 10.1016/

j.jneumeth.2014.04.016

2014;4(1):84-91

10.14738/jbemi.12.146

connectivity maps. NMR in

30

[92] Ehrlicher AJ, Betz T, Stuhrmann B, Koch D, Milner V, Raizen MG, et al. Guiding neuronal growth with light. PNAS. 2002;99(25):16024-16028. DOI: 10.1073/pnas.252631899

[93] Ilina IV, Ovchinnikov AV, Sitnikov DS, Chefonov OV, Agranat MB, Khramova YV, et al. Microsurgery of cell membrane with femtosecond laser pulses for cell fusion and optical injection. In: International Conference Advanced Laser Technologies (ALT '12); 2–6 September 2012. Vol. 1.  DOI: 10.12684/alt.1.61

[94] Nikitin ES, Aseev NA, Balaban PM. Improvements in the optical recording of neuron activity using voltagedependent dyes. Neuroscience and Behavioral Physiology. 2015;45(2): 132-139. DOI: 10.1007/s11055-015- 0050-7

[95] Birbaumer N, Gallegos-Ayala G, Wildgruber M, Silvoni S, Soekadar SR. Direct brain control and communication in paralysis. Brain Topography. 2014; 27(1):4-11. DOI: 10.1007/s10548-013- 0282-1

Chapter 3

Abstract

Speech Enhancement Using

an Iterative Posterior NMF

Over the years, miscellaneous methods for speech enhancement have been proposed, such as spectral subtraction (SS) and minimum mean square error (MMSE) estimators. These methods do not require any prior knowledge about the speech and noise signals nor any training stage beforehand, so they are highly flexible and allow implementation in various situations. However, these algorithms usually assume that the noise is stationary and are thus not good at dealing with

nonstationary noise types, especially under low signal-to-noise (SNR) conditions. To overcome the drawbacks of the above methods, nonnegative matrix factorization (NMF) is introduced. NMF approach is more robust to nonstationary noise. In this chapter, we are actually interested in the application of speech enhancement using NMF approach. A speech enhancement method based on regularized nonnegative matrix factorization (NMF) for nonstationary Gaussian noise is proposed. The spectral components of speech and noise are modeled as Gamma and Rayleigh, respectively. We propose to adaptively estimate the sufficient statistics of these

distributions to obtain a natural regularization of the NMF criterion.

Keywords: nonnegative matrix factorization (NMF), speech enhancement, signal-to-noise ratio (SNR), expectation maximization (EM) algorithms,

Over the past several decades, there has been a large research interest in the problem of single-channel sound source separation. Such work focuses on the task of separating a single mixture recording into its respective sources and is motivated by the fact that real-world sounds are inherently constructed by many individual sounds (e.g., human speakers, musical instruments, background noise, etc.). While source separation is difficult, the topic is highly motivated by many outstanding problems in audio signal processing and machine learning, including the following:

1. Speech denoising and enhancement—the task of removing background noise (e.g., wind, babble, etc.) from recorded speech and improving speech intelligibility for human listeners and/or automatic speech recognizers

processing audio based on semantic properties of the recording such as tempo,

2. Content-based analysis and processing—the task of extracting and/or

Sunnydayal Vanambathina

posterior regularization (PR)

rhythm, and/or pitch

33

1. Introduction

## Chapter 3

## Speech Enhancement Using an Iterative Posterior NMF

Sunnydayal Vanambathina

## Abstract

Over the years, miscellaneous methods for speech enhancement have been proposed, such as spectral subtraction (SS) and minimum mean square error (MMSE) estimators. These methods do not require any prior knowledge about the speech and noise signals nor any training stage beforehand, so they are highly flexible and allow implementation in various situations. However, these algorithms usually assume that the noise is stationary and are thus not good at dealing with nonstationary noise types, especially under low signal-to-noise (SNR) conditions. To overcome the drawbacks of the above methods, nonnegative matrix factorization (NMF) is introduced. NMF approach is more robust to nonstationary noise. In this chapter, we are actually interested in the application of speech enhancement using NMF approach. A speech enhancement method based on regularized nonnegative matrix factorization (NMF) for nonstationary Gaussian noise is proposed. The spectral components of speech and noise are modeled as Gamma and Rayleigh, respectively. We propose to adaptively estimate the sufficient statistics of these distributions to obtain a natural regularization of the NMF criterion.

Keywords: nonnegative matrix factorization (NMF), speech enhancement, signal-to-noise ratio (SNR), expectation maximization (EM) algorithms, posterior regularization (PR)

## 1. Introduction

Over the past several decades, there has been a large research interest in the problem of single-channel sound source separation. Such work focuses on the task of separating a single mixture recording into its respective sources and is motivated by the fact that real-world sounds are inherently constructed by many individual sounds (e.g., human speakers, musical instruments, background noise, etc.). While source separation is difficult, the topic is highly motivated by many outstanding problems in audio signal processing and machine learning, including the following:

## 2. Nonnegative matrix factorization

## 2.1 NMF model

Nonnegative matrix factorization is a process that approximates a single nonnegative matrix as the product of two nonnegative matrices. It is defined by

$$V \approx \text{WH} \tag{1}$$

This process can be seen in Figure 1 [3], where a two-measure piano passage of "Mary Had a Little Lamb" is shown alongside a spectrogram and an NMF factorization. Notice how W captures the harmonic content of the three pitches of the passage and H captures the time onsets and gains of the individual notes. Also note that Nz is typically chosen manually or using a model selection procedure such as cross-validation and Nf and Nt are a function of the overall recording length and

This leads to two related interpretations of how NMF models spectrogram data. The first interpretation is that the columns of V (i.e., short-time segments of the mixture signal) are approximated as a weighted sum of basis vectors as shown in

<sup>j</sup>¼<sup>1</sup>Hj1Wj <sup>∑</sup><sup>K</sup>

The second interpretation is that the entire matrix V is approximated as a sum of

NMF of a piano performing "Mary had a little lamb" for two measures with Nz = 3. Notice how matrix W captures the harmonic content of the three pitches of the passage and matrix H captures the time onsets and gains

NMF interpretation I. the columns of V (i.e., short-time segments of the mixture signal) are approximated as a

<sup>j</sup>¼<sup>1</sup>Hj2Wj <sup>∑</sup><sup>K</sup>

h i (2)

<sup>j</sup>¼<sup>1</sup>HjNtWj

STFT parameters (transform length, zero-padding size, and hop size).

3 7 5≈ ∑Nz

Figure 2 and Eq. (2):

2 6 4

j jj j V1 V2 V3……VNt j jj j

Speech Enhancement Using an Iterative Posterior NMF DOI: http://dx.doi.org/10.5772/intechopen.84976

matrix "layers," as shown in Figure 3 and Eq. (3).

V ≈

Figure 1.

Figure 2.

35

weighted sum or mixture of basis vectors W [3].

of the individual notes [3].

V ∈ RNf �Nt <sup>þ</sup> is a nonnegative input matrix. <sup>W</sup> <sup>∈</sup> <sup>R</sup>Nf �Nz <sup>þ</sup> is a matrix of basis vectors, basis functions, or dictionary elements; H ∈ RNz�Nt <sup>þ</sup> is a matrix of corresponding activations, weights, or gains; and Nf is the number of rows of the input matrix. Nt is the number of columns of the input matrix; Nz is the number of basis vectors [1].

V ∈ RNf �Nt <sup>þ</sup> —original nonnegative input data matrix

W ∈ RNf �Nz <sup>þ</sup> —matrix of basis vectors, basis functions, or dictionary elements.

H ∈ RNz�Nt <sup>þ</sup> —matrix of activations, weights, encodings, or gains.

When used for audio applications, NMF is typically used to model spectrogram data or the magnitude of STFT data [2]. That is, we take a single-channel recording, transform it into the time-frequency domain using the STFT, take the magnitude or power V, and then approximate the result as V ≈WH. In doing so, NMF approximates spectrogram data as a linear combination of prototypical frequencies or spectra (i.e., basis vectors) over time.

Speech Enhancement Using an Iterative Posterior NMF DOI: http://dx.doi.org/10.5772/intechopen.84976

This process can be seen in Figure 1 [3], where a two-measure piano passage of "Mary Had a Little Lamb" is shown alongside a spectrogram and an NMF factorization. Notice how W captures the harmonic content of the three pitches of the passage and H captures the time onsets and gains of the individual notes. Also note that Nz is typically chosen manually or using a model selection procedure such as cross-validation and Nf and Nt are a function of the overall recording length and STFT parameters (transform length, zero-padding size, and hop size).

This leads to two related interpretations of how NMF models spectrogram data. The first interpretation is that the columns of V (i.e., short-time segments of the mixture signal) are approximated as a weighted sum of basis vectors as shown in Figure 2 and Eq. (2):

$$\mathbf{V} \approx \begin{bmatrix} | & | & | & | \\ \mathbf{V1} & \mathbf{V2} & \mathbf{V3} & \dots & \mathbf{V\_{N\_i}} \\ | & | & | & | \end{bmatrix} \approx \begin{bmatrix} \sum\_{j=1}^{N\_i} H\_{j1} \mathbf{W\_j} & \sum\_{j=1}^{K} H\_{j2} \mathbf{W\_j} & \sum\_{j=1}^{K} H\_{jN\_i} \mathbf{W\_j} \end{bmatrix} \tag{2}$$

The second interpretation is that the entire matrix V is approximated as a sum of matrix "layers," as shown in Figure 3 and Eq. (3).

### Figure 1.

3.Music transcription—the task of notating an audio recording into a musical representation such as a musical score, guitar tablature, or other symbolic

4.Audio-based forensics—the task of examining, comparing, and evaluating

5. Audio restoration—the task of removing imperfections such as noise, hiss,

6.Music remixing and content creation—the task of creating a new musical work by manipulating the content of one or more previously existing recordings

Nonnegative matrix factorization is a process that approximates a single non-

corresponding activations, weights, or gains; and Nf is the number of rows of the input matrix. Nt is the number of columns of the input matrix; Nz is the number of

<sup>þ</sup> —matrix of basis vectors, basis functions, or dictionary elements.

• A column represents a basis vector, basis function, or dictionary element.

• Each column is not orthonormal, but commonly normalized to one.

<sup>þ</sup> —matrix of activations, weights, encodings, or gains.

When used for audio applications, NMF is typically used to model spectrogram data or the magnitude of STFT data [2]. That is, we take a single-channel recording, transform it into the time-frequency domain using the STFT, take the magnitude or power V, and then approximate the result as V ≈WH. In doing so, NMF approximates spectrogram data as a linear combination of prototypical frequencies or

• A row represents the gain of a corresponding basis vector.

• Each row is not orthonormal, but sometimes normalized to one.

V ≈WH (1)

<sup>þ</sup> is a matrix of basis

<sup>þ</sup> is a matrix of

negative matrix as the product of two nonnegative matrices. It is defined by

<sup>þ</sup> is a nonnegative input matrix. <sup>W</sup> <sup>∈</sup> <sup>R</sup>Nf �Nz

<sup>þ</sup> —original nonnegative input data matrix

• Each column is an Nf -dimensional data sample.

• Each row represents a data feature.

spectra (i.e., basis vectors) over time.

vectors, basis functions, or dictionary elements; H ∈ RNz�Nt

audio recordings for scientific and/or legal matters

2. Nonnegative matrix factorization

New Frontiers in Brain-Computer Interfaces

pops, and crackles from (typically old) audio recordings

notations

2.1 NMF model

V ∈ RNf �Nt

basis vectors [1]. V ∈ RNf �Nt

W ∈ RNf �Nz

H ∈ RNz�Nt

34

NMF of a piano performing "Mary had a little lamb" for two measures with Nz = 3. Notice how matrix W captures the harmonic content of the three pitches of the passage and matrix H captures the time onsets and gains of the individual notes [3].

### Figure 2.

NMF interpretation I. the columns of V (i.e., short-time segments of the mixture signal) are approximated as a weighted sum or mixture of basis vectors W [3].

Figure 3. NMF interpretation II. The matrix V (i.e., the mixture signal) is approximated as a sum of matrix "layers" [3].

$$\mathbf{V} \approx \begin{bmatrix} | & | & | & | \\ \mathbf{V1} & \mathbf{V2} & \mathbf{V3} & \dots \mathbf{V\_{N\_t}} \\ | & | & | & | \end{bmatrix} \approx \begin{bmatrix} | & | & | & | \\ \mathbf{W1} & \mathbf{W2} & \mathbf{W3} & \dots \mathbf{W\_{N\_x}} \\ | & | & | & | \end{bmatrix} \begin{bmatrix} h\_1^T \\ h\_2^T \\ h\_3^T \\ \vdots \\ h\_{N\_t}^T \end{bmatrix}$$

$$\mathbf{V} \approx \mathbf{W\_1} h\_1^T + \mathbf{W\_2} h\_2^T + \mathbf{W\_3} h\_3^T + \dots + \mathbf{W\_{N\_x}} h\_{N\_x}^T \tag{3}$$

The application of NMF on noisy speech can be seen in Figure 4.

## 2.2 Optimization formulation

To estimate the basis matrix W and the activation matrix H for a given input data matrix V, NMF algorithm is formulated as an optimization problem. This is written as:

$$\underset{\mathbf{W},\mathbf{H}}{\operatorname{argmin}} \, D(\mathbf{V}|\mathbf{W}\mathbf{H})$$

$$\boldsymbol{W} \ge \mathbf{0}, \boldsymbol{H} \ge \mathbf{0} \tag{4}$$

where ½�ft indicates its argument at row f and column t and D Vð Þ jWH is a scalar

Popular cost functions include the Euclidean distance metric, Kullback-Liebler (KL) divergence, and Itakura-Saito (IS) divergence. Both the KL and IS divergences have been found to be well suited for audio purposes. In this work, we focus on the

where ½�ft indicates its argument at row f and column t and dqp ð Þ j is a scalar cost

 <sup>þ</sup> ½ � WH ft

<sup>þ</sup> const

W ≥0, H ≥ 0 (7)

<sup>p</sup> � <sup>q</sup> <sup>þ</sup> <sup>p</sup> (6)

case where d qp ð Þ j is generalized (non-normalized) KL divergence:

This results in the following optimization formulation:

∑ Nf

f¼1 ∑ Nt t¼1

dKLð Þ¼ q p<sup>j</sup> <sup>q</sup> ln <sup>q</sup>

�Vft ln WH ½ �ft

Given this formulation, we notice that the problem is not convex in W and H, limiting our ability to find a globally optimal solution to Eq. (7). It is, however, biconvex or independently convex in W for a fixed value of H and convex in H for a fixed value of W, motivating the use of iterative numerical methods to estimate

To solve Eq. (7), we must use an iterative numerical optimization technique and

The most popular alternative that has been proposed is by Lee and Seung [1, 5] and consists of a fast, simple, and efficient multiplicative gradient descent-based optimization procedure. The method works by breaking down the larger optimization problem into two subproblems and iteratively optimizes over W and then H, back and forth, given an initial feasible solution. The approach monotonically decreases the optimization objective for both the KL divergence and Euclidean cost

The approach is justified using the machinery of majorization-minimization (MM) algorithms [6]. MM algorithms are closely related to expectation maximization (EM) algorithms. In general, MM algorithms operate by approximating an optimization objective with a lower bound auxiliary function. The lower bound is then maximized instead of the original function, which is usually more difficult to

Algorithm 1 shows the complete iterative numerical optimization procedure applied to Eq. (7) with the KL divergence, where the division is element-wise, ⊗ is an element-wise multiplication, and 1 is a vector or matrix of ones with appropri-

hope to find a locally optimal solution. Gradient descent methods are the most common and straightforward for this purpose but typically are slow to converge. Other methods such as Newton's method, interior-point methods, conjugate gradient methods, and similar [4] can converge faster but are typically much more

complicated to implement, motivating alternative approaches.

functions and converges to a local stationary point.

cost function measured between V and WH.

Speech Enhancement Using an Iterative Posterior NMF DOI: http://dx.doi.org/10.5772/intechopen.84976

function measured between q and p.

locally optimal values of W and H.

2.3 Parameter estimation

optimize.

37

ately defined dimensions [5].

Subject to

argmin <sup>W</sup>, <sup>H</sup>

where D Vð Þ jWH is an appropriately defined cost function between V and W H and the inequalities ≥ are element-wise. It is also common to add additional equality constraints to require the columns of W to sum to one, which we enforce. When D Vð Þ jWH is additively separable, the cost function can be reduced to

$$D(\mathbf{V}|\mathbf{W}\mathbf{H}) = \sum\_{f=1}^{N\_f} \sum\_{t=1}^{N\_t} d\left(\mathbf{V}\_{ft} \Big| \mathbf{[W}\mathbf{H}]\_{ft}\right) \tag{5}$$

Figure 4. Applying NMF on noisy speech.

Speech Enhancement Using an Iterative Posterior NMF DOI: http://dx.doi.org/10.5772/intechopen.84976

where ½�ft indicates its argument at row f and column t and D Vð Þ jWH is a scalar cost function measured between V and WH.

Popular cost functions include the Euclidean distance metric, Kullback-Liebler (KL) divergence, and Itakura-Saito (IS) divergence. Both the KL and IS divergences have been found to be well suited for audio purposes. In this work, we focus on the case where d qp ð Þ j is generalized (non-normalized) KL divergence:

$$d\_{\rm KL}(q|p) = q \ln \frac{q}{p} - q + p \tag{6}$$

where ½�ft indicates its argument at row f and column t and dqp ð Þ j is a scalar cost function measured between q and p.

This results in the following optimization formulation:

$$\underset{\text{Wg }\mathbf{H}}{\operatorname{argmin}} \sum\_{f=1}^{N\_f} \sum\_{t=1}^{N\_t} -\mathbf{V}\_{ft} \ln \left| [\mathbf{W}\mathbf{H}]\_{ft} + \left| [\mathbf{W}\mathbf{H}]\_{ft} + \text{const} \right| \right| $$

Subject to

V ≈

Figure 3.

2 6 4

New Frontiers in Brain-Computer Interfaces

2.2 Optimization formulation

written as:

Figure 4.

36

Applying NMF on noisy speech.

j jj j V1 V2 V3……VNt j jj j

V ≈W1h<sup>T</sup>

3 7 5≈

<sup>1</sup> <sup>þ</sup> <sup>W</sup>2hT

The application of NMF on noisy speech can be seen in Figure 4.

argmin <sup>W</sup>, <sup>H</sup>

and the inequalities ≥ are element-wise. It is also common to add additional equality constraints to require the columns of W to sum to one, which we enforce. When D Vð Þ jWH is additively separable, the cost function can be reduced to

Nf

f¼1 ∑ Nt t¼1

D Vð Þ¼ jWH ∑

2 6 4

NMF interpretation II. The matrix V (i.e., the mixture signal) is approximated as a sum of matrix "layers" [3].

<sup>2</sup> <sup>þ</sup> <sup>W</sup>3h<sup>T</sup>

To estimate the basis matrix W and the activation matrix H for a given input data matrix V, NMF algorithm is formulated as an optimization problem. This is

D Vð Þ jWH

where D Vð Þ jWH is an appropriately defined cost function between V and W H

d Vft ½ � WH ft � � � � �

j jj j W1 W2 W3……WNz j jj j

<sup>3</sup> <sup>þ</sup> ::… <sup>þ</sup> WNzhT

W ≥0, H ≥ 0 (4)

3 7 5

hT 1 hT 2 hT 3 : hT Nz

Nz (3)

(5)

$$W \ge 0, H \ge 0 \tag{7}$$

Given this formulation, we notice that the problem is not convex in W and H, limiting our ability to find a globally optimal solution to Eq. (7). It is, however, biconvex or independently convex in W for a fixed value of H and convex in H for a fixed value of W, motivating the use of iterative numerical methods to estimate locally optimal values of W and H.

## 2.3 Parameter estimation

To solve Eq. (7), we must use an iterative numerical optimization technique and hope to find a locally optimal solution. Gradient descent methods are the most common and straightforward for this purpose but typically are slow to converge. Other methods such as Newton's method, interior-point methods, conjugate gradient methods, and similar [4] can converge faster but are typically much more complicated to implement, motivating alternative approaches.

The most popular alternative that has been proposed is by Lee and Seung [1, 5] and consists of a fast, simple, and efficient multiplicative gradient descent-based optimization procedure. The method works by breaking down the larger optimization problem into two subproblems and iteratively optimizes over W and then H, back and forth, given an initial feasible solution. The approach monotonically decreases the optimization objective for both the KL divergence and Euclidean cost functions and converges to a local stationary point.

The approach is justified using the machinery of majorization-minimization (MM) algorithms [6]. MM algorithms are closely related to expectation maximization (EM) algorithms. In general, MM algorithms operate by approximating an optimization objective with a lower bound auxiliary function. The lower bound is then maximized instead of the original function, which is usually more difficult to optimize.

Algorithm 1 shows the complete iterative numerical optimization procedure applied to Eq. (7) with the KL divergence, where the division is element-wise, ⊗ is an element-wise multiplication, and 1 is a vector or matrix of ones with appropriately defined dimensions [5].

## Algorithm 1 KL-NMF parameter estimation

Procedure KL-NMF (V ∈ RNf �Nt <sup>þ</sup> //input data matrix. Nz//number of basic vectors. ) Initialize: W ∈ RNf �Nz <sup>þ</sup> , <sup>H</sup> <sup>∈</sup> <sup>R</sup>Nz�Nt: þ repeat Optimize over W W W ⊗ V WH <sup>H</sup><sup>T</sup> <sup>1</sup>H<sup>T</sup> (8)

Optimize over H

$$H \leftarrow H \otimes \frac{\boldsymbol{W}^T \left( \frac{\boldsymbol{V}}{\boldsymbol{\text{WH}}} \right)}{\boldsymbol{W}^T \mathbf{1}} \tag{9}$$

to p fz ð Þ j , p tz ð Þ j , and/or p zð Þ, would encourage or discourage one source or another to explain a given time-frequency point. We can see this more clearly when we

where X f ð Þ ; t is the observed data that we model as the product of three distinct functions or factors φð Þz , φð Þ f; z , and φð Þ t; z . Note, each factor has different input arguments and each factor has different parameters that we wish to estimate via EM. Also, forget for the moment that the factors must be normalized probabilities. Given this model, if we wish to incorporate additional information, we could

This way of manipulation allows us to maintain our factorized form and can be thought of as prior-based regularization. If we would like to incorporate additional information/regularization that is a function of all three variables z, f, and t, then we must do something else. The first option would be to try to simultaneously modify all factors together to impose regularization that is a function of all three variables. This is unfortunately very difficult—both conceptually difficult to construct and

This motivates the use of posterior regularization (PR). PR provides us with an algorithmic mechanism via EM to incorporate constraints that are complementary to prior-based regularization. Instead of modifying the individual factors of our model as we saw before, we directly modify the posterior distribution of our model. The posterior distribution of our model, very loosely speaking, is a function of all random variables of our model. It is natively computed within each E step of EM and is required to iteratively improve the estimates of our model parameters. In this example, the posterior distribution would be akin to φð Þ z; f; t , which is a function of t, f, and z, as required. We now formally discuss PR below, beginning

The framework of posterior regularization, first introduced by Graca, Ganchev,

In this case, what we do is constrain the distribution q in some way when we maximize the auxiliary bound F qð Þ ; Θ with respect to q in the expectation step of an

and Taskar [9, 10], is a relatively new mechanism for injecting rich, typically data-dependent constraints into latent variable models using the EM algorithm. In contrast to standard Bayesian prior-based regularization, which applies constraints to the model parameters of a latent variable model in the maximization step of EM, posterior regularization applies constraints to the posterior distribution (distribution over the latent variables, conditioned on everything else) computation in the expectation step of EM. The method has found success in many natural language processing tasks, such as statistical word alignment, part-of-speech tagging, and

with a general discussion and concluding with the specific form of PR we

X f ð Þ ; t ≈φð Þz φð Þ f; z φð Þ t; z (11)

consider PLCA to be the following simplified estimation problem:

Speech Enhancement Using an Iterative Posterior NMF DOI: http://dx.doi.org/10.5772/intechopen.84976

• φð Þz to incorporate past knowledge of the variable z

practically difficult to algorithmically solve.

similar tasks that involve latent variable models.

employ within our approach.

3.1 Posterior regularization

EM algorithm, resulting in

39

• φð Þ f; z to incorporate past knowledge of the variable f and z

• φð Þ t; z to incorporate past knowledge of the variable t and z

independently modify:

until convergence return: W and H

NMF is an optimization technique using EM algorithm in terms of matrix, whereas probabilistic latent component analysis (PLCA) is also an optimization technique using EM algorithm in terms of probability. In PLCA, we are going to incorporate probabilities of time and frequency. In the next section, the development of PLCA-based algorithm to incorporate time-frequency constraints is discussed.

## 3. A probabilistic latent variable model with time-frequency constraints

Considering this approach, we now develop a new PLCA-based algorithm to incorporate the time-frequency user-annotations. For clarity, we restate the form of the symmetric two-dimensional PLCA model we use:

$$p(f, t) = \sum\_{x} p(z)p(f|z)p(t|z) \tag{10}$$

Compared to a modified NMF formulation, incorporating optimization constraints as a function of time, frequency, and sound source into the factorized PLCA model is particularly interesting and motivating to our focus.

Incorporating prior information into this model, and PLCA in general, can be done in several ways. The most commonly used methods are by direct observations (i.e., setting probabilities to zero, one, etc.) or by incorporating Bayesian prior probabilities on model parameters. Direct observations do not give us enough control, so we consider incorporating Bayesian prior probabilities. For the case of Eq. (10), this would result in independently modifying the factor terms p fz ð Þ j , p tz ð Þ j , or p zð Þ. Common prior probability distributions used for this purpose include Dirichlet priors [7], gamma priors [8], and others.

Given that we would like to incorporate the user-annotations as a function of time, frequency, and sound source, however, we notice that this is not easily accomplished using standard priors. This is because the model is factorized, and each factor is only a function of one variable and (possibly) conditioned by another, making it difficult to construct a set of prior probabilities that, when jointly applied Algorithm 1 KL-NMF parameter estimation

<sup>þ</sup> , <sup>H</sup> <sup>∈</sup> <sup>R</sup>Nz�Nt: þ

W W ⊗

H H ⊗

NMF is an optimization technique using EM algorithm in terms of matrix, whereas probabilistic latent component analysis (PLCA) is also an optimization technique using EM algorithm in terms of probability. In PLCA, we are going to incorporate probabilities of time and frequency. In the next section, the development of PLCA-based algorithm to incorporate time-frequency constraints is

3. A probabilistic latent variable model with time-frequency constraints

Considering this approach, we now develop a new PLCA-based algorithm to incorporate the time-frequency user-annotations. For clarity, we restate the form of

Compared to a modified NMF formulation, incorporating optimization constraints as a function of time, frequency, and sound source into the factorized PLCA

Incorporating prior information into this model, and PLCA in general, can be done in several ways. The most commonly used methods are by direct observations (i.e., setting probabilities to zero, one, etc.) or by incorporating Bayesian prior probabilities on model parameters. Direct observations do not give us enough control, so we consider incorporating Bayesian prior probabilities. For the case of Eq. (10), this would result in independently modifying the factor terms p fz ð Þ j , p tz ð Þ j , or p zð Þ. Common prior probability distributions used for this purpose include

Given that we would like to incorporate the user-annotations as a function of time, frequency, and sound source, however, we notice that this is not easily accomplished using standard priors. This is because the model is factorized, and each factor is only a function of one variable and (possibly) conditioned by another, making it difficult to construct a set of prior probabilities that, when jointly applied

p f ð Þ¼ ; t ∑zp zð Þp fz ð Þ j p tz ð Þ j (10)

the symmetric two-dimensional PLCA model we use:

Dirichlet priors [7], gamma priors [8], and others.

model is particularly interesting and motivating to our focus.

<sup>þ</sup> //input data matrix.

V WH H<sup>T</sup>

W<sup>T</sup> <sup>V</sup> WH

<sup>1</sup>H<sup>T</sup> (8)

<sup>W</sup><sup>T</sup><sup>1</sup> (9)

Procedure KL-NMF (V ∈ RNf �Nt

New Frontiers in Brain-Computer Interfaces

Nz//number of basic vectors.

Initialize: W ∈ RNf �Nz

Optimize over W

Optimize over H

until convergence return: W and H

discussed.

38

)

repeat

to p fz ð Þ j , p tz ð Þ j , and/or p zð Þ, would encourage or discourage one source or another to explain a given time-frequency point. We can see this more clearly when we consider PLCA to be the following simplified estimation problem:

$$X(f, t) \approx \rho(z)\rho(f, z)\rho(t, z)\tag{11}$$

where X f ð Þ ; t is the observed data that we model as the product of three distinct functions or factors φð Þz , φð Þ f; z , and φð Þ t; z . Note, each factor has different input arguments and each factor has different parameters that we wish to estimate via EM. Also, forget for the moment that the factors must be normalized probabilities.

Given this model, if we wish to incorporate additional information, we could independently modify:

This way of manipulation allows us to maintain our factorized form and can be thought of as prior-based regularization. If we would like to incorporate additional information/regularization that is a function of all three variables z, f, and t, then we must do something else. The first option would be to try to simultaneously modify all factors together to impose regularization that is a function of all three variables. This is unfortunately very difficult—both conceptually difficult to construct and practically difficult to algorithmically solve.

This motivates the use of posterior regularization (PR). PR provides us with an algorithmic mechanism via EM to incorporate constraints that are complementary to prior-based regularization. Instead of modifying the individual factors of our model as we saw before, we directly modify the posterior distribution of our model. The posterior distribution of our model, very loosely speaking, is a function of all random variables of our model. It is natively computed within each E step of EM and is required to iteratively improve the estimates of our model parameters. In this example, the posterior distribution would be akin to φð Þ z; f; t , which is a function of t, f, and z, as required. We now formally discuss PR below, beginning with a general discussion and concluding with the specific form of PR we employ within our approach.

## 3.1 Posterior regularization

The framework of posterior regularization, first introduced by Graca, Ganchev, and Taskar [9, 10], is a relatively new mechanism for injecting rich, typically data-dependent constraints into latent variable models using the EM algorithm. In contrast to standard Bayesian prior-based regularization, which applies constraints to the model parameters of a latent variable model in the maximization step of EM, posterior regularization applies constraints to the posterior distribution (distribution over the latent variables, conditioned on everything else) computation in the expectation step of EM. The method has found success in many natural language processing tasks, such as statistical word alignment, part-of-speech tagging, and similar tasks that involve latent variable models.

In this case, what we do is constrain the distribution q in some way when we maximize the auxiliary bound F qð Þ ; Θ with respect to q in the expectation step of an EM algorithm, resulting in

$$q^{n+1} = \underset{q}{\text{argmin}} \, KL(q||p) + \mathcal{Q}(q) \tag{12}$$

To solve the above optimization problem for a given time-frequency point, we

With γ being a Lagrange multiplier, take the gradient with respect to q and γ:

<sup>¼</sup> <sup>1</sup> � <sup>q</sup><sup>T</sup>

qft <sup>¼</sup> Pft <sup>⊗</sup> exp �λft PT

Notice the result is computed in closed form and does not require any iterative optimization scheme as may be required in the general posterior regularization framework [9], minimizing the computational cost when incorporating the constraints. Also note, however, that this optimization must be solved for each timefrequency point of our spectrogram data for each E step iteration of our final EM

Now knowing the posterior-regularized expectation step optimization, we can derive a complete EM algorithm for a posterior-regularized two-dimensional PLCA

pzf ð Þ j ; <sup>t</sup> p zð Þp fz ð Þ <sup>j</sup> p tz ð Þ <sup>j</sup> <sup>Λ</sup>ftz

Algorithm 2 PR-PLCA with linear grouping expectation constraints

where Λ ¼ exp f g �Λ . The entire algorithm is outlined in Algorithm 2. Notice we continue to maintain closed-form E and M steps, allowing us to optimize further and draw connections to multiplicative nonnegative matrix factorization algorithms.

Precompute: Λ exp ð Þ �Λ (21)

∑<sup>z</sup>0p z<sup>0</sup> ð Þp fzj <sup>0</sup> ð Þp tzj <sup>0</sup> ð ÞΛftz<sup>0</sup>

ftln qft <sup>þ</sup> <sup>q</sup><sup>T</sup>

ftλft <sup>þ</sup> <sup>γ</sup> <sup>1</sup> � <sup>q</sup><sup>T</sup>

¼ �ln pft <sup>þ</sup> <sup>1</sup> <sup>þ</sup> ln qft <sup>þ</sup> <sup>λ</sup>ft � <sup>γ</sup><sup>1</sup> <sup>¼</sup> <sup>0</sup> (17)

ft1

ft1 (16)

<sup>¼</sup> <sup>0</sup> (18)

ft exp �λft (19)

(20)

ftln pft <sup>þ</sup> <sup>q</sup><sup>T</sup>

∇aL qft; γ

where exp{} is an element-wise exponential function.

set Eqs. (17) and (18) equal to zero, and solve for qft, resulting in

form the Lagrangian

L qft; γ

parameter estimation algorithm.

3.3 Parameter estimation

Procedure PLCA (

Nz//number of basis vectors Ns//number of sources Λ∈ RNf �Nt�Nz //penalties

<sup>þ</sup> //observed normalized data

Initialize: feasible p zð Þ,p fz ð Þ j and p tz ð Þ j

V ∈ RNf �Nt

)

41

model (PR-PLCA):

¼ �q<sup>T</sup>

Speech Enhancement Using an Iterative Posterior NMF DOI: http://dx.doi.org/10.5772/intechopen.84976

∇qftL qft; γ

where Ωð Þq constrains the possible space of q.

Note, the only difference between Eq. (12) and our past discussion on EM is the added term Ωð Þq . If Ωð Þq is set to zero, we get back the original formulation and easily solve the optimization by setting q = p without any computation (except computing the posterior p). Also note to denote the use of constraints in this context, the term "weakly supervised" was introduced by Graca [11] and is similarly adopted here.

This method of regularization is in contrast to prior-based regularization, where the modified maximization step would be

$$\Theta^{n+1} = \underset{\theta}{\operatorname{argmax}} \, F(q^{n+1}, \theta) + \Omega(\theta) \tag{13}$$

where Ω Θð Þ constrains the model parameter Θ.

## 3.2 Linear grouping expectation constraints

Given the general framework of posterior regularization, we need to define a meaningful penalty Ωð Þq for which we map our annotations. We do this by mapping the annotation matrices to linear grouping constraints on the latent variable z. To do so, we first notice that Eq. (12) decouples for each time-frequency point for our specific model. Because of this, we can independently solve Eq. (12) for each time-frequency point, making the optimization much simpler. When we rewrite our E step optimization using vector notation, we get

$$\begin{cases} \underset{q}{\arg\min} \ -q\_{ft}^{T} \ln p\_{ft} + q\_{ft}^{T} \ln q\_{ft} \\\\ q\_{ft}^{T} \mathbf{1} = \mathbf{1}, q\_{ft} \ge \mathbf{0} \end{cases} \tag{14}$$

subject to

$$q\_{ft}^{\prime}\mathbf{1} = \mathbf{1}, q\_{ft} \ge \mathbf{0} \tag{14}$$

$$\mathbf{1} \quad \dots \quad \mathbf{1} \quad \dots \quad \dots \quad \dots \quad \dots \quad \dots$$

where q and pz f ð Þ j ; t for a given value of f and t is written as qft and pft without any modification; we note q is optimal when equal to pz f ð Þ j ; t as before.

We then apply our linear grouping constraints independently for each timefrequency point:

$$\underset{q}{\text{argmin}} \, \boldsymbol{q} - \boldsymbol{q}\_{ft}^{T} \boldsymbol{l} \boldsymbol{n} \boldsymbol{p}\_{ft} + \boldsymbol{q}\_{ft}^{T} \boldsymbol{l} \boldsymbol{n} \boldsymbol{q}\_{ft} + \boldsymbol{q}\_{ft}^{T} \boldsymbol{\lambda}\_{ft}$$

Subject to

$$q\_{ft}^T \mathbf{1} = \mathbf{1}, q\_{ft} \ge \mathbf{0},\tag{15}$$

where we define λft ¼ Λft1::……Λft1Λft2………Λft<sup>2</sup> <sup>T</sup> ∈ RNz as the vector of userdefined penalty weights, T is a matrix transpose, ≥ is element-wise greater than or equal to, and 1 is a column vector of ones. In this case, positive-valued penalties are used to decrease the probability of a given source, while negative-valued coefficients are used to increase the probability of a given source. Note the penalty weights imposed on the group of values of z that correspond to a given source s are identical, linear with respect to the z variables, and applied in the E step of EM, hence the name "linear grouping expectation constraints."

Speech Enhancement Using an Iterative Posterior NMF DOI: http://dx.doi.org/10.5772/intechopen.84976

<sup>q</sup>nþ<sup>1</sup> <sup>¼</sup> argmin q

<sup>Θ</sup>nþ<sup>1</sup> <sup>¼</sup> argmax Θ

where Ω Θð Þ constrains the model parameter Θ.

our E step optimization using vector notation, we get

argmin q

where we define λft ¼ Λft1::……Λft1Λft2………Λft<sup>2</sup>

hence the name "linear grouping expectation constraints."

argmin q

�q<sup>T</sup>

qT

any modification; we note q is optimal when equal to pz f ð Þ j ; t as before.

�q<sup>T</sup>

qT

ftlnpft <sup>þ</sup> <sup>q</sup><sup>T</sup>

where q and pz f ð Þ j ; t for a given value of f and t is written as qft and pft without

We then apply our linear grouping constraints independently for each time-

ftlnpft <sup>þ</sup> <sup>q</sup><sup>T</sup>

<sup>T</sup>

defined penalty weights, T is a matrix transpose, ≥ is element-wise greater than or equal to, and 1 is a column vector of ones. In this case, positive-valued penalties are used to decrease the probability of a given source, while negative-valued coefficients are used to increase the probability of a given source. Note the penalty weights imposed on the group of values of z that correspond to a given source s are identical, linear with respect to the z variables, and applied in the E step of EM,

ftlnqft

ftlnqft <sup>þ</sup> <sup>q</sup><sup>T</sup>

ftλft

ft1 ¼ 1, qft ≥0, (15)

∈ RNz as the vector of user-

ft1 ¼ 1, qft ≥0 (14)

3.2 Linear grouping expectation constraints

Note, the only difference between Eq. (12) and our past discussion on EM is the added term Ωð Þq . If Ωð Þq is set to zero, we get back the original formulation and easily solve the optimization by setting q = p without any computation (except computing the posterior p). Also note to denote the use of constraints in this context, the term "weakly supervised" was introduced by Graca [11] and is simi-

This method of regularization is in contrast to prior-based regularization, where

F qnþ<sup>1</sup>

Given the general framework of posterior regularization, we need to define a meaningful penalty Ωð Þq for which we map our annotations. We do this by mapping the annotation matrices to linear grouping constraints on the latent variable z. To do so, we first notice that Eq. (12) decouples for each time-frequency point for our specific model. Because of this, we can independently solve Eq. (12) for each time-frequency point, making the optimization much simpler. When we rewrite

where Ωð Þq constrains the possible space of q.

the modified maximization step would be

New Frontiers in Brain-Computer Interfaces

larly adopted here.

subject to

frequency point:

Subject to

40

KL q p ð Þþ k Ωð Þq (12)

; <sup>Θ</sup> <sup>þ</sup> Ω Θð Þ (13)

To solve the above optimization problem for a given time-frequency point, we form the Lagrangian

$$L\left(q\_{\rm ft},\chi\right) = -q\_{\rm ft}^T \ln p\_{\rm ft} + q\_{\rm ft}^T \ln q\_{\rm ft} + q\_{\rm ft}^T \lambda\_{\rm ft} + \chi \left(1 - q\_{\rm ft}^T \mathbf{1}\right) \tag{16}$$

With γ being a Lagrange multiplier, take the gradient with respect to q and γ:

$$\nabla\_{q\_{\hat{\mathbb{H}}}} L(q\_{\hat{\mathbb{H}}}, \chi) = -\ln p\_{\hat{\mathbb{H}}} + \mathbb{1} + \ln q\_{\hat{\mathbb{H}}} + \lambda\_{\hat{\mathbb{H}}} - \gamma \mathbb{1} = \mathbf{0} \tag{17}$$

$$\nabla\_{d}L\left(q\_{ft},\chi\right) = \left(\mathbf{1} - q\_{ft}^{T}\mathbf{1}\right) = \mathbf{0} \tag{18}$$

set Eqs. (17) and (18) equal to zero, and solve for qft, resulting in

$$q\_{ft} = \frac{P\_{ft} \otimes \exp\left(-\lambda\_{ft}\right)}{P\_{ft}^T \exp\left(-\lambda\_{ft}\right)}\tag{19}$$

where exp{} is an element-wise exponential function.

Notice the result is computed in closed form and does not require any iterative optimization scheme as may be required in the general posterior regularization framework [9], minimizing the computational cost when incorporating the constraints. Also note, however, that this optimization must be solved for each timefrequency point of our spectrogram data for each E step iteration of our final EM parameter estimation algorithm.

## 3.3 Parameter estimation

Now knowing the posterior-regularized expectation step optimization, we can derive a complete EM algorithm for a posterior-regularized two-dimensional PLCA model (PR-PLCA):

$$p(\boldsymbol{z}|f,t) \leftarrow \frac{p(\boldsymbol{z})p(f|\boldsymbol{z})p(t|\boldsymbol{z})\Lambda\_{\text{flx}}}{\sum\_{\boldsymbol{z}'} p(\boldsymbol{z}')p(f|\boldsymbol{z}')p(t|\boldsymbol{z}')\overline{\Lambda}\_{\text{flx}'}}\tag{20}$$

where Λ ¼ exp f g �Λ . The entire algorithm is outlined in Algorithm 2. Notice we continue to maintain closed-form E and M steps, allowing us to optimize further and draw connections to multiplicative nonnegative matrix factorization algorithms.

Algorithm 2 PR-PLCA with linear grouping expectation constraints

Procedure PLCA ( V ∈ RNf �Nt <sup>þ</sup> //observed normalized data Nz//number of basis vectors Ns//number of sources Λ∈ RNf �Nt�Nz //penalties ) Initialize: feasible p zð Þ,p fz ð Þ j and p tz ð Þ j

$$\mathbf{Precompute:}$$

$$\text{Precompute:} \qquad \qquad \qquad \Lambda \leftarrow \exp\left(-\Lambda\right) \tag{21}$$

repeat

Expectation step for all z, f, t do

$$p(\boldsymbol{z}|\boldsymbol{f},t) \leftarrow \frac{p(\boldsymbol{z})p(\boldsymbol{f}|\boldsymbol{z})p(\boldsymbol{t}|\boldsymbol{z})\overline{\Lambda}\_{\boldsymbol{f}\boldsymbol{t}\boldsymbol{z}}}{\sum\_{\boldsymbol{z}'}p(\boldsymbol{z}')p(\boldsymbol{f}|\boldsymbol{z}')p(\boldsymbol{t}|\boldsymbol{z}')\overline{\Lambda}\_{\boldsymbol{f}\boldsymbol{t}\boldsymbol{z}'}}\tag{22}$$

which fully specifies the iterative updates. By putting Eqs. (30) and (31) in matrix notation, we specify the multiplicative form of the proposed method in

Algorithm 3. PR-PLCA with linear grouping expectation constraints in matrix

Λ<sup>s</sup> exp f g �Λ<sup>s</sup> (32) Xs V ⊗ Λ<sup>s</sup> (33)

ð Þ WsHs ⊗ Λ<sup>s</sup> (34)

<sup>Γ</sup> (35)

(37)

(36)

Algorithm 3.

Procedure PLCA (

Initialize: W ∈ RNf �Nz

Nz//number of basis vectors Ns//number of sources

For all s do

End for

For all s do

End for until convergence return: W and H

<sup>þ</sup> //observed normalized data

Speech Enhancement Using an Iterative Posterior NMF DOI: http://dx.doi.org/10.5772/intechopen.84976

, ∀ ∈ f g 1; :…Ns //penalties

<sup>þ</sup> , <sup>H</sup> <sup>∈</sup> <sup>R</sup>Nz�Nt þ

Γ ∑<sup>s</sup>

Zs Xs

ZsH<sup>T</sup> s 1H<sup>T</sup> s

<sup>s</sup> Zs

Wð Þ<sup>s</sup> Ws ⊗

<sup>H</sup>ð Þ<sup>s</sup> Hs <sup>⊗</sup> <sup>W</sup><sup>T</sup>

4. An iterative posterior NMF method for speech enhancement in the

Over the past several years, research has been carried out in single-channel sound source separation methods. This problem is motivated by speech denoising, speech enhancement [13], music transcription [14], audio-based forensic, and music remixing. One of the most effective approach is nonnegative matrix factorization (NMF) [5]. The user-annotations can be used to obtain the PR terms [15]. If the number of sources is more, then it is difficult to identify sources in the spectrogram. In such cases, the user interaction-based constraint approaches are inefficient. In order to avoid the previous problem, in the proposed method, an automatic iterative procedure is introduced. The spectral components of speech and noise are

presence of additive Gaussian noise (proposed algorithm)

modeled as Gamma and Rayleigh, respectively [16].

V ∈ RNf �Nt

Λ<sup>s</sup> ∈ RNf �Nt

Precompute:

Repeat

43

notation

)

end for

Maximization step for all z, f, t do

$$p(f|x) = \frac{\sum\_{t} V\_{ft} p(z|f, t)}{\sum\_{f'} \sum\_{t'} V\_{f't'} p(z|f', t')} \tag{23}$$

$$p(t|x) = \frac{\sum\_{f} \mathcal{V}\_{ft} p(x|f, t)}{\sum\_{f'} \sum\_{t'} \mathcal{V}\_{f't'} p(x|f', t')} \tag{24}$$

$$p(\mathbf{z}) = \frac{\sum\_{f} \sum\_{t} V\_{ft} p(\mathbf{z}|f, t)}{\sum\_{\mathbf{z}'} \sum\_{f'} \sum\_{t'} V\_{f't'} p(\mathbf{z}|f', t')} \tag{25}$$

## end for

until convergence return:p fz ð Þ j , p tz ð Þ j ,p zð Þ and pz f ð Þ j ; t

## • Multiplicative Update Equations

We can rearrange the expressions in Algorithm 2 and convert to a multiplicative form following similar methodology to Smaragdis and Raj [12].

Rearranging the expectation and maximization steps, in conjunction with Bayes' rule, and

$$Z(f,t) = \sum\_{x} p(z)p(f|z)p(t|z)\overline{\Lambda}\_{fix}$$

we get

$$p(\boldsymbol{z}|\boldsymbol{f},t) = \frac{p(\boldsymbol{f}|\boldsymbol{z})p(\boldsymbol{t},\boldsymbol{z})\overline{\Lambda}\_{\text{f\!{x}}}}{Z(\boldsymbol{f},t)}\tag{26}$$

$$p(t, x) = \sum\_{f} V\_{ft} q(x|f, t) \tag{27}$$

$$p(f|\mathbf{z}) = \frac{\sum\_{t} V\_{ft} q(\mathbf{z}|f, t)}{\sum\_{t} p(t, \mathbf{z})} \tag{28}$$

$$p(\mathbf{z}) = \sum\_{\mathbf{t}} p(\mathbf{t}, \mathbf{z}) \tag{29}$$

Rearranging further, we get

$$p(f|z) = \frac{p(f|z) \sum\_{t} \frac{V\_{ft}\overline{A}\_{\theta t}}{Z(f,t)} p(t,z)}{\sum\_{t} p(t,z)}\tag{30}$$

$$p(t, z) = p(t, z) \sum\_{f} p(f|z) \frac{V\_{ft} \overline{A}\_{fix}}{Z(f, t)} \tag{31}$$

which fully specifies the iterative updates. By putting Eqs. (30) and (31) in matrix notation, we specify the multiplicative form of the proposed method in Algorithm 3.

Algorithm 3.  An iterative posterior NMF method for speech enhancement in the presence of additive Gaussian noise (proposed algorithm)

Over the past several years, research has been carried out in single-channel sound source separation methods. This problem is motivated by speech denoising, speech enhancement [13], music transcription [14], audio-based forensic, and music remixing. One of the most effective approach is nonnegative matrix factorization (NMF) [5]. The user-annotations can be used to obtain the PR terms [15]. If the number of sources is more, then it is difficult to identify sources in the spectrogram. In such cases, the user interaction-based constraint approaches are inefficient.

In order to avoid the previous problem, in the proposed method, an automatic iterative procedure is introduced. The spectral components of speech and noise are modeled as Gamma and Rayleigh, respectively [16].

## 4.1 Notation and basic concepts

Let noisy speech signal x[n] be the sum of clean speech s[n] and noise d[n] and their corresponding magnitude spectrogram be represented as

$$|\mathbf{X}(f,\mathbf{t})| = |\mathbf{S}(f,\mathbf{t})| + |\mathbf{D}(f,\mathbf{t})|\tag{38}$$

f xð Þ¼ ; <sup>k</sup>; <sup>θ</sup> xk�1<sup>e</sup>

xi and <sup>k</sup><sup>≈</sup> <sup>3</sup> � <sup>s</sup> <sup>þ</sup>

The regularization term for the speech samples is defined as (by applying nega-

Special case: When we fix k = 1, the Gamma density simplifies to the exponential

The subscript(s) with parenthesis represents corresponding columns or rows of the matrix assigned to a given source. 1 is an approximately sized matrices of ones,

<sup>θ</sup> ,ΛSP � <sup>Λ</sup><sup>S</sup><sup>2</sup> <sup>¼</sup> <sup>x</sup>

with shape parameter k>0 and scale parameter θ>0:

where the auxiliary variable <sup>s</sup> is defined as <sup>s</sup> <sup>¼</sup> ln <sup>1</sup>

<sup>Λ</sup>SP � <sup>Λ</sup><sup>S</sup><sup>2</sup> ¼ � log f xð Þ¼ ; <sup>k</sup>; <sup>θ</sup> <sup>x</sup>

f xð Þ¼ ; 1; θ

and ⊗ represents element-wise multiplication.

X ∈ R<sup>f</sup>�<sup>t</sup>

Λ<sup>S</sup> ∈ R<sup>f</sup>�<sup>t</sup>

<sup>2</sup>σ<sup>2</sup> � log <sup>X</sup>

1 θ e �x

The proposed multiplicative nonnegative matrix factorization method is summarized in Algorithm 4 [16]. In general, like in the specific case of Algorithm 4,

one can only guarantee the monotonous descent of the iteration through a majorization-minimization approach [19] or the convergence to a stationary

Algorithm 4: Gamma-Rayleigh regularized PLCA method (GR-NMF)

<sup>þ</sup> % Observed normalized data

Λs NEW ð Þ ¼ 0

<sup>σ</sup><sup>2</sup> and <sup>Λ</sup><sup>S</sup><sup>2</sup> <sup>¼</sup> <sup>Λ</sup>SP OLD ð Þ <sup>¼</sup> <sup>X</sup>

Λe<sup>s</sup> ¼ ð Þ 1 � μ Λs OLD ð Þ þ μΛs NEW ð Þ %Update penalties using LMS (50)

<sup>θ</sup> <sup>¼</sup> <sup>1</sup> kN <sup>∑</sup> N i¼1

Speech Enhancement Using an Iterative Posterior NMF DOI: http://dx.doi.org/10.5772/intechopen.84976

tive logarithm in both sides of (44))

density and

point [20].

 NS % ΛS-Penalties, NS-Number of sources

<sup>Λ</sup>es OLD ð Þ exp f g �Λ<sup>s</sup> (49)

Λs OLD ð Þ ¼ Λs NEW ð Þ (51)

<sup>θ</sup> � log <sup>X</sup><sup>k</sup>�<sup>1</sup> θk Γð Þk

!

(48)

<sup>Γ</sup>ð Þ<sup>k</sup> (44)

<sup>N</sup> <sup>∑</sup><sup>N</sup>

<sup>θ</sup> , x≥<sup>0</sup> (47)

<sup>i</sup>¼1ln ð Þ xi .

, x≥ 0 (46)

(45)

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð Þ <sup>s</sup> � <sup>3</sup> <sup>2</sup> <sup>þ</sup> <sup>24</sup><sup>s</sup>

> <sup>N</sup> <sup>∑</sup><sup>N</sup> <sup>i</sup>¼<sup>1</sup>xi � � � <sup>1</sup>

> > <sup>θ</sup><sup>k</sup>Γð Þ<sup>k</sup>

!

where f represents the frequency bin and t the frame number. The observed magnitudes in time-frequency are arranged in a matrix X ∈ R <sup>f</sup>�<sup>t</sup> <sup>þ</sup> of nonnegative elements. The source separation algorithms based on NMF pursue the factorization of <sup>X</sup> as a product of two nonnegative matrices, <sup>W</sup> <sup>¼</sup> ½ � <sup>w</sup>1; <sup>w</sup>2; …; <sup>w</sup><sup>R</sup> <sup>∈</sup> <sup>R</sup> <sup>f</sup>�<sup>R</sup> <sup>þ</sup> in which the columns collect the basis vectors and <sup>H</sup> <sup>¼</sup> hT <sup>1</sup> ; hT <sup>2</sup> ; :…; <sup>h</sup><sup>T</sup> R <sup>T</sup> <sup>∈</sup> <sup>R</sup>R�<sup>t</sup> <sup>þ</sup> that collects their respective weights, i.e.,

$$X = \mathbf{W} \mathbf{H} = \sum\_{x=1}^{R} W\_x H\_x \tag{39}$$

where R denotes the number of latent components.

## 4.2 Proposed regularization

There are several ways to incorporate the user-annotations into latent variable models, for instance, by using the suitable regularization functions. For expectation maximization (EM) algorithms, posterior regularization was introduced by [9, 11]. This method is data dependent. This method gives richness and also gives the constraints on the posterior distributions of latent variable models. The applications of this method is used in many natural language processing tasks like statistical word alignment, part-of-speech tagging. The main idea is to constrain on the distribution of posterior, when computing expectation step in EM algorithm.

The prior distributions for the magnitude of the noise spectral components are modeled as Rayleigh probability density function (PDF) with scale parameter σ, which is fitted to the observations by a maximum likelihood procedure [16, 17], i.e.,

$$f(\mathbf{x}; \sigma) = \frac{\mathbf{x}}{\sigma^2} e^{-\mathbf{x}^2 / 2\sigma^2} \text{ for } \mathbf{x} \ge \mathbf{0} \text{ with } \sigma^2 = \frac{1}{2N} \sum\_{i=1}^N \mathbf{x}\_i^2 \tag{40}$$

The above equation can be written as

$$f(\mathbf{x}; \sigma) = e^{\log\left(\frac{\mathbf{x}}{\sigma^2}\right)} e^{-\mathbf{x}^2/2\sigma^2} = e^{\log\left(\frac{\mathbf{x}}{\sigma^2}\right) - \frac{\mathbf{x}^2}{2\sigma^2}} \tag{41}$$

By applying negative logarithm on both sides of (41), we will get

$$-\log\left(f(\mathbf{x};\sigma)\right) = -\log\left(e^{\log\left(\frac{\mathbf{x}}{\sigma^2}\right) - \frac{\mathbf{x}^2}{2\sigma^2}}\right) = \frac{\mathbf{x}^2}{2\sigma^2} - \log\left(\frac{\mathbf{x}}{\sigma^2}\right) \tag{42}$$

Then, the regularization term for the noise is defined as

$$
\Lambda\_N \equiv \Lambda\_{\mathbb{S}\_1} = -\log f(\mathfrak{x}; \sigma) = \frac{\mathfrak{x}^2}{2\sigma^2} - \log \frac{\mathfrak{x}}{\sigma^2}.\tag{43}
$$

The spectral components of speech modeled as Gamma probability density function [16, 18]

Speech Enhancement Using an Iterative Posterior NMF DOI: http://dx.doi.org/10.5772/intechopen.84976

4.1 Notation and basic concepts

New Frontiers in Brain-Computer Interfaces

collects their respective weights, i.e.,

4.2 Proposed regularization

Let noisy speech signal x[n] be the sum of clean speech s[n] and noise d[n] and

where f represents the frequency bin and t the frame number. The observed

elements. The source separation algorithms based on NMF pursue the factorization

R z¼1

There are several ways to incorporate the user-annotations into latent variable models, for instance, by using the suitable regularization functions. For expectation maximization (EM) algorithms, posterior regularization was introduced by [9, 11]. This method is data dependent. This method gives richness and also gives the constraints on the posterior distributions of latent variable models. The applications of this method is used in many natural language processing tasks like statistical word alignment, part-of-speech tagging. The main idea is to constrain on the distribution of posterior, when computing expectation step in EM algorithm.

The prior distributions for the magnitude of the noise spectral components are modeled as Rayleigh probability density function (PDF) with scale parameter σ, which is fitted to the observations by a maximum likelihood procedure [16, 17], i.e.,

for x<sup>≥</sup> <sup>0</sup> with <sup>σ</sup><sup>2</sup> <sup>¼</sup> <sup>1</sup>

<sup>¼</sup> <sup>e</sup>log <sup>x</sup> σ2 � <sup>x</sup><sup>2</sup>

<sup>¼</sup> <sup>x</sup><sup>2</sup>

<sup>2</sup>σ<sup>2</sup> � log <sup>x</sup>

<sup>2</sup><sup>N</sup> <sup>∑</sup> N i¼1 x2

<sup>2</sup>σ<sup>2</sup> � log <sup>x</sup>

<sup>i</sup> (40)

(42)

<sup>2</sup>σ<sup>2</sup> (41)

<sup>σ</sup><sup>2</sup> : (43)

σ2

of <sup>X</sup> as a product of two nonnegative matrices, <sup>W</sup> <sup>¼</sup> ½ � <sup>w</sup>1; <sup>w</sup>2; …; <sup>w</sup><sup>R</sup> <sup>∈</sup> <sup>R</sup> <sup>f</sup>�<sup>R</sup>

X ¼ WH ¼ ∑

j j X f ð Þ ;t ¼ j j S f ð Þ ;t þ j j D f ð Þ ;t (38)

<sup>1</sup> ; hT

<sup>2</sup> ; :…; <sup>h</sup><sup>T</sup> R

WzHz (39)

<sup>T</sup>

<sup>þ</sup> of nonnegative

<sup>þ</sup> in

<sup>∈</sup> <sup>R</sup>R�<sup>t</sup> <sup>þ</sup> that

their corresponding magnitude spectrogram be represented as

magnitudes in time-frequency are arranged in a matrix X ∈ R <sup>f</sup>�<sup>t</sup>

which the columns collect the basis vectors and <sup>H</sup> <sup>¼</sup> hT

where R denotes the number of latent components.

f xð Þ¼ ; <sup>σ</sup> <sup>x</sup>

The above equation can be written as

function [16, 18]

44

σ2 e

f xð Þ¼ ; <sup>σ</sup> <sup>e</sup>log <sup>x</sup>

� log ð Þ¼� f xð Þ ; <sup>σ</sup> log <sup>e</sup>log <sup>x</sup>

Then, the regularization term for the noise is defined as

�x2=2σ<sup>2</sup>

σ2 e �x2=2σ<sup>2</sup>

> σ2 � <sup>x</sup><sup>2</sup> 2σ2

By applying negative logarithm on both sides of (41), we will get

<sup>Λ</sup><sup>N</sup> � <sup>Λ</sup><sup>S</sup><sup>1</sup> ¼ � log f xð Þ¼ ; <sup>σ</sup> <sup>x</sup><sup>2</sup>

The spectral components of speech modeled as Gamma probability density

$$f(\mathbf{x};k,\theta) = \frac{\mathbf{x}^{k-1} e^{\frac{\mathbf{x}}{\sigma}}}{\theta^k \Gamma(k)}\tag{44}$$

with shape parameter k>0 and scale parameter θ>0:

$$\theta = \frac{1}{kN} \sum\_{i=1}^{N} \mathbf{x}\_i \text{ and } k \approx \frac{3 - s + \sqrt{\left(s - 3\right)^2 + 24s}}{12s} \tag{45}$$

where the auxiliary variable <sup>s</sup> is defined as <sup>s</sup> <sup>¼</sup> ln <sup>1</sup> <sup>N</sup> <sup>∑</sup><sup>N</sup> <sup>i</sup>¼<sup>1</sup>xi � � � <sup>1</sup> <sup>N</sup> <sup>∑</sup><sup>N</sup> <sup>i</sup>¼1ln ð Þ xi .

The regularization term for the speech samples is defined as (by applying negative logarithm in both sides of (44))

$$\Lambda\_{\text{SP}} \equiv \Lambda\_{\text{S}\_2} = -\log f(\varkappa; k, \theta) = \frac{\varkappa}{\theta} - \log \left( \frac{\varkappa^{k-1}}{\theta^k \Gamma(k)} \right), \varkappa \ge 0 \tag{46}$$

Special case: When we fix k = 1, the Gamma density simplifies to the exponential density and

$$f(\mathbf{x}; \mathbf{1}, \boldsymbol{\theta}) = \frac{\mathbf{1}}{\boldsymbol{\theta}} e^{\frac{\mathbf{x}}{\boldsymbol{\theta}}}, \Lambda\_{\text{SP}} \equiv \Lambda\_{\text{S}\_2} = \frac{\mathbf{x}}{\boldsymbol{\theta}}, \boldsymbol{\infty} \ge \mathbf{0} \tag{47}$$

The proposed multiplicative nonnegative matrix factorization method is summarized in Algorithm 4 [16]. In general, like in the specific case of Algorithm 4, one can only guarantee the monotonous descent of the iteration through a majorization-minimization approach [19] or the convergence to a stationary point [20].

The subscript(s) with parenthesis represents corresponding columns or rows of the matrix assigned to a given source. 1 is an approximately sized matrices of ones, and ⊗ represents element-wise multiplication.

Algorithm 4: Gamma-Rayleigh regularized PLCA method (GR-NMF)

## Procedure

X ∈ Rf�<sup>t</sup> <sup>þ</sup> % Observed normalized data Λ<sup>S</sup> ∈ R<sup>f</sup>�<sup>t</sup> <sup>þ</sup> , s∈ f g 1; ::…; NS % ΛS-Penalties, NS-Number of sources

$$\mathbf{A}\_{\mathbf{s}(\text{NEW})} = \mathbf{0}$$

$$\Lambda\_{\rm S1} = \Lambda\_{\rm N(OD)} = \frac{X^2}{2\sigma^2} - \log \frac{X}{\sigma^2} \text{ and } \Lambda\_{\rm S1} = \Lambda\_{\rm SP(OD)} = \frac{X}{\theta} - \log \left(\frac{X^{k-1}}{\theta^k \Gamma(k)}\right) \tag{48}$$

$$\ddot{A}\_{s(OLD)} \leftarrow \exp\left\{-A\_s\right\} \tag{49}$$

Repeat

For all s do

$$A\_s = (1 - \mu)\Lambda\_{\text{s}(\text{OD})} + \mu\Lambda\_{\text{s}(\text{NEW})} \text{ \textquotedbl{}GUpdate penalities using LMS} \tag{50}$$

$$
\Lambda\_{\text{s(ODD)}} = \Lambda\_{\text{s(NEW)}} \tag{51}
$$

$$X\_{\mathfrak{s}} \leftarrow X \otimes \tilde{\Lambda}\_{\mathfrak{S}} \tag{52}$$

5. Experimental results

Speech Enhancement Using an Iterative Posterior NMF DOI: http://dx.doi.org/10.5772/intechopen.84976

6. Conclusion

PESQ and SDR for street noise.

Table 2.

47

Table 1.

PESQ and SDR for babble noise.

The speech and noise audio samples were taken from NOIZEUS [21]. Sampling frequency is 8 KHz. The algorithm is iterated until convergence [16]. The proposed method was compared with Euclidean NMF (EUC-NMF) [5], Itakura-Saito NMF (IS-NMF) [22], posterior regularization NMF (PR-NMF) [15], Wiener filtering

implemented by considering nonstationary noise, babble noise and street noise. The performance of proposed method was evaluated by using perceptual evaluation of speech quality (PESQ) [25] and source-to-distortion ratio (SDR) [26]. SDR gives the average quality of separation on dB scale and considers signal distortion as well as noise distortion. For PESQ and SDR, the higher value indicates the better performance. Tables 1 and 2 show the PESQ and SDR values of different NMF algorithms evaluated. The experimental results show that proposed method performs better

A novel speech enhancement method based on an iterative and regularized NMF algorithm for single-channel source separation is proposed. The clean speech and noise magnitude spectra are modeled as Gamma and Rayleigh distributions, respectively. The corresponding log-likelihood functions are used as penalties to

[23], and constrained version of NMF (CNMF)[24]. These methods are

than other existing methods in terms of the PESQ and SDR indices.

End for

$$
\Gamma \leftarrow \sum\_{\mathfrak{s}} \left( \mathcal{W}\_{\mathfrak{s}\mathfrak{l}} H\_{\mathfrak{k}\mathfrak{l}} \right) \otimes \widetilde{\Lambda}\_{\mathfrak{s}} \tag{53}
$$

$$Z\_s \leftarrow \frac{X\_s}{\Gamma} \tag{54}$$

For all s do

$$\mathbf{W}\_{(\circ)} \leftarrow \mathbf{W}\_{(\circ)} \otimes \frac{\mathbf{Z}\_{\circ} \mathbf{H}\_{(\circ)}^{T}}{\mathbf{1} \mathbf{H}\_{(\circ)}^{T}} \tag{55}$$

$$H\_{(\circ)} \leftarrow H\_{(\circ)} \otimes \left(W^T\_{(\circ)} Z\_{\circ}\right) \tag{56}$$

## End for Reconstruction For all s do

$$M\_{\circ} \leftarrow \frac{W\_{(s)}H\_{(s)}}{\text{WH}} \quad \text{@ Compute Filter} \tag{57}$$

$$
\hat{X}\_s \gets M\_s \otimes X \lll \text{Filter Mixture} \tag{58}
$$

$$\mathbf{x}\_{\mathbf{s}} \leftarrow \text{ISTFT}\left(\hat{\mathbf{X}}\_{\mathbf{s}}, \mathcal{L}\mathbf{X}, \boldsymbol{P}\right) \; \forall \; \mathbf{P}-\text{ STFT}\; parameters \tag{59}$$

if update k % Gamma model

$$\mathcal{L} = \ln\left(\frac{1}{N}\sum \hat{X}\_s\right) - \frac{1}{N}\sum \ln\left(\hat{X}\_s\right),\\k \approx \frac{3-s+\sqrt{(s-3)^2+24s}}{12s} \tag{60}$$

else % Exponential model

k = 1,

end

$$\theta = \frac{1}{kN} \sum \hat{X}\_s \tag{61}$$

$$A\_{S\_1} = A\_{N(OLD)} = \frac{\hat{X}\_{s\_1}^2}{2\sigma^2} - \log\frac{\hat{X}\_{s\_1}}{\sigma^2}$$

$$A\_{S\_2} = A\_{SP(OLD)} = \frac{\hat{X}\_{s\_2}}{\theta} - \log\left(\frac{\hat{X}\_{s\_2}^{k-1}}{\theta^k \Gamma(k)}\right) \tag{62}$$

<sup>Λ</sup>s NEW ð Þ <sup>¼</sup> exp �Λs OLD ð Þ � � % <sup>Λ</sup>s OLD ð Þ represents both <sup>Λ</sup>SP and <sup>Λ</sup><sup>N</sup> (63)

End for Until Convergence Return: Time domain signals xs

## 5. Experimental results

Xs X ⊗ Λe<sup>S</sup> (52)

� � <sup>⊗</sup> <sup>Λ</sup>e<sup>S</sup> (53)

<sup>Γ</sup> (54)

(55)

(56)

(60)

Γ ∑<sup>s</sup> Wð Þ<sup>s</sup> Hð Þ<sup>s</sup>

Wð Þ<sup>s</sup> Wð Þ<sup>s</sup> ⊗

<sup>H</sup>ð Þ<sup>s</sup> <sup>H</sup>ð Þ<sup>s</sup> <sup>⊗</sup> <sup>W</sup><sup>T</sup>

Ms <sup>W</sup>ð Þ<sup>s</sup> <sup>H</sup>ð Þ<sup>s</sup>

if update k % Gamma model

<sup>N</sup> <sup>∑</sup>ln <sup>X</sup>^ <sup>s</sup>

Zs Xs

ZsH<sup>T</sup> ð Þs 1H<sup>T</sup> ð Þs

ð Þ<sup>s</sup> Zs � �

WH % Compute Filter (57)

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð Þ <sup>s</sup> � <sup>3</sup> <sup>2</sup> <sup>þ</sup> <sup>24</sup><sup>s</sup>

kN <sup>∑</sup>X^ <sup>s</sup> (61)

1

A (62)

<sup>X</sup>^ <sup>s</sup> Ms <sup>⊗</sup> <sup>X</sup> % Filter Mixture (58)

q

12s

σ2

<sup>X</sup>^ <sup>k</sup>�<sup>1</sup> s2 <sup>θ</sup><sup>k</sup>Γð Þ<sup>k</sup>

0 @

� � % <sup>Λ</sup>s OLD ð Þ represents both <sup>Λ</sup>SP and <sup>Λ</sup><sup>N</sup> (63)

xs ISTFT <sup>X</sup>^ <sup>s</sup>;∠X; <sup>P</sup> � � % <sup>P</sup> � STFT parameters (59)

� �, k<sup>≈</sup> <sup>3</sup> � <sup>s</sup> <sup>þ</sup>

s1 <sup>2</sup>σ<sup>2</sup> � log <sup>X</sup>^ <sup>s</sup><sup>1</sup>

<sup>θ</sup> � log

<sup>θ</sup> <sup>¼</sup> <sup>1</sup>

<sup>Λ</sup><sup>S</sup><sup>1</sup> <sup>¼</sup> <sup>Λ</sup>N OLD ð Þ <sup>¼</sup> <sup>X</sup>^ <sup>2</sup>

<sup>Λ</sup><sup>S</sup><sup>2</sup> <sup>¼</sup> <sup>Λ</sup>SP OLD ð Þ <sup>¼</sup> <sup>X</sup>^ <sup>s</sup><sup>2</sup>

End for

New Frontiers in Brain-Computer Interfaces

For all s do

End for

<sup>s</sup> <sup>¼</sup> ln <sup>1</sup>

end

46

else % Exponential model k = 1,

Λs NEW ð Þ ¼ exp �Λs OLD ð Þ

Return: Time domain signals xs

End for Until Convergence

<sup>N</sup> <sup>∑</sup>X^ <sup>s</sup> � � � 1

For all s do

Reconstruction

The speech and noise audio samples were taken from NOIZEUS [21]. Sampling frequency is 8 KHz. The algorithm is iterated until convergence [16]. The proposed method was compared with Euclidean NMF (EUC-NMF) [5], Itakura-Saito NMF (IS-NMF) [22], posterior regularization NMF (PR-NMF) [15], Wiener filtering [23], and constrained version of NMF (CNMF)[24]. These methods are implemented by considering nonstationary noise, babble noise and street noise. The performance of proposed method was evaluated by using perceptual evaluation of speech quality (PESQ) [25] and source-to-distortion ratio (SDR) [26]. SDR gives the average quality of separation on dB scale and considers signal distortion as well as noise distortion. For PESQ and SDR, the higher value indicates the better performance. Tables 1 and 2 show the PESQ and SDR values of different NMF algorithms evaluated. The experimental results show that proposed method performs better than other existing methods in terms of the PESQ and SDR indices.

Table 1. PESQ and SDR for babble noise.

Table 2.

PESQ and SDR for street noise.

## 6. Conclusion