* [307] C.-F. Lin, S.-M. Yuan, M.-C. Leu, and C.-T. Tsai, “A framework for scalable cloud video recorder system in surveillance environment,” in _Ubiquitous intelligence & computing and 9th international conference on autonomic & trusted computing (UIC/ATC), 2012 9th international conference on_. IEEE, 2012, pp. 655–660.
  * [308] X. Liu, D. Zhao, L. Xu, W. Zhang, J. Yin, and X. Chen, “A distributed video management cloud platform using hadoop,” _IEEE Access_ , vol. 3, pp. 2637–2643, 2015.
  * [309] C. Ryu, D. Lee, M. Jang, C. Kim, and E. Seo, “Extensible video processing framework in apache hadoop,” in _Cloud Computing Technology and Science (CloudCom), 2013 IEEE 5th International Conference on_ , vol. 2. IEEE, 2013, pp. 305–310.
  * [310] G. Bradski and A. Kaehler, “Opencv,” _Dr. Dobb’s journal of software tools_ , vol. 3, 2000.
  * [311] M. Ali, A. Anjum, O. Rana, A. R. Zamani, D. Balouek-Thomert, and M. Parashar, “Res: Real-time video stream analytics using edge enhanced clouds,” _IEEE Transactions on Cloud Computing_ , 2020.
  * [312] B. White, T. Yeh, J. Lin, and L. Davis, “Web-scale computer vision using mapreduce for multimedia data mining,” in _Proceedings of the Tenth International Workshop on Multimedia Data Mining_. ACM, 2010, p. 9.
  * [313] R. Pereira, M. Azambuja, K. Breitman, and M. Endler, “An architecture for distributed high performance video processing in the cloud,” in _Cloud Computing (CLOUD), 2010 IEEE 3rd International Conference on_. IEEE, 2010, pp. 482–489.
  * [314] C. Liu, K. Fan, Z. Yang, and J. Xiu, “A distributed video share system based on hadoop,” in _Cloud Computing and Intelligence Systems (CCIS), 2014 IEEE 3rd International Conference on_. IEEE, 2014, pp. 587–590.
  * [315] M. A. Hossain, “Framework for a cloud-based multimedia surveillance system,” _International Journal of Distributed Sensor Networks_ , vol. 10, no. 5, p. 135257, 2014.
  * [316] L. Valentín, S. A. Serrano, R. O. García, A. Andrade, M. A. Palacios-Alonso, and L. E. Sucar, “a cloud-based architecture for smart video surveillance,” _The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences_ , vol. 42, p. 99, 2017.
  * [317] W. Zhang, P. Duan, Z. Li, Q. Lu, W. Gong, and S. Yang, “A deep awareness framework for pervasive video cloud.” _IEEE Access_ , vol. 3, pp. 2227–2237, 2015.
  * [318] W. Zhang, L. Xu, Z. Li, Q. Lu, and Y. Liu, “A deep-intelligence framework for online video processing,” _IEEE Software_ , vol. 33, no. 2, pp. 44–51, 2016\.
  * [319] M. A. Uddin, A. Alam, N. A. Tu, M. S. Islam, and Y.-K. Lee, “Siat: A distributed video analytics framework for intelligent video surveillance,” _Symmetry_ , vol. 11, no. 7, p. 911, 2019.
  * [320] H. Zhang, J. Yan, and Y. Kou, “Efficient online surveillance video processing based on spark framework,” in _International Conference on Big Data Computing and Communications_. Springer, 2016, pp. 309–318.
  * [321] M. A. Uddin, J. B. Joolee, A. Alam, and Y.-K. Lee, “Human action recognition using adaptive local motion descriptor in spark,” _IEEE Access_ , vol. 5, pp. 21 157–21 167, 2017.
  * [322] H. Wang, X. Zheng, and B. Xiao, “Large-scale human action recognition with spark,” in _Multimedia Signal Processing (MMSP), 2015 IEEE 17th International Workshop on_. IEEE, 2015, pp. 1–6.
  * [323] Q. Huang, P. Ang, P. Knowles, T. Nykiel, I. Tverdokhlib, A. Yajurvedi, P. Dapolito IV, X. Yan, M. Bykov, C. Liang _et al._ , “Sve: Distributed video processing at facebook scale,” in _Proceedings of the 26th Symposium on Operating Systems Principles_. ACM, 2017, pp. 87–103.
  * [324] W. Zhang, Z. Wang, X. Liu, H. Sun, J. Zhou, Y. Liu, and W. Gong, “Deep learning-based real-time fine-grained pedestrian recognition using stream processing,” _IET Intelligent Transport Systems_ , vol. 12, no. 7, pp. 602–609, 2018.
  * [325] Citilog, “Citilog,” 1997.
  * [326] checkvideo, “checkvideo,” 1998, accessed: 2018-12-07. [Online]. Available: https://www.checkvideo.com/
  * [327] I. Vision, “Intelli vision,” 2002. [Online]. Available: https://www.intelli-vision.com/
  * [328] G. Inc, “Video ai,” 2017. [Online]. Available: https://cloud.google.com/video-intelligence/
  * [329] B. LLC, “Usher in the new cognitive era,” 2015. [Online]. Available: https://bluechasm.com/
  * [330] J. H. Julia Kornich, Craig Casey, “Media analytics on the media services platform,” 2010. [Online]. Available: https://docs.microsoft.com/en-gb/azure/media-services/previous/media-services-analytics-overview
  * [331] G. Inc, “Features,” 2017. [Online]. Available: https://cloud.google.com/video-intelligence/docs/features
  * [332] J. G. Collier, “Ibm uses watson’s power to unlock insights from video in the cloud,” 2017. [Online]. Available: https://www-03.ibm.com/press/us/en/pressrelease/52126.wss
  * [333] I. Cloud, “Ibm cloud object storage,” 2017. [Online]. Available: https://www.ibm.com/cloud/object-storage
  * [334] ——, “Cloudant,” 2017. [Online]. Available: https://www.ibm.com/cloud/cloudant
  * [335] G. Ananthanarayanan, P. Bahl, P. Bodík, K. Chintalapudi, M. Philipose, L. Ravindranath, and S. Sinha, “Real-time video analytics: The killer app for edge computing,” _computer_ , vol. 50, no. 10, pp. 58–67, 2017.
  * [336] F. Loewenherz, V. Bahl, and Y. Wang, “Video analytics towards vision zero,” _Institute of Transportation Engineers. ITE Journal_ , vol. 87, no. 3, p. 25, 2017.
  * [337] H. Qiu, X. Liu, S. Rallapalli, A. J. Bency, K. Chan, R. Urgaonkar, B. Manjunath, and R. Govindan, “Kestrel: Video analytics for augmented multi-camera vehicle tracking,” in _2018 IEEE/ACM Third International Conference on Internet-of-Things Design and Implementation (IoTDI)_. IEEE, 2018, pp. 48–59.
  * [338] T. Gao, Z.-G. Liu, S.-H. Yue, J.-Q. Mei, and J. Zhang, “Traffic video-based moving vehicle detection and tracking in the complex environment,” _Cybernetics and Systems: An International Journal_ , vol. 40, no. 7, pp. 569–588, 2009.
  * [339] Y. Lin, L. Li, H. Jing, B. Ran, and D. Sun, “Automated traffic incident detection with a smaller dataset based on generative adversarial networks,” _Accident Analysis & Prevention_, vol. 144, p. 105628, 2020.
  * [340] L. Fridman, D. E. Brown, M. Glazer, W. Angell, S. Dodd, B. Jenik, J. Terwilliger, J. Kindelsberger, L. Ding, S. Seaman _et al._ , “Mit autonomous vehicle technology study: Large-scale deep learning based analysis of driver behavior and interaction with automation,” _arXiv preprint arXiv:1711.06976_ , 2017.
  * [341] W. Xu, H. Zhou, N. Cheng, F. Lyu, W. Shi, J. Chen, and X. Shen, “Internet of vehicles in big data era,” _IEEE/CAA Journal of Automatica Sinica_ , vol. 5, no. 1, pp. 19–35, 2017.
  * [342] J. Wang, G. Bebis, M. Nicolescu, M. Nicolescu, and R. Miller, “Improving target detection by coupling it with tracking,” _Machine Vision and Applications_ , vol. 20, no. 4, pp. 205–223, 2009.
  * [343] Y. J. Tsai, Z. Hu, and C. Alberti, “Detection of roadway sign condition changes using multi-scale sign image matching (m-sim),” _Photogrammetric Engineering & Remote Sensing_, vol. 76, no. 4, pp. 391–405, 2010.
  * [344] R. Ayachi, M. Afif, Y. Said, and M. Atri, “Traffic signs detection for real-world application of an advanced driving assisting system using deep learning,” _Neural Processing Letters_ , vol. 51, no. 1, pp. 837–851, 2020\.
  * [345] A. M. Callan, R. Osu, Y. Yamagishi, D. E. Callan, and N. Inoue, “Neural correlates of resolving uncertainty in driver’s decision making,” _Human brain mapping_ , vol. 30, no. 9, pp. 2804–2812, 2009.
  * [346] C. Yan, F. Coenen, and B. Zhang, “Driving posture recognition by convolutional neural networks,” _IET Computer Vision_ , vol. 10, no. 2, pp. 103–114, 2016\.
  * [347] S. Gite and H. Agrawal, “Early prediction of driver’s action using deep neural networks,” _International Journal of Information Retrieval Research (IJIRR)_ , vol. 9, no. 2, pp. 11–27, 2019.
  * [348] S. Fleck and W. Straßer, “Smart camera based monitoring system and its application to assisted living,” _Proceedings of the IEEE_ , vol. 96, no. 10, pp. 1698–1714, 2008.
  * [349] Z. Zhou, X. Chen, Y.-C. Chung, Z. He, T. X. Han, and J. M. Keller, “Activity analysis, summarization, and visualization for indoor human activity monitoring,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 18, no. 11, pp. 1489–1498, 2008.
  * [350] J. Aertssen, M. Rudinac, and P. Jonker, “Fall and action detection in elderly homes,” _AAATE: Maastricht, The Netherlands_ , 2011.
  * [351] D. Brulin, Y. Benezeth, and E. Courtial, “Posture recognition based on fuzzy logic for home monitoring of the elderly,” _IEEE transactions on information technology in biomedicine_ , vol. 16, no. 5, pp. 974–982, 2012.
  * [352] G. Sacco, V. Joumier, N. Darmon, A. Dechamps, A. Derreumaux, J.-H. Lee, J. Piano, N. Bordone, A. Konig, B. Teboul _et al._ , “Detection of activities of daily living impairment in alzheimer’s disease and mild cognitive impairment using information and communication technology,” _Clinical interventions in aging_ , vol. 7, p. 539, 2012.
  * [353] M. Yu, A. Rhuma, S. M. Naqvi, L. Wang, and J. Chambers, “A posture recognition-based fall detection system for monitoring an elderly person in a smart home environment,” _IEEE transactions on information technology in biomedicine_ , vol. 16, no. 6, pp. 1274–1286, 2012.
  * [354] W. Choi, C. Pantofaru, and S. Savarese, “A general framework for tracking multiple people from a moving camera,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 35, no. 7, pp. 1577–1591, 2012.
  * [355] V. Pragada, “Intrusion detection and video surveillance activation and processing,” Aug. 6 2015, uS Patent App. 14/172,880.
  * [356] Z. Zhao, H. Li, R. Zhao, and X. Wang, “Crossing-line crowd counting with two-phase deep neural networks,” in _European Conference on Computer Vision_. Springer, 2016, pp. 712–726.
  * [357] G. Xu and W. K. Cobb, “Loitering detection in a video surveillance system,” Dec. 8 2015, uS Patent 9,208,675.
  * [358] P. Shivakumara, D. Tang, M. Asadzadehkaljahi, T. Lu, U. Pal, and M. H. Anisi, “Cnn-rnn based method for license plate recognition,” _CAAI Transactions on Intelligence Technology_ , vol. 3, no. 3, pp. 169–175, 2018.
  * [359] B. Krausz and R. Herpers, “Metrosurv: detecting events in subway stations,” _Multimedia Tools and Applications_ , vol. 50, no. 1, pp. 123–147, 2010.
  * [360] J.-L. Shih, C.-c. Han, and K.-C. Yan, “Illegal entrant detection at a restricted area in open spaces using color features,” in _Proceedings 40th Annual 2006 International Carnahan Conference on Security Technology_. IEEE, 2006, pp. 68–74.
  * [361] H. Singh, “Applications of intelligent video analytics in the field of retail management: A study,” in _Supply Chain Management Strategies and Risk Assessment in Retail Environments_. IGI Global, 2018, pp. 42–59.
  * [362] H. Xu, P. Lv, and L. Meng, “A people counting system based on head-shoulder detection and tracking in surveillance video,” in _2010 International Conference On Computer Design and Applications_ , vol. 1. IEEE, 2010, pp. V1–394.
  * [363] I. Haritaoglu and M. Flickner, “Attentive billboards: Towards to video based customer behavior understanding,” in _Sixth IEEE Workshop on Applications of Computer Vision, 2002.(WACV 2002). Proceedings._ IEEE, 2002, pp. 127–131.
  * [364] Y. Hu, L. Cao, F. Lv, S. Yan, Y. Gong, and T. S. Huang, “Action detection in complex scenes with spatial and temporal ambiguities,” in _2009 IEEE 12th International Conference on Computer Vision_. IEEE, 2009, pp. 128–135.
  * [365] R. Sicre and H. Nicolas, “Human behaviour analysis and event recognition at a point of sale,” in _2010 Fourth Pacific-Rim Symposium on Image and Video Technology_. IEEE, 2010, pp. 127–132.
  * [366] A. D. Popescu, A. Balmin, V. Ercegovac, and A. Ailamaki, “Predict: towards predicting the runtime of large scale iterative analytics,” _Proceedings of the VLDB Endowment_ , vol. 6, no. 14, pp. 1678–1689, 2013\.
  * [367] S. Amershi, M. Cakmak, W. B. Knox, and T. Kulesza, “Power to the people: The role of humans in interactive machine learning,” _AI Magazine_ , vol. 35, no. 4, pp. 105–120, 2014.
  * [368] Y.-K. Kwok and I. Ahmad, “Static scheduling algorithms for allocating directed task graphs to multiprocessors,” _ACM Computing Surveys (CSUR)_ , vol. 31, no. 4, pp. 406–471, 1999.
  * [369] X. Zhu, J. Wang, J. Wang, and X. Qin, “Analysis and design of fault-tolerant scheduling for real-time tasks on earth-observation satellites,” in _2014 43rd International Conference on Parallel Processing_. IEEE, 2014, pp. 491–500.
  * [370] E. Bortnikov, A. Frank, E. Hillel, and S. Rao, “Predicting execution bottlenecks in map-reduce clusters,” in _Presented as part of the_ , 2012\.
  * [371] Y. Zhai, Y.-S. Ong, and I. W. Tsang, “The emerging” big dimensionality”,” 2014\.
  * [372] L. Gao, J. Song, X. Liu, J. Shao, J. Liu, and J. Shao, “Learning in high-dimensional multimedia data: the state of the art,” _Multimedia Systems_ , vol. 23, no. 3, pp. 303–313, 2017.
  * [373] A. Guttman, “R-trees: A dynamic index structure for spatial searching,” in _Proceedings of the 1984 ACM SIGMOD International Conference on Management of Data_ , ser. SIGMOD ’84. New York, NY, USA: ACM, 1984, pp. 47–57. [Online]. Available: http://doi.acm.org/10.1145/602259.602266
  * [374] P. Ciaccia, M. Patella, and P. Zezula, “M-tree: An efficient access method for similarity search in metric spaces,” in _Proceedings of the 23rd International Conference on Very Large Data Bases_ , ser. VLDB ’97. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 1997, pp. 426–435. [Online]. Available: http://dl.acm.org/citation.cfm?id=645923.671005
  * [375] K.-I. Lin, H. V. Jagadish, and C. Faloutsos, “The tv-tree: An index structure for high-dimensional data,” _The VLDB Journal_ , vol. 3, no. 4, pp. 517–542, 1994.
  * [376] M. Datar, N. Immorlica, P. Indyk, and V. S. Mirrokni, “Locality-sensitive hashing scheme based on p-stable distributions,” in _Proceedings of the twentieth annual symposium on Computational geometry_. ACM, 2004, pp. 253–262.
  * [377] P. O’Neil, E. Cheng, D. Gawlick, and E. O’Neil, “The log-structured merge-tree (lsm-tree),” _Acta Informatica_ , vol. 33, no. 4, pp. 351–385, 1996.
  * [378] T. Kraska, A. Beutel, E. H. Chi, J. Dean, and N. Polyzotis, “The case for learned index structures,” in _Proceedings of the 2018 International Conference on Management of Data_. ACM, 2018, pp. 489–504.
  * [379] G. De Francisci Morales, “Samoa: A platform for mining big data streams,” in _Proceedings of the 22nd International Conference on World Wide Web_. ACM, 2013, pp. 777–778.
  * [380] J. Lu, S. C. Hoi, J. Wang, P. Zhao, and Z.-Y. Liu, “Large scale online kernel learning,” _The Journal of Machine Learning Research_ , vol. 17, no. 1, pp. 1613–1655, 2016.
  * [381] D. Nallaperuma, R. Nawaratne, T. Bandaragoda, A. Adikari, S. Nguyen, T. Kempitiya, D. De Silva, D. Alahakoon, and D. Pothuhera, “Online incremental machine learning platform for big data-driven smart traffic management,” _IEEE Transactions on Intelligent Transportation Systems_ , 2019\.
  * [382] J. Friedman, T. Hastie, and R. Tibshirani, _The elements of statistical learning_. Springer series in statistics New York, 2001, vol. 1, no. 10.
  * [383] S. Kandel, A. Paepcke, J. M. Hellerstein, and J. Heer, “Enterprise data analysis and visualization: An interview study,” _IEEE Transactions on Visualization and Computer Graphics_ , vol. 18, no. 12, pp. 2917–2926, 2012.
  * [384] A. Kumar, R. McCann, J. Naughton, and J. M. Patel, “Model selection management systems: The next frontier of advanced analytics,” _ACM SIGMOD Record_ , vol. 44, no. 4, pp. 17–22, 2016.
  * [385] Z. Zhang, B. Cui, Y. Shao, L. Yu, J. Jiang, and X. Miao, “Ps2: Parameter server on spark,” in _Proceedings of the 2019 International Conference on Management of Data_ , 2019, pp. 376–388.
  * [386] J. Jiang, F. Fu, T. Yang, and B. Cui, “Sketchml: Accelerating distributed machine learning with data sketches,” in _Proceedings of the 2018 International Conference on Management of Data_ , 2018, pp. 1269–1284.
  * [387] Y. Huang, T. Jin, Y. Wu, Z. Cai, X. Yan, F. Yang, J. Li, Y. Guo, and J. Cheng, “Flexps: Flexible parallelism control in parameter server architecture,” _Proceedings of the VLDB Endowment_ , vol. 11, no. 5, pp. 566–579, 2018.
  * [388] R. M. Haralick, “Performance characterization in computer vision,” in _BMVC92_. Springer, 1992, pp. 1–8.
  * [389] W. Förstner, “Dagm workshop on performance characteristics and quality of computer vision algorithms,” _Technical University of Brunswick, Germany_ , 1997.
  * [390] P. J. Phillips and K. W. Bowyer, “Empirical evaluation of computer vision algorithms,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 21, no. 4, pp. 289–290, 1999.
  * [391] H. Christensen and W. Foerstner, “Special issue on performance evaluation,” _Machine Vision and Applications_ , vol. 9, no. 5, 1997.
  * [392] A. T. Nghiem, F. Bremond, M. Thonnat, and V. Valentin, “Etiseo, performance evaluation for video surveillance systems,” in _Advanced Video and Signal Based Surveillance, 2007. AVSS 2007. IEEE Conference on_. IEEE, 2007, pp. 476–481.
  * [393] A. G. Hauptmann and M. G. Christel, “Successful approaches in the trec video retrieval evaluations,” in _Proceedings of the 12th annual ACM international conference on Multimedia_. ACM, 2004, pp. 668–675.
  * [394] H. O. S. D. Branch, “Imagery library for intelligent detection systems (i-lids),” in _2006 IET Conference on Crime and Security_. IET, 2006, pp. 445–448.
  * [395] D. Singh and C. K. Reddy, “A survey on platforms for big data analytics,” _Journal of big data_ , vol. 2, no. 1, p. 8, 2015.
  * [396] D. Wu, X. Luo, M. Shang, Y. He, G. Wang, and M. Zhou, “A deep latent factor model for high-dimensional and sparse matrices in recommender systems,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , 2019.
  * [397] A. Mazrekaj, I. Shabani, and B. Sejdiu, “Pricing schemes in cloud computing: an overview,” _International Journal of Advanced Computer Science and Applications_ , vol. 7, no. 2, pp. 80–86, 2016.
  * [398] S. Khalid, S. Wu, A. Alam, and I. Ullah, “Real-time feedback query expansion technique for supporting scholarly search using citation network analysis,” _Journal of Information Science_ , p. 0165551519863346, 2019.
  * [399] D. Kang, P. Bailis, and M. Zaharia, “Challenges and opportunities in dnn-based video analytics: A demonstration of the blazeit video query engine.” in _CIDR_ , 2019.
  * [400] Y. Lu, A. Chowdhery, and S. Kandula, “Visflow: a relational platform for efficient large-scale video analytics,” in _ACM Symposium on Cloud Computing (SoCC)_ , 2016.
  * [401] D. Kang, P. Bailis, and M. Zaharia, “Blazeit: Optimizing declarative aggregation and limit queries for neural network-based video analytics.”
  * [402] A. Ghosh and I. Arce, “Guest editors’ introduction: In cloud computing we trust-but should we?” _IEEE security & privacy_, vol. 8, no. 6, pp. 14–16, 2010.
  * [403] S. Khusro, A. Alam, and S. Khalid, “Social question and answer sites: the story so far,” _Program_ , vol. 51, no. 2, pp. 170–192, 2017.
  * [404] A. Alam, S. Khusro, I. Ullah, and M. S. Karim, “Confluence of social network, social question and answering community, and user reputation model for information seeking and experts generation,” _Journal of Information Science_ , vol. 43, no. 2, pp. 260–274, 2017.
  * [405] M. Crosby, P. Pattanayak, S. Verma, V. Kalyanaraman _et al._ , “Blockchain technology: Beyond bitcoin,” _Applied Innovation_ , vol. 2, no. 6-10, p. 71, 2016.

|  Aftab Alam has received his master’s degree in computer science with a
specialization in Web engineering from the Department of Computer Science,
University of Peshawar, Peshawar, Pakistan. He is currently pursuing his Ph.D.
at the Department of Computer Science and Engineering, Kyung Hee University
(Global Campus), South Korea. His research interests in the areas of: Big data
analytics, distributed computing, video retrieval, social computing,
machine/deep learning, information service engineering, knowledge engineering,
knowledge graph and graph mining.
---|---
|  Irfan Ullah has received his master’s degree in computer science from the
National University of Sciences and Technology (NUST), Islamabad, Pakistan.
Currently, he is serving as a research assistant at the Department of Computer
Science and Engineering, Kyung Hee University (Global Campus), South Korea.
His research interests in the areas of: Natural language processing, social
computing, crisis informatics, machine/deep learning, OS design and
optimization on memory systems, big data analytics, and distributed computing.
---|---
|  Young-Koo Lee has received the B.S., M.S., and Ph.D. degrees in computer
science from the Korea Advanced Institute of Science and Technology, Daejeon,
South Korea, in 1992, 1994, and 2002. Since 2004, he has been with the
Department of Computer Science and Engineering, Kyung Hee University. His
researches have been concentrated on data mining, online analytical
processing, and big data processing.
---|---


# VoxGenesis: Unsupervised Discovery of Latent Speaker Manifold for Speech
Synthesis

Weiwei Lin, Chenhang He, Man-Wai Mak & Kong Aik, Lee
Department of Electrical and Electronic Engineering and Department of
Computing
The Hong Kong Polytechnic University
&Jiachen Lian
Department of Electrical Engineering & Computer Science
University of California Berkeley

###### Abstract

Achieving nuanced and accurate emulation of human voice has been a
longstanding goal in artificial intelligence. Although significant progress
has been made in recent years, the mainstream of speech synthesis models still
relies on supervised speaker modeling and explicit reference utterances.
However, there are many aspects of human voice, such as emotion, intonation,
and speaking style, for which it is hard to obtain accurate labels. In this
paper, we propose VoxGenesis, a novel unsupervised speech synthesis framework
that can discover a latent speaker manifold and meaningful voice editing
directions without supervision. VoxGenesis is conceptually simple. Instead of
mapping speech features to waveforms deterministically, VoxGenesis transforms
a Gaussian distribution into speech distributions conditioned and aligned by
semantic tokens. This forces the model to learn a speaker distribution
disentangled from the semantic content. During the inference, sampling from
the Gaussian distribution enables the creation of novel speakers with distinct
characteristics. More importantly, the exploration of latent space uncovers
human-interpretable directions associated with specific speaker
characteristics such as gender attributes, pitch, tone, and emotion, allowing
for voice editing by manipulating the latent codes along these identified
directions. We conduct extensive experiments to evaluate the proposed
VoxGenesis using both subjective and objective metrics, finding that it
produces significantly more diverse and realistic speakers with distinct
characteristics than the previous approaches. We also show that latent space
manipulation produces consistent and human-identifiable effects that are not
detrimental to the speech quality, which was not possible with previous
approaches. Finally, we demonstrate that VoxGenesis can also be used in voice
conversion and multi-speaker TTS, outperforming the state-of-the-art
approaches. Audio samples of VoxGenesis can be found at:
https://bit.ly/VoxGenesis.

##  1 Introduction

Deep generative models have revolutionized multiple fields, marked by several
breakthroughs including the Generative Pretrained Transformer (GPT) (Brown et
al., 2020), Generative Adversarial Network (GAN) (Goodfellow et al., 2014),
Variational Autoencoder (VAE) (Kingma & Welling, 2014), and, more recently,
Denoising Diffusion Models (DDPM) (Dhariwal & Nichol, 2021; Ho et al., 2020).
These models can generate realistic images, participate in conversations with
humans, and compose intricate programs. When utilized in speech synthesis,
they are capable of producing speech that is virtually indistinguishable from
human speech (Shen et al., 2018; Oord et al., 2016; Kim et al., 2021; Wang et
al., ; Tan et al., 2022). However, the success are primarily confined to
replicating the voices of training or reference speakers. In contrast to image
synthesis, where models can produce realistic and unseen scenes and faces, the
majority of speech synthesis models are unable to generate new, unheard
voices. We argue that this limitation predominantly stems from the design of
the speaker encoders and neural vocoders. Typically, they function as
deterministic modules (Polyak et al., 2021; Jia et al., 2018b; Kim et al.,
2021; Qian et al., 2020), mapping speaker embeddings to the target waveforms.

Besides the obvious advantage of being able to generate new objects,
generative models also permit the control over the generation process and
allow for latent space manipulation to edit specific aspects of the generated
objects without the necessity for attribute labels (Härkönen et al., 2020;
Voynov & Babenko, 2020a). This advantage is especially important in speech,
where nuanced characteristics such as emotion, intonation, and speaker styles
are hard to label. The incorporation of a speaker latent space could enable
more sophisticated voice editing and customization, expanding the potential
applications of speech synthesis substantially. However, learning the speaker
distribution is not a straightforward task. This is due to the intrinsic
complexity of speech signals where speaker-specific characteristics are
entangled with the semantic content information. As such, we cannot directly
fit a distribution over speech and expect the model to generate new speakers
while maintaining control over the content information. The disentanglement of
content from speaker features is a necessary first step (Hsu et al., 2017;
Yadav et al., 2023; Qian et al., 2020; Lin et al., 2023). In (Stanton et al.,
2022), the authors proposed TacoSpawn, a method that fits a Gaussian Mixture
Model (GMM) over Tacotron2 speaker embeddings to learn a prior distribution
over speakers. While TacoSpawn (Stanton et al., 2022) does offer the
capability to generate novel speakers, it comes with its own set of
limitations. Firstly, there is a separation in the parameterization of the
speaker embedding table and the speaker generation model, which prevents the
synthesis modules from fully benefiting from the generative approach.
Secondly, in contrast to modern deep generative models, the mixture model in
TacoSpawn is trained to maximize the likelihood of the speaker embeddings
rather than the data likelihood, thereby limiting the representational
capability of the generative model.

In this paper, we introduce VoxGenesis, an unsupervised generative model that
learns a distribution over the voice manifold. At its core, VoxGenesis learns
to transform a Gaussian distribution into a speech distribution conditioned on
semantic tokens. This approach contrasts with conventional GAN vocoders such
as Mel-GAN (Kumar et al., 2019), HIFI-GAN (Kong et al., 2020), and more
recently SpeechResynthesis (Polyak et al., 2021), which learn a deterministic
mapping between speech features and waveforms. Figure 1 illustrates the
architectural differences between VoxGenesis and SpeechResynthesis. VoxGenesis
introduces a mapping network that converts the isotropic Gaussian distribution
into a non-isotropic one, enabling the control module (the yellow box) to
identify major variances. It also features a shared embedding layer for the
discriminator and employs semantic transformation matrices, facilitating
semantic-specific transformations of speaker attributes. Furthermore,
VoxGenesis sets itself apart from image generation GANs like Style-GAN or
BigGAN by integrating a Gaussian constrained encoder into the framework. This
inclusion not only stabilizes training but also enables the encoding of
external speaker representations.

In summary, our contributions are as follows:

  * •

We introduce a general framework for unsupervised voice generation by
transforming Gaussian distribution to speech distribution.

  * •

We demonstrate the potential for unsupervised editing of nuanced speaker
attributes such as gender characteristics, pitch, tone, and emotions.

  * •

We identify the implicit sampling process associated with using speaker
embeddings for GANs and proposed a divergence term to constrain the speaker
embeddings distributions. This allows the conventional speaker encoder to be
incorporated as components of a generative model, thereby facilitating the
encoding and subsequent modification of external speakers.

##  2 Backgroud

Voice Conversion (VC) and Text-to-Speech (TTS). The majority of VC and TTS
models work in the speech feature domain, meaning that the model output are
speech features such as a Mel-spectrogram (Qian et al., 2019; Kaneko et al.,
2019; Shen et al., 2018; Ren et al., 2019). The primary distinction between VC
and TTS is their approach to content representations. While VC strives to
convert speech from one speaker to another without altering the content, TTS
acquires content information from a text encoder. This encoder is trained on
paired text-speech data and can utilize autoregressive modelling (Shen et al.,
2018) or non-autoregressive modelling with external alignments, as seen in
FastSpeech (Ren et al., 2019). Given the necessity for vocoders in both VC and
TTS to invert the spectrogram, there has been a significant effort to improve
them. This has led to the development of autoregressive, flow, GAN, and
diffusion-based vocoders (Kong et al., 2020; Oord et al., 2016; Prenger et
al., 2019; Chen et al., 2020). Recently, VITS (Kim et al., 2021), an end-to-
end TTS model, has been introduced; it utilizes conditional variational
autoencoders in tandem with adversarial training to facilitate the direct
conversion from text to waveform. Building on top of VITS, YourTTS (Casanova
et al., 2022) caters to multilingual scenarios in low-resource languages by
enhancing the input text with language embeddings. Beyond the standard GAN and
VAE models, VoiceBox introduces flow-matching to produce speech when provided
with an audio context and text (Le et al., 2023).

Speaker Modeling in Speech Synthesis. Speaker modeling stands as a crucial
component in speech synthesis. The initial approach to speaker modeling
involved the utilization of a speaker embedding table (Gibiansky et al.,
2017). However, the scalability of this method becomes a concern with the
increase in the number of speakers. Therefore, pretrained speaker encoders
have been introduced into TTS systems to facilitate the transfer of learned
speaker information to the synthesis modules (Jia et al., 2018a). The speaker
embeddings can be combined with conventional speech features like MFCC or with
self-supervised learned (SSL) speech units (Hsu et al., 2021; Schneider et
al., 2019). A demonstration of integrating SSL speech units with speaker
embeddings is presented in SpeechResynthesis (Polyak et al., 2021), where the
authors have proposed a model that re-synthesizes speech utilizing SSL units
and speaker embeddings. Besides the explicit utilization of speaker lookup
tables and speaker embeddings, speaker information can also be incorporated
implicitly. This can be achieved by training autoregressive models on residual
vector quantized (RVQ) representations (Kumar et al., 2023; Défossez et al.,
2022; Zeghidour et al., 2021), exemplified by VALL-E (Wang et al., ), or
through BERT-like masking prediction as in SoundStorm (Borsos et al., 2023).

Speech Style Learning and Editing. Speech conveys multifaceted information
such as speaker identity, pitch, emotion, and intonation. Many elements are
challenging to label, making unsupervised learning a popular approach for
extracting such information. The concept of Style Tokens is introduced in
(Wang et al., 2018), where a bank of global style tokens (GST) is learned
jointly with Tacotron. The authors demonstrate that GST can be employed to
manipulate speech speed and speaking style, independently of text content. In
another development, SpeechSplit (Wang et al., 2018) achieves the
decomposition of speech into timbre, pitch, and rhythm by implementing
information bottlenecks. Consequently, style-transfer can be executed using
the disentangled representations. However, the utilization of information
bottlenecks can potentially result in deteriorated reconstruction quality. To
address this issue, the authors in (Choi et al., 2021) propose NANSY, an
analysis and synthesis framework that employs information perturbation to
disentangle speech features. This method has been successfully applied in
various applications, including voice conversion, pitch shift, and time-scale
modification.

Figure 1:  Illustration of (a) the SpeechResynthesis (excluding pitch module)
and (b) our VoxGenesis. In contrast to SpeechResynthesis, which focuses on
reconstructing the waveforms from semantic tokens and speaker embeddings,
VoxGenesis is a deep generative model that learns to transform a Gaussian
distribution to match the speech distribution conditioned on semantic tokens
either using Gaussian noise as input (the green dashed line), or using
embeddings from a Gaussian-constrained speaker encoder (pink dashed line).

##  3 VoxGenesis

###  3.1 Learning Latent Speaker Distribution with GAN

Generative Adversarial Network (GAN) has been the de facto choice for vocoders
since the advent of Mel-GAN and HiFi-GAN (Kumar et al., 2019; Kong et al.,
2020). However, these GANs are predominantly utilized as spectrogram
inverters, learning deterministic mappings from Mel-spectrogram or other
speech features (Polyak et al., 2021) to waveforms. A notable limitation in
these models is the lack of true “generation”; the vocoders learn to replicate
the voices of the training or reference speakers rather than creating new
voices. This stands in contrast to the application of GANs in computer vision,
where they are primarily utilized to generate new faces and objects (Karras et
al., 2019; Brock et al., 2018). The principal challenge in using GAN as a
generative model arises due to the high semantic variations in speech. This
makes transforming a Gaussian distribution to a speech distribution difficult
without certain constraints. Recently, Self-Supervised Learned (SSL) speech
units have emerged as effective tools for disentangling semantic information
(Hsu et al., 2021; Baevski et al., 2020). This advancement motivates us to
utilize these semantic tokens as conditions for GAN; consequently, a
conditional GAN is employed to transform a Gaussian distribution, rather than
mapping speech features to waveforms. Let’s represent a semantic token
sequence as $Y$ and an acoustic waveform sequence as $X$, originating from the
empirical distribution $p_{\text{data}}(X)$. We aim to train a GAN to
transform a standard Gaussian distribution $p(\bm{\mathbf{z}})$ into speech
distributions $p_{\text{data }}(X)$ conditioned on semantic tokens $Y$. This
is done by solving the following min-max problem with a discriminator $D$ and
a generator $G$:

$\min_{G}\max_{D}V(D,G)=\mathbb{E}_{X\sim p_{\text{data }}(X)}[\log D(X\mid
Y)]+\mathbb{E}_{\bm{\mathbf{z}}\sim{\cal
N}(0,I)}[\log(1-D(G(\bm{\mathbf{z}}\mid Y))],$ (1)

There is no one-to-one correspondence between the latent code and the
generated waveform anymore. Therefore, training the network with the
assistance of Mel-spectrogram loss, as proposed in (Kong et al., 2020), is no
longer feasible here. Instead, the model is trained to minimize discrepancies
only at the distribution level, as opposed to relying on point-wise loss.

Regarding the generator’s design, Figure 1(b) highlights three modules that
are vital for learning: Shared Embedding Layer $e$: Both the generator and the
discriminator leverage a shared embedding layer $e$. It is crucial, in the
absence of Mel-spectrogram loss, for the discriminator to receive semantic
tokens; otherwise, the generator could deceive the discriminator with
intelligible speech. Mapping Network $M$: A mapping network $M$ is integrated,
consisting of seven feedforward layers, to transform the latent code prior to
the deconvolution layers. Drawing inspiration from Style-GAN (Karras et al.,
2019), this enables the generation of more representative latent codes and a
non-isotropic distribution, the output of which will be utilized by the
control module, discussed in later section. Semantic Conditioned
Transformation $T$: Rather than indiscriminately adding the latent codes to
each semantic token embedding, we conditionally transform the latent code
based on the semantic information. This enables semantic-specific
transformations of speaker attributes. Specifically, the generator comprises a
deep deconvolution network $f$, a semantic conditioned feed-forward network
$T$, a shared embedding layer $e$, and a latent code transform network $M$.
The equation for the generator, conditioned on semantic tokens, is represented
as:

$G(\bm{\mathbf{z}}|Y)=f\left(T\big{(}M(\bm{\mathbf{z}}),Y\big{)}+e(Y)\right).$
(2)

Ancestral Sampling for GAN. Transforming random noise to speech distribution
has its disadvantages, one of which is the inability to use specific speakers’
voices post-training due to the absence of an encoder to encode external
speakers. Another notable challenge is the well-known “mode collapse,” an
issue often mitigated in most GAN vocoders due to the stabilizing effect of
the Mel-spectrogram loss during training. To overcome these challenges, we
introduce a probabilistic encoder capable of encoding speaker representation
using posterior inference, $p_{\theta}(\bm{\mathbf{z}}|X)$, while maintaining
the marginal distribution as a standard Gaussian distribution,
$p(\bm{\mathbf{z}})$. The neural factor analysis (NFA) (Lin et al., 2023) is
one of such models. Here we assume the NFA encoder is pre-trained. During GAN
training, ancestral sampling is used; initially, samples are drawn from the
empirical distribution, $p_{\text{data}}(X)$, followed by sampling from the
posterior distribution , $p_{\theta}(\bm{\mathbf{z}}|X)$, parametrized by
$\theta$:

$\displaystyle X$ $\displaystyle\sim p_{\text{data}}(X)$ (3)
$\displaystyle\bm{\mathbf{z}}$ $\displaystyle\sim
p_{\theta}(\bm{\mathbf{z}}|X).$ (4)

Given that the marginal distribution, $p(\bm{\mathbf{z}})$, is Gaussian, the
GAN continues to be trained to transform a Gaussian distribution to a speech
distribution, conditioned on semantic tokens. Here, the one-to-one
correspondence between $\bm{\mathbf{z}}$ and the target waveform, $X$, is re-
established, allowing the usage of Mel-spectrogram loss to stabilize training
and avert mode collapse. During inference, to generate random speakers,
samples can be drawn from the marginal distribution, $p(\bm{\mathbf{z}})$, or,
to encode a specific speaker, the maximum a posteriori estimate of the
conditional distribution, $p_{\theta}(\bm{\mathbf{z}}|X)$, can be used. We
refer to the resulting model as NFA-VoxGenesis.

The ancestral sampling procedures described in Eq. 3 and Eq. 4 can be
generalized to encompass any encoder capable of yielding a conditional
distribution, extending even to discriminative speaker encoders. The essential
prerequisite here is the feasibility of sampling from the marginal
distribution $p(\bm{\mathbf{z}})$, a condition not satisfied by discriminative
speaker encoders as it does not constrain $p(\bm{\mathbf{z}})$ during
training. To address this, a divergence term can be introduced during
discriminative speaker encoders training to ensure that the marginal
distribution $p_{\theta}(\bm{\mathbf{z}})$ approximates a standard Gaussian
distribution:

$\min_{\theta}\lambda{\cal
D}_{KL}(p_{\theta}(\bm{\mathbf{z}})||{\cal{N}}(\bm{\mathbf{0}},\bm{\mathbf{I}})),$
(5)

where ${\cal D}_{KL}$ is the Kullback–Leibler divergence and $\lambda$ control
the strength of divergence relate to other encoder loss. The subscript
$\theta$ denotes the dependence of the implicit distribution
$p(\bm{\mathbf{z}})$ on the parameters of the speaker encoder. Since
$p_{\theta}(\bm{\mathbf{z}})$ is accessible only through ancestral sampling,
Eq. 5 is executed by computing the mean and the standard deviation of the
speaker embeddings within a mini-batch and subsequently computing the
divergence with a standard Gaussian. Incorporating the divergence term during
the training phase of the speaker encoders enables compatibility of our
framework with any speaker encoders. The VoxGenesis model equipped with a
speaker encoder trained via cross-entropy is referred as CE-VoxGenesis, and
when trained with contrastive loss, it is referred as CL-VoxGenesis. Because
all encoders are trained with Gaussian divergence, we refer to them as
Gaussian-constrained speaker encoders as depicted by Figure 1(b). Table 4
illustrates the different variants of VoxGenesis associated with various
speaker encoders.

###  3.2 Interpretable Latent Direction Discovery

GANs are often preferred over denoising diffusion models and flow models (Ho
et al., 2020; Kingma & Dhariwal, 2018) due to their semantically meaningful
latent space. This characteristics enables manipulations to modify various
aspects of the generated object (Härkönen et al., 2020; Voynov & Babenko,
2020b). This feature is particularly invaluable in applications where
obtaining attribute labels is challenging. In this section, we illuminate how
a straightforward application of Principal Component Analysis (PCA) on
intermediate features unveils latent directions instrumental for manipulating
speaker characteristics. PCA is a canonical technique designed to identify the
predominant variations within the data. Our objective is to apply PCA to
latent representations to uncover these significant variations or changes that
are interpretable to humans. As discussed in (Härkönen et al., 2020), the
isotropic distribution of $p(\bm{\mathbf{z}})$ tends to be ineffective for
highlighting the most distinctive change directions, due to its uniform
characteristic in all dimensions. To use PCA effectively, we opt for computing
them on the output of the mapping network $M(\bm{\mathbf{z}})$. We randomly
sample $\bm{\mathbf{z}}$ from a Gaussian distribution and compute the
corresponding $\bm{\mathbf{w}}=M(\bm{\mathbf{z}})$. Singular Value
Decomposition (SVD) is then employed to determine the $N$ bases
$\\{\bm{\mathbf{v}}\\}_{n=1}^{N}$. For any given speaker representation
$\bm{\mathbf{z}}$, modifications can be performed by moving $\bm{\mathbf{w}}$
along the direction outlined by the principal component $\bm{\mathbf{v}}_{n}$:

$\bm{\mathbf{w}}^{\prime}=M(\bm{\mathbf{z}})+s\bm{\mathbf{v}}_{n},$ (6)

where $s$ represents a shift value. $\bm{\mathbf{w}}^{\prime}$ is subsequently
fed through the deconvolution layers to synthesize speech spoken by the
modified speaker. This procedure is applicable to external speaker
representations encoded through encoders like NFA or Gaussian-constrained
speaker encoders. Figure 2 illustrates the effect of modifying latent codes
along the discovered directions.

Figure 2: The effect of manipulating different latent directions on speaker
similarity, pitch, and gender identification.

We find principal directions related to gender characteristics, pitch, tone,
and emotion. Notably, inter-speaker variations like gender are reflected in
the leading Principal Components (PCs), while more subtle intra-speaker
nuances like emotion are captured in the latter PCs. As illustrated in Figure
2 (b), manipulating the latent representation of a male speaker along the
negative direction of first principal component gradually shifts it towards
the sound of a female speaker, as detected by a speech gender classifier. It
is noteworthy that between the region of -0.1 and -0.25, speaker similarity
remains high (0.9 and 0.7, respectively), and the classifier exhibits
ambiguity regarding the speaker’s gender. This implies minimal alteration in
speaker identity, while rendering the speaker more feminine sounding with
subtle modifications. In Figure 2 (a), we demonstrate the effects of modifying
the third principal direction, responsible for controlling pitch. The shift in
the latent code along this PC (depicted by the green line) apparently lowers
the pitch across the entire recording, compared to the original waveform
represented by the blue line. Conversely, altering along the opposite
direction (illustrated by the red line) elevates the pitch. Figure 2 (c)
evaluates the ramifications of shifting different PCs on speaker similarity.
It’s evident that manipulations employing leading PCs influence speaker
similarity more substantially compared to those utilizing later PCs. This
phenomenon suggests a potential application of later PCs in refining subtle
speaker attributes like emotion and intonation, allowing for nuanced
adjustments while preserving the inherent characteristics of the speaker.

###  3.3 Voice Conversion and Multi-speaker TTS with VoxGenesis

With the integration of NFA (Lin et al., 2023) or Gaussian-constrained speaker
encoders, VoxGenesis can be effectively employed for voice conversion and
multi-speaker Text-to-Speech (TTS). Given a speaker reference waveform,
denoted as $X_{b}$, and a content reference waveform, denoted as $X_{a}$,
VoxGenesis enables the conversion of the speaker identity in $X_{a}$ to that
in $X_{b}$, while the speech content in $X_{a}$ remains unchanged. This is
represented mathematically as:

$\hat{X}_{a\rightarrow b}=G\left(\bm{\mathbf{z}}_{b}\mid
Y_{a}\right),\quad\text{where}\quad\bm{\mathbf{z}}_{b}=\underset{\bm{\mathbf{z}}}{\arg\max}\phantom{s}p_{\theta}(\bm{\mathbf{z}}\mid
X_{b}).$ (7)

$Y_{a}$ represents the semantic token that is extracted from the content
reference waveform $X_{a}$. Essentially, this capability allows for the
transformation of speaker identity of the given speech content without
altering the content of the speech. We can also sample from a Gaussian
distribution to generate a novel speaker speaking the content in $X_{a}$:

$\hat{X}_{a\rightarrow\text{?}}=G\left(\bm{\mathbf{z}}\mid
Y_{a}\right),\quad\text{where}\quad\bm{\mathbf{z}}\sim\mathcal{N}(\bm{\mathbf{0}},\,\bm{\mathbf{I}}).$
(8)

VoxGenesis can also be deployed as a speaker encoder and vocoder for a multi-
speaker TTS. For this application, we initially discretize speech features
utilizing HuBERT (Hsu et al., 2021) and subsequently train a Tacotron model to
predict the discrete tokens.

##  4 Experimental Setup

###  4.1 Model Configuration and Training Details

All VoxGenesis variants and baseline models including TacoSpawn, VITS, and
SpeechResynthesis were trained using the train-clean-100 and train-clean-360
split of LibriTTS-R (Zen et al., 2019; Koizumi et al., 2023). Audio files are
downsampled to 16kHz to ensure compatibility with the 16kHz models. For
training vanilla-VoxGenesis, NFA-VoxGenesis, and CL-VoxGenesis, we did not use
speaker labels or any meta data. For CE-VoxGenesis, we used the speaker
labels. We used HuBERT Large as semantic encoder (Ott et al., 2019). Different
from (Lin et al., 2023), NFA was trained with an EM algorithm using HuBERT’s
features and discrete tokens. The embeddings dimension of NFA speak vector is
300. For CE-VoxGenesis and CL-VoxGenesis, the speaker encoders were trained
using cross-entropy and contrastive loss on a X-vector network (Snyder et al.,
2018), respectively. Because the HuBERT features have a larger time span than
the MFCC features used in the original HiFi-GAN, we adjusted the upsample
parameters in the transpose convolution layer to [10, 4, 2, 2]. We used the
Adam optimizer with a learning rate of 0.0002 and the betas set to 0.8 and
0.99. The training segment length was set to 8,960 frames.

###  4.2 Evaluation Metrics for Speaker Generation

We used Fréchet Inception distance (FID) (Heusel et al., 2017) on speaker
embeddings to compare the generated speaker distribution and the training
speaker distribution. We used 50,000 randomly sampled utterances to evaluate
the FID score. Because a model that simply memorizes the training speakers
would achieve a very low FID score and it is easy to memorize speaker
embeddings with speaker labels, which would not align with our goal of novel
speaker generation. To complement FID, we used an additional subjective metric
that measures the similarity between the generated speakers and the training
speakers. Specifically, we used a pre-trained x-vector network (Snyder et al.,
2018) to retrieve the top-3 most similar speech segments, and then asked 20
human evaluators to assess the similarity between the generated speaker and
the retrieved ones. We used a three-point scale from 0 to 1 to represent the
evaluators’ opinions, with 1 being that theretrieved audio is very likely from
the generated speaker and 0 being that the retrieved audio is unlikely to be
from the generated speaker. We refer to the score as the speaker similarity
score. Additionally, we asked the evaluators to rate the diversity of the
generated speakers on a scale from 0 to 5, with 0 indicating no diversity and
5 indicating that every utterance sounded like it was spoken by a different
speaker. We refer to this metric as the diversity score. Finally, we asked the
evaluators to rate the naturalness of the speech using the standard MOS scale,
with an interval of 0.5. We utilized crowd-sourcing for the subjective
evaluations.

##  5 Results

Given the nature of our work, we believe that it would be more informative for
readers to listen to the audio samples for comparisons. The demo page is
available at https://bit.ly/VoxGenesis.

###  5.1 Speaker Generation Evaluation

Table 1: Speaker generation evaluation. Subjective metrics results are reported with a 95% confidence interval. Method | FID $\downarrow$ | Spk. Similarity $\downarrow$ | Spk. Diversity $\uparrow$ | MOS $\uparrow$
---|---|---|---|---
Ground Truth | - | 0.22 | 4.22$\pm$0.07 | $4.3\pm 0.09$
TacoSpawn (Stanton et al., 2022) | 0.18 | 0.59 | 3.85$\pm$0.09 | $3.54\pm 0.09$
Vanilla-VoxGenesis | 0.17 | 0.38 | 3.96$\pm$0.07 | 3.92$\pm 0.07$
NFA-VoxGenesis | 0.14 | 0.30 | 4.17$\pm$0.09 | 4.22$\pm$0.06
CL-VoxGenesis | 0.16 | 0.36 | 4.02$\pm$0.12 | 3.74$\pm$0.08
CE-VoxGenesis | 0.11 | 0.28 | 4.11$\pm$0.07 | 4.13$\pm$0.09

In this section, we evaluate the diversity, speech quality, and similarity of
the generated speakers in comparison to the training speakers. Table 1
presents these evaluations for TacoSpawn (Stanton et al., 2022) and four
VoxGenesis variants, with ground truth included for reference.

We can see that all four variants of VoxGenesis produce lower FID scores than
TacoSpawn. This suggestes that VoxGenesis is more effective in capturing the
speaker distribution. Moreover, VoxGenesis speaker similarity is also
significantly lower than TacoSpawn, suggesting that the generative process
relies less on memorization of the training speakers. Among the variants, the
unsupervised version of VoxGenesis registers a higher FID score than its
supervised counterpart—a foreseeable outcome given its lack of access to
speaker labels. Despite no discernible difference in speech quality, Vanilla-
VoxGenesis records the lowest diversity score, as reflected by both FID and
speaker diversity score, signaling some degree of mode collapse occurring in
Vanilla-VoxGenesis training. During the auditory evaluation, we found that
VoxGenesis tends to generate speakers who exhibit distinct characteristics and
speak with better intonation and emotion, in contrast to the more neutral-
toned speakers produced by TacoSpawn. This distinction is reflected in the MOS
and speaker diversity scores, where all four VoxGenesis variants outperform
TacoSpawn.

###  5.2 Edibility of the Latent Space

In this section, we evaluate the edibility of the VoxGenesis latent space.
Specifically, the objectives are to investigate: (1) whether editing impacts
the speech quality of the recording, (2) the extent to which editing alters
the identity of the speaker, and (3) whether the editing along a direction has
consistent effects and generalizes to both the internal latent code and
externally encoded speakers. To address these questions, we provided 10 edited
sample for each editing direction, and engaged human assessors to evaluate
both the speech quality, measured by MOS, and identifiability, measured by the
”successful ID rate”. Additionally, a pre-trained speaker classifier (Snyder
et al., 2018) was employed to assess the similarity among the edited speakers.
For the identifiability experiments, assessors were asked to identify the
changes induced by editings, given 10 options, which included 4 real
directions utilized in the experiment and 6 random distractors. We conducted
experiments on both internal representations (samples from Gaussian prior) and
external speakers (extracted from test speech files).

Figure 3: Barplot showing the effect of editing the latent representations on
speaker similarity, speech quality, and identifiability.

The outcomes of these experiments are shown in Figure 3, where the first row
shows the results of internal speaker editing and the second row show the
results of external speaker editing, and each column shows the different
aspects of evaluation. In the column (a), it is observed that, with the
exception of gender, most edits retain high speaker similarity, indicating
that the majority of the latent directions induced are within-speaker changes.
In the column (b), which measures speech quality, it is evident that editing
does not compromise the quality of the speech much; most MOS of the edited
files exceed 4. In the column (c), which examines the identifiability of the
editing, it is apparent that changes in internal representation are quite
noticeable to the listener, as indicated by the very high successful ID rate
in the first row of the (c) column. Nevertheless, we noticed that manipulating
external code is notably more challenging than adjusting internal code. This
is particularly apparent for more nuanced attributes such as tone and emotion,
where the successful ID rate experiences a significant drop between the two
rows of the column (c). Although Success ID Rate decreases in all instances,
different encoders exhibit distinct behaviors, with NFA and supervised speaker
encoder demonstrating more robust editing capabilities.

###  5.3 Zero-shot Voice Conversion and Multi-Speaker TTS Performance

In addition to speaker generation and editing, it is straightforward to apply
VoxGenesis to voice conversion and multi-speaker TTS tasks. Given that the
majority of VC and TTS systems utilize embeddings from discriminative speaker
encoders, exploring the performance of an unsupervised approach like NFA-
VoxGenesis, which is trained without using any speaker labels, is quite
interesting. Therefore, the focus of this evaluation is primarily on NFA-
VoxGenesis, and its performance is compared with the state-of-the-art the
voice conversion system, Speech Resynthesis (Polyak et al., 2021), and the
state-of-the-art multi-speaker TTS system, VITS (Kim et al., 2021).

For zero-shot voice conversion, we randomly selected 15 speakers from
LibriTTS-R (Koizumi et al., 2023) test split. We assessed the capability of
the model to retain content and maintain speaker fidelity. This is measured by
the Word Error Rate (WER) and Equal Error Rate (EER) using a pre-trained ASR
(Ravanelli et al., 2021) and an (Snyder et al., 2018) model, respectively,
alongside the speech naturalness, measured by MOS. The results are documented
in Table 2. As can be seen from Table 2, VoxGenesis and Speech Resynthesis
exhibit comparable performance in content preservation, as measured by WER.
This is anticipated since both VoxGenesis and Speech Resynthesis employ a
HuBERT-based model to extract content information. Regarding speaker fidelity,
NFA-VoxGenesis surpasses Speech Resynthesis in terms of EER, indicating that
the generative speaker encoder of NFA maintains speaker information more
effectively than the discriminative speaker encoder in Speech Resynthesis.
Additionally, the overall speech quality of NFA-VoxGenesis is superior to that
of Speech Resynthesis, as reflected by the higher MOS score. For multi-speaker
TTS, we assess the generated speech with a focus on speaker MOS, where
evaluators appraise the similarity between the generated speakers and the
ground truth speakers, putting aside other aspects such as content and
grammar. Additionally, we employ general MOS to measure the overall quality
and naturalness of the speech. As indicated in Table 3, VoxGenesis achieves
higher MOS scores in both speaker similarity and naturalness. We observed that
VoxGenesis preserves speakers characteristics and intonation better, despite
the absence of speaker labels during the training.

Method | WER $\downarrow$ | EER $\downarrow$ | MOS $\uparrow$
---|---|---|---
Ground Truth | 4.14 | 4.62 | $4.3\pm 0.09$
Speech Resynthesis (Polyak et al., 2021) | 7.54 | 6.23 | $3.77\pm 0.08$
NFA-VoxGenesis | 7.56 | 5.75 |  4.21$\pm$0.07
Table 2: Zero-shot VC results. Method | Spk. MOS $\uparrow$ | MOS $\uparrow$
---|---|---
NFA-VoxGenesis + Discrete-Tacotron2 | 4.03$\pm$0.09 | 4.15$\pm$0.08
VITS (Kim et al., 2021) | 3.63$\pm$0.20 | 3.8$\pm$0.09
Table 3: Multi-speaker TTS evaluation.

##  6 Conclusions

In this paper, we introduced VoxGenesis, a deep generative model tailored for
voice generation and editing. We demonstrated that VoxGenesis is capable of
generating realistic speakers with distinct characteristics. It can also
uncover significant, human-interpretable speaker variations that are hard to
obtain labels. Furthermore, we demonstrated that VoxGenesis is adept at
performing zero-shot voice conversion and can be effectively utilized as both
a vocoder and a speaker encoder in multi-speaker TTS.

## References

  * Baevski et al. (2020) Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  In _Proc. Advances in Neural Information Processing Systems_ , 2020\.
  * Borsos et al. (2023) Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi.  Soundstorm: Efficient parallel audio generation.  _arXiv preprint arXiv:2305.09636_ , 2023.
  * Brock et al. (2018) Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  _arXiv preprint arXiv:1809.11096_ , 2018.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in Neural Information Processing Systems_ , 33:1877–1901, 2020.
  * Casanova et al. (2022) Edresson Casanova, Julian Weber, Christopher D Shulby, Arnaldo Candido Junior, Eren Gölge, and Moacir A Ponti.  Yourtts: Towards zero-shot multi-speaker TTS and zero-shot voice conversion for everyone.  In _International Conference on Machine Learning_ , pp. 2709–2720, 2022.
  * Chen et al. (2020) Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan.  Wavegrad: Estimating gradients for waveform generation.  _arXiv preprint arXiv:2009.00713_ , 2020.
  * Choi et al. (2021) Hyeong-Seok Choi, Juheon Lee, Wansoo Kim, Jie Lee, Hoon Heo, and Kyogu Lee.  Neural analysis and synthesis: Reconstructing speech from self-supervised representations.  _Advances in Neural Information Processing Systems_ , 34:16251–16265, 2021.
  * Défossez et al. (2022) Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi.  High fidelity neural audio compression.  _arXiv preprint arXiv:2210.13438_ , 2022.
  * Dhariwal & Nichol (2021) Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat GANs on image synthesis.  _Advances in Neural Information Processing Systems_ , 34:8780–8794, 2021.
  * Gibiansky et al. (2017) Andrew Gibiansky, Sercan Arik, Gregory Diamos, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, and Yanqi Zhou.  Deep voice 2: Multi-speaker neural text-to-speech.  _Advances in Neural Information Processing Systems_ , 30, 2017.
  * Goodfellow et al. (2014) Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio.  Generative adversarial nets.  In _Advances in Neural Information Processing Systems_ , pp. 2672–2680, 2014.
  * Härkönen et al. (2020) Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris.  Ganspace: Discovering interpretable gan controls.  _Advances in neural information processing systems_ , 33:9841–9850, 2020.
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  GANs trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in Neural Information Processing Systems_ , 30, 2017.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing systems_ , 33:6840–6851, 2020.
  * Hsu et al. (2017) Wei-Ning Hsu, Yu Zhang, and James Glass.  Unsupervised learning of disentangled and interpretable representations from sequential data.  _Advances in Neural Information Processing systems_ , 30, 2017.
  * Hsu et al. (2021) Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE ACM Trans. Audio Speech Lang. Process._ , 29:3451–3460, 2021.
  * Jia et al. (2018a) Ye Jia, Yu Zhang, Ron Weiss, Quan Wang, Jonathan Shen, Fei Ren, Patrick Nguyen, Ruoming Pang, Ignacio Lopez Moreno, Yonghui Wu, et al.  Transfer learning from speaker verification to multispeaker text-to-speech synthesis.  _Advances in Neural Information Processing Systems_ , 31, 2018a.
  * Jia et al. (2018b) Ye Jia, Yu Zhang, Ron J. Weiss, Quan Wang, Jonathan Shen, Fei Ren, Zhifeng Chen, Patrick Nguyen, Ruoming Pang, Ignacio Lopez-Moreno, and Yonghui Wu.  Transfer learning from speaker verification to multispeaker text-to-speech synthesis.  In _Advances in Neural Information Processing Systems_ , pp. 4485–4495, 2018b.
  * Kaneko et al. (2019) Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, and Nobukatsu Hojo.  CycleGAN-VC2: Improved CycleGAN-based non-parallel voice conversion.  In _IEEE International Conference on Acoustics, Speech and Signal Processing_ , pp. 6820–6824, 2019.
  * Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 4401–4410, 2019.
  * Kim et al. (2021) Jaehyeon Kim, Jungil Kong, and Juhee Son.  Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech.  In _International Conference on Machine Learning_ , pp. 5530–5540. PMLR, 2021.
  * Kingma & Welling (2014) Diederik P. Kingma and Max Welling.  Auto-encoding variational bayes.  In _2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings_ , 2014.
  * Kingma & Dhariwal (2018) Durk P Kingma and Prafulla Dhariwal.  Glow: Generative flow with invertible 1x1 convolutions.  _Advances in Neural Information Processing Systems_ , 31, 2018.
  * Koizumi et al. (2023) Yuma Koizumi, Heiga Zen, Shigeki Karita, Yifan Ding, Kohei Yatabe, Nobuyuki Morioka, Michiel Bacchiani, Yu Zhang, Wei Han, and Ankur Bapna.  Libritts-r: A restored multi-speaker text-to-speech corpus.  _arXiv preprint arXiv:2305.18802_ , 2023.
  * Kong et al. (2020) Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae.  Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis.  _Advances in Neural Information Processing Systems_ , 33:17022–17033, 2020.
  * Kumar et al. (2019) Kundan Kumar, Rithesh Kumar, Thibault De Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron C Courville.  Melgan: Generative adversarial networks for conditional waveform synthesis.  _Advances in neural information processing systems_ , 32, 2019.
  * Kumar et al. (2023) Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar.  High-fidelity audio compression with improved rvqgan.  _arXiv preprint arXiv:2306.06546_ , 2023.
  * Le et al. (2023) Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al.  Voicebox: Text-guided multilingual universal speech generation at scale.  _arXiv preprint arXiv:2306.15687_ , 2023.
  * Lin et al. (2023) Weiwei Lin, Chenhang He, Man Wai Mak, and Youzhi Tu.  Self-supervised Neural Factor Analysis for Disentangling Utterance-level Speech Representations.  In Proceedings of the International Conference on Machine Learning, 2023.
  * Oord et al. (2016) Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu.  Wavenet: A generative model for raw audio.  _arXiv preprint arXiv:1609.03499_ , 2016.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli.  fairseq: A fast, extensible toolkit for sequence modeling.  _arXiv preprint arXiv:1904.01038_ , 2019.
  * Polyak et al. (2021) Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux.  Speech resynthesis from discrete disentangled self-supervised representations.  _arXiv preprint arXiv:2104.00355_ , 2021.
  * Prenger et al. (2019) Ryan Prenger, Rafael Valle, and Bryan Catanzaro.  Waveglow: A flow-based generative network for speech synthesis.  In _IEEE International Conference on Acoustics, Speech and Signal Processing_ , pp. 3617–3621, 2019.
  * Qian et al. (2019) Kaizhi Qian, Yang Zhang, Shiyu Chang, Xuesong Yang, and Mark Hasegawa-Johnson.  AutoVC: Zero-shot voice style transfer with only autoencoder loss.  In _International Conference on Machine Learning_ , pp. 5210–5219. PMLR, 2019.
  * Qian et al. (2020) Kaizhi Qian, Yang Zhang, Shiyu Chang, Mark Hasegawa-Johnson, and David Cox.  Unsupervised speech decomposition via triple information bottleneck.  In _International Conference on Machine Learning_ , pp. 7836–7846. PMLR, 2020.
  * Ravanelli et al. (2021) Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, Ju-Chieh Chou, Sung-Lin Yeh, Szu-Wei Fu, Chien-Feng Liao, Elena Rastorgueva, François Grondin, William Aris, Hwidong Na, Yan Gao, Renato De Mori, and Yoshua Bengio.  SpeechBrain: A general-purpose speech toolkit, 2021.  arXiv:2106.04624.
  * Ren et al. (2019) Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu.  Fastspeech: Fast, robust and controllable text to speech.  _Advances in neural information processing systems_ , 32, 2019.
  * Schneider et al. (2019) Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli.  wav2vec: Unsupervised pre-training for speech recognition.  In _Proc. Interspeech 2019_ , pp. 3465–3469, 2019.
  * Shen et al. (2018) Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al.  Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions.  In _International Conference on Acoustics, Speech and Signal Processing_ , pp. 4779–4783, 2018.
  * Snyder et al. (2018) David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur.  X-vectors: Robust DNN embeddings for speaker recognition.  In _International Conference on Acoustics, Speech and Signal Processing_ , pp. 5329–5333, 2018.
  * Stanton et al. (2022) Daisy Stanton, Matt Shannon, Soroosh Mariooryad, RJ Skerry-Ryan, Eric Battenberg, Tom Bagby, and David Kao.  Speaker generation.  In _IEEE International Conference on Acoustics, Speech and Signal Processing_ , pp. 7897–7901, 2022.
  * Tan et al. (2022) Xu Tan, Jiawei Chen, Haohe Liu, Jian Cong, Chen Zhang, Yanqing Liu, Xi Wang, Yichong Leng, Yuanhao Yi, Lei He, et al.  Naturalspeech: End-to-end text to speech synthesis with human-level quality.  _arXiv preprint arXiv:2205.04421_ , 2022.
  * Voynov & Babenko (2020a) Andrey Voynov and Artem Babenko.  Unsupervised discovery of interpretable directions in the gan latent space.  In _International Conference on Machine Learning_ , pp. 9786–9796. PMLR, 2020a.
  * Voynov & Babenko (2020b) Andrey Voynov and Artem Babenko.  Unsupervised discovery of interpretable directions in the gan latent space.  In _International conference on machine learning_ , pp. 9786–9796. PMLR, 2020b.
  * (45) Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al.  Neural codec language models are zero-shot text to speech synthesizers, 2023.  _URL: https://arxiv. org/abs/2301.02111. doi: doi_ , 10.
  * Wang et al. (2018) Yuxuan Wang, Daisy Stanton, Yu Zhang, RJ-Skerry Ryan, Eric Battenberg, Joel Shor, Ying Xiao, Ye Jia, Fei Ren, and Rif A Saurous.  Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis.  In _International Conference on Machine Learning_ , pp. 5180–5189. PMLR, 2018.
  * Yadav et al. (2023) Amit Kumar Singh Yadav, Kratika Bhagtani, Ziyue Xiang, Paolo Bestagini, Stefano Tubaro, and Edward J Delp.  Dsvae: Interpretable disentangled representation for synthetic speech detection.  _arXiv preprint arXiv:2304.03323_ , 2023.
  * Zeghidour et al. (2021) Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi.  Soundstream: An end-to-end neural audio codec.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 30:495–507, 2021.
  * Zen et al. (2019) Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu.  LibriTTS: A corpus derived from LibriSpeech for text-to-speech.  _arXiv preprint arXiv:1904.02882_ , 2019.

##  Appendix A Appendix

Model | Input to | Encoder | Speaker | Speaker | Speaker | Speaker
---|---|---|---|---|---|---
| GAN | Objective | Labels | Generation | Encoding | Edit
Vanilla-VoxGenesis | Noise | NA | No | Yes | No | Internal
NFA-VoxGenesis | Emb. | Likelihood | No | Yes | Yes | Any
CL-VoxGenesis | Emb. | Contrastive | No | Yes | Yes | Any
\+ KL-divergence
CE-VoxGenesis | Emb. | Cross-entropy | Required | Yes | Yes | Any
\+ KL-divergence
Table 4: Summary of VoxGenesis models with different speaker encoders.