--- library_name: transformers license: llama3.1 language: - th - en pipeline_tag: text-generation --- # Typhoon2-Audio
Typhoon2-Audio
**Typhoon2-Audio** is an end-to-end speech-to-speech model architecture capable of processing audio, speech, and text inputs and generating both text and speech outputs simultaneously. It is optimized specifically for Thai and English languages. - **GitHub**: https://github.com/scb-10x/typhoon2-audio/ - **Demo**: [Not ready yet] - **Paper**: https://arxiv.org/abs/2412.13702 ## Model Description - **Model type**: The LLM is based on Typhoon2 LLM. - **Requirement**: transformers 4.38.0 or newer. - **Primary Language(s)**: Thai 🇹🇭 and English 🇬🇧 - **License-Speech-Input & LLM**: [Llama 3.1 Community License](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/LICENSE) - **License-Speech-Output**: [CC-BY-NC](https://creativecommons.org/licenses/by-nc/4.0/) ## Usage ### Load Model ```python import torch from transformers import AutoModel model = AutoModel.from_pretrained( "scb10x/llama3.1-typhoon2-audio-8b-instruct", torch_dtype=torch.float16, trust_remote_code=True ) model.to("cuda") ``` ### Inference - Single turn example ```python conversation = [ {"role": "system", "content": "You are a helpful female assistant named ไต้ฝุ่น."}, { "role": "user", "content": [ { "type": "audio", "audio_url": "examples/tmp-2860cd0a094b64043226167340af03a3.wav", }, {"type": "text", "text": "Transcribe this audio"}, ], }, ] x = model.generate( conversation=conversation, max_new_tokens=500, do_sample=True, num_beams=1, top_p=0.9, repetition_penalty=1.0, length_penalty=1.0, temperature=0.7, ) # x => x['text'] (text), x['audio'] (numpy array) # to save the audio output # import soundfile as sf # sf.write("examples/speechout.wav", x["audio"]["array"], x["audio"]["sampling_rate"]) ``` ### Inference - Multi turn example ```python conversation_multi_turn = [ { "role": "system", "content": "You are a helpful female assistant named ไต้ฝุ่น. Respond conversationally to the speech provided in the language it is spoken in.", }, { "role": "user", "content": [ { "type": "audio", "audio_url": "examples/tmp-2860cd0a094b64043226167340af03a3.wav", # บอกชื่อเมืองใหญ่ๆในอเมริกามาให้หน่อยสิ -- "List some names of US cities" }, { "type": "text", "text": "", }, ], }, { "role": "assistant", "content": [ { "type": "text", "text": "โอเคค่ะ, ฉันจะบอกชื่อเมืองใหญ่ๆ ในอเมริกาให้คุณฟัง:\n\n1. นิวยอร์ก\n2. ลอสแอนเจลิส\n3. ชิคาโก\n4. ฮิวสตัน\n5. ฟิลาเดลเฟีย\n6. บอสตัน\n7. ซานฟรานซิสโก\n8. วอชิงตัน ดี.ซี. (Washington D.C.)\n9. แอตแลนต้า\n10. ซีแอตเทิล\n\nถ้าคุณต้องการข้อมูลเพิ่มเติมหรือมีคำถามอื่นๆ กรุณาถามได้เลยค่ะ'", }, ], }, { "role": "user", "content": [ { "type": "audio", "audio_url": "examples/tmp-2284cd76e1c875525ff75327a2fc3610.wav", # แล้วถ้าเป็นประเทศอังกฤษล่ะ -- "How about the UK" }, ], }, ] x = model.generate(conversation=conversation_multi_turn) # x => x['text'] (text), x['audio'] (numpy array) # to save the audio output # import soundfile as sf # sf.write("examples/speechout.wav", x["audio"]["array"], x["audio"]["sampling_rate"]) ``` ### TTS functionality ```python y = model.synthesize_speech("Hello, my name is ไต้ฝุ่น I am a language model specialized in Thai") # y => numpy array ``` ## Evaluation Results - **1) Audio and Speech Understanding** | Model | ASR-en (WER↓) | ASR-th (WER↓) | En2Th (BLEU↑) | X2Th (BLEU↑) | Th2En (BLEU↑) | |:----------------------------|:-------------------|:--------------|:--------------|:-------------|:--------------| | SALMONN-13B | 5.79 | 98.07 | 0.07 | 0.10 | 14.97 | | DiVA-8B | 30.28 | 65.21 | 9.82 | 5.31 | 7.97 | | Gemini-1.5-pro-001 | 5.98 | 13.56 | 20.69 | 13.52 | 22.54 | | Typhoon-Audio | 8.72 | 14.17 | 17.52 | 10.67 | 24.14 | | Typhoon2-Audio | 5.83 | 14.04 | 27.15 | 15.93 | 33.25 | | Model | Gender-th (Acc) | SpokenQA-th (F1) | SpeechInstruct-(en,th) | |:-------------------------------|:---------------|:-------------------|:-------------------| | SALMONN-13B | 93.26 | 2.95 | 2.47, 1.18 | | DiVA-8B | 50.12 | 15.13 | 6.81, 2.68 | | Gemini-1.5-pro-001 | 81.32 | 62.10 | 3.24, 3.93 | | Typhoon-Audio | 93.74 | 64.60 | 5.62, 6.11 | | Typhoon2-Audio | 75.65 | 70.01 | 6.00, 6.79 | - **2) Speech-to-Speech Evaluation** - 2.1) *Content Generation* | Model | SpeechIF(En)-Quality | SpeechIF(En)-Style | SpeechIF(Th)-Quality | SpeechIF(Th)-Style | |:------------------------------|:---------------|:-------------------|:-------------------|:-------------------| | Llama-Omni | 5.15 | 5.79 | 1.71 | 2.14 | | GPT-4o-Audio | 6.82 | 7.86 | 6.66 | 8.07 | | Typhoon2-Audio | 4.92 | 5.39 | 7.19 | 8.04 | - 2.2) *Speech Quality* | Model | SpeechIF(En)-CER | SpeechIF(En)-UTMOS | SpeechIF(Th)-CER | SpeechIF(Th)-UTMOS | |:------------------------------|:---------------|:-------------------|:-------------------|:-------------------| | Llama-Omni* | 3.40 | 3.93 | 6.30 | 3.93 | | GPT-4o-Audio | 3.20 | 3.65 | 8.05 | 3.46 | | Typhoon2-Audio | 26.50 | 2.29 | 8.67 | 2.35 | *Note that Llama-Omni does not generate Thai text/speech, so it has low CER and high UTMOS due to the outputs being English. ## Intended Uses & Limitations This model is experimental and may not always follow human instructions accurately, making it prone to generating hallucinations. Additionally, the model lacks moderation mechanisms and may produce harmful or inappropriate responses. Developers should carefully assess potential risks based on their specific applications. ## Follow us & Support - https://twitter.com/opentyphoon - https://discord.gg/CqyBscMFpg ## Acknowledgements We would like to thank the SALMONN team and the Llama-Omni team for open-sourcing their code and data, and thanks to the Biomedical and Data Lab at Mahidol University for releasing the fine-tuned Whisper that allowed us to adopt its encoder. Thanks to many other open-source projects for their useful knowledge sharing, data, code, and model weights. ## Typhoon Team Potsawee Manakul, Warit Sirichotedumrong, Kunat Pipatanakul, Pittawat Taveekitworachai, Natapong Nitarach, Surapon Nonesung, Teetouch Jaknamon, Parinthapat Pengpun, Pittawat Taveekitworachai, Adisai Na-Thalang, Sittipong Sripaisarnmongkol, Krisanapong Jirayoot, Kasima Tharnpipitchai ## **Citation** - If you find Typhoon2 useful for your work, please cite it using: ``` @misc{typhoon2, title={Typhoon 2: A Family of Open Text and Multimodal Thai Large Language Models}, author={Kunat Pipatanakul and Potsawee Manakul and Natapong Nitarach and Warit Sirichotedumrong and Surapon Nonesung and Teetouch Jaknamon and Parinthapat Pengpun and Pittawat Taveekitworachai and Adisai Na-Thalang and Sittipong Sripaisarnmongkol and Krisanapong Jirayoot and Kasima Tharnpipitchai}, year={2024}, eprint={2412.13702}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2412.13702}, } ```