Update README.md

README.md

@@ -1,208 +1,246 @@
 ---
-license: cc-by-nc-sa-4.0
-language:
-- en
-- zh
+license: apache-2.0
 tags:
-- text-to-speech
-library_tag: spark-tts
-base_model:
-- SparkAudio/Spark-TTS-0.5B
----
-<div>
-  <p style="margin-bottom: 0; margin-top: 0;">
-    <strong>See <a href="https://huggingface.co/collections/unsloth/text-to-speech-tts-models-68007ab12522e96be1e02155">our collection</a> for all our TTS model uploads.</strong>
-  </p>
-  <p style="margin-bottom: 0;">
-    <em>Learn to fine-tune TTS models - <a href="https://docs.unsloth.ai/basics/text-to-speech-tts-fine-tuning">Read our Guide</a>.</em>
-  </p>
-<p style="margin-top: 0;margin-bottom: 0;">
-    <em><a href="https://docs.unsloth.ai/basics/unsloth-dynamic-v2.0-gguf">Unsloth Dynamic 2.0</a> achieves superior accuracy & outperforms other leading quants.</em>
-  </p>
-  <div style="display: flex; gap: 5px; align-items: center; ">
-    <a href="https://github.com/unslothai/unsloth/">
-      <img src="https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png" width="133">
-    </a>
-    <a href="https://discord.gg/unsloth">
-      <img src="https://github.com/unslothai/unsloth/raw/main/images/Discord%20button.png" width="173">
-    </a>
-    <a href="https://docs.unsloth.ai/basics/text-to-speech-tts-fine-tuning">
-      <img src="https://raw.githubusercontent.com/unslothai/unsloth/refs/heads/main/images/documentation%20green%20button.png" width="143">
-    </a>
-  </div>
-<h1 style="margin-top: 0rem;">✨ Run & Fine-tune TTS models with Unsloth!</h1>
-</div>
-
-- Fine-tune TTS models for free using our Google [Colab notebooks here](https://docs.unsloth.ai/get-started/unsloth-notebooks#text-to-speech-tts-notebooks)!
-- Read our Blog about TTS support: [unsloth.ai/blog/tts](https://docs.unsloth.ai/basics/text-to-speech-tts-fine-tuning)
-
-| Unsloth supports          |    Free Notebooks                                                                                           | Performance | Memory use |
-|-----------------|--------------------------------------------------------------------------------------------------------------------------|-------------|----------|
-| **Spark-TTS**      | [▶️ Start on Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Spark_TTS_(0_5B).ipynb)               | 1.5x faster | 58% less |
-| **Whisper Large V3**      | [▶️ Start on Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Whisper.ipynb)               | 1.5x faster | 50% less |
-| **Qwen3 (14B)**      | [▶️ Start on Colab](https://docs.unsloth.ai/get-started/unsloth-notebooks)               | 2x faster | 70% less |
-| **Llama 3.2 Vision (11B)**      | [▶️ Start on Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(11B)-Vision.ipynb)               | 1.8x faster | 50% less |
-
-<div align="center">
-    <h1>
-    Spark-TTS
-    </h1>
-    <p>
-    Official model for <br>
-    <b><em>Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens</em></b>
-    </p>
-    <p>
-    <img src="src/logo/SparkTTS.jpg" alt="Spark-TTS Logo" style="width: 200px; height: 200px;">
-    </p>
-</div>
-
-
-## Spark-TTS 🔥
-
-### 👉🏻 [Spark-TTS Demos](https://sparkaudio.github.io/spark-tts/) 👈🏻
-
-### 👉🏻 [Github Repo](https://github.com/SparkAudio/Spark-TTS) 👈🏻
-
-### 👉🏻 [Paper](https://arxiv.org/pdf/2503.01710) 👈🏻
-
-### Overview
-
-Spark-TTS is an advanced text-to-speech system that uses the power of large language models (LLM) for highly accurate and natural-sounding voice synthesis. It is designed to be efficient, flexible, and powerful for both research and production use.
-
-### Key Features
-
-- **Simplicity and Efficiency**: Built entirely on Qwen2.5, Spark-TTS eliminates the need for additional generation models like flow matching. Instead of relying on separate models to generate acoustic features, it directly reconstructs audio from the code predicted by the LLM. This approach streamlines the process, improving efficiency and reducing complexity.
-- **High-Quality Voice Cloning**: Supports zero-shot voice cloning, which means it can replicate a speaker's voice even without specific training data for that voice. This is ideal for cross-lingual and code-switching scenarios, allowing for seamless transitions between languages and voices without requiring separate training for each one.
-- **Bilingual Support**: Supports both Chinese and English, and is capable of zero-shot voice cloning for cross-lingual and code-switching scenarios, enabling the model to synthesize speech in multiple languages with high naturalness and accuracy.
-- **Controllable Speech Generation**: Supports creating virtual speakers by adjusting parameters such as gender, pitch, and speaking rate.
-
+  - spark-tts
+  - text-to-speech
+  - nonverbal
+  - emotional
+  - audio
+  - speech-synthesis
+  - huggingface
+language:
+  - en
+model-index:
+  - name: SparkNV-Voice
+    results: []
 ---
 
-<table align="center">
-  <tr>
-    <td align="center"><b>Inference Overview of Voice Cloning</b><br><img src="src/figures/infer_voice_cloning.png" width="80%" /></td>
-  </tr>
-  <tr>
-    <td align="center"><b>Inference Overview of Controlled Generation</b><br><img src="src/figures/infer_control.png" width="80%" /></td>
-  </tr>
-</table>
-
-
-## Install
-**Clone and Install**
-
-- Clone the repo
-``` sh
-git clone https://github.com/SparkAudio/Spark-TTS.git
-cd Spark-TTS
-```
-
-- Install Conda: please see https://docs.conda.io/en/latest/miniconda.html
-- Create Conda env:
-
-``` sh
-conda create -n sparktts -y python=3.12
-conda activate sparktts
-pip install -r requirements.txt
-# If you are in mainland China, you can set the mirror as follows:
-pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host=mirrors.aliyun.com
-```
-
-**Model Download**
-
-Download via python:
-```python
-from huggingface_hub import snapshot_download
-
-snapshot_download("SparkAudio/Spark-TTS-0.5B", local_dir="pretrained_models/Spark-TTS-0.5B")
-```
+# 🔊 SparkNV-Voice
 
-Download via git clone:
-```sh
-mkdir -p pretrained_models
+**SparkNV-Voice** is a fine-tuned version of the [Spark-TTS](https://huggingface.co/suno-ai/spark-tts) model trained on the [NonverbalTTS](https://huggingface.co/datasets/deepvk/NonverbalTTS) dataset. It enables expressive speech synthesis with **nonverbal cues** (like laughter, sighs, sneezing, etc.) and rich emotional tone.
 
-# Make sure you have git-lfs installed (https://git-lfs.com)
-git lfs install
+Built for applications that require **natural, human-like vocalization**, this model produces speech with **semantic tokens** and **global prosody control** using BiCodec detokenization.
 
-git clone https://huggingface.co/SparkAudio/Spark-TTS-0.5B pretrained_models/Spark-TTS-0.5B
-```
+---
 
-**Basic Usage**
+## 🧾 Model Details
 
-You can simply run the demo with the following commands:
-``` sh
-cd example
-bash infer.sh
-```
+- **Base**: `suno-ai/spark-tts`
+- **Dataset**: [`deepvk/NonverbalTTS`](https://huggingface.co/datasets/deepvk/NonverbalTTS)
+- **Architecture**: Causal Language Model + BiCodec for audio token generation
+- **Language**: English
+- **Voice**: Single-speaker (no multi-speaker conditioning)
 
-Alternatively, you can directly execute the following command in the command line to perform inference：
+---
 
-``` sh
-python -m cli.inference \
-    --text "text to synthesis." \
-    --device 0 \
-    --save_dir "path/to/save/audio" \
-    --model_dir pretrained_models/Spark-TTS-0.5B \
-    --prompt_text "transcript of the prompt audio" \
-    --prompt_speech_path "path/to/prompt_audio"
-```
+## 🛠 Installation
 
-**UI Usage**
+To run this model, install the required dependencies:
 
-You can start the UI interface by running `python webui.py`, which allows you to perform Voice Cloning and Voice Creation. Voice Cloning supports uploading reference audio or directly recording the audio.
+```bash
+pip install --no-deps bitsandbytes accelerate xformers==0.0.29.post3 peft trl triton cut_cross_entropy unsloth_zoo
+pip install sentencepiece protobuf "datasets>=3.4.1,<4.0.0" "huggingface_hub>=0.34.0" hf_transfer 
+pip install --no-deps unsloth 
+git clone https://github.com/SparkAudio/Spark-TTS
+pip install omegaconf einx
+````
 
 
-| **Voice Cloning** | **Voice Creation** |
-|:-------------------:|:-------------------:|
-| ![Image 1](src/figures/gradio_TTS.png) | ![Image 2](src/figures/gradio_control.png) |
+---
 
+## 🚀 Inference Code
 
-## To-Do List
+```python
+import torch
+import re
+import numpy as np
+from typing import Dict, Any
+import torchaudio.transforms as T
+from unsloth import FastModel
+import sys
+sys.path.append('Spark-TTS')
+from sparktts.models.audio_tokenizer import BiCodecTokenizer
+from huggingface_hub import snapshot_download
 
-- [x] Release the Spark-TTS paper.
-- [ ] Release the training code.
-- [ ] Release the training dataset, VoxBox.
+# Download model and code
+snapshot_download("yasserrmd/SparkNV-Voice", local_dir = "SparkNV-Voice")
+
+
+max_seq_length = 2048 # Choose any for long context!
+model, tokenizer = FastModel.from_pretrained(
+    model_name = "SparkNV-Voice",
+    max_seq_length = max_seq_length,
+    dtype = torch.float32, # Spark seems to only work on float32 for now
+    full_finetuning = True, # We support full finetuning now!
+    load_in_4bit = False,
+    #token = "hf_...", # use one if using gated models like meta-llama/Llama-2-7b-hf
+)
+
+FastModel.for_inference(model) # Enable native 2x faster inference
+
+audio_tokenizer = BiCodecTokenizer("SparkNV-Voice", "cuda")
+audio_tokenizer.model.to("cuda")
+
+input_text = "Hey there, my name is Yasser, and I'm a 🌬️ speech generation model that can sound like a person."
+chosen_voice = None # None for single-speaker
+
+@torch.inference_mode()
+def generate_speech_from_text(
+    text: str,
+    temperature: float = 0.8,   # Generation temperature
+    top_k: int = 50,            # Generation top_k
+    top_p: float = 1,        # Generation top_p
+    max_new_audio_tokens: int = 2048, # Max tokens for audio part
+    device: torch.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+) -> np.ndarray:
+    """
+    Generates speech audio from text using default voice control parameters.
+
+    Args:
+        text (str): The text input to be converted to speech.
+        temperature (float): Sampling temperature for generation.
+        top_k (int): Top-k sampling parameter.
+        top_p (float): Top-p (nucleus) sampling parameter.
+        max_new_audio_tokens (int): Max number of new tokens to generate (limits audio length).
+        device (torch.device): Device to run inference on.
+
+    Returns:
+        np.ndarray: Generated waveform as a NumPy array.
+    """
+
+    torch.compiler.reset()
+
+    prompt = "".join([
+        "<|task_tts|>",
+        "<|start_content|>",
+        text,
+        "<|end_content|>",
+        "<|start_global_token|>"
+    ])
+
+    model_inputs = tokenizer([prompt], return_tensors="pt").to(device)
+
+    print("Generating token sequence...")
+    generated_ids = model.generate(
+        **model_inputs,
+        max_new_tokens=max_new_audio_tokens, # Limit generation length
+        do_sample=True,
+        temperature=temperature,
+        top_k=top_k,
+        top_p=top_p,
+        eos_token_id=tokenizer.eos_token_id, # Stop token
+        pad_token_id=tokenizer.pad_token_id # Use models pad token id
+    )
+    print("Token sequence generated.")
+
+
+    generated_ids_trimmed = generated_ids[:, model_inputs.input_ids.shape[1]:]
+
+
+    predicts_text = tokenizer.batch_decode(generated_ids_trimmed, skip_special_tokens=False)[0]
+    # print(f"\nGenerated Text (for parsing):\n{predicts_text}\n") # Debugging
+
+    # Extract semantic token IDs using regex
+    semantic_matches = re.findall(r"<\|bicodec_semantic_(\d+)\|>", predicts_text)
+    if not semantic_matches:
+        print("Warning: No semantic tokens found in the generated output.")
+        # Handle appropriately - perhaps return silence or raise error
+        return np.array([], dtype=np.float32)
+
+    pred_semantic_ids = torch.tensor([int(token) for token in semantic_matches]).long().unsqueeze(0) # Add batch dim
+
+    # Extract global token IDs using regex (assuming controllable mode also generates these)
+    global_matches = re.findall(r"<\|bicodec_global_(\d+)\|>", predicts_text)
+    if not global_matches:
+         print("Warning: No global tokens found in the generated output (controllable mode). Might use defaults or fail.")
+         pred_global_ids = torch.zeros((1, 1), dtype=torch.long)
+    else:
+         pred_global_ids = torch.tensor([int(token) for token in global_matches]).long().unsqueeze(0) # Add batch dim
+
+    pred_global_ids = pred_global_ids.unsqueeze(0) # Shape becomes (1, 1, N_global)
+
+    print(f"Found {pred_semantic_ids.shape[1]} semantic tokens.")
+    print(f"Found {pred_global_ids.shape[2]} global tokens.")
+
+
+    # 5. Detokenize using BiCodecTokenizer
+    print("Detokenizing audio tokens...")
+    # Ensure audio_tokenizer and its internal model are on the correct device
+    audio_tokenizer.device = device
+    audio_tokenizer.model.to(device)
+    # Squeeze the extra dimension from global tokens as seen in SparkTTS example
+    wav_np = audio_tokenizer.detokenize(
+        pred_global_ids.to(device).squeeze(0), # Shape (1, N_global)
+        pred_semantic_ids.to(device)           # Shape (1, N_semantic)
+    )
+    print("Detokenization complete.")
+
+    return wav_np
+
+if __name__ == "__main__":
+    print(f"Generating speech for: '{input_text}'")
+    text = f"{chosen_voice}: " + input_text if chosen_voice else input_text
+    generated_waveform = generate_speech_from_text(input_text)
+
+    if generated_waveform.size > 0:
+        import soundfile as sf
+        output_filename = "generated_speech_controllable.wav"
+        sample_rate = audio_tokenizer.config.get("sample_rate", 16000)
+        sf.write(output_filename, generated_waveform, sample_rate)
+        print(f"Audio saved to {output_filename}")
+
+        # Optional: Play in notebook
+        from IPython.display import Audio, display
+        display(Audio(generated_waveform, rate=sample_rate))
+    else:
+        print("Audio generation failed (no tokens found?).")
+````
 
-## Citation
+---
 
-```
-@misc{wang2025sparktts,
-      title={Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens}, 
-      author={Xinsheng Wang and Mingqi Jiang and Ziyang Ma and Ziyu Zhang and Songxiang Liu and Linqin Li and Zheng Liang and Qixi Zheng and Rui Wang and Xiaoqin Feng and Weizhen Bian and Zhen Ye and Sitong Cheng and Ruibin Yuan and Zhixian Zhao and Xinfa Zhu and Jiahao Pan and Liumeng Xue and Pengcheng Zhu and Yunlin Chen and Zhifei Li and Xie Chen and Lei Xie and Yike Guo and Wei Xue},
-      year={2025},
-      eprint={2503.01710},
-      archivePrefix={arXiv},
-      primaryClass={cs.SD},
-      url={https://arxiv.org/abs/2503.01710}, 
-}
-```
+## 🎛️ Supported Nonverbal Cues
 
+The model is fine-tuned on sequences containing:
 
-## ⚠ License Update
+* `<|laughing|>`
+* `<|sighing|>`
+* `<|groaning|>`
+* `<|grunting|>`
+* `<|sniffing|>`
+* `<|sneezing|>`
+* `<|breathing|>`
+* `<|coughing|>`
+* `<|snoring|>`
+* `<|throat_clearing|>`
 
-The model's license has been updated from Apache 2.0 to CC BY-NC-SA due to the licensing terms of some training data.
+You can combine these with your prompt to guide tone/emotion or rely on semantic token generation.
 
-Key Changes:
+---
 
-- The model can only be used for non-commercial purposes.
+## 🧠 Dataset Highlights: `NonverbalTTS`
 
-- Any modifications or derivatives must also be released under CC BY-NC-SA 4.0.
+* 17+ hours of annotated emotional & nonverbal English speech
+* Automatic + human-validated labels
+* Sources: VoxCeleb, Expresso
+* Paper: [arXiv:2507.13155](https://arxiv.org/abs/2507.13155)
 
-- Proper attribution is required when using or modifying the model.
+---
 
-Please ensure compliance with the new license terms.
+## 📜 License
 
+Apache 2.0 — free for commercial and academic use.
 
-## ⚠️ Usage Disclaimer
+---
 
-This project provides a zero-shot voice cloning TTS model intended for academic research, educational purposes, and legitimate applications, such as personalized speech synthesis, assistive technologies, and linguistic research.
+## 🤝 Credits
 
-Please note:
+* Base model: [`suno-ai/spark-tts`](https://huggingface.co/suno-ai/spark-tts)
+* Dataset: [`deepvk/NonverbalTTS`](https://huggingface.co/datasets/deepvk/NonverbalTTS)
+* Author: [`@yasserrmd`](https://huggingface.co/yasserrmd)
 
-- Do not use this model for unauthorized voice cloning, impersonation, fraud, scams, deepfakes, or any illegal activities.
+---
 
-- Ensure compliance with local laws and regulations when using this model and uphold ethical standards.
+## 💬 Feedback & Contributions
 
-- The developers assume no liability for any misuse of this model.
+Open a discussion or issue on this repo. Contributions are welcome!
 
-We advocate for the responsible development and use of AI and encourage the community to uphold safety and ethical principles in AI research and applications. If you have any concerns regarding ethics or misuse, please contact us.