Upload 3 files

README.md

@@ -0,0 +1,148 @@
+---
+license: apache-2.0
+pipeline_tag: text-to-speech
+library_name: zonos
+---
+# Zonos-v0.1
+
+<div align="center">
+<img src="https://github.com/Zyphra/Zonos/blob/main/assets/ZonosHeader.png?raw=true"
+     alt="Title card" 
+     style="width: 500px;
+            height: auto;
+            object-position: center top;">
+</div>
+
+---
+
+Zonos-v0.1 is a leading open-weight text-to-speech model trained on more than 200k hours of varied multilingual speech, delivering expressiveness and quality on par with—or even surpassing—top TTS providers.
+
+Our model enables highly natural speech generation from text prompts when given a speaker embedding or audio prefix, and can accurately perform speech cloning when given a reference clip spanning just a few seconds. The conditioning setup also allows for fine control over speaking rate, pitch variation, audio quality, and emotions such as happiness, fear, sadness, and anger. The model outputs speech natively at 44kHz.
+
+##### For more details and speech samples, check out our blog [here](https://www.zyphra.com/post/beta-release-of-zonos-v0-1)
+
+##### We also have a hosted version available at [playground.zyphra.com/audio](https://playground.zyphra.com/audio)
+
+---
+
+Zonos follows a straightforward architecture: text normalization and phonemization via eSpeak, followed by DAC token prediction through a transformer or hybrid backbone. An overview of the architecture can be seen below.
+
+<div align="center">
+<img src="https://github.com/Zyphra/Zonos/blob/main/assets/ArchitectureDiagram.png?raw=true"
+     alt="Architecture diagram" 
+     style="width: 1000px;
+            height: auto;
+            object-position: center top;">
+</div>
+
+---
+
+## Usage
+
+### Python
+
+```python
+import torch
+import torchaudio
+from zonos.model import Zonos
+from zonos.conditioning import make_cond_dict
+
+model = Zonos.from_pretrained("Zyphra/Zonos-v0.1-hybrid", device="cuda")
+# model = Zonos.from_pretrained("Zyphra/Zonos-v0.1-transformer", device="cuda")
+
+wav, sampling_rate = torchaudio.load("assets/exampleaudio.mp3")
+speaker = model.make_speaker_embedding(wav, sampling_rate)
+
+cond_dict = make_cond_dict(text="Hello, world!", speaker=speaker, language="en-us")
+conditioning = model.prepare_conditioning(cond_dict)
+
+codes = model.generate(conditioning)
+
+wavs = model.autoencoder.decode(codes).cpu()
+torchaudio.save("sample.wav", wavs[0], model.autoencoder.sampling_rate)
+```
+
+### Gradio interface (recommended)
+
+```bash
+uv run gradio_interface.py
+# python gradio_interface.py
+```
+
+This should produce a `sample.wav` file in your project root directory.
+
+_For repeated sampling we highly recommend using the gradio interface instead, as the minimal example needs to load the model every time it is run._
+
+## Features
+
+- Zero-shot TTS with voice cloning: Input desired text and a 10-30s speaker sample to generate high quality TTS output
+- Audio prefix inputs: Add text plus an audio prefix for even richer speaker matching. Audio prefixes can be used to elicit behaviours such as whispering which can otherwise be challenging to replicate when cloning from speaker embeddings
+- Multilingual support: Zonos-v0.1 supports English, Japanese, Chinese, French, and German
+- Audio quality and emotion control: Zonos offers fine-grained control of many aspects of the generated audio. These include speaking rate, pitch, maximum frequency, audio quality, and various emotions such as happiness, anger, sadness, and fear.
+- Fast: our model runs with a real-time factor of ~2x on an RTX 4090
+- Gradio WebUI: Zonos comes packaged with an easy to use gradio interface to generate speech
+- Simple installation and deployment: Zonos can be installed and deployed simply using the docker file packaged with our repository.
+
+## Installation
+
+**At the moment this repository only supports Linux systems (preferably Ubuntu 22.04/24.04) with recent NVIDIA GPUs (3000-series or newer, 6GB+ VRAM).**
+
+See also [Docker Installation](#docker-installation)
+
+#### System dependencies
+
+Zonos depends on the eSpeak library phonemization. You can install it on Ubuntu with the following command:
+
+```bash
+apt install -y espeak-ng
+```
+
+#### Python dependencies
+
+We highly recommend using a recent version of [uv](https://docs.astral.sh/uv/#installation) for installation. If you don't have uv installed, you can install it via pip: `pip install -U uv`.
+
+##### Installing into a new uv virtual environment (recommended)
+
+```bash
+uv sync
+uv sync --extra compile
+```
+
+##### Installing into the system/actived environment using uv
+
+```bash
+uv pip install -e .
+uv pip install -e .[compile]
+```
+
+##### Installing into the system/actived environment using pip
+
+```bash
+pip install -e .
+pip install --no-build-isolation -e .[compile]
+```
+
+##### Confirm that it's working
+
+For convenience we provide a minimal example to check that the installation works:
+
+```bash
+uv run sample.py
+# python sample.py
+```
+
+## Docker installation
+
+```bash
+git clone https://github.com/Zyphra/Zonos.git
+cd Zonos
+
+# For gradio
+docker compose up
+
+# Or for development you can do
+docker build -t Zonos .
+docker run -it --gpus=all --net=host -v /path/to/Zonos:/Zonos -t Zonos
+cd /Zonos
+python sample.py # this will generate a sample.wav in /Zonos
+```

config.json

@@ -0,0 +1,117 @@
+{
+    "backbone": {
+        "d_model": 2048,
+        "d_intermediate": 0,
+        "attn_mlp_d_intermediate": 8192,
+        "n_layer": 46,
+        "ssm_cfg": {
+            "layer": "Mamba2"
+        },
+        "attn_layer_idx": [
+            0,
+            4,
+            8,
+            12,
+            16,
+            20,
+            24,
+            28,
+            32,
+            36,
+            40,
+            44
+        ],
+        "attn_cfg": {
+            "causal": true,
+            "num_heads": 16,
+            "num_heads_kv": 4,
+            "rotary_emb_dim": 128,
+            "qkv_proj_bias": false,
+            "out_proj_bias": false
+        },
+        "rms_norm": false,
+        "residual_in_fp32": false,
+        "norm_epsilon": 1e-05
+    },
+    "prefix_conditioner": {
+        "conditioners": [
+            {
+                "type": "EspeakPhonemeConditioner",
+                "name": "espeak"
+            },
+            {
+                "cond_dim": 128,
+                "uncond_type": "learned",
+                "projection": "linear",
+                "type": "PassthroughConditioner",
+                "name": "speaker"
+            },
+            {
+                "input_dim": 8,
+                "uncond_type": "learned",
+                "type": "FourierConditioner",
+                "name": "emotion"
+            },
+            {
+                "min_val": 0,
+                "max_val": 24000,
+                "uncond_type": "learned",
+                "type": "FourierConditioner",
+                "name": "fmax"
+            },
+            {
+                "min_val": 0,
+                "max_val": 400,
+                "uncond_type": "learned",
+                "type": "FourierConditioner",
+                "name": "pitch_std"
+            },
+            {
+                "min_val": 0,
+                "max_val": 40,
+                "uncond_type": "learned",
+                "type": "FourierConditioner",
+                "name": "speaking_rate"
+            },
+            {
+                "min_val": -1,
+                "max_val": 126,
+                "uncond_type": "learned",
+                "type": "IntegerConditioner",
+                "name": "language_id"
+            },
+            {
+                "input_dim": 8,
+                "min_val": 0.5,
+                "max_val": 0.8,
+                "uncond_type": "learned",
+                "type": "FourierConditioner",
+                "name": "vqscore_8"
+            },
+            {
+                "min_val": -1.0,
+                "max_val": 1000,
+                "uncond_type": "learned",
+                "type": "FourierConditioner",
+                "name": "ctc_loss"
+            },
+            {
+                "min_val": 1,
+                "max_val": 5,
+                "uncond_type": "learned",
+                "type": "FourierConditioner",
+                "name": "dnsmos_ovrl"
+            },
+            {
+                "min_val": 0,
+                "max_val": 1,
+                "uncond_type": "learned",
+                "type": "IntegerConditioner",
+                "name": "speaker_noised"
+            }
+        ],
+        "projection": "linear"
+    },
+    "eos_token_id": 1024,
+    "masked_token_id": 1025
+}

gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text