|
--- |
|
license: mit |
|
language: |
|
- en |
|
pipeline_tag: text-to-audio |
|
tags: |
|
- audiocraft |
|
- audiogen |
|
- styletts2 |
|
- shift |
|
- audeering |
|
- sound |
|
- audio-generation |
|
- text-to-speech |
|
- mimic3 |
|
--- |
|
|
|
|
|
# Affective TTS / SoundScapes |
|
|
|
- [SHIFT TTS tool](https://github.com/audeering/shift) |
|
- Analysis of TTS emotionality [#1](https://huggingface.co/dkounadis/artificial-styletts2/discussions/2) |
|
- Soundscapes `trees, water, ..` via [AudioGen](https://huggingface.co/dkounadis/artificial-styletts2/discussions/3) |
|
- `landscape2soundscape.py` generates soundscape / overlays TTS / creates video from image. |
|
|
|
## Available Voices |
|
|
|
<a href="https://audeering.github.io/shift/">Listen to available voices!</a> |
|
|
|
## Flask API |
|
|
|
<details> |
|
<summary> |
|
Create virtualenv |
|
</summary> |
|
|
|
Clone |
|
|
|
``` |
|
git clone https://huggingface.co/dkounadis/artificial-styletts2 |
|
``` |
|
Install |
|
|
|
``` |
|
virtualenv --python=python3 ~/.envs/.my_env |
|
source ~/.envs/.my_env/bin/activate |
|
cd artificial-styletts2/ |
|
pip install -r requirements.txt |
|
``` |
|
|
|
</details> |
|
|
|
Start Flask |
|
|
|
``` |
|
CUDA_DEVICE_ORDER=PCI_BUS_ID HF_HOME=./hf_home CUDA_VISIBLE_DEVICES=2 python api.py |
|
``` |
|
|
|
## Landscape 2 Soundscape |
|
|
|
The following needs `api.py` to be already running on a tmux session. |
|
|
|
```python |
|
# TTS & soundscape - overlay to .mp4 |
|
python landscape2soundscape.py |
|
``` |
|
|
|
# Videos / Examples |
|
|
|
Video where Native voice is replaced with English TTS voice |
|
|
|
|
|
[](https://www.youtube.com/watch?v=geI1Vqn4QpY) |
|
|
|
## Joint Application of D3.1 & D3.2 |
|
|
|
<a href="https://youtu.be/wWC8DpOKVvQ" rel="Subtitles to Video"></a> |
|
|
|
|
|
From an image and text create a video: |
|
|
|
```python |
|
|
|
python tts.py --text sample.txt --image assets/image_from_T31.jpg |
|
``` |
|
|
|
## Landscape 2 Soundscape |
|
|
|
|
|
|
|
|
|
|
|
```python |
|
# Loads image & text & sound-scene text and creates .mp4 |
|
python landscape2soundscape.py |
|
``` |
|
|
|
For SHIFT demo / Collaboration with [SMB](https://www.smb.museum/home/) |
|
- YouTube Videos |
|
|
|
|
|
[](https://youtu.be/SSi3gUO4GtY) |
|
|
|
[](https://youtu.be/2YjxAPkdXIc) |
|
|
|
[](https://youtu.be/BhMh02knkco) |
|
|
|
|
|
|
|
[](https://youtu.be/a3qk9S87v60) |
|
|
|
[](https://youtu.be/3M0y9OYzDfU) |
|
|
|
[](https://youtu.be/OBY666_By1k) |
|
|
|
[](https://youtu.be/gnGCYLcdLsA) |
|
|
|
[](https://www.youtube.com/watch?v=Y8QyYUgLaCg) |
|
|
|
[](https://youtu.be/XDDzxDSrhb0) |
|
|
|
[](https://youtu.be/I3YYKiUzHpA) |
|
|
|
|
|
|
|
|
|
# Live Demo - Paplay |
|
|
|
Flask |
|
|
|
```python |
|
CUDA_DEVICE_ORDER=PCI_BUS_ID HF_HOME=/data/dkounadis/.hf7/ CUDA_VISIBLE_DEVICES=4 python live_api.py |
|
``` |
|
|
|
Client - Describe any sound with words and it will be played back to you. |
|
|
|
```python |
|
python live_demo.py # will ask text input & play soundscape |
|
``` |
|
|
|
# Simple Demo |
|
|
|
```python |
|
CUDA_DEVICE_ORDER=PCI_BUS_ID HF_HOME=/data/dkounadis/.hf7/ CUDA_VISIBLE_DEVICES=4 python demo.py |
|
``` |