File size: 2,923 Bytes
d0e7717 0755ed0 d0e7717 4d75853 d0e7717 0755ed0 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 |
---
license: mit
tags:
- art
- music
- midi
- emotion
- clip
- multimodal
---
# ARIA - Artistic Rendering of Images into Audio
ARIA is a multimodal AI model that generates MIDI music based on the emotional content of artwork. It uses a CLIP-based image encoder to extract emotional valence and arousal from images, then generates emotionally appropriate music using conditional MIDI generation.
## Model Description
- **Developed by:** Vincent Amato
- **Model type:** Multimodal (Image-to-MIDI) Generation
- **Language(s):** English
- **License:** MIT
- **Parent Model:** Uses CLIP for image encoding and midi-emotion for music generation
- **Repository:** [GitHub](https://github.com/vincentamato/aria)
### Model Architecture
ARIA consists of two main components:
1. A CLIP-based image encoder fine-tuned to predict emotional valence and arousal from images
2. A transformer-based MIDI generation model (midi-emotion) that conditions on these emotional values
The model offers three different conditioning modes:
- `continuous_concat`: Emotions as continuous vectors concatenated to all tokens
- `continuous_token`: Emotions as continuous vectors prepended to sequence
- `discrete_token`: Emotions quantized into discrete tokens
### Usage
The repository contains three variants of the MIDI generation model, each trained with a different conditioning strategy. Each variant includes:
- `model.pt`: The trained model weights
- `mappings.pt`: Token mappings for MIDI generation
- `model_config.pt`: Model configuration
Additionally, `image_encoder.pt` contains the CLIP-based image emotion encoder.
## Intended Use
This model is designed for:
- Generating music that matches the emotional content of artwork
- Exploring emotional transfer between visual and musical domains
- Creative applications in art and music generation
### Limitations
- Music generation quality depends on the emotional interpretation of input images
- Generated MIDI may require human curation for professional use
- Model's emotional understanding is limited to valence-arousal space
## Training Data
The model combines:
1. Image encoder: Uses ArtBench with emotional annotations
2. MIDI generation: Uses the Lakh-Spotify dataset as processed by the midi-emotion project
## Attribution
This project builds upon:
- **midi-emotion** by Serkan Sulun et al. ([GitHub](https://github.com/serkansulun/midi-emotion))
- Paper: "Symbolic music generation conditioned on continuous-valued emotions" ([IEEE Access](https://ieeexplore.ieee.org/document/9762257))
- Citation: S. Sulun, M. E. P. Davies and P. Viana, "Symbolic Music Generation Conditioned on Continuous-Valued Emotions," in IEEE Access, vol. 10, pp. 44617-44626, 2022
- **CLIP** by OpenAI for the base image encoder architecture
## License
This model is released under the MIT License. However, usage of the midi-emotion component should comply with its GPL-3.0 license. |