---
license: mit
tags:
- art
- music
- midi
- emotion
- clip
- multimodal
---

# ARIA - Artistic Rendering of Images into Audio

ARIA is a multimodal AI model that generates MIDI music based on the emotional content of artwork. It uses a CLIP-based image encoder to extract emotional valence and arousal from images, then generates emotionally appropriate music using conditional MIDI generation.

## Model Description

- **Developed by:** Vincent Amato
- **Model type:** Multimodal (Image-to-MIDI) Generation
- **Language(s):** English
- **License:** MIT
- **Parent Model:** Uses CLIP for image encoding and midi-emotion for music generation
- **Repository:** [GitHub](https://github.com/vincentamato/aria)

### Model Architecture

ARIA consists of two main components:
1. A CLIP-based image encoder fine-tuned to predict emotional valence and arousal from images
2. A transformer-based MIDI generation model (midi-emotion) that conditions on these emotional values

The model offers three different conditioning modes:
- `continuous_concat`: Emotions as continuous vectors concatenated to all tokens
- `continuous_token`: Emotions as continuous vectors prepended to sequence
- `discrete_token`: Emotions quantized into discrete tokens

### Usage

The repository contains three variants of the MIDI generation model, each trained with a different conditioning strategy. Each variant includes:
- `model.pt`: The trained model weights
- `mappings.pt`: Token mappings for MIDI generation
- `model_config.pt`: Model configuration

Additionally, `image_encoder.pt` contains the CLIP-based image emotion encoder.

## Intended Use

This model is designed for:
- Generating music that matches the emotional content of artwork
- Exploring emotional transfer between visual and musical domains
- Creative applications in art and music generation

### Limitations

- Music generation quality depends on the emotional interpretation of input images
- Generated MIDI may require human curation for professional use
- Model's emotional understanding is limited to valence-arousal space

## Training Data

The model combines:
1. Image encoder: Uses ArtBench with emotional annotations
2. MIDI generation: Uses the Lakh-Spotify dataset as processed by the midi-emotion project

## Attribution

This project builds upon:
- **midi-emotion** by Serkan Sulun et al. ([GitHub](https://github.com/serkansulun/midi-emotion))
  - Paper: "Symbolic music generation conditioned on continuous-valued emotions" ([IEEE Access](https://ieeexplore.ieee.org/document/9762257))
  - Citation: S. Sulun, M. E. P. Davies and P. Viana, "Symbolic Music Generation Conditioned on Continuous-Valued Emotions," in IEEE Access, vol. 10, pp. 44617-44626, 2022
- **CLIP** by OpenAI for the base image encoder architecture

## License

This model is released under the MIT License. However, usage of the midi-emotion component should comply with its GPL-3.0 license.