--- language: - en tags: - image-generation - text-to-image - conditional-generation - generative-modeling - image-synthesis - image-manipulation - design-prototyping - research - educational license: mit metrics: - FID - KID - HWD - CER --- # VATr++ (Hugging Face Version) This is a re-upload of the **VATr++** styled handwritten text generation model to the Hugging Face Model Hub. The original code and more detailed documentation can be found in the [VATr-pp GitHub repository](https://github.com/EDM-Research/VATr-pp). > **Note**: Please refer to the original repo for: > - Full training instructions > - In-depth code details > - Extended usage and references This Hugging Face version allows you to directly load the **VATr++** model with `AutoModel.from_pretrained(...)` and use it in your pipelines or scripts without manually handling checkpoints. The usage differs slightly from the original GitHub repository, primarily because we leverage Hugging Face’s `transformers` interface here. --- ## Installation 1. **Create a conda environment (recommended)**: ```bash conda create --name vatr python=3.9 conda activate vatr ``` 2. **Install PyTorch and CUDA (if available)**: ```bash conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia ``` 3. **Install additional requirements** (including `transformers`, `opencv`, etc.): ```bash pip install transformers opencv-python ``` *You may need to adjust or add libraries based on your specific environment needs.* --- ## Loading the Model #### **VATr++** To load the **VATr++** version: ```python from transformers import AutoModel model_vatr_pp = AutoModel.from_pretrained( "blowing-up-groundhogs/vatrpp", trust_remote_code=True ) ``` #### **VATr (original)** To load the **original VATr** model (instead of VATr++), specify the `subfolder` argument: ```python model_vatr = AutoModel.from_pretrained( "blowing-up-groundhogs/vatrpp", subfolder="vatr", trust_remote_code=True ) ``` --- ## Usage (Inference Example) Below is a **minimal** usage example that demonstrates how to: 1. Load the VATr++ model from the Hugging Face Hub. 2. Preprocess a style image (an image of handwriting). 3. Generate a new handwritten line of text in the style of the provided image. > **Important**: This model requires `trust_remote_code=True` to properly load its custom generation logic. ```python import numpy as np from PIL import Image import torch from torchvision import transforms as T from transformers import AutoModel # 1. Load the model (VATr++) model = AutoModel.from_pretrained("blowing-up-groundhogs/vatrpp", trust_remote_code=True) # 2. Helper functions to load and process style images def load_image(img, chunk_width=192): # Convert to grayscale and resize to height 32 img = img.convert("L") img = img.resize((img.width * 32 // img.height, 32)) arr = np.array(img) # Setup transforms: invert + normalize transform = T.Compose([ T.Grayscale(num_output_channels=1), T.ToTensor(), T.Normalize((0.5,), (0.5,)) ]) # Pad / chunk the image to a fixed width arr = 255 - arr height, width = arr.shape out = np.zeros((height, chunk_width), dtype="float32") out[:, :width] = arr[:, :chunk_width] out = 255 - out # Apply transforms out = transform(Image.fromarray(out.astype(np.uint8))) return out, width def load_image_line(img, chunk_width=192, style_imgs_count=15): # Convert to grayscale and resize img = img.convert("L") img = img.resize((img.width * 32 // img.height, 32)) arr = np.array(img) # Split into fixed-width chunks chunks = [] for start in range(0, arr.shape[1], chunk_width): chunk = arr[:, start:start+chunk_width] chunks.append(chunk) # Transform each chunk transformed = [] for c in chunks: t, _ = load_image(Image.fromarray(c), chunk_width) transformed.append(t) # If fewer than `style_imgs_count` chunks, repeat them while len(transformed) < style_imgs_count: transformed += transformed transformed = transformed[:style_imgs_count] # Combine return torch.cat(transformed, 0) # 3. Load a style image of your handwriting (or any handwriting sample) style_image_path = "path/to/your_style_image.png" img = Image.open(style_image_path) style_imgs = load_image_line(img) # 4. Generate text in the style of `style_image_path` generated_pil_image = model.generate( gen_text="This is a test", # Text to generate style_imgs=style_imgs, # Preprocessed style chunks align_words=True, # Align words at baseline at_once=True, # Generate line at once ) # 5. Save the generated image generated_pil_image.save("generated_output.png") ``` - **`style_imgs`**: A batch of fixed-width image chunks from your style reference. In practice, you can supply multiple small style samples or a single line image split into chunks. - **`gen_text`**: The text to render in the given style. - **`align_words`** and **`at_once`**: Optional arguments that control how the text is laid out and generated. --- ## Original Repository This model is built upon the code from [**EDM-Research/VATr-pp**](https://github.com/EDM-Research/VATr-pp), which is itself an improvement on the [VATr](https://github.com/aimagelab/VATr) project. If you need to: - Train your own model from scratch - Explore advanced features (like style cycle loss, punctuation modes, or advanced augmentation) - Examine experimental details or replicate the original paper's setup Please visit the original GitHub repos for comprehensive documentation and support files. --- ## License and Acknowledgments - The original code and model are under the license found in [the GitHub repository](https://github.com/EDM-Research/VATr-pp). - All credit goes to the original authors and maintainers for creating VATr++ and releasing it openly. - This Hugging Face re-upload is merely intended to **simplify inference** and **model sharing**; no changes have been made to the core training code or conceptual pipeline. --- **Enjoy generating styled handwritten text!** For any issues specific to this Hugging Face version, feel free to open an issue or pull request here. Otherwise, for deeper technical questions, please consult the original repository or its authors.