HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videos

Jinglei Zhang¹ Jiankang Deng² Chao Ma¹ Rolandos Alexandros Potamias²

¹Shanghai Jiao Tong University, China ²Imperial College London, UK

This is the official implementation of HaWoR, a hand reconstruction model in the world coordinates:

Installation

git clone --recursive https://github.com/ThunderVVV/HaWoR.git
cd HaWoR

The code has been tested with PyTorch 1.13 and CUDA 11.7. It is suggested to use an anaconda environment to install the the required dependencies:

conda create --name hawor python=3.10
conda activate hawor

pip install torch==1.13.0+cu117 torchvision==0.14.0+cu117 --extra-index-url https://download.pytorch.org/whl/cu117
# Install requirements
pip install -r requirements.txt
pip install pytorch-lightning==2.2.4 --no-deps
pip install lightning-utilities torchmetrics==1.4.0

Install masked DROID-SLAM:

cd thirdparty/DROID-SLAM
python setup.py install

Download DROID-SLAM official weights droid.pth, put it under ./weights/external/.

Install Metric3D

Download Metric3D official weights metric_depth_vit_large_800k.pth, put it under thirdparty/Metric3D/weights.

Download the model weights

wget https://huggingface.co/spaces/rolpotamias/WiLoR/resolve/main/pretrained_models/detector.pt -P ./weights/external/
wget https://huggingface.co/ThunderVVV/HaWoR/resolve/main/hawor/checkpoints/hawor.ckpt -P ./weights/hawor/checkpoints/
wget https://huggingface.co/ThunderVVV/HaWoR/resolve/main/hawor/checkpoints/infiller.pt -P ./weights/hawor/checkpoints/
wget https://huggingface.co/ThunderVVV/HaWoR/resolve/main/hawor/model_config.yaml -P ./weights/hawor/

It is also required to download MANO model from MANO website. Create an account by clicking Sign Up and download the models (mano_v*_*.zip). Unzip and put the hand model to the _DATA/data/mano/MANO_RIGHT.pkl and _DATA/data_left/mano_left/MANO_LEFT.pkl.

Note that MANO model falls under the MANO license.

Demo

For visualizaiton in world view, run with:

python demo.py --video_path ./example/video_0.mp4  --vis_mode world

For visualizaiton in camera view, run with:

python demo.py --video_path ./example/video_0.mp4 --vis_mode cam

Training

The training code will be released soon.

Acknowledgements

Parts of the code are taken or adapted from the following repos:

License

HaWoR models fall under the CC-BY-NC--ND License. This repository depends also on MANO Model, which are fall under their own licenses. By using this repository, you must also comply with the terms of these external licenses.

Citing

If you find HaWoR useful for your research, please consider citing our paper: