Add comprehensive model card for Multi-View 3D Point Tracking (#1)

- Add comprehensive model card for Multi-View 3D Point Tracking (027783913e6d93e789b7fe23cd1263b14a415f44)


Co-authored-by: Niels Rogge <[email protected]>

README.md

@@ -0,0 +1,98 @@
+---
+license: apache-2.0
+pipeline_tag: keypoint-detection
+library_name: pytorch
+---
+
+<div align="center" style="line-height:1.2; margin:0; padding:0;">
+<h1 style="margin-bottom:0em;">Multi-View 3D Point Tracking</h1>
+
+<a href="https://huggingface.co/papers/2508.21060"><img src="https://img.shields.io/badge/Paper-2508.21060-b31b1b" alt="Paper"></a>
+<a href="https://ethz-vlg.github.io/mvtracker/"><img src="https://img.shields.io/badge/Project%20Page-009688?logo=internetcomputer&logoColor=white" alt="Project Page"></a>
+<a href="https://github.com/ethz-vlg/mvtracker"><img src="https://img.shields.io/badge/GitHub-Code-blue.svg?logo=github&logoColor=white" alt="GitHub"></a>
+[![](https://img.shields.io/badge/%F0%9F%A4%97%20Demo-Coming%20soon…-ffcc00)](#)
+<br>
+[**Frano Rajič**](https://m43.github.io/)<sup>1</sup> · 
+[**Haofei Xu**](https://haofeixu.github.io/)<sup>1</sup> · 
+[**Marko Mihajlovic**](https://markomih.github.io/)<sup>1</sup> · 
+[**Siyuan Li**](https://siyuanliii.github.io/)<sup>1</sup> · 
+[**Irem Demir**](https://github.com/iremddemir)<sup>1</sup>  
+[**Emircan Gündoğdu**](https://github.com/emircangun)<sup>1</sup> · 
+[**Lei Ke**](https://www.kelei.site/)<sup>2</sup> · 
+[**Sergey Prokudin**](https://vlg.inf.ethz.ch/team/Dr-Sergey-Prokudin.html)<sup>1,3</sup> · 
+[**Marc Pollefeys**](https://people.inf.ethz.ch/marc.pollefeys/)<sup>1,4</sup> · 
+[**Siyu Tang**](https://vlg.inf.ethz.ch/team/Prof-Dr-Siyu-Tang.html)<sup>1</sup>
+<br>
+<sup>1</sup>[ETH Zürich](https://vlg.inf.ethz.ch/) &emsp;
+<sup>2</sup>[Carnegie Mellon University](https://www.cmu.edu/) &emsp;
+<sup>3</sup>[Balgrist University Hospital](https://www.balgrist.ch/) &emsp;
+<sup>4</sup>[Microsoft](https://www.microsoft.com/)
+</div>
+
+MVTracker is the first **data-driven multi-view 3D point tracker** for tracking arbitrary 3D points across multiple cameras. It fuses multi-view features into a unified 3D feature point cloud, within which it leverages kNN-based correlation to capture spatiotemporal relationships across views. A transformer then iteratively refines the point tracks, handling occlusions and adapting to varying camera setups without per-sequence optimization.
+
+<p float="left">
+  <img alt="selfcap" src="https://github.com/user-attachments/assets/b502d193-c37c-43be-af6c-653b5de7597e" width="48%" /> 
+  <img alt="dexycb" src="https://github.com/user-attachments/assets/d14d4c6c-152e-4040-b29b-3da4b7e8b913" width="48%" /> 
+  <img alt="4d-dress-stretching" src="https://github.com/user-attachments/assets/f3eabdda-59e1-4032-b345-c4603ea86fc0" width="48%" />
+  <img alt="4d-dress-avatarmove" src="https://github.com/user-attachments/assets/3fef9924-84ad-4295-95e2-5b82ae7c3053" width="48%" />
+</p>
+
+## Abstract
+We introduce the first data-driven multi-view 3D point tracker, designed to track arbitrary points in dynamic scenes using multiple camera views. Unlike existing monocular trackers, which struggle with depth ambiguities and occlusion, or prior multi-camera methods that require over 20 cameras and tedious per-sequence optimization, our feed-forward model directly predicts 3D correspondences using a practical number of cameras (e.g., four), enabling robust and accurate online tracking. Given known camera poses and either sensor-based or estimated multi-view depth, our tracker fuses multi-view features into a unified point cloud and applies k-nearest-neighbors correlation alongside a transformer-based update to reliably estimate long-range 3D correspondences, even under occlusion. We train on 5K synthetic multi-view Kubric sequences and evaluate on two real-world benchmarks: Panoptic Studio and DexYCB, achieving median trajectory errors of 3.1 cm and 2.0 cm, respectively. Our method generalizes well to diverse camera setups of 1-8 views with varying vantage points and video lengths of 24-150 frames. By releasing our tracker alongside training and evaluation datasets, we aim to set a new standard for multi-view 3D tracking research and provide a practical tool for real-world applications.
+
+## Quick Start
+
+This repo was validated on **Python 3.10.12**, **PyTorch 2.3.0** (CUDA 12.1), **cuDNN 8903**, and **gcc 11.3.0**. If you want a fresh minimal environment that runs the Hub demo and `demo.py`:
+```bash
+conda create -n 3dpt python=3.10.12 -y
+conda activate 3dpt
+conda install pytorch==2.3.0 torchvision==0.18.0 torchaudio==2.3.0 pytorch-cuda=12.1 -c pytorch -c nvidia -y
+pip install -r https://raw.githubusercontent.com/ethz-vlg/mvtracker/refs/heads/main/requirements.txt
+
+# Optional, speeds up the model
+pip install --upgrade --no-build-isolation flash-attn==2.5.8  # Speeds up attention
+pip install "git+https://github.com/ethz-vlg/pointcept.git@2082918#subdirectory=libs/pointops"  # Speeds up kNN search; may require gcc 11.3.0: conda install -c conda-forge gcc_linux-64=11.3.0 gxx_linux-64=11.3.0 gcc=11.3.0 gxx=11.3.0
+```
+
+With the minimal dependencies in place, you can try MVTracker directly via **PyTorch Hub**:
+```python
+import torch
+import numpy as np
+from huggingface_hub import hf_hub_download
+
+device = "cuda" if torch.cuda.is_available() else "cpu"
+mvtracker = torch.hub.load("ethz-vlg/mvtracker", "mvtracker", pretrained=True, device=device)
+
+# Example input from demo sample (downloaded automatically)
+sample = np.load(hf_hub_download("ethz-vlg/mvtracker", "data_sample.npz"))
+rgbs = torch.from_numpy(sample["rgbs"]).float()
+depths = torch.from_numpy(sample["depths"]).float()
+intrs = torch.from_numpy(sample["intrs"]).float()
+extrs = torch.from_numpy(sample["extrs"]).float()
+query_points = torch.from_numpy(sample["query_points"]).float()
+
+with torch.no_grad():
+    results = mvtracker(
+        rgbs=rgbs[None].to(device) / 255.0,
+        depths=depths[None].to(device),\
+        intrs=intrs[None].to(device),
+        extrs=extrs[None].to(device),
+        query_points_3d=query_points[None].to(device),
+    )
+
+pred_tracks = results["traj_e"].cpu()  # [T,N,3]
+pred_vis = results["vis_e"].cpu()      # [T,N]
+print(pred_tracks.shape, pred_vis.shape)
+```
+
+## Citation
+If you find our repository useful, please consider giving it a star ⭐ and citing our work:
+```bibtex
+@inproceedings{rajic2025mvtracker,
+  title     = {Multi-View 3D Point Tracking},
+  author    = {Raji{\v{c}}, Frano and Xu, Haofei and Mihajlovic, Marko and Li, Siyuan and Demir, Irem and G{\"u}ndo{\u{g}}du, Emircan and Ke, Lei and Prokudin, Sergey and Pollefeys, Marc and Tang, Siyu},
+  booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
+  year      = {2025}
+}
+```