Real-Time High-Resolution Background Matting

We introduce a real-time, high-resolution background replacement technique which operates at 30fps in 4K resolution, and 60fps for HD on a modern GPU. Our technique is based on <PRE_TAG><PRE_TAG>background matting</POST_TAG></POST_TAG>, where an additional frame of the background is captured and used in recovering the <PRE_TAG><PRE_TAG>alpha matte</POST_TAG></POST_TAG> and the <PRE_TAG>foreground layer</POST_TAG>. The main challenge is to compute a high-quality <PRE_TAG><PRE_TAG>alpha matte</POST_TAG></POST_TAG>, preserving strand-level hair details, while processing high-resolution images in real-time. To achieve this goal, we employ two <PRE_TAG>neural networks</POST_TAG>; a base network computes a <PRE_TAG>low-resolution result</POST_TAG> which is refined by a second network operating at high-resolution on selective patches. We introduce two largescale video and image matting datasets: <PRE_TAG>VideoMatte240K</POST_TAG> and <PRE_TAG>PhotoMatte13K/85</POST_TAG>. Our approach yields higher quality results compared to the previous state-of-the-art in <PRE_TAG><PRE_TAG>background matting</POST_TAG></POST_TAG>, while simultaneously yielding a dramatic boost in both speed and resolution.