VOID: Advanced Video Object and Interaction Removal

VOID is a sophisticated tool designed to eliminate objects from videos along with their interactions, such as shadows, reflections, and physical effects like falling objects. Built on CogVideoX, VOID uses interaction-aware mask conditioning to ensure seamless video inpainting. For instance, if a person holding a guitar is removed, the guitar will naturally fall as if the person was never there.

## Models

VOID operates with two transformer checkpoints trained sequentially. Pass 1 can be used alone for basic inpainting, while Pass 2 enhances temporal consistency for longer clips. Users can download these models from HuggingFace and integrate them into their workflows using specific configuration paths.

## Quick Start

The fastest way to explore VOID is through the provided notebook, which automates setup, model downloads, and inference on sample videos. A powerful GPU with at least 40GB VRAM is required for optimal performance. For more detailed control, users can customize the pipeline with their own videos and mask generation.

## Setup and Pipeline

Setting up VOID involves installing dependencies, configuring the environment, and downloading necessary models. The process includes generating quadmasks that define regions in the video for object removal and interaction effects. The pipeline is divided into stages: mask generation using SAM2 and Gemini, and inference with two passes for video processing.

## Training

VOID's training involves generating data through HUMOTO and Kubric pipelines, which create counterfactual videos with and without objects. Training is conducted in two stages: Pass 1 focuses on base inpainting, while Pass 2 refines results using warped noise for better temporal consistency.

## Community and Acknowledgements

The community is encouraged to build upon VOID, with existing demos like the Gradio Demo showcasing its capabilities. The project acknowledges contributions from various open-source projects and encourages users to cite their work if found useful.