A differentiable point-based rendering framework
Document🏠 | 中文文档🏠 | Paper(Comming soon)📄 | Msplat Backend🌐 | 教程视频🔗

News

• 2024-09-11: We have included instructions for hyperparameter search in the documentation.
• 2024-09-02: We support all hyperparameter turning based on wandb, including "random", "grid" and "bayes" sweep configuration.
• 2024-08-18: We have released Pointrix: v1.0
• 2024-08-18: We add exporter which supports metric/video/mesh export.

Features

Pointrix is a differentiable point-based rendering framework which has following properties:

• Highly Extensible:
  • Python API
  • Modular design for both researchers and beginners
  • Implementing your own method without touching CUDA
• Powerful Backend:
  • CUDA Backend
  • Forward Anything: rendering image, depth, normal, optical flow, etc.
  • Backward Anything: optimizing even intrinsics and extrinsics.
• Rich Features:
  • Support camera parameters optimization.
  • Support Dynmamic scene reconstruction task and Generation task (WIP).
  • Support mesh extraction and different type of initialization.

Quickstart

Installation

Clone pointrix:

git clone https://github.com/pointrix-project/pointrix.git  --recursive
cd pointrix

Create a new conda environment with pytorch:

conda create -n pointrix python=3.9
conda activate pointrix
conda install pytorch==2.1.1 torchvision==0.16.1 pytorch-cuda=12.1 -c pytorch -c nvidia

Install Pointrix and MSplat:

cd msplat
pip install .

cd ..
pip install -r requirements.txt
pip install -e .

(Optional) You can also install gsplat or diff-gaussian-rasterization:

pip install gsplat

git clone https://github.com/graphdeco-inria/diff-gaussian-rasterization.git
cd diff-gaussian-rasterization
python setup.py install
pip install .

Train Your First 3D Gaussian

Tanks and Temples Dataset Demo (Colmap format dataset)

Download the demo truck scene data and run:

cd examples/gaussian_splatting
# For Tanks and Temples data which have high-res images and need to downsample.
python launch.py --config ./configs/colmap.yaml trainer.datapipeline.dataset.data_path=your_data_path trainer.datapipeline.dataset.scale=0.5 trainer.output_path=your_log_path

# you can also use GaussianSplatting renderer or GSplat renderer
python launch.py --config ./configs/colmap.yaml trainer.datapipeline.dataset.data_path=your_data_path trainer.datapipeline.dataset.scale=0.5 trainer.output_path=your_log_path trainer.model.renderer.name=GaussianSplattingRender

python launch.py --config ./configs/colmap.yaml trainer.datapipeline.dataset.data_path=your_data_path trainer.datapipeline.dataset.scale=0.5 trainer.output_path=your_log_path trainer.controller.normalize_grad=True trainer.model.renderer.name=GsplatRender

The scale should be set as 0.25 for mipnerf 360 datasets.

For other colmap dataset which do not need to downsample:

python launch.py --config ./configs/colmap.yaml trainer.datapipeline.dataset.data_path=your_data_path trainer.datapipeline.dataset.scale=1.0 trainer.output_path=your_log_path

if you want test your model:

cd examples/gaussian_splatting
# For Tanks and Temples data which have high-res images and need to downsample.
python launch.py --config ./configs/colmap.yaml trainer.datapipeline.dataset.data_path=your_data_path trainer.datapipeline.dataset.scale=0.25 trainer.output_path=your_log_path trainer.training=False trainer.test_model_path=your_model_path

NeRF-Lego (NeRF-Synthetic format dataset)

Download the lego data:

wget http://cseweb.ucsd.edu/~viscomp/projects/LF/papers/ECCV20/nerf/nerf_example_data.zip

Run the following (with adjusted data path):

cd examples/gaussian_splatting
python launch.py --config ./configs/nerf.yaml trainer.datapipeline.dataset.data_path=your_data_path trainer.output_path=your_log_path

if you want to test the model:

python launch.py --config ./configs/nerf.yaml trainer.training=False trainer.datapipeline.dataset.data_path=your_data_path trainer.test_model_path=your_model_path

Advanced Approaches

Turning your hyperparameters

Pointrix support turning of hyperparameters based on sweep configuration in wandb, try this feature by running following command:

cd examples/gaussian_splatting_sweep
python launch_sweep.py --config configs/colmap.yaml --config_sweep configs/colmap_sweep.yaml trainer.datapipeline.dataset.data_path=your_data_path  trainer.output_path=your_log_path

Camera optimization

To enable camera optimization, you should set trainer.model.camera_model.enable_training=True and trainer.optimizer.optimizer_1.camera_params.lr=1e-3: The renderer must be setted as MsplatRender.

python launch.py --config ./configs/colmap.yaml trainer.datapipeline.dataset.data_path=your_data_path trainer.datapipeline.dataset.scale=1.0 trainer.output_path=your_log_path trainer.model.renderer.name=MsplatRender trainer.model.camera_model.enable_training=True trainer.optimizer.optimizer_1.camera_params.lr=1e-3

Post-Processing Results Extraction (Metric, Mesh, Video)

Pointrix uses exporters to obtain desired post-processing results, such as mesh and video. The relevant configuration is as follows:

trainer:
    exporter:
        exporter_a:
            type: MetricExporter
        exporter_b:
            type: TSDFFusion
            extra_cfg:
                voxel_size: 0.02
                sdf_trunc: 0.08
                total_points: 8_000_000 
        exporter_c:
            type: VideoExporter

Users can specify multiple exporters to obtain various post-processing results. For example, with the above configuration, users can get Metric and Mesh extraction results as well as Video post-processing results. Mesh is obtained using the TSDF fusion method by default. The renderer must be set as MsplatRender or GsplatRender. You need to set trainer.model.renderer.render_depth as True to enable TSDFFusion.

cd pointrix/projects/gaussian_splatting
python launch.py --config ./configs/nerf.yaml trainer.training=False trainer.datapipeline.dataset.data_path=your_data_path trainer.test_model_path=your_model_path trainer.model.renderer.render_depth=True

Dust3r initialization (Beta)

1. Switch to the Beta branch.

2. Download Dust3r to examples/dust3r_init and follow the installation instructions.

3. Move convert_dust3r.py to the examples/dust3r_init/dust3r folder.

4. Navigate to examples/dust3r_init/dust3r, and then use Dust3r to extract point cloud priors and camera priors:

python convert_dust3r.py --model_path your_dust3r_weights --filelist your_image_path

5. Run the program

python launch.py --config config.yaml trainer.datapipeline.dataset.data_path=your_data_path trainer.output_path=your_log_path

Release Plans

• Nerf_synthetic dataset (this week).
• Dust3r initialization (this week).
• Mesh exstraction (this week).
• Introduction video (this week)
• reformat the document (this week)
• Dynamic Gaussian Project(next week).

Welcome to discuss with us and submit PR on new ideas and methods.

Acknowledgment

Thanks to the developers and contributors of the following open-source repositories, whose invaluable work has greatly inspire our project:

• 3D Gaussian Splatting: 3D Gaussian Splatting for Real-Time Radiance Field Rendering.
• Threestudio: A unified framework for 3D content creation
• OmegaConf: Flexible Python configuration system.
• SSIM: pytorch SSIM loss implemetation.
• GSplat: An open-source library for CUDA accelerated rasterization of gaussians with python bindings.
• detectron2: Detectron2 is Facebook AI Research's next generation library that provides state-of-the-art detection and segmentation algorithms.
• DN-Splatter: Depth and Normal Priors for Gaussian Splatting and Meshing
• GOF: Efficient and Compact Surface Reconstruction in Unbounded Scenes

This is project is licensed under Apache License. However, if you use MSplat or the original 3DGS kernel in your work, please follow their license.

Contributors

Made with contrib.rocks.