Injae Kim*, Minhyuk Choi*, Hyunwoo J. Kim†.
This repository is an official implementation of the NeurIPS 2023 paper UP-NeRF (Unconstrained Pose-Prior-Free Neural Radiance Fields) using pytorch-lightning.
git clone https://github.com/mlvlab/UP-NeRF.git
cd UP-NeRF
We recommend using Anaconda to set up the environment.
conda create -n upnerf python=3.8 -y
conda activate upnerf
conda install pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=11.3 -c pytorch -y
pip install -r requirements.txt
Download the Phototourism data of the scene that you want to train from here and train/val split file of the data from here.
And place them as shown below.
UP-NeRF
├── data
│ └── phototourism
│ ├── brandenburg_gate
│ │ ├── brandenburg_gate.tsv
│ │ └── dense
│ ├── taj_mahal
│ │ ├── taj_mahal.tsv
│ │ └── dense
│ ...
...
Or you can simply use our automated download script.
# Example
sh scripts/download_phototourism.sh brandenburg_gate
Scenes provided are {brandenburg_gate, british_museum, lincoln_memorial_statue, pantheon_exterior, sacre_coeur, st_pauls_cathedral, taj_mahal, trevi_fountain}
To run with your own dataset, please check the format of metadata data/example/metadata.json and configuration file configs/custom.yaml (You can omit c2w fields in metadata.json if pose evaluation is not necessary. In addition, c2w matrices must be right up back format) You must put images in dense/images (mandatory for compatability).
Before training you need to save DINO feature maps and DPT mono-depth maps.
Initialize the external submodule. (Last sanity check on Jan 2st, 2024)
git submodule update --init --recursive
Run the script for preprocessing. (example of Brandenburg Gate scene)
sh ./preprocess/preprocess_all.sh brandenburg_gate
❗ Our script includes downloading checkpoint of DPT. If download fails due to some reasons, you can do it manually by downloading it from here, and move the weight to "./DPT/weights/".
Additionally, we highly recommend saving the cache data.
Run (example)
python prepare_phototourism.py --config configs/brandenburg_gate.yaml
Script is slightly different, so we separate the script file.
sh ./preprocess/preprocess_all_custom.sh data/example # you have to specify root directory of dataset
python prepare_phototourism.py --config configs/example.yaml
# If you saved the cache data.
python train.py --config configs/brandenburg_gate.yaml
# If you did not save the cache data.
python train.py --config configs/brandenburg_gate.yaml phototourism.use_cache False
You can change the yaml file to change the scene. Check the config files in ./configs
Use tto.py for test time optimization. It optimizes camera poses and appearance embeddings for test images.
Run (example)
python tto.py \
--result_dir ./outputs/brandenburg_gate/UP-NeRF \
--ckpt last \
--optimize_num -1 \
--wandb
eval.py prints results (PSNR, SSIM, LPIPS, rotation & translation errors).
Run (example)
python eval.py \
--result_dir ./outputs/brandenburg_gate/UP-NeRF \
--ckpt last
You can download pretrained weights from here. (brandenburg_gate, sacre_coeur, taj_mahal, trevi_fountain)
@inproceedings{kim2023upnerf,
title={UP-NeRF: Unconstrained Pose-Prior-Free Neural Radiance Fields},
author={Kim, Injae and Choi, Minhyuk and Kim, Hyunwoo J},
booktitle={Advances in Neural Information Processing Systems},
year={2023}
}
Our code is based on the implementation of NeRF in the Wild (NeRF-W) and BARF (BARF).