NS3D: Neuro-Symbolic Grounding of 3D Objects and Relations
Joy Hsu,
Jiayuan Mao,
Jiajun Wu
In Conference on Computer Vision and Pattern Recognition (CVPR) 2023
Our dataset download process follows the ReferIt3D benchmark.
Specifically, you will need to
- (1) Download
sr3d_train.csvandsr3d_test.csvfrom this link - (2) Download scans from ScanNet and process them according to this link. This should result in a
keep_all_points_with_global_scan_alignment.pklfile.
Run the following commands to install necessary dependencies.
conda create -n ns3d python=3.7.11
conda activate ns3d
pip -r requirements.txtInstall Jacinle.
git clone https://github.com/vacancy/Jacinle --recursive
export PATH=<path_to_jacinle>/bin:$PATHInstall the referit3d python package from ReferIt3D.
git clone https://github.com/referit3d/referit3d
cd referit3d
pip install -e .Compile CUDA layers for PointNet++.
cd models/scene_graph/point_net_pp/pointnet2
python setup.py installTo evaluate NS3D:
scannet=<path_to/keep_all_points_with_global_scan_alignment.pkl>
referit=<path_to/sr3d_train.csv>
load_path=<path_to/model_to_evaluate.pth>
jac-run ns3d/trainval.py --desc ns3d/desc_ns3d.py --scannet-file $scannet --referit3D-file $referit --load $load_path --evaluateWeights for our trained NS3D model can be found at trained_ns3d.pth and loaded into load_path.
To train NS3D:
scannet=<path_to/keep_all_points_with_global_scan_alignment.pkl>
referit=<path_to/sr3d_train.csv>
load_path=<path_to/pretrained_classification_model.pth>
jac-run ns3d/trainval.py --desc ns3d/desc_ns3d.py --scannet-file $scannet --referit3D-file $referit --load $load_path --lr 0.0001 --epochs 5000 --save-interval 1 --validation-interval 1Weights for our pretrained classification model can be found at pretrained_cls.pth and loaded into load_path.
Our codebase is built on top of NSCL and ReferIt3D. Please feel free to email me at [email protected] if any problems arise.