[CVPRF 26] PASR: Pose-Aware 3D Shape Retrieval from Occluded Single Views

Pose-aware feature-level analysis-by-synthesis for single-view 3D shape retrieval.

Overview

PASR reformulates single-view 3D shape retrieval as a pose-aware feature-level analysis-by-synthesis problem. Instead of directly matching a query image to 3D shapes, PASR distills 2D foundation-model knowledge into point-level 3D representations and performs test-time pose optimization to align rendered mesh features with query image features.

This design enables robust 3D shape retrieval from occluded single-view images and improves generalization to unseen shapes.

Highlights

• Pose-aware retrieval. PASR explicitly optimizes object pose at test time rather than relying only on fixed-view matching.
• Feature-level analysis-by-synthesis. Query image features are compared with rendered mesh features in the foundation-model feature space.
• Point-level 3D representation learning. 2D foundation-model knowledge is distilled into 3D point features.
• Robustness under occlusion. PASR supports evaluation on original and occluded Pascal3D / Pix3D settings.
• Generalization to unseen shapes. The method is designed for retrieval beyond the training shape distribution.

Installation

Clone the repository and create the environment:

cd PASR

conda create -n pasr python=3.10 -y
conda activate pasr

Install PyTorch with CUDA 12.8:

pip install torch==2.7.1 torchvision==0.22.1 \
  --index-url https://download.pytorch.org/whl/cu128

Install Python dependencies:

pip install accelerate omegaconf timm numpy scipy scikit-learn \
  pillow matplotlib tqdm trimesh plyfile wandb plotly ninja multimethod \
  termcolor shortuuid easydict

Install PyTorch3D:

export CC=gcc-12
export CXX=g++-12
pip install "git+https://github.com/facebookresearch/pytorch3d.git" \
  --no-build-isolation

Install PointNeXt extensions:

cd openpoints/cpp/pointnet2_batch
python setup.py install
cd ../

cd pointops
python setup.py install
cd ..

cd chamfer_dist
python setup.py install --user

cd ../../../

Data Preparation

The default configs assume that all datasets are placed under data/.

Preprocessed Pascal3D and Pix3D data can be downloaded from JiaxinShi/PASR_DATASETS on Hugging Face.

The expected directory structure is:

data/
├── pascal3d/
├── pascal3d_processed/
│   ├── train/original/
│   ├── val/original/
│   └── camera_para.json
├── pascal3d_train.json
├── pascal3d_val.json
│
├── pix3d/
├── pix3d_processed/
│   ├── original/
│   └── camera_para.json
├── pix3d_train.json
└── pix3d_test.json

If your data is stored elsewhere, update the corresponding paths in the config files under configs/.

Training

Train on Pascal3D:

accelerate launch train.py \
  --config configs/pascal3d_dino_tune.yaml \
  --exp_name pascal3d \
  --output_dir exp

Train on Pix3D:

accelerate launch train.py \
  --config configs/pix3d_dino_tune.yaml \
  --exp_name pix3d \
  --output_dir exp

Useful flags:

--visualize true                 # Save/log feature visualizations
--save_checkpoint_start_epoch N  # Start saving checkpoints from epoch N
--eval_start_epoch N             # Start training-time evaluation from epoch N

Training outputs are saved to:

exp/<exp_name>/

Checkpoints are saved as:

epoch_<N>.pth
final.pth

Inference and Evaluation

PASR performs inference in four steps. Use the same config and checkpoint for all steps.

Example setup:

CKPT=exp/pascal3d/epoch_100.pth
CFG=configs/pascal3d_dino_tune.yaml
EXP=pascal3d

Step 1: Extract query image features

accelerate launch inference_1_extract_img_feature.py \
  --config $CFG \
  --exp_name $EXP \
  --load_checkpoint $CKPT \
  --wandb false

Step 2: Extract mesh features over predefined poses

accelerate launch inference_2_extract_mesh_feature.py \
  --config $CFG \
  --exp_name $EXP \
  --load_checkpoint $CKPT \
  --wandb false

Step 3: Run initial search

accelerate launch inference_3_init_search.py \
  --config $CFG \
  --exp_name $EXP \
  --wandb false

Step 4: Run test-time pose optimization

accelerate launch inference_4_retrieval.py \
  --config $CFG \
  --exp_name $EXP \
  --load_checkpoint $CKPT

Evaluation results are written to:

exp/<exp_name>/search_acc.txt
exp/<exp_name>/retrieval_acc_pytorch.txt

When enabled, per-query retrieval results are saved as pickle files:

--save_retrieval_pickle true

Evaluation on Occluded Pascal3D and Pix3D

To evaluate under different occlusion levels, modify test.query_dataset.dataset.occlusion_level in the corresponding config.

Supported values:

original, L1, L2, L3

Example:

test:
  query_dataset:
    dataset:
      class_name: dataloader.pascal3d.query_dataset
      processed_dataset_path: data/pascal3d_processed/val
      original_dataset_path: data/pascal3d
      categories: [
        'aeroplane', 'bicycle', 'boat', 'bottle', 'bus',
        'car', 'chair', 'diningtable', 'motorbike', 'train',
        'sofa', 'tvmonitor'
      ]
      meta_path: data/pascal3d_val.json
      occlusion_level: 'original'  # choices: original, L1, L2, L3
      feature_map_path: cache/img_feat/pascal3d
      search_res_path: cache/search_res/pascal3d

After changing the occlusion level, rerun the inference and evaluation pipeline.

Citation

If you find this project useful, please cite:

@misc{shi2026pasrposeaware3dshape,
  title         = {PASR: Pose-Aware 3D Shape Retrieval from Occluded Single Views},
  author        = {Jiaxin Shi and Guofeng Zhang and Wufei Ma and Naifu Liang and Adam Kortylewski and Alan Yuille},
  year          = {2026},
  eprint        = {2604.22658},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2604.22658}
}