KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Control

Tomas Jakab1,4, Richard Tucker4, Ameesh Makadia4, Jiajun Wu3, Noah Snavely4, Angjoo Kanazawa2,4
1University of Oxford, 2UC Berkeley, 3Stanford University, 4Google Research
In CVPR 2021 (Oral presentation)

We present KeypointDeformer, a novel unsupervised method for shape control through automatically discovered 3D keypoints. Our approach produces intuitive and semantically consistent control of shape deformations. Moreover, our discovered 3D keypoints are consistent across object category instances despite large shape variations. Since our method is unsupervised, it can be readily deployed to new object categories without requiring expensive annotations for 3D keypoints and deformations.

Code is now released!

Paper Demo Code and models BibTeX

Quick Explanation

Approach

Our method analyzes the difference between the shapes of the two objects by comparing their latent representations. This latent representation is in the form of 3D keypoints that are learned in an unsupervised way. The difference between the 3D keypoints of the source and the target objects then informs the shape deformation algorithm that deforms the source object into the target object. The whole model is learned end-to-end on this pair-wise shape alignment task and simultaneously discovers 3D keypoints while learning to use them for deforming object shapes.

Automatic Shape Augmentation

Our method can also learn a categorical shape prior that allows us to automatically sample new augmented shapes.

Wing angle change

Wing span change

Unsupervised 3D Keypoints

The unsupervised keypoints that our method discovers are semantically consistent across different instances with highly varying shapes.

Our method also works on real-world 3D scans of shoes from Google scanned objects.

Interactive Shape Control

User guided shape deformation using our discovered keypoint as handles.

Interactive Demo

Drag keypoints to deform the shape. The demo does not employ the optional deformation prior. Different models can be selected in the side menu. Dragging is not supported on mobile devices. Full screen version.

Paper

paper thumbnail
arXiv Preprint

BibTeX

@inproceedings{jakab2021keypointdeformer,
  title={KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Control},
  author={Jakab, Tomas and Tucker, Richard and Makadia, Ameesh and Wu, Jiajun and Snavely, Noah and Kanazawa, Angjoo},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2020}
}