Self-Supervised 3D Hand Pose Estimation from monocular RGB via Contrastive Learning

authors: Adrian Spurr*, Aneesh Dahiya*, Xi Wang, Xucong Zhang, and Otmar Hilliges
publication: International Conference on Computer Vision (ICCV), October 2021, oral
*Authors contributed equally.


Model Overview

The proposed PeCLR framework for 3D hand pose estimation.




Abstract

Encouraged by the success of contrastive learning on image classification tasks, we propose a new self-supervised method for the structured regression task of 3D hand pose estimation. Contrastive learning makes use of unlabeled data for the purpose of representation learning via a loss formulation that encourages the learned feature representations to be invariant under any image transformation. For 3D hand pose estimation, it too is desirable to have invariance to appearance transformation such as color jitter. However, the task requires equivariance under affine transformations, such as rotation and translation. To address this issue, we propose an equivariant contrastive objective and demonstrate its effectiveness in the context of 3D hand pose estimation. We experimentally investigate the impact of invariant and equivariant contrastive objectives and show that learning equivariant features leads to better representations for the task of 3D hand pose estimation. Furthermore, we show that standard ResNets with sufficient depth, trained on additional unlabeled data, attain improvements of up to 14.5% in PA-EPE on FreiHAND and thus achieves state-of-the-art performance without any task specific, specialized architectures.




Blog post

Please see the blog post for a high-level introduction to PeCLR.




Video




Code and Models

Qries
Pre-trained models coming soon!




Citation

@inProceedings{spurr2021peclr,
  title={Self-Supervised 3D Hand Pose Estimation from monocular RGB via Contrastive Learning},
  author={Spurr, Adrian and Dahiya, Aneesh and Wang, Xi and Zhang, Xucong and Hilliges, Otmar},
  booktitle={International Conference on Computer Vision (ICCV)},
  year={2021}
}



Acknowledgments

We are grateful to Thomas Langerak for the aid in figure creation and Marcel Buehler for helpful discussions and comments.