Implicit Mesh Reconstruction
from Unannotated Image Collections


Shubham Tulsiani1
Nilesh Kulkarni2
Abhinav Gupta1,3

1Facebook AI Research
2University of Michigan
3CMU



Given a single input image, we can infer the shape, texture and camera viewpoint for the underlying object. In rows 1 and 2, we show the input image, inferred 3D shape and texture from the predicted viewpoint, and three novel viewpoints. We can learn 3D inference using only in-the-wild image collections with approximate instance segmentations, our approach can be easily applied across a diverse set of categories. Rows 3 and 4 show sample predictions across a broad set of categories, with the predicted 3D shape overlaid on the input image.

We present an approach to infer the 3D shape, texture, and camera pose for an object from a single RGB image, using only category-level image collections with foreground masks as supervision. We represent the shape as an image-conditioned implicit function that transforms the surface of a sphere to that of the predicted mesh, while additionally predicting the corresponding texture. To derive supervisory signal for learning, we enforce that: a) our predictions when rendered should explain the available image evidence, and b) the inferred 3D structure should be geometrically consistent with learned pixel to surface mappings. We empirically show that our approach improves over prior work that leverages similar supervision, and in fact performs competitively to methods that use stronger supervision. Finally, as our method enables learning with limited supervision, we qualitatively demonstrate its applicability over a set of about 30 object categories.



Overview





Paper

S. Tulsiani, N. Kulkarni, A. Gupta.

Implicit Mesh Reconstruction
from Unannotated Image Collections.

arXiv preprint, 2020.

[pdf]     [Bibtex]




Results


Acknowledgements

We would like to thank the members of the Fouhey AI lab (FAIL) and CMU Visual Robot Learning lab for helpful discussions and feedback. This webpage template was borrowed from some colorful folks.