Home Abstract Demo Paper Code ☰

Abstract

Despite that significant progress has been made in room layout estimation, most methods aim to reduce the loss in the 2D pixel coordinate rather than exploiting the room structure in the 3D space. Towards reconstructing the room layout in 3D, we formulate the task of 360˚ layout estimation as a problem of predicting depth on the horizon line of a panorama. Specifically, we propose the Differentiable Depth Rendering procedure to make the conversion from layout to depth prediction differentiable, thus making our proposed model end-to-end trainable while leveraging the 3D geometric information, without the need of providing the ground truth depth. We show that our method achieves state-of-the-art performance on numerous 360˚ layout benchmark datasets. Moreover, our formulation enables a pre-training step on the depth dataset, which further improves the generalizability of our layout estimation model.

Upload your own panorama!

You can capture your own panorama, and upload it to see the 3D result.

* When capturing, try your best to keep the camera parallel to the ground like the demo figures below. This will improve the accuracy!

* It takes about 10 seconds. If you wait longer than 20 sec, our server may be dead! RIP my server...

Image Format: jpg or png.

Size limit: 2MB

Online Demo

Click the link for 3D visualization!

Example 1 (link)
Example 2 (link)
Example 3 (link)
Example 4 (link)
Example 5 (link)
Example 6 (link)
Example 7 (link)
Example 8 (link)

CVPR 2021 Oral

LED2-Net: Monocular 360˚ Layout Estimation via Differentiable Depth

Fu-En Wang*, Yu-Hsuan Yeh*, Min Sun, Wei-Chen Chiu, Yi-Hsuan Tsai

Links

[Paper (arXiv)] [Supplementary Marterial] [Code]

Citation

@InProceedings{Wang_2021_CVPR,
    author    = {Wang, Fu-En and Yeh, Yu-Hsuan and Sun, Min and Chiu, Wei-Chen and Tsai, Yi-Hsuan},
    title     = {LED2-Net: Monocular 360deg Layout Estimation via Differentiable Depth Rendering},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2021},
    pages     = {12956-12965}
}