Color-NeuS: Reconstructing Neural Implicit Surfaces with Color

1Shanghai Jiao Tong University, 2Shanghai Qi Zhi Institute, 3National University of Singapore
3DV 2024


The reconstruction of object surfaces from multi-view images or monocular video is a fundamental issue in computer vision. However, much of the recent research concentrates on reconstructing geometry through implicit or explicit methods. In this paper, we shift our focus towards reconstructing mesh in conjunction with color. We remove the view-dependent color from neural volume rendering while retaining volume rendering performance through a relighting network. Mesh is extracted from the signed distance function (SDF) network for the surface, and color for each surface vertex is drawn from the global color network. To evaluate our approach, we conceived a in hand object scanning task featuring numerous occlusions and dramatic shifts in lighting conditions. We've gathered several videos for this task, and the results surpass those of any existing methods capable of reconstructing mesh alongside color. Additionally, our method's performance was assessed using public datasets, including DTU, BlendedMVS, and OmniObject3D. The results indicated that our method performs well across all these datasets.



Color-NeuS employs a Signed Distance Function (SDF) network to grasp the implicit geometry. Following that, it utilizes a global color network to learn view-independent color characteristics. Additionally, a relight network is deployed to adjust for variations that correlate with the viewing direction. During the inference stage, only the global view-independent color is used, ensuring a consistent visual representation regardless of perspective.

Interactive Results
IHO Video OmniObject3D DTU BlendedMVS
ghost bear pink peach game box drink
doll 002 doll 008 doll 037 doll 049 doll 062 doll 074 doll 085 toy_animals 001 toy_animals 005 toy_animals 016 toy_animals 019 toy_animals 059
dtu 24 dtu 37 dtu 40 dtu 55 dtu 63 dtu 65 dtu 69 dtu 83 dtu 97 dtu 105 dtu 106 dtu 110 dtu 114 dtu 118 dtu 122
bear clock dog durian jade man sculpture stone
Note: It may be slow to load models.



ghost bear pink peach game box drink
Structure light
INT solution
Naive solution
Our solution
Note: It may be slow to load models.

Ablation Study
w/o inv sigmoid
w/o grad
Whole Model


        title     = {Color-NeuS: Reconstructing Neural Implicit Surfaces with Color},
        author    = {Zhong, Licheng and Yang, Lixin and Li, Kailin and Zhen, Haoyu and Han, Mei and Lu, Cewu},
        booktitle = {International Conference on 3D Vision (3DV)},
        year      = {2024}