Reconstruction and Simulation of Elastic Objects with Spring-Mass 3D Gaussians

1Stanford University, 2University of Illinois Urbana-Champaign
ECCV 2024

image

Spring-Gaus reconstructs the appearance, geometry, and physical dynamics properties of elastic objects from video observations. Spring-Gaus enables future prediction and simulation under different initial states and environmental parameters.



Abstract

Reconstructing and simulating elastic objects from visual observations is crucial for applications in computer vision and robotics. Existing methods, such as 3D Gaussians, model 3D appearance and geometry, but lack the ability to estimate physical properties for objects and simulate them. The core challenge lies in integrating an expressive yet efficient physical dynamics model. We propose Spring-Gaus, a 3D physical object representation for reconstructing and simulating elastic objects from videos of the object from multiple viewpoints. In particular, we develop and integrate a 3D Spring-Mass model into 3D Gaussian kernels, enabling the reconstruction of the visual appearance, shape, and physical dynamics of the object. Our approach enables future prediction and simulation under various initial states and environmental properties. We evaluate Spring-Gaus on both synthetic and real-world datasets, demonstrating accurate reconstruction and simulation of elastic objects.


Pipeline

image

(a) Static Scene Reconstruction: We start by reconstructing static 3D Gaussians from the first frames of the multiview videos. (b) Refining 3D Gaussians: We extract a set of anchor points to allow efficient simulation, which leads to appearance drift. We refine the 3D Gaussians to better model the appearance during simulation. (c) Dynamic Reconstruction: Our 3D Spring-Mass model simulates anchor points and updates the positions of Gaussian kernels. Upon completion of optimization, we obtain a simulatable 3D object that accurately models its dynamics.

Dynamic Reconstruction & Future Prediction
Synthetic
torus paste apple banana cross duck cream
doraemon chess droplet rope E C V
Observation
Reconstructon
Prediction

Real Capture
burger dog bun pig potato
Observation
Reconstructon
Prediction


Simulatable Digital Assets
Edit Ground Height (Interactive)
Trained Assets
Ground 0.0m
Simulate Results
Loading...

Drag the progress bar to control the height of the ground.


Edit Gravity
Normal
Zero Gravity
Hypergravity

Normal
Zero Gravity
Hypergravity

Edit Boundary Condition
Normal
Smooth Ground
Sticky Ground

Normal
Smooth Ground
Sticky Ground

Edit Physical Property
Softer
Normal
Harder

Random Init Velocity

Movable Ground

BibTeX

@article{zhong2024springgaus,
        title     = {Reconstruction and Simulation of Elastic Objects with Spring-Mass 3D Gaussians},
        author    = {Zhong, Licheng and Yu, Hong-Xing and Wu, Jiajun and Li, Yunzhu},
        journal   = {European Conference on Computer Vision (ECCV)},
        year      = {2024}
    }