GaussianDreamerPro
Text to Manipulable 3D Gaussians with Highly Enhanced Quality
Abstract
Recently, 3D Gaussian splatting (3D-GS) has achieved great success in reconstructing and rendering real-world scenes. To transfer the high rendering quality to generation tasks, a series of research works attempt to generate 3D-Gaussian assets from text. However, the generated assets have not achieved the same quality as those in reconstruction tasks. We observe that Gaussians tend to grow without control as the generation process may cause indeterminacy. Aiming at highly enhancing the generation quality, we propose a novel framework named GaussianDreamerPro. The main idea is to bind Gaussians to reasonable geometry, which evolves over the whole generation process. Along different stages of our framework, both the geometry and appearance can be enriched progressively. The final output asset is constructed with 3D Gaussians bound to mesh, which shows significantly enhanced details and quality compared with previous methods. Notably, the generated asset can also be seamlessly integrated into downstream manipulation pipelines, e.g. animation, composition, and simulation etc., greatly promoting its potential in wide applications.
Framework
Our framework can be divided into two parts: basic 3D asset generation and quality enhancement with geometry-bound Gaussians. In the basic 3D asset generation stage, we generate initial 3D assets, which are used to initialize 2D Gaussians, obtain basic 3D assets under the optimization of the 2D diffusion model, and export as a mesh. In the quality enhancement with geometry-bound Gaussians stage, we bind 3D Gaussians to the mesh, and also obtain enhanced 3D assets under the optimization of the 2D diffusion model.
Comparison Results
Qualitative comparisons between our method and GaussianDreamer, LucidDreamer and DreamCraft3D.
Application
Animate and simulate the generated 3D assets.
More Generated Samples
More generated samples by our GaussianDreamerPro.
Elsa in Frozen Disney, head
Flying Dragon, highly detailed, breathing fire
Joker wearing top hat, head, photorealistic, Fujifilm XT5, 8K, HD, raw
Zeus, head
A fuzzy pink flamingo lawn ornament
a boy in mohawk hairstyle, head only, 4K, HD, raw
a DSLR photo of a pair of tan cowboy boots
Viking axe, fantasy, weapon, blender, 8k, HD
a DSLR photo of a car made out of sushi
A chameleon perched on a tree branch
Deadpool
a DSLR photo of a plate of fried chicken and waffles with maple syrup on them
a DSLR photo of an ice cream sundae
a DSLR photo of an origami motorcycle
An intricate ceramic vase with peonies painted on it
More Research
BibTeX
@article{GaussianDreamerPro,
title={GaussianDreamerPro: Text to Manipulable 3D Gaussians with Highly Enhanced Quality},
author={Yi, Taoran and Fang, Jiemin and Zhou, Zanwei and Wang, Junjie and Wu, Guanjun and Xie, Lingxi and Zhang, Xiaopeng and Liu, Wenyu and Wang, Xinggang and Tian, Qi},
journal={arXiv:2406.18462},
year={2024}
}