date
type
status
slug
summary
tags
category
icon
password
October 13, 2023 • 3 min read
by Simon Meng, See original
Since I released the 3D generation tool Dreamfields-3D last October (I, a programming novice, built an AI tool for generating 3D models from text (open-sourced)), it's hard to believe a year has passed. Recently, numerous AIGC 3D generation algorithms have emerged, and the latest release, DreamGaussian, feels like we've nearly reached the stable diffusion moment for 3D generation—challenging the "quality, speed, and generalization" triangle of impossibility!
Example of mesh quality exported from the DreamGaussian generation model.
🥰DreamGaussian = Dreamfusion + 3D Gaussian Splatting iteration + mesh export + SD optimized texture.
DreamGaussian technical framework.
✨Dreamfusion is the first algorithm to transplant the 2D generation capabilities of stable diffusion to 3D Nerf generation (using an iterative approach known as SDS loss, detailed in my earlier popular science article: From Hand Modeling to Mouth Modeling: An Overview of the Latest AI Algorithms for Generating 3D Models from Text).
👀3D Gaussian Splatting is currently the state-of-the-art method for 3D scene representation using neural radiance fields, achieving the best quality and speed in 3D scene reconstruction (see this popular science article: Next-Generation Nerf, Gaussian Splatting is Here~).
💕By combining these, we can use SD + 3D Gaussian to quickly (within 1 minute) iterate a rough Gaussian field from text/image input; then export the Gaussian field as a mesh and UV texture, and use SD to continue optimizing the texture without changing the mesh (also within 1 minute).
😍The result of this entire process is: achieving SOTA (state-of-the-art) quality + short iteration time (about 1.5 minutes on a 3090) + support for various tasks (text to 3D & image to 3D) + strong generalization (leveraging the generalization ability of the SD model).
Examples of DreamGaussian converting images to 3D models.
🤔However, considering that 3D Gaussian Splatting has only been open-sourced for three months, there is still significant room for optimization. After the release of the Dreamfusion algorithm, many other algorithms leveraging SD model capabilities for 3D generation have emerged, incorporating various tricks to improve performance, which have yet to be introduced. Therefore, in the foreseeable year ahead, 3D generation algorithms will rapidly iterate, transitioning from the "SD 1.0" era to the "controlnet + SDXL" era. (P.S.: Who remembers that stable diffusion was only released last August, and it's just over a year now? We use SD so naturally, it's almost like breathing. 🐶)
Example of DreamGaussian converting text to 3D models.
Example of DreamGaussian converting text to images + 3D models.
🤗Related resources:
_▶ demo (play instantly): https://huggingface.co/spaces/jiawei011/dreamgaussian_
_▶ site: https://dreamgaussian.github.io/_
_▶ paper: https://arxiv.org/pdf/2309.16653.pdf_
_▶ colab - image to 3D: https://colab.research.google.com/drive/1sLpYmmLS209-e5eHgcuqdryFRRO6ZhFS?usp=sharing_
- 作者:Simon Shengyu Meng
- 链接:https://shengyu.me//article/3D-moment4SD-en
- 声明:本文采用 CC BY-NC-SA 4.0 许可协议,转载请注明出处。
相关文章
How I Used AI to Create a Promotional Video for Xiaomi's Daniel Arsham Limited Edition Smartphone
3D scene editing has entered the era of AI text interaction
The 2022 Venice - Metaverse Art Annual Exhibition: How Nature Inspires Design
Hidden Time Space 5: Cloudscape Artistry — "Vintage" AI Model Generates Retro Chinese-Style Landscape Animation
The Basic Principles of ChatGPT
From Hand Modeling to Text Modeling: A Comprehensive Explanation of the Latest AI Algorithms for Generating 3D Models from Text