DreamFusion

DreamFusion is a product that utilizes a text-to-image generative model called Imagen to optimize a 3D scene. The product features Score Distillation Sampling (SDS), a method for generating samples from a diffusion model by optimizing a loss function. SDS enables the optimization of samples in an arbitrary parameter space, including a 3D space. The product employs a 3D scene parameterization similar to Neural Radiance Fields to define this differentiable mapping. DreamFusion incorporates additional regularizers and optimization strategies to enhance geometry, resulting in coherent, high-quality normals, surface geometry, depth, and relightable NeRFs with a Lambertian shading model.

The use cases for DreamFusion involve optimizing 3D scenes for various applications, including video games, virtual reality experiences, and film production.