Learn about the AI model DreamEditor that can edit 3D scenes through text prompts

撰文：Ekrem Chetinkaya

Source: MarkTechpost

Image source: Generated by Unbounded AI tool

In recent years, the field of 3D computer vision has been flooded with NeRF technology. A breakthrough in this field, NeRF is able to reconstruct and synthesize novel views of a scene. It can capture and model underlying geometric and appearance information from multi-view image collections.

Through neural networks, NeRF provides a way beyond traditional data-driven approaches. The neural network in NeRF learns to represent complex relationships between scene geometry, lighting, and gaze-dependent appearance, enabling highly detailed and realistic scene reconstructions. A key strength of NeRFs is their ability to generate realistic images from any desired viewpoint in a scene, even regions not captured by the original image set.

The success of NeRF has opened up new possibilities in computer graphics, virtual reality and augmented reality, enabling people to create immersive and interactive virtual environments that closely resemble real-world scenarios. Therefore, there is great interest in further advancing the field of NeRF.

But some disadvantages of NeRF also limit their application in real-world scenarios. For example, editing neural fields is a significant challenge due to the implicit encoding of shape and texture information in high-dimensional neural network features. Although some methods try to address this problem with exploratory editing techniques, they often require a lot of user input and it is difficult to achieve precise and high-quality results.

The ability to edit NeRF can open possibilities for real-world applications. However, all attempts so far have been insufficient to address these issues. But now, there's a new player in the game -- DreamEditor.

DreamEditor allows editing 3D NeRF. source:

DreamEditor is a user-friendly framework that allows intuitive and easy modification of neural fields using text cues. By using a grid-based neural field to represent the scene, and adopting a step-by-step editing framework, DreamEditor can achieve a variety of editing effects, including remapping, object replacement, and object insertion.

Mesh representation facilitates precise local editing by converting 2D editing masks into 3D editing regions, while also separating geometry and textures to prevent excessive deformation. The step-by-step framework combines a pre-trained diffusion model with SDS (score distillation sampling), allowing efficient and accurate editing based on simple text prompts.

DreamEditor follows three key stages to facilitate intuitive and precise text-guided 3D scene editing. In the initial stage, the original neural radiation field is transformed into a grid-based neural field. This grid representation enables spatially selective editing. After conversion, it employs a custom text-to-image (T2I) model trained on a specific scene to capture the semantic relationship between keywords in text cues and the visual content of the scene. Finally, the edited modification is applied to the target object within the neural field using T2I diffusion mode.

DreamEditor enables accurate, step-by-step editing of 3D scenes while maintaining a high level of fidelity and realism. This step-by-step approach, from grid-based representation to precise positioning and controlled editing through diffuse patterns, enables DreamEditor to achieve highly realistic editing results while minimizing unnecessary modifications in irrelevant areas.