There’s yet another way to generate 3D meshes from 2D images: PartCrafter.
The explosion of AI generative tools has taken hold in the 2D space, but is still working forward in the 3D space. There are several text-to-3D applications, but one that will also be required is “image-to-3D”.
Image to 3D involves submitting a single 2D image of a subject to an AI. The AI would then recognize the image and generate a full 3D model for the subject, including the parts not shown in the 2D image. This is much like your own understanding of the unseen parts of a person, building, or vehicle.
There are a few image to 3D experiments going on, but few are ready for prime time.
A new approach is described in a research paper by the unlikely combination of Peking University, ByteDance, and Carnegie Mellon University entitled “PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformers”.
PartCrafter differs from previous approaches in that it accepts a 2D image and generates multiple meshes from the components identified in the image. Previous systems handling complex objects would require the object to be segmented first, and then each would individually be transformed into 3D meshes.
Their approach is to represent the components of a structure with a set of “disentangled tokens”, and developing a “hierarchical attention mechanism” that provides communication across and between parts during generation.
This required an unusual training approach whereby they used component descriptions in addition to training on entire objects. In all, they trained on 50,000 objects containing 300,000 parts in this manner.
The results showed improvements in quality scores for mesh fidelity, geometry independence between parts, inference speed, and better occlusion handling.
Work continues on this important feature, which I believe will become a key technology in the near future. It will be combined with generative image technologies to create a two-stage method to create object designs: text to image creates an image of the part, then image to 3D converts it into a usable mesh for printing.
Via ArXiv (PDF) and GitHub
