An AI generated mug might look magnificent, until you try to pour something into it.
That’s because today’s AI 3D model generators are quite good at producing objects that appear correct, but they may not actually function. That’s because the AI’s understanding how those objects work in the real world isn’t complete.
A team of researchers from MIT, Google, and Northeastern University has developed an interesting solution to this problem. Their system, called “InstructMesh”, allows users to repair AI generated 3D models without requiring conventional 3D modeling skills.
The approach is described in a research paper, “InstructMesh: Selective Refinement of Generative 3D Models for Fabrication”.
The researchers first investigated how serious the problem actually is. They selected 120 popular 3D models from Thingiverse, rendered images of them, and then used Microsoft’s TRELLIS AI system to reconstruct the models from those images.
An astonishing 78.3% of the generated models contained multiple types of geometric flaws. On average, each model had 2.4 problems.
The most common problem was unwanted geometric features, appearing in 65.8% of the models. Missing openings affected 48.6%, while incorrect hollowing appeared in 36.9%.
These weren’t just minor cosmetic issues. Imagine a whistle without an internal chamber, a mug with its opening covered, or a mechanical component with two moving parts fused together.
All might look perfectly reasonable in a rendered image, but would be completely useless after 3D printing. AI generated models of today look great, but unless they are just a decoration, they probably don’t work.
InstructMesh attempts to solve this by allowing users to identify the problematic region on the generated 3D model and describe what needs to change.
For example, a user might highlight the sealed opening of a mug and request that it be opened (see image at top). Alternatively, they could select a thin section and request additional thickness. Or whatever else might be wrong with the generated 3D model.
The system combines TRELLIS with GPT-4, but the interesting part is how the changes are actually made.
Instead of directly manipulating the triangles of the generated mesh, InstructMesh operates on an intermediate voxel representation used internally by TRELLIS.
The researchers developed six geometric operations, including extrusion, expansion, filling, trimming, erosion, and flattening. These can add or remove material in selected regions.
GPT-4 interprets the user’s instruction and selects an appropriate operation and its parameters. The modified voxel representation is then processed by TRELLIS to regenerate the corrected mesh.
The user can preview the proposed changes before accepting them, and can also use sliders to adjust the operations directly.
The AI doesn’t have to invent complicated geometry operations. It only has to interpret what the user wants and select from a limited collection of its known operations.
The researchers tested the approach with twelve participants who had no previous experience with CAD or 3D modeling.
These beginners successfully identified 90.4% of the visible fabrication problems and repaired 89.7% of the annotated flaws, according to an independent expert’s evaluation. Not perfect, but clearly a vast improvement over the raw AI generated 3D models.
The team also fabricated several corrected objects using a Stratasys J55, including a dragon-themed mug, a whistle, eyeglass frames, and an unusual octopus-shaped liquid dispenser.
Their current system uses a relatively coarse voxel representation, which limits the smallest geometric features that can be modified: you can’t operate on less than a voxel. It also depends on users recognizing problems themselves, and can’t handle precision mechanical features like moving joints.
Nevertheless, I think this approach could lead to an important change in AI-assisted 3D printing.
Currently, AI 3D generation is mainly useful for producing interesting shapes. Making those shapes functional often requires transferring them into conventional modeling software, where users must possess an entirely different set of skills. But a system like InstructMesh could start to eliminate that second requirement, which is not achievable by many people.
I suspect future AI 3D design systems will increasingly incorporate this kind of interactive geometric correction, perhaps eventually combining it with automated dimensional checking, slicing analysis, mechanical simulation and other capabilities. Eventually, it may be entirely automated and you will actually get a mug that you can use in the real world.
