- Published on
AI agents build 3D scenes from photos but have no idea if they got it right
- Authors
- Name
- DeepHorus
Researchers at the University of Maryland and AWS have introduced LEGO-Anything, a project that enables AI coding agents to reconstruct editable 3D scenes from a single photograph, alongside a benchmark revealing that these agents struggle to judge whether their reconstructions are accurate. The work highlights self-assessment and geometric accuracy as the two biggest bottlenecks in automated 3D scene generation.
The core approach, called "Image-to-Code," gives a coding agent one image and asks it to write code for Blender, the widely used 3D software. Instead of generating a scene in a single pass, the agent works iteratively: it writes code, runs it, inspects the result, and revises until the scene matches the original. Because the final output is an executable program, it explicitly captures objects, geometry, layout, and camera position, so users can inspect, edit, and query it like any other code.
To evaluate performance, the team built LEGO-Bench, which contains 208 images drawn from 104 indoor and outdoor scenes and uses 443 registered assets. Real photos lack precise 3D ground truth, while purely synthetic scenes look unrealistic, so the benchmark renders images from professionally built simulator scenes. This gives natural-looking inputs while keeping exact geometry, depth, and object assignments hidden as an answer key. Each scene is scored on three axes: validity, reconstruction accuracy, and pixel-level appearance match.
All six tested GPT configurations delivered a usable scene almost every time, but accuracy varied widely. GPT-6 Astra, the strongest tested model, reached 53.4 percent on indoor scenes and 39.6 percent outdoors, while weaker configurations scored around 15 percent. Increasing the models' reasoning budget helped substantially: Astra's score on an office subset jumped from 32.3 to 61.8 percent. Complexity and outdoor settings both degraded performance.
Analysis of the agents' work steps found three recurring failures: poor initial attempts, revisions that undid earlier progress, and unreliable self-assessment. When models had to choose which of two versions better matched the original, their geometric judgments landed near or below chance level. The researchers conclude that refinement should rely on concrete measurements rather than the agent's own judgment.
This insight led to LEGO-Plugin, a training-free extension that anchors the starting scene in the reference image, replaces self-judgment with measurements, and shields correct progress from regressions. The plugin improved all six models, with weaker agents gaining up to 62.7 percent and the top model gaining only about two points. Because each reconstructed scene is an executable program, the team also tested it for object detection, segmentation, and depth estimation without extra training. Results were usable but unremarkable, suggesting reconstructed scenes are promising foundations for vision tasks but still need refinement before they can reliably power downstream applications.
