Meet LegoGPT that thinks in bricks: AI that builds real-world lego models from text prompts

A new artificial intelligence model namedLegoGPT has been introduced by researchers at Carnegie Mellon University, offering a fascinating look into the potential of generative AI in physical design. The system is capable of producing three-dimensional Lego structures based on written descriptions—ensuring they are not only visually accurate but also structurally sound.

Built upon a fine-tuned version of theLLaMA-3.2-Instruct model with one billion parameters, LegoGPT is designed to interpret natural language prompts and produce corresponding Lego builds that obey the laws of physics. To validate structural stability, the model incorporates a mathematical optimisation tool calledGurobi, which evaluates each design’s resilience and balance.

The AI is trained on a purpose-built dataset calledStableText2Lego, which includes over 47,000 Lego constructions and more than 28,000 distinct 3D models. Each entry in the dataset is accompanied by detailed captions, design code, and 3D model files to enrich training accuracy.

LegoGPT has been released as an open-source project under the permissive MIT licence and is now available for public use via GitHub. Users can prompt the system with imaginative requests such as a “streamline elongated vessel” or a “backless bench with armrest,” and receive ready-to-build designs that are both aesthetically pleasing and physically viable.

To ensure the model’s outputs could withstand real-world construction, researchers tested the AI-generated designs using a dual robotic assembly system. The robots were instructed to physically construct the designs, evaluating their stability post-build. Human participants also attempted assembly, helping the team assess performance with less precise manipulation. Impressively, the study reports that 99.8 per cent of the generated structures passed the stability benchmark.

This breakthrough not only highlights the intersection of generative AI and engineering but also paves the way for future research in AI-assisted construction and design. By making both the model and dataset publicly accessible, the team hopes to foster innovation in physics-aware generative modelling and inspire new applications across education, design, and robotics.