Farm3D

Farm3D is a research project from the Visual Geometry Group at the University of Oxford, authored by Tomas Jakab, Ruining Li, Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi. It trains a monocular reconstruction network that predicts the 3D shape, texture, viewpoint, and lighting of an articulated animal from a single input photo, covering categories such as cows, horses, sheep, pigs, and dogs. Rather than relying on captured 3D scans or paired real-image datasets, the method uses a pretrained 2D diffusion model, specifically Stable Diffusion, to generate synthetic training views and to critique the network's outputs during training. The paper was published at 3DV 2024 and is available as an arXiv preprint. It targets computer vision researchers and graphics practitioners rather than end consumers, since the output is a reconstruction network and dataset, not a consumer app. Once trained, the network can reconstruct a 3D animal from a photo in seconds and supports downstream synthesis tasks like relighting, texture swapping between animals of the same category, and simple animation. The team released Animodel, a benchmark of textured 3D animal meshes, alongside code on GitHub. Because the model never trains on real 3D animal scans, its accuracy depends entirely on how well the diffusion model's implicit shape knowledge transfers, and its reconstructions are not validated against ground-truth 3D captures.