Learning Visual and Motion Aware 3D Model Representations
for Semantic Retrieval

TVCG 2026: IEEE Transactions on Visualization and Computer Graphics, presented at ISMAR 2026

Sruti Srinidhi Carnegie Mellon University
Anthony Rowe Carnegie Mellon University
Bosch Research
Paper Code Dataset Video
Examples of 3D models retrieved from text, image, and video prompts

3D models retrieved for text, image, and video prompts, including ones that describe motion, like "a wooden desk that is retractable" or a video of a person running.

Abstract

Virtual Reality (VR) content creators today face lengthy trial-and-error cycles when searching for 3D models, which relies heavily on manually annotated metadata. However, this approach struggles to capture the rich visual and dynamic details inherent to 3D models. Prior work has attempted to address this by rendering models as sets of images, but has largely remained restricted to category-level search and static geometry. In VR environments where asset behavior is often as important as appearance, we present a visual and motion aware 3D retrieval method that represents each model through multiview renders and short animations, extending search beyond static geometry to include dynamic behaviors. Pretrained image–text and video–text encoders extract features that are combined into a unified embedding by training a lightweight multimodal fusion model. This enables 3D model retrieval from text, images, and videos without requiring abundant and detailed ground truth data, supporting more nuanced and semantically meaningful search. On text queries referencing both appearance and motion, our method achieves a 23-percentage-point improvement in top-5 retrieval accuracy on the Objaverse dataset, while also attaining competitive accuracy on the ModelNet40 dataset and maintaining robustness as the model library size scales. A user study further shows that participants strongly preferred our retrievals compared to the one used by Sketchfab, one of today's most widely used 3D repositories. To support future work, we release 26k multi-view renders, animation clips, and 500 manually annotated visual and motion aware captions for the Objaverse dataset.

87.3%Top-5 accuracy on Objaverse
+23 ptsTop-5 over the next best prior work
57%of Top-5 accuracy retained at 26k models
3.96 vs 1.72Motion relevance vs. Sketchfab (out of 5)

How It Works

Multiview images and an animation video of a 3D model are encoded by frozen pre-aligned encoders and fused into a unified embedding space shared with text prompts

Unlike images and videos on the Internet, 3D models rarely come with detailed captions, especially ones that describe motion, which makes it difficult to train a dedicated 3D–text encoder. Instead, we adopt a proxy strategy: each 3D model is rendered from six canonical views (±x, ±y, ±z) and as a short animation clip, and encoded with frozen, pre-aligned LanguageBind image and video encoders. A lightweight image-fusion model, trained without any captions using cosine-similarity and batch-wise contrastive self-consistency losses, pools the six view embeddings into one multiview image embedding. This is then averaged with the video embedding to form the final 3D model representation. Surprisingly, this simple mean outperforms a trained multimodal fusion model, suggesting that the pretrained encoders are already well aligned across modalities.

Because the final embedding lives in the same space as the text encoder, models can be retrieved directly by cosine similarity to a text, image, or video query. All embeddings are stored in a ChromaDB vector database, giving fast nearest-neighbor search across tens of thousands of models. The method is encoder-agnostic: any model with a unified image–video–text space can be swapped in.

Scaling to Larger Model Libraries

Top-5 retrieval accuracy retained as the model library grows from 500 to 26k models

Top-5 accuracy retained (relative to a 500-model library) as the library grows to 1k, 10k, and 26k models. Baselines such as OpenShape and ULIP drop sharply, while our method retains 57% of its top-5 accuracy at 26k candidates.

Granularity Stress Test

Relative similarity scores for three prompts over six variants of one character: punching, jumping, or standing, with a red or blue cap

Six variants of the same character performing three actions (punching, jumping jacks, T-pose), each with a red or blue cap. Scores are row-normalized relative similarity. Action-only prompts rank both matching animations on top. Prompts that combine action and color isolate the exact model. Visual attributes like cap color often act as stronger anchors than the motion, since they show up in both the image and the video embeddings.

Video

Dataset

We release six canonical multiview renders and a 6-second animation clip for over 26,000 Objaverse models. To our knowledge, this is the first large-scale release of rendered videos for 3D models. We also release a benchmark of 500 manually annotated captions that describe both appearance and motion, plus the Blender rendering pipelines used to make the dataset. Everything is available on Hugging Face, and the code is on GitHub.

Try It

git clone --recurse-submodules https://github.com/srutisrinidhi/3DModelSearch
cd 3DModelSearch
export LANGUAGEBIND_PATH="$(pwd)"

python scripts/main.py --query "a red sports car" \
    --models_csv out/models_data.csv --persist_directory databases/model_database

See the README for full setup, precomputed downloads, and the interactive Gradio demo.

Acknowledgments

This work was supported by the NSF under award CNS-1956095 and Bosch Research.