NVIDIA Omniverse Audio2Face

NVIDIA Audio2Face-3D is a facial animation microservice within the NVIDIA ACE and Omniverse platform that converts a spoken audio track into real-time 3D facial animation. The system outputs ARKit-compatible blendshapes and includes an embedded emotion-recognition model, Audio2Emotion, that automatically detects the emotional tone of the input speech and uses it to drive expressive facial movement, removing the need for an animator to hand-key each frame. The service is built for game studios, virtual production teams, and developers building interactive digital assistants who need a character's mouth and face to sync convincingly with dialogue. It communicates over a gRPC API, supports both local inference on a developer's own machine and remote inference through a hosted microservice, and can process multiple audio streams at once so many users or characters can be animated concurrently. Developers can tune face parameters, emotion weights, and blendshape multipliers at runtime, and the service supports several output audio sample rates including 16kHz, 22.05kHz, 44.1kHz, and 48kHz. One limitation is that Audio2Face-3D is a developer-facing API and toolset rather than a finished consumer app — using it requires integration work to connect its blendshape output to a 3D character rig.