Unreal Engine developer Epic Games has released Live Link Face, an iPhone app that uses the front 3D sensors on the smartphone to capture live motion for facial animations in 3D projects like video games, animations or movies.
The app uses tools from Apple's ARKit framework and the iPhone's TrueDepth sensor to stream live motion capture of a user looking at their smartphone to capture 3D characters in the Unreal Engine running on a nearby workstation. Captures facial expressions as well as head and neck rotation.
Live Link Face can stream to multiple machines at the same time, and "robust timecode support and precise frame accuracy allow for seamless synchronization with other stage components such as cameras and body motion capture," according to Epic's blog post in the app's announcement. Users get a CSV of the merge shape raw data and a MOV from the smartphone's front video camera, with time codes.
To many, the iPhone's TrueDepth sensor suite didn't seem necessary, everyone was happy that Touch ID worked well for most people's purposes on previous iPhones, and there weren't many clear applications of the technology other than Face ID and Animojis. and nothing else that would justify the technology.
Now with this app, albeit for a restricted audience: Game developers and independent filmmakers can use apps like this on the iPhone for motion capture, bringing virtual characters to life with facial expressions and other movements of real human actors. This is something that would traditionally involve expensive studios out of reach for everyone but the big players.
The motion capture that the iPhone's sensors can manage isn't as accurate as that used by "triple A" game development studios or major movies, of course, so these high-end studios don't abandon their mo-cap-equipped studios to iPhones now. But now, individual creators and smaller teams can do something that was previously cost-prohibitive.
Apple recently added a lidar sensor to the iPad Pro, and there are many rumors on the internet that it will also reach at least one of the variants in the new iPhone lineup in 2020.
