Matthew Asanuma and Professor Brent Adams, Animation

Facial motion capture is used to capture the intimate and subtle motions of the facial features and apply them to a digital character’s face. Animation for the character, Davy Jones, in the Pirates of the Caribbean movies was achieved using this technique. In Overcoming the Obstacles of Motion Capture, I’ve tried to develop a relatively low-cost, yet high-performing technique suitable for a target market of students and independent artists.

There are two parts to transferring captured motion data to a digital character rig. In one part, you need to capture and track the footage. In the other part, you need to have a digital character rig that has been built to accept such motion.

As I commenced my research, a large portion of my time was invested in the part that involves the character model and rig. Over the course of many months I developed two rigs each with different approaches. For the student film, Chasm, I modeled the protagonist, Addison. I developed a rather sophisticated facial control rig for her that I hoped would someday be accepting tracked motion data. After finishing the rig I had broadened my understanding of facial motion and how to make a more successful facial rig in the future. For the student film, Owned, I modeled the supporting character, Abby, and I created the control rig for her body. I approached the creation of Abby’s control rig with scripting on my mind. I knew I would need a strong ability to script for rigs in the next facial rigs I’d be building and so I put great emphasis on learning these skills. I also applied the lessons I learned from Addison onto Abby and I feel like her topology was greatly improved over that of Addison for all types of motion.

I eventually turned my efforts onto the practical side of this project – that of capturing facial performance and tracking the motion. In order to make this technology appealing to the target market, the solution must involve four things. First, the solution must be relatively low cost. Second, any materials, hardware or software used must be widely available. Third, the compromise on quality must be minimal (in other words, results should be comparable to high-end industry systems). Fourth, results must have a reasonable turnaround time (at least somewhat faster than standard keyframe animation).

Initial attempts at facial motion capture involved a single webcam which recorded the face from head on, and some free tracking software I discovered online. However, the results were not as high quality as I would have hoped and the turnaround time was somewhat slow. I decided that my method wasn’t necessarily the problem, but the execution was. I would need to make it much more robust if it had hopes of solving the four issues mentioned earlier.

Three GoPro cameras would replace the single webcam from the first trials. This camera was necessary for three reasons: It is lightweight, and so can be easily mounted and carried on the head; it can record at various levels of high-resolution; and it can record at high frame-rates. The resolution and the frame-rate of these particular cameras both help solve the same problem – tracking the data. A recording that is too low in resolution or that has too few frames per second does not provide enough visual information to track the rapid and subtle motions of the face.

Cameras were mounted onto a salvaged hard-hat. Quarter-inch aluminum rods (bought at a hardware store) served as adjustable boom arms, holding the cameras at variable distances from the face. White air-soft BBs, glued to small patches of electrical tape were used as adhesive markers and were placed at key points on the face. All materials and hardware used are available at common stores.

The cameras and their placement would be the main solution for ending up with a high-quality result. Front and side-facing cameras would not only allow for the recording of x-and y-axis motion (as with my preliminary tests) but would also allow the user to capture depth motion along the z-axis. I then would use Match Mover (a software program that is usually bundled with Maya) to track the motion from the digital video. From there a free software program would be used to transfer that captured motion and transfer it into Maya where it could be used to drive a digital model of a face.

While developing this system, I encountered great difficulty when it came to tracking the motion and putting it into a digitally usable medium. Facial motions are fast. Despite recording at 120 frames per second at a resolution of 1280 x 720, the tracking software I had at my disposal still had difficulty at keeping up with the lightning movements of the lips and the eyes (which coincidentally are most necessary for conveying emotion). I also struggled with proper brightness and contrast in the video which hindered successful tracking.

In the end, I did not achieve what I set out to do. Although I learned much about rigging and produced a couple successful rigs in the process, I did not finish my project with tangible proof of a usable motion capture solution. I spent a great deal of time tracking and re-tracking data. Forty facial markers, recorded at 120 frames per second, from three different angles, meant that for one minute of footage I’d be tracking through over a half million frames. For much of it, the software could track automatically (a process that in and of itself was time consuming), but for quick movements, the software would lose the point in the image and I’d have to re-align it. This meant that I had to do a lot of baby-sitting. Much of my tracks were not usable, and with every failure I learned a great deal of how to get better and more-usable captures. I learned that a few seconds of footage require a good many hours of preparation and post-processing. My methodology for creating ideal capture settings has shifted greatly and I will surely continue to learn as I continue refining this process.

In conclusion, this story and this experiment are truly far from conclusion. I still feel like there is a great deal to be learned and I still feel passionate about finding an ideal solution. I feel as though time is the only limitation that kept me from achieving my goal. I’m grateful to have the resources I need to further my research, and I fully intend to do so. I will continue to dedicate time to this project in the hope that one day I will be able to present a solution viable for the targeted market.