fluidImages MiniFace - Motionbuilder Plug in

[seanboyette]

Well Guys -

I am moving forward nicely on my first motionbuilder/optitrack plugin.

For those who do not know -
Its a small motionbuilder plugin, that uses one camera to create a 2d pointcloud for facial anim.

Current Status:
I have added just about all the controls/ui that I think will be needed for v1.0.
Just about all of the error checking is complete. (in the midst of unit testing myself pretty heavily - and will have another collegue begin on the first, with a proposed release of the 7th or so of January.

A full rendered animation of a facial pointcloud should be complete tommorow and I will post it here. I will also apply it to a character to see how it fares.

You can record the 2d point/cloud animation.
You can have any number of markers up to 100. (may drop this to 50 for speed)
You can create an optical point cloud.
You can change the key options for an optitrack camera - these currently include Threshhold, Intensity, FPS, Exposure.

You can change Optical Scale - You could do this via scaling the point cloud post process as well, but why not have it in the plugin?
You can Change also change the location (this is kinda my calibration if you will) of the pointcloud -

To Do:
Most Importantly
Ohh Acutal Point Tracking - right now I use the base SDK marker count and just iterate through that - assigning the count number as the marker number/name - not acceptable of course - useable, but not acceptable. I am researching techniques on how to do this - if anyone has any good reading, let me know.

Grayscale cam preview - this may be for 1.5 or shortly after a release - not sure exactly how I will do it as of yet. (Any MB programmers out there streaming stuff to the FBImage control?)

Multiple cam support for 3d space? Not sure just yet, if this is necessary - we shall see, once I start playing with it a bit more for actual work!

Any other items that unit testing deems necessary, and any requests you guys may have?

Sean

[DanB]

Hi,

Any news on this plugin? Can you post a video of the facial animation captured in motion builder?

Can your method export the point cloud as .bvh, .c3d, or .fbx?

Thanks, very intrigued about this idea.

DanB

[seanboyette]

Hey dan,

Sure - hopefully have a smaple today!

Its a motionbuilder plugin - not stand alone, so you can export to bvh and c3d, while FBX is the native format.

Sean

[seanboyette]

Update - the Actual Tracking is somewhat a necessity!
I placed 11 points, and was able to track them, however the swapping is riddiculous - I will implement a quick proximity tracker just to keep the data in the correct marker.

sean

[Jim]

Just want to clarify that Sean's app is based on the OptiTrack Free SDK, not a plug-in to Arena. We do plan on making a plug in interface to Arena in the future.

I think this app sounds like a lot of fun, can't wait to see the video. It will be interesting to see if 2D tracking is enough for you, or if 3D is really needed.

[leith]

I suspect that with some clever rigging, 2d will be enough to get some utility, as long as the head/neck movement is kept to a minimum (don't turn away from camera). There are a lot of commercial packages out there that use that kind of solution and I know of at least one major motion picture coming out in the next year that is using that kind of approach. If you think about it, the human face doesn't do much depth movement. Its mostly about swimming along the surface of the skull/musculature. A lip pucker could be an issue, but if you take it into account that the lips scrunching that close together would necessitate poking out, you can get a lot of free depth movement based on assumptions. Its not ideal for performance. You want your actors to be able to walk about a mocap volume and act as they wish without restriction. Really though, thats not possible without cameras with resolutions over 2Kx2K.

Obviously the next step up is a full 3d extraction, at least allowing for more head/neck/upper body movement. This allows for a more comfortable working environment for the talent, though the ability to capture the head/neck movement is not as advantageous as it seems. In practice, grafting a moving head onto a moving full body looks wrong. The head doesn't move in concert with the body and its just plain creepy. Its often better to graft the face movement on and go with the existing head movement that was done with the body in the first place.

Its probably worth noting that you could actually try to head mount the mocap camera for face extraction and still let your actor move around in 3d space. I know this has been done before and is currently in use on some major projects. Though I honestly don't like the idea as a general solution. Its cumbersome in its own way. I may eat my own words in the future though. Who knows?

[seanboyette]

Well Guys -

Alpha 1 of my miniFace is complete!
And - hey it works! Still needs some stability work - a little bit more refine on the tracking and some better camera on/off checking.

here is a sample with 12 points and the standard face from Motionbuilder.

http://www.fluidimages.com/OurWork/flui ... fault.aspx

I am using a brute force tracker - nothing fancy at all.

Tommorow - I want to add a few more features, and some more error tracking and will provide on the forums within the a week or so.

Sean

[Jim]

Really cool, shows what a motivated programmer can do with the OptiTrack SDK.

[seanboyette]

Yup - and to clarify even further, it is not a stand alone app, but rather a plugin based on the OptiTrack Free SDK for Motionbuilder.

Adding a few more features and should be able to publish a release soon!

Sean

[seanboyette]

Update - Grayscale/2 bit preview are now complete in Motionbuilder!

Sean