Real time face mocap streaming with NatNet

[davidtheory]

By error I have put this post previously in a topic called "Streaming facial data from Expression to 3ds Max". So sorry for double posting :S

----

I want to do real time face animation without motion builder (suppose we have a library for animating a rigged face model and displaying it with openGL).

So I am using NatNet 2.2 to stream markers from a face template in Arena 1.7.1. and I have some questions:

1. Is there an option in the streaming tab in Arena or a function with NatNet, to get the head rigid motion (position & orientation) directly? how can I get the head rigid motion?

2. When getting the 3D marker data with NatNet, am I getting the local positions relative to a face local reference system? or am I getting them in global world coordinates? If they are local, how can I get the base vectors that define this face local reference system?

Any help would be really appreciated.

[morgan]

Hey David,

1. Re head pos/ori - yes. You can get the pos/ori of the head by checking "Rigid Body Data" in Arena, then using NatNet to query for the rigid body list/data. See the SampleClient3D for a code example of how to do this.

2. Re marker position coord system, the Arena "Local Marker Position" check box influences whether marker data is relative to the root bone (head rigid bopdy) or global. So for local, the base vector / coordinate system would be the head rigid body / root bone you got in 1. above.

Again I would play with the SampleClient3D example and see the code and how it effects the data.

hope this helps,

Morgan

[davidtheory]

Thanks! it worked by checking the rigid body, the thing was that I did not know that sections in a face template are treated as rigid bodies when streaming with NatNet.

[davidtheory]

Hi Morgan, I have related questions on this process of streaming facial motion capture.

I did the following simple experiments.

1. I created the most simple face template composed only by the 4 headband markers (the root bone). Then I stream the facial mocap checking the boxes a) Right Hand Coordinates, b)3D markers and c.) Rigid Body Data. I wanted with this to see how to get the head motion, and I got it, in the data descriptions I get 1 rigid body and I get the position and orientation of this rigid body, that is the head motion. I was able to get the correct corresponding motion in my openGL model. OK.

2. Now I wanted to check the facial motion without the head rigid
motion. For this I created the face template as normally is done (it is not relevant how many markers I used), and I stream data by checking a) Right Hand Coordinates, b)3D markers and c.) Local Markers Positions. In my c++ NatNet client I get 1 data description which is composed of all the markers of my face template, I ignored the root bone markers and parse the rest of the markers positions which I retarget to the rig of my model, and I get exactly what I wanted, just the facial motion without the head motion. OK.

3. Now I wanted to get all at once, so I checked the boxes on the streaming tab a) Right Hand Coordinates, b)3D markers, c) local marker pos, and d.) Rigid Body Data. With this I get in my data descriptions object, 2 descriptions corresponding to rigid body and all the markers. But when my client uses the GetLastFrameOfData() which triggers its dataHandler callback function the sFrameOfMocapData* argument of this function has no rigid bodies and just a marker set (that is nMarkerSets=1, nRigidBodies=0), So I am unable to get the two motions separately. How can I do this?