Real time motion capture in Unity3D with OptiTrack

[davidskp]

I am currently involved in a project where the goal is to make a live performance where the spectators see two 3D characters who are played by actors in mocap suits behind the scenes.
Right now we are using Motionbuilder, but we would like all the possibilities that Unity provides. I have worked with Unity for a couple of years now, but not with motion capture or the OptiTrack system.

Can someone say if this is even possible to do real time motion capture in Unity with the OptiTrack system and what it would require?

Thanks,
Dave

[Seth Steiling]

Hi Dave,

A real-time plugin to Unity doesn't exist at the moment, but you should be able to create your own with our NatNet SDK, which makes marker, rigid body, and skeleton data available for streaming from ARENA to custom applications. This integration would require some programming capability.

[davidskp]

Thanks alot - I will look into this and probably return with more questions

[seanboyette]

Dave,

You could certainly write/add something similar to how the motionbuilder plug works -
In short - it takes the output data (pointcloud) from the system and brings it into motionbuilder. The complexity is driving the character with the point cloud/rigid data.

The SDK is pretty straight forward to use, if you need any assistance, let me know...

Sean

[davidskp]

[quote=seanboyette]Dave,

You could certainly write/add something similar to how the motionbuilder plug works -
In short - it takes the output data (pointcloud) from the system and brings it into motionbuilder. The complexity is driving the character with the point cloud/rigid data.

The SDK is pretty straight forward to use, if you need any assistance, let me know...

Sean [/quote]

Thanks a lot - as you say the biggest problem is probably going to be applying the data to the characters in Unity. I am not a hardcore programmer and I'm hoping that we can get one on the team to help out on this.
If you have any more input please don't hesitate to share

Btw a newbie question: Does each point in the point cloud only contain coordinates (x,y,z) or also orientation or similar? As mentioned I have never worked with OptiTrack so I guess it's a good starting point to know what kind of data I'll be working with

[neill_solow]

Hi davidskp,
Are you using Arena ? If so, there is some plugin for MotionBuilder on the optitrack website.

[seanboyette]

Dave,

Optitrack is optical mocap, so it just captures markers and their x,y,z data. A c3d point cloud is also just x,y,z data of the output points, a rigid would provide you with rotation (orientation) of that data.. typically the application in use takes the input of points and creates a group of select rigids, and these rigids drive characters.. this is how arena does it (being very very very high level) and even motionbuilder.
for real time, you would take a group of rigids, providing you with the x,y,z, and rotation in order to drive the bones in unity. for speed you could even do a grouping of two point rigids. (mapping each of those rigids to a joint)

I hope this babbly makes sense..

Sean

[davidskp]

Yea i think it makes sense Its just hard for me to try things hands on since i dont have the Optitrack system with me atm. I will probably work on it next week and hopefully we'll make some kind of progress

[AlexC]

Hi,

I run a mocap facility for the University of Portsmouth in the UK.

We are trying to do something similar in Unity but using a Vicon system running their Blade software, which instead of streaming a point cloud for markers, can stream a RT skeletal hierarchy, so bone rotations can be streamed to the rigged character in Unity, no need to write a solver.

I would be very interested in collaborating with you an mybe together we could crack this.

How are you getting on with the API?

Please PM me if you are interested.

[morgan]

Hey Alex,

Arena streams marker data (3d points), skeleton data (skeleton hierarchy, joint centers, joint angles), and rigid body data (e.g. for props (swords, etc.).

The Unity client should be able to listen to any/all of these streams, so yeah a solver is not required.

The MotionBuilder plugin uses the marker data just because MotionBuilder folks are used to using the MotionBuilder solver in their pipeline.

lets us know if you guys need any help using the SDKs.

Morgan