temporary and per take kinematic association

[leith]

it would be nice to be able to establish temporary and per take kinematic association of markers for the purpose of gap filling.

scenario 1:

I am capturing a particularly difficult take that involves the talent rolling on his back while holding a bulky prop. I wish to augment the existing calibrated marker set with extra markers to aid in the reconstruction of the calibrated markers when they inevitably become occluded.

I therefore add markers to the talent to aid in being able to reconstruct a given limb from a particular difficult occlusion situation. I then have the talent quickly enter the volume and capture the new augmented marker set. I assign the new markers to the appropriate calibrated markers as a take long association.

I record the take. the solvers recognize the new markers and use them to reconstruct the calibrated markers and joints live as well as in editing mode.

I am finished with this particular take and this particular augmentation. I remove the markers. I delete the association from the live session, as it is no longer valid. It remains in the captured take, which is already on disk, safe from a system crash.

end scenario 1

scenario 2:

I have already captured a complex take with lots of occlusion. In some areas, I have added extra markers. However, I have not set up a formal association. They're just extra markers.

At this point I may want to associate them with joints, simply to allow the system to label the marker through the take. Depending on the marker's overall stability I may or may not want the solver to use it to track the skeleton.

While editing, I select the occluded marker and set in and out points for the time I wish to fill. I also set the length of blend handles to blend back into existing data. I select any number of other markers to use as associated, and set off the fill operation.

if only 1 associated marker is chosen, the occluded marker is locked to it with an offset in worldspace from either the head of the gap or the tail of the gap, and regenerated. Optionally, it may set an offset for the head and and offset for the tail, and interpolate that offset over time.

if only 2 associated markers are chosen, the occluded marker is locked to an offset from a point between them. That point is pointing to an associated marker and using an upvector for its orientation, that can interpolate from the location of the occluded marker before the gap, to its location after the gap, or optionally, it can use a static up vector. likewise, its offset may be interpolated from the head of the gap to the tail of the gap, or chosen to be static from either end.

if 3 or more markers are chosen, the occluded marker is reconstructed as a rigid body relative to the other markers. likewise, its offset may be interpolated from the head of the gap to the tail of the gap or chosen to be static from either end.

once the gap is filled, I move on to other gaps either for this marker, or other markers.

end scenario 2

notes:

markers can be purposely placed in areas that they serve two joints. For example, a marker on or close to the knee or elbow joint can be used to reconstruct markers on both the humorous and forearm, or thigh and shin respectively. A marker on the knee can be used to help reconstruct a marker on the ankle, or on the hip. such markers are both helpful and problematic. They can be used in the solving of multiple kinematic objects but probably should not be associated with those objects all of the time. Therefore, its best to not have them contribute to the solve at all times but rather be pulled in to aid in sticky gap solves when the user feels it appropriate. Its also important to note that these markers are the most subject to drift due to flex and flesh movement. They are situated on the most complex areas of flesh on the body and they simply have a tendency to move in odd ways relative to the bones and underlying joints... but they can be the difference between getting a reasonable take, and not getting a take at all.

[Jim]

Brad:

Thanks for the detailed post. I think you explain the feature request quite nicely.

We have some of this now, in a limited sense. You can over assign a part of the body with markers, and the solver will use these extra markers to position the segment when others are not in view. We also have the 45 marker skeleton, which has redundant markers by default.

Currently, you would need to assign the marks to the skeleton and use them for the entire take, but you could have a different skeleton for each take, that isn't too hard or painful to do.

The level of editing tools you are suggesting is a ways out for us and we would probably consider them to be quite complex for most of our users. Our overall goal is to have an easy to use system that just works well and is robust for capturing, as to avoid the need to deal with overmarkering. We have had success with that in testing, even rolling around on the floor.

You could also over assign markers, then just export into MB and use those tools to correct for gaps, which I think is similar to what you want. I do think we are moving towards having tools like these, but I want to set your expectation level for them as being a ways off.

PS. If we had some takes to view that showed where the current 45 overmarker setup failed and we couldn't fix it using our current tools, then we could work backwards into tools to deal with the data and offer options to advanced users to fix the capture, or just make it more robust.