Significant slow down and lag with moving trackables & VRPN

gkfricke
Posts: 31
Joined: Sun Feb 14, 2010 5:42 pm
Location: Durham, NC
Contact:

Significant slow down and lag with moving trackables & VRPN

Post by gkfricke » Sun Aug 11, 2013 6:46 pm

I'm trying to track up to ten different trackables simultaneously.
These trackables are independent robots within my ~4m x 4m x 3m lab space, and each trackable is made up of 3 markers.

I have 22 V100:R2 cameras on 4 OptiHubs, and I'm currently running TrackingTools 2.4.0-beta1 64-bit. It's running on Windows7 Pro 64 with a 2010-era Xeon quad-core processor and 24GB RAM.

I have noticed that whenever I start my experiments (and my robots start moving) the VRPN stream becomes quiet laggy (up to 2-second dropout), and the indicated camera frame rate drops sharply from 100fps to as low as ~30fps.

Each robot initiates VRPN communication for each trackable -- I'm not sure exactly what happens on the TT side, but could this be the source of the lag, or is it because I have too many trackables moving?

gkfricke
Posts: 31
Joined: Sun Feb 14, 2010 5:42 pm
Location: Durham, NC
Contact:

Re: Significant slow down and lag with moving trackables & V

Post by gkfricke » Tue Sep 10, 2013 12:33 pm

I've done a little more work trying to figure this out.
My problem seems to be related to the VRPN streaming engine, somehow.

I'm also going to post this problem to the VRPN listserv, but can anyone help me with this?

Currently, I have up to 6 moving trackables (robots), and each of them subscribes to the VRPN for each trackable (up to 7 total).

Does this mean that the VRPN is "broadcasting" 42 streams, i.e., 6 identical copies of all 7 trackables?

I'm including a screenshot of the network traffic from Windows Task Manager. Not the periodic nature of the slowdowns; is that some kind of clue? Each time block is approximately 3 seconds, so the span where data is "good" is about 1 minute. (TrackingTools reports a significantly decreased Camera frame rate during the "bad" spans.)
Attachments
NetworkHistory_6bots7tracks_notMoving.png
6x7 subscriptions, nothing moving
NetworkHistory_6bots7tracks_notMoving.png (65.86 KiB) Viewed 5629 times

gkfricke
Posts: 31
Joined: Sun Feb 14, 2010 5:42 pm
Location: Durham, NC
Contact:

Re: Significant slow down and lag with moving trackables & V

Post by gkfricke » Tue Sep 10, 2013 12:33 pm

In comparison, if I only activate 1 robot (subscribing to 7 trackables) the stream stays fine, at least for about 5 minutes:
Attachments
NetworkHistory_1bot7tracks_notMoving.png
1x7 subscriptions
NetworkHistory_1bot7tracks_notMoving.png (93.82 KiB) Viewed 5629 times

gkfricke
Posts: 31
Joined: Sun Feb 14, 2010 5:42 pm
Location: Durham, NC
Contact:

Re: Significant slow down and lag with moving trackables & V

Post by gkfricke » Tue Sep 10, 2013 2:27 pm

Is there any possibility that upgrading to Motive would solve this problem?

NaturalPoint-Dustin
Posts: 612
Joined: Tue Mar 19, 2013 5:03 pm

Re: Significant slow down and lag with moving trackables & V

Post by NaturalPoint-Dustin » Tue Sep 10, 2013 2:50 pm

Hello,

Thank you for submitting a post on our forums. I will look into this and have an answer for you shortly.

Best Regards,
Dustin
Technical Support Engineer
OptiTrack | TrackIR | SmartNav

beckdo
NaturalPoint Employee
NaturalPoint Employee
Posts: 519
Joined: Tue Jan 02, 2007 2:02 pm

Re: Significant slow down and lag with moving trackables & V

Post by beckdo » Tue Sep 10, 2013 2:59 pm

gkfricke,

I think the key thing to determine is when you're tracking the rigid bodies from within the Tracking Tools, is it laggy? Also, at the bottom of the Trackables Panel you'll see processing time shown. Take a look at that and see what the time is that it takes to solve all those rigid bodies.

If you're good to go in Tracking Tools, then you should definitely be able to stream VRPN.

The latest Motive release (v1.5 Beta1) has improved solving both in performance and in the quality of the solution. It's basically a completely new solving engine. More improvements are coming as well. It's much higher performing and if since you've been using 3 marker rigid bodies in Tracking Tools you've probably noticed sometimes the solver can get stuck on inverted solutions. This is not the case in Motive.

However, it important to assess your current situation in Tracking Tools.

The other likely cause of this is that in your client software you're not servicing the VRPN client quickly enough. If you get behind, so with the data.

gkfricke
Posts: 31
Joined: Sun Feb 14, 2010 5:42 pm
Location: Durham, NC
Contact:

Re: Significant slow down and lag with moving trackables & V

Post by gkfricke » Tue Sep 10, 2013 3:32 pm

Thank you for that reply, beckdo.

When I am not streaming (i.e., VRPN broadcast checkbox checked but clients not connected) the camera frame rate is rocksteady at 100FPS; Point Cloud: 1.2-1.9ms; Latency 10.0-11.2ms.

The Trackable Engine Statistics at the bottom of the Trackables pane & tab shows a steady Process Time of 0.310+/-0.005ms with all 7 defined trackables tracked, with occasional jumps to 0.400ms.

With one machine requesting seven streams: no change
With six machines each requesting seven streams, the TT GUI suddenly gets very laggy, and the reported Camera frame rate drops below 60FPS (sometimes down under 40). Process Time remains ~0.315ms, Latency jumps as high as 300ms (but doesn't seem to update), Point Cloud stays at or below 2ms (but doesn't update), Display drops from ~35FPS typical to 10-15FPS.

On the client side, the lags are consistent with what I see in Task Manager. As far as I can tell, the client side data handler always discards old data if there's newer data in the queue; could a queue backup on the client side cause TT to bog?

beckdo
NaturalPoint Employee
NaturalPoint Employee
Posts: 519
Joined: Tue Jan 02, 2007 2:02 pm

Re: Significant slow down and lag with moving trackables & V

Post by beckdo » Wed Sep 11, 2013 4:50 pm

Ok, this sounds like a bottleneck with the VRPN library unfortunately. We can continue to pursue the cause via working with the guys behind VRPN.

You might want to consider using NatNet if that's possible. It optionally uses multicast (multicast is the default), so there is no penalty for additional clients. You can download and run the latest NatNet (v2.4) sample client applications to confirm you're receiving the tracking data. Then I would suggest running that sample client on all your machines to confirm you can receive the data without slow down. If that proves fruitful, which it likely will, you might want to consider using NatNet.

Also, another thing to consider is to give Motive v1.5 Beta 1 a test to see if the problem is exhibited in our latest software.

gkfricke
Posts: 31
Joined: Sun Feb 14, 2010 5:42 pm
Location: Durham, NC
Contact:

Re: Significant slow down and lag with moving trackables & V

Post by gkfricke » Thu Sep 12, 2013 10:01 am

Thanks for the reply.

I am using Robot Operating System (ROS) packages on my robots, and it does look like there's a NatNet "mocap" package, supporting v2.2. With Motive 1.0.2, the NatNet version is 2.4.0.0; will this be a problem?

I will try to implement that this afternoon and see how it goes.

beckdo
NaturalPoint Employee
NaturalPoint Employee
Posts: 519
Joined: Tue Jan 02, 2007 2:02 pm

Re: Significant slow down and lag with moving trackables & V

Post by beckdo » Fri Sep 13, 2013 10:33 am

You should run the latest NatNet 2.4.0, it should work with your current Tracking Tools install or the latest Motive. Keep us posted.

Post Reply