very slow NPObject property accessors

[leith]

So, I'm writing in C#. Here is my current code:

Code: Select all

/*
 * Created by SharpDevelop.
 * User: leith
 * Date: 11/19/2007
 * Time: 2:20 PM
 * 
 * To change this template use Tools | Options | Coding | Edit Standard Headers.
 */

using System;
using OptiTrack;

namespace FIEMCOptiTrack
{
    /// <summary>
    /// Description of Server2D.
    /// </summary>
    /// 
    
    public class CameraFrameJob{
        public CameraFrameJob(){}
        public FIEIPC.PostalSystem postalSystem;
        public FIEMCMessaging.CameraFrame camFrame;
        public System.Runtime.Serialization.Formatters.Binary.BinaryFormatter formatter;
        
        public void DoJob(object info){            
            FIEIPC.Message mes = new FIEIPC.Message();
            mes.Type = (int)FIEMCMessaging.MessageTypes.CameraFrame;                            
            System.IO.MemoryStream ms = new System.IO.MemoryStream();
            lock (formatter){
                formatter.Serialize(ms, this.camFrame);                            
            }
            mes.Data = ms.ToArray();
            this.postalSystem.SendMessage(mes);
            
        }
    }
    
    public class Server2D
    {
        
        private int port;
        
        public Server2D(int port)
        {
            this.port = port;
        }
        
        public void Start(){
            
            FIEIPC.PostalSystem postalSystem = new FIEIPC.PostalSystem();
            postalSystem.Listen(this.port);
            System.Runtime.Serialization.Formatters.Binary.BinaryFormatter formatter = new System.Runtime.Serialization.Formatters.Binary.BinaryFormatter();                        
            OptiTrack.NPCameraCollectionClass cameras = new NPCameraCollectionClass();
            cameras.Enum();
            cameras.Synchronize();
            foreach (NPCamera cam in cameras){
                InitCamera(cam);
            }
            FIEMCMessaging.CameraFrame camFrame = new FIEMCMessaging.CameraFrame();
            bool closeRequested = false;
            while (!closeRequested){
                int frames = 0;
                int points = 0;
                foreach (NPCamera cam in cameras){
                    OptiTrack.NPCameraFrame frame = null;
                    frame = cam.GetFrame(0);
                    while (frame != null){                        
                        if (frame.IsEmpty){
                            // TODO: handle empty frame
                        } else if (frame.IsCorrupt){
                            // TODO: handle corrupt frame
                        } else if (frame.IsGreyscale){
                            // TODO: handle greyscale frame
                        } else {
                            // frame is normal
                            camFrame.BlipCollection.Clear();
                            foreach (NPObject trkobj in frame){
                                FIEMCMessaging.Blip blip = new FIEMCMessaging.Blip();
                                blip.area = (double)trkobj.Area;
                                blip.height = trkobj.Height;
                                blip.width = trkobj.Width;
                                blip.rank = (double)trkobj.Rank;
                                blip.score = (double)trkobj.Score;
                                blip.x = (double)trkobj.X;
                                blip.y = (double)trkobj.Y;
                                camFrame.BlipCollection.Add(blip);
                            }
                            camFrame.id = frame.Id;
                            camFrame.timestamp = (double)frame.TimeStamp;
                            camFrame.freq = (double)frame.TimeStampFrequency;                            
                            camFrame.serial = cam.SerialNumber;
                            
                            frames++;
                            points = points + frame.Count;
                            CameraFrameJob job = new CameraFrameJob();
                            job.postalSystem = postalSystem;
                            job.camFrame = camFrame;
                            job.formatter = formatter;
                            System.Threading.ThreadPool.QueueUserWorkItem(new System.Threading.WaitCallback(job.DoJob));
                            
                        }                        
                        frame.Free();
                        frame = null;
                        frame = cam.GetFrame(0);
                    }
                    FIEIPC.Message inmes = postalSystem.GetMessage();
                    while (inmes != null){                        
                        //deal with requests here
                        switch (inmes.Type){
                            case (int)FIEMCMessaging.MessageTypes.CloseRequest:
                                closeRequested = true;
                            break;                            
                        }                        
                        inmes = postalSystem.GetMessage();
                    }                    
                }        
                
            }
            postalSystem.Stop();
            
        }
        private void InitCamera(NPCamera cam){
            cam.SetLED(0, true);
            cam.SetLED(1, true);
            cam.SetLED(2, false);            
            cam.Open();
            cam.Start();
        }
        
    }
}

I've found that the accessors for the properties on NPObjects are very slow. Though, I've not completely isolated them as being the only slow part of the SDK. However, if I comment out all the "blip.x = trkobj.x" lines, my loop runs fine. If however, I leave them in, the code does not execute fast enough to empty the camera queues. If I leave 1-2 in on my current machine, it barely empties them. If I leave 2-4 in, it gets about 90% of the frames. It seems to scale linearly with regard to how many times I access. To give context, on my brand spankin new quad core pentium, with 4 cameras, I get about 1/4 the frames I should due to slow execution when these lines are active. I am able to easily empty the queue and run at full framerate when they are commented. Due to the heavily mulitreaded nature of the rest of my program, I actually don't max out any core on the system at any time.

To diagnose this, I moved my serialization code to a worker thread at first, thinking it was my slow code getting in the way. However when that didn't work, I started commenting out lines until I found that it was specifically the lines that call the NPObject properties that are the culprits.

furthermore, I've run this piece of code both as a thread within a consumer application, and as its own process, wrapped in its own exe (both with a consumer app connected and without). The communication is via network socket in both cases. The consumer app does not call the naturalpoint sdk at all. And no matter the config, the NPObject accessors are the difference.

Granted, those property accessors are being called WAY more often than anything else in the NaturalPoint SDK and anything else that is coming through COM, so it could be that its a general SDK performance problem or a general COM performance problem rather than being specific to the NPObject object.

So I'm kinda stumped as to where to go now. My next approach would be to go with c++ directly in hopes of jumping over a .netCOM bottleneck that may or may not be there, but I'd rather not go that route. What do you guys think is the performance issue here? What kind of potential solutions do you see?

[leith]

not to be obnoxious but umm...

[bump]

this is kinda killing me in the long run. I've moved onto other areas of the application but I'll have to deal with this issue eventually. Any thoughts?

[beckdo]

Hey Brad,

We have seen the same thing internally through the framework. It's unfortunate and we don't have a simple fix for performance improvement.

We do however interface through the same OptiTrack COM interface for our full body motion capture as well as other products. So acceptable performance can be achieved through a native C/C++ implementation. We don't have an immediate answer for a purely C#/VB.NET application that needs both high camera and marker counts.

There are two solutions I would suggest that would yield an immediate solution to your issue. Create a native interface to the cameras in C/C++ and then wrap and interface to the native code with a managed C++/CLI wrapper. Alternatively you could create a native C/C++ camera interface in a DLL and use the framework's [DllImport()] to pull the data up to .NET. I prefer the C++/CLI approach, it's cleaner, but both should yield appropriate performance.

[leith]

Thanks Doug. I'll try and move in that direction. Thanks.

-brad

[leith]

have you guys considered doing a frame dump function?

you could return a single object that contains all the frame data that was in the queue to a static number of arrays.

something like:

Code: Select all

class frameDump{
    int[] serails; //array of serial numbers, one per frame in the dump
    int[] id; //array of frame ids one per frame
    int[] objectCount; //array of counts of objects per frame
    double[] objX; //array of x locations for all the objects
    double[] objY; //array of y locations for all the objects

}

you get the idea. there would be an array for each property in the current NPCameraFrame and NPObject objects. You'd iterate over the object arrays using the objectCount to know which object properties belong to which object in in which frame.

the theoretical advantage, would be that there is only one call per property per dump. Once that data has moved across the COM barrier, it should be native to the consuming language, as the data types are all arrays of primitive data types. Therefore, calls will not be made per property per object per frame, but rather, per property per frame dump. Perhaps cutting 8 properties per 10 objects per 120 frames per 4 cameras (38400 calls a second) into 8 properties per 60 dumps (480 calls per second). Also, its possible that an increase in cameras and objects does not increase the COM overhead with something like this.

Is this nuts?

[beckdo]

Yes, this is a great idea and we have talked about it internally. I think this would be a good stop gap to boost performance. I will see if we can get some priority on this and see if we can make it happen.

[leith]

that is great. Thanks Doug\all.

[amir]

Any new code sampels???