forums.naturalpoint.com

Discussion and Support for the OptiTrack, SmartNav and TrackIR brands by NaturalPoint

Camera images to OpenCV

Share source code samples for use with the Camera SDK

by LinusA » Fri Feb 04, 2011 11:21 am

Hi, for all of you who'd like to get the bitmap of a camera frame into an OpenCV matrix, have a look at the code sample below. This post applies to OpenCV 2.2 and CameraSDK 1.0, but should be adaptable to other versions easily.


Code: Select all
// pMyCam is of type CameraLibrary::Camera *
int camWidth  = pMyCam->Width();
int camHeight = pMyCam->Height();

// we set up a new OpenCV matrix for our frame image
// note that we use 1 channel with a depth of 8 bit,
// i.e. we want to get the data as grayscale picture
cv::Mat matFrame(cv::Size(camWidth, camHeight), CV_8UC1);

// which is why we also set this constant to 8
const int BACKBUFFER_BITSPERPIXEL = 8;


// later on, when we get the frame as usual:
CameraLibrary::Frame * pFrame = pMyCam->GetFrame();

// as usual, check pFrame against NULL, and then the main call:
pFrame->Rasterize(camWidth, camHeight, matFrame.step, BACKBUFFER_BITSPERPIXEL, matFrame.data);

// now we can do everything we want with matFrame, for example just:
cv::imshow("My Window Name", matFrame);

// as always, don't forget this:
pFrame->Release();



If you want the frame as 3-channel color RGB matrix (for further processing), use:
Code: Select all
// 3 channels
cv::Mat matFrame(cv::Size(camWidth, camHeight), CV_8UC3);

// 3*8 = 24
const int BACKBUFFER_BITSPERPIXEL = 24;


Please note that for any bit-depth != 8 (i.e. anything else than the first example), the call to Rasterize() will be slower. On my quadcore, it takes about 0.5 ms per call (with an V100R2 cam, MJPEG Mode), and about twice as long when I use more than 1 channel...

Hope this helps, feel free to comment etc.

Linus
LinusA
 
Posts: 37
Joined: Mon Nov 08, 2010 8:43 am
Location: Aachen, Germany

by davidtheory » Mon Feb 28, 2011 9:57 am

Hey Linus, can you get the images from more than 1 camera at the same time? I have 6 cameras connected to a OptiHub, how can I choose which camera to use at a specific time?

This is useful, thanks!
davidtheory
 
Posts: 10
Joined: Thu Jan 13, 2011 4:51 am
Location: Porto, Portugal

by Birch » Mon Feb 28, 2011 3:32 pm

David,

One thing to keep in mind is the raw grayscale images at full frame rate and image size can consume a lot of USB bandwidth. If you are only capturing from a single camera at a time then you may be able to achieve full frame rate, but you might also see some dropped frames. If you are trying to stream from multiple cameras then you might want to consider using the temporal or spatial decimation settings, or switching to MJPEG grayscale mode, as a way to reduce the bandwidth consumption.

Here are two FAQ entries on the topic.
http://www.naturalpoint.com/optitrack/s ... .html#q204
http://www.naturalpoint.com/optitrack/s ... .html#q205

For information about how to enumerate and use multiple cameras you can check the CameraList sample and the "Multiple Cameras" documentation.
Birch
NaturalPoint Employee
NaturalPoint Employee
 
Posts: 1139
Joined: Thu Jan 30, 2003 5:00 am
Location: Corvallis, Oregon

by LinusA » Wed Mar 02, 2011 8:50 am

Hi David,

as Birch suggested, I did look at the examples from the SDK. I also browsed the header files of the camera SDK. It happens that I also use 6 cameras. It all works flawlessly.

I stripped everything down to the bare minimum for my needs, that means, I initialize the camera manager:
Code: Select all
CameraLibrary::CameraManager::X().WaitForInitialization();
// + errorhandling & sanity checking


I then specify the serial numbers for my cam (via user input or configuration files) and use:
Code: Select all
CameraLibrary::Camera *pMyCam = CameraLibrary::CameraManager::X().GetCameraBySerial(serialNo);


Just make sure to call "->Release()" of the cam pointers once you're done, and then shut down the CameraManager. Just as in the examples, nothing fancy.

By keeping N OpenCV matrices available (type cv::Mat, see above), you can use the exact same code for each of the N cams.

I can use MJPEG or Precision mode with 100 FPS and 6 V100R2 cams in real time on my machine...
LinusA
 
Posts: 37
Joined: Mon Nov 08, 2010 8:43 am
Location: Aachen, Germany

by davidtheory » Wed Mar 09, 2011 8:38 am

Hi guys, thanks for the help.

My settings for one cam are:

mycam->SetVideoType(CameraLibrary::MJPEGMode);
mycam->SetAEC(true);
mycam->SetAGC(true);

is this the best configuration for speed increasing in openCV? for real time? I saw the image processing example and the way they render the image with openGL seems (my perception) to be faster than showing the image in an openCV window.
davidtheory
 
Posts: 10
Joined: Thu Jan 13, 2011 4:51 am
Location: Porto, Portugal

by LinusA » Thu Mar 10, 2011 8:39 am

[quote=davidtheory]Hi guys, thanks for the help.

My settings for one cam are:

mycam->SetVideoType(CameraLibrary::MJPEGMode);
mycam->SetAEC(true);
mycam->SetAGC(true);

is this the best configuration for speed increasing in openCV? for real time?
[/quote]
I personally found AEC and AGC of no help, so I always keep them off, but YMMV. The camera settings don't have an effect on speed with OpenCV. The "CPU expensive" operation (but still affordable) is the call to rasterize(). If you only need marker positions, you might want to go without pictures (using Precision mode).

[quote=davidtheory]
I saw the image processing example and the way they render the image with openGL seems (my perception) to be faster than showing the image in an openCV window.
[/quote]
Yes of course using OpenGL is another way, and GPU acceleration is always good. As usual, it depends on what you want to do.

I just showed a way to get the camera picture into OpenCV. I personally use OpenCV windows to display my OpenCV images (such as camera frames), but it's fast enough (2 Windows, 50 FPS is possible). I use the Qt backend of OpenCV with OpenGL support enabled anyway (compiled using WITH_QT and WITH_QT_OPENGL). You can see screenshots here:http://forum.naturalpoint.com/forum/ubbthreads.php?ubb=showflat&Number=46150#Post46150 )

If speed was important, I'd probably switch to "plain OpenGL" or Qt's QGLWidget. As I said, it depends :-)
LinusA
 
Posts: 37
Joined: Mon Nov 08, 2010 8:43 am
Location: Aachen, Germany

by zzzamarron » Mon Sep 12, 2011 3:59 am

Hi this is the code i am trying Thanks to LinusA but i only get gray image in the imshow window. I use a Trackir 4 camera, any idea? Thank you

int camWidth =camera->Width();
int camHeight = camera->Height();

Mat matFrame(Size(camWidth, camHeight), CV_8UC1);
const int BACKBUFFER_BITSPERPIXEL = 8;
namedWindow("Track",CV_WINDOW_AUTOSIZE);

// later on, when we get the frame as usual:
while(1)
{
Frame *frame = camera->GetFrame();

if(frame) {

frame->Rasterize(camWidth, camHeight, matFrame.step, BACKBUFFER_BITSPERPIXEL, matFrame.data);

imshow("Track", matFrame);


frame->Release();
}

}
zzzamarron
 
Posts: 13
Joined: Thu Jan 04, 2007 11:22 am

by LinusA » Tue Sep 13, 2011 11:34 am

[quote=malboro]Hi this is the code i am trying Thanks to LinusA but i only get gray image in the imshow window. I use a Trackir 4 camera, any idea? Thank you
[/quote]
You're using a single-channel cv::Mat, which is a gray image. I wrote above how to retrieve the data so you can do "color processing" in OpenCV with that cv::Mat object:
Code: Select all
// 3 channels
cv::Mat matFrame(cv::Size(camWidth, camHeight), CV_8UC3);

// 3*8 = 24
const int BACKBUFFER_BITSPERPIXEL = 24;


But please note that this doesn't change the factsthat some cameras only return grayscale values. I don't know the cam you're using, but I'd guess it doesn't return color images!
LinusA
 
Posts: 37
Joined: Mon Nov 08, 2010 8:43 am
Location: Aachen, Germany

by wuzle » Thu Sep 15, 2011 12:12 pm

Can you give some pointers as to your project settings for this? I am having a ton weird runtime errors which I suspect has to do with be libs I am using.

So:
-What version of Visual Studio are you using?
-Are you still using OpenCV 2.2, and Optitrack SDK 1.0?
-Are you using the precompiled OpenCV libraries(lib or staticlib), or did you compile yourself?

Any help is appreciated.
wuzle
 
Posts: 1
Joined: Thu Sep 15, 2011 12:07 pm

by LinusA » Mon Sep 19, 2011 11:23 am

I've been compiling my project on three machines: Visual Studio 2008 on WinXP SP3 32bit, VS 2008 in Win7 32bit, and VS 2010 Win7 64bit.

In all cases I'm using Camera SDK 1.0 (but this doesn't matter, all 1.x versions work), OpenCV 2.2 with a custom patch, and Qt 4.7.3. I always compiled OpenCV myself, yes -- I get static .lib files from that. I also use OpenGL and GLEW headers and libs.

The custom OpenCV patch I use is https://code.ros.org/trac/opencv/ticket/919 , but I don't think you need that.


You should make test-projects to test all things independently, one lib at a time. Can you create an OpenCV project and display a cv::Mat? Can you create a project using Camera SDK and retrieve frame objects?

I did so myself (also including Qt tests and so on), then put everything together.
LinusA
 
Posts: 37
Joined: Mon Nov 08, 2010 8:43 am
Location: Aachen, Germany


Return to Camera SDK Code Samples

cron