Page 1 of 2

Camera images to OpenCV

Posted: Fri Feb 04, 2011 11:21 am
by LinusA
Hi, for all of you who'd like to get the bitmap of a camera frame into an OpenCV matrix, have a look at the code sample below. This post applies to OpenCV 2.2 and CameraSDK 1.0, but should be adaptable to other versions easily.

Code: Select all

// pMyCam is of type CameraLibrary::Camera * 
int camWidth  = pMyCam->Width();
int camHeight = pMyCam->Height();

// we set up a new OpenCV matrix for our frame image
// note that we use 1 channel with a depth of 8 bit,
// i.e. we want to get the data as grayscale picture
cv::Mat matFrame(cv::Size(camWidth, camHeight), CV_8UC1);

// which is why we also set this constant to 8 
const int BACKBUFFER_BITSPERPIXEL = 8;


// later on, when we get the frame as usual:
CameraLibrary::Frame * pFrame = pMyCam->GetFrame();

// as usual, check pFrame against NULL, and then the main call:
pFrame->Rasterize(camWidth, camHeight, matFrame.step, BACKBUFFER_BITSPERPIXEL, matFrame.data);

// now we can do everything we want with matFrame, for example just:
cv::imshow("My Window Name", matFrame);

// as always, don't forget this:
pFrame->Release();

If you want the frame as 3-channel color RGB matrix (for further processing), use:

Code: Select all

// 3 channels
cv::Mat matFrame(cv::Size(camWidth, camHeight), CV_8UC3);

// 3*8 = 24
const int BACKBUFFER_BITSPERPIXEL = 24;
Please note that for any bit-depth != 8 (i.e. anything else than the first example), the call to Rasterize() will be slower. On my quadcore, it takes about 0.5 ms per call (with an V100R2 cam, MJPEG Mode), and about twice as long when I use more than 1 channel...

Hope this helps, feel free to comment etc.

Linus

Re: Camera images to OpenCV

Posted: Mon Feb 28, 2011 9:57 am
by davidtheory
Hey Linus, can you get the images from more than 1 camera at the same time? I have 6 cameras connected to a OptiHub, how can I choose which camera to use at a specific time?

This is useful, thanks!

Re: Camera images to OpenCV

Posted: Mon Feb 28, 2011 3:32 pm
by Birch
David,

One thing to keep in mind is the raw grayscale images at full frame rate and image size can consume a lot of USB bandwidth. If you are only capturing from a single camera at a time then you may be able to achieve full frame rate, but you might also see some dropped frames. If you are trying to stream from multiple cameras then you might want to consider using the temporal or spatial decimation settings, or switching to MJPEG grayscale mode, as a way to reduce the bandwidth consumption.

Here are two FAQ entries on the topic.
http://www.naturalpoint.com/optitrack/s ... .html#q204
http://www.naturalpoint.com/optitrack/s ... .html#q205

For information about how to enumerate and use multiple cameras you can check the CameraList sample and the "Multiple Cameras" documentation.

Re: Camera images to OpenCV

Posted: Wed Mar 02, 2011 8:50 am
by LinusA
Hi David,

as Birch suggested, I did look at the examples from the SDK. I also browsed the header files of the camera SDK. It happens that I also use 6 cameras. It all works flawlessly.

I stripped everything down to the bare minimum for my needs, that means, I initialize the camera manager:

Code: Select all

CameraLibrary::CameraManager::X().WaitForInitialization();
// + errorhandling & sanity checking
I then specify the serial numbers for my cam (via user input or configuration files) and use:

Code: Select all

CameraLibrary::Camera *pMyCam = CameraLibrary::CameraManager::X().GetCameraBySerial(serialNo);
Just make sure to call "->Release()" of the cam pointers once you're done, and then shut down the CameraManager. Just as in the examples, nothing fancy.

By keeping N OpenCV matrices available (type cv::Mat, see above), you can use the exact same code for each of the N cams.

I can use MJPEG or Precision mode with 100 FPS and 6 V100R2 cams in real time on my machine...

Re: Camera images to OpenCV

Posted: Wed Mar 09, 2011 8:38 am
by davidtheory
Hi guys, thanks for the help.

My settings for one cam are:

mycam->SetVideoType(CameraLibrary::MJPEGMode);
mycam->SetAEC(true);
mycam->SetAGC(true);

is this the best configuration for speed increasing in openCV? for real time? I saw the image processing example and the way they render the image with openGL seems (my perception) to be faster than showing the image in an openCV window.

Re: Camera images to OpenCV

Posted: Thu Mar 10, 2011 8:39 am
by LinusA
[quote=davidtheory]Hi guys, thanks for the help.

My settings for one cam are:

mycam->SetVideoType(CameraLibrary::MJPEGMode);
mycam->SetAEC(true);
mycam->SetAGC(true);

is this the best configuration for speed increasing in openCV? for real time?
[/quote]
I personally found AEC and AGC of no help, so I always keep them off, but YMMV. The camera settings don't have an effect on speed with OpenCV. The "CPU expensive" operation (but still affordable) is the call to rasterize(). If you only need marker positions, you might want to go without pictures (using Precision mode).

[quote=davidtheory]
I saw the image processing example and the way they render the image with openGL seems (my perception) to be faster than showing the image in an openCV window.
[/quote]
Yes of course using OpenGL is another way, and GPU acceleration is always good. As usual, it depends on what you want to do.

I just showed a way to get the camera picture into OpenCV. I personally use OpenCV windows to display my OpenCV images (such as camera frames), but it's fast enough (2 Windows, 50 FPS is possible). I use the Qt backend of OpenCV with OpenGL support enabled anyway (compiled using WITH_QT and WITH_QT_OPENGL). You can see screenshots here:http://forum.naturalpoint.com/forum/ubb ... #Post46150 )

If speed was important, I'd probably switch to "plain OpenGL" or Qt's QGLWidget. As I said, it depends :-)

Re: Camera images to OpenCV

Posted: Mon Sep 12, 2011 3:59 am
by zzzamarron
Hi this is the code i am trying Thanks to LinusA but i only get gray image in the imshow window. I use a Trackir 4 camera, any idea? Thank you

int camWidth =camera->Width();
int camHeight = camera->Height();

Mat matFrame(Size(camWidth, camHeight), CV_8UC1);
const int BACKBUFFER_BITSPERPIXEL = 8;
namedWindow("Track",CV_WINDOW_AUTOSIZE);

// later on, when we get the frame as usual:
while(1)
{
Frame *frame = camera->GetFrame();

if(frame) {

frame->Rasterize(camWidth, camHeight, matFrame.step, BACKBUFFER_BITSPERPIXEL, matFrame.data);

imshow("Track", matFrame);


frame->Release();
}

}

Re: Camera images to OpenCV

Posted: Tue Sep 13, 2011 11:34 am
by LinusA
[quote=malboro]Hi this is the code i am trying Thanks to LinusA but i only get gray image in the imshow window. I use a Trackir 4 camera, any idea? Thank you
[/quote]
You're using a single-channel cv::Mat, which is a gray image. I wrote above how to retrieve the data so you can do "color processing" in OpenCV with that cv::Mat object:

Code: Select all

// 3 channels
cv::Mat matFrame(cv::Size(camWidth, camHeight), CV_8UC3);

// 3*8 = 24
const int BACKBUFFER_BITSPERPIXEL = 24;
But please note that this doesn't change the factsthat some cameras only return grayscale values. I don't know the cam you're using, but I'd guess it doesn't return color images!

Re: Camera images to OpenCV

Posted: Thu Sep 15, 2011 12:12 pm
by wuzle
Can you give some pointers as to your project settings for this? I am having a ton weird runtime errors which I suspect has to do with be libs I am using.

So:
-What version of Visual Studio are you using?
-Are you still using OpenCV 2.2, and Optitrack SDK 1.0?
-Are you using the precompiled OpenCV libraries(lib or staticlib), or did you compile yourself?

Any help is appreciated.

Re: Camera images to OpenCV

Posted: Mon Sep 19, 2011 11:23 am
by LinusA
I've been compiling my project on three machines: Visual Studio 2008 on WinXP SP3 32bit, VS 2008 in Win7 32bit, and VS 2010 Win7 64bit.

In all cases I'm using Camera SDK 1.0 (but this doesn't matter, all 1.x versions work), OpenCV 2.2 with a custom patch, and Qt 4.7.3. I always compiled OpenCV myself, yes -- I get static .lib files from that. I also use OpenGL and GLEW headers and libs.

The custom OpenCV patch I use is https://code.ros.org/trac/opencv/ticket/919 , but I don't think you need that.


You should make test-projects to test all things independently, one lib at a time. Can you create an OpenCV project and display a cv::Mat? Can you create a project using Camera SDK and retrieve frame objects?

I did so myself (also including Qt tests and so on), then put everything together.