Setting and understanding framerates V120

[beckdo]

Also,

#8: Please take the FrameRateCalculation Camera SDK sample for a spin and see how it does with your V120.

Thx,
Doug

[LamarLatrell]

Hi Doug,

Thanks for getting back !

Some quick replies:

#1 >> Sleep(1) caused the frame rate to halve, i.e. worse. Sleep(0) no difference.
#2 >> 2012 MacBookPro retina with 16Gb RAM - would have thought so ?
#3 >> No change for either
#4 >> Same FPS for grayscale mode (my preferred mode anyway)
#5 >> Not sure what this is - and what you mean by 'why your UI is not updating' - I see a line in the FrameRateCalculation sample referring to it, adding the line to my code gives an unresolved external symbol error (haven't tried to solve that as there are other things to concentrate on)
#6 >> Might look into this next, see below though - but first, I'm interested; do you suggest that it's simply better to use TimeStamp() for more accuracy in general … or that my current timing estimate is actually leading me to believe I'm not at 120 fps and I actually am - and therefore there is no problem? (wouldn't that be great )
#8 >> SDK samples are working now (!) no idea why they didn't a week ago - can confirm I see 119fps stabilizing to 120 fps - and changing frame rates also works.

If I throw in my old GetSystemTime style function it'll tell me that the FPS is 62fps (meanwhile the FrameRateCalculation still says 120fps) - this makes me think maybe there has been no issue all along but the fact that my timing method was at fault.

However, if I place a big ol' Sleep(1000); in the while(1) loop of the FrameRateCalculation it tells me the FPS is 9fps …

Another test of adding Sleep(100) right after while(1) should give 10fps maximum (yes?) - FrameRateCalculation gives 72fps - however using GetSystemTime gives 8fps

This concerns me

Maybe Sleep() in the loop confused FrameRateCalculation ?

btw. the popWaitingDialog issue seems to spring from selecting a 64bit config.

Feel like I'm going in circles again - maybe I'm over thinking this - I'd just like a basic boilerplate VS console app project that got V120 frames into an openCV Mat as nicely as possible and gave feedback of the FPS of the loop as I added further processing within it. Maybe I already have that, maybe not

[LamarLatrell]

I've tried CameraManager::X().TimeStamp() and as you suggest it gives more resolution of time than my other method, but results are the same (just more decimal places).

Code: Select all

double camTime = CameraManager::X().Timestamp();
double prevCamTime = 0;
double camFPS = 0;

while(1){
    prevCamTime = camTime;
    camTime = CameraManager::X().Timestamp();
    camFPS = 1/(camTime - prevCamTime);

    //...the rest of the loop as in the FrameRateCalculation sample

I will see 120FPS from the original app, but 63~64FPS from the code above.

Adding Sleep(10) at the top of the loop gives 120FPS from the original app, but 31~32FPS from the code above.
Adding Sleep(50) at the top of the loop gives 115FPS from the original app, but 12.7~12.8FPS from the code above.

(Sleep() models other stuff I'll be doing in the loop)

The image stream is clearly following the lower estimate.

I wonder, what am I doing incorrectly?

The original fps function seems up to a point unconcerned with the addition of Sleep() to the loop, so it's not helpful to me as I'm after an estimate of the loop rate (I'll be adding processing and need to keep it within a spec).

Using CameraManager::X().Timestamp(); seems to follow the logical slowing down of FPS when I add a Sleep() to the loop, which is great, however, it tells me the FPS is almost half of what it should be when I'm running 'free' with no Sleep() (and the original fps calc is saying it's at 120fps).

It seems relatively correct, but absolutely not (by close to a factor of 2)

Either way, it's incorrect

Keen to understand this.

[beckdo]

Hey Lamar,

The FrameRateCalculation sample application is calculating the frame rate correctly. Let me clear up a few points of confusion for you.

First off, I assume that your impression was that if you put a sleep for 1 second in the main loop that the calculation should yield 1 FPS. That would be true if the loop only fetched one frame each time through the loop. If you look, every time through the loop, it will fetch all available frames from the camera. The cameras themselves and relatively robust for users of the Camera SDK in that the cameras will queue up multiple frames if the application is not fetching them fast enough or perhaps the application stalls momentarily. The camera is capable of queuing up roughly 10 frames before it will start dropping frames.

By putting a Sleep(1000) in the main loop, you are effectively measuring how many frames will the camera queue up for the application if it doesn't service it in a timely fashion.

Please use CameraManager::X().TimeStamp() for your timing measurements as GetSystemTime() is not accurate enough for the type of time measurement you are performing. I recently confirmed this in the process of debugging inaccurate timing measurements for another customer.

[LamarLatrell]

Hi,

Thanks for getting back, I'm not sure you read both of my posts (?) as I am using CameraManager::X().Timestamp():

Using CameraManager::X().Timestamp() seems to follow the logical slowing down of FPS when I add a Sleep() to the loop, which is great, however, it tells me the FPS is almost half of what it should be when I'm running 'free' with no Sleep() (and the original fps calc is saying it's at 120fps).

It seems relatively correct, but absolutely not (by close to a factor of 2)

Thanks for the info re. the FPS measurement - I understand more about what is going on there now, but I am still unsure how to go about finding a close representation of the rate that a while(1) loop will run at. That's what I'm actually interested in.

Maybe we can refer to it as 'code loop rate' instead of FPS to avoid confusion.

With your base camera SDK sample which I'd assume is about as boilerplate as you can get, using CameraManager::X().Timestamp() I see a code loop rate of 64, which isn't 120fps.

i.e. it would seem that the cameras ability to capture at 120fps is mostly redundant as the actual rate at which the data is made available to an application isn't comparable, by almost a factor of half.

The question is:

Have I timed it incorrectly? (see code from earlier post)
if no -> how to time it correctly?
if yes -> how am I to achieve 120 code loop rate?

It's a MBP retina with 16Gb RAM, would have thought it's fast enough?

Thanks for all your time on this one.

PS. I'm about to build a quick LED strobe to see if I can figure out its FPS that way... maybe it wont work as I expect with the frame queue system but it'll be interesting regardless - I wonder, what is the cameras effective shutter angle?

[beckdo]

Hi Lamar,

I have a MBP Retina here as well and it tests out running in MJPEG mode at 120 FPS. (attached screenshot)

Let me request you do a test for me on your MBP Retina:

1. Uninstall the Camera SDK and delete any left-over files from building out of the samples directory.
2. Do a fresh install of the latest Camera SDK.
3. Open the FrameRateCalculation sample app.
4. Update the Set Video Mode line to:

Code: Select all

camera->SetVideoType(Core::MJPEGMode);

5. Run the application.

What Frame Rate does the application give you?

Also, you are concerned about the time it takes to go through the main loop. There are a couple things to consider here. First off, the application is displaying via OpenGL. So the time through the main loop is affected by your video settings. The main one to watch out for is the vsync enabled. This will tend to block when updating the frame via OpenGL until the next vsync triggers.

It would be good to measure the amount of time it takes for your machine to decompress a frame. Measure the time it takes to call frame->Rasterize(framebuffer); (0.7 milliseconds for MJPEG decode on my MBP) and that will tell you how fast your machine can decode the MJPEG frame. Also measure DrawGLScene(&Texture). (0.4 milliseconds on my MBP) It will spike if your video card is locking to vsync.

Thx,
Doug

[beckdo]

Also, I'm probably not the best person to answer your query about shutter angle. I would suggest you email support with regards to your specific camera order and lens selection to get that information.

[LamarLatrell]

Hi Doug,

With SDK reinstalled I see 120fps in MJPEG Mode.

I set timers up as per the following code:

Code: Select all

while(1)	{
		prevCamTime = camTime;
		camTime = CameraManager::X().TimeStamp();
		camPeriod = (camTime - prevCamTime);
		//== Fetch a new frames from the camera and count them for fps calculation ===---

		Frame *newFrame = camera->GetFrame();

		while(newFrame)
		{
			frameCount++;

			if(frame)
			{
				frame->Release();
			}

			frame = newFrame;

			newFrame = camera->GetFrame();
		}

		double currentTime = timer.Elapsed();

		//== Every second update the FPS reported ==--

		if(currentTime-startTime>1.0)
		{
			double latestFPS = frameCount/(currentTime-startTime);

			startTime  = currentTime;
			frameCount = 0;

			if(fps==0)
				fps = latestFPS;
			else
				fps = (0.5*fps)+(0.5*latestFPS);
		}

		if(frame)
		{
			//== Lets have the Camera Library raster the camera's
			//== image into our texture.

			rasterBefore = CameraManager::X().TimeStamp();
			frame->Rasterize(framebuffer);
			rasterAfter = CameraManager::X().TimeStamp();

			//== Overlay the camera frames per second onto the image ==--

			char st[80];
			sprintf_s(st,80,"Camera FPS: %.1f", fps);

			framebuffer->PrintLarge((cameraWidth-((int)strlen(st)*12))/2,cameraHeight/2,st);

			//== Display Camera Image ============--

			drawGLbefore = CameraManager::X().TimeStamp();
			if(!DrawGLScene(&Texture))  
				break;
			drawGLafter = CameraManager::X().TimeStamp();

			printf("Loop period: %f  Raster period: %f  DrawGL period: %f\n", camPeriod*1000, (rasterAfter - rasterBefore)*1000, (drawGLafter - drawGLbefore)*1000); 

			//== Escape key to exit application ==--

			if (keys[VK_ESCAPE])
				break;
		}

		Sleep(2);

		//== Service Windows Message System ==--

		if(!PumpMessages())
			break;
	}

I get:

normal.jpg (145.25 KiB) Viewed 8288 times

If I comment out "if(frameThrottler) return true;" in supportcode.cpp I get:

commentedOUT.jpg (134.49 KiB) Viewed 8288 times

You'll notice interesting things happening in the drawGL timings. hopefully it's helpful to you? But please keep in mind I need this to be in openCV...

Also my jpeg decompress seems a lot slower than yours (or my timing method is incorrect).

When searching for how to disable vsync in windows 8.1 I'm getting results that say it can't be done? It's new to me, I'm not a gamer . How is it altered ?

If it means anything, the images are only important to the code (openCV and/or DLIB) and it is for those purposes I am chasing 120fps - i.e. I'd like a code loop rate that made best use of the FPS of the camera. Having an image to see is much further down the list of priorities, it really only there as visual feedback that things are working and so tearing isn't such a bad thing visually.

Also. as an aside the code gets more complex I see that the processing may become the bottleneck, in that case, I'd rather have low latency, so I'd simply drop frames than let them queue up. (but I'd certainly would rather avoid this in the first instance)

Thanks again

[NaturalPoint-Dustin]

Hi Lamar,

We would like to get you on a remote desktop session or on the phone. Could I have you submit a ticket with us at help.naturalpoint.com with your contact details?

[beckdo]

Excellent, Lamar.

It's important to understand how rendering and video card settings can impact the timing of execution. OpenGL and DirectX will block execution when swapping the back buffer to what is visible on the screen if vertical sync is enabled. For example, you can't achieve a visual update rate in your application of 120 FPS if you're running on a 60Hz monitor with vertical sync enabled.

The easiest way to disable vertical sync is through your video card settings globally on your computer.

It's also possible to change the setting in code within your application. In the case of OpenGL, changing vsync is not in the original OpenGL spec so OpenGL extensions would need to be employed which is a topic outside the scope of the sample application.

The topic of displaying information effectively is a separate topic from receiving data from the Camera SDK. It sounds like because you do not have significant exposure to the nuances of rendering visuals you might want to separate your work from visualizing the output in real-time because that is a separate topic all together.

Another thing I see from your timing measurements is that my MBP Retina is about twice as fast as your MBP Retina. I believe my MBP Retina is a newer model.

Finally, in regards to your query about latency and frame queuing. There is no penalty for fetching a frame from a camera and immediately releasing it. This is intentional. If you look at the sample application, you'll see that every time through the loop, it doesn't just get one frame. In fact, it grabs every frame that is available and releases them all except the most recently received frame. This is effectively an implementation of "get latest frame", just what you are looking for. So long as processing is happening in a timely fashion, there is no frame drop. It's only when processing takes a significant amount of time do frames start dropping.

It's important to be realistic about how much processing you will be able to do at 120FPS. We know you only have a little over 8 milliseconds total. On your MBP the JPEG decode takes roughly 2ms (approximately 25% of your total budget). Depending on what OpenCV algorithms you employ and the speed of those methods will determine the rate at which you'll be able to process frames.