How can I get the image from camera faster?

[Mitsuru Takeda]

Hi!
I'm working on object recognition and stereo vision with cameraSDK.
Now, I have a problem that frame->Rasterlize() takes too much time.
In my PC (cpu: core-i9-9900, gpu: RTX-2080), it takes 20ms every frame.
I want to realize 240fps. So, it should be in 4ms.
Ideally, it should be in 2ms because I use 3 cameras.

I get images in this way.

cv::Mat framebuffer(cv::Size(cameraWidth, cameraHeight), CV_8UC4);
FrameGroup* frameGroup = sync->GetFrameGroup();
Frame* frame = frameGroup->GetFrame(i);
frame->Rasterize(cameraWidth, cameraHeight, 0, 32, framebuffer.data);

Is there any way to get image faster?
I know that Rasterize takes 15ms, construction and destruction of frame class takes 5ms and other functions take no more than 1ms.

Here is my code.
//=================================================================================-----
//== NaturalPoint 2017
//== Camera Library SDK Sample
//==
//== This sample connects to a single OptiTrack color camera, decodes, and displays
//== the camera's H.264 video.
//==
//== you will need to download the latest libav prebuilt binaries from the libav
//== web site for this sample application to compile and run properly.
//==
//== 1. download the binaries from: http://builds.libav.org/windows/release-lgpl/
//== note: make sure you download binaries that match your 32bit or 64bit target.
//== if there is a mismatch, you'll get link errors for unknown libav symbols.
//== 2. unzip the download file into this sample application's folder ./ColorVideo/
//== 3. rename the resulting folder to 'libav' removing platform & build details
//== from the folder name.
//== 4. The resulting file folder tree should look as follows:
//==
//== ./ColorVideo/
//== libav/
//== usr/
//== bin/
//== include/
//== lib/
//== share/
//==
//== 5. Compile & run the sample application with at least one OptiTrack color
//== camera connected.
//==
//=================================================================================-----

#include "cameralibrary.h" //== Camera Library header file ======================---
#include "supportcode.h" //== Boiler-plate code for application window init ===---
#include "videodecoder.h" //== libav h.264 decoder implementation ==============---
#include <opencv2/opencv.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/objdetect/objdetect.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <windows.h>
#include <conio.h>
#include <tchar.h>
#include <iostream>
#include <stdio.h>
#include <vector>
#include <algorithm>
#include <functional>
#include <string.h>
#include <time.h>

using namespace CameraLibrary;
using namespace std;
using namespace cv;

#define exposure 2000
#define framerate 240

int main(int argc, char* argv[])
{
CameraLibrary_EnableDevelopment();

CameraLibrary::CameraManager::X();

PopWaitingDialog();

//== Get a connected camera ================----
if (CameraManager::X().WaitForInitialization())
printf("complete\n\n");
else
{
printf("failed (return key to exit)");
getchar();
return 1;
}

//== Connect to all connected cameras ==----

Camera* camera[kMaxCameras];
int cameraCount = 0;

CameraList list;

printf("Cameras:\n");

for (int i = 0; i < list.Count(); i++)
{
printf("Camera %d >> %s\n", i, list.Name());

camera = CameraManager::X().GetCamera(list.UID());

if (camera == 0)
{
printf("unable to connected to camera...\n");
}
else
{
cameraCount++;
}
}

if (cameraCount == 0)
{
printf("no cameras (return key to exit)");
getchar();
return 1;
}

int cameraWidth = camera[0]->Width();
int cameraHeight = camera[0]->Height();

//== Color Video Mode ==--
for (int i = 0; i < cameraCount; i++)
{
camera->SetVideoType(Core::VideoMode); //== Select Color Video ==============---
camera->AttachModule(new cModuleVideoDecompressorLibav()); //== create & attach video decoder ==--
camera->SetLateDecompression(true);

//== Set camera settings ==--

camera->SetFrameRate(framerate);
camera->SetExposure(exposure);

//== Start camera output ==--

camera->Start();

//== Turn on some overlay text so it's clear things are ===---
//== working even if there is nothing in the camera's view. ===---

camera[i]->SetTextOverlay(true);
}

//== camera frames. ===---
cModuleSync* sync = cModuleSync::Create();

for (int i = 0; i < cameraCount; i++)
{
sync->AddCamera(camera[i]);
}
cv::Mat framebuffer(cv::Size(cameraWidth, cameraHeight), CV_8UC4);
cv::Vec4b* buffer_ptr = framebuffer3.ptr<cv::Vec4b>(0);
clock_t start_clock;
clock_t end_clock;
int times = 0;
while(!_kbhit())
{
all_found = false;
//== Fetch a new frame from the camera ===---
FrameGroup* frameGroup = sync->GetFrameGroup();
if (frameGroup)
{
if (frameGroup->Count() == cameraCount) {
all_found = true;
if (times == 0) {
start_clock = clock();
std::cout << "start" << endl;
}
if (times == 100) {
end_clock = clock();
std::cout << "end" << endl;
double lap_time = static_cast<double>(end_clock - start_clock) / CLOCKS_PER_SEC * 1000.0;
std::cout << lap_time << endl;
times = -1;
}
times++;
for (int i = 0; i < cameraCount; i++) {
Frame* frame = frameGroup->GetFrame(i);
if (frame) {
pos_now[i][0] = static_cast<float>(frame->FrameID());
vector<vector<float>> pos_cd;
frame->Rasterize(cameraWidth, cameraHeight, 0, 32, framebuffer.data);

//== do some image processing =========--

frame->Release();
}
else
all_found = false;
}
}
frameGroup->Release();
}
if (all_found) {
//== do stereo vision =========--
}
//== Escape key to exit application ==--

if (keys[VK_ESCAPE])
break;
}
//== Stop video stream ==--
for (int i = 0; i < cameraCount; i++) {
camera[i]->Stop();
Sleep(250);

//== Release camera ==--

camera[i]->Release();
}
//== Shutdown Camera Library ==--

CameraManager::X().Shutdown();

//== Exit the application. Simple! ==--

return 1;
}

Please give me some advice!
Thank you in advance!

[Mitsuru Takeda]

Now, I'm working on multi-threading to get camera image faster.
But, When I define another Frame* before Frame*->Release(), there is an error.
How can I solve this problem?