Arena to Unity streaming

NatNet, VRPN, TrackD, and Plugins
johny3212
Posts: 10
Joined: Wed Oct 02, 2013 3:39 am

Re: Arena to Unity streaming

Post by johny3212 »

morgan Thank you for your infos.

So I have a two separate ways how to implement client for OptiTrack systems:
First way is use NatNetML.dll and create my own plugin (requires Unity Pro) or second is directly depacketize of NatNET packets within Unity C# script (it means create my own wrapper, yes?).

First way is nicest and better but i have some question:
1. if I use NatNetML.dll and create my own plugin, this will work on Windows system. But I need compile/build Unity app for the Android system. Do you think this way will work on the Android system in mobile device? Or I will have to use second way and create my own wrapper? I ask this question because I don't know which one I can start firstly because I don't have a time. I am in Erasmus as PhD student and my time is running out.
2. For first way why I will have to use Unity Pro? I can't create my own plugin on free version of Unity?

Thank you.

VRDave your work is very amazing. In my home university in cooperation with private company we are developing similar systems and its own graphical plarformu (engyne) but with a focus on industrial engineering, logistic or ergonomic systems, augmented reality and etc. We have 3D CAVE and we are waiting for Oculus Rift. I am looking for the student placements next year :D.
Sorry for my English.
johny3212
Posts: 10
Joined: Wed Oct 02, 2013 3:39 am

Re: Arena to Unity streaming

Post by johny3212 »

Hello,

I have some first results. So I thing the NatNetML.dll is not directly compatible with the Unity3D. I din'd find solution how to links this lib directly to a C# scripts in the Unity project.
So my solution is to create my own plugin "Optitrack_UnityPlugin.dll" in C++ language and use NatNetLib.dll. I used this tutorial:
http://docs.unity3d.com/Documentation/M ... ugins.html
So I develop simple C++ plugin for NatNet Client. Everything is running fain and fast without latency.
This first version of my plugin can be free so I can publish it. On the link below you can download my plugin with source code and scripts for Unity3D. Plugin library must be copied in the folder "Assest/Plugins".
In (Asert/Scripts/Optitrack) folder you can find OptitrackManager.cs script. This script initialize my C++ Unity plugin, connect to the NatNet server (Motive) and use first RigidBody for object movement. So you can create Cube and only Add Component script and set IP addresses in script.
Next there is a folder Optitrack_UnityPlugin. You can import this folder to the Eclipse as project (Import/General/Existing Project into Workspace) and after compile with Microsoft Visual C++ Compiler to Optitrack_UnityPlugin.dll. I don's use Microsoft Visual Studio :).
http://sourceforge.net/projects/optitra ... z/download
For compile and run my lib you need NatNetLib.dll and NatNet SDK.

There is a one problem for me. This solution is available only for Unity Windows desktop app. Because OptiTrack NatNet lib is compiled to the "*.dll" lib format and Android is Linux, so I thing is impossible to use this way for the Android. Because in Android I need "*.so" lib format. So I will have to use second way. It is directly depacketize our NatNET UDP packets within Unity script. So I try to use script be Morgan. Or have somebady some other ideas ? Thank you.
morgan
NaturalPoint Employee
NaturalPoint Employee
Posts: 199
Joined: Tue Jun 24, 2008 2:01 pm
Location: Corvallis, OR, USA
Contact:

Re: Arena to Unity streaming

Post by morgan »

Hi there,

The NatNet SDK contains a direct depacketization sample (PacketClient.cpp), that shows how to decode NatNet UDP packets directly without needing NatNet lib. If you updated your plugin to use this approach, you could then recompile for Unix (Linux/android/etc).

Alternatively, you could use the script only path from my earlier post. This script may need to be updated to be compatible with the current Motive 1.5 bitstream syntax. Again, the PacketClient.cpp sample in the latest NatNet SDK shows how to decode the UDP packets directly and correctly.

Hope this helps,

Morgan
johny3212
Posts: 10
Joined: Wed Oct 02, 2013 3:39 am

Re: Arena to Unity streaming

Post by johny3212 »

morgan:

I can't just update my plugin with "PacketClient.cpp" samples. It is impossible for Android because it is Linux and this example use win sockets. I will have to use Linux sockets. But it doesn't matter. Easier is implement all functionality to the .NET sockets script for Unity3D with my own NatNet parser until the time when Optitrack gives new release included Unity3D support for Android. :). Next problem is that NatNetML.dll is only wrapper for NatNet.dll, so It does not use .NET sockets. Morgan your code doesn't work on Android. I thing it is only problem with IP options.
So I developed my own C# script for Unity3D. This script was testing on Android device and is running fain. There is implement only RigidBody parser no skeleton. I can publish some basic code.

DirectMulticastSocketClient.cs

Code: Select all

using UnityEngine;
using System.Collections;

using System;
using System.Net;
using System.Net.Sockets;
using System.Threading;
using System.Text;

namespace OptitrackManagement
{

public class DirectStateObject 
{
    public Socket workSocket = null;
    public const int BufferSize = 65507;
    public byte[] buffer = new byte[BufferSize];
    public StringBuilder sb = new StringBuilder();
}

public static class DirectMulticastSocketClient {
			
	private static Socket client;
	private static bool _isInitRecieveStatus = false;
	private static bool _isIsActiveThread = false;
	private static StreemData _streemData = null;
	private static String _strFrameLog = String.Empty;
		
	//bool returnValue = false;
	private static int _dataPort = 1511;
    private static int _commandPort = 1510;
	private static string _multicastIPAddress = "239.255.42.99";

	private static void StartClient() 
	{
        // Connect to a remote device.
        try
		{
			
			Debug.Log("[UDP] Starting client");
			_streemData = new StreemData();
            client = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
			client.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true);
			//client.ExclusiveAddressUse = false;

			IPEndPoint ipep=new IPEndPoint(IPAddress.Any, _dataPort);
			client.Bind(ipep);
			
			IPAddress ip=IPAddress.Parse(_multicastIPAddress);
			client.SetSocketOption(SocketOptionLevel.IP, SocketOptionName.AddMembership, new MulticastOption(ip,IPAddress.Any));
			
			_isInitRecieveStatus = Receive(client);
			_isIsActiveThread = _isInitRecieveStatus;
            
        } catch (Exception e) 
		{
            Debug.LogError("[UDP] DirectMulticastSocketClient: " + e.ToString());
        }
    }
	
	private static bool Receive(Socket client) 
	{
        try 
		{
            // Create the state object.
            DirectStateObject state = new DirectStateObject();
            state.workSocket = client;
			
			Debug.Log("[UDP multicast] Receive");

            // Begin receiving the data from the remote device.
            client.BeginReceive( state.buffer, 0, DirectStateObject.BufferSize, 0,
                new AsyncCallback(ReceiveCallback), state);
			
        } catch (Exception e) 
		{
            Debug.Log(e.ToString());
			return false;
        }
			
	return true;
    }
	
	private static void ReceiveCallback( IAsyncResult ar ) 
	{
        try {
			//Debug.Log("[UDP multicast] Start ReceiveCallback");
            // Retrieve the state object and the client socket 
            // from the asynchronous state object.
            DirectStateObject state = (DirectStateObject) ar.AsyncState;
            Socket client = state.workSocket;

            // Read data from the remote device.
            int bytesRead = client.EndReceive(ar);

            if (bytesRead > 0 && _isIsActiveThread) 
			{				
				ReadPacket(state.buffer);
				
				//client.Shutdown(SocketShutdown.Both);
				//client.Close();	
				
                client.BeginReceive(state.buffer,0,DirectStateObject.BufferSize,0,
                    new AsyncCallback(ReceiveCallback), state);
            } else 
			{
				Debug.LogWarning("[UDP] - End ReceiveCallback");
					
					if(_isIsActiveThread == false)
					{
					Debug.LogWarning("[UDP] - Closing port");
					_isInitRecieveStatus = false;
					//client.Shutdown(SocketShutdown.Both);
					client.Close();	
					}
					
                // Signal that all bytes have been received.
            }
        } catch (Exception e) 
		{
            Debug.LogError(e.ToString());
        }
		
    }
	
	private static void ReadPacket(Byte[] b)
    {
        int offset = 0;
        int nBytes = 0;
        int[] iData = new int[100];
        float[] fData = new float[500];
        char[] cData = new char[500];

        Buffer.BlockCopy(b, offset, iData, 0, 2); offset += 2;
        int messageID = iData[0];

        Buffer.BlockCopy(b, offset, iData, 0, 2); offset += 2;
        nBytes = iData[0];

        //Debug.Log("[UDPClient] Processing Received Packet (Message ID : " + messageID + ")");
        if (messageID == 5)      // Data descriptions
        {
			Debug.Log("DirectParseClient: Data descriptions");
				
		}else if (messageID == 7)   // Frame of Mocap Data
        {
			_strFrameLog = String.Format("DirectParseClient: [UDPClient] Read FrameOfMocapData: {0}\n", nBytes);
            Buffer.BlockCopy(b, offset, iData, 0, 4); offset += 4;
            _strFrameLog += String.Format("Frame # : {0}\n", iData[0]);
				
            //number of data sets (markersets, rigidbodies, etc)
            Buffer.BlockCopy(b, offset, iData, 0, 4); offset += 4;
            int nMarkerSets = iData[0];
            _strFrameLog += String.Format("MarkerSets # : {0}\n", iData[0]);
				
			for (int i = 0; i < nMarkerSets; i++)
            {
                String strName = "";
                int nChars = 0;
                while (b[offset + nChars] != '\0')
                {
                    nChars++;
                }
                strName = System.Text.Encoding.ASCII.GetString(b, offset, nChars);
                offset += nChars + 1;

                Buffer.BlockCopy(b, offset, iData, 0, 4); offset += 4;
                _strFrameLog += String.Format("{0}:" + strName + ": marker count : {1}\n", i, iData[0]);

                nBytes = iData[0] * 3 * 4;
                Buffer.BlockCopy(b, offset, fData, 0, nBytes); offset += nBytes;
				//do not need	
            }
			
			 // Other Markers - All 3D points that were triangulated but not labeled for the given frame.
            Buffer.BlockCopy(b, offset, iData, 0, 4); offset += 4;
            int nOtherMarkers = iData[0];
            _strFrameLog += String.Format("Other Markers : {0}\n", iData[0]);
            nBytes = iData[0] * 3 * 4;
            Buffer.BlockCopy(b, offset, fData, 0, nBytes); offset += nBytes;	
				
			// Rigid Bodies
            //RigidBody rb = new RigidBody();
            Buffer.BlockCopy(b, offset, iData, 0, 4); offset += 4;
            _streemData._nRigidBodies = iData[0];
            _strFrameLog += String.Format("Rigid Bodies : {0}\n", iData[0]);
            for (int i = 0; i < _streemData._nRigidBodies; i++)
            {
                ReadRigidBody(b, ref offset, _streemData._rigidBody[i]);
				//_streemData._rigidBody[0];
            }	
			
			//Debug.Log(_strFrameLog);	
				
		}else if (messageID == 100)
        {

		}
		
	}
		
	 // Unpack RigidBody data
	 private static void ReadRigidBody(Byte[] b, ref int offset, RigidBody rb)
	    {
			try
			{
			
		        int[] iData = new int[100];
		        float[] fData = new float[100];
		
		        // RB ID
		        Buffer.BlockCopy(b, offset, iData, 0, 4); offset += 4;
		        int iSkelID = iData[0] >> 16;           // hi 16 bits = ID of bone's parent skeleton
		        int iBoneID = iData[0] & 0xffff;       // lo 16 bits = ID of bone
		        rb.ID = iData[0]; // already have it from data descriptions
		
		        // RB pos
		        float[] pos = new float[3];
		        Buffer.BlockCopy(b, offset, pos, 0, 4 * 3); offset += 4 * 3;
		        rb.pos.x = pos[0]; rb.pos.y = pos[1]; rb.pos.z = pos[2];
		
		        // RB ori
		        float[] ori = new float[4];
		        Buffer.BlockCopy(b, offset, ori, 0, 4 * 4); offset += 4 * 4;
		        rb.ori.x = ori[0]; rb.ori.y = ori[1]; rb.ori.z = ori[2]; rb.ori.w = ori[3];
		        Buffer.BlockCopy(b, offset, iData, 0, 4); offset += 4;
		        int nMarkers = iData[0];
		        Buffer.BlockCopy(b, offset, fData, 0, 4 * 3 * nMarkers); offset += 4 * 3 * nMarkers;
		
		        Buffer.BlockCopy(b, offset, iData, 0, 4 * nMarkers); offset += 4 * nMarkers;
		
		        Buffer.BlockCopy(b, offset, fData, 0, 4 * nMarkers); offset += 4 * nMarkers;
		
		        Buffer.BlockCopy(b, offset, fData, 0, 4); offset += 4;			
			} catch (Exception e)
			{
            	Debug.LogError(e.ToString());
        	}
	    }
	
	// Use this for initialization
	public static void Start () 
	{
		StartClient();
	}
	// Update is called once per frame
	public static void Update () 
	{
	}	
	public static void Close () 
	{
	_isIsActiveThread = false;
	}	
	public static bool IsInit()
	{
		return _isInitRecieveStatus;
	}
	public static StreemData GetStreemData()
	{
		return _streemData;
	}	
}
}
SkeletonClass.cs

Code: Select all

using UnityEngine;
using System;
using System.Collections;
using System.Collections.Generic;
using System.Net;

namespace OptitrackManagement
{

// marker
public class Marker
{
public int ID = -1;
public Vector3 pos;		
}
	
// bone
public class RigidBody
{
    public string name = "";
    public int ID = -1;
    public int parentID = -1;
    public float length = 1.0f;
    public Vector3 pos;
    public Quaternion ori;
    public Quaternion initOri;
    public Transform trans;
}
	
public class SkeletonData 
	{
		

	}

}
StreemData.cs - include all parse data

Code: Select all

using UnityEngine;
using System;
using System.Collections;
using System.Collections.Generic;

namespace OptitrackManagement
{

public class StreemData {
		
		public RigidBody[] _rigidBody = new RigidBody[200];
		public int _nRigidBodies = 0;
		//public List<RigidBody> _listRigidBody = new List<RigidBody>();
			
		public StreemData ()
		{
						
			//Debug.Log("StreemData: Construct");
			InitializeRigidBody();
		}
		
		public bool InitializeRigidBody()
		{
			_nRigidBodies = 0;
			for (int i = 0; i < 200; i++)
        	{
            _rigidBody[i] = new RigidBody();
        	}	
			return true;
		}
		
		public bool InitializeSkeleton()
		{	
			return true;
		}
		
		public bool InitializeMarkerSet()
		{		
			return true;
		}	
}
}
OptotrackManager.cs - Unity3D load script

Code: Select all

using UnityEngine;
using System;
using System.Collections;
//using NatNetML;
using OptitrackManagement;

public class OptitrackManager : MonoBehaviour 
{
	public string myName;
	//public string _clientIPadres;//=192.168.1.2
	//public string _serverIPadres;//=192.168.1.3
	private Vector3 _moveVector;
	public bool _deinitValue = false;
	
	~OptitrackManager ()
	{		
		Debug.Log("OptitrackManager: Destruct");
		OptitrackManagement.DirectMulticastSocketClient.Close();
	}
	
	void Start () 
	{
		Debug.Log(myName + ": i am alive");
			
		OptitrackManagement.DirectMulticastSocketClient.Start();
		_moveVector = transform.position;

	}
	
	// Update is called once per frame
	void Update () 
	{
		OptitrackManagement.DirectMulticastSocketClient.Update();
		///_netClient.Step();
					
		if(OptitrackManagement.DirectMulticastSocketClient.IsInit())
			{
				StreemData networkData = OptitrackManagement.DirectMulticastSocketClient.GetStreemData();
			
				_moveVector = networkData._rigidBody[0].pos * 2.0f;	
			}		
			
		transform.position = _moveVector;
		
		if(_deinitValue)
			{
			_deinitValue = false;
			OptitrackManagement.DirectMulticastSocketClient.Close();
			}
	}
	
}
There is a android application. You need only run the Motive+server with RigidBody and stream this data via the network with Multicast group IP 239.255.42.99. App during the start will automatically connect this server via the WIFI connection and use first RgidBody pos for set pos of the cube in the Unity environment.
http://sourceforge.net/projects/optitra ... z/download
Aranda
Posts: 3
Joined: Sun Jan 12, 2014 10:02 pm

Re: Arena to Unity streaming

Post by Aranda »

Thanks all for the Unity scripts. I've managed to integrate it nicely with Motive Tracker. To help others who need to do this, a couple of gotchas I cam across:
  • Need to set Motive's streaming options to stream the data and since I was running Unity on the same PC as Motive I had to use the Local Loopback option.
  • Getting the orientations correct was a big headache. Turns out I needed to reorder the quaternion components and then negate the euler orientation:

Code: Select all

         // Rigid body rotation streamed from Motive Tracker
         Quaternion rot = lastRigidBody.ori;

         // Re-order quaternion components
         rot = new Quaternion(rot.z, rot.y, rot.x, rot.w);
         
         // Invert pitch and yaw
         Vector3 euler = rot.eulerAngles;
         rot.eulerAngles = new Vector3(-euler.x, -euler.y, euler.z);

         // Apply rotation
         transform.rotation = rot;
VRDave
Posts: 11
Joined: Mon Mar 04, 2013 11:29 am
Location: Seattle, WA
Contact:

Re: Arena to Unity streaming

Post by VRDave »

Aranda,

This is one way to do it, but the underlying problem is that the native handedness of Motive and Unity are not the same. Motive, by default, outputs right-handed, whereas Unity uses a left-handed system. Lucky for us, Motive has an option to change the output stream to a left-handed coordinate system. I believe it's under the Application Options, but I can't remember for sure, as it's muscle memory by now.

This makes it play very nicely with the mocap plugin I've been writing over the last year and does not require additional conversion on the Unity end. My hope is that the NaturalPoint conversion to left-handedness is computationally faster than the equivalent conversion if done on the Unity end (similar to the one in your post). I guess someone could bench-mark it some day...
VRcade - Full Motion Virtual Reality
VRcade.com | facebook.com/VRcade
kumarsmurthy
Posts: 1
Joined: Sun Jan 26, 2014 10:17 am

Re: Arena to Unity streaming

Post by kumarsmurthy »

Hey guys,
Thanks for all the information. It's pretty sad to know that the dll provided by NatNet SDK cant be used by a C# script in Unity directly. Even after going through Johny3212's code I am just too new to all this to implement it. Maybe if I spend more time on it, I can. :roll:
Is it possible to modify the sample C apps to write the information into a text file and read that text file directly from unity? I have a 15 day-deadline and I am just lazy to go through dll hell/plugin development!
Has anybody tried this?
(I am pretty sure this isn't a good approach :? )
johny3212
Posts: 10
Joined: Wed Oct 02, 2013 3:39 am

Re: Arena to Unity streaming

Post by johny3212 »

Hello guys,
I finished some work and created my first solution for Augmeneted Reality application based on OptiTrack, Vuforia, Unity and Android mobile device so I wanna show to you my first results. I write some paper to Journal about it now. There is some video presentation of the work:

http://www.youtube.com/watch?v=SvVtebRKDsE
v3_XD
Posts: 4
Joined: Wed Oct 02, 2013 12:46 pm

Re: Arena to Unity streaming

Post by v3_XD »

How do you get the rigidbody's name from the udp code?
VRDave
Posts: 11
Joined: Mon Mar 04, 2013 11:29 am
Location: Seattle, WA
Contact:

Re: Arena to Unity streaming

Post by VRDave »

Hey v3_XD,

If you are using the code that johny3212 supplied, you cannot get the rigidbody name from the frame data. It has been intentionally left out for performance reasons. Instead, you have to add a remote call to Motive to get the data descriptions, which has the RigidBody ID -> name mapping you are looking for. He has a place-holder for receiving the result here:

Code: Select all

if (messageID == 5) {
...
}
You will want to fill this out and parse the bits you require. You can use the PacketClient.cpp example to get the byte syntax for reading data descriptions.

You will also have to request that information from Motive by setting up an additional socket on the command port (he has it listed as _commandPort, but it is currently unused). Then you can send the request for the data descriptions and handle it in the existing async callback.
VRcade - Full Motion Virtual Reality
VRcade.com | facebook.com/VRcade
Post Reply