The Ventuz Video Input Output API, short VIO, allows Ventuz users to program applications that send video streams towards Ventuz. In addition to the color frames, alpha (key) and depth buffer (z buffer) are also supported as well as ancillary data such as Camera or Projection Matrices. On the Output side you can provide the client application with ancillary data from Ventuz.
This allows for the integration of hardware and software that Ventuz usually does not support out of the box. The availability of the depth buffer makes it possible to combine the Ventuz engine with more specialized engines, like real-time ray-tracers, map engines and windows user interface applications. Latency is much better than with SDI or DVI capture since the frames can be generated on the same GPU.
VIO is an additional License Option with certain special terms and conditions. Please contact us via sales@ventuz.com.
The VIO can transfer frames directly with OpenGL, Direct3D9 or Direct3D11 contexts, or via CPU-mapped memory.
Ancillary data can be attached to the stream on both sides. This allows e.g. to pass the view and projection matrices are used to render the frame so that the receiver can continue rendering. Only if the matrices are 100% correct the depth buffer works as expected.
Although you can send custom ancillary data to the client application over the VIO Interface, sending textures is not possible! You can only receive textures in Ventuz.
The API is exposed as C *.dll, *.lib and *.h file. It should be no problem to use that in most programming languages. The dll is compiled without dependencies to the C/C++ runtime, so it will link with any compiler. We will assume that Visual Studio 2015 is used.
The header file contains detailed documentation of the functions, their parameters and all structure members and enumerations. This text will explain the general operation of the video engine.
Note that we avoid the terms input and output as they are confusing: What is an input for Ventuz is an output for the client.
Client: The application that communicates with Ventuz using the VIO API.
To Ventuz: Transfer from the client to Ventuz.
From Ventuz: Transfer from Ventuz to the client.
Frame: A color buffer with alpha, and possibly a fitting depth buffer.
Ancillary data or Anc: Data that is attached to the frames. We avoid the term meta-data as it is used by the video replay for metadata of the stream, not single frames.
Stream: The connection between Ventuz and the client.
To use a VIO input (to Ventuz) a stream has to be created in the Ventuz configuration editor. This is done by dragging the VIO device into the input or output area. See Audio / Video configuration.
The most important option here is synchronous/asynchronous.
For inputs, it is best to configure the resolution to auto detect and let the client define the resolution. The client also chooses many other aspects, like if OpenGL or DirectX is used and what kind of depth buffer (if any) is used.
The View and Projection Matrices are automatically set to the default camera. Make sure that the Layer3D is set to the default camera. Instead you can also use a Tracked Camera.
To composite the VIO Input into a Layer3D, use the Vio Input Node. See the Node Page for more information on its properties.
To get correct alpha-sorting, think about what to render before the compositing node and what to render afterward.
You can use an Alpha node to set an alpha value for compositing. Other render and material options are ignored.
Following is a basic guide to what to consider when implementing the VIO API in a client application.
To use the VIO API, include the header and link the library. Make sure the DLL is in the executable path of the application
#include "VioApi.h" #pragma comment(lib,"VioApi")
To Initialize the VIO dll, call vInit(). If the client uses DirectX, pass the IDirect3DDevice9Ex interface. The VIO API assumes that the OpenGL context is current when any function is called. When transfer is done via CPU memory, no DirectX or OpenGL context is required as argument for vInit()
When no DirectX device was specified, VioApi will create it's own device. If a windows handle was passed to vInit(), this will be used to create the device. Otherwise an invisible dummy window is created.
To open a stream, call vOpen(). Fill out all fields of the vOpenPara structure. Some of these fields are only required for future extensions, like the color space.
vOpenPara para;
memset(¶,0,sizeof(para));
para.Channel = 0;
para.Mode = VM_ToVentuz;
para.Transfer = VTM_Direct3D9;
para.Color = VCP_BGRA_8;
para.Depth = CONFIG_DEPTH;
para.ColorSpace = VCS_RGB;
para.SizeX = WindowSizeX;
para.SizeY = WindowSizeY;
if(VE_OK==vOpen(¶,&VioHandle))
{
// success
}
It does not matter if Ventuz or the Client has started first. Both ends of the connection will try to connect repeatedly until a connection is established, and will try to re-connect if the connection was interrupted.
Multiple streams can be opened by different client processes.
To close a stream call vClose(), to de-initialize the whole dll after all streams have been closed call vExit().
The procedure to render a frame is as follows:
vFrame frame;
if(VE_Ok==vLockFrame(VioHandle,&frame))
{
// work with the frame
vUnlockFrame(VioHandle);
}
The timing is determined by Ventuz. The client should render as fast as possible and not wait for vertical sync. Ventuz will block the client if it renders faster than Ventuz.
The Ventuz Configuration Editor can configure the stream as either synchronized or not. In a synchronized stream, Ventuz will wait for the client and buffer up a few frames to prevent frame drops. In async mode, Ventuz will always use the latest frame, possibly dropping or duplicating frames.
If the transfer mode is VTM_Direct3D9, the vFrame provides IDirect3DSurface9 interfaces for color and depth buffer.
If the transfer mode is VTM_OpenGl, the vFrame provides OpenGL Texture2D names that can be bound either to GL_TEXTURE2D or to GL_FRAMEBUFFER.
If the transfer mode is VTM_Cpu, the vFrame provides pointers to memory. The stride (pitch) of these buffers is aligned to 4 bytes. This is important for 16-bit depth buffers with odd (not even) widths.
The client is expected to render to or read from these resources between (and only between) vLockFrame() and vUnlockFrame().
On a stream to Ventuz, the client can add ancillary data using vAncToVentuz(). The data provided will be copied, so the pointer does not need to be valid after the call.
On a stream from Ventuz, the client can extract ancillary data using vAncFromVentuz(). This function provides a pointer that will remain valid only between lock and unlock.
Ancillary data consists of a FourCC code, the data and the size. Multiple such packets can be sent or received, even of the same FourCC code. When sending ancillary to Ventuz, the LiveVideo node can be used to extract the packets for binding with other Ventuz nodes.
The ancillary data that flows in the same direction as the frames is synchronized with the frames. The frame inputs have synchronized vAncToVentuz(). But vAncFromVentuz() and vAncToVentuz() are not synchronized.
Unsynchronized data may have packets merged. This means, for example, if an output calls vAncToVentuz() every frame, timing issues may cause Ventuz to provide no data in one frame and the merged data of two frames in the next frame.
Ancillary data is currently limited to 16 packets and 32KB of data for all packets.
OpenGL usually uses a right-handed coordinate system and z-clips from -w to w. DirectX usually uses a left-handed coordinate system and z-clips from 0 to w.
You need a mirror-z matrix and then multiply it before and after the camera matrix:
| 1 0 0 0 |
mz = | 0 1 0 0 | ; cam' = mz * cam * mz
| 0 0 -1 0 |
| 0 0 0 1 |
This can be done cheaper like this:
cam.k.x *= -1; cam.i.z *= -1; cam.k.y *= -1; cam.j.z *= -1; cam.k.w *= -1; cam.l.z *= -1;
This uses a matrix class, that consists of four vectors i, j, k, and l that each have a float of x, y, z and w. Note that cam.k.z is inverted twice, so that goes away too.
The w-range is a bigger problem. The easiest solution is to create a correct DirectX projection matrix along with the OpenGl projection matrix.
Direct3D9 does not support reading the depth buffer as a texture. Various hacks are around, and we use the INTZ FourCC code. For an explanation of how this works see Advanced DX9 Capabilities for ATI Radeon Cards. The trick works also with all relevant NVidia and Intel GPUs.
The color surface provided by vFrame can be used directly as render-target. The depth surface can not, as it is an D3DFMT_R32F or D3DFMT_L16 surface. A real depth surface of type INTZ has to be created and then blitted to the provided surface.
This has to be done by rendering a rectangle, and can not be done with StretchRect(). When doing so, be sure to remember the half-pixel offset required by Direct3D9 rasterization rules.
When rendering with DirectX on a different GPU, the frame has to be transferred to memory. This involves creating an additional texture in D3DPOOL_SYSTEMMEM. To transfer from GPU to CPU memory, use GetRendertargetData(). First copy from an INTZ depth buffer to a D3DFMT_R32F texture in D3DPOOL_DEFAULT by drawing a rectangle, then use GetRendertargetData() to blit that to a D3DFMT_R32F texture in D3DPOOL_SYSTEMMEM, then lock that surface and memcpy() it to the buffer provided by the VIO API. In other words: prefer staying on the same GPU with DirectX.
As with DirectX, a real depth buffer has to be created, and then copied into the provided OpenGL texture, by rendering a rectangle. OpenGL rasterization rules do not require any offsets.
When reading from memory, use glReadPixels(). In the OpenGL case, no extra system memory textures are required, glReadPixels() can copy directly into the buffers provided by the VIO API.
As a side effect of the blitting, the depth or color buffer may be upside down, or the red and blue color channels may be swapped. This is easily compensated with the Vio Input Node. Otherwise you can also handle this on the client side.
If a DirectX device is passed to vInit(), it must use the same GPU as Ventuz. If Ventuz is already running, vGetVentuzAdapter() can be used to find out on which GPU Ventuz is running. If the VioApi creates it's own device, it will automatically use the correct GPU. This will not work correctly if the video configuration has changed between starting Ventuz and the Client. For example: Changing the main display will change the order of DirectX adapters, and the correct GPU can not be found any more.
When rendering on a different GPU than Ventuz, memory transfers (VTM_Cpu) have to be used and no DirectX device may be passed to vInit()
Rendering on a different monitor than Ventuz is not a problem if both monitors are attached to the same GPU.