The Encoded Stream Out outputs provide a simple pipe based streaming protocol to other processes running on the same machine. This way 3rd party applications can process or distribute the video coming out of Ventuz in real time.
There is a comprehensive example, client library and test application, written in C#, on GitHub here: https://github.com/VentuzTechnology/StreamOutExample
Ventuz makes the encoded streams available to other processes via a named pipe. These processes will need the proper permissions to access the pipe (usually this means running as the same user or at a higher privilege level than the Ventuz runtime).
The pipe's name is VentuzOutA for the first output and B etc. for any additional outputs. Ventuz accepts an arbitrary number of connections to the named pipes, so for example a stream server can choose whether to handle multiple incoming connections itself or to connect to the pipe once per client served.
Although Ventuz allows arbitrarily many connections to each pipe, all connections will share the same output stream; you can't eg. set a different bitrate for each connection.
All data Ventuz sends through the pipe is organized in chunks. All chunks start with an 8 byte chunk header structure that looks like this:
offset | type | description |
---|---|---|
+00 | FourCC | Chunk type |
+04 | uint32 | Chunk size in bytes |
("uint32" means 32 bit unsigned little endian integer; "FourCC" means Four Character Code, ie. a sequence of 4 character bytes)
The chunk header is directly followed by the chunk data with the given size. For compatibility with future versions please note:
As soon as a client connects to the pipe, Ventuz will first send a Pipe Header chunk with information about the audio and video streams.
offset | type | description |
---|---|---|
+00 | uint32 | Protocol version |
+04 | FourCC | Video format |
+08 | uint32 | Video width (pixels) |
+12 | uint32 | Video height (pixels) |
+16 | uint32 | Video frame rate numerator |
+20 | uint32 | Video frame rate denominator |
+24 | FourCC | Audio format |
+28 | uint32 | Audio sample rate (Hz) |
+32 | uint32 | Audio channel count |
Notes:
After sending the pipe header Ventuz will start transmitting video and audio data through the pipe. Every frame begins with a frame header chunk, followed by video and audio frame payload chunks.
offset | type | description |
---|---|---|
+00 | uint32 | Frame index |
+04 | uint32 | Frame flags |
The frame index is a monotonically increasing number.
Currently the following frame flags are defined:
value | description |
---|---|
0x00000001 | This video frame is an IDR frame |
This chunk contains exactly one frame worth of encoded video. If the IDR frame flag is set in the frame header it's guaranteed that this frame contains everything that's needed to start or restart a video stream, i.e. Sequence and Picture information, and no reference to past frames.
The first frame sent over a newly opened connection will always be an IDR frame so clients don't need to wait out for it.
The audio chunk contains "one video frame" worth of PCM encoded audio. Which means, the audio chunk corresponds to the exact point and duration in time the video frame is supposed to be displayed. It's therefore guaranteed that audio and video are always in sync with each other.
Even though audio is uncompressed and audio and video always come together, don't rely on audio chunks having a fixed size. When rendering at NTSCish (/1001) frame rates the audio chunk size is going to vary.
While receiving audio and video clients can send commands to Ventuz to set encoding parameters or inject keyboard and touch input. Each command consists of a command byte followed by an optional command payload structure. The following commands are defined:
byte | followed by | description |
---|---|---|
0x00 | - | No operation |
0x01 | - | Request IDR frame |
0x10 | Touch parameters | Begin Touch |
0x11 | Touch parameters | Move Touch |
0x12 | Touch parameters | End Touch |
0x13 | Touch parameters | Cancel Touch |
0x20 | Unicode character (uint32) | Key Pressed |
0x21 | VirtualKey code (uint32) | Key Down |
0x22 | VirtualKey code (uint32) | Key Up |
0x28 | Mouse XY parameters | Mouse position |
0x29 | Mouse Button parameters | Mouse buttons |
0x2a | Mouse XY parameters | Mouse wheel |
0x30 | Encode parameters | Set encode parameters |
This command instructs the encoder inside Ventuz to emit an IDR frame. Due to the asynchronous nature of the encoding process this may take a frame or two; don't assume the frame received right after sending this command will already be an IDR one.
To inject touch input clients can use the four touch commands. All of them require for following payload:
offset | type | description |
---|---|---|
+00 | uint32 | Touch ID |
+04 | int32 | X coordinate (pixels from left side of viewport) |
+08 | int32 | Y coordinate (pixels from upper side of viewport) |
Each "touch" is a sequence of one Begin Touch command, followed by zero or more Move Touch commands, and ended with either an End Touch or Cancel Touch command. For this each touch needs to be assigned a unique ID. Clients can eg. either assign IDs incrementally or randomly whenever a new touch is begun or have a "finger number". The important thing is that it's consistent for one touch and there aren't two touches at the same time with the same ID.
Clients can inject text characters with the Key Pressed command followed by this structure:
offset | type | description |
---|---|---|
+00 | uint32 | UTF32 codepoint |
Control keys that correspond to control characters such as Backspace or Enter will also work. This command only works with nodes that accept text input, i.e. Text Fields and Web Browsers.
Alternatively, clients can use the Key Up and Key Down events that supply a Windows VirtualKey as described in https://learn.microsoft.com/en-us/windows/win32/inputdev/virtual-key-codes
offset | type | description |
---|---|---|
+00 | uint32 | VirtualKey code |
Note that this command doesn't work in conjunction with the DirectInput Key node.
Injecting mouse updates works with these three messages:
A mouse position update has the following parameters:
offset | type | description |
---|---|---|
+00 | int32 | X coordinate (pixels from left side of viewport) |
+04 | int32 | Y coordinate (pixels from upper side of viewport) |
Any X or Y values outside the viewport (such as negative values) will also be treated as the mouse leaving the window or output.
A mouse button update updates all buttons at once. The message payload looks like this:
offset | type | description |
---|---|---|
+00 | uint32 | Buttons bitfield |
The buttons bitfield is the logical addition of all currently pressed buttons.
bit | value | description |
---|---|---|
0 | 0x01 | Left |
1 | 0x02 | Right |
2 | 0x04 | Middle |
3 | 0x08 | X1 |
4 | 0x10 | X2 |
Mouse wheel messages have the following payload:
offset | type | description |
---|---|---|
+00 | int32 | Reserved (set to zero) |
+04 | int32 | Wheel delta |
The wheel delta follows Windows conventions, i.E. usually positive and negative 120 units per click.
Note that the mouse commands doen't work in conjunction with the DirectInput Mouse node.
This command sets certain encode parameters without interrupting the video stream. This way eg. a server can dynamically adjust the bitrate to match client or transport constraints. The parameter structure for this command looks like this:
offset | type | description |
---|---|---|
+00 | uint32 | Rate control mode |
+04 | uint32 | Bitrate or QP |
There are two values for the rate control mode:
value | description |
---|---|
0x00000000 | Encode with constant QP (Second field specifies QP (0..51)) |
0x00000001 | Encode with constant bitrate (Second field specifies bitrate in kbits/s) |
The encoder will change its behavior as quickly as possible upon receiving this command. Due to the asynchronous nature of the encoding process it may take a few frames unil the new settings are applied.
The following source code opens a connection to Ventuz and dumps the video and audio streams as "test.264" and "test.pcm" into the computer's TEMP directory, and lets you send commands to Ventuz by pressing keys. You're free to copy it partially or in total, or to use it as reference for your own implementation.
using System; using System.IO; using System.IO.Pipes; using System.Runtime.InteropServices; /// <summary> /// /// Copyright (C) Ventuz 2020-2024. All rights reserved. /// /// Example for reading from the Ventuz StreamOut pipe. /// Opens the pipe and dumps the audio and video streams into files. /// /// Changelog /// 2024-07-23 [TH] Added mouse messages /// 2019-08-29 [TH] Initial public version /// /// </summary> namespace StreamOutPipeExample { // The data coming from the pipe is organized in chunks. every chunk starts with this, followed by the chunk data [StructLayout(LayoutKind.Sequential)] public struct ChunkHeader { public uint fourCC; // chunk type (four character code) public int size; // size of chunk data } // Pipe header with general information (VVSP chunk) [StructLayout(LayoutKind.Sequential)] public struct PipeHeader { public static readonly int VERSION = 2; public uint hdrVersion; // should be VERSION public uint videoCodecFourCC; // used video codec, currently 'h264' aka h.264 public uint videoWidth; // width of image public uint videoHeight; // height of image public uint videoFrameRateNum; // frame rate numerator public uint videoFrameRateDen; // frame rate denominator public uint audioCodecFourCC; // used audio codec, currently 'pc16' aka 16 bit signed little endian PCM public uint audioRate; // audio sample rate, currently fixed at 48000 public uint audioChannels; // audio channel count, currently 2 } // Frame header, gets sent every frame, followed by video and then audio data (fhdr chunk) [StructLayout(LayoutKind.Sequential)] public struct FrameHeader { [Flags] public enum FrameFlags : uint { IDR_FRAME = 0x01, // frame is IDR frame (aka key frame / stream restart/sync point) } public uint frameIndex; // frame index. If this isn't perfectly contiguous there was a frame drop in Ventuz public FrameFlags flags; // flags, see above } // Commands to send back to the encoder. enum PipeCommand : byte { Nop = 0x00, // Do nothing RequestIDRFrame = 0x01, // Request an IDR frame. it may take a couple of frames until the IDR frame arrives due to latency reasons. TouchBegin = 0x10, // Start a touch. Must be followed by a TouchPara structure TouchMove = 0x11, // Move a touch. Must be followed by a TouchPara structure TouchEnd = 0x12, // Release a touch. Must be followed by a TouchPara structure TouchCancel = 0x13, // Cancal a touch if possible. Must be followed by a TouchPara structure Char = 0x20, // Send a keystroke. Must be followed by a KeyPara structure KeyDown = 0x21, // Send a key down event. Must be followed by a KeyPara structure KeyUp = 0x22, // Send a key up event. Must be followed by a KeyPara structure MouseMove = 0x28, // Mouse positon update. Must be followed by a MouseXYPara structure MouseButtons = 0x29, // Mouse buttons update. Must be followed by a MouseButtonsPara structure MouseWheel = 0x2a, // Mouse wheel update. Must be followed by a MouseXYPara structure // NOTE: Currently only the Y value of the wheel is supported SetEncodePara = 0x30, // Send encoder parameters. Must be followed by an EncodePara structure } // Parameters for Touch* PipeCommands [StructLayout(LayoutKind.Sequential)] public struct TouchPara { public uint id; // numerical id. Must be unique per touch (eg. finger # or something incremental) public int x; // x coordinate in pixels from the left side of the viewport public int y; // y coordinate in pixels from the upper side of the viewport }; // Parameters for the Key PipeCommand [StructLayout(LayoutKind.Sequential)] public struct KeyPara { public uint Code; // UTF32 codepoint for Char command; Windows VK_* for KeyUp/KeyDown }; // Parameters for the MouseMove and MouseWheel PipeCommands [StructLayout(LayoutKind.Sequential)] public struct MouseXYPara { public int x; // x coordinate in pixels from the left side of the viewport, or horizontal wheel delta public int y; // y coordinate in pixels from the upper side of the viewport, or vertical wheel delta } // Mouse buttons bitfield [Flags] public enum MouseButtons : uint { Left = 0x01, Right = 0x02, Middle = 0x04, X1 = 0x08, X2 = 0x10, }; // Parameters for the MouseButtons PipeCommand [StructLayout(LayoutKind.Sequential)] public struct MouseButtonsPara { public MouseButtons Buttons; // bit field of all buttons pressed at the time }; // Parameters for the SetEncodePara PipeCommand [StructLayout(LayoutKind.Sequential)] public struct EncodePara { public enum RateControlMode : uint { ConstQP = 0, // BitrateOrQP is QP (0..51) ConstRate = 1, // BitrateOrQP is rate in kBits/s } public RateControlMode Mode; public uint BitrateOrQP; }; class Program { // The following helper functions are only meant as a reference. // In a full implementation these should be asynchronous or // at least have some kind of timeout/error handling. /// <summary> /// Read a number of bytes from a stream and return as array /// </summary> public static byte[] ReadBytes(Stream stream, int length) { var bytes = new byte[length]; int done = 0; while (done < length) { int read = stream.Read(bytes, done, length-done); if (read == 0) throw new IOException("stream ended"); done += read; } return bytes; } /// <summary> /// Read a struct from a stream by just blitting the data /// </summary> public static T ReadStruct<T>(Stream stream, int size = 0) { var tsize = Marshal.SizeOf(typeof(T)); if (size <= 0) size = tsize; if (size < tsize) throw new ArgumentException("size is too small"); var bytes = ReadBytes(stream, size); GCHandle handle = GCHandle.Alloc(bytes, GCHandleType.Pinned); T theStructure = (T)Marshal.PtrToStructure(handle.AddrOfPinnedObject(), typeof(T)); handle.Free(); return theStructure; } /// <summary> /// Send a command to the pipe /// </summary> /// <param name="cmd"></param> /// <returns></returns> public static void SendCommand(Stream stream, PipeCommand cmd) { var bytes = new byte[] { (byte)cmd }; stream.Write(bytes, 0, 1); } /// <summary> /// Send a command to the pipe, with data struct /// </summary> public static void SendCommand<T>(Stream stream, PipeCommand cmd, T para) { int size = Marshal.SizeOf(para); byte[] arr = new byte[size+1]; arr[0] = (byte)cmd; var h = GCHandle.Alloc(arr, GCHandleType.Pinned); Marshal.StructureToPtr(para, h.AddrOfPinnedObject()+1, false); h.Free(); stream.Write(arr, 0, arr.Length); } /// <summary> /// Make uint FourCC from four characters /// </summary> public static uint FourCC(char a, char b, char c, char d) => ((uint)a << 24) | ((uint)b << 16) | ((uint)c << 8) | ((uint)d); /// <summary> /// Entry point here /// </summary> static void Main(string[] args) { Console.WriteLine("Ventuz video stream pipe receiver example\n"); // The pipe name is "VentuzOutA" (or B etc for additional outputs), local only. // You can connect to the pipe as many times as you wish (eg. once per client you serve), // or you can connect once and handle distribution yourself. using (var stream = new NamedPipeClientStream("VentuzOutA")) { Console.WriteLine("Connecting...\n"); // connect to the pipe stream.Connect(); // try to get first chunk var chunk = ReadStruct<ChunkHeader>(stream); if (chunk.fourCC != FourCC('V', 'V', 'S', 'P')) throw new Exception("invalid header from pipe"); // get header var header = ReadStruct<PipeHeader>(stream, chunk.size); if (header.hdrVersion != PipeHeader.VERSION) throw new Exception("wrong protocol version"); // check codecs // let's only accept h.264 here, but it works the same with HEVC if (header.videoCodecFourCC != FourCC('h', '2', '6', '4') || header.audioCodecFourCC != FourCC('p', 'c', '1', '6')) throw new Exception("unsupported video or audio codec"); Console.WriteLine($"video: {header.videoWidth}x{header.videoHeight} @ {(float)header.videoFrameRateNum / header.videoFrameRateDen}fps"); Console.WriteLine($"audio: {header.audioChannels}ch @ {header.audioRate}Hz"); Console.WriteLine("Press ESC to quit.\n"); // video should always start with a full IDR frame (key frame with decoder reset, // contains metadata such as picture and sequence information). Let's check if this is true. bool expectIDRFrame = true; float bytespersec = 0; // let's open some files to dump the streams into. You can open the .264 file with // VLC or Media Player Classic, or mux video and audio using FFMpeg. FileStream videoFile = null; FileStream audioFile = null; try { var temppath = Path.GetTempPath(); videoFile = File.Create(temppath + "test.264"); audioFile = File.Create(temppath + "test.pcm"); } catch (Exception ex) { Console.WriteLine($"ERROR: Can't create output file(s): {ex.Message}\n"); } // now let's receive some stuff! Forever! bool quit = false; while (!quit) { while (Console.KeyAvailable) { var rk = Console.ReadKey(); var key = rk.Key; // On second thought, quit if esc is pressed. if ( key == ConsoleKey.Escape ) { quit = true; break; } else if (key == ConsoleKey.Spacebar) { // If the space bar is pressed, request an IDR frame // (not setting expectIDRFrame here; it might take a frame or two until the IDR frame arrives) SendCommand(stream, PipeCommand.RequestIDRFrame); } else if (key == ConsoleKey.F1) { // simulate a singular touch var para = new TouchPara { id = 12345, x = 100, y = 100, }; SendCommand(stream, PipeCommand.TouchBegin, para); SendCommand(stream, PipeCommand.TouchEnd, para); } else if (rk.KeyChar >= '1' && rk.KeyChar <= '9') { // numeric keys: switch encoder bitrate to 1 mbit per key :) SendCommand(stream, PipeCommand.SetEncodePara, new EncodePara { Mode = EncodePara.RateControlMode.ConstRate, BitrateOrQP = ((uint)rk.KeyChar - '0') * 1000, }); } else { // forward all other keys to Ventuz SendCommand(stream, PipeCommand.Char, new KeyPara { Code = rk.KeyChar }); } } try { // frame header first chunk = ReadStruct<ChunkHeader>(stream); if (chunk.fourCC != FourCC('f', 'h', 'd', 'r')) { // skip unknown chunks var _ = ReadBytes(stream, chunk.size); continue; } var frameheader = ReadStruct<FrameHeader>(stream, chunk.size); if (expectIDRFrame) { if (!frameheader.flags.HasFlag(FrameHeader.FrameFlags.IDR_FRAME)) throw new Exception("IDR frame expected"); expectIDRFrame = false; } // read video data chunk = ReadStruct<ChunkHeader>(stream); if (chunk.fourCC != FourCC('f', 'v', 'i', 'd')) throw new Exception("video frame expected"); byte[] videoFrame = ReadBytes(stream, chunk.size); // read audio data chunk = ReadStruct<ChunkHeader>(stream); if (chunk.fourCC != FourCC('f', 'a', 'u', 'd')) throw new Exception("audio frame expected"); byte[] audioFrame = ReadBytes(stream, chunk.size); // calc video bitrate (exponential moving average over packet size times frame rate) bytespersec += 0.1f * ((float)videoFrame.Length * header.videoFrameRateNum / header.videoFrameRateDen - bytespersec); // print stuff and dump streams into files if (frameheader.flags.HasFlag(FrameHeader.FrameFlags.IDR_FRAME)) Console.WriteLine($"Received IDR frame {frameheader.frameIndex} "); Console.Write($"got frame {frameheader.frameIndex}, {bytespersec * 8 / 1000.0f}kbits/s \r"); videoFile?.Write(videoFrame, 0, videoFrame.Length); audioFile?.Write(audioFrame, 0, audioFrame.Length); } catch (Exception e) { Console.WriteLine($"pipe read failed: {e.Message}"); break; } } Console.WriteLine("good bye."); videoFile?.Close(); audioFile?.Close(); } } } }