HowTo: Cluster Rendering

Introduction

If you have a large Ventuz Scene which spans over multiple screens you can try to use one PC with one graphics card and span all outputs together to get one large desktop (render output). Due to some architectural changes, on Windows 7 this is currently only possible with certain graphics card drivers. Make sure your chosen graphics card supports spanning. The main advantage of this solution is that you can be sure that all screens are in sync.

If your setup requires more than one machine, synchronizing animation and video playback becomes a problem. You can use OpenSoundControl (OSC) to start your animations/videos at the same time, but since all PC´s are running on different time-bases the animations/videos will drift and get out of sync. To avoid this, Ventuz provides the possibility of using a Cluster Clock, which ensures that all machines connected to the same cluster are receiving an identical timing.

Cluster Clock and Cluster ID

In order to set up a clustered environment, check the Enable Cluster Clock Synchronization checkbox in the Machine Configuration on all machines in your cluster. You can do this either under Menu/Tools/Machine Configuration in Ventuz or by using the Machine Configuration Tool in the Ventuz Technology program folder.

Make sure that all machines which are supposed to use the same cluster clock have the same Group ID (must be greater than zero!) and a different Machine ID applied. The machine with the lowest Machine ID within a group will serve as the Cluster Clock Master. The algorithm used for interchanging the cluster clock information is easy and transparent: Every machine sends out its current time right after it has finished rendering a frame but before the vertical retrace of the graphics card has occurred. As soon the vertical retrace releases the renderer from its wait state every machine receives all clock information from all other machines - include its own. Every machine simply selects the clock from the lowest machine-ID is has received and adds a value of one to enter the next rendering cycle. This makes the lowest ID the Master but if the master stops rendering or has been disconnected from network another machine will become the Master on-the-fly!

Note: All changes done in the Machine Configuration dialog require a restart of Ventuz to take effect.

To achieve a properly working cluster clock it is highly recommended that:

1. You are using machines with identical hardware.
2. You are using the same scene on all machines (use the SystemID node to apply machine specific settings e.g. projection center shift)
3. That your scene is running properly with the maximum frame-rate.
4. You are using identical project and graphics card settings(a different anti-aliasing setting will bring the clock out of sync).

Once the cluster is set up properly, all timing related nodes except for DirectShow movie and audio nodes are affected by the cluster timing. Please note that the QuickTime Movie node provides the possibility to disable the cluster timing by checking the Use Audio Clock property. Some nodes, like Scroll Text, Counter or Ticker can be configured to work either frame based or time based. In order to sync those nodes via the cluster clock the Progress property has to be set to time based.

When activating the statistics in your render window, you will notice some additional information:

ID 01.02 means that this machine has the Group-ID 01 and the Machine-ID 02, the following number shows the current value of the cluster clock, and the -01 indicates that the current clock master has Machine-ID 01.
On a slave system the type of the cluster clock is supposed to be green colored, on a master system it is white.

A font color change on a slave system from green to white indicates that the slave has lost its master clock and is running independently. Make sure that your slave systems show a constant green font without any flickering between green and white.

Whereas Movers in Absolute mode are running exactly in sync no matter when the scene was started, nodes which need to be triggered (e.g. Mover in OneShot mode) are supposed to receive the trigger signal simultaneously. This can be achieved by using e.g. OSC multicasting.

Remoting

As soon as multiple Ventuz machines are running in a cluster the .net Remoting Interface can use the when parameter to synchronize execution of commands. The commands have to be scheduled slightly in the future (1 to 4 frame) to give Ventuz and the network infrastructure some time to process the communication. As long commands are issued to all machines with same when value they will be processed in-sync. See also Remoting.

Network multicast cluster setup

You should generally try to run the Cluster Clock on a separate network with a separate switch (or hub). Since every PC in your Ventuz Cluster broadcasts the Cluster Clock every sixtieth of a second into your network, it is a good idea to detach the Cluster network from the rest of your network.

Multicast messaging with multiple network interface cards

You have to setup the network card (NIC) which is connected to the cluster clock network to the lowest metric. Use the route command in the cmd shell. Make sure that you run the cmd shell as Administrator (right click "cmd prompt", choose run as Administrator).

route  change 224.0.0.0 mask 240.0.0.0 <IP of network card here> metric 25

If you want this to persist then add -p, otherwise changes will be lost with the next reboot.

Frame Synchronization

Frame synchronization is the process of synchronizing display pixel scanning to a synchronization source. When several systems are connected, a sync signal is fed from a master system to the other systems in the network, and the displays are synchronized with each other.

The NVidia Quadro® G-Sync offers frame synchronization to enable multiple systems synchronizing frame rendering across multiple displays plus synchronizing one or many systems to a common sync source.

Frame lock uses hardware to synchronize the frames on each display and to redraw onto multiple displays at the same time. When Ventuz is displayed across multiple monitors, frame-locked systems help maintain image continuity.

Frame locking several systems requires that the systems are connected with the sync signal fed from the master to the other systems in the group. A network of frame-locked systems can be synchronized by connecting the master system to an external synchronization source.

An NVidia Quadro card together with a G-Sync II board or a ATI 8800 together with a S400 board can framelock your cluster such that each display runs exactly the same frequency and phase as the master.

Making one system the server (master)

• Right click desktop / NVidia Control Panel / Synchronize Displays: – Chose timing server – Choose internal timing (server settings 60 Hz)

(If you have live video you can also use the house sync to synchronize the master)

For further details please refer to the NVidia and ATI graphics card documentation.

Client systems

On all other systems go to Synchronize displays and check this server: "Select displays to lock to the server"

Check on the "View System Topology" tab that the framelock sync puls is "Present".

Setup external sync

If you have live video in (Deltacast SDI Board) you should sync your cluster to the house sync (black burst). To achieve this the server(master) of your cluster has to be genlocked to the external house sync using the BNC input socket, all the clients in the cluster will still receive the sync signal via frame lock.

Copyright 2011 Ventuz Technology Group