Concepts

Transports

A transport moves the messages of a channel and allocates their buffers. FairMQ has two transports: zeromq sends the data through sockets, and shmem keeps the data in shared memory and sends only metadata. You select the transport for each channel in the configuration, so the code of your device does not change.

The transports and their names

Each transport is a subclass of fair::mq::TransportFactory. A transport factory makes the messages, the sockets of the channels, the pollers and the unmanaged regions of one transport. Transports.h maps the names to the values of the enum fair::mq::Transport:

Name	Enum value	Implementation	Use
`zeromq`	`Transport::ZMQ`	`fair::mq::zmq::TransportFactory` in `fairmq/zeromq/`	Data through zeromq sockets, in one process, on one host, or between hosts.
`shmem`	`Transport::SHM`	`fair::mq::shmem::TransportFactory` in `fairmq/shmem/`	Data in shared memory, between processes on one host.
`default`	`Transport::DEFAULT`	none	Channel property only: the channel uses the default transport of the device.

TransportName() converts an enum value to its name. TransportType() converts a name to the enum value and throws fair::mq::TransportError for an unknown name. GetEnabledTransports() returns Transport::ZMQ and Transport::SHM.

Make a transport factory

TransportFactory::CreateTransportFactory(type, id, config) returns a std::shared_ptr<TransportFactory>. It accepts the names zeromq and shmem (TransportFactory.cxx). For all other names, it logs an error and throws fair::mq::TransportFactoryError.

id is the id of the factory in the topology. A device gives its device id. If id is empty, the function makes a UUID.
config is a fair::mq::ProgOptions pointer. The factory reads its properties from it, for example io-threads. Without config, the factory uses the defaults.

A device makes its factories itself. You call CreateTransportFactory() yourself when you use channels without a device.

How a device selects the transport

The device selects the transports in the state INITIALIZING DEVICE (Device.cxx):

The device reads the property transport, which is the option --transport. The default is zeromq. This is the default transport of the device.
The device makes the factory of the default transport. Transport() returns it, and NewMessage() uses it.
For each sub-channel, the device reads the channel property transport. The default is default, which selects the default transport of the device.
The device gets the factory for the transport of the sub-channel with AddTransport(). This function makes a factory only if the device has no factory of this transport yet.

The result: a device has one factory for each transport that it uses. All channels with the same transport share this factory, and with it one zeromq context or one shared memory session. Each factory gets the device id and the configuration of the device.

An unknown name in --transport or in a channel property stops the initialization, and the device goes to the state ERROR.

The sink of the multiple-transports example uses shmem as the default transport and zeromq for the channel ack:

SINK="fairmq-ex-multiple-transports-sink"
SINK+=" --id sink1"
SINK+=" --severity debug"
SINK+=" --transport shmem"
SINK+=" --channel-config name=data1,type=pull,method=connect,address=tcp://127.0.0.1:5555"
SINK+="                  name=data2,type=pull,method=connect,address=tcp://127.0.0.1:5556,transport=shmem"
SINK+="                  name=ack,type=pub,method=connect,address=tcp://127.0.0.1:5557,transport=zeromq"

examples/multiple-transports/fairmq-start-ex-multiple-transports.sh.in:20–26

With --severity debug, the sink logs its choices between other debug lines. The block shows these lines without the prefix of time and severity. The order of the channels can change:

Setting 'shmem' as default transport for the device
Adding 'shmem' transport
Initializing transport for channel ack[0]: zeromq
Adding 'zeromq' transport
Initializing transport for channel data1[0]: default
Reusing existing 'shmem' transport
Initializing transport for channel data2[0]: shmem
Reusing existing 'shmem' transport

The transports during the life of a device

On each new transition request, the device calls Interrupt() on all its factories. A blocking Send() or Receive() then returns TransferCode::interrupted within about 100 ms. When the next state starts, the device calls Resume(). In the state RESETTING DEVICE, the device calls Reset() on all factories and destroys them with the channels.

Compare the transports

	`zeromq`	`shmem`
Data path	The data goes through the zeromq socket.	The data stays in a shared memory segment. A small metadata message goes through a zeromq socket.
Copies of the data	For `ipc://` and `tcp://`, the receiver gets its own copy.	None. The receiver reads the buffer that the sender allocated.
Peers	One process (`inproc://`), one host (`ipc://`, `tcp://`), or other hosts (`tcp://`).	Processes on one host with the same session.
Socket types	`push`, `pull`, `pub`, `sub`, `xpub`, `xsub`, `req`, `rep`, `dealer`, `router`, `pair`	The same, without `pub` and `sub`.
Required configuration	None.	The same `--session` on all peers, and a segment that is large enough (`--shm-segment-size`, default 2 GiB).
Message from your buffer	No copy. The transport calls your deleter after the transfer.	The transport copies the buffer into the segment and calls your deleter at once.
Message from an unmanaged region	The transport copies the block and calls the region callback at once.	No copy. The callback runs when the receiver releases the block.
`Copy()` of a message	The messages share the buffer in one process.	The messages share the buffer through a reference count, also between processes.
Clean-up	Nothing to clean up.	The last shmem factory of the session removes the shared memory, unless `--shm-no-cleanup` is `true`. After a crash, use the monitor.
Tools	none	`fairmq-shmmonitor`

An inproc:// address connects only the channels of one factory: each factory has its own zeromq context, and ZeroMQ connects inproc:// endpoints only inside one context. Thus both channels must be in the same process and use the same factory.

The zeromq transport

The zeromq transport (fairmq/zeromq/) uses the libzmq library:

Each factory has one zeromq context with --io-threads I/O threads (default 1). The context accepts up to 10000 sockets (zeromq/Context.h).
Each sub-channel has one zeromq socket of its type. The socket identity is <factory id>.<channel name>.<type>.
A sub socket subscribes to all messages.
A socket waits for 100 ms at most in each call to libzmq. Between the calls, it checks for an interruption and for the timeout of the transfer.

The shmem transport

The shmem transport (fairmq/shmem/) puts the data of each message into a managed segment of the session. The segment has a default size of 2 GiB. A send transfers only the metadata of the message through a zeromq socket: 40 bytes for a single message on a 64-bit platform, or the value of --shm-metadata-msg-size if it is larger. The receiver finds the same buffer in the segment. The factory has its own zeromq context with --io-threads I/O threads.

The devices that exchange shmem messages must run on one host and use the same --session (default default). A pub or sub socket throws fair::mq::SocketError. For the details, see How shared memory works and Sessions and segments. All devices of a session must use compatible FairMQ versions: see Mixed versions.

Tune the zeromq transport

The zeromq transport has few settings of its own. Most of the tuning is in the channel properties and in the selection of the transport. Measure each change with Run a benchmark: give the same options to fairmq-bsampler and fairmq-sink, and compare the rate lines.

I/O threads

--io-threads sets the number of I/O threads of the zeromq context (ZMQ_IO_THREADS, default 1). In code, SetNumIoThreads() sets the same property. The factory reads the value one time, when it makes its context in INITIALIZING DEVICE (zeromq/TransportFactory.h). All sockets of the factory share these threads.

ZeroMQ needs at least one I/O thread for tcp:// and ipc://. A program that uses only inproc:// can set 0 (ZeroMQ behavior).
If ZeroMQ refuses the value, the zeromq transport throws ContextError with failed configuring context, reason: <text> (zeromq/Context.h). The shmem transport only logs the error.
The shmem transport uses --io-threads for its own context, which sends the metadata.

If one I/O thread does not move the data of all channels fast enough, try 2 or more threads, and compare the rates.

Number of sockets

FairMQ sets ZMQ_MAX_SOCKETS of each context to 10000 (zeromq/Context.h). The ZeroMQ default is 1023. No option changes this value. Each FairMQ socket also makes a monitor for GetNumberOfConnectedPeers() (zeromq/Common.h). zmq_socket_monitor() makes one pair socket in the context (ZeroMQ behavior), and FairMQ makes a second one that reads the events. Thus one FairMQ socket uses three ZeroMQ sockets of the context.

Queues and kernel buffers

The channel properties set the buffers of each socket. See Channel properties.

sndBufSize and rcvBufSize (default 1000) limit the number of messages in the queues. ZeroMQ applies the limit to each peer of the socket (ZeroMQ behavior). With the zeromq transport, a queue holds the messages with their data: 1000 messages of 1 MB in one queue use 1 GB of memory. Smaller values limit this memory. A push socket then waits earlier for a slow receiver. See Behavior of each socket type.
sndKernelSize and rcvKernelSize (default 0, the system default) set the kernel buffers of the tcp:// connections in bytes. ZeroMQ does not apply them to ipc:// connections (ZeroMQ behavior). See sndKernelSize and rcvKernelSize. Linux limits the values to net.core.wmem_max and net.core.rmem_max (Linux behavior).

To send a large buffer without a copy into the message, make the message from your buffer with a deleter. See Your own buffer.

When to use shmem

Use the shmem transport for large messages between processes on one host. With ipc:// or tcp://, the zeromq transport moves each byte through the kernel, and the receiver gets its own copy. The shmem transport sends only the metadata, and the receiver reads the buffer of the sender.

The saved copy grows with the size of the message. The cost of each transfer stays: the shmem transport also sends one ZeroMQ message for each transfer, and it allocates the buffer in the segment. Measure both transports with the message sizes of your application.
The shmem transport needs the same session on all peers and a segment that holds all messages in flight. See Tune and debug.
Between hosts, and for pub and sub, use the zeromq transport.

A device can use both transports: shmem for the channels on the host, zeromq for the others. See Use several transports.

The removed ofi transport

FairMQ v1.5.0 removed the experimental ofi transport. No transport replaces it. The name ofi is an unknown transport now: CreateTransportFactory("ofi") throws TransportFactoryError, and a device with --transport ofi goes to the state ERROR. Channel::Validate() still accepts the verbs:// addresses of the ofi transport, but the zeromq and shmem transports cannot bind or connect to them. Use zeromq with tcp:// for transfers between hosts.

Messages and multipart: what a transport does with your buffers.
Use several transports: one device with zeromq and shmem channels.
How shared memory works: the shmem transport in detail.
Use channels without a device: make a transport factory in your own program.
Command-line options: --transport, --io-threads and the --shm-* options.
Run a benchmark: measure the two transports and the effect of the settings.
Use std::pmr memory resources: fill a container in the message memory of a transport.