OpenOnload
OpenOnload can reduce network latency for AMPS deployments that use supported Solarflare or Xilinx network adapters. OpenOnload provides a user-space network stack for selected sockets, which can reduce operating system overhead in latency-sensitive deployments.
Use OpenOnload when the application requires the lowest practical TCP latency and the host network hardware, driver, operating system, and operational model support OpenOnload. For most deployments, start with the general Host Guidance and Linux OS Settings recommendations, then test OpenOnload with the complete AMPS workload before deploying it in production.
Runtime and Driver Compatibility
Keep the OpenOnload user-space libraries compatible with the OpenOnload driver on the host. Avoid mixing a container image library with a different host driver version unless that combination is explicitly tested and supported by the operations team.
Visit the Onload user guide to learn more about Onload installation and configuration.
AMPS Configuration
AMPS uses OpenOnload when the AMPS process is started with the OpenOnload runtime active. In that environment, AMPS detects OpenOnload and uses OpenOnload stacks for eligible TCP sockets.
The AMPS configuration file can include the OpenOnload transport setting when the configuration should explicitly show that a transport is intended to run with OpenOnload. This setting is optional and does not load OpenOnload by itself.
<AMPSConfig>
...
<Transports>
<Transport>
<Name>amps-tcp</Name>
<Type>tcp</Type>
<InetAddr>nic-ip:9007</InetAddr>
<Protocol>amps</Protocol>
<OpenOnload>enabled</OpenOnload>
</Transport>
</Transports>
...
</AMPSConfig>
The setting above does not load OpenOnload by itself. The AMPS process must still be started with the OpenOnload runtime active, as shown in the next section.
Starting AMPS with OpenOnload
On a physical host, the onload command is the most direct way to start AMPS with OpenOnload. The command configures the process environment and preloads the OpenOnload library before starting AMPS.
onload ampServer config.xml
Setting LD_PRELOAD directly also loads the OpenOnload runtime when the deployment does not use the onload wrapper.
LD_PRELOAD=libonload.so ./ampServer config.xml
Optionally, an OpenOnload profile can be provided to tune behavior. The following settings can be applied by supplying --profile= with the onload command.
onload --profile=/path/to/amps.opf ampServer config.xml
The following is the AMPS low-latency Onload profile that can be used as a starting point. Save the contents to amps.opf, test the settings with the complete workload, and pass the profile to onload.
# SPDX-License-Identifier: BSD-2-Clause
# X-SPDX-Copyright-Text: (c) Solarflare Communications Inc
# 60East AMPS Low Latency Profile
# Enable CTPIO even if using full-featured firmware. N.B. This is
# incompatible with multicast loopback.
onload_set EF_CTPIO_SWITCH_BYPASS 1
# Assume that application does not invoke calls on a single epoll object
# concurrently.
onload_set EF_EPOLL_MT_SAFE 1
# Enable polling / spinning. When the application makes a blocking call
# such as recv() or poll(), this causes Onload to busy wait for up to 100ms
# before blocking.
onload_set EF_POLL_USEC 100000
# Disable FASTSTART when connection is new or has been idle for a while.
# The additional acks it causes add latency on the receive path.
onload_set EF_TCP_FASTSTART_INIT 0
onload_set EF_TCP_FASTSTART_IDLE 0
# Use a large initial congestion window so that the slow-start algorithm
# doesn't cause delays. We don't enable this by default because it breaks
# the TCP specs, and could cause congestion in your network. Uncomment if
# you think you need this.
# onload_set EF_TCP_INITIAL_CWND 1048576
# When TCP_NODELAY is used, always kick packets out immediately. This is
# not enabled by default because most apps benefit from the default
# behaviour.
onload_set EF_NONAGLE_INFLIGHT_MAX 65535
Alternatively, set the same OpenOnload options as environment variables in the AMPS process environment before starting ampServer:
export LD_PRELOAD=libonload.so
export EF_CTPIO_SWITCH_BYPASS=1
export EF_EPOLL_MT_SAFE=1
export EF_POLL_USEC=100000
export EF_TCP_FASTSTART_INIT=0
export EF_TCP_FASTSTART_IDLE=0
# export EF_TCP_INITIAL_CWND=1048576 # optional
export EF_NONAGLE_INFLIGHT_MAX=65535
After setting the variables, start AMPS normally:
./ampServer config.xml
Monitoring
At AMPS startup, confirm that the runtime environment is active. AMPS logs environment variables such as LD_PRELOAD, and this can be used to verify that the process started with libonload.so loaded.
The environment variable 'LD_PRELOAD' is set to 'libonload.so'
Use OpenOnload tools such as onload_stackdump to confirm that AMPS has OpenOnload stacks and to inspect the OpenOnload environment used by the process.
onload_stackdump
Example output with AMPS running under Onload:
#stack-id stack-name pids
1 AMPS:DEFAULT-p1 1
The absence of AMPS stacks could indicate could indicate that AMPS did not start under Onload.
As AMPS clients connect, run onload_stackdump again and confirm that new stack names beginning with AMPS: appear.
#stack-id stack-name pids
19 AMPS:0-t230 1
1 AMPS:DEFAULT-p1 1
For OpenOnload deployments, include the following in the production monitoring plan:
- OpenOnload runtime verification at process startup.
- NIC counters, packet drops, retransmits, and link status.
- End-to-end publish and subscribe latency from the application perspective.
For general AMPS, host, container, and VM monitoring guidance, see Monitoring and Virtualization Considerations.
Virtual Machines
OpenOnload is primarily a physical-host acceleration technology. In a virtual machine, OpenOnload is only appropriate when the VM has the required access to supported network hardware, such as through a hardware passthrough configuration that exposes the NIC and required driver behavior to the guest.
Do not assume that OpenOnload improves an AMPS deployment just because the physical host has supported hardware. Test the exact VM type, host configuration, NIC exposure model, CPU allocation, and AMPS workload. Validate Onload using onload_stackdump to confirm AMPS is running with the accelerated path. If the VM cannot use the accelerated path, follow the standard virtual machine guidance and run AMPS with the normal TCP stack.
Containers
OpenOnload can be used from a container when the container has access to the OpenOnload runtime, the host network path, and the required device nodes. The Onload container guidance uses Docker; the example below shows the same general requirements using Podman. Adjust the image, paths, and runtime options for the deployment.
podman run -it \
--name amps \
--network host \
--device=/dev/onload \
--device=/dev/onload_epoll \
-v /path/to/amps:/amps \
-v /usr/lib64/libonload.so:/usr/lib64/libonload.so \
-w /amps/bin \
-e LD_PRELOAD=libonload.so \
-e EF_CTPIO_SWITCH_BYPASS=1 \
-e EF_EPOLL_MT_SAFE=1 \
-e EF_POLL_USEC=100000 \
-e EF_TCP_FASTSTART_INIT=0 \
-e EF_TCP_FASTSTART_IDLE=0 \
-e EF_NONAGLE_INFLIGHT_MAX=65535 \
fedora:latest \
./ampServer config.xml
The example mounts the host OpenOnload shared library at /usr/lib64/libonload.so into the same path in the container. Another valid approach is to build the matching OpenOnload user-space components into the container image, provided that the image contents are kept compatible with the host driver.
Prefer explicit device mappings when the container runtime and host policy allow them. Some environments require --privileged for OpenOnload because device, capability, or namespace access is otherwise incomplete. Use --privileged only when the security model for the host and container has been reviewed.
Host networking and OpenOnload device access give the AMPS container direct access to host networking resources. Review the host networking security guidance in Host Guidance before exposing the container to production networks.