Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages


home | help
NETMAP(4)	       FreeBSD Kernel Interfaces Manual		     NETMAP(4)

     netmap -- a framework for fast packet I/O

     device netmap

     userspace and kernel clients, and for Virtual Machines.  It runs on
     FreeBSD Linux and some versions of	Windows, and supports a	variety	of
     netmap ports, including

     physical NIC ports
	   to access individual	queues of network interfaces;

     host ports
	   to inject packets into the host stack;

     VALE ports
	   implementing	a very fast and	modular	in-kernel software switch/dat-

     netmap pipes
	   a shared memory packet transport channel;

     netmap monitors
	   a mechanism similar to bpf(4) to capture traffic

     All these netmap ports are	accessed interchangeably with the same API,
     and are at	least one order	of magnitude faster than standard OS mecha-
     nisms (sockets, bpf, tun/tap interfaces, native switches, pipes).	With
     suitably fast hardware (NICs, PCIe	buses, CPUs), packet I/O using netmap
     on	supported NICs reaches 14.88 million packets per second	(Mpps) with
     much less than one	core on	10 Gbit/s NICs;	35-40 Mpps on 40 Gbit/s	NICs
     (limited by the hardware);	about 20 Mpps per core for VALE	ports; and
     over 100 Mpps for netmap pipes.  NICs without native netmap support can
     still use the API in emulated mode, which uses unmodified device drivers
     and is 3-5	times faster than bpf(4) or raw	sockets.

     Userspace clients can dynamically switch NICs into	netmap mode and	send
     and receive raw packets through memory mapped buffers.  Similarly,	VALE
     switch instances and ports, netmap	pipes and netmap monitors can be cre-
     ated dynamically, providing high speed packet I/O between processes, vir-
     tual machines, NICs and the host stack.

     netmap supports both non-blocking I/O through ioctl(2), synchronization
     such as select(2),	poll(2), kqueue(2) and epoll(7).  All types of netmap
     ports and the VALE	switch are implemented by a single kernel module,
     which also	emulates the netmap API	over standard drivers.	For best per-
     formance, netmap requires native support in device	drivers.  A list of
     such devices is at	the end	of this	document.

     netmap and	VALE architecture, features and	usage.


     receive packets in	batches	through	the rings, and possibly	implement
     zero-copy forwarding between ports.

     to	all processes who own /dev/netmap file descriptors bound to NICs.  In-
     dependent VALE and	netmap pipe ports by default use separate memory re-
     gions, but	can be independently configured	to share memory.

     The following section describes the system	calls to create	and control
     level functions are described in the LIBRARIES section.

     Ports and rings are created and controlled	through	a file descriptor,
     created by	opening	a special device
	   fd =	open("/dev/netmap");
     and then bound to a specific port with an

     netmap has	multiple modes of operation controlled by the struct nmreq ar-
     gument.  arg.nr_name specifies the	netmap port name, as follows:

     OS	network	interface name (e.g., 'em0', 'eth1', ...)
	   the host stack;

	   the file descriptor is bound	to port	PPP of VALE switch SSS.
	   Switch instances and	ports are dynamically created if necessary.

	   Both	SSS and	PPP have the form [0-9a-zA-Z_]+	, the string cannot
	   exceed IFNAMSIZ characters, and PPP cannot be the name of any ex-
	   isting OS network interface.

     accessed by mmapping the memory

     the file descriptor permit	blocking I/O.

     NIC to normal mode	(reconnecting the data path to the host	stack),	or de-
     stroys the	virtual	port.

     The data structures in the	mmapped	memory region are detailed in
     <sys/net/netmap.h>, which is the ultimate reference for the netmap	API.
     The main structures and fields are	indicated below:

     struct netmap_if (one per interface)

	  struct netmap_if {
	      const uint32_t   ni_flags;      /* properties		 */
	      const uint32_t   ni_tx_rings;   /* NIC tx	rings		 */
	      const uint32_t   ni_rx_rings;   /* NIC rx	rings		 */
	      uint32_t	       ni_bufs_head;  /* head of extra bufs list */

	  Indicates the	number of available rings (struct netmap_rings)	and
	  (ni_tx_rings,	ni_rx_rings) normally depends on the hardware.	NICs
	  NIOCREGIF can	also request additional	unbound	buffers	in the same
	  memory space,	to be used as temporary	storage	for packets.  The num-
	  ber of extra buffers is specified in the arg.nr_arg3 field.  On suc-
	  cess,	the kernel writes back to arg.nr_arg3 the number of extra buf-
	  fers actually	allocated (they	may be less than the amount requested
	  if the memory	space ran out of buffers).  ni_bufs_head contains the
	  index	of the first of	these extra buffers, which are connected in a
	  list (the first uint32_t of each buffer being	the index of the next
	  buffer in the	list).	A 0 indicates the end of the list.  The	appli-
	  cation is free to modify this	list and use the buffers (i.e.,	bind-
	  ing them to the slots	of a netmap ring).  When closing the netmap
	  file descriptor, the kernel frees the	buffers	contained in the list
	  pointed by ni_bufs_head , irrespectively of the buffers originally
	  provided by the kernel on NIOCREGIF.

     struct netmap_ring	(one per ring)

	  struct netmap_ring {
	      const uint32_t num_slots;	  /* slots in each ring		   */
	      const uint32_t nr_buf_size; /* size of each buffer	   */
	      uint32_t	     cur;	  /* (u) wakeup	position	   */
	      uint32_t	     flags;

	  Implements transmit and receive rings, with read/write pointers,
	  metadata and an array	of slots describing the	buffers.

     struct netmap_slot	(one per buffer)

	  struct netmap_slot {
	      uint32_t buf_idx;		  /* buffer index		  */
	      uint16_t len;		  /* packet length		  */
	      uint16_t flags;		  /* buf changed, etc.		  */
	      uint64_t ptr;		  /* address for indirect buffers */

	  Describes a packet buffer, which normally is identified by an	index
	  and resides in the mmapped region.

     packet buffers

     all other objects are reachable through relative references (offsets or
     them into actual pointers:

	   struct netmap_if *nifp = NETMAP_IF(mem, arg.nr_offset);
	   struct netmap_ring *txr = NETMAP_TXRING(nifp, ring_index);
	   struct netmap_ring *rxr = NETMAP_RXRING(nifp, ring_index);

	   char	*buf = NETMAP_BUF(ring,	buffer_index);

     should not	be assumed to be a power of two.

     head is the first slot available to userspace;


     tail is the first slot reserved to	the kernel.

     Slot indexes must only move forward; for convenience, the function
	   nm_ring_next(ring, index)
     returns the next index modulo the ring size.

     are explicitly assigned to	the kernel.

     the slots sequentially and	advance	head and cur past slots	ready to

     Below is an example of the	evolution of a TX ring:

		   head=cur   tail
		    |	       |
		    v	       v

	 user creates new packets to (T)ransmit
		     head=cur tail
			 |     |
			 v     v

		     head=cur	   tail
			 |	    |
			 v	    v

     select() and poll() will block if there is	no space in the	ring, i.e.,
	   ring->cur ==	ring->tail
     and return	when new slots have become available.

     preparing as many packets as possible before issuing them.

     A transmit	ring with pending transmissions	has
     The function int nm_tx_pending(ring) implements this test.

     packets without returning all the previous	slots to the kernel.

     incoming packets.

     Below is an example of the	evolution of an	RX ring:

		head  cur     tail
		 |     |       |
		 v     v       v

	 user advances head and	cur, releasing some slots and holding others
		    head cur  tail
		      |	 |     |
		      v	 v     v

		    head cur	    tail
		      |	 |	     |
		      v	 v	     v

     Normally, packets should be stored	in the netmap-allocated	buffers	as-
     signed to slots when ports	are bound to a file descriptor.	 One packet is
     fully contained in	a single buffer.

     The following flags affect	slot and buffer	processing:

	  must be used when the	buf_idx	in the slot is changed.	 This can be

	  reports when this buffer has been transmitted.  Normally, netmap no-
	  tifies transmit completions in batches, hence	signals	can be delayed
	  indefinitely.	 This flag helps detect	when packets have been sent
	  and a	file descriptor	can be closed.

	  When a ring is in 'transparent' mode,	packets	marked with this flag
	  by the user application are forwarded	to the other endpoint at the
	  between a NIC	and the	host stack.

	  tells	the forwarding code that the source MAC	address	for this
	  packet must not be used in the learning bridge code.

	  indicates that the packet's payload is in a user-supplied buffer
	  size can reach 65535 bytes.

	  helps	reducing data copies in	the interconnection of virtual ma-

	  indicates that the packet continues with subsequent buffers; the
	  last buffer in a packet must have the	flag clear.

     normally used with	VALE ports when	connecting virtual machines, as	they
     cal device.

     no	place with the total length of a packet.

     greater than 1 also have NS_MOREFRAG set.

     to	query and configure ports, with	the following argument:

     struct nmreq {
	 uint32_t  nr_version;	      /* (i) API version		*/
	 uint16_t  nr_rx_rings;	      /* (i/o) number of rx rings	*/
	 uint16_t  nr_cmd;	      /* (i) special command		*/

     supported by network devices, see netintro(4).

	   that	all the	information below can change before the	interface is
	   actually put	in netmap mode.

	       indicates the size of the netmap	memory region.	NICs in	netmap
	       mode all	share the same memory region, whereas VALE ports have
	       independent regions for each port.

	   nr_tx_slots,	nr_rx_slots
	       indicate	the size of transmit and receive rings.

	   nr_tx_rings,	nr_rx_rings
	       indicate	the number of transmit and receive rings.  Both	ring
	       number and sizes	may be configured at runtime using interface-
	       specific	functions (e.g., ethtool(8) ).

	   physical device this	also switches it into netmap mode, disconnect-
	   ing it from the host	stack.	Multiple file descriptors can be bound
	   to the same port, with proper synchronization left to the user.

	   The recommended way to bind a file descriptor to a port is to use
	   function nm_open(..)	(see LIBRARIES)	which parses names to access
	   specific port types and enable features.  In	the following we docu-
	   ment	the main features.

	   NIOCREGIF can also bind a file descriptor to	one endpoint of	a
	   port, and is	meant to enable	configuration where a master process
	   acts	as a dispatcher	towards	slave processes.

	   to use, and reserve extra space in the memory region.

	   nr_flags indicating the identity of the rings controlled through
	   the file descriptor.

	   nr_flags nr_ringid selects which rings are controlled through this
	   file	descriptor.  Possible values of	nr_flags are indicated below,
	   together with the naming schemes that application libraries (such
	   as the nm_open indicated below) can use to indicate the specific
	   netmap port name.

	   NR_REG_ALL_NIC netmap:foo
		  (default) all	hardware ring pairs

	   NR_REG_SW netmap:foo^

	   NR_REG_NIC_SW netmap:foo+
		  all hardware rings and the host rings

	   NR_REG_ONE_NIC netmap:foo-i

	   NR_REG_PIPE_MASTER netmap:foo{i
		  in nr_ringid;

	   NR_REG_PIPE_SLAVE netmap:foo}i

		  pipe will only have a	single ring pair with index 0, irre-
		  spective of the value	of i.

	   fied.  The feature can be disabled by or-ing	NETMAP_NO_TX_POLL to

	   When	registering a virtual interface	that is	dynamically created to
	   nr_rx_rings fields.

	   ber of slots	available for transmission.

	   able	packets.

     form, epoll(7) and	kqueue(2) are supported	too.

     even without requesting write events.  Passing the	NETMAP_NO_TX_POLL flag
     cessed only if read events	are requested.	Passing	the NETMAP_DO_RX_POLL
     flag to NIOCREGIF updates receive rings even without read events. Note
     that on epoll(7) and kqueue(2), NETMAP_NO_TX_POLL and NETMAP_DO_RX_POLL
     only have an effect when some event is posted for the file	descriptor.

     plicity and for efficient integration with	applications.

     For convenience, the <net/netmap_user.h> header provides a	few macros and
     of	libpcap-based applications to netmap.  To use these extra functions,
     programs should
	   #define NETMAP_WITH_LIBS
	   #include <net/netmap_user.h>

     The following functions are available:

	    uint64_t flags, const struct nm_desc *arg)
	    similar to pcap_open_live(3), binds	a file descriptor to a port.

		is a port name,	in the form "netmap:PPP" for a NIC and "va-
		leSSS:PPP" for a VALE port.

		provides the initial values for	the argument to	the NIOCREGIF
		ioctl.	The nm_flags and nm_ringid values are overwritten by
		parsing	ifname and flags, and other fields can be overridden
		through	the other two arguments.

		points to a struct nm_desc containing arguments	(e.g., from a
		previously open	file descriptor) that should override the de-
		faults.	 The fields are	used as	described below

		can be set to a	combination of the following flags:
		NETMAP_NO_TX_POLL, NETMAP_DO_RX_POLL (copied into nr_ringid);
		NM_OPEN_NO_MMAP	(if arg	points to the same memory region,
		avoids the mmap	and uses the values from it); NM_OPEN_IFNAME
		NM_OPEN_ARG2, NM_OPEN_ARG3 (uses the fields from arg);
		NM_OPEN_RING_CFG (uses the ring	number and sizes from arg).

     int nm_close(struct nm_desc *d)
	    closes the file descriptor,	unmaps memory, frees resources.

	    size of the	packet is successful, or 0 on error;

	    similar to pcap_dispatch(),	applies	a callback to incoming packets

	    similar to pcap_next(), fetches the	next packet

     netmap natively supports the following devices:

     On	FreeBSD: cxgbe(4), em(4), iflib(4) (providing igb, em and lem),
     ixgbe(4), ixl(4), re(4), vtnet(4).

     On	Linux e1000, e1000e, i40e, igb,	ixgbe, ixgbevf,	r8169, virtio_net,

     cantly higher than	various	raw socket types (bpf, PF_PACKET, etc.).  Note
     that for slow devices (such as 1 Gbit/s and slower	NICs, or several 10
     Gbit/s NICs whose hardware	is unable to sustain line rate), emulated and
     native mode will likely have similar or same throughput.

     When emulation is in use, packet sniffer programs such as tcpdump could
     see received packets before they are diverted by netmap.  This behaviour
     is	not intentional, being just an artifact	of the implementation of emu-
     lation.  Note that	in case	the netmap application subsequently moves
     packets received from the emulated	adapter	onto the host RX ring, the
     sniffer will intercept those packets again, since the packets are in-
     jected to the host	stack as they were received by the network interface.

     Emulation is also available for devices with native netmap	support, which
     can be used for testing or	performance comparison.	 The sysctl variable
     dev.netmap.admode globally	controls how netmap mode is implemented.

     Some aspect of the	operation of netmap are	controlled through sysctl

     dev.netmap.admode:	0
	     Controls the use of native	or emulated adapter mode.

	     0 uses the	best available option;

	     1 forces native mode and fails if not available;

	     2 forces emulated hence never fails.

     dev.netmap.generic_rings: 1
	     Number of rings used for emulated netmap mode

     dev.netmap.generic_ringsize: 1024
	     Ring size used for	emulated netmap	mode

     dev.netmap.generic_mit: 100000
	     Controls interrupt	moderation for emulated	mode

     dev.netmap.mmap_unreg: 0

     dev.netmap.fwd: 0
	     Forces NS_FORWARD mode

     dev.netmap.flags: 0

     dev.netmap.txsync_retry: 2

     dev.netmap.no_pendintr: 1
	     Forces recovery of	transmit buffers on system calls

     dev.netmap.mitigate: 1
	     Propagates	interrupt mitigation to	user processes

     dev.netmap.no_timestamp: 0
	     Disables the update of the	timestamp in the netmap	ring

     dev.netmap.verbose: 0
	     Verbose kernel messages

     dev.netmap.buf_num: 163840

     dev.netmap.buf_size: 2048

     dev.netmap.ring_num: 200

     dev.netmap.ring_size: 36864

     dev.netmap.if_num:	100

     dev.netmap.if_size: 1024
	     the global	memory region.	The only parameter worth modifying is
	     dev.netmap.buf_num	as it impacts the total	amount of memory used
	     by	netmap.

     dev.netmap.buf_curr_num: 0

     dev.netmap.buf_curr_size: 0

     dev.netmap.ring_curr_num: 0

     dev.netmap.ring_curr_size:	0

     dev.netmap.if_curr_num: 0

     dev.netmap.if_curr_size: 0
	     Actual values in use.

     dev.netmap.bridge_batch: 1024
	     Batch size	used when moving packets across	a VALE switch.	Values
	     above 64 generally	guarantee good performance.

     dev.netmap.ptnet_vnet_hdr:	1
	     Allow ptnet devices to use	virtio-net headers

     netmap uses select(2), poll(2), epoll(7) and kqueue(2) to wake up pro-
     cesses when significant events occur, and mmap(2) to map memory.
     ioctl(2) is used to configure ports and VALE switches.

     Applications may need to create threads and bind them to specific cores
     particular, pthread_setaffinity_np(3) may be of use.

     applications.  See	the examples/ directory	in netmap distributions, or
     tools/tools/netmap/ directory in FreeBSD distributions.

     pkt-gen(8)	is a general purpose traffic source/sink.

     As	an example
	   pkt-gen -i ix0 -f tx	-l 60
     can generate an infinite stream of	minimum	size packets, and
	   pkt-gen -i ix0 -f rx
     the system	performs.

     pkt-gen(8)	has many options can be	uses to	set packet sizes, addresses,
     rates, and	use multiple send/receive threads and cores.

     bridge(4) is another test program which interconnects two netmap ports.
     It	can be used for	transparent forwarding between interfaces, as in
	   bridge -i netmap:ix0	-i netmap:ix1
     or	even connect the NIC to	the host stack using netmap
	   bridge -i netmap:ix0

     The following code	implements a traffic generator

     #include <net/netmap_user.h>
     void sender(void)
	 struct	netmap_if *nifp;
	 struct	netmap_ring *ring;
	 struct	nmreq nmr;
	 struct	pollfd fds;

	 bzero(&nmr, sizeof(nmr));
	 strcpy(nmr.nr_name, "ix0");
	 nmr.nm_version	= NETMAP_API;
	 ioctl(fd, NIOCREGIF, &nmr);
	 p = mmap(0, nmr.nr_memsize, fd);
	 nifp =	NETMAP_IF(p, nmr.nr_offset);
	 ring =	NETMAP_TXRING(nifp, 0);
	 fds.fd	= fd; = POLLOUT;
	 for (;;) {
	     poll(&fds,	1, -1);
	     while (!nm_ring_empty(ring)) {
		 i = ring->cur;
		 ... prepare packet in buf ...

     A simple receiver can be implemented using	the helper functions
     #define NETMAP_WITH_LIBS
     #include <net/netmap_user.h>
     void receiver(void)
	 struct	nm_desc	*d;
	 struct	pollfd fds;
	 u_char	*buf;
	 struct	nm_pkthdr h;
	 fds.fd	= NETMAP_FD(d); = POLLIN;
	 for (;;) {
	     poll(&fds,	1, -1);
		 consume_pkt(buf, h->len);

     do	packet forwarding between ports	swapping buffers.  The buffer from the
     transmit ring is used to replenish	the receive ring:
	 uint32_t tmp;
	 struct	netmap_slot *src, *dst;
	 tmp = dst->buf_idx;
	 dst->buf_idx =	src->buf_idx;
	 dst->len = src->len;
	 dst->flags = NS_BUF_CHANGED;
	 src->buf_idx =	tmp;
	 src->flags = NS_BUF_CHANGED;

     which you can access with the netmap API (e.g., with
	   nm_open("netmap:eth0^", ...);
     up	to the host stack.

     sender and	a receiver to it, e.g.,	running	the following in two different
     port names, e.g.,
	   pkt-gen -i vale2:x{3	-f rx #	receiver on the	master side
	   pkt-gen -i vale2:x}3	-f tx #	sender on the slave side

     The following command attaches an interface and the host stack to a
	   vale-ctl -h vale2:em0
     Other netmap clients attached to the same switch can now communicate with
     the network card or the host.

     vale(4), vale-ctl(4), bridge(8), lb(8), nmreplay(8), pkt-gen(8)


     ATC'12, June 2012,	Boston

     Luigi Rizzo, Giuseppe Lettieri, VALE, a switched ethernet for virtual ma-
     chines, ACM CoNEXT'12, December 2012, Nice


     The netmap	framework has been originally designed and implemented at the
     help from Matteo Landi, Gaetano Catalli, Giuseppe Lettieri, and Vincenzo

     netmap and	VALE have been funded by the European Commission within	FP7
     Projects CHANGE (257422) and OPENLAB (287581).

     faster interfaces requires	hardware with sufficient performance.  Several
     cient PCIe	or memory bandwidth can	also cause reduced performance.

     able flow control when running high speed experiments.

     packet filters.

     Multiple transmit and receive rings are supported natively	and can	be
     configured	with ordinary OS tools,	such as	ethtool(8) or device-specific
     filtering of incoming traffic.

     netmap does not use features such as checksum offloading, TCP
     segmentation offloading, encryption, VLAN encapsulation/decapsulation,
     etc.  When	using netmap to	exchange packets with the host stack, make
     sure to disable these features.

FreeBSD	13.0		       November	20, 2018		  FreeBSD 13.0


Want to link to this manual page? Use this URL:

home | help