Vector Packet Processing (VPP) portability on FreeBSD

Post by Francois ten Krooden
Greetings
We have a vested interest in high-speed IPsec VPN on FreeBSD. We have started with the porting of VPP (https://fd.io/) to FreeBSD.
Currently we have VPP compiled and running with netmap. The speeds we measure are nowhere near the performance of a 10Gbps link, at around 350kpps for 1500 byte IPv4 packets. We suspect the biggest issue is related to how VPP implements huge pages (Linux) and our modifications to support super pages on FreeBSD.
Apart from the above, there are remaining issues we need to sort out and "Linuxisms" that need porting to FreeBSD, but this is going reasonably well. We are working in a public Github repository and have started listing our issues there alongside the code. Our main working branch is "freebsd" (https://github.com/ftk-ntq/vpp/tree/freebsd).
Our aim with this mail is to get the discussion started on porting VPP to FreeBSD and to invite interested parties to help with the effort. We intend to upstream the work hoping that the original authors will adopt our ported code and continue maintaining future compatibility with FreeBSD.
1. netmap vs. DPDK (VPP relies on DPDK by default with the netmap integration deprecated). Which will be the best to choose?
2. How to correctly implement using super pages / huge pages in FreeBSD in order to allow VPP to allocate contiguous memory blocks for packet buffers to process packets from the packet handling framework (netmap/DPDK)?
3. What are suitable alternatives for reading information from procfs and sysfs on FreeBSD?
4. Functionality relying on Linux epoll is currently supported using epoll-shim. Is this the correct approach?
Any help and input to aid in the effort will be greatly appreciated.

Jacques Fourie

2021-05-10 13:41:27 UTC

Post by Francois ten Krooden
Greetings
We have a vested interest in high-speed IPsec VPN on FreeBSD. We have

started with the porting of VPP (https://fd.io/) to FreeBSD.

Post by Francois ten Krooden
Currently we have VPP compiled and running with netmap. The speeds we

measure are nowhere near the performance of a 10Gbps link, at around
350kpps for 1500 byte IPv4 packets. We suspect the biggest issue is related
to how VPP implements huge pages (Linux) and our modifications to support
super pages on FreeBSD.

Post by Francois ten Krooden
Apart from the above, there are remaining issues we need to sort out and

"Linuxisms" that need porting to FreeBSD, but this is going reasonably
well. We are working in a public Github repository and have started listing
our issues there alongside the code. Our main working branch is "freebsd" (
https://github.com/ftk-ntq/vpp/tree/freebsd).

Post by Francois ten Krooden
Our aim with this mail is to get the discussion started on porting VPP

to FreeBSD and to invite interested parties to help with the effort. We
intend to upstream the work hoping that the original authors will adopt our
ported code and continue maintaining future compatibility with FreeBSD.

Post by Francois ten Krooden
1. netmap vs. DPDK (VPP relies on DPDK by default with the netmap

integration deprecated). Which will be the best to choose?

Post by Francois ten Krooden
2. How to correctly implement using super pages / huge pages in FreeBSD

in order to allow VPP to allocate contiguous memory blocks for packet
buffers to process packets from the packet handling framework (netmap/DPDK)?

Post by Francois ten Krooden
3. What are suitable alternatives for reading information from procfs

and sysfs on FreeBSD?

Post by Francois ten Krooden
4. Functionality relying on Linux epoll is currently supported using

epoll-shim. Is this the correct approach?

Post by Francois ten Krooden
Any help and input to aid in the effort will be greatly appreciated.

DPDK is already supported on FreeBSD :
https://doc.dpdk.org/guides/freebsd_gsg/install_from_ports.html. It has
been a long time since I tried using DPDK on FreeBSD but the basics should
work.

Post by Francois ten Krooden
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-net

Francois ten Krooden

2021-05-11 06:25:19 UTC

Am 10.05.2021 um 13:08 schrieb Francois ten Krooden
Greetings
We have a vested interest in high-speed IPsec VPN on FreeBSD. We have

started with the porting of VPP (https://fd.io/) to FreeBSD.

Currently we have VPP compiled and running with netmap. The speeds we

Apart from the above, there are remaining issues we need to sort out and

"Linuxisms" that need porting to FreeBSD, but this is going reasonably well.
We are working in a public Github repository and have started listing our
issues there alongside the code. Our main working branch is "freebsd"
(https://github.com/ftk-ntq/vpp/tree/freebsd).

Our aim with this mail is to get the discussion started on porting VPP to

FreeBSD and to invite interested parties to help with the effort. We intend
to upstream the work hoping that the original authors will adopt our ported
code and continue maintaining future compatibility with FreeBSD.

1. netmap vs. DPDK (VPP relies on DPDK by default with the netmap

integration deprecated). Which will be the best to choose?

2. How to correctly implement using super pages / huge pages in FreeBSD

in order to allow VPP to allocate contiguous memory blocks for packet
buffers to process packets from the packet handling framework
(netmap/DPDK)?

3. What are suitable alternatives for reading information from procfs and

sysfs on FreeBSD?

4. Functionality relying on Linux epoll is currently supported using epoll-

shim. Is this the correct approach?

Any help and input to aid in the effort will be greatly appreciated.

IIRC, somebody from Netgate (pfSense) had porting this and/or DPDK as a
„spare time project“.
Not sure how far that actually proceeded. Maybe somebody else knows more?
Thanks for your efforts, in any case.
https://doc.dpdk.org/guides/freebsd_gsg/install_from_ports.html. It has
been a long time since I tried using DPDK on FreeBSD but the basics should
work.

I did have a look at the port of DPDK and you are correct the basics does work.
On FreeBSD however it seems to require the use of the contigmem driver to allocate memory. Which is not a problem when DPDK is used on it's own.
However VPP has a lot of memory management code as well and I suspect there might be some issues with how VPP and DPDK handles their own memory.

Post by Francois ten Krooden
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-net

Important Notice:

This e-mail and its contents are subject to the Nanoteq (Pty) Ltd e-mail legal notice available at:
http://www.nanoteq.com/AboutUs/EmailDisclaimer.aspx

Konstantin Belousov

2021-05-10 14:10:15 UTC

Superpage CPU mappings, as opposed to small pages mappings, typically
give you several units of percents improvement in best setup. Having
large page support in hardware for things like memory registration/rings
mapping could give you some more if hardware DMA engine is limited in
capacity, e.g. have fixed number of descriptors.

To allocate physically-contiguous superpage-mapped regions in userspace,
create special posix shmfd with shm_create_largepage(), then map it
with mmap(2). The backing pages are allocated on creation, so you can
e.g. pass them to hardware without mmaping into userspace at all, if
needed/preferred.

Post by Francois ten Krooden
3. What are suitable alternatives for reading information from procfs and sysfs on FreeBSD?

Understand what information is obtained, then what for is it actually used,
then match it against equivalent FreeBSD approach, then gather the required
information.

Post by Francois ten Krooden
4. Functionality relying on Linux epoll is currently supported using epoll-shim. Is this the correct approach?
Any help and input to aid in the effort will be greatly appreciated.
Regards
Francois ten Krooden
http://www.nanoteq.com/AboutUs/EmailDisclaimer.aspx
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-net

Francois ten Krooden

2021-05-11 12:43:10 UTC

Post by Francois ten Krooden
Greetings
We have a vested interest in high-speed IPsec VPN on FreeBSD. We have

started with the porting of VPP (https://fd.io/) to FreeBSD.

Post by Francois ten Krooden
Currently we have VPP compiled and running with netmap. The speeds we

Post by Francois ten Krooden
Apart from the above, there are remaining issues we need to sort out and

"Linuxisms" that need porting to FreeBSD, but this is going reasonably well.
We are working in a public Github repository and have started listing our
issues there alongside the code. Our main working branch is "freebsd"
(https://github.com/ftk-ntq/vpp/tree/freebsd).

Post by Francois ten Krooden
Our aim with this mail is to get the discussion started on porting VPP to

Post by Francois ten Krooden
1. netmap vs. DPDK (VPP relies on DPDK by default with the netmap

integration deprecated). Which will be the best to choose?

Post by Francois ten Krooden
2. How to correctly implement using super pages / huge pages in FreeBSD

in order to allow VPP to allocate contiguous memory blocks for packet
buffers to process packets from the packet handling framework
(netmap/DPDK)?
Superpage CPU mappings, as opposed to small pages mappings, typically
give you several units of percents improvement in best setup. Having large
page support in hardware for things like memory registration/rings mapping
could give you some more if hardware DMA engine is limited in capacity,
e.g. have fixed number of descriptors.
To allocate physically-contiguous superpage-mapped regions in userspace,
create special posix shmfd with shm_create_largepage(), then map it with
mmap(2). The backing pages are allocated on creation, so you can e.g. pass
them to hardware without mmaping into userspace at all, if
needed/preferred.

Thank you I will have a look at this as well. I knew about memfd_create which was also added in FreeBSD 13.0 which I did use.

Post by Francois ten Krooden
3. What are suitable alternatives for reading information from procfs and

sysfs on FreeBSD?
Understand what information is obtained, then what for is it actually used,
then match it against equivalent FreeBSD approach, then gather the
required information.

Thank you. This was basically what we suspected.
One of the ones we are unsure about is what the equivalent of /proc/self/pagemap on Linux would be.
The one idea we had is using procstat_getvmmap from libprocstat, but haven't finished investigating yet.

Regards
Francois

Post by Francois ten Krooden
4. Functionality relying on Linux epoll is currently supported using epoll-

shim. Is this the correct approach?

Post by Francois ten Krooden
Any help and input to aid in the effort will be greatly appreciated.
Regards
Francois ten Krooden
This e-mail and its contents are subject to the Nanoteq (Pty) Ltd e-mail
http://www.nanoteq.com/AboutUs/EmailDisclaimer.aspx
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-net

Important Notice:

This e-mail and its contents are subject to the Nanoteq (Pty) Ltd e-mail legal notice available at:
http://www.nanoteq.com/AboutUs/EmailDisclaimer.aspx

Mark Johnston

2021-05-11 13:04:14 UTC

Post by Francois ten Krooden
3. What are suitable alternatives for reading information from procfs and

sysfs on FreeBSD?
Understand what information is obtained, then what for is it actually used,
then match it against equivalent FreeBSD approach, then gather the
required information.

I believe DPDK's libeal uses /proc/pagemap to look up the physical
address of large page mappings. Assuming you want to do the same thing,
there is the MEM_EXTRACT_PADDR /dev/mem ioctl. It was added
specifically for DPDK. See the mem(4) man page for details on its
usage.

Jim Thompson via freebsd-net

2021-05-11 14:27:46 UTC

Post by Mark Johnston

Post by Francois ten Krooden
3. What are suitable alternatives for reading information from procfs and

sysfs on FreeBSD?
Understand what information is obtained, then what for is it actually used,
then match it against equivalent FreeBSD approach, then gather the
required information.

Upthread mentioned Netgate (we do pfSense) as being possibly involved in a port of VPP to FreeBSD. We do have some interest, and I had once handed this to a contractor, but
not much progress was made, and we had our hands full with pfSense and our VPP-based product (tnsr). VPP has been discussed on freebsd-net before.

You and your team have made it much farther.

I’d spoken to the VPP ‘core team’ about the sustainability of a port of VPP to FreeBSD, and they didn’t say ’No’. The lead VPP person at Cisco used to be a FreeBSD src committer, and he basically said, “Patches welcome”. Someone will probably want to bring CSIT along to help prevent regressions on FreeBSD, but that team also seemed open when I checked.

IMO, netmap seems a better approach on FreeBSD, as you’ll eventually want something like the ‘router plugin’ to be able to communicate between the kernel and VPP, obtain FIBs, etc.
We’ve been strong proponents of (and a main contributor to) the ‘linux-cp” plugin, (a better version of the router plugin, partially upstreamed to VPP), but it’s based on netlink, and is has more linux-isms than the main body of VPP. Netmap has a very nice architecture for building something similar, where packets can be directly sent to and received from the host rings by VPP, and these host rings are associated with an interface ring. Netmap-fwd uses this to good effect.

Netmap is slightly slower in maximum PPS than DPDK, but for your use case (IPsec), you won't see a difference (assuming some things when I say this.)

Jim

Francois ten Krooden

2021-05-14 06:46:40 UTC

Post by Mark Johnston

On Mon, May 10, 2021 at 11:08:18AM +0000, Francois ten Krooden

Post by Francois ten Krooden
3. What are suitable alternatives for reading information from procfs and

sysfs on FreeBSD?
Understand what information is obtained, then what for is it
actually used, then match it against equivalent FreeBSD approach,
then gather the required information.

Thank you. This was basically what we suspected.
One of the ones we are unsure about is what the equivalent of

/proc/self/pagemap on Linux would be.

Post by Mark Johnston

Post by Francois ten Krooden
The one idea we had is using procstat_getvmmap from libprocstat, but

haven't finished investigating yet.

Post by Mark Johnston
I believe DPDK's libeal uses /proc/pagemap to look up the physical
address of large page mappings. Assuming you want to do the same
thing, there is the MEM_EXTRACT_PADDR /dev/mem ioctl. It was added
specifically for DPDK. See the mem(4) man page for details on its
usage.

Upthread mentioned Netgate (we do pfSense) as being possibly involved in
a port of VPP to FreeBSD. We do have some interest, and I had once handed
this to a contractor, but not much progress was made, and we had our hands
full with pfSense and our VPP-based product (tnsr). VPP has been discussed
on freebsd-net before.
You and your team have made it much farther.
I’d spoken to the VPP ‘core team’ about the sustainability of a port of VPP to
FreeBSD, and they didn’t say ’No’. The lead VPP person at Cisco used to be a
FreeBSD src committer, and he basically said, “Patches welcome”.
Someone will probably want to bring CSIT along to help prevent regressions
on FreeBSD, but that team also seemed open when I checked.
IMO, netmap seems a better approach on FreeBSD, as you’ll eventually want
something like the ‘router plugin’ to be able to communicate between the
kernel and VPP, obtain FIBs, etc.
We’ve been strong proponents of (and a main contributor to) the ‘linux-cp”
plugin, (a better version of the router plugin, partially upstreamed to VPP),
but it’s based on netlink, and is has more linux-isms than the main body of
VPP. Netmap has a very nice architecture for building something similar,
where packets can be directly sent to and received from the host rings by
VPP, and these host rings are associated with an interface ring. Netmap-fwd
uses this to good effect.
Netmap is slightly slower in maximum PPS than DPDK, but for your use case
(IPsec), you won't see a difference (assuming some things when I say this.)
Jim

Thank you for the feedback Jim.
It is appreciated.
Will have a look. I might have some questions about who to contact when we reach a point where we think it can be upstreamed to VPP.

Regards
Francois

Important Notice:

This e-mail and its contents are subject to the Nanoteq (Pty) Ltd e-mail legal notice available at:
http://www.nanoteq.com/AboutUs/EmailDisclaimer.aspx

Francois ten Krooden

2021-05-14 06:41:32 UTC

On Mon, May 10, 2021 at 11:08:18AM +0000, Francois ten Krooden

Post by Francois ten Krooden
3. What are suitable alternatives for reading information from procfs and

sysfs on FreeBSD?
Understand what information is obtained, then what for is it
actually used, then match it against equivalent FreeBSD approach,
then gather the required information.

Thank you. This was basically what we suspected.
One of the ones we are unsure about is what the equivalent of

/proc/self/pagemap on Linux would be.

Post by Francois ten Krooden
The one idea we had is using procstat_getvmmap from libprocstat, but

haven't finished investigating yet.
I believe DPDK's libeal uses /proc/pagemap to look up the physical address
of large page mappings. Assuming you want to do the same thing, there is
the MEM_EXTRACT_PADDR /dev/mem ioctl. It was added specifically for
DPDK. See the mem(4) man page for details on its usage.

Thank you Mark.
Will have a look at that.

Regards
Francois

Important Notice:

This e-mail and its contents are subject to the Nanoteq (Pty) Ltd e-mail legal notice available at:
http://www.nanoteq.com/AboutUs/EmailDisclaimer.aspx

Luigi Rizzo

2021-05-10 13:39:18 UTC

Post by Francois ten Krooden
Greetings
We have a vested interest in high-speed IPsec VPN on FreeBSD. We have

started with the porting of VPP (https://fd.io/) to FreeBSD.

Post by Francois ten Krooden
Currently we have VPP compiled and running with netmap. The speeds we

350kpps is way too low for being related to tlb
and page size issues.

I suspect that you are
either using the "emulated" netmap mode,
which runs on top of mbufs and may involve
extra data copies on the receive path,
or the test itself is e.g. using TCP and congestion
control or limited window size throttle down
the rate.

I'd retry the test with some open loop traffic
source/sink, and using first all possible low level
APIs (sockets/mbufs; emulated netmap; native netmap; your vpp port) with
small and large packets and increasing packet rates, to see where the
limits are with each. That should give you good hints to figure out what is
making the performance so bad.
Even better if you can run the test between one known-good endpoint so you
can test separately the tx and tx sides.

Cheers
Luigi

Post by Francois ten Krooden
Apart from the above, there are remaining issues we need to sort out and

Post by Francois ten Krooden
Our aim with this mail is to get the discussion started on porting VPP

Post by Francois ten Krooden
1. netmap vs. DPDK (VPP relies on DPDK by default with the netmap

integration deprecated). Which will be the best to choose?

Post by Francois ten Krooden
2. How to correctly implement using super pages / huge pages in FreeBSD

in order to allow VPP to allocate contiguous memory blocks for packet
buffers to process packets from the packet handling framework (netmap/DPDK)?

Post by Francois ten Krooden
3. What are suitable alternatives for reading information from procfs

and sysfs on FreeBSD?

Post by Francois ten Krooden
4. Functionality relying on Linux epoll is currently supported using

epoll-shim. Is this the correct approach?

Post by Francois ten Krooden
Any help and input to aid in the effort will be greatly appreciated.

IIRC, somebody from Netgate (pfSense) had porting this and/or DPDK as a
„spare time project“.
Not sure how far that actually proceeded. Maybe somebody else knows more?
Thanks for your efforts, in any case.
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-net

--
-----------------------------------------+-------------------------------
Prof. Luigi RIZZO, ***@iet.unipi.it . Dip. di Ing. dell'Informazione
http://www.iet.unipi.it/~luigi/ . Universita` di Pisa
TEL +39-050-2217533 . via Diotisalvi 2
Mobile +39-338-6809875 . 56122 PISA (Italy)
-----------------------------------------+-------------------------------

Luigi Rizzo

2021-05-10 14:39:18 UTC

[repost since it appears to be blocked]

350kpps is way too low for being related to tlb
and page size issues.

I suspect that you are
either using the "emulated" netmap mode,
which runs on top of mbufs and may involve
extra data copies on the receive path,
or the test itself is e.g. using TCP and congestion
control or limited window size throttle down
the rate.

I'd retry the test with some open loop traffic
source/sink, and using first all possible low level
APIs (sockets/mbufs; emulated netmap; native netmap; your vpp port)
with small and large packets and increasing packet rates, to see where
the limits are with each. That should give you good hints to figure
out what is making the performance so bad.
Even better if you can run the test between one known-good endpoint so
you can test separately the tx and tx sides.

Cheers
Luigi

Francois ten Krooden

2021-05-11 06:18:03 UTC

-----Original Message-----
Sent: Monday, 10 May 2021 16:39
Subject: Re: Vector Packet Processing (VPP) portability on FreeBSD
[repost since it appears to be blocked]

Am 10.05.2021 um 13:08 schrieb Francois ten Krooden
Greetings
We have a vested interest in high-speed IPsec VPN on FreeBSD. We have

started with the porting of VPP (https://fd.io/) to FreeBSD.

Currently we have VPP compiled and running with netmap. The speeds

we measure are nowhere near the performance of a 10Gbps link, at around
350kpps for 1500 byte IPv4 packets. We suspect the biggest issue is related
to how VPP implements huge pages (Linux) and our modifications to support
super pages on FreeBSD.
350kpps is way too low for being related to tlb and page size issues.
I suspect that you are
either using the "emulated" netmap mode, which runs on top of mbufs and
may involve extra data copies on the receive path, or the test itself is e.g.
using TCP and congestion control or limited window size throttle down the
rate.

The NIC's we are using is the 'Intel X552' (10 GbE SFP+) on the Supermicro X10SDV-8C-TLN4F+ board.
I know this uses the 'ixgbe' driver in FreeBSD which is netmap enabled.
Is there a way I can confirm if it is running in emulated netmap mode?

I'd retry the test with some open loop traffic source/sink, and using first all
possible low level APIs (sockets/mbufs; emulated netmap; native netmap;
your vpp port) with small and large packets and increasing packet rates, to
see where the limits are with each. That should give you good hints to figure
out what is making the performance so bad.
Even better if you can run the test between one known-good endpoint so you
can test separately the tx and tx sides.

Thanks for these suggestions. I will rerun the tests again and just check if I can make some headway on this.
One additional test I have done previously is to use 'netmap-fwd' from https://github.com/Netgate/netmap-fwd.
When I executed the tests with netmap-fwd I was able to reach around 800kpps for 1500 byte packets, which was almost at the maximum speed. (I will just confirm this number again)

Cheers
Francois

Cheers
Luigi

Important Notice:

This e-mail and its contents are subject to the Nanoteq (Pty) Ltd e-mail legal notice available at:
http://www.nanoteq.com/AboutUs/EmailDisclaimer.aspx

Francois ten Krooden

2021-05-13 08:31:54 UTC

-----Original Message-----
From: Francois ten Krooden
Sent: Tuesday, 11 May 2021 08:18
Subject: RE: Vector Packet Processing (VPP) portability on FreeBSD

-----Original Message-----
Sent: Monday, 10 May 2021 16:39
Subject: Re: Vector Packet Processing (VPP) portability on FreeBSD
[repost since it appears to be blocked]
On Mon, May 10, 2021 at 2:42 PM Rainer Duffner

Am 10.05.2021 um 13:08 schrieb Francois ten Krooden
Greetings
We have a vested interest in high-speed IPsec VPN on FreeBSD. We
have

started with the porting of VPP (https://fd.io/) to FreeBSD.

Currently we have VPP compiled and running with netmap. The speeds

we measure are nowhere near the performance of a 10Gbps link, at
around 350kpps for 1500 byte IPv4 packets. We suspect the biggest
issue is related to how VPP implements huge pages (Linux) and our
modifications to support super pages on FreeBSD.
350kpps is way too low for being related to tlb and page size issues.
I suspect that you are
either using the "emulated" netmap mode, which runs on top of mbufs
and may involve extra data copies on the receive path, or the test itself is

e.g.

using TCP and congestion control or limited window size throttle down
the rate.

The NIC's we are using is the 'Intel X552' (10 GbE SFP+) on the Supermicro
X10SDV-8C-TLN4F+ board.
I know this uses the 'ixgbe' driver in FreeBSD which is netmap enabled.
Is there a way I can confirm if it is running in emulated netmap mode?

I'd retry the test with some open loop traffic source/sink, and using
first all possible low level APIs (sockets/mbufs; emulated netmap;
native netmap; your vpp port) with small and large packets and
increasing packet rates, to see where the limits are with each. That
should give you good hints to figure out what is making the performance

so bad.

Even better if you can run the test between one known-good endpoint so
you can test separately the tx and tx sides.

Thanks for these suggestions. I will rerun the tests again and just check if I
can make some headway on this.
One additional test I have done previously is to use 'netmap-fwd' from
https://github.com/Netgate/netmap-fwd.
When I executed the tests with netmap-fwd I was able to reach around
800kpps for 1500 byte packets, which was almost at the maximum speed. (I
will just confirm this number again)
Cheers
Francois

Cheers
Luigi

Important Notice:

This e-mail and its contents are subject to the Nanoteq (Pty) Ltd e-mail legal notice available at:
http://www.nanoteq.com/AboutUs/EmailDisclaimer.aspx

Francois ten Krooden

2021-05-13 11:27:09 UTC

On Thu, May 13, 2021 at 10:42 AM Francois ten Krooden

Post by Francois ten Krooden
Hi
Just for info I ran a test using TREX (https://trex-tgn.cisco.com/)
Where I just sent traffic in one direction through the box running FreeBSD

with VPP using the netmap interfaces.

Post by Francois ten Krooden
These were the results we found before significant packet loss started

occuring.

Those numbers are way too low for netmap.
I believe you are either using the emulated mode, or issuing a system call on
every single packet.
I am not up to date (Vincenzo may know better) but there used to be a sysctl
https://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4
SYSCTL VARIABLES AND MODULE PARAMETERS
Some aspects of the operation of netmap and VALE are controlled
through
sysctl variables on FreeBSD (dev.netmap.*) and module parameters on
Linux
dev.netmap.admode: 0
Controls the use of native or emulated adapter mode.
0 uses the best available option;
1 forces native mode and fails if not available;
2 forces emulated hence never fails.
If it still exists, try set it to 1. If the program fails, then you should figure out
why native netmap support is not compiled in.

Thank you. I did set this to 1 specifically now and it still works. So then it should be running in native mode.

I will dig a bit into the function that processes the incoming packets.
The code I currently use was added to VPP in somewhere before 2016, so it might be that there is a bug in that code.

Will try and see if I can find anything interesting there.

cheers
luigi

Important Notice:

This e-mail and its contents are subject to the Nanoteq (Pty) Ltd e-mail legal notice available at:
http://www.nanoteq.com/AboutUs/EmailDisclaimer.aspx

Jacques Fourie

2021-05-13 11:58:39 UTC

On Thu, May 13, 2021 at 10:42 AM Francois ten Krooden

Post by Francois ten Krooden
Hi
Just for info I ran a test using TREX (https://trex-tgn.cisco.com/)
Where I just sent traffic in one direction through the box running

FreeBSD

with VPP using the netmap interfaces.

Post by Francois ten Krooden
These were the results we found before significant packet loss started

occuring.

Those numbers are way too low for netmap.
I believe you are either using the emulated mode, or issuing a system

call on

every single packet.
I am not up to date (Vincenzo may know better) but there used to be a

sysctl

https://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4
SYSCTL VARIABLES AND MODULE PARAMETERS
Some aspects of the operation of netmap and VALE are controlled
through
sysctl variables on FreeBSD (dev.netmap.*) and module parameters on
Linux
dev.netmap.admode: 0
Controls the use of native or emulated adapter mode.
0 uses the best available option;
1 forces native mode and fails if not available;
2 forces emulated hence never fails.
If it still exists, try set it to 1. If the program fails, then you

should figure out

why native netmap support is not compiled in.

Thank you. I did set this to 1 specifically now and it still works. So
then it should be running in native mode.
I will dig a bit into the function that processes the incoming packets.
The code I currently use was added to VPP in somewhere before 2016, so it
might be that there is a bug in that code.
Will try and see if I can find anything interesting there.

cheers
luigi

A couple of questions / suggestions:
Will it be possible to test using the netmap bridge app or a vale switch
instead of vpp?
Did you verify that the TREX setup can perform at line rate when connected
back to back?
Which NICs are you using?

Post by Francois ten Krooden
This e-mail and its contents are subject to the Nanoteq (Pty) Ltd e-mail
http://www.nanoteq.com/AboutUs/EmailDisclaimer.aspx
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-net

Francois ten Krooden

2021-05-13 13:02:04 UTC

On Thursday, 13 May 2021 13:59 Jacques Fourie

On Thu, May 13, 2021 at 10:42 AM Francois ten Krooden

Post by Francois ten Krooden
Hi
Just for info I ran a test using TREX (https://trex-tgn.cisco.com/)
Where I just sent traffic in one direction through the box

running FreeBSD

with VPP using the netmap interfaces.

Post by Francois ten Krooden
These were the results we found before significant packet loss started

occuring.

Those numbers are way too low for netmap.
I believe you are either using the emulated mode, or issuing a system call

every single packet.
I am not up to date (Vincenzo may know better) but there used to be a

sysctl

out

why native netmap support is not compiled in.

Thank you. I did set this to 1 specifically now and it still works. So then it
should be running in native mode.
I will dig a bit into the function that processes the incoming packets.
The code I currently use was added to VPP in somewhere before 2016, so it
might be that there is a bug in that code.
Will try and see if I can find anything interesting there.

cheers
luigi

Thank you for the suggestions.

Post by Francois ten Krooden
Will it be possible to test using the netmap bridge app or a vale switch
instead of vpp?

I did perform a test using netmap-fwd (https://github.com/Netgate/netmap-fwd)
I did look at the code and it appears that the packets are processed as a batch in the application. But each packet is passed through the complete IP stack in the application, before the next one is processed.
With this application it was possible to reach about 1.4Mpps for 64-byte packets, and 812 kpps for 1518 byte packets
I haven't done any other tweaking on the FreeBSD box yet. It is running FreeBSD 13.0

Post by Francois ten Krooden
Did you verify that the TREX setup can perform at line rate when connected
back to back?

Post by Francois ten Krooden
Which NICs are you using?

We are using Intel X552 10 GbE SFP+ NIC's which is part of the Intel Xeon D-1537 SoC, on a SuperMicro X10SDV-8C-TLN4F+ Board.

I will also put the results on the github repository
https://github.com/ftk-ntq/vpp/wiki
and will update as we get some more information

Kind Regards
Francois

Post by Francois ten Krooden
This e-mail and its contents are subject to the Nanoteq (Pty) Ltd e-mail legal
http://www.nanoteq.com/AboutUs/EmailDisclaimer.aspx
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-net

Important Notice:

This e-mail and its contents are subject to the Nanoteq (Pty) Ltd e-mail legal notice available at:
http://www.nanoteq.com/AboutUs/EmailDisclaimer.aspx

Jim Thompson via freebsd-net

2021-05-14 14:11:59 UTC

Post by Francois ten Krooden
Thank you. I did set this to 1 specifically now and it still works. So then it
should be running in native mode.
I will dig a bit into the function that processes the incoming packets.
The code I currently use was added to VPP in somewhere before 2016, so it
might be that there is a bug in that code.
Will try and see if I can find anything interesting there.

It was added to plumb in a TCP stack, (mstack, iirc) prior to the VPP group writing their own. L

Your netmap-fed results are also quite low. loos@ (who wrote netmap-fwd) was getting 2.23Mpps single core on a xeon D-1540.

Jim

Francois ten Krooden

2021-05-17 09:46:44 UTC

Post by Jim Thompson via freebsd-net

It was added to plumb in a TCP stack, (mstack, iirc) prior to the VPP group writing their own. L

Post by Jim Thompson via freebsd-net
http://youtu.be/cAVgvzivzII
Jim

Important Notice:

This e-mail and its contents are subject to the Nanoteq (Pty) Ltd e-mail legal notice available at:
http://www.nanoteq.com/AboutUs/EmailDisclaimer.aspx

Vincenzo Maffione

2021-05-16 07:22:25 UTC

Hi,
Yes, you are not using emulated netmap mode.

In the test setup depicted here
https://github.com/ftk-ntq/vpp/wiki/VPP-throughput-using-netmap-interfaces#test-setup
I think you should really try to replace VPP with the netmap "bridge"
application (tools/tools/netmap/bridge.c), and see what numbers you get.

You would run the application this way
# bridge -i ix0 -i ix1
and this will forward any traffic between ix0 and ix1 (in both directions).

These numbers would give you a better idea of where to look next (e.g. VPP
code improvements or system tuning such as NIC interrupts, CPU binding,
etc.).

Cheers,
Vincenzo

Il giorno gio 13 mag 2021 alle ore 15:02 Francois ten Krooden <

Post by Francois ten Krooden
On Thursday, 13 May 2021 13:59 Jacques Fourie

On Thu, May 13, 2021 at 10:42 AM Francois ten Krooden

Post by Francois ten Krooden
Hi
Just for info I ran a test using TREX (https://trex-tgn.cisco.com/)
Where I just sent traffic in one direction through the box

running FreeBSD

with VPP using the netmap interfaces.

Post by Francois ten Krooden
These were the results we found before significant packet loss

started

occuring.

Those numbers are way too low for netmap.
I believe you are either using the emulated mode, or issuing a system

call

Post by Francois ten Krooden
on

every single packet.
I am not up to date (Vincenzo may know better) but there used to be a

sysctl

Linux
dev.netmap.admode: 0
Controls the use of native or emulated adapter mode.
0 uses the best available option;
1 forces native mode and fails if not available;
2 forces emulated hence never fails.
If it still exists, try set it to 1. If the program fails, then you

should figure

Post by Francois ten Krooden
out

why native netmap support is not compiled in.

Thank you. I did set this to 1 specifically now and it still works. So

then it

Post by Francois ten Krooden
should be running in native mode.
I will dig a bit into the function that processes the incoming packets.
The code I currently use was added to VPP in somewhere before 2016, so it
might be that there is a bug in that code.
Will try and see if I can find anything interesting there.

cheers
luigi

Thank you for the suggestions.

Post by Francois ten Krooden
Will it be possible to test using the netmap bridge app or a vale switch
instead of vpp?

I did perform a test using netmap-fwd (
https://github.com/Netgate/netmap-fwd)
I did look at the code and it appears that the packets are processed as a
batch in the application. But each packet is passed through the complete
IP stack in the application, before the next one is processed.
With this application it was possible to reach about 1.4Mpps for 64-byte
packets, and 812 kpps for 1518 byte packets
I haven't done any other tweaking on the FreeBSD box yet. It is running FreeBSD 13.0

Post by Francois ten Krooden
Did you verify that the TREX setup can perform at line rate when

connected

Post by Francois ten Krooden
back to back?

Post by Francois ten Krooden
Which NICs are you using?

We are using Intel X552 10 GbE SFP+ NIC's which is part of the Intel Xeon
D-1537 SoC, on a SuperMicro X10SDV-8C-TLN4F+ Board.
I will also put the results on the github repository
https://github.com/ftk-ntq/vpp/wiki
and will update as we get some more information
Kind Regards
Francois

Post by Francois ten Krooden
This e-mail and its contents are subject to the Nanoteq (Pty) Ltd e-mail

legal

Post by Francois ten Krooden
http://www.nanoteq.com/AboutUs/EmailDisclaimer.aspx
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-net

This e-mail and its contents are subject to the Nanoteq (Pty) Ltd e-mail
http://www.nanoteq.com/AboutUs/EmailDisclaimer.aspx

Francois ten Krooden

2021-05-17 09:53:25 UTC

Post by Vincenzo Maffione
Hi,
Yes, you are not using emulated netmap mode.
In the test setup depicted here
https://github.com/ftk-ntq/vpp/wiki/VPP-throughput-using-netmap-
interfaces#test-setup
I think you should really try to replace VPP with the netmap "bridge"
application (tools/tools/netmap/bridge.c), and see what numbers you get.
You would run the application this way
# bridge -i ix0 -i ix1
and this will forward any traffic between ix0 and ix1 (in both directions).
These numbers would give you a better idea of where to look next (e.g. VPP
code improvements or system tuning such as NIC interrupts, CPU binding,
etc.).

Thank you for the suggestion.
I did run a test with the bridge this morning, and updated the results as well.
+-------------+------------------+
| Packet Size | Throughput (pps) |
+-------------+------------------+
| 64 bytes | 7.197 Mpps |
| 128 bytes | 7.638 Mpps |
| 512 bytes | 2.358 Mpps |
| 1280 bytes | 964.915 kpps |
| 1518 bytes | 815.239 kpps |
+-------------+------------------+
Besides for the 64-byte and 128-byte packets the other sizes where matching the maximum rates possible on 10Gbps.
This was when the bridge application was running on a single core, and the cpu core was maxing out at a 100%.

I think there might be a bit of system tuning needed, but I suspect most of the improvement would be needed in VPP.

Regards
Francois

Post by Vincenzo Maffione
Cheers,
Vincenzo
Il giorno gio 13 mag 2021 alle ore 15:02 Francois ten Krooden <

Post by Francois ten Krooden
On Thursday, 13 May 2021 13:59 Jacques Fourie

On Thu, May 13, 2021 at 7:27 AM Francois ten Krooden

On Thu, May 13, 2021 at 10:42 AM Francois ten Krooden

Post by Francois ten Krooden
Hi
Just for info I ran a test using TREX
(https://trex-tgn.cisco.com/) Where I just sent traffic in one
direction through the box

running FreeBSD

with VPP using the netmap interfaces.

Post by Francois ten Krooden
These were the results we found before significant packet loss

started

occuring.

Those numbers are way too low for netmap.
I believe you are either using the emulated mode, or issuing a system

call

every single packet.
I am not up to date (Vincenzo may know better) but there used to be a

sysctl

should figure

out

why native netmap support is not compiled in.

Thank you. I did set this to 1 specifically now and it still works.
So

then it

should be running in native mode.
I will dig a bit into the function that processes the incoming packets.
The code I currently use was added to VPP in somewhere before 2016,
so it might be that there is a bug in that code.
Will try and see if I can find anything interesting there.

cheers
luigi

Thank you for the suggestions.

Will it be possible to test using the netmap bridge app or a vale
switch instead of vpp?

I did perform a test using netmap-fwd (
https://github.com/Netgate/netmap-fwd)
I did look at the code and it appears that the packets are processed
as a batch in the application. But each packet is passed through the
complete IP stack in the application, before the next one is processed.
With this application it was possible to reach about 1.4Mpps for
64-byte packets, and 812 kpps for 1518 byte packets I haven't done any
other tweaking on the FreeBSD box yet. It is running FreeBSD 13.0

Did you verify that the TREX setup can perform at line rate when

connected

back to back?

We did tests with TREX back to back yesterday and we reached the

following.

Which NICs are you using?

We are using Intel X552 10 GbE SFP+ NIC's which is part of the Intel Xeon
D-1537 SoC, on a SuperMicro X10SDV-8C-TLN4F+ Board.
I will also put the results on the github repository
https://github.com/ftk-ntq/vpp/wiki
and will update as we get some more information
Kind Regards
Francois

This e-mail and its contents are subject to the Nanoteq (Pty) Ltd e-mail

legal

http://www.nanoteq.com/AboutUs/EmailDisclaimer.aspx
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-net

This e-mail and its contents are subject to the Nanoteq (Pty) Ltd
http://www.nanoteq.com/AboutUs/EmailDisclaimer.aspx

_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-net

Important Notice:

This e-mail and its contents are subject to the Nanoteq (Pty) Ltd e-mail legal notice available at:
http://www.nanoteq.com/AboutUs/EmailDisclaimer.aspx

Marko Zec

2021-05-17 17:20:54 UTC

On Mon, 17 May 2021 09:53:25 +0000

Post by Vincenzo Maffione
Hi,
Yes, you are not using emulated netmap mode.
In the test setup depicted here
https://github.com/ftk-ntq/vpp/wiki/VPP-throughput-using-netmap-
interfaces#test-setup
I think you should really try to replace VPP with the netmap
"bridge" application (tools/tools/netmap/bridge.c), and see what
numbers you get.
You would run the application this way
# bridge -i ix0 -i ix1
and this will forward any traffic between ix0 and ix1 (in both directions).
These numbers would give you a better idea of where to look next
(e.g. VPP code improvements or system tuning such as NIC
interrupts, CPU binding, etc.).

I assume you're on 13.0 where netmap throughput is lower compared to
11.x due to migration of most drivers to iflib (apparently increased
overhead) and different driver defaults. On 11.x I could move 10G line
rate from one ix to another at low CPU freqs, where on 13.x the CPU
must be set to max speed, and still can't do 14.88 Mpps.

#1 thing which changed: default # of packets per ring dropped down from
2048 (11.x) to 1024 (13.x). Try changing this in /boot/loader.conf:

dev.ixl.0.iflib.override_nrxds=2048
dev.ixl.0.iflib.override_ntxds=2048
dev.ixl.1.iflib.override_nrxds=2048
dev.ixl.1.iflib.override_ntxds=2048
etc.

For me this increases the throughput of
bridge -i netmap:ixl0 -i netmap:ixl1
from 9.3 Mpps to 11.4 Mpps

#2: default interrupt moderation delays seem to be too long. Combined
with increasing the ring sizes, reducing dev.ixl.0.rx_itr from 62
(default) to 40 increases the throughput further from 11.4 to 14.5 Mpps

Hope this helps,

Marko

Post by Francois ten Krooden
Besides for the 64-byte and 128-byte packets the other sizes where
matching the maximum rates possible on 10Gbps. This was when the
bridge application was running on a single core, and the cpu core was
maxing out at a 100%.
I think there might be a bit of system tuning needed, but I suspect
most of the improvement would be needed in VPP.
Regards
Francois

Kevin Bowling

2021-05-18 07:32:05 UTC

Post by Marko Zec
On Mon, 17 May 2021 09:53:25 +0000

Post by Vincenzo Maffione
Hi,
Yes, you are not using emulated netmap mode.
In the test setup depicted here
https://github.com/ftk-ntq/vpp/wiki/VPP-throughput-using-netmap-
interfaces#test-setup
I think you should really try to replace VPP with the netmap
"bridge" application (tools/tools/netmap/bridge.c), and see what
numbers you get.
You would run the application this way
# bridge -i ix0 -i ix1
and this will forward any traffic between ix0 and ix1 (in both directions).
These numbers would give you a better idea of where to look next
(e.g. VPP code improvements or system tuning such as NIC
interrupts, CPU binding, etc.).

I believe this issue is in the combined txrx interrupt filter. It is
causing a bunch of unnecessary tx re-arms.

Post by Marko Zec
#1 thing which changed: default # of packets per ring dropped down from
dev.ixl.0.iflib.override_nrxds=2048
dev.ixl.0.iflib.override_ntxds=2048
dev.ixl.1.iflib.override_nrxds=2048
dev.ixl.1.iflib.override_ntxds=2048
etc.
For me this increases the throughput of
bridge -i netmap:ixl0 -i netmap:ixl1
from 9.3 Mpps to 11.4 Mpps
#2: default interrupt moderation delays seem to be too long. Combined
with increasing the ring sizes, reducing dev.ixl.0.rx_itr from 62
(default) to 40 increases the throughput further from 11.4 to 14.5 Mpps
Hope this helps,
Marko

_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-net

Vincenzo Maffione

2021-05-18 21:11:03 UTC

Il giorno mar 18 mag 2021 alle ore 09:32 Kevin Bowling <

Post by Marko Zec
On Mon, 17 May 2021 09:53:25 +0000

Post by Vincenzo Maffione
Hi,
Yes, you are not using emulated netmap mode.
In the test setup depicted here
https://github.com/ftk-ntq/vpp/wiki/VPP-throughput-using-netmap-
interfaces#test-setup
I think you should really try to replace VPP with the netmap
"bridge" application (tools/tools/netmap/bridge.c), and see what
numbers you get.
You would run the application this way
# bridge -i ix0 -i ix1
and this will forward any traffic between ix0 and ix1 (in both directions).
These numbers would give you a better idea of where to look next
(e.g. VPP code improvements or system tuning such as NIC
interrupts, CPU binding, etc.).

I believe this issue is in the combined txrx interrupt filter. It is
causing a bunch of unnecessary tx re-arms.

Could you please elaborate on that?

TX completion is indeed the one thing that changed considerably with the
porting to iflib. And this could be a major contributor to the performance
drop.
My understanding is that TX interrupts are not really used anymore on
multi-gigabit NICs such as ix or ixl. Instead, "softirqs" are used, meaning
that a timer is used to perform TX completion. I don't know what the
motivations were for this design decision.
I had to decrease the timer period to 90us to ensure timely completion (see
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248652). However, the
timer period is currently not adaptive.

_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-net

Kevin Bowling

2021-05-25 06:16:48 UTC

I don't fully understand the issue, but in iflib_fast_intr_rxtx
https://cgit.freebsd.org/src/tree/sys/net/iflib.c#n1581 it seems like
we end up re-enabling interrupts per course instead of only handling
spurious cases or some low water threshold (which seems like it would
be tricky to do here). The idea is we want to pump interrupts by
disabling them in the msix_que handler, and then wait to re-enable
only when we have more work to do in the ift_task grouptask.

It was a lot easier to reason about this with separate TX and RX
interrupts. Doing the combined TXRX is definitely a win in terms of
reducing msi-x vector usage (which is important in a lot of FreeBSD
use cases), but it's tricky to understand.

My time has been sucked away due to work, so I haven't been looking at
this problem to the depth I want to. I'd be interested in discussing
it further with anyone that is interested in it.

Regards,
Kevin

Post by Vincenzo Maffione

Post by Marko Zec
On Mon, 17 May 2021 09:53:25 +0000

Post by Vincenzo Maffione
Hi,
Yes, you are not using emulated netmap mode.
In the test setup depicted here
https://github.com/ftk-ntq/vpp/wiki/VPP-throughput-using-netmap-
interfaces#test-setup
I think you should really try to replace VPP with the netmap
"bridge" application (tools/tools/netmap/bridge.c), and see what
numbers you get.
You would run the application this way
# bridge -i ix0 -i ix1
and this will forward any traffic between ix0 and ix1 (in both directions).
These numbers would give you a better idea of where to look next
(e.g. VPP code improvements or system tuning such as NIC
interrupts, CPU binding, etc.).

I believe this issue is in the combined txrx interrupt filter. It is causing a bunch of unnecessary tx re-arms.

Could you please elaborate on that?
TX completion is indeed the one thing that changed considerably with the porting to iflib. And this could be a major contributor to the performance drop.
My understanding is that TX interrupts are not really used anymore on multi-gigabit NICs such as ix or ixl. Instead, "softirqs" are used, meaning that a timer is used to perform TX completion. I don't know what the motivations were for this design decision.
I had to decrease the timer period to 90us to ensure timely completion (see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248652). However, the timer period is currently not adaptive.

_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-net

Kevin Bowling

2021-05-25 06:34:16 UTC

The one other thing I want to mention, what this means in effect is every
que ends up limited by EITR on ixgbe (around 30kps with the default
settings) whether itâs a TX or RX workload. This ends up working ok if you
have sufficient CPU but seems awkward. On the TX workload we should need a
magnitude less interrupts to do 10g. There was some work to adapt AIM to
this new combined handler but it is not properly tuned and Iâm not sure it
should consider TX at all.

Regards,
Kevin

Post by Kevin Bowling
I don't fully understand the issue, but in iflib_fast_intr_rxtx
https://cgit.freebsd.org/src/tree/sys/net/iflib.c#n1581 it seems like
we end up re-enabling interrupts per course instead of only handling
spurious cases or some low water threshold (which seems like it would
be tricky to do here). The idea is we want to pump interrupts by
disabling them in the msix_que handler, and then wait to re-enable
only when we have more work to do in the ift_task grouptask.
It was a lot easier to reason about this with separate TX and RX
interrupts. Doing the combined TXRX is definitely a win in terms of
reducing msi-x vector usage (which is important in a lot of FreeBSD
use cases), but it's tricky to understand.
My time has been sucked away due to work, so I haven't been looking at
this problem to the depth I want to. I'd be interested in discussing
it further with anyone that is interested in it.
Regards,
Kevin

Post by Vincenzo Maffione
Il giorno mar 18 mag 2021 alle ore 09:32 Kevin Bowling <

Post by Marko Zec
On Mon, 17 May 2021 09:53:25 +0000

Post by Vincenzo Maffione
Hi,
Yes, you are not using emulated netmap mode.
In the test setup depicted here
https://github.com/ftk-ntq/vpp/wiki/VPP-throughput-using-netmap-
interfaces#test-setup
I think you should really try to replace VPP with the netmap
"bridge" application (tools/tools/netmap/bridge.c), and see what
numbers you get.
You would run the application this way
# bridge -i ix0 -i ix1
and this will forward any traffic between ix0 and ix1 (in both directions).
These numbers would give you a better idea of where to look next
(e.g. VPP code improvements or system tuning such as NIC
interrupts, CPU binding, etc.).

I believe this issue is in the combined txrx interrupt filter. It is

causing a bunch of unnecessary tx re-arms.

Post by Vincenzo Maffione
Could you please elaborate on that?
TX completion is indeed the one thing that changed considerably with the

porting to iflib. And this could be a major contributor to the performance
drop.

Post by Vincenzo Maffione
My understanding is that TX interrupts are not really used anymore on

multi-gigabit NICs such as ix or ixl. Instead, "softirqs" are used, meaning
that a timer is used to perform TX completion. I don't know what the
motivations were for this design decision.

Post by Vincenzo Maffione
I had to decrease the timer period to 90us to ensure timely completion

(see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248652). However,
the timer period is currently not adaptive.

Post by Vincenzo Maffione

_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-net

Vincenzo Maffione

2021-05-18 20:17:27 UTC

+1

Thanks,
Vincenzo

Post by Marko Zec
On Mon, 17 May 2021 09:53:25 +0000

Post by Vincenzo Maffione
Hi,
Yes, you are not using emulated netmap mode.
In the test setup depicted here
https://github.com/ftk-ntq/vpp/wiki/VPP-throughput-using-netmap-
interfaces#test-setup
I think you should really try to replace VPP with the netmap
"bridge" application (tools/tools/netmap/bridge.c), and see what
numbers you get.
You would run the application this way
# bridge -i ix0 -i ix1
and this will forward any traffic between ix0 and ix1 (in both directions).
These numbers would give you a better idea of where to look next
(e.g. VPP code improvements or system tuning such as NIC
interrupts, CPU binding, etc.).

I assume you're on 13.0 where netmap throughput is lower compared to
11.x due to migration of most drivers to iflib (apparently increased
overhead) and different driver defaults. On 11.x I could move 10G line
rate from one ix to another at low CPU freqs, where on 13.x the CPU
must be set to max speed, and still can't do 14.88 Mpps.
#1 thing which changed: default # of packets per ring dropped down from
dev.ixl.0.iflib.override_nrxds=2048
dev.ixl.0.iflib.override_ntxds=2048
dev.ixl.1.iflib.override_nrxds=2048
dev.ixl.1.iflib.override_ntxds=2048
etc.
For me this increases the throughput of
bridge -i netmap:ixl0 -i netmap:ixl1
from 9.3 Mpps to 11.4 Mpps
#2: default interrupt moderation delays seem to be too long. Combined
with increasing the ring sizes, reducing dev.ixl.0.rx_itr from 62
(default) to 40 increases the throughput further from 11.4 to 14.5 Mpps
Hope this helps,
Marko

Francois ten Krooden

2021-05-20 05:10:26 UTC

Thanks will give these a shot.
Yes we are working on FreeBSD 13.0 as there is some of the memory management functionality which is required.
I suspected there might have been some impact with implementing iflib.

Post by Marko Zec
On Mon, 17 May 2021 09:53:25 +0000

Post by Vincenzo Maffione
Hi,
Yes, you are not using emulated netmap mode.
In the test setup depicted here
https://github.com/ftk-ntq/vpp/wiki/VPP-throughput-using-netmap-
interfaces#test-setup
I think you should really try to replace VPP with the netmap
"bridge" application (tools/tools/netmap/bridge.c), and see what
numbers you get.
You would run the application this way # bridge -i ix0 -i ix1 and
this will forward any traffic between ix0 and ix1 (in both
directions).
These numbers would give you a better idea of where to look next
(e.g. VPP code improvements or system tuning such as NIC interrupts,
CPU binding, etc.).

I assume you're on 13.0 where netmap throughput is lower compared to
11.x due to migration of most drivers to iflib (apparently increased
overhead) and different driver defaults. On 11.x I could move 10G line rate
from one ix to another at low CPU freqs, where on 13.x the CPU must be set
to max speed, and still can't do 14.88 Mpps.
#1 thing which changed: default # of packets per ring dropped down from
dev.ixl.0.iflib.override_nrxds=2048
dev.ixl.0.iflib.override_ntxds=2048
dev.ixl.1.iflib.override_nrxds=2048
dev.ixl.1.iflib.override_ntxds=2048
etc.
For me this increases the throughput of
bridge -i netmap:ixl0 -i netmap:ixl1
from 9.3 Mpps to 11.4 Mpps
#2: default interrupt moderation delays seem to be too long. Combined with
increasing the ring sizes, reducing dev.ixl.0.rx_itr from 62
(default) to 40 increases the throughput further from 11.4 to 14.5 Mpps
Hope this helps,
Marko

Important Notice:

This e-mail and its contents are subject to the Nanoteq (Pty) Ltd e-mail legal notice available at:
http://www.nanoteq.com/AboutUs/EmailDisclaimer.aspx

Francois ten Krooden

2021-05-13 13:04:28 UTC

On Thursday, 13 May 2021 15:02, Luigi Rizzo

On Thu, May 13, 2021 at 1:27 PM Francois ten Krooden