IPsec performace - netisr hits %100

Discussion:

Özkan KIRIK

2021-04-30 20:11:48 UTC

Hello,

I'm using FreeBSD stable/12 built world on 12 April 2021.
my setup is:
[freebsd host cc0] <--------> [cc1 - same freebsd, but jail]

without IPsec, I can achieve easily to 20Gbps. (test was run with different
source IPs using multiple iperf to scale across multiple queues)
My hardware is Xeon D-2146NT (8 core + SoC Qat), cc0 and cc1 is Chelsio
T62100-LP-CR.

But with IPsec, throughput is limited to 2Gbps (with ccr) and only one
netisr thread hits %100 cpu.
with aesni throughput is 1,4 Gbps
with QAT throughput is 1,6 Gbps (qat0 C62x, qat1 C62x)
with CCR throughput is 2,0 Gbps (t6nex0)
But always bottleneck is netisr.

Is there any way to workaround this netisr bottleneck ?
I tried to switch net.isr.dispatch to deferred and hybrid, but performance
drops a bit.

my configuration is below:

net.isr.numthreads: 4
net.isr.maxprot: 16
net.isr.defaultqlimit: 8192
net.isr.maxqlimit: 1000000
net.isr.bindthreads: 1
net.isr.maxthreads: 4
net.isr.dispatch: direct

net.inet.ipsec.async_crypto: 1

FreeBSD Host:
kldload ccr # Chelsio Crypto Accelerator
ifconfig lo1 create 172.16.68.1/24 up
ifconfig cc0 192.168.1.3/24 up
ifconfig ipsec0 create reqid 100
ifconfig ipsec0 inet tunnel 192.168.1.3 192.168.1.5
ifconfig ipsec0 inet 172.16.0.3/16 172.16.0.5
setkey -c << EOF
add 192.168.1.3 192.168.1.5 esp 10000 -m tunnel -u 100 -E aes-gcm-16
"VerySecureKey!!10000";
add 192.168.1.5 192.168.1.3 esp 10001 -m tunnel -u 100 -E aes-gcm-16
"VerySecureKey!!20000";
EOF
route add 172.16.70.0/24 172.16.0.5
iperf -s

FreeBSD Jail:
jail -c name=client persist vnet vnet.interface=cc1 host.hostname=client
jexec client sysctl net.inet.ipsec.async_crypto=1
jexec client ifconfig lo1 create 172.16.70.1/24 up
jexec client bash -c 'for i in $(seq 2 10); do ifconfig lo1 172.16.70.$i/32
alias; done'
jexec client ifconfig cc1 192.168.1.5/24 up
jexec client ifconfig ipsec0 create reqid 200
jexec client ifconfig ipsec0 inet tunnel 192.168.1.5 192.168.1.3
jexec client ifconfig ipsec0 inet 172.16.0.5/16 172.16.0.3
jexec client setkey -c
add 192.168.1.3 192.168.1.5 esp 10000 -m tunnel -u 200 -E aes-gcm-16
"VerySecureKey!!10000";
add 192.168.1.5 192.168.1.3 esp 10001 -m tunnel -u 200 -E aes-gcm-16
"VerySecureKey!!20000";
^D
jexec client route add 172.16.68.0/24 172.16.0.3

jexec client bash -c 'for i in $(seq 1 10); do (iperf -B 172.16.70.$i -c
172.16.68.1 -P 2 | grep SUM &); done'
---------------------------------

top -azSHj
PID JID USERNAME PRI NICE SIZE RES STATE C TIME WCPU
COMMAND
11 0 root -72 - 0B 1120K CPU2 2 0:26 100.00%
[intr{swi1: netisr 2}]
11 0 root -92 - 0B 1120K CPU4 4 0:36 72.55%
[intr{irq295: t6nex0:0a0}]
3 0 root -16 - 0B 16K CPU8 8 0:13 51.11%
[crypto returns 0]
11 0 root -92 - 0B 1120K WAIT 11 0:18 43.40%
[intr{irq297: t6nex0:0a2}]
14 0 root -16 - 0B 16K crypto 14 0:09 33.43%
[crypto returns 8]
11 0 root -92 - 0B 1120K WAIT 12 0:11 21.17%
[intr{irq307: t6nex0:1a2}]
9049 1 root 32 0 23M 4356K CPU14 14 0:00 7.50%
iperf -B 172.16.70.9 -c 172.16.68.1 -P 2{iperf}
9040 1 root 30 0 23M 4356K sbwait 8 0:00 6.92%
iperf -B 172.16.70.6 -c 172.16.68.1 -P 2{iperf}
9043 1 root 33 0 23M 4356K sbwait 8 0:00 6.90%
iperf -B 172.16.70.7 -c 172.16.68.1 -P 2{iperf}
9046 1 root 32 0 23M 4356K sbwait 11 0:00 6.72%
iperf -B 172.16.70.8 -c 172.16.68.1 -P 2{iperf}
9031 1 root 24 0 23M 4356K sbwait 6 0:00 6.40%
iperf -B 172.16.70.3 -c 172.16.68.1 -P 2{iperf}
9037 1 root 29 0 23M 4356K sbwait 11 0:00 6.05%
iperf -B 172.16.70.5 -c 172.16.68.1 -P 2{iperf}
9037 1 root 29 0 23M 4356K sbwait 6 0:00 5.82%
iperf -B 172.16.70.5 -c 172.16.68.1 -P 2{iperf}
9025 1 root 22 0 23M 4356K sbwait 14 0:00 5.71%
iperf -B 172.16.70.1 -c 172.16.68.1 -P 2{iperf}
9052 1 root 32 0 23M 4356K sbwait 10 0:00 5.24%
iperf -B 172.16.70.10 -c 172.16.68.1 -P 2{iperf}

If you need more information, i can provide.

Regards,
Özkan

Mark Johnston

2021-04-30 20:32:52 UTC

Permalink

Post by Ãzkan KIRIK
Hello,
I'm using FreeBSD stable/12 built world on 12 April 2021.
[freebsd host cc0] <--------> [cc1 - same freebsd, but jail]
without IPsec, I can achieve easily to 20Gbps. (test was run with different
source IPs using multiple iperf to scale across multiple queues)
My hardware is Xeon D-2146NT (8 core + SoC Qat), cc0 and cc1 is Chelsio
T62100-LP-CR.
But with IPsec, throughput is limited to 2Gbps (with ccr) and only one
netisr thread hits %100 cpu.
with aesni throughput is 1,4 Gbps
with QAT throughput is 1,6 Gbps (qat0 C62x, qat1 C62x)
with CCR throughput is 2,0 Gbps (t6nex0)
But always bottleneck is netisr.
Is there any way to workaround this netisr bottleneck ?
I tried to switch net.isr.dispatch to deferred and hybrid, but performance
drops a bit.

I can suggest a couple of things to try. First, we configure only one
netisr thread by default. You can create more by setting the
net.isr.maxthreads loader tunable. I believe netipsec selects a thread
using the SPI so adding more threads might not help much depending on
your configuration, but testing with e.g., maxthreads = ncpu could be
illuminating.

Second, netipsec unconditionally hands rx processing off to netisr
threads for some reason, that's why changing the dispatch policy doesn't
help. Maybe it's to help avoid running out of kernel stack space or to
somehow avoid packet reordering in some case that is not clear to me. I
tried a patch (see below) which eliminates this and it helped somewhat.
If anyone can provide an explanation for the current behaviour I'd
appreciate it.

Could you try both approaches and report back? It would also be
interesting to know how your results compare with 13.0, if possible.

commit 618ab87449d412a74bfee4932d84a6fc17afce6c
Author: Mark Johnston <***@FreeBSD.org>
Date: Thu Jan 7 11:29:14 2021 -0500

netipsec: Avoid deferred dispatch on the input path

diff --git a/sys/netipsec/ipsec_input.c b/sys/netipsec/ipsec_input.c
index 48acba68a1fe..98d0954c4c53 100644
--- a/sys/netipsec/ipsec_input.c
+++ b/sys/netipsec/ipsec_input.c
@@ -425,7 +425,7 @@ ipsec4_common_input_cb(struct mbuf *m, struct secasvar *sav, int skip,
error = ipsec_if_input(m, sav, af);
if (error == 0) {
NET_EPOCH_ENTER(et);
- error = netisr_queue_src(isr_prot, (uintptr_t)sav->spi, m);
+ error = netisr_dispatch_src(isr_prot, (uintptr_t)sav->spi, m);
NET_EPOCH_EXIT(et);
if (error) {
IPSEC_ISTAT(sproto, qfull);
@@ -624,7 +624,7 @@ ipsec6_common_input_cb(struct mbuf *m, struct secasvar *sav, int skip,
error = ipsec_if_input(m, sav, af);
if (error == 0) {
NET_EPOCH_ENTER(et);
- error = netisr_queue_src(isr_prot,
+ error = netisr_dispatch_src(isr_prot,
(uintptr_t)sav->spi, m);
NET_EPOCH_EXIT(et);
if (error) {

Özkan KIRIK

2021-04-30 20:40:43 UTC

Permalink

Thank you Mark,

I'm going to recompile kernel with this patch and share results.
After a few days, I can install stable/13 on same hardware and then I'll
repeat same tests.

Thanks again,
Cheers

Post by Ãzkan KIRIK

different

Post by Ãzkan KIRIK
source IPs using multiple iperf to scale across multiple queues)
My hardware is Xeon D-2146NT (8 core + SoC Qat), cc0 and cc1 is Chelsio
T62100-LP-CR.
But with IPsec, throughput is limited to 2Gbps (with ccr) and only one
netisr thread hits %100 cpu.
with aesni throughput is 1,4 Gbps
with QAT throughput is 1,6 Gbps (qat0 C62x, qat1 C62x)
with CCR throughput is 2,0 Gbps (t6nex0)
But always bottleneck is netisr.
Is there any way to workaround this netisr bottleneck ?
I tried to switch net.isr.dispatch to deferred and hybrid, but

performance

Post by Ãzkan KIRIK
drops a bit.

I can suggest a couple of things to try. First, we configure only one
netisr thread by default. You can create more by setting the
net.isr.maxthreads loader tunable. I believe netipsec selects a thread
using the SPI so adding more threads might not help much depending on
your configuration, but testing with e.g., maxthreads = ncpu could be
illuminating.
Second, netipsec unconditionally hands rx processing off to netisr
threads for some reason, that's why changing the dispatch policy doesn't
help. Maybe it's to help avoid running out of kernel stack space or to
somehow avoid packet reordering in some case that is not clear to me. I
tried a patch (see below) which eliminates this and it helped somewhat.
If anyone can provide an explanation for the current behaviour I'd
appreciate it.
Could you try both approaches and report back? It would also be
interesting to know how your results compare with 13.0, if possible.
commit 618ab87449d412a74bfee4932d84a6fc17afce6c
Date: Thu Jan 7 11:29:14 2021 -0500
netipsec: Avoid deferred dispatch on the input path
diff --git a/sys/netipsec/ipsec_input.c b/sys/netipsec/ipsec_input.c
index 48acba68a1fe..98d0954c4c53 100644
--- a/sys/netipsec/ipsec_input.c
+++ b/sys/netipsec/ipsec_input.c
@@ -425,7 +425,7 @@ ipsec4_common_input_cb(struct mbuf *m, struct secasvar *sav, int skip,
error = ipsec_if_input(m, sav, af);
if (error == 0) {
NET_EPOCH_ENTER(et);
- error = netisr_queue_src(isr_prot, (uintptr_t)sav->spi, m);
+ error = netisr_dispatch_src(isr_prot, (uintptr_t)sav->spi,
m);
NET_EPOCH_EXIT(et);
if (error) {
IPSEC_ISTAT(sproto, qfull);
@@ -624,7 +624,7 @@ ipsec6_common_input_cb(struct mbuf *m, struct secasvar *sav, int skip,
error = ipsec_if_input(m, sav, af);
if (error == 0) {
NET_EPOCH_ENTER(et);
- error = netisr_queue_src(isr_prot,
+ error = netisr_dispatch_src(isr_prot,
(uintptr_t)sav->spi, m);
NET_EPOCH_EXIT(et);
if (error) {

Özkan KIRIK

2021-04-30 21:31:57 UTC

Permalink

Hello again,

patch is applied, now netisr is not eating CPU. but performance drops
around 0.2Gbps according to previous kernel.

I tried also both net.isr.maxthreads=1 and net.isr.maxthreads=4 . results
are same

Results are:

- with CCR - 1.8Gbps
top:
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
14 root -16 - 0B 16K CPU5 5 1:38 100.00% [crypto
returns 8]
3 root -16 - 0B 16K CPU1 1 0:58 77.83% [crypto
returns 0]
11 root -92 - 0B 1120K WAIT 4 0:46 41.77%
[intr{irq295: t6nex0:0a0}]
11 root -92 - 0B 1120K WAIT 8 0:37 36.83%
[intr{irq297: t6nex0:0a2}]
11 root -92 - 0B 1120K WAIT 12 0:19 21.05%
[intr{irq307: t6nex0:1a2}]
5376 root 23 0 23M 4348K CPU10 10 0:00 3.77% iperf -B
172.16.70.10 -c 172.16.68.1 -P 2 -t 20{iperf}
5376 root 23 0 23M 4348K sbwait 3 0:00 3.72% iperf -B
172.16.70.10 -c 172.16.68.1 -P 2 -t 20{iperf}
5358 root 22 0 23M 4348K sbwait 12 0:00 3.72% iperf -B
172.16.70.4 -c 172.16.68.1 -P 2 -t 20{iperf}
...

- with aesni - 1.25Gbps
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
14 root -16 - 0B 16K CPU7 7 1:59 99.66% [crypto
returns 8]
3 root -16 - 0B 16K CPU3 3 1:10 46.09% [crypto
returns 0]
11 root -92 - 0B 1120K CPU14 14 0:45 38.65%
[intr{irq297: t6nex0:0a2}]
0 root -20 - 0B 2144K - 8 0:02 20.42%
[kernel{crypto_12}]
0 root -20 - 0B 2144K - 5 0:02 18.54%
[kernel{crypto_7}]
0 root -20 - 0B 2144K CPU4 4 0:02 18.28%
[kernel{crypto_10}]
0 root -20 - 0B 2144K - 9 0:01 17.87%
[kernel{crypto_5}]
0 root -20 - 0B 2144K - 11 0:02 17.66%
[kernel{crypto_14}]
0 root -20 - 0B 2144K CPU15 15 0:02 16.94%
[kernel{crypto_15}]
0 root -20 - 0B 2144K CPU2 2 0:00 16.91%
[kernel{crypto_4}]
11 root -92 - 0B 1120K WAIT 12 0:23 16.41%
[intr{irq307: t6nex0:1a2}]
0 root -20 - 0B 2144K - 6 0:02 16.28%
[kernel{crypto_13}]
0 root -20 - 0B 2144K CPU14 14 0:02 14.86%
[kernel{crypto_11}]
0 root -20 - 0B 2144K - 11 0:01 7.37%
[kernel{crypto_9}]
5761 root 22 0 23M 4348K sbwait 13 0:00 3.96% iperf -B
172.16.70.6 -c 172.16.68.1 -P 2 -t 20{iperf}
5761 root 22 0 23M 4348K sbwait 5 0:00 3.86% iperf -B
172.16.70.6 -c 172.16.68.1 -P 2 -t 20{iperf}
...

- with QAT - 1.46 Gbps
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
3 root -16 - 0B 16K RUN 0 1:23 93.46% [crypto
returns 0]
14 root -16 - 0B 16K crypto 2 2:10 35.31% [crypto
returns 8]
11 root -92 - 0B 1664K WAIT 8 0:50 21.83%
[intr{irq297: t6nex0:0a2}]
11 root -92 - 0B 1664K WAIT 12 0:25 9.72%
[intr{irq307: t6nex0:1a2}]
11 root -92 - 0B 1664K WAIT 8 0:00 4.12%
[intr{irq368: qat1}]
6103 root 21 0 23M 4348K sbwait 12 0:00 3.88% iperf -B
172.16.70.1 -c 172.16.68.1 -P 2 -t 20{iperf}
6124 root 22 0 23M 4348K sbwait 4 0:00 3.63% iperf -B
172.16.70.8 -c 172.16.68.1 -P 2 -t 20{iperf}

Post by Ãzkan KIRIK
Thank you Mark,
I'm going to recompile kernel with this patch and share results.
After a few days, I can install stable/13 on same hardware and then I'll
repeat same tests.
Thanks again,
Cheers

Post by Ãzkan KIRIK

different

performance

Post by Ãzkan KIRIK
drops a bit.

I can suggest a couple of things to try. First, we configure only one
netisr thread by default. You can create more by setting the
net.isr.maxthreads loader tunable. I believe netipsec selects a thread
using the SPI so adding more threads might not help much depending on
your configuration, but testing with e.g., maxthreads = ncpu could be
illuminating.
Second, netipsec unconditionally hands rx processing off to netisr
threads for some reason, that's why changing the dispatch policy doesn't
help. Maybe it's to help avoid running out of kernel stack space or to
somehow avoid packet reordering in some case that is not clear to me. I
tried a patch (see below) which eliminates this and it helped somewhat.
If anyone can provide an explanation for the current behaviour I'd
appreciate it.
Could you try both approaches and report back? It would also be
interesting to know how your results compare with 13.0, if possible.
commit 618ab87449d412a74bfee4932d84a6fc17afce6c
Date: Thu Jan 7 11:29:14 2021 -0500
netipsec: Avoid deferred dispatch on the input path
diff --git a/sys/netipsec/ipsec_input.c b/sys/netipsec/ipsec_input.c
index 48acba68a1fe..98d0954c4c53 100644
--- a/sys/netipsec/ipsec_input.c
+++ b/sys/netipsec/ipsec_input.c
@@ -425,7 +425,7 @@ ipsec4_common_input_cb(struct mbuf *m, struct
secasvar *sav, int skip,
error = ipsec_if_input(m, sav, af);
if (error == 0) {
NET_EPOCH_ENTER(et);
- error = netisr_queue_src(isr_prot, (uintptr_t)sav->spi, m);
+ error = netisr_dispatch_src(isr_prot,
(uintptr_t)sav->spi, m);
NET_EPOCH_EXIT(et);
if (error) {
IPSEC_ISTAT(sproto, qfull);
@@ -624,7 +624,7 @@ ipsec6_common_input_cb(struct mbuf *m, struct
secasvar *sav, int skip,
error = ipsec_if_input(m, sav, af);
if (error == 0) {
NET_EPOCH_ENTER(et);
- error = netisr_queue_src(isr_prot,
+ error = netisr_dispatch_src(isr_prot,
(uintptr_t)sav->spi, m);
NET_EPOCH_EXIT(et);
if (error) {

Mark Johnston

2021-04-30 21:45:28 UTC

Permalink

Post by Ãzkan KIRIK
Hello again,
patch is applied, now netisr is not eating CPU. but performance drops
around 0.2Gbps according to previous kernel.
I tried also both net.isr.maxthreads=1 and net.isr.maxthreads=4 . results
are same
- with CCR - 1.8Gbps
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
14 root -16 - 0B 16K CPU5 5 1:38 100.00% [crypto
returns 8]
3 root -16 - 0B 16K CPU1 1 0:58 77.83% [crypto
returns 0]

Could you also try this patch? It won't help with aesni.

diff --git a/sys/opencrypto/crypto.c b/sys/opencrypto/crypto.c
index dfd22662e87f..bc23056e86ad 100644
--- a/sys/opencrypto/crypto.c
+++ b/sys/opencrypto/crypto.c
@@ -94,6 +94,7 @@ struct crypto_session {
void *softc;
uint32_t hid;
uint32_t capabilities;
+ uint64_t id;
};

SDT_PROVIDER_DEFINE(opencrypto);
@@ -572,6 +573,7 @@ crypto_select_driver(const struct cryptoini *cri, int flags)
int
crypto_newsession(crypto_session_t *cses, struct cryptoini *cri, int crid)
{
+ static uint64_t sessid = 0;
crypto_session_t res;
void *softc_mem;
struct cryptocap *cap;
@@ -616,6 +618,7 @@ crypto_newsession(crypto_session_t *cses, struct cryptoini *cri, int crid)
softc_mem = malloc(softc_size, M_CRYPTO_DATA, M_WAITOK | M_ZERO);
res = uma_zalloc(cryptoses_zone, M_WAITOK | M_ZERO);
res->softc = softc_mem;
+ res->id = atomic_fetchadd_64(&sessid, 1);

CRYPTO_DRIVER_LOCK();
cap = crypto_checkdriver(hid);
@@ -1016,7 +1019,7 @@ crypto_dispatch(struct cryptop *crp)
binuptime(&crp->crp_tstamp);
#endif

- crp->crp_retw_id = ((uintptr_t)crp->crp_session) % crypto_workers_num;
+ crp->crp_retw_id = crp->crp_session->id % crypto_workers_num;

if (CRYPTOP_ASYNC(crp)) {
if (crp->crp_flags & CRYPTO_F_ASYNC_KEEPORDER) {

Özkan KIRIK

2021-04-30 22:26:40 UTC

Permalink

Here is fresh results;
a bit performance gain achieved. But strangely QAT is better now.

- with CCR - 2.14 Gbps
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
15 root -16 - 0B 16K CPU7 7 0:11 100.00% [crypto
returns 9]
11 root -92 - 0B 1120K CPU4 4 0:10 98.74%
[intr{irq295: t6nex0:0a0}]
14 root -16 - 0B 16K CPU11 11 0:09 95.32% [crypto
returns 8]
11 root -92 - 0B 1120K CPU8 8 0:07 63.49%
[intr{irq297: t6nex0:0a2}]
11 root -92 - 0B 1120K WAIT 12 0:05 42.73%
[intr{irq307: t6nex0:1a2}]
16 root -16 - 0B 16K CPU15 15 0:04 33.82% [crypto
returns 10]
13 root -16 - 0B 16K RUN 3 0:02 21.78% [crypto
returns 7]
4317 root 22 0 23M 4348K sbwait 14 0:01 7.24% iperf -B
172.16.70.6 -c 172.16.68.1 -P 2 -t 20{iperf}
4329 root 23 0 23M 4348K sbwait 14 0:01 7.18% iperf -B
172.16.70.10 -c 172.16.68.1 -P 2 -t 20{iperf}
...
11 root -92 - 0B 1120K WAIT 3 0:00 0.31%
[intr{irq294: t6nex0:evt}]
0 root -92 - 0B 2144K - 14 0:00 0.30%
[kernel{t6nex0 tq0}]
...

- with QAT - 2.63 Gbps
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
8 root -16 - 0B 16K CPU7 7 0:31 92.05% [crypto
returns 5]
7 root -16 - 0B 16K CPU0 0 0:29 86.03% [crypto
returns 4]
9 root -16 - 0B 16K RUN 15 0:15 43.32% [crypto
returns 6]
11 root -92 - 0B 1664K RUN 4 0:15 43.03%
[intr{irq297: t6nex0:0a2}]
6 root -16 - 0B 16K crypto 4 0:09 26.01% [crypto
returns 3]
11 root -92 - 0B 1664K WAIT 12 0:07 19.88%
[intr{irq307: t6nex0:1a2}]
11 root -92 - 0B 1664K WAIT 8 0:02 7.22%
[intr{irq368: qat1}]
4313 root 22 0 23M 4348K sbwait 2 0:01 6.51% iperf -B
172.16.70.8 -c 172.16.68.1 -P 2 -t 20{iperf}
4316 root 22 0 23M 4348K sbwait 3 0:01 6.18% iperf -B
172.16.70.9 -c 172.16.68.1 -P 2 -t 20{iperf}
...
11 root -92 - 0B 1664K WAIT 12 0:02 4.52%
[intr{irq355: qat0}]
...
11 root -92 - 0B 1664K WAIT 12 0:01 1.89%
[intr{irq372: qat1}]
11 root -92 - 0B 1664K WAIT 0 0:00 1.12%
[intr{irq360: qat1}]
11 root -92 - 0B 1664K WAIT 2 0:00 1.11%
[intr{irq362: qat1}]
11 root -92 - 0B 1664K WAIT 14 0:00 1.10%
[intr{irq374: qat1}]
11 root -92 - 0B 1664K WAIT 4 0:00 1.10%
[intr{irq364: qat1}]
11 root -92 - 0B 1664K WAIT 10 0:00 1.10%
[intr{irq370: qat1}]
11 root -92 - 0B 1664K WAIT 7 0:00 1.09%
[intr{irq367: qat1}]
11 root -92 - 0B 1664K WAIT 11 0:00 1.07%
[intr{irq371: qat1}]
11 root -92 - 0B 1664K WAIT 5 0:00 1.04%
[intr{irq365: qat1}]
11 root -92 - 0B 1664K WAIT 15 0:00 1.04%
[intr{irq375: qat1}]
11 root -92 - 0B 1664K WAIT 1 0:00 1.04%
[intr{irq361: qat1}]
11 root -92 - 0B 1664K WAIT 6 0:00 1.03%
[intr{irq366: qat1}]

Post by Ãzkan KIRIK

COMMAND

Post by Ãzkan KIRIK
14 root -16 - 0B 16K CPU5 5 1:38 100.00%

[crypto

Post by Ãzkan KIRIK
returns 8]
3 root -16 - 0B 16K CPU1 1 0:58 77.83%

[crypto

Post by Ãzkan KIRIK
returns 0]

Could you also try this patch? It won't help with aesni.
diff --git a/sys/opencrypto/crypto.c b/sys/opencrypto/crypto.c
index dfd22662e87f..bc23056e86ad 100644
--- a/sys/opencrypto/crypto.c
+++ b/sys/opencrypto/crypto.c
@@ -94,6 +94,7 @@ struct crypto_session {
void *softc;
uint32_t hid;
uint32_t capabilities;
+ uint64_t id;
};
SDT_PROVIDER_DEFINE(opencrypto);
@@ -572,6 +573,7 @@ crypto_select_driver(const struct cryptoini *cri, int flags)
int
crypto_newsession(crypto_session_t *cses, struct cryptoini *cri, int crid)
{
+ static uint64_t sessid = 0;
crypto_session_t res;
void *softc_mem;
struct cryptocap *cap;
@@ -616,6 +618,7 @@ crypto_newsession(crypto_session_t *cses, struct
cryptoini *cri, int crid)
softc_mem = malloc(softc_size, M_CRYPTO_DATA, M_WAITOK | M_ZERO);
res = uma_zalloc(cryptoses_zone, M_WAITOK | M_ZERO);
res->softc = softc_mem;
+ res->id = atomic_fetchadd_64(&sessid, 1);
CRYPTO_DRIVER_LOCK();
cap = crypto_checkdriver(hid);
@@ -1016,7 +1019,7 @@ crypto_dispatch(struct cryptop *crp)
binuptime(&crp->crp_tstamp);
#endif
- crp->crp_retw_id = ((uintptr_t)crp->crp_session) %
crypto_workers_num;
+ crp->crp_retw_id = crp->crp_session->id % crypto_workers_num;
if (CRYPTOP_ASYNC(crp)) {
if (crp->crp_flags & CRYPTO_F_ASYNC_KEEPORDER) {

Özkan KIRIK

2021-05-01 12:39:30 UTC

Permalink

I've pulled the latest stable/13 branch and make buildworld.

Same configuration and same usage, ping works between two sides, iperf can
connect to server but no data transferred. throughput is around 10Kbps.
I tried both with and without netipsec/ipsec_input.c patch, but no change.
There is something wrong with ipsec (or ipsec in jail) in stable/13.
iperf between host and jail works without ipsec.

I saw some errors in truss output show as a brief:
write(3,"@\^A\0x\0\0\0\^A\0\0\^S\M^I\0\0"...,131072) ERR#35 'Resource
temporarily unavailable'
recvfrom(3,0x8018048c0,131072,0,0x0,0x0) ERR#35 'Resource temporarily
unavailable'
_umtx_op(0x80026e008,UMTX_OP_WAIT_UINT_PRIVATE,0x0,0x18,0x7fffdfffdec8)
ERR#60 'Operation timed out'

Full output of truss output is below:

***@host # jexec client bash
[***@client /]# truss iperf -B 172.16.70.1 -c 172.16.68.1 2>&1 | grep -v
clock_nanosleep
mmap(0x0,135168,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =
34362150912 (0x80024d000)
mprotect(0x80024a000,4096,PROT_READ) = 0 (0x0)
issetugid() = 0 (0x0)
sigfastblock(0x1,0x80024c510) = 0 (0x0)
open("/etc/libmap.conf",O_RDONLY|O_CLOEXEC,011136710) = 3 (0x3)
fstat(3,{ mode=-rw-r--r-- ,inode=482,size=47,blksize=4096 }) = 0 (0x0)
read(3,"# $FreeBSD$\nincludedir /usr/loc"...,47) = 47 (0x2f)
close(3) = 0 (0x0)
open("/usr/local/etc/libmap.d",O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC,0165)
ERR#2 'No such file or directory'
open("/var/run/ld-elf.so.hints",O_RDONLY|O_CLOEXEC,010015464) = 3 (0x3)
read(3,"Ehnt\^A\0\0\0\M^@\0\0\0\M-U\0\0"...,128) = 128 (0x80)
fstat(3,{ mode=-r--r--r-- ,inode=187,size=341,blksize=4096 }) = 0 (0x0)
pread(3,"/lib:/usr/lib:/usr/local/lib:/us"...,213,0x80) = 213 (0xd5)
close(3) = 0 (0x0)
open("/lib/libthr.so.3",O_RDONLY|O_CLOEXEC|O_VERIFY,00) = 3 (0x3)
fstat(3,{ mode=-r--r--r-- ,inode=186701,size=125952,blksize=32768 }) = 0
(0x0)
mmap(0x0,4096,PROT_READ,MAP_PRIVATE|MAP_PREFAULT_READ,3,0x0) = 34362286080
(0x80026e000)
mmap(0x0,184320,PROT_NONE,MAP_GUARD,-1,0x0) = 34362290176 (0x80026f000)
mmap(0x80026f000,53248,PROT_READ,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x0)
= 34362290176 (0x80026f000)
mmap(0x80027c000,73728,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0xc000)
= 34362343424 (0x80027c000)
mmap(0x80028e000,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x1d000)
= 34362417152 (0x80028e000)
mmap(0x80028f000,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x1d000)
= 34362421248 (0x80028f000)
mmap(0x800291000,45056,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANON,-1,0x0)
= 34362429440 (0x800291000)
munmap(0x80026e000,4096) = 0 (0x0)
close(3) = 0 (0x0)
open("/lib/librt.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,066000) ERR#2 'No such
file or directory'
open("/usr/lib/librt.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,066000) = 3 (0x3)
fstat(3,{ mode=-r--r--r-- ,inode=411606,size=22824,blksize=32768 }) = 0
(0x0)
mmap(0x0,4096,PROT_READ,MAP_PRIVATE|MAP_PREFAULT_READ,3,0x0) = 34362286080
(0x80026e000)
mmap(0x0,36864,PROT_NONE,MAP_GUARD,-1,0x0) = 34362474496 (0x80029c000)
mmap(0x80029c000,12288,PROT_READ,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x0)
= 34362474496 (0x80029c000)
mmap(0x80029f000,12288,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x2000)
= 34362486784 (0x80029f000)
mmap(0x8002a2000,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x4000)
= 34362499072 (0x8002a2000)
mmap(0x8002a3000,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x4000)
= 34362503168 (0x8002a3000)
mmap(0x8002a4000,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANON,-1,0x0)
= 34362507264 (0x8002a4000)
munmap(0x80026e000,4096) = 0 (0x0)
close(3) = 0 (0x0)
open("/lib/libc++.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,00) ERR#2 'No such file
or directory'
open("/usr/lib/libc++.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,00) = 3 (0x3)
fstat(3,{ mode=-r--r--r-- ,inode=411469,size=824056,blksize=32768 }) = 0
(0x0)
mmap(0x0,4096,PROT_READ,MAP_PRIVATE|MAP_PREFAULT_READ,3,0x0) = 34362286080
(0x80026e000)
mmap(0x0,860160,PROT_NONE,MAP_GUARD,-1,0x0) = 34362511360 (0x8002a5000)
mmap(0x8002a5000,376832,PROT_READ,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x0)
= 34362511360 (0x8002a5000)
mmap(0x800301000,425984,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x5b000)
= 34362888192 (0x800301000)
mmap(0x800369000,24576,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0xc2000)
= 34363314176 (0x800369000)
mmap(0x80036f000,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0xc7000)
= 34363338752 (0x80036f000)
mmap(0x800371000,24576,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANON,-1,0x0)
= 34363346944 (0x800371000)
munmap(0x80026e000,4096) = 0 (0x0)
close(3) = 0 (0x0)
open("/lib/libcxxrt.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,041400) = 3 (0x3)
fstat(3,{ mode=-r--r--r-- ,inode=186668,size=113688,blksize=32768 }) = 0
(0x0)
mmap(0x0,4096,PROT_READ,MAP_PRIVATE|MAP_PREFAULT_READ,3,0x0) = 34362286080
(0x80026e000)
mmap(0x0,143360,PROT_NONE,MAP_GUARD,-1,0x0) = 34363371520 (0x800377000)
mmap(0x800377000,53248,PROT_READ,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x0)
= 34363371520 (0x800377000)
mmap(0x800384000,61440,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0xc000)
= 34363424768 (0x800384000)
mmap(0x800393000,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x1a000)
= 34363486208 (0x800393000)
mmap(0x800394000,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x1a000)
= 34363490304 (0x800394000)
mmap(0x800396000,16384,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANON,-1,0x0)
= 34363498496 (0x800396000)
munmap(0x80026e000,4096) = 0 (0x0)
close(3) = 0 (0x0)
open("/lib/libm.so.5",O_RDONLY|O_CLOEXEC|O_VERIFY,00) = 3 (0x3)
fstat(3,{ mode=-r--r--r-- ,inode=186671,size=214192,blksize=32768 }) = 0
(0x0)
mmap(0x0,4096,PROT_READ,MAP_PRIVATE|MAP_PREFAULT_READ,3,0x0) = 34362286080
(0x80026e000)
mmap(0x0,225280,PROT_NONE,MAP_GUARD,-1,0x0) = 34363514880 (0x80039a000)
mmap(0x80039a000,73728,PROT_READ,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x0)
= 34363514880 (0x80039a000)
mmap(0x8003ac000,143360,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x11000)
= 34363588608 (0x8003ac000)
mmap(0x8003cf000,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x33000)
= 34363731968 (0x8003cf000)
mmap(0x8003d0000,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x33000)
= 34363736064 (0x8003d0000)
munmap(0x80026e000,4096) = 0 (0x0)
close(3) = 0 (0x0)
open("/lib/libgcc_s.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,020643400) = 3 (0x3)
fstat(3,{ mode=-r--r--r-- ,inode=186687,size=93448,blksize=32768 }) = 0
(0x0)
mmap(0x0,4096,PROT_READ,MAP_PRIVATE|MAP_PREFAULT_READ,3,0x0) = 34362286080
(0x80026e000)
mmap(0x0,106496,PROT_NONE,MAP_GUARD,-1,0x0) = 34363740160 (0x8003d1000)
mmap(0x8003d1000,36864,PROT_READ,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x0)
= 34363740160 (0x8003d1000)
mmap(0x8003da000,57344,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x8000)
= 34363777024 (0x8003da000)
mmap(0x8003e8000,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x15000)
= 34363834368 (0x8003e8000)
mmap(0x8003ea000,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x16000)
= 34363842560 (0x8003ea000)
munmap(0x80026e000,4096) = 0 (0x0)
close(3) = 0 (0x0)
open("/lib/libc.so.7",O_RDONLY|O_CLOEXEC|O_VERIFY,00) = 3 (0x3)
fstat(3,{ mode=-r--r--r-- ,inode=186667,size=1904424,blksize=32768 }) = 0
(0x0)
mmap(0x0,4096,PROT_READ,MAP_PRIVATE|MAP_PREFAULT_READ,3,0x0) = 34362286080
(0x80026e000)
mmap(0x0,4182016,PROT_NONE,MAP_GUARD,-1,0x0) = 34363846656 (0x8003eb000)
mmap(0x8003eb000,520192,PROT_READ,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x0)
= 34363846656 (0x8003eb000)
mmap(0x80046a000,1331200,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x7e000)
= 34364366848 (0x80046a000)
mmap(0x8005af000,36864,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x1c2000)
= 34365698048 (0x8005af000)
mmap(0x8005b8000,28672,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x1ca000)
= 34365734912 (0x8005b8000)
mmap(0x8005bf000,2265088,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANON,-1,0x0)
= 34365763584 (0x8005bf000)
munmap(0x80026e000,4096) = 0 (0x0)
close(3) = 0 (0x0)
mprotect(0x80028e000,4096,PROT_READ) = 0 (0x0)
mprotect(0x8002a2000,4096,PROT_READ) = 0 (0x0)
mprotect(0x800369000,20480,PROT_READ) = 0 (0x0)
mprotect(0x800393000,4096,PROT_READ) = 0 (0x0)
mprotect(0x8003cf000,4096,PROT_READ) = 0 (0x0)
mprotect(0x8003e8000,4096,PROT_READ) = 0 (0x0)
mprotect(0x8005af000,36864,PROT_READ) = 0 (0x0)
mprotect(0x8005af000,36864,PROT_READ|PROT_WRITE) = 0 (0x0)
mprotect(0x8005af000,36864,PROT_READ) = 0 (0x0)
readlink("/etc/malloc.conf",0x7fffffffd5c0,1024) ERR#2 'No such file or
directory'
issetugid() = 0 (0x0)
__sysctl("vm.overcommit",2,0x7fffffffbb54,0x7fffffffbb48,0x0,0) = 0 (0x0)
mmap(0x0,2097152,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(21),-1,0x0)
= 34368126976 (0x800800000)
mmap(0x0,2097152,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0)
= 34370224128 (0x800a00000)
mmap(0x0,4194304,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(21),-1,0x0)
= 34372321280 (0x800c00000)
__sysctl("kern.usrstack",2,0x8002903c8,0x7fffffffd878,0x0,0) = 0 (0x0)
getrlimit(RLIMIT_STACK,{ cur=536870912,max=536870912 }) = 0 (0x0)
thr_self(0x800a12000) = 0 (0x0)
mmap(0x7fffdfffe000,4096,PROT_NONE,MAP_ANON,-1,0x0) = 140736951476224
(0x7fffdfffe000)
rtprio_thread(RTP_LOOKUP,102215,0x7fffffffd838) = 0 (0x0)
sigaction(SIGTHR,{ 0x8002884d0 SA_SIGINFO ss_t },0x0) = 0 (0x0)
sigprocmask(SIG_UNBLOCK,{ },0x0) = 0 (0x0)
_umtx_op(0x7fffffffd830,UMTX_OP_WAKE,0x1,0x0,0x0) = 0 (0x0)
mprotect(0x0,0,PROT_NONE) = 0 (0x0)
getpid() = 7030 (0x1b76)
getpid() = 7030 (0x1b76)
sigprocmask(SIG_BLOCK,{
SIGHUP|SIGINT|SIGQUIT|SIGILL|SIGTRAP|SIGABRT|SIGEMT|SIGFPE|SIGKILL|SIGBUS|SIGSEGV|SIGSYS|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2
},{ }) = 0 (0x0)
sigfastblock(0x3,0x0) = 0 (0x0)
sigprocmask(SIG_SETMASK,{ },0x0) = 0 (0x0)
sigfastblock(0x1,0x800a12038) = 0 (0x0)
getcontext(0x7fffffffcbd0) = 0 (0x0)
sysarch(AMD64_GET_XFPUSTATE,0x7fffffffcb98) = 0 (0x0)
mmap(0x0,135168,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =
34376515584 (0x801000000)
mprotect(0x22c000,4096,PROT_READ) = 0 (0x0)
sigprocmask(SIG_SETMASK,{
SIGHUP|SIGINT|SIGQUIT|SIGILL|SIGTRAP|SIGABRT|SIGEMT|SIGFPE|SIGKILL|SIGBUS|SIGSEGV|SIGSYS|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2
},{ }) = 0 (0x0)
sigaction(SIGTERM,{ 0x800288380 SA_RESTART|SA_SIGINFO ss_t },{ SIG_DFL 0x0
ss_t }) = 0 (0x0)
sigprocmask(SIG_SETMASK,{ },0x0) = 0 (0x0)
sigprocmask(SIG_SETMASK,{
SIGHUP|SIGINT|SIGQUIT|SIGILL|SIGTRAP|SIGABRT|SIGEMT|SIGFPE|SIGKILL|SIGBUS|SIGSEGV|SIGSYS|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2
},{ }) = 0 (0x0)
sigaction(SIGINT,{ 0x800288380 SA_RESTART|SA_SIGINFO ss_t },{ SIG_DFL 0x0
ss_t }) = 0 (0x0)
sigprocmask(SIG_SETMASK,{ },0x0) = 0 (0x0)
sigprocmask(SIG_SETMASK,{
SIGHUP|SIGINT|SIGQUIT|SIGILL|SIGTRAP|SIGABRT|SIGEMT|SIGFPE|SIGKILL|SIGBUS|SIGSEGV|SIGSYS|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2
},{ }) = 0 (0x0)
sigaction(SIGALRM,{ 0x800288380 SA_SIGINFO ss_t },{ SIG_DFL 0x0 ss_t }) = 0
(0x0)
sigprocmask(SIG_SETMASK,{ },0x0) = 0 (0x0)
sigprocmask(SIG_SETMASK,{
SIGHUP|SIGINT|SIGQUIT|SIGILL|SIGTRAP|SIGABRT|SIGEMT|SIGFPE|SIGKILL|SIGBUS|SIGSEGV|SIGSYS|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2
},{ }) = 0 (0x0)
sigaction(SIGPIPE,{ SIG_IGN SA_RESTART ss_t },{ SIG_DFL 0x0 ss_t }) = 0
(0x0)
sigprocmask(SIG_SETMASK,{ },0x0) = 0 (0x0)
mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =
34362286080 (0x80026e000)
mmap(0x0,147456,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =
34376650752 (0x801021000)
mmap(0x7fffdfdfd000,2101248,PROT_READ|PROT_WRITE,MAP_STACK,-1,0x0) =
140736949374976 (0x7fffdfdfd000)
mprotect(0x7fffdfdfd000,4096,PROT_NONE) = 0 (0x0)
thr_new(0x7fffffffe880,0x68) = 0 (0x0)
<new thread 102901>
sigfastblock(0x1,0x800a12738) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_UNLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_WRLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_UNLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_RDLOCK,0x0,0x0,0x0) = 0 (0x0)
mmap(0x7fffdfbfc000,2101248,PROT_READ|PROT_WRITE,MAP_STACK,-1,0x0) =
140736947273728 (0x7fffdfbfc000)
mprotect(0x7fffdfbfc000,4096,PROT_NONE) = 0 (0x0)
thr_new(0x7fffffffe880,0x68) = 0 (0x0)
<new thread 102902>
sigfastblock(0x1,0x800a12e38) = 0 (0x0)
mmap(0x0,2097152,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(21),-1,0x0)
= 34378612736 (0x801200000)
mmap(0x0,2097152,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(21),-1,0x0)
= 34380709888 (0x801400000)
mmap(0x0,2097152,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0)
= 34382807040 (0x801600000)
mmap(0x0,2097152,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0)
= 34384904192 (0x801800000)
socket(PF_INET,SOCK_STREAM,0) = 3 (0x3)
bind(3,{ AF_INET 172.16.70.1:0 },16) = 0 (0x0)
connect(3,{ AF_INET 172.16.68.1:5001 },16) = 0 (0x0)
setsockopt(3,SOL_SOCKET,SO_SNDTIMEO,0x7fffdfdfced0,16) = 0 (0x0)
getsockname(3,{ AF_INET 172.16.70.1:34901 },0x800a1b200) = 0 (0x0)
getpeername(3,{ AF_INET 172.16.68.1:5001 },0x800a1b178) = 0 (0x0)
getsockname(3,{ AF_INET 172.16.70.1:34901 },0x7fffdfdfce3c) = 0 (0x0)
__sysctl("net.routetable.0.0.5.0",6,0x0,0x7fffdfdfcdc8,0x0,0) = 0 (0x0)
__sysctl("net.routetable.0.0.5.0",6,0x801826000,0x7fffdfdfcdc8,0x0,0) = 0
(0x0)
_umtx_op(0x80026e008,UMTX_OP_WAKE_PRIVATE,0x7fffffff,0x0,0x0) = 0 (0x0)
_umtx_op(0x80026e008,UMTX_OP_WAIT_UINT_PRIVATE,0x0,0x18,0x7fffdfffdec8) = 0
(0x0)
getpid() = 7030 (0x1b76)
getsockopt(3,IPPROTO_TCP,TCP_MAXSEG,0x7fffdfdfceb4,0x7fffdfdfceb0) = 0 (0x0)
getsockopt(3,SOL_SOCKET,SO_SNDBUF,0x7fffdfdfceb4,0x7fffdfdfceb0) = 0 (0x0)
fstat(1,{ mode=p--------- ,inode=12766,size=0,blksize=4096 }) = 0 (0x0)
mmap(0x0,2621440,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0)
= 34387001344 (0x801a00000)
sendto(3,"@\^A\0x\0\0\0\^A\0\0\^S\M^I\0\0"...,60,MSG_DONTWAIT,NULL,0) = 60
(0x3c)
getsockopt(3,SOL_SOCKET,SO_SNDBUF,0x7fffdfffde64,0x7fffdfffde60) = 0 (0x0)
setsockopt(3,SOL_SOCKET,SO_SNDTIMEO,0x7fffdfdfced0,16) = 0 (0x0)
------------------------------------------------------------
Client connecting to 172.16.68.1, TCP port 5001
TCP window size: 33.2 KByte (default)
------------------------------------------------------------
write(1,"--------------------------------"...,208) = 208 (0xd0)
[ 1] local 172.16.70.1 port 34901 connected with 172.16.68.1 port 5001
write(1,"[ 1] local 172.16.70.1 port 349"...,72) = 72 (0x48)
write(3,"@\^A\0x\0\0\0\^A\0\0\^S\M^I\0\0"...,131072) = 34000 (0x84d0)
write(3,"@\^A\0x\0\0\0\^A\0\0\^S\M^I\0\0"...,131072) ERR#35 'Resource
temporarily unavailable'
setitimer(0,{ 0.000000, 0.000000 },0x0) = 0 (0x0)
shutdown(3,SHUT_WR) = 0 (0x0)
setsockopt(3,SOL_SOCKET,SO_RCVTIMEO,0x7fffdfdfce60,16) = 0 (0x0)
recvfrom(3,"\0\0\0\^B\0\0\0\^\\0\0\0\0\0\^B"...,131072,0,NULL,0x0) = 28
(0x1c)
recvfrom(3,0x8018048c0,131072,0,0x0,0x0) ERR#35 'Resource temporarily
unavailable'
_umtx_op(0x80026e020,UMTX_OP_WAKE_PRIVATE,0x7fffffff,0x0,0x0) = 0 (0x0)
_umtx_op(0x80026e020,UMTX_OP_WAIT_UINT_PRIVATE,0x0,0x18,0x7fffdfdfcd38) = 0
(0x0)
[ ID] Interval Transfer Bandwidth
[ 1] 0.00-20.05 sec 33.3 KBytes 13.6 Kbits/sec
write(1,"[ ID] Interval Transfer "...,94) = 94 (0x5e)
_umtx_op(0x80026e020,UMTX_OP_WAKE_PRIVATE,0x7fffffff,0x0,0x0) = 0 (0x0)
_umtx_op(0x80026e020,UMTX_OP_WAIT_UINT_PRIVATE,0x0,0x18,0x7fffdfdfcd38) = 0
(0x0)
close(3) = 0 (0x0)
_umtx_op(0x800a130c0,UMTX_OP_NWAKE_PRIVATE,0x1,0x0,0x0) = 0 (0x0)
_umtx_op(0x8002908a8,UMTX_OP_WAIT_UINT_PRIVATE,0x0,0x0,0x0) = 0 (0x0)
__sysctl("kern.ostype",2,0x80024be6a,0x7fffdfdfb720,0x0,0) = 0 (0x0)
__sysctl("kern.hostname",2,0x80024bf6a,0x7fffdfdfb720,0x0,0) = 0 (0x0)
__sysctl("kern.osrelease",2,0x80024c06a,0x7fffdfdfb720,0x0,0) = 0 (0x0)
__sysctl("kern.version",2,0x80024c16a,0x7fffdfdfb720,0x0,0) = 0 (0x0)
__sysctl("hw.machine",2,0x80024c26a,0x7fffdfdfb720,0x0,0) = 0 (0x0)
open("/lib/libgcc_s.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,00) = 3 (0x3)
fstat(3,{ mode=-r--r--r-- ,inode=186687,size=93448,blksize=32768 }) = 0
(0x0)
close(3) = 0 (0x0)
madvise(0x801a00000,1839104,MADV_FREE) = 0 (0x0)
madvise(0x80184f000,4096,MADV_FREE) = 0 (0x0)
madvise(0x801851000,36864,MADV_FREE) = 0 (0x0)
madvise(0x801801000,274432,MADV_FREE) = 0 (0x0)
<thread 102902 exited>
_umtx_op(0x80026e008,UMTX_OP_WAIT_UINT_PRIVATE,0x0,0x18,0x7fffdfffdec8)
ERR#60 'Operation timed out'
_umtx_op(0x800a129c0,UMTX_OP_NWAKE_PRIVATE,0x1,0x0,0x0) = 0 (0x0)
_umtx_op(0x8002908a8,UMTX_OP_WAIT_UINT_PRIVATE,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_UNLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_WRLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_UNLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_RDLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_UNLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_WRLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_RDLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_UNLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_WRLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_UNLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_RDLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_WRLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_UNLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_RDLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_UNLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_WRLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_RDLOCK,0x0,0x0,0x0) = 0 (0x0)
<thread 102901 exited>
exit(0x0)
process exit, rval = 0

Post by Ãzkan KIRIK
Here is fresh results;
a bit performance gain achieved. But strangely QAT is better now.
- with CCR - 2.14 Gbps
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
15 root -16 - 0B 16K CPU7 7 0:11 100.00% [crypto
returns 9]
11 root -92 - 0B 1120K CPU4 4 0:10 98.74%
[intr{irq295: t6nex0:0a0}]
14 root -16 - 0B 16K CPU11 11 0:09 95.32% [crypto
returns 8]
11 root -92 - 0B 1120K CPU8 8 0:07 63.49%
[intr{irq297: t6nex0:0a2}]
11 root -92 - 0B 1120K WAIT 12 0:05 42.73%
[intr{irq307: t6nex0:1a2}]
16 root -16 - 0B 16K CPU15 15 0:04 33.82% [crypto
returns 10]
13 root -16 - 0B 16K RUN 3 0:02 21.78% [crypto
returns 7]
4317 root 22 0 23M 4348K sbwait 14 0:01 7.24% iperf
-B 172.16.70.6 -c 172.16.68.1 -P 2 -t 20{iperf}
4329 root 23 0 23M 4348K sbwait 14 0:01 7.18% iperf
-B 172.16.70.10 -c 172.16.68.1 -P 2 -t 20{iperf}
...
11 root -92 - 0B 1120K WAIT 3 0:00 0.31%
[intr{irq294: t6nex0:evt}]
0 root -92 - 0B 2144K - 14 0:00 0.30%
[kernel{t6nex0 tq0}]
...
- with QAT - 2.63 Gbps
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
8 root -16 - 0B 16K CPU7 7 0:31 92.05% [crypto
returns 5]
7 root -16 - 0B 16K CPU0 0 0:29 86.03% [crypto
returns 4]
9 root -16 - 0B 16K RUN 15 0:15 43.32% [crypto
returns 6]
11 root -92 - 0B 1664K RUN 4 0:15 43.03%
[intr{irq297: t6nex0:0a2}]
6 root -16 - 0B 16K crypto 4 0:09 26.01% [crypto
returns 3]
11 root -92 - 0B 1664K WAIT 12 0:07 19.88%
[intr{irq307: t6nex0:1a2}]
11 root -92 - 0B 1664K WAIT 8 0:02 7.22%
[intr{irq368: qat1}]
4313 root 22 0 23M 4348K sbwait 2 0:01 6.51% iperf
-B 172.16.70.8 -c 172.16.68.1 -P 2 -t 20{iperf}
4316 root 22 0 23M 4348K sbwait 3 0:01 6.18% iperf
-B 172.16.70.9 -c 172.16.68.1 -P 2 -t 20{iperf}
...
11 root -92 - 0B 1664K WAIT 12 0:02 4.52%
[intr{irq355: qat0}]
...
11 root -92 - 0B 1664K WAIT 12 0:01 1.89%
[intr{irq372: qat1}]
11 root -92 - 0B 1664K WAIT 0 0:00 1.12%
[intr{irq360: qat1}]
11 root -92 - 0B 1664K WAIT 2 0:00 1.11%
[intr{irq362: qat1}]
11 root -92 - 0B 1664K WAIT 14 0:00 1.10%
[intr{irq374: qat1}]
11 root -92 - 0B 1664K WAIT 4 0:00 1.10%
[intr{irq364: qat1}]
11 root -92 - 0B 1664K WAIT 10 0:00 1.10%
[intr{irq370: qat1}]
11 root -92 - 0B 1664K WAIT 7 0:00 1.09%
[intr{irq367: qat1}]
11 root -92 - 0B 1664K WAIT 11 0:00 1.07%
[intr{irq371: qat1}]
11 root -92 - 0B 1664K WAIT 5 0:00 1.04%
[intr{irq365: qat1}]
11 root -92 - 0B 1664K WAIT 15 0:00 1.04%
[intr{irq375: qat1}]
11 root -92 - 0B 1664K WAIT 1 0:00 1.04%
[intr{irq361: qat1}]
11 root -92 - 0B 1664K WAIT 6 0:00 1.03%
[intr{irq366: qat1}]

Post by Mark Johnston

results

Post by Ãzkan KIRIK
are same
- with CCR - 1.8Gbps
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU

COMMAND

Post by Ãzkan KIRIK
14 root -16 - 0B 16K CPU5 5 1:38 100.00%

[crypto

Post by Ãzkan KIRIK
returns 8]
3 root -16 - 0B 16K CPU1 1 0:58 77.83%

[crypto

Post by Ãzkan KIRIK
returns 0]

Could you also try this patch? It won't help with aesni.
diff --git a/sys/opencrypto/crypto.c b/sys/opencrypto/crypto.c
index dfd22662e87f..bc23056e86ad 100644
--- a/sys/opencrypto/crypto.c
+++ b/sys/opencrypto/crypto.c
@@ -94,6 +94,7 @@ struct crypto_session {
void *softc;
uint32_t hid;
uint32_t capabilities;
+ uint64_t id;
};
SDT_PROVIDER_DEFINE(opencrypto);
@@ -572,6 +573,7 @@ crypto_select_driver(const struct cryptoini *cri, int flags)
int
crypto_newsession(crypto_session_t *cses, struct cryptoini *cri, int crid)
{
+ static uint64_t sessid = 0;
crypto_session_t res;
void *softc_mem;
struct cryptocap *cap;
@@ -616,6 +618,7 @@ crypto_newsession(crypto_session_t *cses, struct
cryptoini *cri, int crid)
softc_mem = malloc(softc_size, M_CRYPTO_DATA, M_WAITOK | M_ZERO);
res = uma_zalloc(cryptoses_zone, M_WAITOK | M_ZERO);
res->softc = softc_mem;
+ res->id = atomic_fetchadd_64(&sessid, 1);
CRYPTO_DRIVER_LOCK();
cap = crypto_checkdriver(hid);
@@ -1016,7 +1019,7 @@ crypto_dispatch(struct cryptop *crp)
binuptime(&crp->crp_tstamp);
#endif
- crp->crp_retw_id = ((uintptr_t)crp->crp_session) %
crypto_workers_num;
+ crp->crp_retw_id = crp->crp_session->id % crypto_workers_num;
if (CRYPTOP_ASYNC(crp)) {
if (crp->crp_flags & CRYPTO_F_ASYNC_KEEPORDER) {

Özkan KIRIK

2021-05-01 13:30:59 UTC

Permalink

This bug is related to CCR. @Navdeep Parhar <***@freebsd.org> , @John Baldwin
<***@freebsd.org> if you are interested to fix this bug related with CCR, I
can test if you provide patches. Test environment is explained in my first
email on this thread.

@Mark Johnston <***@freebsd.org> Now again on stable/13,
- with aesni, without netipsec/ipsec_input.c patch - 1.44Gbps - single
netisr thread eats %100 cpu
- with qat, without netipsec/ipsec_input.c patch - 1.88Gbps - single netisr
thread eats %100 cpu

- with aesni, with netipsec/ipsec_input.c patch - 1.33Gbps
PID JID USERNAME PRI NICE SIZE RES STATE C TIME WCPU
COMMAND
7 0 root -16 - 0B 16K CPU4 4 0:39 98.62%
[crypto returns 3]
6 0 root -16 - 0B 16K CPU5 5 0:32 84.14%
[crypto returns 2]
11 0 root -92 - 0B 1152K CPU8 8 0:28 72.17%
[intr{irq97: t6nex0:0a2}]
11 0 root -92 - 0B 1152K CPU3 3 0:20 51.22%
[intr{irq107: t6nex0:1a2}]
8 0 root -16 - 0B 16K crypto 7 0:12 30.68%
[crypto returns 4]
0 0 root -20 - 0B 13M RUN 7 0:07 21.52%
[kernel{crypto_2}]
0 0 root -20 - 0B 13M CPU12 12 0:06 20.53%
[kernel{crypto_15}]
0 0 root -20 - 0B 13M CPU9 9 0:05 19.97%
[kernel{crypto_0}]
0 0 root -20 - 0B 13M - 0 0:05 19.86%
[kernel{crypto_14}]
5 0 root -16 - 0B 16K CPU2 2 0:08 19.75%
[crypto returns 1]
0 0 root -20 - 0B 13M - 6 0:05 19.56%
[kernel{crypto_9}]
0 0 root -20 - 0B 13M - 7 0:07 18.74%
[kernel{crypto_3}]
0 0 root -20 - 0B 13M - 13 0:07 18.68%
[kernel{crypto_5}]
0 0 root -20 - 0B 13M CPU1 1 0:07 18.12%
[kernel{crypto_11}]
0 0 root -20 - 0B 13M - 2 0:07 17.46%
[kernel{crypto_8}]
0 0 root -20 - 0B 13M CPU10 10 0:05 17.31%
[kernel{crypto_1}]
0 0 root -20 - 0B 13M - 4 0:06 17.08%
[kernel{crypto_7}]
0 0 root -20 - 0B 13M - 14 0:07 16.07%
[kernel{crypto_12}]
0 0 root -20 - 0B 13M - 11 0:07 15.39%
[kernel{crypto_6}]
0 0 root -20 - 0B 13M - 6 0:07 12.09%
[kernel{crypto_4}]

- with qat, with netipsec/ipsec_input.c patch - 2.85Gbps -
PID JID USERNAME PRI NICE SIZE RES STATE C TIME WCPU
COMMAND
20 0 root -16 - 0B 16K CPU6 6 0:18 89.69%
[crypto returns 13]
19 0 root -16 - 0B 16K CPU4 4 0:17 80.25%
[crypto returns 12]
11 0 root -92 - 0B 1696K CPU1 1 0:52 43.90%
[intr{irq97: t6nex0:0a2}]
21 0 root -16 - 0B 16K crypto 0 0:09 43.43%
[crypto returns 14]
18 0 root -16 - 0B 16K crypto 2 0:05 24.94%
[crypto returns 11]
11 0 root -92 - 0B 1696K WAIT 12 0:35 20.18%
[intr{irq107: t6nex0:1a2}]
11 0 root -92 - 0B 1696K WAIT 8 0:02 7.44%
[intr{irq168: qat1}]
4202 1 root 23 0 32M 4480K sbwait 14 0:01 5.57%
iperf -B 172.16.70.5 -c 172.16.68.1 -P 2 -t 30{iperf}
4217 1 root 22 0 32M 4480K sbwait 5 0:01 5.52%
iperf -B 172.16.70.10 -c 172.16.68.1 -P 2 -t 30{iperf}
...
11 0 root -92 - 0B 1696K WAIT 12 0:01 4.39%
[intr{irq155: qat0}]
4182 0 root 21 0 105M 40M sbwait 4 0:01 4.04%
iperf -s{iperf}
4182 0 root 21 0 105M 40M sbwait 5 0:01 3.99%
iperf -s{iperf}
...
11 0 root -92 - 0B 1696K WAIT 12 0:00 1.84%
[intr{irq172: qat1}]
11 0 root -92 - 0B 1696K WAIT 15 0:00 1.07%
[intr{irq175: qat1}]
11 0 root -92 - 0B 1696K WAIT 4 0:00 1.04%
[intr{irq164: qat1}]
11 0 root -92 - 0B 1696K CPU6 6 0:00 1.02%
[intr{irq166: qat1}]
11 0 root -92 - 0B 1696K WAIT 14 0:00 1.00%
[intr{irq174: qat1}]
11 0 root -92 - 0B 1696K WAIT 10 0:00 0.98%
[intr{irq170: qat1}]
11 0 root -92 - 0B 1696K WAIT 5 0:00 0.98%
[intr{irq165: qat1}]
11 0 root -92 - 0B 1696K WAIT 3 0:00 0.97%
[intr{irq163: qat1}]
11 0 root -92 - 0B 1696K WAIT 7 0:00 0.95%
[intr{irq167: qat1}]
11 0 root -92 - 0B 1696K WAIT 2 0:00 0.93%
[intr{irq162: qat1}]

stable/13 results are better then stable/12 but not enough fast. There is
something makes bottleneck for IPsec.

Post by Ãzkan KIRIK
I've pulled the latest stable/13 branch and make buildworld.
Same configuration and same usage, ping works between two sides, iperf can
connect to server but no data transferred. throughput is around 10Kbps.
I tried both with and without netipsec/ipsec_input.c patch, but no change.
There is something wrong with ipsec (or ipsec in jail) in stable/13.
iperf between host and jail works without ipsec.
temporarily unavailable'
recvfrom(3,0x8018048c0,131072,0,0x0,0x0) ERR#35 'Resource temporarily
unavailable'
_umtx_op(0x80026e008,UMTX_OP_WAIT_UINT_PRIVATE,0x0,0x18,0x7fffdfffdec8)
ERR#60 'Operation timed out'
clock_nanosleep
mmap(0x0,135168,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =
34362150912 (0x80024d000)
mprotect(0x80024a000,4096,PROT_READ) = 0 (0x0)
issetugid() = 0 (0x0)
sigfastblock(0x1,0x80024c510) = 0 (0x0)
open("/etc/libmap.conf",O_RDONLY|O_CLOEXEC,011136710) = 3 (0x3)
fstat(3,{ mode=-rw-r--r-- ,inode=482,size=47,blksize=4096 }) = 0 (0x0)
read(3,"# $FreeBSD$\nincludedir /usr/loc"...,47) = 47 (0x2f)
close(3) = 0 (0x0)
open("/usr/local/etc/libmap.d",O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC,0165)
ERR#2 'No such file or directory'
open("/var/run/ld-elf.so.hints",O_RDONLY|O_CLOEXEC,010015464) = 3 (0x3)
fstat(3,{ mode=-r--r--r-- ,inode=187,size=341,blksize=4096 }) = 0 (0x0)
pread(3,"/lib:/usr/lib:/usr/local/lib:/us"...,213,0x80) = 213 (0xd5)
close(3) = 0 (0x0)
open("/lib/libthr.so.3",O_RDONLY|O_CLOEXEC|O_VERIFY,00) = 3 (0x3)
fstat(3,{ mode=-r--r--r-- ,inode=186701,size=125952,blksize=32768 }) = 0
(0x0)
mmap(0x0,4096,PROT_READ,MAP_PRIVATE|MAP_PREFAULT_READ,3,0x0) = 34362286080
(0x80026e000)
mmap(0x0,184320,PROT_NONE,MAP_GUARD,-1,0x0) = 34362290176 (0x80026f000)
mmap(0x80026f000,53248,PROT_READ,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x0)
= 34362290176 (0x80026f000)
mmap(0x80027c000,73728,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0xc000)
= 34362343424 (0x80027c000)
mmap(0x80028e000,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x1d000)
= 34362417152 (0x80028e000)
mmap(0x80028f000,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x1d000)
= 34362421248 (0x80028f000)
mmap(0x800291000,45056,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANON,-1,0x0)
= 34362429440 (0x800291000)
munmap(0x80026e000,4096) = 0 (0x0)
close(3) = 0 (0x0)
open("/lib/librt.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,066000) ERR#2 'No such
file or directory'
open("/usr/lib/librt.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,066000) = 3 (0x3)
fstat(3,{ mode=-r--r--r-- ,inode=411606,size=22824,blksize=32768 }) = 0
(0x0)
mmap(0x0,4096,PROT_READ,MAP_PRIVATE|MAP_PREFAULT_READ,3,0x0) = 34362286080
(0x80026e000)
mmap(0x0,36864,PROT_NONE,MAP_GUARD,-1,0x0) = 34362474496 (0x80029c000)
mmap(0x80029c000,12288,PROT_READ,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x0)
= 34362474496 (0x80029c000)
mmap(0x80029f000,12288,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x2000)
= 34362486784 (0x80029f000)
mmap(0x8002a2000,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x4000)
= 34362499072 (0x8002a2000)
mmap(0x8002a3000,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x4000)
= 34362503168 (0x8002a3000)
mmap(0x8002a4000,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANON,-1,0x0)
= 34362507264 (0x8002a4000)
munmap(0x80026e000,4096) = 0 (0x0)
close(3) = 0 (0x0)
open("/lib/libc++.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,00) ERR#2 'No such
file or directory'
open("/usr/lib/libc++.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,00) = 3 (0x3)
fstat(3,{ mode=-r--r--r-- ,inode=411469,size=824056,blksize=32768 }) = 0
(0x0)
mmap(0x0,4096,PROT_READ,MAP_PRIVATE|MAP_PREFAULT_READ,3,0x0) = 34362286080
(0x80026e000)
mmap(0x0,860160,PROT_NONE,MAP_GUARD,-1,0x0) = 34362511360 (0x8002a5000)
mmap(0x8002a5000,376832,PROT_READ,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x0)
= 34362511360 (0x8002a5000)
mmap(0x800301000,425984,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x5b000)
= 34362888192 (0x800301000)
mmap(0x800369000,24576,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0xc2000)
= 34363314176 (0x800369000)
mmap(0x80036f000,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0xc7000)
= 34363338752 (0x80036f000)
mmap(0x800371000,24576,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANON,-1,0x0)
= 34363346944 (0x800371000)
munmap(0x80026e000,4096) = 0 (0x0)
close(3) = 0 (0x0)
open("/lib/libcxxrt.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,041400) = 3 (0x3)
fstat(3,{ mode=-r--r--r-- ,inode=186668,size=113688,blksize=32768 }) = 0
(0x0)
mmap(0x0,4096,PROT_READ,MAP_PRIVATE|MAP_PREFAULT_READ,3,0x0) = 34362286080
(0x80026e000)
mmap(0x0,143360,PROT_NONE,MAP_GUARD,-1,0x0) = 34363371520 (0x800377000)
mmap(0x800377000,53248,PROT_READ,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x0)
= 34363371520 (0x800377000)
mmap(0x800384000,61440,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0xc000)
= 34363424768 (0x800384000)
mmap(0x800393000,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x1a000)
= 34363486208 (0x800393000)
mmap(0x800394000,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x1a000)
= 34363490304 (0x800394000)
mmap(0x800396000,16384,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANON,-1,0x0)
= 34363498496 (0x800396000)
munmap(0x80026e000,4096) = 0 (0x0)
close(3) = 0 (0x0)
open("/lib/libm.so.5",O_RDONLY|O_CLOEXEC|O_VERIFY,00) = 3 (0x3)
fstat(3,{ mode=-r--r--r-- ,inode=186671,size=214192,blksize=32768 }) = 0
(0x0)
mmap(0x0,4096,PROT_READ,MAP_PRIVATE|MAP_PREFAULT_READ,3,0x0) = 34362286080
(0x80026e000)
mmap(0x0,225280,PROT_NONE,MAP_GUARD,-1,0x0) = 34363514880 (0x80039a000)
mmap(0x80039a000,73728,PROT_READ,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x0)
= 34363514880 (0x80039a000)
mmap(0x8003ac000,143360,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x11000)
= 34363588608 (0x8003ac000)
mmap(0x8003cf000,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x33000)
= 34363731968 (0x8003cf000)
mmap(0x8003d0000,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x33000)
= 34363736064 (0x8003d0000)
munmap(0x80026e000,4096) = 0 (0x0)
close(3) = 0 (0x0)
open("/lib/libgcc_s.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,020643400) = 3 (0x3)
fstat(3,{ mode=-r--r--r-- ,inode=186687,size=93448,blksize=32768 }) = 0
(0x0)
mmap(0x0,4096,PROT_READ,MAP_PRIVATE|MAP_PREFAULT_READ,3,0x0) = 34362286080
(0x80026e000)
mmap(0x0,106496,PROT_NONE,MAP_GUARD,-1,0x0) = 34363740160 (0x8003d1000)
mmap(0x8003d1000,36864,PROT_READ,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x0)
= 34363740160 (0x8003d1000)
mmap(0x8003da000,57344,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x8000)
= 34363777024 (0x8003da000)
mmap(0x8003e8000,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x15000)
= 34363834368 (0x8003e8000)
mmap(0x8003ea000,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x16000)
= 34363842560 (0x8003ea000)
munmap(0x80026e000,4096) = 0 (0x0)
close(3) = 0 (0x0)
open("/lib/libc.so.7",O_RDONLY|O_CLOEXEC|O_VERIFY,00) = 3 (0x3)
fstat(3,{ mode=-r--r--r-- ,inode=186667,size=1904424,blksize=32768 }) = 0
(0x0)
mmap(0x0,4096,PROT_READ,MAP_PRIVATE|MAP_PREFAULT_READ,3,0x0) = 34362286080
(0x80026e000)
mmap(0x0,4182016,PROT_NONE,MAP_GUARD,-1,0x0) = 34363846656 (0x8003eb000)
mmap(0x8003eb000,520192,PROT_READ,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x0)
= 34363846656 (0x8003eb000)
mmap(0x80046a000,1331200,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x7e000)
= 34364366848 (0x80046a000)
mmap(0x8005af000,36864,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x1c2000)
= 34365698048 (0x8005af000)
mmap(0x8005b8000,28672,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x1ca000)
= 34365734912 (0x8005b8000)
mmap(0x8005bf000,2265088,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANON,-1,0x0)
= 34365763584 (0x8005bf000)
munmap(0x80026e000,4096) = 0 (0x0)
close(3) = 0 (0x0)
mprotect(0x80028e000,4096,PROT_READ) = 0 (0x0)
mprotect(0x8002a2000,4096,PROT_READ) = 0 (0x0)
mprotect(0x800369000,20480,PROT_READ) = 0 (0x0)
mprotect(0x800393000,4096,PROT_READ) = 0 (0x0)
mprotect(0x8003cf000,4096,PROT_READ) = 0 (0x0)
mprotect(0x8003e8000,4096,PROT_READ) = 0 (0x0)
mprotect(0x8005af000,36864,PROT_READ) = 0 (0x0)
mprotect(0x8005af000,36864,PROT_READ|PROT_WRITE) = 0 (0x0)
mprotect(0x8005af000,36864,PROT_READ) = 0 (0x0)
readlink("/etc/malloc.conf",0x7fffffffd5c0,1024) ERR#2 'No such file or
directory'
issetugid() = 0 (0x0)
__sysctl("vm.overcommit",2,0x7fffffffbb54,0x7fffffffbb48,0x0,0) = 0 (0x0)
mmap(0x0,2097152,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(21),-1,0x0)
= 34368126976 (0x800800000)
mmap(0x0,2097152,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0)
= 34370224128 (0x800a00000)
mmap(0x0,4194304,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(21),-1,0x0)
= 34372321280 (0x800c00000)
__sysctl("kern.usrstack",2,0x8002903c8,0x7fffffffd878,0x0,0) = 0 (0x0)
getrlimit(RLIMIT_STACK,{ cur=536870912,max=536870912 }) = 0 (0x0)
thr_self(0x800a12000) = 0 (0x0)
mmap(0x7fffdfffe000,4096,PROT_NONE,MAP_ANON,-1,0x0) = 140736951476224
(0x7fffdfffe000)
rtprio_thread(RTP_LOOKUP,102215,0x7fffffffd838) = 0 (0x0)
sigaction(SIGTHR,{ 0x8002884d0 SA_SIGINFO ss_t },0x0) = 0 (0x0)
sigprocmask(SIG_UNBLOCK,{ },0x0) = 0 (0x0)
_umtx_op(0x7fffffffd830,UMTX_OP_WAKE,0x1,0x0,0x0) = 0 (0x0)
mprotect(0x0,0,PROT_NONE) = 0 (0x0)
getpid() = 7030 (0x1b76)
getpid() = 7030 (0x1b76)
sigprocmask(SIG_BLOCK,{
SIGHUP|SIGINT|SIGQUIT|SIGILL|SIGTRAP|SIGABRT|SIGEMT|SIGFPE|SIGKILL|SIGBUS|SIGSEGV|SIGSYS|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2
},{ }) = 0 (0x0)
sigfastblock(0x3,0x0) = 0 (0x0)
sigprocmask(SIG_SETMASK,{ },0x0) = 0 (0x0)
sigfastblock(0x1,0x800a12038) = 0 (0x0)
getcontext(0x7fffffffcbd0) = 0 (0x0)
sysarch(AMD64_GET_XFPUSTATE,0x7fffffffcb98) = 0 (0x0)
mmap(0x0,135168,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =
34376515584 (0x801000000)
mprotect(0x22c000,4096,PROT_READ) = 0 (0x0)
sigprocmask(SIG_SETMASK,{
SIGHUP|SIGINT|SIGQUIT|SIGILL|SIGTRAP|SIGABRT|SIGEMT|SIGFPE|SIGKILL|SIGBUS|SIGSEGV|SIGSYS|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2
},{ }) = 0 (0x0)
sigaction(SIGTERM,{ 0x800288380 SA_RESTART|SA_SIGINFO ss_t },{ SIG_DFL 0x0
ss_t }) = 0 (0x0)
sigprocmask(SIG_SETMASK,{ },0x0) = 0 (0x0)
sigprocmask(SIG_SETMASK,{
SIGHUP|SIGINT|SIGQUIT|SIGILL|SIGTRAP|SIGABRT|SIGEMT|SIGFPE|SIGKILL|SIGBUS|SIGSEGV|SIGSYS|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2
},{ }) = 0 (0x0)
sigaction(SIGINT,{ 0x800288380 SA_RESTART|SA_SIGINFO ss_t },{ SIG_DFL 0x0
ss_t }) = 0 (0x0)
sigprocmask(SIG_SETMASK,{ },0x0) = 0 (0x0)
sigprocmask(SIG_SETMASK,{
SIGHUP|SIGINT|SIGQUIT|SIGILL|SIGTRAP|SIGABRT|SIGEMT|SIGFPE|SIGKILL|SIGBUS|SIGSEGV|SIGSYS|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2
},{ }) = 0 (0x0)
sigaction(SIGALRM,{ 0x800288380 SA_SIGINFO ss_t },{ SIG_DFL 0x0 ss_t }) =
0 (0x0)
sigprocmask(SIG_SETMASK,{ },0x0) = 0 (0x0)
sigprocmask(SIG_SETMASK,{
SIGHUP|SIGINT|SIGQUIT|SIGILL|SIGTRAP|SIGABRT|SIGEMT|SIGFPE|SIGKILL|SIGBUS|SIGSEGV|SIGSYS|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2
},{ }) = 0 (0x0)
sigaction(SIGPIPE,{ SIG_IGN SA_RESTART ss_t },{ SIG_DFL 0x0 ss_t }) = 0
(0x0)
sigprocmask(SIG_SETMASK,{ },0x0) = 0 (0x0)
mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =
34362286080 (0x80026e000)
mmap(0x0,147456,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =
34376650752 (0x801021000)
mmap(0x7fffdfdfd000,2101248,PROT_READ|PROT_WRITE,MAP_STACK,-1,0x0) =
140736949374976 (0x7fffdfdfd000)
mprotect(0x7fffdfdfd000,4096,PROT_NONE) = 0 (0x0)
thr_new(0x7fffffffe880,0x68) = 0 (0x0)
<new thread 102901>
sigfastblock(0x1,0x800a12738) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_UNLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_WRLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_UNLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_RDLOCK,0x0,0x0,0x0) = 0 (0x0)
mmap(0x7fffdfbfc000,2101248,PROT_READ|PROT_WRITE,MAP_STACK,-1,0x0) =
140736947273728 (0x7fffdfbfc000)
mprotect(0x7fffdfbfc000,4096,PROT_NONE) = 0 (0x0)
thr_new(0x7fffffffe880,0x68) = 0 (0x0)
<new thread 102902>
sigfastblock(0x1,0x800a12e38) = 0 (0x0)
mmap(0x0,2097152,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(21),-1,0x0)
= 34378612736 (0x801200000)
mmap(0x0,2097152,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(21),-1,0x0)
= 34380709888 (0x801400000)
mmap(0x0,2097152,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0)
= 34382807040 (0x801600000)
mmap(0x0,2097152,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0)
= 34384904192 (0x801800000)
socket(PF_INET,SOCK_STREAM,0) = 3 (0x3)
bind(3,{ AF_INET 172.16.70.1:0 },16) = 0 (0x0)
connect(3,{ AF_INET 172.16.68.1:5001 },16) = 0 (0x0)
setsockopt(3,SOL_SOCKET,SO_SNDTIMEO,0x7fffdfdfced0,16) = 0 (0x0)
getsockname(3,{ AF_INET 172.16.70.1:34901 },0x800a1b200) = 0 (0x0)
getpeername(3,{ AF_INET 172.16.68.1:5001 },0x800a1b178) = 0 (0x0)
getsockname(3,{ AF_INET 172.16.70.1:34901 },0x7fffdfdfce3c) = 0 (0x0)
__sysctl("net.routetable.0.0.5.0",6,0x0,0x7fffdfdfcdc8,0x0,0) = 0 (0x0)
__sysctl("net.routetable.0.0.5.0",6,0x801826000,0x7fffdfdfcdc8,0x0,0) = 0
(0x0)
_umtx_op(0x80026e008,UMTX_OP_WAKE_PRIVATE,0x7fffffff,0x0,0x0) = 0 (0x0)
_umtx_op(0x80026e008,UMTX_OP_WAIT_UINT_PRIVATE,0x0,0x18,0x7fffdfffdec8) =
0 (0x0)
getpid() = 7030 (0x1b76)
getsockopt(3,IPPROTO_TCP,TCP_MAXSEG,0x7fffdfdfceb4,0x7fffdfdfceb0) = 0 (0x0)
getsockopt(3,SOL_SOCKET,SO_SNDBUF,0x7fffdfdfceb4,0x7fffdfdfceb0) = 0 (0x0)
fstat(1,{ mode=p--------- ,inode=12766,size=0,blksize=4096 }) = 0 (0x0)
mmap(0x0,2621440,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0)
= 34387001344 (0x801a00000)
(0x3c)
getsockopt(3,SOL_SOCKET,SO_SNDBUF,0x7fffdfffde64,0x7fffdfffde60) = 0 (0x0)
setsockopt(3,SOL_SOCKET,SO_SNDTIMEO,0x7fffdfdfced0,16) = 0 (0x0)
------------------------------------------------------------
Client connecting to 172.16.68.1, TCP port 5001
TCP window size: 33.2 KByte (default)
------------------------------------------------------------
write(1,"--------------------------------"...,208) = 208 (0xd0)
[ 1] local 172.16.70.1 port 34901 connected with 172.16.68.1 port 5001
write(1,"[ 1] local 172.16.70.1 port 349"...,72) = 72 (0x48)
temporarily unavailable'
setitimer(0,{ 0.000000, 0.000000 },0x0) = 0 (0x0)
shutdown(3,SHUT_WR) = 0 (0x0)
setsockopt(3,SOL_SOCKET,SO_RCVTIMEO,0x7fffdfdfce60,16) = 0 (0x0)
recvfrom(3,"\0\0\0\^B\0\0\0\^\\0\0\0\0\0\^B"...,131072,0,NULL,0x0) = 28
(0x1c)
recvfrom(3,0x8018048c0,131072,0,0x0,0x0) ERR#35 'Resource temporarily
unavailable'
_umtx_op(0x80026e020,UMTX_OP_WAKE_PRIVATE,0x7fffffff,0x0,0x0) = 0 (0x0)
_umtx_op(0x80026e020,UMTX_OP_WAIT_UINT_PRIVATE,0x0,0x18,0x7fffdfdfcd38) =
0 (0x0)
[ ID] Interval Transfer Bandwidth
[ 1] 0.00-20.05 sec 33.3 KBytes 13.6 Kbits/sec
write(1,"[ ID] Interval Transfer "...,94) = 94 (0x5e)
_umtx_op(0x80026e020,UMTX_OP_WAKE_PRIVATE,0x7fffffff,0x0,0x0) = 0 (0x0)
_umtx_op(0x80026e020,UMTX_OP_WAIT_UINT_PRIVATE,0x0,0x18,0x7fffdfdfcd38) =
0 (0x0)
close(3) = 0 (0x0)
_umtx_op(0x800a130c0,UMTX_OP_NWAKE_PRIVATE,0x1,0x0,0x0) = 0 (0x0)
_umtx_op(0x8002908a8,UMTX_OP_WAIT_UINT_PRIVATE,0x0,0x0,0x0) = 0 (0x0)
__sysctl("kern.ostype",2,0x80024be6a,0x7fffdfdfb720,0x0,0) = 0 (0x0)
__sysctl("kern.hostname",2,0x80024bf6a,0x7fffdfdfb720,0x0,0) = 0 (0x0)
__sysctl("kern.osrelease",2,0x80024c06a,0x7fffdfdfb720,0x0,0) = 0 (0x0)
__sysctl("kern.version",2,0x80024c16a,0x7fffdfdfb720,0x0,0) = 0 (0x0)
__sysctl("hw.machine",2,0x80024c26a,0x7fffdfdfb720,0x0,0) = 0 (0x0)
open("/lib/libgcc_s.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,00) = 3 (0x3)
fstat(3,{ mode=-r--r--r-- ,inode=186687,size=93448,blksize=32768 }) = 0
(0x0)
close(3) = 0 (0x0)
madvise(0x801a00000,1839104,MADV_FREE) = 0 (0x0)
madvise(0x80184f000,4096,MADV_FREE) = 0 (0x0)
madvise(0x801851000,36864,MADV_FREE) = 0 (0x0)
madvise(0x801801000,274432,MADV_FREE) = 0 (0x0)
<thread 102902 exited>
_umtx_op(0x80026e008,UMTX_OP_WAIT_UINT_PRIVATE,0x0,0x18,0x7fffdfffdec8)
ERR#60 'Operation timed out'
_umtx_op(0x800a129c0,UMTX_OP_NWAKE_PRIVATE,0x1,0x0,0x0) = 0 (0x0)
_umtx_op(0x8002908a8,UMTX_OP_WAIT_UINT_PRIVATE,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_UNLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_WRLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_UNLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_RDLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_UNLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_WRLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_RDLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_UNLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_WRLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_UNLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_RDLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_WRLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_UNLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_RDLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_UNLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_WRLOCK,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800290d80,UMTX_OP_RW_RDLOCK,0x0,0x0,0x0) = 0 (0x0)
<thread 102901 exited>
exit(0x0)
process exit, rval = 0

Post by Ãzkan KIRIK
Here is fresh results;
a bit performance gain achieved. But strangely QAT is better now.
- with CCR - 2.14 Gbps
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
15 root -16 - 0B 16K CPU7 7 0:11 100.00%
[crypto returns 9]
11 root -92 - 0B 1120K CPU4 4 0:10 98.74%
[intr{irq295: t6nex0:0a0}]
14 root -16 - 0B 16K CPU11 11 0:09 95.32%
[crypto returns 8]
11 root -92 - 0B 1120K CPU8 8 0:07 63.49%
[intr{irq297: t6nex0:0a2}]
11 root -92 - 0B 1120K WAIT 12 0:05 42.73%
[intr{irq307: t6nex0:1a2}]
16 root -16 - 0B 16K CPU15 15 0:04 33.82%
[crypto returns 10]
13 root -16 - 0B 16K RUN 3 0:02 21.78%
[crypto returns 7]
4317 root 22 0 23M 4348K sbwait 14 0:01 7.24% iperf
-B 172.16.70.6 -c 172.16.68.1 -P 2 -t 20{iperf}
4329 root 23 0 23M 4348K sbwait 14 0:01 7.18% iperf
-B 172.16.70.10 -c 172.16.68.1 -P 2 -t 20{iperf}
...
11 root -92 - 0B 1120K WAIT 3 0:00 0.31%
[intr{irq294: t6nex0:evt}]
0 root -92 - 0B 2144K - 14 0:00 0.30%
[kernel{t6nex0 tq0}]
...
- with QAT - 2.63 Gbps
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
8 root -16 - 0B 16K CPU7 7 0:31 92.05%
[crypto returns 5]
7 root -16 - 0B 16K CPU0 0 0:29 86.03%
[crypto returns 4]
9 root -16 - 0B 16K RUN 15 0:15 43.32%
[crypto returns 6]
11 root -92 - 0B 1664K RUN 4 0:15 43.03%
[intr{irq297: t6nex0:0a2}]
6 root -16 - 0B 16K crypto 4 0:09 26.01%
[crypto returns 3]
11 root -92 - 0B 1664K WAIT 12 0:07 19.88%
[intr{irq307: t6nex0:1a2}]
11 root -92 - 0B 1664K WAIT 8 0:02 7.22%
[intr{irq368: qat1}]
4313 root 22 0 23M 4348K sbwait 2 0:01 6.51% iperf
-B 172.16.70.8 -c 172.16.68.1 -P 2 -t 20{iperf}
4316 root 22 0 23M 4348K sbwait 3 0:01 6.18% iperf
-B 172.16.70.9 -c 172.16.68.1 -P 2 -t 20{iperf}
...
11 root -92 - 0B 1664K WAIT 12 0:02 4.52%
[intr{irq355: qat0}]
...
11 root -92 - 0B 1664K WAIT 12 0:01 1.89%
[intr{irq372: qat1}]
11 root -92 - 0B 1664K WAIT 0 0:00 1.12%
[intr{irq360: qat1}]
11 root -92 - 0B 1664K WAIT 2 0:00 1.11%
[intr{irq362: qat1}]
11 root -92 - 0B 1664K WAIT 14 0:00 1.10%
[intr{irq374: qat1}]
11 root -92 - 0B 1664K WAIT 4 0:00 1.10%
[intr{irq364: qat1}]
11 root -92 - 0B 1664K WAIT 10 0:00 1.10%
[intr{irq370: qat1}]
11 root -92 - 0B 1664K WAIT 7 0:00 1.09%
[intr{irq367: qat1}]
11 root -92 - 0B 1664K WAIT 11 0:00 1.07%
[intr{irq371: qat1}]
11 root -92 - 0B 1664K WAIT 5 0:00 1.04%
[intr{irq365: qat1}]
11 root -92 - 0B 1664K WAIT 15 0:00 1.04%
[intr{irq375: qat1}]
11 root -92 - 0B 1664K WAIT 1 0:00 1.04%
[intr{irq361: qat1}]
11 root -92 - 0B 1664K WAIT 6 0:00 1.03%
[intr{irq366: qat1}]

Post by Mark Johnston

results

Post by Ãzkan KIRIK
are same
- with CCR - 1.8Gbps
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU

COMMAND

Post by Ãzkan KIRIK
14 root -16 - 0B 16K CPU5 5 1:38 100.00%

[crypto

Post by Ãzkan KIRIK
returns 8]
3 root -16 - 0B 16K CPU1 1 0:58 77.83%

[crypto

Post by Ãzkan KIRIK
returns 0]

Could you also try this patch? It won't help with aesni.
diff --git a/sys/opencrypto/crypto.c b/sys/opencrypto/crypto.c
index dfd22662e87f..bc23056e86ad 100644
--- a/sys/opencrypto/crypto.c
+++ b/sys/opencrypto/crypto.c
@@ -94,6 +94,7 @@ struct crypto_session {
void *softc;
uint32_t hid;
uint32_t capabilities;
+ uint64_t id;
};
SDT_PROVIDER_DEFINE(opencrypto);
@@ -572,6 +573,7 @@ crypto_select_driver(const struct cryptoini *cri, int flags)
int
crypto_newsession(crypto_session_t *cses, struct cryptoini *cri, int crid)
{
+ static uint64_t sessid = 0;
crypto_session_t res;
void *softc_mem;
struct cryptocap *cap;
@@ -616,6 +618,7 @@ crypto_newsession(crypto_session_t *cses, struct
cryptoini *cri, int crid)
softc_mem = malloc(softc_size, M_CRYPTO_DATA, M_WAITOK | M_ZERO);
res = uma_zalloc(cryptoses_zone, M_WAITOK | M_ZERO);
res->softc = softc_mem;
+ res->id = atomic_fetchadd_64(&sessid, 1);
CRYPTO_DRIVER_LOCK();
cap = crypto_checkdriver(hid);
@@ -1016,7 +1019,7 @@ crypto_dispatch(struct cryptop *crp)
binuptime(&crp->crp_tstamp);
#endif
- crp->crp_retw_id = ((uintptr_t)crp->crp_session) %
crypto_workers_num;
+ crp->crp_retw_id = crp->crp_session->id % crypto_workers_num;
if (CRYPTOP_ASYNC(crp)) {
if (crp->crp_flags & CRYPTO_F_ASYNC_KEEPORDER) {

Mark Johnston

2021-05-01 14:51:20 UTC

Permalink

Post by Ãzkan KIRIK
can test if you provide patches. Test environment is explained in my first
email on this thread.
@Mark Johnston <***@freebsd.org> Now again on stable/13,
- with aesni, without netipsec/ipsec_input.c patch - 1.44Gbps - single
netisr thread eats %100 cpu
- with qat, without netipsec/ipsec_input.c patch - 1.88Gbps - single netisr
thread eats %100 cpu
- with aesni, with netipsec/ipsec_input.c patch - 1.33Gbps
- with qat, with netipsec/ipsec_input.c patch - 2.85Gbps -
stable/13 results are better then stable/12 but not enough fast. There is
something makes bottleneck for IPsec.

So with these results it looks like we have 4 crypto threads running,
which is what I'd expect for two pairs of IP addresses. There is still
a single-threaded bottleneck. I would suggest generating a flame graph
using DTrace and https://github.com/brendangregg/FlameGraph to see where
we're spending CPU time. It would also be useful to know if we're
getting errors or drops anywhere. The QAT (sysctl dev.qat.*.stats) and
ESP/AH (netstat -s -p (esp|ah)) counters would be a useful start, in
addition to counters from cxgbe.

Özkan KIRIK

2021-05-01 21:18:04 UTC

Permalink

the previous flamegraph is captured while iperf client (jail) sends to
iperf server.
the attached flamegraph to this mail is captured while iperf configured
full-duplex mode. Throughput is about up: 1.5 Gbps down: 1.5 Gbps total: 3
Gbps

Post by Ãzkan KIRIK
Hello,
The flamegraph is attached.
# netstat -s
...
0 inbound packets violated process security policy
0 inbound packets failed due to insufficient memory
0 invalid inbound packets
0 outbound packets violated process security policy
0 outbound packets with no SA available
0 outbound packets failed due to insufficient memory
0 outbound packets with no route available
0 invalid outbound packets
0 outbound packets with bundled SAs
0 spd cache hits
0 spd cache misses
0 clusters copied during clone
0 mbufs inserted during makespace
0 packets shorter than header shows
0 packets dropped; protocol family not supported
0 packets dropped; no TDB
0 packets dropped; bad KCR
0 packets dropped; queue full
0 packets dropped; no transform
0 replay counter wraps
0 packets dropped; bad authentication detected
0 packets dropped; bad authentication length
0 possible replay packets detected
0 packets in
0 packets out
0 packets dropped; invalid TDB
0 bytes in
0 bytes out
0 packets dropped; larger than IP_MAXPACKET
0 packets blocked due to policy
0 crypto processing failures
0 tunnel sanity check failures
aes-gmac-128: 35517864
0 packets shorter than header shows
0 packets dropped; protocol family not supported
0 packets dropped; no TDB
0 packets dropped; bad KCR
0 packets dropped; queue full
20 packets dropped; no transform
0 packets dropped; bad ilen
0 replay counter wraps
0 packets dropped; bad encryption detected
0 packets dropped; bad authentication detected
0 possible replay packets detected
23598941 packets in
11918943 packets out
0 packets dropped; invalid TDB
32247932688 bytes in
630318292 bytes out
0 packets dropped; larger than IP_MAXPACKET
0 packets blocked due to policy
0 crypto processing failures
0 tunnel sanity check failures
aes-gcm-16: 35517864
dev.qat.1.stats.sym_alloc_failures: 0
dev.qat.1.stats.ring_full: 1267
dev.qat.1.stats.gcm_aad_updates: 0
dev.qat.1.stats.gcm_aad_restarts: 0
dev.qat.1.%domain: 0
dev.qat.1.%parent: pci16
dev.qat.1.%pnpinfo: vendor=0x8086 device=0x37c8 subvendor=0x8086
subdevice=0x0000 class=0x0b4000
dev.qat.1.%location: slot=0 function=0 dbsf=pci0:182:0:0
dev.qat.1.%driver: qat
dev.qat.1.%desc: Intel C620/Xeon D-2100 QuickAssist PF
dev.qat.0.stats.sym_alloc_failures: 0
dev.qat.0.stats.ring_full: 0
dev.qat.0.stats.gcm_aad_updates: 0
dev.qat.0.stats.gcm_aad_restarts: 0
dev.qat.0.%domain: 0
dev.qat.0.%parent: pci15
dev.qat.0.%pnpinfo: vendor=0x8086 device=0x37c8 subvendor=0x8086
subdevice=0x0000 class=0x0b4000
dev.qat.0.%location: slot=0 function=0 dbsf=pci0:181:0:0
dev.qat.0.%driver: qat
dev.qat.0.%desc: Intel C620/Xeon D-2100 QuickAssist PF

Post by Ãzkan KIRIK
Baldwin
CCR, I

Post by Ãzkan KIRIK
can test if you provide patches. Test environment is explained in my

first

Post by Ãzkan KIRIK
email on this thread.
@Mark Johnston <***@freebsd.org> Now again on stable/13,
- with aesni, without netipsec/ipsec_input.c patch - 1.44Gbps - single
netisr thread eats %100 cpu
- with qat, without netipsec/ipsec_input.c patch - 1.88Gbps - single

netisr

Post by Ãzkan KIRIK
thread eats %100 cpu
- with aesni, with netipsec/ipsec_input.c patch - 1.33Gbps
- with qat, with netipsec/ipsec_input.c patch - 2.85Gbps -
stable/13 results are better then stable/12 but not enough fast. There

Post by Ãzkan KIRIK
something makes bottleneck for IPsec.

Andrey V. Elsukov

2021-05-02 13:08:18 UTC

Permalink

Post by Mark Johnston
Second, netipsec unconditionally hands rx processing off to netisr
threads for some reason, that's why changing the dispatch policy doesn't
help. Maybe it's to help avoid running out of kernel stack space or to
somehow avoid packet reordering in some case that is not clear to me. I
tried a patch (see below) which eliminates this and it helped somewhat.
If anyone can provide an explanation for the current behaviour I'd
appreciate it.

Previously we have reports about kernel stack overflow during IPsec
processing. In your example there is only one IPsec transform is
configured, but it is possible to configure several in the bundle,
AFAIR, it is limited to 4 transforms. E.g. if you configure ESP+AH - it
is bundle of two transforms and this will grow kernel stack requirements.

--
WBR, Andrey V. Elsukov

Mark Johnston

2021-05-03 19:31:09 UTC

Permalink

Post by Andrey V. Elsukov

Is it only a problem for synchronous crypto ops? With hardware drivers
I'd expect the stack usage to be reset after each transform, since
completions are handled by a dedicated thread. There is also the
net.inet.ipsec.async_crypto knob, which has a similar effect I think.

Özkan KIRIK

2021-05-06 14:07:30 UTC

Permalink

I wonder that if you received the flame graphs ?

I also tested system with multiple if_ipsec interfaces using different
source-dst tunnel address.
By this way, system can utilize all cpu cores.
But for single if_ipsec interface, is there a way to speed up transfer ?

Thanks!

Post by Mark Johnston

Post by Andrey V. Elsukov

Post by Mark Johnston
Second, netipsec unconditionally hands rx processing off to netisr
threads for some reason, that's why changing the dispatch policy

doesn't

Post by Andrey V. Elsukov

Post by Mark Johnston
help. Maybe it's to help avoid running out of kernel stack space or to
somehow avoid packet reordering in some case that is not clear to me.

Post by Andrey V. Elsukov

Post by Mark Johnston
tried a patch (see below) which eliminates this and it helped somewhat.
If anyone can provide an explanation for the current behaviour I'd
appreciate it.

Mark Johnston

2021-05-06 15:56:44 UTC

Permalink

Post by Ãzkan KIRIK
I wonder that if you received the flame graphs ?

Yes, there's nothing obviously problematic there.

Post by Ãzkan KIRIK
I also tested system with multiple if_ipsec interfaces using different
source-dst tunnel address.
By this way, system can utilize all cpu cores.
But for single if_ipsec interface, is there a way to speed up transfer ?

Did you see Andrey's comment regarding the MTU on the ipsec interface?

Andrey V. Elsukov

2021-05-02 13:48:32 UTC

Permalink

I suspect you are using 9k MTU on cc(4) interfaces.
If you set bigger MTU on the if_ipsec(4) interfaces, this can increase
throughput.

--
WBR, Andrey V. Elsukov