[Bug 230465] ixl: not working in netmap mode

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=230465

Charles Goncalves <***@halfling.com.br> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@FreeBSD.org

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=230465

Michael <***@spam-fetish.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@spam-fetish.org

--- Comment #1 from Michael <***@spam-fetish.org> ---
X710 has a couple of problem with netmap.
Also experienced it with all available firmware versions for the NIC.

I just downgraded to X520 or Chelsio.

For me the NIC just freezes, carrier active and seeing incoming packets via
tcpdump but nothing else ...

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=230465

Mark Linimon <***@FreeBSD.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
Keywords| |IntelNetworking
CC| |***@FreeBSD.org

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=230465

--- Comment #2 from Charles Goncalves <***@halfling.com.br> ---
If I run pkt-gen with rate option then pkt-gen sends packets.

Like this: pkg-gen -f tx -R 150000

If I change value greater than 150k i.e: 160000 I got same error:

/pkt-gen -i ixl0 -f tx -R 160000
812.653021 main [2593] interface is ixl0
812.653078 main [2727] running on 1 cpus (have 8)
812.653279 extract_ip_range [468] range is 10.0.0.1:1234 to 10.0.0.1:1234
812.653296 extract_ip_range [468] range is 10.1.0.1:1234 to 10.1.0.1:1234
812.826944 main [2822] mapped 294020KB at 0x801600000
Sending on netmap:ixl0: 8 queues, 1 threads and 1 cpus.
10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> ff:ff:ff:ff:ff:ff)
812.827043 main [2919] Sending 512 packets every 0.003200000 s
812.827085 start_threads [2274] Wait 2 secs for phy reset
814.836422 start_threads [2276] Ready...
814.836644 sender_body [1464] start, fd 3 main_fd 3
815.838007 main_thread [2364] 0.000 pps (0.000 pkts 0.000 bps in 1001364 usec)
0.00 avg_batch 0 min_space
816.134343 sender_body [1538] poll error on 3 ring 0-7
816.886366 main_thread [2364] 1.954 Kpps (2.048 Kpkts 983.040 Kbps in 1048359
usec) 341.33 avg_batch 99999 min_space
817.061069 main_thread [2391] ouch, thread 0 exited with error
Sent 2048 packets 122880 bytes 6 events 60 bytes each in -1534269816.00
seconds.
Speed: -0.000 pps Bandwidth: -0.001 bps (raw -0.001 bps). Average batch: 341.33
pkts


And kernel log:
Aug 14 15:03:36 rt1 kernel: 816.016016 [1637] nm_txsync_prologue ixl0
TX1: fail 'kring->nr_hwcur >= n || kring->rhead >= n || kring->rtail >= n ||
kring->nr_hwtail >= n' h 275 c 275 t -278400513 rh 274 rc 274 rt -278400513 hc
274 ht -278400513
Aug 14 15:03:36 rt1 kernel: 816.074020 [1758] netmap_ring_reinit called
for ixl0 TX1
Aug 14 15:03:36 rt1 kernel: 816.090925 [1783] netmap_ring_reinit total 1
errors
Aug 14 15:03:36 rt1 kernel: 816.106519 [1787] netmap_ring_reinit ixl0
TX1 reinit, cur 275 -> 274 tail -278400513 -> -278400513



These errors seem to be related to synchronization of HEAD/CUR/TAIL in netmap
ring

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=230465

--- Comment #3 from Charles Goncalves <***@halfling.com.br> ---
I've noted that in POLLIN this:
poll(pfd, rxrings, -1);
Doesn't work, need it to be something like:
poll(pfd, rxrings, 1);
But with this I have a latency problem.

And I also have noted that I only can TX from TX-RING-0 if I use TX-RING more
than 0, doesn't work. In my test scenario my ixl NIC has 8 rings.

Can anyone understand the reason?

Thanks!

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=230465

***@zxy.spb.ru changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@zxy.spb.ru

--- Comment #43 from ***@zxy.spb.ru ---
(In reply to Vincenzo Maffione from comment #42)

Looks like netmap don't worked:

# /usr/obj/usr/src/amd64.amd64/tools/tools/netmap/pkt-gen -i ixl1 -f tx
321.990539 main [2921] interface is ixl1
321.990568 main [3044] using default burst size: 512
321.990573 main [3052] running on 1 cpus (have 24)
321.990640 extract_ip_range [476] range is 10.0.0.1:1234 to 10.0.0.1:1234
321.990645 extract_ip_range [476] range is 10.1.0.1:1234 to 10.1.0.1:1234
Sending on netmap:ixl1: 5 queues, 1 threads and 1 cpus.
10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> ff:ff:ff:ff:ff:ff)
322.096770 main [3255] Sending 512 packets every 0.000000000 s
322.096813 start_threads [2580] Wait 2 secs for phy reset
324.222299 start_threads [2582] Ready...
324.222365 sender_body [1599] start, fd 3 main_fd 3
324.222392 sender_body [1657] frags 1 frag_size 60
324.234391 sender_body [1695] drop copy
325.285776 main_thread [2671] 2.794 Mpps (2.971 Mpkts 1.341 Gbps in 1063411
usec) 15.05 avg_batch 0 min_space
326.348859 main_thread [2671] 0.000 pps (0.000 pkts 0.000 bps in 1063084 usec)
0.00 avg_batch 99999 min_space
326.472386 sender_body [1682] poll error on queue 0: timeout
327.411859 main_thread [2671] 0.000 pps (0.000 pkts 0.000 bps in 1063000 usec)
0.00 avg_batch 99999 min_space
328.473456 sender_body [1682] poll error on queue 0: timeout
328.474874 main_thread [2671] 0.000 pps (0.000 pkts 0.000 bps in 1063015 usec)
0.00 avg_batch 99999 min_space
329.537820 main_thread [2671] 0.000 pps (0.000 pkts 0.000 bps in 1062945 usec)
0.00 avg_batch 99999 min_space
330.474386 sender_body [1682] poll error on queue 0: timeout
330.600771 main_thread [2671] 0.000 pps (0.000 pkts 0.000 bps in 1062951 usec)
0.00 avg_batch 99999 min_space
331.663860 main_thread [2671] 0.000 pps (0.000 pkts 0.000 bps in 1063090 usec)
0.00 avg_batch 99999 min_space
332.475381 sender_body [1682] poll error on queue 0: timeout
332.726861 main_thread [2671] 0.000 pps (0.000 pkts 0.000 bps in 1063001 usec)
0.00 avg_batch 99999 min_space
^C333.671467 sigint_h [573] received control-C on thread 0x800a12000
333.671475 main_thread [2671] 0.000 pps (0.000 pkts 0.000 bps in 944614 usec)
0.00 avg_batch 99999 min_space
334.476434 sender_body [1737] flush tail 576 head 576 on thread 0x800a12700
334.734834 main_thread [2671] 0.000 pps (0.000 pkts 0.000 bps in 1063359 usec)
0.00 avg_batch 99999 min_space
Sent 2971392 packets 178283520 bytes 197414 events 60 bytes each in 10.25
seconds.
Speed: 289.777 Kpps Bandwidth: 139.093 Mbps (raw 139.093 Mbps). Average batch:
15.05 pkts


Additional, in my application I am see logical errors from kernel:

I a send 3 packets in ring 0, c/h/t is 3/3/2047
do NIOCTXSYNC, c/h/t is 3/3/0
do not send any packets, just do NIOCTXSYNC, c/h/t is 3/3/3 now!
i.e. like TX ring is full and stalled. Any transmission staled after this.

13-stable.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=230465

--- Comment #44 from Vincenzo Maffione <***@FreeBSD.org> ---
What if you set
hw.ixl.enable_head_writeback = 0
in /boot/loader.conf and reboot?

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=230465

***@Nanoteq.com changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@Nanoteq.com

--- Comment #45 from ***@Nanoteq.com ---
(In reply to Vincenzo Maffione from comment #44)

I was testing a different application on a box with a ixl card.
I also noticed the drop in the tx packets.
Setting hw.ixl.enable_head_writeback = 0 in /boot/loader.conf seems to have
resolved the issue with the preliminary tests I have done.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=230465

--- Comment #46 from ***@Nanoteq.com ---
(In reply to strongswan from comment #45)
I did some more testing, and even with the setting hw.ixl.enable_head_writeback
= 0 I still get to a situation where no packets are transmitted.
There is however a much longer interval between when the issues are occurring.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=230465

--- Comment #47 from Vincenzo Maffione <***@FreeBSD.org> ---
What is the state of the TX ring (head, cur, tail) when stalling?