Intel 4-port ethernet adaptor link aggregation issue

[Joe Moog]

We have an iXsystems 1U server (E5) with an Intel 4-port ethernet NIC installed, model I350-T4 (manufactured May of 2013). We're trying to bind the 4 ports on this NIC together into a single lagg port, connected LACP to a distribution switch (Cisco 4900-series). We are able to successfully bind the 2 on-board ethernet ports to a single lagg, however the NIC is not so cooperative. At first we thought we had a bad NIC, but a replacement has not fixed the issue. We are thinking there may be a driver limitation with these Intel ethernet NICs when attempting to bind more than 2 ports to a lagg.

FreeBSD version:
FreeBSD 9.1-PRERELEASE #0 r244125: Wed Dec 12 11:47:47 CST 2012

rc.conf:
# LINK AGGREGATION
ifconfig_igb2="UP"
ifconfig_igb3="UP"
ifconfig_igb4="UP"
ifconfig_igb5="UP"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto lacp laggport igb2 laggport igb3 laggport igb4 laggport igb5"
ifconfig_lagg0="inet 192.168.1.14 netmask 255.255.255.0"

We've confirmed that the lagg module is loaded (clearly, as the pair of on-board ethernet ports can be bound successfully). Binding various combinations of ports on the NIC yields odd results, as sometimes the first one in the list does not negotiate properly, sometimes the last one in the list fails negotiation. Adding interfaces to lagg individually versus all at the same time does not seem to make any difference. At one point we even tried to assign unique and separate IP addresses to the ethernet ports individually, and only a couple of the ports would actually come active and respond to any sort of network activity. Due to this issue with the number of "usable" ports even beyond the link aggregation failure, this is sort of what leads us to believe there may be an issue with the drive
rs for this card.

We've searched the 'net/lists fairly extensively, and have seen very few instances where people have tried to bind more than 2 ports to a lagg with FreeBSD. Again, 2 ports is no problem, so long as we use the on-board ports; it's the introduction of the Intel NIC and 2 more ports that has us stuck.

Has anybody had any success with such a setup?

Joe

[Ryan Stone]

Post by Joe Moog
We have an iXsystems 1U server (E5) with an Intel 4-port ethernet NIC
installed, model I350-T4 (manufactured May of 2013). We're trying to bind
the 4 ports on this NIC together into a single lagg port, connected LACP to
a distribution switch (Cisco 4900-series). We are able to successfully bind
the 2 on-board ethernet ports to a single lagg, however the NIC is not so
cooperative. At first we thought we had a bad NIC, but a replacement has
not fixed the issue. We are thinking there may be a driver limitation with
these Intel ethernet NICs when attempting to bind more than 2 ports to a
lagg.
FreeBSD 9.1-PRERELEASE #0 r244125: Wed Dec 12 11:47:47 CST 2012
# LINK AGGREGATION
ifconfig_igb2="UP"
ifconfig_igb3="UP"
ifconfig_igb4="UP"
ifconfig_igb5="UP"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto lacp laggport igb2 laggport igb3 laggport igb4 laggport igb5"
ifconfig_lagg0="inet 192.168.1.14 netmask 255.255.255.0"
We've confirmed that the lagg module is loaded (clearly, as the pair of
on-board ethernet ports can be bound successfully). Binding various
combinations of ports on the NIC yields odd results, as sometimes the first
one in the list does not negotiate properly, sometimes the last one in the
list fails negotiation. Adding interfaces to lagg individually versus all
at the same time does not seem to make any difference. At one point we even
tried to assign unique and separate IP addresses to the ethernet ports
individually, and only a couple of the ports would actually come active and
respond to any sort of network activity. Due to this issue with the number
of "usable" ports even beyond the link aggregation failure, this is sort of
what leads us to believe there may be an issue with the drivers for this
card.
We've searched the 'net/lists fairly extensively, and have seen very few
instances where people have tried to bind more than 2 ports to a lagg with
FreeBSD. Again, 2 ports is no problem, so long as we use the on-board
ports; it's the introduction of the Intel NIC and 2 more ports that has us
stuck.
Has anybody had any success with such a setup?
Joe
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-net

Have you tried using only two ports, but both from the NIC? My suspicion
would be that the problem is in the lagg's handling of more than 2 ports
rather than the driver, especially given that it is the igb driver in all
cases.

[Joe Moog]

Have you tried using only two ports, but both from the NIC? My suspicion would be that the problem is in the lagg's handling of more than 2 ports rather than the driver, especially given that it is the igb driver in all cases.

Ryan:

We have done this successfully with two ports on the NIC, on another hardware-identical host. That said, it is entirely possible that this is a shortcoming of lagg.

Can you think of any sort of workaround? Our desired implementation really requires the inclusion of all 4 ports in the lagg. Failing this we're looking at the likelihood of 10G ethernet, but with that comes significant overhead, both cost and administration (before anybody tries to force the cost debate, remember that there are 10G router modules and 10G-capable distribution switches involved, never mind the cabling and SFPs -- it's not just a $600 10G card for the host). I'd like to defer that requirement as long as possible. 4 aggregated gig ports would serve us perfectly well for the near-term.

Thanks

Joe

[Joe Moog]

Post by Joe Moog

Have you tried using only two ports, but both from the NIC? My suspicion would be that the problem is in the lagg's handling of more than 2 ports rather than the driver, especially given that it is the igb driver in all cases.

We have done this successfully with two ports on the NIC, on another hardware-identical host. That said, it is entirely possible that this is a shortcoming of lagg.
Can you think of any sort of workaround? Our desired implementation really requires the inclusion of all 4 ports in the lagg. Failing this we're looking at the likelihood of 10G ethernet, but with that comes significant overhead, both cost and administration (before anybody tries to force the cost debate, remember that there are 10G router modules and 10G-capable distribution switches involved, never mind the cabling and SFPs -- it's not just a $600 10G card for the host). I'd like to defer that requirement as long as possible. 4 aggregated gig ports would serve us perfectly well for the near-term.
Thanks
Joe

UPDATE: After additional testing, I'm beginning to suspect the igb driver. With our setup, ifconfig identifies all the ethernet ports as igb(0-5). I configured igb0 with a single static IP address (say, 192.168.1.10), and was able to connect to the host administratively. While connected, I enabled another port as a second standalone port, again with a unique address (say, 192.168.1.20), and was able to access the host via that interface as well. The problem arises when we attempt to similarly add a third interface to the mix -- and it doesn't seem to matter what interface(s) we use, or in what order we activate them. Always on the third interface, that third interface fails to respond despite showing "active" both in ifconfig and on the switch.

If there is anything else I could try that would be useful to help identify where the issue may reside, please let me know.

Thanks

Joe

[Sean Bruno]

Post by Joe Moog

Post by Joe Moog

Have you tried using only two ports, but both from the NIC? My suspicion would be that the problem is in the lagg's handling of more than 2 ports rather than the driver, especially given that it is the igb driver in all cases.

We have done this successfully with two ports on the NIC, on another hardware-identical host. That said, it is entirely possible that this is a shortcoming of lagg.
Can you think of any sort of workaround? Our desired implementation really requires the inclusion of all 4 ports in the lagg. Failing this we're looking at the likelihood of 10G ethernet, but with that comes significant overhead, both cost and administration (before anybody tries to force the cost debate, remember that there are 10G router modules and 10G-capable distribution switches involved, never mind the cabling and SFPs -- it's not just a $600 10G card for the host). I'd like to defer that requirement as long as possible. 4 aggregated gig ports would serve us perfectly well for the near-term.
Thanks
Joe

UPDATE: After additional testing, I'm beginning to suspect the igb driver. With our setup, ifconfig identifies all the ethernet ports as igb(0-5). I configured igb0 with a single static IP address (say, 192.168.1.10), and was able to connect to the host administratively. While connected, I enabled another port as a second standalone port, again with a unique address (say, 192.168.1.20), and was able to access the host via that interface as well. The problem arises when we attempt to similarly add a third interface to the mix -- and it doesn't seem to matter what interface(s) we use, or in what order we activate them. Always on the third interface, that third interface fails to respond despite showing "active" both in ifconfig and on the switch.
If there is anything else I could try that would be useful to help identify where the issue may reside, please let me know.
Thanks
Joe

Your test seems to indicate that the *first* port on the quad-port card
is causing you issues as the on-board interfaces igb0/1 are working
fine.

Can you bring up *any* ports on the quad-port card?

Are you sure that device enumeration is correct in the host o/s and that
port 1 on the aud-port card is really igb2, port 2 is igb3, etc ?

Sean

[Joe Moog]

Post by Sean Bruno

Post by Joe Moog
UPDATE: After additional testing, I'm beginning to suspect the igb driver. With our setup, ifconfig identifies all the ethernet ports as igb(0-5). I configured igb0 with a single static IP address (say, 192.168.1.10), and was able to connect to the host administratively. While connected, I enabled another port as a second standalone port, again with a unique address (say, 192.168.1.20), and was able to access the host via that interface as well. The problem arises when we attempt to similarly add a third interface to the mix -- and it doesn't seem to matter what interface(s) we use, or in what order we activate them. Always on the third interface, that third interface fails to respond despite showing "active" both in ifconfig and on the switch.
If there is anything else I could try that would be useful to help identify where the issue may reside, please let me know.
Thanks
Joe

Your test seems to indicate that the *first* port on the quad-port card
is causing you issues as the on-board interfaces igb0/1 are working
fine.
Can you bring up *any* ports on the quad-port card?
Are you sure that device enumeration is correct in the host o/s and that
port 1 on the aud-port card is really igb2, port 2 is igb3, etc ?
Sean

Sean:

It is not always the first port on the NIC. The host maps the ports the same way every time, in the same order, so this doesn't appear to be of any consequence. We can enable any one port on the host (on-board or NIC), and then enable another (again, on-board or NIC), and both appear to function as expected. The problem arises when we enable a third port -- any port, in any order. That third port always fails to respond appropriately in our setup, despite appearing to be active according to ifconfig and the interface status on the switch. Any port activated after the second one fails to respond to any sort of network activity.

Is it possible there is a sysctl option that is restricting igb from allowing more than two active ethernet ports?

Thanks

Joe

[Jack Vogel]

Post by Joe Moog

Post by Ryan Stone
Have you tried using only two ports, but both from the NIC? My

suspicion would be that the problem is in the lagg's handling of more than
2 ports rather than the driver, especially given that it is the igb driver
in all cases.

Post by Joe Moog
We have done this successfully with two ports on the NIC, on another

hardware-identical host. That said, it is entirely possible that this is a
shortcoming of lagg.

Post by Joe Moog
Can you think of any sort of workaround? Our desired implementation

really requires the inclusion of all 4 ports in the lagg. Failing this
we're looking at the likelihood of 10G ethernet, but with that comes
significant overhead, both cost and administration (before anybody tries to
force the cost debate, remember that there are 10G router modules and
10G-capable distribution switches involved, never mind the cabling and SFPs
-- it's not just a $600 10G card for the host). I'd like to defer that
requirement as long as possible. 4 aggregated gig ports would serve us
perfectly well for the near-term.

Post by Joe Moog
Thanks
Joe

UPDATE: After additional testing, I'm beginning to suspect the igb driver.
With our setup, ifconfig identifies all the ethernet ports as igb(0-5). I
configured igb0 with a single static IP address (say, 192.168.1.10), and
was able to connect to the host administratively. While connected, I
enabled another port as a second standalone port, again with a unique
address (say, 192.168.1.20), and was able to access the host via that
interface as well. The problem arises when we attempt to similarly add a
third interface to the mix -- and it doesn't seem to matter what
interface(s) we use, or in what order we activate them. Always on the third
interface, that third interface fails to respond despite showing "active"
both in ifconfig and on the switch.
If there is anything else I could try that would be useful to help
identify where the issue may reside, please let me know.

Well, you're using a PRERELEASE of 9.1, leads me to wonder how old the igb
driver is also. First step would be to try all this
on more recent bits... I'd go for HEAD or at least 9.2 BETA as a start. We
don't use or test lagg within Intel, but we've tested the
quad port adapter and not seen an issue with a third port not working.

Good luck,

Jack

[Pieper, Jeffrey E]

-----Original Message-----
From: owner-freebsd-***@freebsd.org [mailto:owner-freebsd-***@freebsd.org] On Behalf Of Jack Vogel
Sent: Thursday, August 01, 2013 3:49 PM
To: Joe Moog
Cc: freebsd-net; Ryan Stone
Subject: Re: Intel 4-port ethernet adaptor link aggregation issue

Post by Joe Moog

Post by Ryan Stone
Have you tried using only two ports, but both from the NIC? My

suspicion would be that the problem is in the lagg's handling of more than
2 ports rather than the driver, especially given that it is the igb driver
in all cases.

Post by Joe Moog
We have done this successfully with two ports on the NIC, on another

hardware-identical host. That said, it is entirely possible that this is a
shortcoming of lagg.

Post by Joe Moog
Can you think of any sort of workaround? Our desired implementation

really requires the inclusion of all 4 ports in the lagg. Failing this
we're looking at the likelihood of 10G ethernet, but with that comes
significant overhead, both cost and administration (before anybody tries to
force the cost debate, remember that there are 10G router modules and
10G-capable distribution switches involved, never mind the cabling and SFPs
-- it's not just a $600 10G card for the host). I'd like to defer that
requirement as long as possible. 4 aggregated gig ports would serve us
perfectly well for the near-term.

Post by Joe Moog
Thanks
Joe

UPDATE: After additional testing, I'm beginning to suspect the igb driver.
With our setup, ifconfig identifies all the ethernet ports as igb(0-5). I
configured igb0 with a single static IP address (say, 192.168.1.10), and
was able to connect to the host administratively. While connected, I
enabled another port as a second standalone port, again with a unique
address (say, 192.168.1.20), and was able to access the host via that
interface as well. The problem arises when we attempt to similarly add a
third interface to the mix -- and it doesn't seem to matter what
interface(s) we use, or in what order we activate them. Always on the third
interface, that third interface fails to respond despite showing "active"
both in ifconfig and on the switch.
If there is anything else I could try that would be useful to help
identify where the issue may reside, please let me know.
Well, you're using a PRERELEASE of 9.1, leads me to wonder how old the igb
driver is also. First step would be to try all this
on more recent bits... I'd go for HEAD or at least 9.2 BETA as a start. We
don't use or test lagg within Intel, but we've tested the
quad port adapter and not seen an issue with a third port not working.
Good luck,
Jack

I can ping to 4 different subnets using all 4 ports simultaneously with an I350-T4 using 9.1-RELEASE and igb-2.3.9 (in-kernel driver) connected to a Cisco 4948, so the issue is definitely not with the igb driver.

Jeff
_______________________________________________
freebsd-***@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-***@freebsd.org"

[Adrian Chadd]

Well, isn't there MAC address reprogramming and such going on when one
enables LAGG on an interface?



-adrian

[John-Mark Gurney]

Post by Joe Moog

Post by Joe Moog

Have you tried using only two ports, but both from the NIC? My suspicion would be that the problem is in the lagg's handling of more than 2 ports rather than the driver, especially given that it is the igb driver in all cases.

We have done this successfully with two ports on the NIC, on another hardware-identical host. That said, it is entirely possible that this is a shortcoming of lagg.
Can you think of any sort of workaround? Our desired implementation really requires the inclusion of all 4 ports in the lagg. Failing this we're looking at the likelihood of 10G ethernet, but with that comes significant overhead, both cost and administration (before anybody tries to force the cost debate, remember that there are 10G router modules and 10G-capable distribution switches involved, never mind the cabling and SFPs -- it's not just a $600 10G card for the host). I'd like to defer that requirement as long as possible. 4 aggregated gig ports would serve us perfectly well for the near-term.
Thanks
Joe

UPDATE: After additional testing, I'm beginning to suspect the igb driver. With our setup, ifconfig identifies all the ethernet ports as igb(0-5). I configured igb0 with a single static IP address (say, 192.168.1.10), and was able to connect to the host administratively. While connected, I enabled another port as a second standalone port, again with a unique address (say, 192.168.1.20), and was able to access the host via that interface as well. The problem arises when we attempt to similarly add a third interface to the mix -- and it doesn't seem to matter what interface(s) we use, or in what order we activate them. Always on the third interface, that third interface fails to respond despite showing "active" both in ifconfig and on the switch.

Can you show an ifconfig -au from the host when it fails, and which was
the third interface that you added? Above, you talk about adding ips in
the same subnet to different interfaces, which with modern switchs can
cause issues with which port to deliver packets, etc.

Do you have any firewalling enabled on the host?

Post by Joe Moog
If there is anything else I could try that would be useful to help identify where the issue may reside, please let me know.
Thanks
Joe
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-net

[Joe Moog]

Post by John-Mark Gurney

Post by Joe Moog

Post by Joe Moog

Have you tried using only two ports, but both from the NIC? My suspicion would be that the problem is in the lagg's handling of more than 2 ports rather than the driver, especially given that it is the igb driver in all cases.

We have done this successfully with two ports on the NIC, on another hardware-identical host. That said, it is entirely possible that this is a shortcoming of lagg.
Can you think of any sort of workaround? Our desired implementation really requires the inclusion of all 4 ports in the lagg. Failing this we're looking at the likelihood of 10G ethernet, but with that comes significant overhead, both cost and administration (before anybody tries to force the cost debate, remember that there are 10G router modules and 10G-capable distribution switches involved, never mind the cabling and SFPs -- it's not just a $600 10G card for the host). I'd like to defer that requirement as long as possible. 4 aggregated gig ports would serve us perfectly well for the near-term.
Thanks
Joe

UPDATE: After additional testing, I'm beginning to suspect the igb driver. With our setup, ifconfig identifies all the ethernet ports as igb(0-5). I configured igb0 with a single static IP address (say, 192.168.1.10), and was able to connect to the host administratively. While connected, I enabled another port as a second standalone port, again with a unique address (say, 192.168.1.20), and was able to access the host via that interface as well. The problem arises when we attempt to similarly add a third interface to the mix -- and it doesn't seem to matter what interface(s) we use, or in what order we activate them. Always on the third interface, that third interface fails to respond despite showing "active" both in ifconfig and on the switch.

Can you show an ifconfig -au from the host when it fails, and which was
the third interface that you added? Above, you talk about adding ips in
the same subnet to different interfaces, which with modern switchs can
cause issues with which port to deliver packets, etc.
Do you have any firewalling enabled on the host?

There are no firewalls enabled on the host.

I don't know that I see the switch as being the weak point in this setup as we have been very successful multihoming boxes with these switches for a variety of other purposes. I will collect and forward "ifconfig -au" output from the host in a couple of days, as we have had to fall back on the 2-port lagg to get this particular host in service until such time the 4-port lagg issue can be resolved. We will be setting up another hardware-identical host in a lab for further testing and info gathering.

Thanks

Joe

[Hooman Fazaeli]

Post by Joe Moog

Post by Joe Moog

Have you tried using only two ports, but both from the NIC? My suspicion would be that the problem is in the lagg's handling of more than 2 ports rather than the driver, especially given that it is the igb driver in all cases.

We have done this successfully with two ports on the NIC, on another hardware-identical host. That said, it is entirely possible that this is a shortcoming of lagg.
Can you think of any sort of workaround? Our desired implementation really requires the inclusion of all 4 ports in the lagg. Failing this we're looking at the likelihood of 10G ethernet, but with that comes significant overhead, both cost and administration (before anybody tries to force the cost debate, remember that there are 10G router modules and 10G-capable distribution switches involved, never mind the cabling and SFPs -- it's not just a $600 10G card for the host). I'd like to defer that requirement as long as possible. 4 aggregated gig ports would serve us perfectly well for the near-term.
Thanks
Joe

UPDATE: After additional testing, I'm beginning to suspect the igb driver. With our setup, ifconfig identifies all the ethernet ports as igb(0-5). I configured igb0 with a single static IP address (say, 192.168.1.10), and was able to connect to the host administratively. While connected, I enabled another port as a second standalone port, again with a unique address (say, 192.168.1.20), and was able to access the host via that interface as well. The problem arises when we attempt to similarly add a third interface to the mix -- and it doesn't seem to matter what interface(s) we use, or in what order we activate them. Always on the third interface, that third interface fails to respond despite showing "active" both in ifconfig and on the switch.
If there is anything else I could try that would be useful to help identify where the issue may reside, please let me know.
Thanks
Joe
_______________________________________________

Assign IP addresses from __different__ subnets to the four NIC ports and re-test.
(e.g., 192.168.0.10/24, 1.10/24, 2.10/24, 3.10/24).

[Joe Moog]

All:

Thanks again to everybody for the responses and suggestions to our 4-port lagg issue. The solution (for those that may find the information of some value) was to set the value for kern.ipc.nmbclusters to a higher value than we had initially. Our previous tuning had this value set at 25600, but following a recommendation from the good folks at iXSystems we bumped this to a value closer to 2000000, and the 4-port lagg is functioning as expected now.

Thank you all.

Joe

[Barney Cordoba]

That's way too high. Your base rx requirement is 

Ports * queues * rxd 

With a quad card you shouldn't be using more than 2 queues, so your requirement
with 5 ports is 10,240 just for the receive setup. If you're using 4 queues that
number doubles, which would make 25,600 not enough. 

Note that setting mbufs to a huge number doesn't allocate the buffers; they'll be
allocated as needed. It's a ceiling. The reason for the ceiling is so that you don't
blow up your memory. If your system is using 2 million mbuf clusters then you
have much bigger problems than LAGG.

Anyone who recommends 2 million clearly has no idea what they're doing.

BC


________________________________
From: Joe Moog <***@ebureau.com>
To: freebsd-net <freebsd-***@freebsd.org>
Sent: Wednesday, August 28, 2013 9:36 AM
Subject: Re: Intel 4-port ethernet adaptor link aggregation issue


All:

Thanks again to everybody for the responses and suggestions to our 4-port lagg issue. The solution (for those that may find the information of some value) was to set the value for kern.ipc.nmbclusters to a higher value than we had initially. Our previous tuning had this value set at 25600, but following a recommendation from the good folks at iXSystems we bumped this to a value closer to 2000000, and the 4-port lagg is functioning as expected now.

Thank you all.

Joe

_______________________________________________
freebsd-***@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-***@freebsd.org"

[Steve Read]

Post by Joe Moog
We have an iXsystems 1U server (E5) with an Intel 4-port ethernet NIC installed, model I350-T4 (manufactured May of 2013). We're trying to bind the 4 ports on this NIC together into a single lagg port, connected LACP to a distribution switch (Cisco 4900-series). We are able to successfully bind the 2 on-board ethernet ports to a single lagg, however the NIC is not so cooperative. At first we thought we had a bad NIC, but a replacement has not fixed the issue. We are thinking there may be a driver limitation with these Intel ethernet NICs when attempting to bind more than 2 ports to a lagg.
FreeBSD 9.1-PRERELEASE #0 r244125: Wed Dec 12 11:47:47 CST 2012
# LINK AGGREGATION
ifconfig_igb2="UP"
ifconfig_igb3="UP"
ifconfig_igb4="UP"
ifconfig_igb5="UP"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto lacp laggport igb2 laggport igb3 laggport igb4 laggport igb5"
ifconfig_lagg0="inet 192.168.1.14 netmask 255.255.255.0"

Am I the only one who noticed that you replaced the value of
$ifconfig_lagg0 that specifies the proto and the ports with one that
specifies just the address?

Merge the two ifconfig_lagg0 lines into one, and it will work infinitely
better, or at least no worse.

ifconfig_lagg0="laggproto lacp laggport igb2 laggport igb3 laggport igb4 laggport igb5 inet 192.168.1.14 netmask 255.255.255.0"


-- Steve

[Freddie Cash]

Post by Steve Read

Post by Joe Moog
We have an iXsystems 1U server (E5) with an Intel 4-port ethernet NIC
installed, model I350-T4 (manufactured May of 2013). We're trying to bind
the 4 ports on this NIC together into a single lagg port, connected LACP to
a distribution switch (Cisco 4900-series). We are able to successfully bind
the 2 on-board ethernet ports to a single lagg, however the NIC is not so
cooperative. At first we thought we had a bad NIC, but a replacement has
not fixed the issue. We are thinking there may be a driver limitation with
these Intel ethernet NICs when attempting to bind more than 2 ports to a
lagg.
FreeBSD 9.1-PRERELEASE #0 r244125: Wed Dec 12 11:47:47 CST 2012
# LINK AGGREGATION
ifconfig_igb2="UP"
ifconfig_igb3="UP"
ifconfig_igb4="UP"
ifconfig_igb5="UP"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto lacp laggport igb2 laggport igb3 laggport igb4 laggport igb5"
ifconfig_lagg0="inet 192.168.1.14 netmask 255.255.255.0"

Am I the only one who noticed that you replaced the value of
$ifconfig_lagg0 that specifies the proto and the ports with one that
specifies just the address?

Good catch!

Post by Steve Read
Merge the two ifconfig_lagg0 lines into one, and it will work infinitely
better, or at least no worse.
ifconfig_lagg0="laggproto lacp laggport igb2 laggport igb3 laggport igb4
laggport igb5 inet 192.168.1.14 netmask 255.255.255.0"
Or, if you want to keep them split into two parts (initialise lagg0, then

add IP):

create_args_lagg0="laggproto lacp laggport igb2 laggport igb3 laggport igb4
laggport igb5"

ifconfig_lagg0="inet 192.168.1.14 netmask 255.255.255.0"

create_args_* are run first, then ifconfig_* are run. I like this setup,
as it separates "create and initialise" from "configure" for cloned/virtual
interfaces like vlans, laggs, etc.

[Zaphod Beeblebrox]

On several machines with large numbers of IGBx interfaces, I've found that
hw.igb.enable_msix=0 is necessary to ensure proper operation.

Post by Joe Moog

Post by Steve Read

Post by Joe Moog
We have an iXsystems 1U server (E5) with an Intel 4-port ethernet NIC
installed, model I350-T4 (manufactured May of 2013). We're trying to

bind

Post by Steve Read

Post by Joe Moog
the 4 ports on this NIC together into a single lagg port, connected

LACP to

Post by Steve Read

Post by Joe Moog
a distribution switch (Cisco 4900-series). We are able to successfully

bind

Post by Steve Read

Post by Joe Moog
the 2 on-board ethernet ports to a single lagg, however the NIC is not

so

Post by Steve Read

Post by Joe Moog
cooperative. At first we thought we had a bad NIC, but a replacement has
not fixed the issue. We are thinking there may be a driver limitation

with

Post by Steve Read

Post by Joe Moog
these Intel ethernet NICs when attempting to bind more than 2 ports to a
lagg.
FreeBSD 9.1-PRERELEASE #0 r244125: Wed Dec 12 11:47:47 CST 2012
# LINK AGGREGATION
ifconfig_igb2="UP"
ifconfig_igb3="UP"
ifconfig_igb4="UP"
ifconfig_igb5="UP"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto lacp laggport igb2 laggport igb3 laggport igb4
laggport igb5"
ifconfig_lagg0="inet 192.168.1.14 netmask 255.255.255.0"

Am I the only one who noticed that you replaced the value of
$ifconfig_lagg0 that specifies the proto and the ports with one that
specifies just the address?

Good catch!

Post by Steve Read
Merge the two ifconfig_lagg0 lines into one, and it will work infinitely
better, or at least no worse.
ifconfig_lagg0="laggproto lacp laggport igb2 laggport igb3 laggport igb4
laggport igb5 inet 192.168.1.14 netmask 255.255.255.0"
Or, if you want to keep them split into two parts (initialise lagg0, then

create_args_lagg0="laggproto lacp laggport igb2 laggport igb3 laggport igb4
laggport igb5"
ifconfig_lagg0="inet 192.168.1.14 netmask 255.255.255.0"
create_args_* are run first, then ifconfig_* are run. I like this setup,
as it separates "create and initialise" from "configure" for cloned/virtual
interfaces like vlans, laggs, etc.
--
Freddie Cash
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-net

[Adrian Chadd]

Post by Zaphod Beeblebrox
On several machines with large numbers of IGBx interfaces, I've found that
hw.igb.enable_msix=0 is necessary to ensure proper operation.

ixgbe behaves much better now. As of like two days ago.



-adrian

[Barney Cordoba]

The stock igb driver binds to all cores, so with multiple igbs you have multiple
nics binding to the same cores. I suppose that might create issues in a lagg setup.
Try 1 queue  and/or comment out the bind code.

BC


________________________________
From: Zaphod Beeblebrox <***@gmail.com>
To: Freddie Cash <***@gmail.com>
Cc: Steve Read <***@netasq.com>; freebsd-net <freebsd-***@freebsd.org>
Sent: Friday, August 2, 2013 5:41 PM
Subject: Re: Intel 4-port ethernet adaptor link aggregation issue


On several machines with large numbers of IGBx interfaces, I've found that
hw.igb.enable_msix=0 is necessary to ensure proper operation.

Post by Joe Moog

Post by Steve Read

Post by Joe Moog
We have an iXsystems 1U server (E5) with an Intel 4-port ethernet NIC
installed, model I350-T4 (manufactured May of 2013). We're trying to

bind

Post by Steve Read

Post by Joe Moog
the 4 ports on this NIC together into a single lagg port, connected

LACP to

Post by Steve Read

Post by Joe Moog
a distribution switch (Cisco 4900-series). We are able to successfully

bind

Post by Steve Read

Post by Joe Moog
the 2 on-board ethernet ports to a single lagg, however the NIC is not

so

Post by Steve Read

Post by Joe Moog
cooperative. At first we thought we had a bad NIC, but a replacement has
not fixed the issue. We are thinking there may be a driver limitation

with

Post by Steve Read

Post by Joe Moog
these Intel ethernet NICs when attempting to bind more than 2 ports to a
lagg.
FreeBSD 9.1-PRERELEASE #0 r244125: Wed Dec 12 11:47:47 CST 2012
# LINK AGGREGATION
ifconfig_igb2="UP"
ifconfig_igb3="UP"
ifconfig_igb4="UP"
ifconfig_igb5="UP"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto lacp laggport igb2 laggport igb3 laggport igb4
laggport igb5"
ifconfig_lagg0="inet 192.168.1.14  netmask 255.255.255.0"

Am I the only one who noticed that you replaced the value of
$ifconfig_lagg0 that specifies the proto and the ports with one that
specifies just the address?

Good catch!

Post by Steve Read
Merge the two ifconfig_lagg0 lines into one, and it will work infinitely
better, or at least no worse.
ifconfig_lagg0="laggproto lacp laggport igb2 laggport igb3 laggport igb4
laggport igb5 inet 192.168.1.14  netmask 255.255.255.0"
Or, if you want to keep them split into two parts (initialise lagg0, then

create_args_lagg0="laggproto lacp laggport igb2 laggport igb3 laggport igb4
laggport igb5"
ifconfig_lagg0="inet 192.168.1.14  netmask 255.255.255.0"
create_args_* are run first, then ifconfig_* are run.  I like this setup,
as it separates "create and initialise" from "configure" for cloned/virtual
interfaces like vlans, laggs, etc.
--
Freddie Cash
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-net

_______________________________________________
freebsd-***@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-***@freebsd.org"

[Adrian Chadd]

Post by Barney Cordoba
The stock igb driver binds to all cores, so with multiple igbs you have multiple
nics binding to the same cores. I suppose that might create issues in a lagg setup.
Try 1 queue and/or comment out the bind code.

I have thrashed the hell out of 2-port ixgbe and 4-port chelsio
(cxgbe) on 4-core device all with lagg. All is great.

There's apparently some more igb improvements coming in the pipeline. Fear not!


-adrian

[Barney Cordoba]

You can create your own pipeline with some minor modifications. Why wait months
for the guys who did it wrong to make changes?

BC


________________________________
From: Adrian Chadd <***@freebsd.org>
To: Barney Cordoba <***@yahoo.com>
Cc: Zaphod Beeblebrox <***@gmail.com>; Freddie Cash <***@gmail.com>; Steve Read <***@netasq.com>; freebsd-net <freebsd-***@freebsd.org>
Sent: Saturday, August 3, 2013 12:21 AM
Subject: Re: Intel 4-port ethernet adaptor link aggregation issue

Post by Barney Cordoba
The stock igb driver binds to all cores, so with multiple igbs you have multiple
nics binding to the same cores. I suppose that might create issues in a lagg setup.
Try 1 queue  and/or comment out the bind code.

I have thrashed the hell out of 2-port ixgbe and 4-port chelsio
(cxgbe) on 4-core device all with lagg. All is great.

There's apparently some more igb improvements coming in the pipeline. Fear not!


-adrian

[Adrian Chadd]

Post by Barney Cordoba
You can create your own pipeline with some minor modifications. Why wait months
for the guys who did it wrong to make changes?

Because I get paid now to ensure that this stuff gets done well, the
vendor gets engaged, the fixes get pushed into the upstream
codebase(s) as well as merged into FreeBSD-HEAD.

In any case, the OP problem looks more like rc.conf problems than a
driver issue. There are likely driver issues and they'll either get
fixed or I'll hack on it and shout at people until they get fixed.



-adrian

[Joe Moog]

Date: Fri, 02 Aug 2013 09:36:30 +0200
Subject: Re: Intel 4-port ethernet adaptor link aggregation issue
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

Post by Joe Moog
We have an iXsystems 1U server (E5) with an Intel 4-port ethernet NIC installed, model I350-T4 (manufactured May of 2013). We're trying to bind the 4 ports on this NIC together into a single lagg port, connected LACP to a distribution switch (Cisco 4900-series). We are able to successfully bind the 2 on-board ethernet ports to a single lagg, however the NIC is not so cooperative. At first we thought we had a bad NIC, but a replacement has not fixed the issue. We are thinking there may be a driver limitation with these Intel ethernet NICs when attempting to bind more than 2 ports to a lagg.
FreeBSD 9.1-PRERELEASE #0 r244125: Wed Dec 12 11:47:47 CST 2012
# LINK AGGREGATION
ifconfig_igb2="UP"
ifconfig_igb3="UP"
ifconfig_igb4="UP"
ifconfig_igb5="UP"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto lacp laggport igb2 laggport igb3 laggport igb4 laggport igb5"
ifconfig_lagg0="inet 192.168.1.14 netmask 255.255.255.0"

Am I the only one who noticed that you replaced the value of
$ifconfig_lagg0 that specifies the proto and the ports with one that
specifies just the address?
Merge the two ifconfig_lagg0 lines into one, and it will work infinitely
better, or at least no worse.
ifconfig_lagg0="laggproto lacp laggport igb2 laggport igb3 laggport igb4 laggport igb5 inet 192.168.1.14 netmask 255.255.255.0"
-- Steve

Steve:

We have tried the configuration 3 different ways, all taken directly from examples found in various BSD forums and wikis:
1) Specifying the IP address for the lagg port as seen above (which has worked with other FreeBSD lagg configs for us)
2) Specifying the IP address in line with the "laggproto" config as directed above
3) Using the line ipv4_addrs_lagg0="192.168.1.14/24" in place of the ifconfig_lagg0="inet....." line above

Each option yields the same results -- all will work with a 2-port lagg, but none will enable a lagg with more than 2 ports (igb). FreeBSD doesn't seem to care which one we use.

Thanks

Joe