vale-ctl(-8), ifconfig(8), SIOCAIFADDR: Invalid argument [utilizing netmap(4) providing virtual switches+interfaces to BHyVe]

Dear all,

I found great papers about netmap(4)s desigen and implementation
details, and I'm sure it's one other masterpeace of rizzo-quality :-)
Thanks to all participants for that great code!

To be honest, I haven't read all of that, because I'm short in time and
my first mission is to see if FreeBSD 11 will replace some of my ESXi
machines.

One key element seems netmap(4).
It's quiet hard to find userland documentation.

So far, I've discovered that there are three essential tools waiting in
_usr/src/tools/tools/netmap_ to be compiled
(resulting in *./vale-ctl*, *./bridge*, *./pkt-gen*)

While the latter is often referenced in netmap(4) documentation, it's
not of interest for me, because I'll be doing real-world performance
tests and I'm convinced that all the impressive numbers presented in the
netmap documentation are valid :-)

So *vale-ctl(-8)* seems to be of interest (I'm using (-8) becaus
currently there is no man8 part (I guess that's the reason for these
tools not beeing integrated into base binaries))

Accidentally I found out that 'vale-ctl -n testif0' creates a artificial
interface, which is reported by ifconfig(8):
testif0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
ether 00:be:eb:8d:f8:00
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

But I can't assign a IP address: 'ifconfig testif0 203.0.113.1/24'
ifconfig: ioctl (SIOCAIFADDR): Invalid argument

I guess couldn't geti the picture of the netmap(4) world yet.
Probably, testif0 is available only in netmap(4) world, not in "host world".
I'm assuming, because I found vale-ctl(-8)s "-h" switch.

So another very little peace I'm aware of the netmap(4) world, is how to
attach physical interfaces to virtual switches:
'/usr/src/tools/tools/netmap/vale-ctl -a vale0:em1'
Now vale-ctl(-8) shows:
bdg_ctl [149] bridge:0 port:0 vale0:em1

/*
To share my experience: One cannot use any other than vale[[:digit:]]
for defining the on-demand to be created virtual switch instance, so
e.g. "vale-ctl -a vale-test:em1" doesn't work, although found in
netmap(4) man page in FreeBSD-11:
»valeXXX:YYY (arbitrary XXX and YYY)
the file descriptor is bound to port YYY of a VALE switch called
XXX, both dynamically created if necessary. The string cannot
exceed IFNAMSIZ characters, and YYY cannot be the name of any
existing OS network interface«

I was about to give up on netmap(4) investigations because I thought it
isn't production ready yet (in FreeBSD), since even andding the first
physical interface fails: '/usr/src/tools/tools/netmap/vale-ctl -a
vale-test:em1'
vale-test:em1: Invalid argument

Probably accidentally I used vale[[:digit:]] instead and wondered whay
it suddenly works…

To get back to vale-ctl(-8)s "-h" switch:
*/

If I add a physical interface with -h instead of -a, the host's IP stack
doesn't get disconnected from the interface, so it's still usable by
host applications and vale-ctl(-8) lists one line more:
bdg_ctl [149] bridge:0 port:0 vale0:em1
bdg_ctl [149] bridge:0 port:1 vale0:em1^
So my assumption that netmap(4) lives decapsuled from the well known
FreeBSD IP world.

Now my question:

How can I plug a jail's or vmm's artificial interface to a VALE virtual
switch, bridging frames to real-world via physical interfaces?
(the latter part should work with vale-ctl -h vale0:em1, but what
interface to use for jail(8) vnet.interface and how to create/attach?)

Thanks,

-harry

Vincenzo Maffione

2016-10-14 13:08:55 UTC

Hi,

Thanks for your feedback.

Post by Harry Schmalzbauer
Dear all,
I found great papers about netmap(4)s desigen and implementation
details, and I'm sure it's one other masterpeace of rizzo-quality :-)
Thanks to all participants for that great code!
To be honest, I haven't read all of that, because I'm short in time and
my first mission is to see if FreeBSD 11 will replace some of my ESXi
machines.
One key element seems netmap(4).
It's quiet hard to find userland documentation.
So far, I've discovered that there are three essential tools waiting in
_usr/src/tools/tools/netmap_ to be compiled
(resulting in *./vale-ctl*, *./bridge*, *./pkt-gen*)
While the latter is often referenced in netmap(4) documentation, it's
not of interest for me, because I'll be doing real-world performance
tests and I'm convinced that all the impressive numbers presented in the
netmap documentation are valid :-)
So *vale-ctl(-8)* seems to be of interest (I'm using (-8) becaus
currently there is no man8 part (I guess that's the reason for these
tools not beeing integrated into base binaries))
Accidentally I found out that 'vale-ctl -n testif0' creates a artificial
testif0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
ether 00:be:eb:8d:f8:00
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
But I can't assign a IP address: 'ifconfig testif0 203.0.113.1/24'
ifconfig: ioctl (SIOCAIFADDR): Invalid argument
I guess couldn't geti the picture of the netmap(4) world yet.
Probably, testif0 is available only in netmap(4) world, not in "host world".
I'm assuming, because I found vale-ctl(-8)s "-h" switch.

Yes, those are the "persistent" VALE ports. They are a recent feature, and
probably you don't need to use them if you are going to play with Virtual
Machines and jails (see below).

Post by Harry Schmalzbauer
So another very little peace I'm aware of the netmap(4) world, is how to
'/usr/src/tools/tools/netmap/vale-ctl -a vale0:em1'
bdg_ctl [149] bridge:0 port:0 vale0:em1
/*
To share my experience: One cannot use any other than vale[[:digit:]]
for defining the on-demand to be created virtual switch instance, so
e.g. "vale-ctl -a vale-test:em1" doesn't work, although found in
»valeXXX:YYY (arbitrary XXX and YYY)
the file descriptor is bound to port YYY of a VALE switch called
XXX, both dynamically created if necessary. The string cannot
exceed IFNAMSIZ characters, and YYY cannot be the name of any
existing OS network interface«
I was about to give up on netmap(4) investigations because I thought it
isn't production ready yet (in FreeBSD), since even andding the first
physical interface fails: '/usr/src/tools/tools/netmap/vale-ctl -a
vale-test:em1'
vale-test:em1: Invalid argument
Probably accidentally I used vale[[:digit:]] instead and wondered whay
it suddenly works…

Correct, this seems to be an inconsistency between the manual and the
implementation, we will fix the manual.

Post by Harry Schmalzbauer
*/
If I add a physical interface with -h instead of -a, the host's IP stack
doesn't get disconnected from the interface, so it's still usable by
bdg_ctl [149] bridge:0 port:0 vale0:em1
bdg_ctl [149] bridge:0 port:1 vale0:em1^
So my assumption that netmap(4) lives decapsuled from the well known
FreeBSD IP world.
How can I plug a jail's or vmm's artificial interface to a VALE virtual
switch, bridging frames to real-world via physical interfaces?
(the latter part should work with vale-ctl -h vale0:em1, but what
interface to use for jail(8) vnet.interface and how to create/attach?)

If you use bhyve/vmm, you can attach the VM TAP interface to the VALE
switch, as you would do for "em1". Regarding jails, I don't know exactly
how networking works there, but I guess epair(4) interface (or similar) are
used. If this is the case, then you would have one end of the epair only
visible in the jail, and the other end only visible in the "host"; then you
could attach the host end to a VALE switch again with "vale-ctl -a".
Unfortunately, the performance you would get in any case is not great,
because TAP and epair interface do not have netmap "native support".
Moreover, when using bhyve, you have to pay the cost of the emulation of
the vtnet device, since each packet passes through this device (other than
passing across netmap).

However, consider the following: a consistent netmap update is going to
happen in FreeBSD-CURRENT, in short. This is going to align the netmap code
which is now in FreeBSD to the code on the official github repository (
https://github.com/luigirizzo/netmap). Among the new features, there is a
new solution for bhyve networking, which will let you attach your bhyve VMs
directly to a VALE switch, without paying additional overheads related to
TAPs, epairs, and vtnet emulation. You can find additional information,
code and performance numbers here:
https://wiki.freebsd.org/SummerOfCode2016/PtnetDriverAndDeviceModel.

Cheers,
Vincenzo

Post by Harry Schmalzbauer
Thanks,
-harry
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-net

--
Vincenzo Maffione

Harry Schmalzbauer

2016-10-14 13:38:01 UTC

Post by Vincenzo Maffione
Hi,
Thanks for your feedback.

…

Post by Harry Schmalzbauer
Accidentally I found out that 'vale-ctl -n testif0' creates a artificial
testif0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
ether 00:be:eb:8d:f8:00
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
But I can't assign a IP address: 'ifconfig testif0 203.0.113.1/24'
ifconfig: ioctl (SIOCAIFADDR): Invalid argument
I guess couldn't geti the picture of the netmap(4) world yet.
Probably, testif0 is available only in netmap(4) world, not in "host world".
I'm assuming, because I found vale-ctl(-8)s "-h" switch.

Yes, those are the "persistent" VALE ports. They are a recent feature, and
probably you don't need to use them if you are going to play with Virtual
Machines and jails (see below).

Hello Vincenzo,

thank you very much for your help!!!

…

Post by Harry Schmalzbauer
How can I plug a jail's or vmm's artificial interface to a VALE virtual
switch, bridging frames to real-world via physical interfaces?
(the latter part should work with vale-ctl -h vale0:em1, but what
interface to use for jail(8) vnet.interface and how to create/attach?)

I'm familar with epair(4), but not with tap(4).
I don't understand the man page for tap, perhaps I should read pty(4)…
But I guess I don't have to know the details of tap(4), since you
confirmed that it can be connected to VALE.

So one could summarize:
VALE (as part of netmap(4)) can act as a if_bridge(4) replacement in
FreeBSD-10/11, keeping everything else involved untouched.
Please correct me if I'm wrong.

Post by Vincenzo Maffione
could attach the host end to a VALE switch again with "vale-ctl -a".
Unfortunately, the performance you would get in any case is not great,
because TAP and epair interface do not have netmap "native support".
Moreover, when using bhyve, you have to pay the cost of the emulation of
the vtnet device, since each packet passes through this device (other than
passing across netmap).

I understand, thanks.
In fact, I expected that at first hand, but have had some oddities with
if_bridge(4) some years ago, so I thought I'd better try something new ;-)
Can I expect any resource savings over if_bridge(4)? I guess if so, the
ammount isn't relevant considering the whole bhyve scenarium.

Post by Vincenzo Maffione
However, consider the following: a consistent netmap update is going to
happen in FreeBSD-CURRENT, in short. This is going to align the netmap code
which is now in FreeBSD to the code on the official github repository (
https://github.com/luigirizzo/netmap). Among the new features, there is a
new solution for bhyve networking, which will let you attach your bhyve VMs
directly to a VALE switch, without paying additional overheads related to
TAPs, epairs, and vtnet emulation. You can find additional information,
https://wiki.freebsd.org/SummerOfCode2016/PtnetDriverAndDeviceModel.

Thanks for that hint!
I guess it's about ptnetmap(4)? I read papers but haven't considered it
could be production-ready for FreeBSD in the near future.
It's extremely interesting and I'd love to be eraly adopter, but my
(ESXi) setups are currently doing well and I don't have spare time or
any business project to try out… :-(

Is it likely that there will a MFC happen? Or will it be a exclusive
12.0 feature? If ptnetmap will be MFCd I'll definitely give it a try
next summer and stay with 11.0 for my replacement machines for now.
Otherwise I'm unsure…

best,

-Harry

Vincenzo Maffione

2016-10-15 07:32:04 UTC

Post by Vincenzo Maffione
Hi,
Thanks for your feedback.

…

Yes, those are the "persistent" VALE ports. They are a recent feature,

and

Post by Vincenzo Maffione
probably you don't need to use them if you are going to play with Virtual
Machines and jails (see below).

Hello Vincenzo,
thank you very much for your help!!!
…

are

Post by Vincenzo Maffione
used. If this is the case, then you would have one end of the epair only
visible in the jail, and the other end only visible in the "host"; then

you
I'm familar with epair(4), but not with tap(4).
I don't understand the man page for tap, perhaps I should read pty(4)…
But I guess I don't have to know the details of tap(4), since you
confirmed that it can be connected to VALE.

It's not necessary to understand the details. However, a TAP device is
conceptually similar to the two ends of an epair, with the difference that
in the TAP a network interface (e.g. tap0) is conecptually "connected"
back-to-back to a file descriptor. The file descriptor is written/read by
the hypervisor (to inject/intercept packets to/from the network stack),
while the tap0 interface can be attached to if_bridge.

Post by Harry Schmalzbauer
VALE (as part of netmap(4)) can act as a if_bridge(4) replacement in
FreeBSD-10/11, keeping everything else involved untouched.
Please correct me if I'm wrong.

For simple cases yes. if_bridge may have features that are not supported by
netmap (i.e. configure ports as VLAN access ports). Moreover, if_bridge has
a interface (br0), whereas VALE bridges doesn't.

than

Post by Vincenzo Maffione
passing across netmap).

I don't think you can expect resource savings in terms of memory, since
netmap is trading off memory for performance, using pre-allocation rather
than dynamic allocation (of each packet).
I don't know what happens in terms of CPU saving, but using TAPs, epair and
vtnet makes the system as a whole not really efficient (as if_bridge), so
you may end up with nothing.

Post by Vincenzo Maffione
However, consider the following: a consistent netmap update is going to
happen in FreeBSD-CURRENT, in short. This is going to align the netmap

code

Post by Vincenzo Maffione
which is now in FreeBSD to the code on the official github repository (
https://github.com/luigirizzo/netmap). Among the new features, there is

Post by Vincenzo Maffione
new solution for bhyve networking, which will let you attach your bhyve

VMs

Post by Vincenzo Maffione
directly to a VALE switch, without paying additional overheads related to
TAPs, epairs, and vtnet emulation. You can find additional information,
https://wiki.freebsd.org/SummerOfCode2016/PtnetDriverAndDeviceModel.

Yes, it's ptnetmap. However, bhyve is going to have support for VALE ports
anyway (even without ptnetmap), as QEMU already does, so at least you will
be able to replace TAPs with VALE ports (while still using vtnet devices
for the VM).

Post by Harry Schmalzbauer
Is it likely that there will a MFC happen? Or will it be a exclusive
12.0 feature? If ptnetmap will be MFCd I'll definitely give it a try
next summer and stay with 11.0 for my replacement machines for now.
Otherwise I'm unsure…

No idea, I guess this is up to the committers. First step, however, is to
merge in CURRENT :)

Cheers,
Vincenzo

Post by Harry Schmalzbauer
best,
-Harry

--
Vincenzo Maffione

Harry Schmalzbauer

2016-10-15 09:42:06 UTC

…

Post by Harry Schmalzbauer
I'm familar with epair(4), but not with tap(4).
I don't understand the man page for tap, perhaps I should read pty(4)…
But I guess I don't have to know the details of tap(4), since you
confirmed that it can be connected to VALE.

Hi Vincenzo, thanks for your explanation!

Post by Harry Schmalzbauer
VALE (as part of netmap(4)) can act as a if_bridge(4) replacement in
FreeBSD-10/11, keeping everything else involved untouched.
Please correct me if I'm wrong.

Again, thank you for your time! (R)STP comes to my mind (which I don't
need any more). And I'm not sure if VALE really lacks that, but I guess
it wouldn't match VALEs philosophy/design at all…

…

Post by Vincenzo Maffione
https://github.com/luigirizzo/netmap). Among the new features, there is

Post by Vincenzo Maffione
new solution for bhyve networking, which will let you attach your bhyve

VMs

Oic, I wasn't aware that there will be a VALE-vtnet direct path! That is
really great news :-) And a big achievment for guests preferring
"standard" drivers, ptnetmap could limit the guest OS choice I guess.

For now, I'm happy having been in touch with netmap(4) – at least with a
very little fraction of natmap – but I'll stay the legacy way utilizing
if_bridge(4) and see if there are still oddities and try to find some
time to track them down (involving LACP, VLANs, Jumbo-Frames and IPv6 –
that was the problematic constellation)

Since I have extra PHYs, I can do PCIe-passthrough like before (with
ESXi) for some special guests. I'm looking forward to find out how this
works with bhyve!

Best,

-Harry

Vincenzo Maffione

2016-10-16 09:29:03 UTC

Post by Harry Schmalzbauer
…

It's not necessary to understand the details. However, a TAP device is
conceptually similar to the two ends of an epair, with the difference

that

Post by Vincenzo Maffione
in the TAP a network interface (e.g. tap0) is conecptually "connected"
back-to-back to a file descriptor. The file descriptor is written/read by
the hypervisor (to inject/intercept packets to/from the network stack),
while the tap0 interface can be attached to if_bridge.

Hi Vincenzo, thanks for your explanation!

Post by Harry Schmalzbauer
VALE (as part of netmap(4)) can act as a if_bridge(4) replacement in
FreeBSD-10/11, keeping everything else involved untouched.
Please correct me if I'm wrong.

For simple cases yes. if_bridge may have features that are not supported

Post by Vincenzo Maffione
netmap (i.e. configure ports as VLAN access ports). Moreover, if_bridge

has

Post by Vincenzo Maffione
a interface (br0), whereas VALE bridges doesn't.

Again, thank you for your time! (R)STP comes to my mind (which I don't
need any more). And I'm not sure if VALE really lacks that, but I guess
it wouldn't match VALEs philosophy/design at all…

Well, VALE is a programmable switch, meaning that you can plug in a custom
forwarding function (which is just a C function). The default forwarding
function implements an L2 learning bridge without STP, because it is
thought to be used as the preferred hypervisor virtual switch, so no L2
connectivity loops, and the STP comes with an overhead. If you really wish
to have VALE with STP, you could define your own custom forwarding function
that implements it.

Post by Harry Schmalzbauer
…

Post by Vincenzo Maffione
https://github.com/luigirizzo/netmap). Among the new features, there

Post by Harry Schmalzbauer
a

Post by Vincenzo Maffione
new solution for bhyve networking, which will let you attach your bhyve

VMs

Post by Vincenzo Maffione
directly to a VALE switch, without paying additional overheads related