Cloud1 is not forwarding ARP packets

[joris]

Hi there,

I've kind of a strange problem and am hoping someone will know what's going on. I'm trying to connect a external machine to my EVE topology, simple as that.

My EVE-NG is build from OVA on VMware ESXi, the host I'm trying to connect to also.

I've change the /etc/network/interfaces file to reflect this:

Code: Select all

iface eth1 inet manual
auto pnet1
iface pnet1 inet static
    address 10.10.100.1/24
    netmask 255.255.255.0
    bridge_ports eth1
    bridge_stp off

The VM on the VLAN attached to eth1/pnet1 does respond to the EVE machine, so that part is fully functional, as expected:

Code: Select all

root@eve-ng:~# ping 10.10.100.2 -c 1
PING 10.10.100.2 (10.10.100.2) 56(84) bytes of data.
64 bytes from 10.10.100.2: icmp_seq=1 ttl=64 time=0.180 ms

--- 10.10.100.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.180/0.180/0.180/0.000 ms


root@eve-ng:~# ip neigh show 10.10.100.2
10.10.100.2 dev pnet1 lladdr 00:50:56:94:77:e0 REACHABLE

Once I started a couple of instances within EVE-NG I see the bridge is getting filled with these virtual interfaces:

Code: Select all

root@eve-ng:~# brctl show pnet1
bridge name	bridge id		STP enabled	interfaces
pnet1		8000.00505694deb4	no		eth1
							vunl0_2_0
							vunl0_7_0

The strange part is that it seems that the ARP entries are not forwarded toward the instance on EVE-NG. While capturing the eth1 interface, I see the arp requests going out and receiving the reply:

Code: Select all

root@eve-ng:~# tcpdump -s0 -pni eth1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes

14:56:34.129862 ARP, Request who-has 10.10.100.2 tell 10.10.100.3, length 46
14:56:34.129992 ARP, Request who-has 10.10.100.2 tell 10.10.100.3, length 46
14:56:34.130022 ARP, Reply 10.10.100.2 is-at 00:50:56:94:77:e0, length 46

But the kicker is, this reply is not forwarded to the interface nor the bridge:

Code: Select all

root@eve-ng:~# tcpdump -s0 -pni vunl0_2_0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vunl0_2_0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:07:31.711695 LLDP, length 161: localhost
15:07:33.033970 ARP, Request who-has 10.10.100.2 tell 10.10.100.3, length 46
15:07:33.034065 ARP, Request who-has 10.10.100.2 tell 10.10.100.3, length 46
15:07:34.032684 ARP, Request who-has 10.10.100.2 tell 10.10.100.3, length 46
15:07:34.032798 ARP, Request who-has 10.10.100.2 tell 10.10.100.3, length 46

When I manually add the ARP entry in the router (arista in this case) it does fully function.
It does seem that the bridge is dropping the ARP entries, although I cannot find proof of this nor can I fix it by editting procfs of sysfs.

Could anyone tell me what it might be or what I could do? I would appreciate the help.

Best regards,

Joris

[Uldis (UD)]

first question, is your ESXi server is connected to LAN using dual NIC bundle ???
If yes then it is an issue:
https://github.com/InfraSIM/infrasim-co ... -situation

If you have single NIC connected to LAN, then usually problem is that is not enabled promiscue mode on Esxi vswitch.

All this stuff is described in eve Cookbook section 10.
http://www.eve-ng.net/images/EVE-COOK-BOOK-1.7.pdf

[joris]

Hi Uldis,

Thanks for your reply!

first question, is your ESXi server is connected to LAN using dual NIC bundle ???

Nope, the VMware environment does not use bundling.

If you have single NIC connected to LAN, then usually problem is that is not enabled promiscue mode on Esxi vswitch.

Promiscuous mode is turned on. The issue is not that the packets aren't arriving at the EVE-NG, they are (as shown in the tcpdump). It's just that the linux bridge does not seem to forward those ARP requests to the Arista instance interface in the bridge. The same goes for DHCP-offers is found out yesterday, the arrive at eth1, but are not forwarded.

[Uldis (UD)]

I dont know then..
Sometimes simple EVE reboot can help, especially if upyiu made some settings on the esxi vswitch...
then I recommend reboot EVE
and reconnect your devices to cloud1...

Do you using esxi vswitch with vlan tag?
My experience I never using on vSwitch any vlan tags, it is just like pipe, allowed all 4094 LANs on it...
Second for such external VM connection to cloud1, you neednot configure any IP on your pnet1 at all...
vSwitch acting simply as the bridge

And no issues such way
BTW what is that external OVA you trying connect?


Uldis

[joris]

VMware is using VLAN's, but I'm confident this works, especially since with a static ARP entry everything is working as expected.

I've already tried rebooting and reconnecting (even recreating a whole new lab) the Cloud1 instance to nodes.

I'll search some further, i guess it's some kind of security / isolation feature in the kernel which I haven't found yet.

Thanks for the help so far!

[Uldis (UD)]

Anyway you can live chat with me in EVE Live Help desk
http://www.eve-ng.net/live-helpdesk
Use your google account or create new for chat.
My chat name UD-EVE

Uldis

[sven]

Hi,

I'm facing the exact same issue. Did this get resolved?

Thanks,
Sven.

[joris]

Hi Sven,

Yes, it appears it was a bug in VMware.

Something with NIC teaming in VMware on the distributed switch that resulted in this behaviour.
I (or actually my VMware colleague) fixed it by providing a new (DV)switch for me without the teaming or with LACP, not sure which, both should work.

Hope this helps!

Best regards,

Joris

[sven]

Hi Joris,

thanks for replying.

The Vswitch I created has only one physical interface (vmnet1) attached to it, but it's in 'down' state (no cable plugged in). I was hoping that that isn't an issue since the switch is virtual.

So just for the fun of it I new created another vSwitch & Port Group without any vmnic in it.. and bingo.. it works

Thanks,
Sven.

[darko]

We are facing this exact same issue. What was the fix for this problem?