NAT-like Behavior on VNET

[Torc]

I have an odd issue I haven't seen before and so far I haven't seen any references to it on the forum. I have a baremetal install of EVE-NG (I was on 2.0.3-60 before upgrading to 2.0.3-68 with no change in behavior). I have two XRv appliances connected with a standard bridge link.

I started troubleshooting why I couldn't ping across the link and using the remote packet capture feature, I see on the side originating the echo request, the source IP matches what is configured on the XRv appliance.

However, the packet capture on the destination side of the link shows the source IP address has been rewritten to the host server's IP address. Even though the IP address is rewritten, the source mac address still matches the address listed inside the XRv appliance (not the physical mac of the host server). In a BGP packet, the TCP checksum has been recalculated but the source/destination ports are unmodified.

I checked the brctl output and see that only the two "vunl" interfaces tied to the bridge network. At this point, I'm not sure where else to look. In the output below, the 172.17.1.0/30 network is the subnet used by the XRv appliances, and the 10.100.100.200 IP address is the host EVE-NG NIC (pnet0).

Any pointers would be appreciated. Thanks!

Code: Select all

||/ Name                                                Version                        Architecture                   Description
+++-===================================================-==============================-==============================-===========================================================================================================
ii  eve-ng                                              2.0.3-68                       amd64                          A new generation software for networking labs.

Code: Select all

# brctl show vnet1_16
bridge name     bridge id               STP enabled     interfaces
vnet1_16          8000.56fe3c381042       no              vunl1_14_3
                                                          vunl1_1_3
# brctl showmacs vnet1_16
port no mac addr                is local?       ageing timer
  1     56:fe:3c:38:10:42       yes                0.00
  1     56:fe:3c:38:10:42       yes                0.00
  2     8a:b9:af:1c:2f:a9       yes                0.00
  2     8a:b9:af:1c:2f:a9       yes                0.00

SOURCE PACKET

Frame 509: 62 bytes on wire (496 bits), 62 bytes captured (496 bits) on interface 0
Ethernet II, Src: 50:01:00:0e:00:03 (50:01:00:0e:00:03), Dst: 50:01:00:01:00:03 (50:01:00:01:00:03)
Destination: 50:01:00:01:00:03 (50:01:00:01:00:03)
Source: 50:01:00:0e:00:03 (50:01:00:0e:00:03)
Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 172.17.1.1, Dst: 172.17.1.2
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0xc0 (DSCP: CS6, ECN: Not-ECT)
Total Length: 48
Identification: 0x6e9b (28315)
Flags: 0x00
Fragment offset: 0
Time to live: 1
Protocol: TCP (6)
Header checksum: 0xf047 [validation disabled]
[Header checksum status: Unverified]
Source: 172.17.1.1
Destination: 172.17.1.2
[Source GeoIP: Unknown]
[Destination GeoIP: Unknown]
Transmission Control Protocol, Src Port: 58430, Dst Port: 179, Seq: 0, Len: 0
Source Port: 58430
Destination Port: 179
[Stream index: 56]
[TCP Segment Len: 0]
Sequence number: 0 (relative sequence number)
Acknowledgment number: 0
Header Length: 28 bytes
Flags: 0x002 (SYN)
Window size value: 16384
[Calculated window size: 16384]
Checksum: 0xc989 [unverified]
[Checksum Status: Unverified]
Urgent pointer: 0
Options: (8 bytes), Maximum segment size, Window scale, End of Option List (EOL)
[SEQ/ACK analysis]

DESTINATION PACKET

Frame 157: 62 bytes on wire (496 bits), 62 bytes captured (496 bits) on interface 0
Ethernet II, Src: 50:01:00:0e:00:03 (50:01:00:0e:00:03), Dst: 50:01:00:01:00:03 (50:01:00:01:00:03)
Destination: 50:01:00:01:00:03 (50:01:00:01:00:03)
Source: 50:01:00:0e:00:03 (50:01:00:0e:00:03)
Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 10.100.100.200, Dst: 172.17.1.2
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0xc0 (DSCP: CS6, ECN: Not-ECT)
Total Length: 48
Identification: 0x6e9b (28315)
Flags: 0x00
Fragment offset: 0
Time to live: 1
Protocol: TCP (6)
Header checksum: 0x908c [validation disabled]
[Header checksum status: Unverified]
Source: 10.6.2.200
Destination: 172.17.1.2
[Source GeoIP: Unknown]
[Destination GeoIP: Unknown]
Transmission Control Protocol, Src Port: 58430, Dst Port: 179, Seq: 0, Len: 0
Source Port: 58430
Destination Port: 179
[Stream index: 29]
[TCP Segment Len: 0]
Sequence number: 0 (relative sequence number)
Acknowledgment number: 0
Header Length: 28 bytes
Flags: 0x002 (SYN)
Window size value: 16384
[Calculated window size: 16384]
Checksum: 0x69ce [unverified]
[Checksum Status: Unverified]
Urgent pointer: 0
Options: (8 bytes), Maximum segment size, Window scale, End of Option List (EOL)
[SEQ/ACK analysis]

[Uldis (UD)]

Can you simply you questions what is actually not working?

You have simply 2 XRV connected with each other?

Or you have XRv connected to EVE cloud interface?
to touch eve inside no need at all....

Just tested your scenario, I hope, and works perfectly...packet capture as well and all is pinging each other
Destination packet capturing showing perfect info, source MAC and IP are correct

show your topology pls

[ecze]

please show us your topology.....

E.

[Torc]

Sorry for not sharing the topology, it had a lot of proprietary info in it. However, I was I was able to replicate the same behavior in a simple two-node topology. I've attached a screenshot of this.





This is the testing I've done today.

On the baremetal server, I have:

-- Created the two node XRv topology and verified that I have the problem when I use the same IP addressing on the XRv appliances (172.17.1.0/30).
-- Replaced an XRv appliance with a CSR1000V appliance and used the same subnet. Confirmed I had the same issue.
-- If I change the subnet to 172.16.1.0/30, the issue disappears, the nodes can ping each other, and the packet captures look correct.
-- I exported the configs and enabled the startup config option. I cloned the lab and did the same testing under a different user with the same result.

I exported the lab and created a new server on an ESXI host via the OVA image. When I tested it with the topology and config that had the issue on the baremetal server, I did not have the problem. Besides being a virtual server, the only other things that are different are the host server IPs and the fact the baremetal server was using a bond interface since it's a Cisco UCS blade and I set up NIC teaming on the two fabric uplinks.

I am continuing with my testing using a different subnet, but I'm still scratching my head about why I ran into this on the baremetal server. I was able to look at the linux bridge config, but the mac-addresses I saw didn't match the mac addresses in the appliances. I'm not an expert at Linux virtual networking but I wonder if there's some other overlay that I'm missing, maybe the TAP interface.

[ecze]

now show us the out put of :

iptables -L -n -v -t nat

E.

[Torc]

Wow, there it is. I can't believe I didn't check that. I just knew it was a completely fresh install of Ubuntu and didn't expect to see much going on in here.

I guess when I tried to follow the docker installation guide (before the support was removed), it added that rule. I'll be the first to admit I have a lot to learn about routing on Linux, but I thought since the interfaces were bridged, layer 3 processing (like NAT) wouldn't be a concern.

Thanks!

Code: Select all

# iptables -L -n -v -t nat
Chain PREROUTING (policy ACCEPT 598K packets, 29M bytes)
 pkts bytes target     prot opt in     out     source               destination         
 4200  252K DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 9743 packets, 964K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 155K packets, 9342K bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 181K packets, 11M bytes)
 pkts bytes target     prot opt in     out     source               destination         
  209 10755 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0           

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0