EPG w/ Intra Isolation VLAN Encap

a12288 · ‎11-12-2021

Good day, community.

I have a strange problem here, one Oracle RAC VM server has certain virtual silent IPs, somehow after DBA did a upgrade ACI cannot reach certain silent IPs. We can still reach real IP but just cannot reach virtual IPs, they are using the same MACs.

TAC was involved and so far we have not found the "root cause" yet, so far we know:

1. ACI does send out ARP request but VM never receive it.

2. If silent VM initiate some traffics, then ACI will learn those silent VM but they will be timed out soon.

One particular both TAC and us cannot understand is that, the packet capture did at VPC ports to Fabric Interact we saw some "incorrect" VLAN encap.

For all the ARP requests which ACI never receives response are encapsulated with a non-exist VLAN, 519, we have no where to define VLAN519 in the fabric. Our P VLANs are 2801 / 2802.

Has anyone experienced anything similar?

Leo

Sergiu.Daniluk · ‎11-13-2021

Hi @a12288

I think what you see in the erspan is something cosmetic, due to this defect:

CSCvm88208 - ERSPAN shows ACI sending ARP GLEAN over 'unknown' VLAN (cosmetic) [Flood in Encap]
"""

In the SPAN capture of ACI leaf, we see GLEANs sourced from the BD SVI - the ERSPAN header lists these GLEANs as going over a seemingly 'unknown' VLAN in the 500 range (e.g. 515, 529, etc.)

"""

This is just a cosmetic issue of the ERSPAN. The vlan in which ARP is sent should be the real vlan.

So far, you know that - broadcast in the direction ACI -> VM is broken.

Few questions which can help in continuing the tshoot:

1. Is unicast also problematic? You can test by configuring static ARP for GW on the VM, and also static EP in ACI for the VIP.

2. All the VMs which are configured with VIP are unreachable? If not, what are the differences between the working and non working VMs?

Stay safe,

Sergiu

a12288 · ‎11-13-2021

Thanks, Sergiu.

That cosmetic bug explains the "wrong" vlan encap.

1. Is unicast also problematic? You can test by configuring static ARP for GW on the VM, and also static EP in ACI for the VIP.

Unicast is having problem, we cannot reach those IPs, and ACI did not learn them as EP, and according to our Oracle DBA those IPs are silent and will receive traffics only but won't initiate traffics. We noticed VM did not receive ARP request during troubleshooting.

We can "force" ACI to learn those IPs by manually ping from those silent IPs i.e. ping -i command, then ACI will learn those IPs but eventually it will be timed out (idle time out) as this is our stage Oracle RAC without production traffic.

2. All the VMs which are configured with VIP are unreachable? If not, what are the differences between the working and non working VMs?

It's two VMs with multiple IPs, those IPs on the same VM share the same MAC. I post the ifconfig and ping result from Leaf (VPC to FI to UCS to VM) at bellow.

root@ceadv-idracpd1 sysconfig]# ifconfig -a

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 172.18.34.30 netmask 255.255.255.0 broadcast 172.18.34.255

ether 00:50:56:84:da:12 txqueuelen 1000 (Ethernet)

RX packets 101279113 bytes 130270443339 (121.3 GiB)

RX errors 0 dropped 18 overruns 0 frame 0

TX packets 93740941 bytes 1568705984966 (1.4 TiB)

TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

LEAF113# iping -V CCS:VRF-CCS 172.18.34.30
PING 172.18.34.30 (172.18.34.30) from 172.18.34.1: 56 data bytes
64 bytes from 172.18.34.30: icmp_seq=0 ttl=64 time=0.778 ms
64 bytes from 172.18.34.30: icmp_seq=1 ttl=64 time=0.492 ms
64 bytes from 172.18.34.30: icmp_seq=2 ttl=64 time=0.524 ms

eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 172.18.34.31 netmask 255.255.255.0 broadcast 172.18.34.255

ether 00:50:56:84:da:12 txqueuelen 1000 (Ethernet)

LEAF113# iping -V CCS:VRF-CCS 172.18.34.31
PING 172.18.34.31 (172.18.34.31) from 172.18.34.1: 56 data bytes
Request 0 timed out
Request 1 timed out

eth0:3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 172.18.34.33 netmask 255.255.255.0 broadcast 172.18.34.255

ether 00:50:56:84:da:12 txqueuelen 1000 (Ethernet)

LEAF113# iping -V CCS:VRF-CCS 172.18.34.33
PING 172.18.34.33 (172.18.34.33) from 172.18.34.1: 56 data bytes
Request 0 timed out
Request 1 timed out

eth0:5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 172.18.34.34 netmask 255.255.255.0 broadcast 172.18.34.255

ether 00:50:56:84:da:12 txqueuelen 1000 (Ethernet)

LEAF113# iping -V CCS:VRF-CCS 172.18.34.34
PING 172.18.34.34 (172.18.34.34) from 172.18.34.1: 56 data bytes
64 bytes from 172.18.34.34: icmp_seq=0 ttl=64 time=0.702 ms
64 bytes from 172.18.34.34: icmp_seq=1 ttl=64 time=0.573 ms

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000

inet 172.16.6.106 netmask 255.255.255.0 broadcast 172.16.6.255

ether 00:50:56:84:99:37 txqueuelen 1000 (Ethernet)

RX packets 1380941429 bytes 15551753132787 (14.1 TiB)

RX errors 0 dropped 192 overruns 0 frame 0

TX packets 1490506028 bytes 2797307512454 (2.5 TiB)

TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000

inet 10.199.34.30 netmask 255.255.255.0 broadcast 10.199.34.255

ether 00:50:56:84:4f:58 txqueuelen 1000 (Ethernet)

RX packets 1018312243 bytes 1498559448745 (1.3 TiB)

RX errors 0 dropped 212 overruns 0 frame 0

TX packets 988109652 bytes 1288270584280 (1.1 TiB)

TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth2:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000

inet 169.254.23.1 netmask 255.255.224.0 broadcast 169.254.31.255

ether 00:50:56:84:4f:58 txqueuelen 1000 (Ethernet)

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 16436

inet 127.0.0.1 netmask 255.0.0.0

loop txqueuelen 1000 (Local Loopback)

RX packets 572509891 bytes 358567272659 (333.9 GiB)

RX errors 0 dropped 0 overruns 0 frame 0

TX packets 572509891 bytes 358567272659 (333.9 GiB)

TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

[root@ceadv-idracpd2 ~]# ifconfig -a

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 172.18.34.35 netmask 255.255.255.0 broadcast 172.18.34.255

ether 00:50:56:84:9b:09 txqueuelen 1000 (Ethernet)

RX packets 59733998 bytes 18130605769 (16.8 GiB)

RX errors 0 dropped 0 overruns 0 frame 0

TX packets 59440714 bytes 129011683492 (120.1 GiB)

TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

LEAF113# iping -V CCS:VRF-CCS 172.18.34.35
PING 172.18.34.35 (172.18.34.35) from 172.18.34.1: 56 data bytes
64 bytes from 172.18.34.35: icmp_seq=0 ttl=64 time=0.864 ms
64 bytes from 172.18.34.35: icmp_seq=1 ttl=64 time=0.539 ms

eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 172.18.34.36 netmask 255.255.255.0 broadcast 172.18.34.255

ether 00:50:56:84:9b:09 txqueuelen 1000 (Ethernet)

LEAF113# iping -V CCS:VRF-CCS 172.18.34.36
PING 172.18.34.36 (172.18.34.36) from 172.18.34.1: 56 data bytes
Request 0 timed out
Request 1 timed out

eth0:2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 172.18.34.32 netmask 255.255.255.0 broadcast 172.18.34.255

ether 00:50:56:84:9b:09 txqueuelen 1000 (Ethernet)

LEAF113# iping -V CCS:VRF-CCS 172.18.34.32
PING 172.18.34.32 (172.18.34.32) from 172.18.34.1: 56 data bytes
64 bytes from 172.18.34.32: icmp_seq=0 ttl=64 time=0.713 ms
64 bytes from 172.18.34.32: icmp_seq=1 ttl=64 time=0.572 ms

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000

inet 172.16.6.107 netmask 255.255.255.0 broadcast 172.16.6.255

ether 00:50:56:84:bd:99 txqueuelen 1000 (Ethernet)

RX packets 573244575 bytes 4262157336785 (3.8 TiB)

RX errors 0 dropped 202 overruns 0 frame 0

TX packets 661519779 bytes 1036513767848 (965.3 GiB)

TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000

inet 10.199.34.35 netmask 255.255.255.0 broadcast 10.199.34.255

ether 00:50:56:84:f1:a6 txqueuelen 1000 (Ethernet)

RX packets 988724880 bytes 1288390562326 (1.1 TiB)

RX errors 0 dropped 225 overruns 0 frame 0

TX packets 1009989469 bytes 1497936272377 (1.3 TiB)

TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth2:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000

inet 169.254.12.73 netmask 255.255.224.0 broadcast 169.254.31.255

ether 00:50:56:84:f1:a6 txqueuelen 1000 (Ethernet)

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 16436

inet 127.0.0.1 netmask 255.0.0.0

loop txqueuelen 1000 (Local Loopback)

RX packets 205217881 bytes 61272919769 (57.0 GiB)

RX errors 0 dropped 0 overruns 0 frame 0

TX packets 205217881 bytes 61272919769 (57.0 GiB)

TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Leo