Same MAC address on multiple physical L3 interfaces, Cisco 4503 (SUP7L-E, running IOS-XE 03.04.02.SG)

[bazzaroo]

Hi All,

I've encountered an interesting scenario today relating to some recently commissioned pairs of 4500 series L3 switches in a VSS configuration. Since deployment, there have been reports of random disconnects from users accessing resources at other sites.

While investigating today, I've noticed that the MAC address of every port configured as an L3 interface on each VSS is the same and there doesn't appear to be a way to change it.

Topology-wise, there's a single SP router at each site with one L3 interface on chassis 1 of the VSS connected to g0/0 on the SP's router with it's own /30 subnet. One L3 interface on chassis 2 of the VSS is connected to g0/1 on their SP's router, again with it's own /30 subnet to provide a level of resilience. EIGRP is running on the LAN side to advertise routes, so the SP router is aware of two equal cost routes and is balancing traffic across them. However, I've had confirmation today that the ARP table on the SP router has the same MAC address entry for the next hop addresses for both /30 networks, which is what I think is causing the disconnects.

I've had a good hunt around to see if there's any way to change this behaviour but I've come up blank. Does anyone know if it's possible to change this? I've got a few ideas up my sleeve to work around it but they all involve making one of the interfaces unusable in some way and I'd really like to continue having both up if possible.

Thanks in advance for any help/advice,

Barry

[Reza Sharifi]

Hi Barry,

The BIA address for all L3 and SVI interfaces is the same.  This is the default functionality of all 6500, 4500 switches.  You can change the mac access for each interface by using command "mac-address xxxx.xxxx.xxxx" under the interface.

What makes you think that the disconnect is because of the MAC address?

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6000-series-switches/41263-catmac-41263.html#topic1

HTH

[bazzaroo]

Hi Reza,

Thanks for your input and sorry for the delay in coming back to you. The mac-address sub-interface command doesn't seem to be available neither on L2 nor L3 interfaces: -

switch(config-if)#mac-?
% Unrecognized command
switch(config-if)#mac-

Any other suggestions?

My theory regarding the disconnections, having thought about it, might be a bit sketchy. The flow as I understand it would be as follows: -

Packet arrives at CPE router: -
 L2 header stripped.
 Destination address of packet read and a lookup against routing table performed.
 Best matching route found, next hop IP address identified.
 ARP table consulted for L3->L2 mapping. If no matching entry, ARP broadcast performed.
 New L2 header with new destination L2 MAC address added to packet and forwarded on.

So in my case, the ARP table on the CPE looks like this: -

Protocol Address Age (min) Hardware Addr Type Interface
Internet 99.99.98.170 - a46c.2a8f.3461 ARPA GigabitEthernet0/1
Internet 99.99.98.171 134 0025.8429.fcc0 ARPA GigabitEthernet0/1
Internet 10.1.2.3 127 0008.e3ff.fc04 ARPA GigabitEthernet0/0
Internet 10.1.2.4 - a46c.2a8f.3460 ARPA GigabitEthernet0/0
Internet 10.1.2.7 114 0008.e3ff.fc04 ARPA GigabitEthernet0/2
Internet 10.1.2.8 - a46c.2a8f.3462 ARPA GigabitEthernet0/2

0008.e3ff.fc04 appears twice, which is the MAC address of the connected L3 switch. When the router performs the routing lookup for, say, 10.8.30.10 and finds the next hop as 10.1.2.3, when it consults the ARP table for the corresponding L2 address it will find 0008.e3ff.fc04. Because it knows about that via g0/0, will it always use that interface or could it potentially also be using g0/2 (same MAC address but associated with 10.1.2.7)? I guess the end product is the same whichever interface is used - the frame will still get to the device on the end of 0008.e3ff.fc04. But could this be causing problems or am I barking up the wrong tree?

Thanks,

Barry

[bazzaroo]

Hello,

Some more info to add to this...

When the random packet loss occurs, if I run a traceroute from my machine. It gets as far as the far end 4500 pair. I've checked routing, all fine. I've checked the arp table, there's an ARP entry for the target address of the device I'm trying to reach. I've checked the CEF FIB and there's an entry for the target address with a 32-bit mask. From the other end, while remotely connected to a device that is contactable on the same LAN, I am not able to trace to my machine from the remote device; the traffic gets as far as the 4500 and goes nowhere. It's not just this one device either, there's more than one and they're all completely different types of device. CPU load on the 4500 pair is very low.

Regards,

Barry