Nexus 3548 SVI and HSRP routing

[actyler1001]

Hello Cisco Community!  I'm in the process of designing a datacenter routing change and using SVIs with HSRP to handle routing rather than an edge UTM device.  I ran into the below behavior while testing and wondered if anyone could explain why I might be seeing this...

host contained within a VLAN and routed out by a pair of Nexus switches using SVI/HSRP.

NX1 SVI IP: 10.10.35.2/24

NX1 SVI IP: 10.10.35.3/24

HSRP VIP: 10.10.35.1

I was testing failover and monkeyed with the priority of the HSRP group, all seems to work no matter which Nexus switch is the active HSRP partner, works as expected.

If I shut down one of the SVIs to simulate a failure, all seems to work.  Host inside VLAN is able to route out.  I can even enable the SVI, then disable the SVI on the second Nexus switch and all is still well.

If I shut down both SVIs, traffic stops for the host inside the VLAN, totally expected.  However, I can’t get things flowing again until both SVIs are enabled.  If just enabling one, the host still fails to route.

Have any idea why that would be happening?

Here is a snip of the vPC config.

vcp domain 1

peer-switch

role priority (1|2) <- depending on what switch you are looking at in the pair

peer-keepalive destination x.x.x.x

peer-gateway

layer3 peer-router

auto-recovery

I tried to make this bite sized and not include the entirety of the config.  Hopefully this description has the important parts, but if you need more info before you can comment, just let me know.  Thanks again for your thoughts!

[Reza Sharifi]

Hi,

Can you post the HSRP config from both nexus switches?

Do you have the same issue if you use VRRP instead?

HTH

[actyler1001]

Hi Reza.  I have not tried VRRP yet.  I can if you think it would be a good idea.

Here is the HSRP configuration along with the SVI config too..

NX1
interface Vlan22
  description LAN-VLAN-22
  no shutdown
  bandwidth 40000000
  vrf member LAN-VRF-28
  no ip redirects
  ip address 10.10.35.2/24
  ip ospf authentication message-digest
  ip ospf message-digest-key 1 md5 3 <redacted>
  ip router ospf 1 area 0.0.0.0
  hsrp version 2
  hsrp 22
    authentication md5 key-string <Redacted>
    preempt delay minimum 30
    ip 10.10.35.1

NX2
interface Vlan22
  description LAN-VLAN-22
  no shutdown
  bandwidth 40000000
  vrf member LAN-VRF-28
  no ip redirects
  ip address 10.10.35.3/24
  ip ospf authentication message-digest
  ip ospf message-digest-key 1 md5 3 <Redacted>
  ip router ospf 1 area 0.0.0.0
  hsrp version 2
  hsrp 22
    authentication md5 key-string <Redacted>
    preempt delay minimum 30
    priority 90
    ip 10.10.35.1

[Reza Sharifi]

Config looks good. So, in this case NX1 it primary HSRP and NX2 is the backup. After shutting down both SVIs, when you enable NX1 SVI no traffic gets forwarded?

HTH

[actyler1001]

That's correct.  However if I use the command "clear mac address-table dynamic interface Po1" on the vPC peer-link it seems to get things working again.  The results have been somewhat intermittent, but what seems to break routing is shutting down both SVIs and bringing one or the other online.  It seems to be more problematic when the NX2 switch SVI is the first one to be enabled.

It's also worth mentioning that while the host is unable to route out of the VLAN using the HSRP VIP, I can successfully complete a ping from the Nexus router using something like this:

ping 10.10.10.10 vrf LAN-VRF-28 source 10.10.35.3

So this looks like the router itself is able to route out of the SVI, but the host using HSRP cannot.

-Adam

Regards,

Adam Tyler

[Reza Sharifi]

Adam,

Is the host a server or a laptop? Is it connected to both Nexus or just one of them? Are the configs for the host port alike if the 2 switches are vPCed together? 

Once more thing, is it possible to configure a simple hsrp with no VRF and no authentication and test?

HTH

[actyler1001 · Accepted solution]

The "host" is a VMware VM.

I just got off the phone with Cisco TAC regarding this issue.  Good conversation.  Basically the answer was, don't shut down SVIs.  Lol..  It isn't a Cisco accepted failure simulation.  With "peer-gateway" enabled on the vPC, the peer Nexus device is still going to try and forward a packet destined for the other Nexus device to the disabled SVI.  It doesn't know it has been disabled and weirdness occurs.

They still recommend enabling peer-gateway as a best practice, it just doesn't work right if you admin shut down an SVI interface.

So..  My failure testing scenarios are going to have to include a different approach.

Regards,

Adam Tyler

[Reza Sharifi]

Thanks for the update Adam!

Reza

[actyler1001]

Incidentally, running the “clear mac address-table interface Po1” command on the vPC peer link port channel seems to have an impact on this behavior. Hm…

-Adam