N7k to 6500 OSPF EXSTART (non MTU issue)

[naser.ranjha]

Hi,

I am posting here after raking my brain and the internet for a possible answer short of opening a TAC case.

So as in the diagram attached I have a pair of 6500 and a pair of N7K talk over a vlan on non vpc trunks between the 4 switches, although the Vlan itself exists on some N5Ks down stream, but the SVIs on the 6500s and N7K are physically talking to each other via non-vpc trunks.

1 - I have IP connectivity among all the 4 swtiches above on VLAN 100.

2 - I am running OSPF on broadcast

3 - 6500-One is the DR, N7k-One is the BDR, 6500-2 is in Full adjacency with the DR and BDR and 2way state with N7K-Two as N7K-Two is niether a DR nor a BDR.

4 - All is good except adjacency between 6500-One (DR) and N7K-Two is getting stuck at EXSTART and 6500-One tears down the neighourship after too many retransmission (about 25).

5 - Nope its not the MTU mismatch I have run debugs and both are operating at 1500.

6 - There is no unicast issue I can ping to both the 6500-One (DR) and the N7k-Two sourcing from SVI 100.

7 - Both sides never go above EXSTART, none of the two problem switches go to Exchange state.

8 - Both sides claim to be master with the flag set at 0x7 and initiate DBD exchange.

9 - The debug shows that 6500-One (DR) is receiving unicast from N7K-Two, however there is no indication that N7K-2 is getting any unicasts from 6500-One at EXSTART, although I am able to ping to N7k-Two sourcing from 6500-One. Also N7k-Two is able to form Full and stable adjacency with the the BDR N7K-One, and stable 2-way with non DR/non BDR 6500-Two.

Will appreciate your input fellows.

[Vasilii Mikhailovskii]

Hello.

IGP was not supported on VPC-enabled VLANs (VL100 in your case) of Nx7K having peer-gateway feature enabled.

It became supported as "Layer 3 over vPC for F2, F2E and F3 modules" since 7.2(0)D1(1)

Please provide your Nx7K OS version along with "show vpc br" for futher discussion.

[naser.ranjha]

Hi Mikhail, 

I am running 6.2

Thanks, you have pointed me to the right direction and very helpful knowing there that Cisco has come up with a band aid in their newer  OS. Also what I noticed is that there is a dearth of info on why peer-gateway does it what it does and why TTL is lost when the peering hosts are in the same subnet across the peer link.

Thinking about it, it makes sense that one of the OSPF neighbours across the VPC peer link could establish an OSPF 2-way adjacency as it was all done over multicast, only when DBD needed to be exchange did the problem DR couldn't send unicast to the N7K-Two.

So my understanding apart from its well published function with respect to FHRP is that the peer gateway function rather than switching the frame from N7K-One to N7K-Two IP forwards (like a gateway does) to N7K-Two thus the unicast losing its TTL of 1 to 0 and packet getting dropped even though the next hop SVI is also in the same subnet. This is done so not to populate the CAM table on N7K-Two with the MAC address of the source host residing in an orphan port in N7K-One, which in a VPC set up would cause loops.

This is also why I found that in an non-VPC  HSRP setup on N7Ks with peer links,  If  one was to shut down the SVI on the primary peer the secondary HSRP after assuming the HSRP primary role won't be able to forward via the peer link as the forwarding is done at Layer 3 rather than layer 2 and with the SVI down on the primary there won't be any next hop l3 interface to forward onto.

[Vasilii Mikhailovskii]

Hello.

So, you are running NXOS 6.2.

If you have peer-gateway been enabled on VPC, then IGP routing is not supported and this is explicitly mentioned in design documents for Nx7K.

Once you upgrade to 7.2(0)D1(1)+ and have the hardware - you will be able to use the feature.

Also please note, that peer-gateway may be disable on per-VLAN basis, but it's a short-term workaround, and the design would still be not supported.

Correct solution (on 6.2) is to use non-vpc enabled VLAN or direct L3 interfaces for IGP peering.