NX-OS OSPF Routing - Directly on Ethernet Interface vs. VLAN Interface

[mbrown-revitycu]

We are encountering an issue with setting up OSPF routing on two Nexus9000 switches we recently put into service.

We have several branch locations in our network. Each has its own branch router or L3 device. These devices use OSPF to form neighbor relationships.

When we first setup the two Nexus9000 devices, we attempted to set up the OSPF routing on a VLAN interface. This interface was set up on both devices with HSRP applied.

The config was very much like this:

Nexus A

interface Vlan990
description CharterMPLS
no shutdown
no ip redirects
ip address 172.16.99.237/24
no ipv6 redirects
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 3 keygoeshere
no ip ospf passive-interface
ip router ospf 10 area 0.0.0.0
hsrp 90
priority 150
timers 1 3
ip 172.16.99.236

Nexus B

interface Vlan990
description CharterMPLS
no shutdown
no ip redirects
ip address 172.16.99.238/24
no ipv6 redirects
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 3 keygoeshere
no ip ospf passive-interface
ip router ospf 10 area 0.0.0.0
hsrp 90
priority 150
timers 1 3
ip 172.16.99.236

Ethernet interface for connection:

interface Ethernet1/6
switchport
switchport mode trunk
switchport trunk allowed vlan 990

Under this configuration the Nexus devices would not form neighbor relationships with the other devices in the network.

However, if we configured an ethernet interface directly with the OSPF info, neighboring came right up and works perfectly.

Config on the interface:

interface Ethernet1/45
description CHARTER_MPLS
no ip redirects
ip address 172.16.99.236/24
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 3 keygoeshere
no ip ospf passive-interface
ip router ospf 10 area 0.0.0.0
no shutdown

Do we have errors in our config somewhere such that setting OSPF on the VLAN interface causes the neighbor relationships to not be built?

Thanks!

[Giuseppe Larosa]

Hello @mbrown-revitycu ,

 >> Under this configuration the Nexus devices would not form neighbor relationships with the other devices in the network.

>> However, if we configured an ethernet interface directly with the OSPF info, neighboring came right up and works perfectly.

Check the MTU when you use the SVI and when you use the physical interface it is the first thing I would look at.

show ip interface vlan 990 compared to show ip interface eth1/45

Hope to help

Giuseppe

[MHM Cisco World]

You can use VLAN' 

But Nexus not like other SW that auto add vlan to db

So what you need is only add vlan by

Vlan 990

And the ospf will be work

MHM

[mbrown-revitycu]

I'm sorry, I don't understand this reply.

"You can use VLAN' 

But Nexus not like other SW that auto add vlan to db

So what you need is only add vlan by

Vlan 990

And the ospf will be work"

We do have this also in the config:

vlan 1,5,10,30,35,66,150,160,252-254,990,999-1000

[REDACTED]

vlan 990
name Charter_MPLS
vlan 999
name SilverpeakLAN

spanning-tree vlan 5,10,30,35,66,150,160,990,999 priority 4096

Is any of that what you mean? Or is there a separate place to define the VLAN?

Thanks!

[MHM Cisco World]

show ip interface brief <<- share this check if VLAN 990 is UP/UP or not 

MHM

[mbrown-revitycu]

It's not currently listed in a show ip int brief, because we don't actually have an IP address on the VLAN interface right now, only this is configured:

interface Vlan990
description Charter MPLS
no ip redirects
no ipv6 redirects

However, when we have attempted to use the VLAN for OSPF, the configuration we tried is something like this:

interface Vlan990
description Charter MPLS
no ip redirects
ip address 172.16.99.235/24
no ipv6 redirects
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 3 authkeygoeshere
no ip ospf passive-interface
no shutdown

[MHM Cisco World]

With sure 

Ip router ospf <> area <>

?

If Yes I will run lab and use VLAN between two NSK and share result here.

One more not' are both NSK is vPC?

Are vlan is vpc vlan or non-vpc vlan?

MHM

[mbrown-revitycu]

The OSPF config is:

router ospf 10
router-id 172.16.99.236
log-adjacency-changes
area 0.0.0.0 authentication
passive-interface default

We have some vPC VLANs, we have some non-vPC VLANs.

This one - when we tried to use it - was set up as non-vPC VLAN.

[MHM Cisco World]

And this vlan use vpc peer-link not separate trunk between two SW?

MHM

[mbrown-revitycu]

We have one peer link set up between the two Nexus switches.

interface port-channel1000
description ***VPC PEER LINKS***
switchport
switchport mode trunk
spanning-tree port type network
vpc peer-link

interface Ethernet1/53
description ***VPC PEER LINKS***
switchport
switchport mode trunk
channel-group 1000 mode active
no shutdown

interface Ethernet1/54
description ***VPC PEER LINKS***
switchport
switchport mode trunk
channel-group 1000 mode active
no shutdown

[MHM Cisco World]

So this trunk peer-link for both vpc vlan and non-vpc vlan

But all vlan allow so all vlan is vpc vlan

Anyway I will run lab with same topology you have abd check results

MHM

[mbrown-revitycu]

Thank you for all the suggestions and help.

I want to clarify something though, that I don't think I did in my earlier posts.

This pair we are trying to fix are the only Nexus switches we have.

All other branches we are connecting by OSPF are using Cisco ISR 4321 or 4321.

[MHM Cisco World]

this is my lab 
trunk allow all vlan between two NSK, this trunk is peer-link 
the OSPF is OK between two SW after I sure that the vlan is UP and appear in show vpc brief and STP not BLK this VLAN

NOW, additional point about the ISR  I dont get it? can you more elaborate 

MHM

[shoppman]

Hi Mbrown,

Can you clarify if you are also making changes to the router side configurations when you make the following changes to the Nexus?

"Under this configuration the Nexus devices would not form neighbor relationships with the other devices in the network.

However, if we configured an ethernet interface directly with the OSPF info, neighboring came right up and works perfectly."

The frame leaving a routed layer 3 port versus a layer 2 trunk port is very different.  In the SVI configurations you provided, you are using VLAN 990.  Naturally, you have also configured the layer 2 port as a trunk carrying VLAN 990.  In this scenario the frames leaving the port to form the OSPF neighborship will be tagged with 802.1q for VLAN 990.  When you are in this configuration is the router port configured to accept these VLAN tagged frames with something like a dot1.q encapsulation for vlan 990? 

Conversely, when you have the Nexus port configured as a layer 3 port, the frames will not be sent with a dot1.q vlan tag.  So the router will consume it differently. 

Just wanted some clarification on this one point before we continue taking a look at other potential causes. If the router configuration is not changing between the above two scenarios, I would suggest moving the layer 2 port to an access port for vlan 990 instead of a trunk, or adding dot1q encapsulation to the router config.

Also, since it has not been discussed yet, please be sure you have your VPC domain with 'layer3 peer-router', as the OSPF neighborships may not be stable unless this is configured.  This would not prevent the neighborships in your current scenario from forming, but could cause issues down the road or with with OSPF neighborships to just 1 of the peers. This configuration needs to be on both VPC peers.

Thanks!

Scott Hoppmann

Cisco HTTS - Data Center Route and Switch

RTP, NC

[mbrown-revitycu]

I'll do my best to respond to everyone coherently in one post here.

Our network consists of eight branch locations. Each of these locations has a L3 switch or a simpler dedicated ISR that does the OSPF routing.

Branch A = Nexus9000 pair
Branch B = Meraki MS350-48FP stack
Branch C = Meraki MS350-48FP stack
Branch D, E, F, G, H = Cisco ISR 4331
Branch J = Cisco ISR 4321

Each of these devices is performing the OSPF routing for its branch. There is not an additional router at Branch A - the Nexus is the router.

Each is in the OSPF network currently, neighboring up with all its peers as we want - this minus the SD-WAN device being included at Branch A. When we attempt to add the SD-WAN device to the Branch A network, OSPF breaks down at Branch A.

As it is, we have not been making any changes to any of the routers when attempting to add the SD-WAN device.

We set up a subnet specific to this traffic, that being 172.16.99.0/24. Each branch device has an address in that subnet that is being used to pass the OSPF traffic.

At the branches with the ISR 4321/4331, the config on the interface performing the OSPF routing does not use a dot1q subinterface. The interface is assigned an IP address in the subnet.

Example config shown below:

interface GigabitEthernet0/0/1
ip address 172.16.99.250 255.255.255.0
ip access-group CUI-IN in
ip access-group CUI-OUT out
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 7 011F570A501F5642331F5D1D
negotiation auto

At the branches with the Meraki core switches, an interface is set up on the stack with an address in the subnet, and using VLAN 990.

Interface Editor
Switch or switch stack
MAIN-CORE-S
Name
Charter MPLS
VLAN
990
Subnet
172.16.99.0/24
Interface IP
172.16.99.253
Multicast routing
Disabled
OSPF settings
Area
0: Backbone
Cost
1
Passive?
No

On the Nexus9000 stack, "layer3 peer-router" is configured as shown here.

Switch A
vpc domain 1
peer-switch
role priority 100
peer-keepalive destination 10.255.255.2 source 10.255.255.1 vrf keepalive
peer-gateway
layer3 peer-router
auto-recovery

Switch B
vpc domain 1
peer-switch
peer-keepalive destination 10.255.255.1 source 10.255.255.2 vrf keepalive
peer-gateway
layer3 peer-router

Routing thru OSPF is working on the Nexus as is, which is a direct connection from the Spectrum WAN router to the Eth 1/45 interface of one of the Nexus switches.

interface Ethernet1/45
description CHARTER_MPLS
no ip redirects
ip address 172.16.99.236/24
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 3 5fbc4a00c637609c98e546fb504bde1e
no ip ospf passive-interface
ip router ospf 10 area 0.0.0.0
no shutdown


At the risk of overwhelming with info that may not even be necessary, I'll end this post here.

Please let me know if I can provide anything else though. I do sincerely appreciate everyone taking the time on this.