VRFs Not Neighbouring OSPFV2

[jbulloch]

Hi cisco community,

Perhaps i am overlooking something simple here. This morning i arrived on site to discover my VRFS will no longer ospf neighbour with the next device. The debug shows hello being sent, but i cannot reach the next device nor the OSPF multicast or receive a reponse. I've been over the configuration and i am not sure what would have changed to cause this. I've attempted resetting the process, verifed the router IDs and password is still good, mtu matches, acl is good and i can reach the hosts from the default/outside vrfs. I would welcome any input on where to countine troubleshooting.

The OSPF configuration:

router ospf 3 vrf MWE
router-id <snipped>
capability vrf-lite
area 3 authentication message-digest
passive-interface Vlan19
passive-interface Vlan83
passive-interface Vlan106
passive-interface Vlan114
passive-interface Vlan116
passive-interface Vlan150
passive-interface Vlan160
passive-interface Vlan1666
passive-interface Vlan2120

My uplink interface:

interface Vlan6
description MWE FW connection
vrf forwarding MWE
ip flow monitor IPv4_NETFLOW input
ip address x.x.6.1 255.255.255.248
ip mtu 1500
ip access-group <acl name> in
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 7 <key snipped>
ip ospf 3 area 3

debug:

Jun 13 12:07:41.204 edt: OSPF-3 ADJ Vl6: Send with youngest Key 1
Jun 13 12:07:41.204 edt: OSPF-3 HELLO Vl6: Send hello to 224.0.0.5 area 3 from x.x.6.1
Jun 13 12:07:41.204 edt: OSPF-3 PAK : Vl6: OUT: x.x.6.1->224.0.0.5: ver:2 type:1 len:44 rid:x.x.6.1 area:0.0.0.3 chksum:0 auth:2 keyid:1 seq:0x684C

[Jens Albrecht]

Hello @jbulloch,

you mention that 'i cannot reach the next device nor the OSPF multicast or receive a reponse.'

Does this mean that you do not have IP connectivity to the next device? In that case it is logical that OSPF does not come up.

So what is this 'next device'?
If the config looks good on your side - and it does based on the snippet you posted unless the ACL is messed up - then it is time to check the other side and restore IP connectivity as the first step.

HTH!

[Richard Burts]

I agree with Jens that it sounds like a connectivity issue. In addition to his suggestions I would suggest these:

- does show ip interface brief show vlan 6 as up?

- does show cdp neighbor show any neighbor on vlan 6?

- if you look in the arp table are there any entries for vlan 6 (other than your vlan interface)?

[jbulloch]

Hi richard,

vlan 6 is up up, but i cannot get a show ip arp. edit: to mean, other than gateway.

I am not sure if i can run cdp neigh on a vlan, unless you're refering to the interface?

Thank you for your assistance.

[Richard Burts]

Thanks for the response. Here are my responses:

- show arp should always show the mac of the interface. If it also show the mac of a connected device (gateway) then that helps demonstrate that the interface is working.

- you do not run show cdp neighbor on an interface (or on a vlan). You run the command and look in the output to see if there is any reference to a neighbor on that vlan/vlan interface. 

[jbulloch]

Hi richard, 

Yes i can see the SVI as the mac address of the gateway IP as expected.

Internet 157.141.6.1 - f4bd.9e49.f4f7 ARPA Vlan6, Vlan6 is up, line protocol is up , Autostate Enabled
Hardware is Ethernet SVI, address is f4bd.9e49.f4f7 (bia f4bd.9e49.f4f7)

I often use cdp neigh (and lldp neigh) for layer 2/mgmt ip verification and some others, but running it only shows connected switches. I am not sure how to show anything on a vlan interface(perhaps i have not done so before). Perhaps since the neighbour is not cisco device (palo alto) i should use LLDP. But i do not see anything there either.

[jbulloch]

Hi jens, no ACL is very simple and not a factor.

[Jens Albrecht]

Hi @jbulloch,

so the next device is a PaloAlto firewall. What is the IP address of this device?

The IP and MAC address of the PA have to be listed in the ARP table in order to have connectivity.
Do a "show arp" and check whether there is an entry for the PA firewall.
If not, then try to ping the PA. Even if the firewall is configured not to respond to pings it must answer the initial ARP request.
If the ping fails, do a "show arp" again and look for the entry of the PA.

If there is still no entry in the ARP table then the admin of the PA firewall needs to check on their side what is wrong with the PA device.

As previously mentioned you must have IP connectivity for OSPF to do its job and without an ARP entry for the PA device your switch will not be able to communicate with the PA.

If you can successfully ping the PA device or have at least an ARP entry for it, then you can do a "debug ip ospf packet" on your switch and check whether you see any OSPF packet coming from the PA device.
If not, then its time again to ask the PA firewall admin to check the OSPF config on their side.

HTH!

[paul driver]

Hello
If the OSPF adj's have suddenly been torn down then something has changed, even though you state no manually changes have been applied>
Your debug suggests so as your not receiving any hellos, so as stated by others you need to check for any broken L2/3 connectivity

Can you also debug the adjacency please< debug ip ospf adj> and post the results.

[jbulloch]

Hi paul,

It does appear that area 0 is functioning, but i am otherwise unable to get out of the process 3 on vlan 6:

Jun 14 11:06:50.372 edt: OSPF-1 ADJ Te1/1/6: Send with youngest Key 9
Jun 14 11:06:50.372 edt: OSPF-1 HELLO Te1/1/6: Send hello to 224.0.0.5 area 0 from 157.141.201.18
Jun 14 11:06:50.372 edt: OSPF-1 PAK : Te1/1/6: OUT: 157.141.201.18->224.0.0.5: ver:2 type:1 len:48 rid:157.141.0.1 area:0.0.0.0 chksum:0 auth:2 keyid:9 seq:0x684B
Jun 14 11:06:50.568 edt: OSPF-3 ADJ Vl6: Send with youngest Key 1
Jun 14 11:06:50.569 edt: OSPF-3 HELLO Vl6: Send hello to 224.0.0.5 area 3 from 157.141.6.1
Jun 14 11:06:50.569 edt: OSPF-3 PAK : Vl6: OUT: 157.141.6.1->224.0.0.5: ver:2 type:1 len:44 rid:157.141.6.1 area:0.0.0.3 chksum:0 auth:2 keyid:1 seq:0x684D
Jun 14 11:06:50.794 edt: OSPF-1 PAK : Te1/1/6: IN: 157.141.201.17->224.0.0.5: ver:2 type:1 len:48 rid:157.141.200.3 area:0.0.0.0 chksum:0 auth:2 keyid:9 seq:0x6845
Jun 14 11:06:50.795 edt: OSPF-1 HELLO Te1/1/6: Rcv hello from 157.141.200.3 area 0 157.141.201.17
Jun 14 11:06:50.976 edt: OSPF-1 PAK : Te2/1/2: IN: 157.141.201.21->224.0.0.5: ver:2 type:1 len:48 rid:157.141.200.5 area:0.0.0.0 chksum:0 auth:2 keyid:9 seq:0x6845
Jun 14 11:06:50.977 edt: OSPF-1 HELLO Te2/1/2: Rcv hello from 157.141.200.5 area 0 157.141.201.21
 

[jbulloch]

However, this vlan (6) is trunked out to area 0/upstream.

[paul driver]

Hello
Suggest remove the acl from vlan6 svi - i have experienced in the past when an acl was applied after the original adjacency was already established and due to some outage the adjacency was then lost but then couldn’t establish due the now applied acl .

[MHM Cisco World]

Hi

To all I am back 

@jbulloch share 

Show ip ospf traffic 

Show ip ospf interface breif 

For both ospf peers

MHM

[jbulloch · Accepted solution]

Hi all,

 I realized this was a L2 problem eventually, has a device way down stream was pruning the vlans between it and the next device. I noticed it appeared to be sending ARPs fine but no matter how i changed things like vtp pruning/trunking/etc it would never resolve the pruning. And both devices had L2 entries for the vlans. I got with TAC eventually and they identified that the devices were dropping the VTP updates all together and it appeared to be  causing the pruning. After some hours, we resolved issue with only up/down of link - simple shut no shut. 

Thank you everyone for your help.

[Richard Burts]

Thanks for the update. Glad to know that you and TAC have resolved the issue.