No OSPF peering between a Catalyst 9300 and an ASA 5516-X

[ppsilva02]

Hi

I have 5516-X ASA version 9.8(2) on one side and a 9300-NM-4G IOS XE version 16.8.1r [FC4] switch on the other side.

My configuration is as simple as (I just copied the relevant part of it):

On the 9300 switch:

ip routing
vtp mode transparent
!
spanning-tree mode pvst
spanning-tree portfast bpdufilter default
spanning-tree extend system-id
!
vlan 100
!
interface GigabitEthernet1/0/2
 description === Connection to Firewall
 switchport access vlan 100
!
interface Vlan100
 description ===INTERNAL VLAN===
 ip address 10.255.4.1 255.255.255.128
!
router ospf 1
 router-id 10.255.4.1
 redistribute connected subnets
 redistribute static subnets
 network 10.255.4.0 0.0.1.255 area 10.255.4.0
!

On 5516-X ASA

interface GigabitEthernet1/2
 nameif INSIDE_LAN
 security-level 100
 ip address 10.255.4.9 255.255.255.128 
!
router ospf 1
 router-id 10.255.4.9
 network 10.255.4.0 255.255.255.128 area 10.255.4.0
 area 10.255.4.0
 log-adj-changes
!

The problem is the they never peer with one another in OSPF

"show ip ospf interface" on the switch side gives:

Vlan100 is up, line protocol is up
  Internet Address 10.255.4.1/25, Interface ID 77, Area 10.255.4.0
  Attached via Network Statement
  Process ID 1, Router ID 10.255.4.1, Network Type BROADCAST, Cost: 1
  Topology-MTID    Cost    Disabled    Shutdown      Topology Name
        0           1         no          no            Base
  Transmit Delay is 1 sec, State DR, Priority 1
  Designated Router (ID) 10.255.4.1, Interface address 10.255.4.1
  No backup designated router on this network
  Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
    oob-resync timeout 40
    Hello due in 00:00:00
  Supports Link-local Signaling (LLS)
  Cisco NSF helper support enabled
  IETF NSF helper support enabled
  Can be protected by per-prefix Loop-Free FastReroute
  Can be used for per-prefix Loop-Free FastReroute repair paths
  Not Protected by per-prefix TI-LFA
  Index 1/1/1, flood queue length 0
  Next 0x0(0)/0x0(0)/0x0(0)
  Last flood scan length is 1, maximum is 1
  Last flood scan time is 0 msec, maximum is 0 msec
  Neighbor Count is 1, Adjacent neighbor count is 0
  Suppress hello for 0 neighbor(s)

On the ASA side:

show ospf interface

INSIDE_LAN is up, line protocol is up
  Internet Address 10.255.4.9 mask 255.255.255.128, Area 10.255.4.0
  Process ID 1, Router ID 10.255.4.9, Network Type BROADCAST, Cost: 10
  Transmit Delay is 1 sec, State DR, Priority 1
  Designated Router (ID) 10.255.4.9, Interface address 10.255.4.9
  No backup designated router on this network
  Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
    oob-resync timeout 40
    Hello due in 0:00:08
  Supports Link-local Signaling (LLS)
  Cisco NSF helper support enabled
  IETF NSF helper support enabled
  Index 1/1, flood queue length 0
  Next 0x0(0)/0x0(0)
  Last flood scan length is 0, maximum is 0
  Last flood scan time is 0 msec, maximum is 0 msec
  Neighbor Count is 0, Adjacent neighbor count is 0
  Suppress hello for 0 neighbor(s)

Then "show ip ospf neighbor" on the switch side:

Neighbor ID     Pri   State           Dead Time   Address         Interface
10.255.4.9      1   INIT/DROTHER    00:00:39    10.255.104.9    Vlan100

"show ospf neighbor" on the ASA:

(empty answer)

When I run a "debug ip ospf hello" in the 9300 side I get:

OSPF-1 HELLO Vl100: Send hello to 224.0.0.5 area 10.255.4.0 from 10.255.4.1
OSPF-1 HELLO Vl100: Rcv hello from 10.255.4.9 area 10.255.4.0 10.255.4.9
OSPF-1 HELLO Vl100:  No more immediate hello for nbr 10.255.4.9, which has been sent on this intf 2 times

And on the 5516-X I get:

OSPF: Send hello to 224.0.0.5 area 10.255.4.0 on INSIDE_LAN from 10.255.4.9
OSPF: Send hello to 224.0.0.5 area 10.255.4.0 on INSIDE_LAN from 10.255.4.9

Now for the funniest thing (or not....) :
If I try the same between the 9300 and an old 5510 ASA, or between the the 5516-X and a Catalyst 3560 EVERYTHING works fine !!!!

My question is:
Has anyone had this problem with this machines ?
Did someone found a workaround for this ?

Is it a question of the firmware versions ?

Would really appreciate some help !!

Thank you all

[Sebastian Fernandez]

Hi @ppsilva02 

Its clear that ASA does not see the OSPF hello from the Cisco 9300, so now the point of investigation is which side is the culprit. Can you trying to ping 224.0.0.5 from the Cisco 9300 and see if you get a response from 10.255.4.9

Also run a monitor capture on the ASA to see if OSPF hello is even making to ASA. Its clearly something on the layer 2 side that is broken. Config seems to be fine, although on the 9300 side your network statement matches /23. That's not an issue as it covers the VLAN100.  But the funny part of both devices working with other units is weird, this is as simple the configuration can get. 

Please mark this post helpful if you find it interesting. 

- 

Sebastian

[ppsilva02]

Hi Sebastian

Thank you for your cooperation.

Unfortunately I was already out of the office when I saw your reply.

I will try to do both things sunday and I'll post the result here as soon as I have it.

Thank you

[paul driver]

Hello
Could you try amending the network statement,  it shouldn't make any difference as the mask  incorporates the ASA subnet but it is incorrect.

Can you also post debug of:
debug ip ospf aj

9300
router ospf 100
network 10.255.4.0 0.0.0.127 area 0

[ppsilva02]

Hi

Thanks to everybody for the kind cooperation.

Fist: about the 0.0.1.255 wildcard mask:
It is there for a purpose: there are other interfaces that must be in the OSPF process but have no neighbors (I just didn't put their
configuration to keep it simple as it has no relevance to the problem.)

Second: Ping from the 9300 to ASA in 224.0.0.5 (ALL_OSPF_ROUTERS)
No results

Third: debug ip ospf adj (on 9300 side)

000237: *May  5 15:57:31: %LINK-3-UPDOWN: Interface Vlan100, changed state to up
000238: *May  5 15:57:32: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan100, changed state to up
000239: *May  5 15:57:32: OSPF-1 ADJ   Vl100: Route adjust notification: UP/UP
000240: *May  5 15:57:32: OSPF-1 ADJ   Vl100: Interface going Up
000241: *May  5 15:57:32: OSPF-1 ADJ   Vl100: Interface state change to UP, new ospf state WAIT
000242: *May  5 15:57:36: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel104, changed state to up
000243: *May  5 15:58:12: OSPF-1 ADJ   Vl100: end of Wait on interface
000244: *May  5 15:58:12: OSPF-1 ADJ   Vl100: DR/BDR election
000245: *May  5 15:58:12: OSPF-1 ADJ   Vl100: Elect BDR 10.255.4.1
000246: *May  5 15:58:12: OSPF-1 ADJ   Vl100: Elect DR 10.255.4.1
000247: *May  5 15:58:12: OSPF-1 ADJ   Vl100: Elect BDR 0.0.0.0
000248: *May  5 15:58:12: OSPF-1 ADJ   Vl100: Elect DR 10.255.4.1
000249: *May  5 15:58:12: OSPF-1 ADJ   Vl100: DR: 10.255.4.1 (Id)
000250: *May  5 15:58:12: OSPF-1 ADJ   Vl100:    BDR: none

Forth: about the capture

capture OSPF trace match ospf any any
capture OSPF int INSIDE_
capture OSPF int INSIDE_LAN

sh capture OSPF

123 packets captured

   1: 17:57:36.283371       10.255.4.1 > 224.0.0.5:  ip-proto-89, length 80
   2: 17:57:40.739951       10.255.4.9 > 224.0.0.5:  ip-proto-89, length 56
   3: 17:57:45.866852       10.255.4.1 > 224.0.0.5:  ip-proto-89, length 80
   4: 17:57:50.739951       10.255.4.9 > 224.0.0.5:  ip-proto-89, length 56
   5: 17:57:55.200932       10.255.4.1 > 224.0.0.5:  ip-proto-89, length 80
   6: 17:58:00.249971       10.255.4.9 > 224.0.0.5:  ip-proto-89, length 56
   7: 17:58:04.565339       10.255.4.1 > 224.0.0.5:  ip-proto-89, length 80
   8: 17:58:09.949932       10.255.4.9 > 224.0.0.5:  ip-proto-89, length 56
   9: 17:58:14.300140       10.255.4.1 > 224.0.0.5:  ip-proto-89, length 80
   .......
(output omited)
123 packets shown

So .. the OSPF packets arrive at the ASA !!

I then runned a packet tracer on the ASA to see what is happening, and this was the result:

packet-tracer input INSIDE_LAN udp 10.255.4.1 2050 224.0.0.5 89 detail

Phase: 1
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7f50608b0670, priority=1, domain=permit, deny=false
        hits=3907, user_data=0x0, cs_id=0x0, l3_type=0x8
        src mac=0000.0000.0000, mask=0000.0000.0000
        dst mac=0000.0000.0000, mask=0100.0000.0000
        input_ifc=INSIDE_LAN, output_ifc=any

Phase: 2
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 10.255.4.1 using egress ifc  INSIDE_LAN

Phase: 3
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7f505fb2e0e0, priority=0, domain=nat-per-session, deny=true
        hits=489, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
        input_ifc=any, output_ifc=any

Phase: 4
Type: ACCESS-LIST
Subtype:
Result: DROP
Config:
Implicit Rule
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7f50608b1aa0, priority=0, domain=permit, deny=true
        hits=3, user_data=0xa, cs_id=0x0, use_real_addr, flags=0x1000, protocol=0
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
        input_ifc=INSIDE_LAN, output_ifc=any

Result:
input-interface: INSIDE_LAN
input-status: up
input-line-status: up
output-interface: INSIDE_LAN
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

So, from the output of the packet-tracer, it looks like the OSPF Hellos are DROPed by an implicit rule of an ACL.

I DO NOT have ANY access-lint defined in the INSIDE_LAN interface.

I even inserted an ACL allowing all IP traffic in that interface, but the results where the same.

Any ideas ?

[Sebastian Fernandez]

Hi @ppsilva02 

Ok so now we know that atleast OSPF hello's are making it to the firewall but getting dropped. BTW that packet-tracer is incorrect because OSPF is not a UDP but an IP protocol number 89. 

The fact that firewall sends out hello to 9300 but drops in incoming hello suggests that there is some policy dropping the traffic. Without the full configuration my options are limited. But if don't have inbound ACL on the interface, then maybe the global ACL might be restricting it. Please check and post your firewall config (exclude or mask any sensitive content such as password or public IP)

***** Please rate this post if it helped ******

- 

Sebastian

[Georg Pauwen]

Hello,

on a side note, you have  a wildcard that doesn't match the actual interface, not sure if that affects anything:

interface Vlan100
description ===INTERNAL VLAN===
ip address 10.255.4.1 255.255.255.128
!
router ospf 1
router-id 10.255.4.1
redistribute connected subnets
redistribute static subnets
network 10.255.4.0 0.0.1.255 area 10.255.4.0 <-- this should be 0.0.0.127 in order to match your interface

[ppsilva02 · Accepted solution]

Well .. I figured that out !!!!!
A colleague did, to tell the truth.

The culprit is, in fact, the 9300 (well, the IOS XE 16.8.1a in this case, but it also exists in other more recent versions) !

The problem has to do with LLS (Link-Local Signaling)
I found a reference to the problem in here: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvg85146 (even if it isn't in a 9300)

So, the work around is to go to the interface where you are doing the peering on the 9300 side (in my case it was Vlan100) and enter:

int vlan 100
 ip ospf lls disable

The neighbor relationship immediately comes up !

I have attached 3 files with the capture of the OSPF Hello packets:

The "OSPF_Hello_Bad_Packet_9300.pdf" shows the wrong formed Hello packet. You can see the error on the "OSPF LLS Data Block" as "Unknown LLS TLV"
The "OSPF_Hello_Good_Packet_3560.pdf" shows the Hello packet with the LLS Block well formed from a 3560.
The last file "OSPF_Hello_Good_Packet.pdf" shows the Hello packet from the 9300 after using the command "ip ospf lls disable" on interface Vlan 100. As you can see there is no longer the LLS Data Block.

Nonetheless I want to sinceriously thank everyone for the kind cooperation.

[Sebastian Fernandez]

Hi @ppsilva02 

This is interesting, thank you for sharing the workaround.  I wasn't aware of this bug. 

[rauchi]

Thank you for saving my day! I had exact the some problem (C9300 with IOS XE 16.12.01 & ASA 5545-X with IOS 9.4.4.(37)).

using "ip ospf lls disable" on the C9300 side solved the problem immediately :-)

Best regards

Marco