05-31-2018 02:09 PM - edited 03-08-2019 03:12 PM
Recently started upgrading our 3850's to 16.3.6 and now seeing OSPF failures every 2-4 days. Randomly the adjacency will fail after the Palo is not seeing 4 hello. Then it takes 20-30 minutes for the adjacency to come back.
Any else seeing this behavior? Palos are running 7.1.10 except for one that is running 8.0.9
Solved! Go to Solution.
07-30-2018 08:34 AM
Working with Cisco they have created the following bug CSCvk51328 and is supposed to be fixed in 16.3.7
05-31-2018 02:25 PM
Hello,
Could you provide the OSPF configuration as well as the interface configuration for the port that connects to the Palo Alto?
Also, are you seeing packet loss across that interface?
05-31-2018 08:09 PM
Physical Interface:
sh interface GigabitEthernet1/0/3 GigabitEthernet1/0/3 is up, line protocol is up (connected) Hardware is Gigabit Ethernet, address is 00c1.b103.3503 (bia 00c1.b103.3503) Description: Connection to Palo Alto e6 MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX input flow-control is off, output flow-control is unsupported ARP type: ARPA, ARP Timeout 04:00:00 Last input never, output 00:00:00, output hang never Last clearing of "show interface" counters never Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 44022 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 35000 bits/sec, 13 packets/sec 5 minute output rate 63000 bits/sec, 47 packets/sec 5035478537 packets input, 6210967859353 bytes, 0 no buffer Received 547748 broadcasts (492918 multicasts) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog, 492918 multicast, 0 pause input 0 input packets with dribble condition detected 3632703425 packets output, 2112890463418 bytes, 0 underruns 0 output errors, 0 collisions, 2 interface resets 0 unknown protocol drops 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier, 0 pause output 0 output buffer failures, 0 output buffers swapped out
Vlan Interface:
interface Vlan1000 ip address 10.x.x.x 255.255.255.240 ip ospf network point-to-point
OSPF:
router ospf 1 router-id 10.x.x.x redistribute connected subnets redistribute eigrp 100 metric 200 subnets network 10.x.x.0 0.0.0.15 area 0
sh ip ospf:
sh ip ospf Routing Process "ospf 1" with ID 10.x.x.x Start time: 00:05:41.080, Time elapsed: 7w3d Supports only single TOS(TOS0) routes Supports opaque LSA Supports Link-local Signaling (LLS) Supports area transit capability Supports NSSA (compatible with RFC 3101) Supports Database Exchange Summary List Optimization (RFC 5243) Event-log enabled, Maximum number of events: 1000, Mode: cyclic It is an autonomous system boundary router Redistributing External Routes from, connected, includes subnets in redistribution eigrp with metric mapped to 200, includes subnets in redistribution Router is not originating router-LSAs with maximum metric Initial SPF schedule delay 5000 msecs Minimum hold time between two consecutive SPFs 10000 msecs Maximum wait time between two consecutive SPFs 10000 msecs Incremental-SPF disabled Minimum LSA interval 5 secs Minimum LSA arrival 1000 msecs LSA group pacing timer 240 secs Interface flood pacing timer 33 msecs Retransmission pacing timer 66 msecs EXCHANGE/LOADING adjacency limit: initial 300, process maximum 300 Number of external LSA 505. Checksum Sum 0x1051AA8 Number of opaque AS LSA 0. Checksum Sum 0x000000 Number of DCbitless external and opaque AS LSA 1 Number of DoNotAge external and opaque AS LSA 0 Number of areas in this router is 1. 1 normal 0 stub 0 nssa Number of areas transit capable is 0 External flood list length 0 IETF NSF helper support enabled Cisco NSF helper support enabled Reference bandwidth unit is 100 mbps Area BACKBONE(0) Number of interfaces in this area is 1 Area has no authentication SPF algorithm last executed 03:16:27.172 ago SPF algorithm executed 52 times Area ranges are Number of LSA 2. Checksum Sum 0x019B48 Number of opaque link LSA 0. Checksum Sum 0x000000 Number of DCbitless LSA 1 Number of indication LSA 0 Number of DoNotAge LSA 0 Flood list length 0
05-31-2018 11:32 PM
Hi!
Are you running qos in the Cisco swith IF it so how is your configuration, you do have some output drops om the Interface you posted?
Can you post show run interface GigabitEthernet1/0/3?
HTH
/Mohammed
06-01-2018 08:30 AM
interface GigabitEthernet1/0/3
description Connection to Palo Alto
switchport mode trunk spanning-tree portfast
06-01-2018 10:10 AM
Hi!
Can you post show spanning-tree interface GigabitEthernet1/0/3?
How many active vlans have you between Palo Alto to 3850?
Are you running som mls command on the 3850? If is yes post it.
HTH
/Mohammed
06-01-2018 01:49 PM
#show spanning-tree interface GigabitEthernet1/0/3 Vlan Role Sts Cost Prio.Nbr Type ------------------- ---- --- --------- -------- -------------------------------- VLAN0001 Desg FWD 4 128.3 P2p VLAN0100 Desg FWD 4 128.3 P2p VLAN0101 Desg FWD 4 128.3 P2p VLAN0142 Desg FWD 4 128.3 P2p VLAN0143 Desg FWD 4 128.3 P2p VLAN0150 Desg FWD 4 128.3 P2p VLAN0160 Desg FWD 4 128.3 P2p VLAN0200 Desg FWD 4 128.3 P2p VLAN0400 Desg FWD 4 128.3 P2p VLAN0500 Desg FWD 4 128.3 P2p VLAN1000 Desg FWD 4 128.3 P2p VLAN1100 Desg FWD 4 128.3 P2p VLAN1200 Desg FWD 4 128.3 P2p
Only 3 of the vlans are on the Palo.
switch#sh run | in mls switch#
06-01-2018 09:20 AM - edited 06-01-2018 09:22 AM
If this issue started after the upgrade then this is most likely due to a bug in the 16.3.6 version.
interface GigabitEthernet1/0/3
description Connection to Palo Alto
switchport mode trunk spanning-tree portfast
Also, you should not configure portfast on a trunk port.
no spanning-tree portfast
06-04-2018 12:58 PM
Any other ideas?
06-04-2018 11:35 PM
Hi!
Like Reza Sharifi sa:
no spanning-tree portfast
Have you tried?
And allow only vlans that is needed to communicate to Palo only:
switchport trunk allowed vlan x,y,z
If is this on production, please don´t forget to use this command:
switchport trunk allowd vlan add x,y,z
Otherwise it could couse spanning-tree calculation:
HTH
/Mohammed
07-18-2018 01:42 AM
I found the same issue with you. How can you fix it ?
07-30-2018 08:34 AM
Working with Cisco they have created the following bug CSCvk51328 and is supposed to be fixed in 16.3.7
08-27-2018 07:39 AM
04-25-2019 04:38 PM - edited 04-25-2019 04:43 PM
We're having a similar issue. In our case, we're running a pair of Palo 3220's in an Active/Passive HA running 8.1.6-h2 and a pair of 3850-24-XUs running 16.3.6. The Palo Alto is configured with two OSPF areas: 0 and xx which is a stub area. We're seeing OSPF adjacency going down every 12-20 hours for about 9-10 minutes each time for the xx area only. After working with Palo TAC and doing several packet captures, they determined that the Cisco 3850 stops sending hello packets with the "Active Neighbor" specified.
Palo Alto IP - 1.1.1.1/30
3850 IP - 1.1.1.2/30
Last hello packet from 3850 before adjacency was lost: Frame 110186: 94 bytes on wire (752 bits), 94 bytes captured (752 bits) on interface 0 Ethernet II, Src: Cisco_93:b1:5c (00:bf:77:93:b1:5c), Dst: IPv4mcast_05 (01:00:5e:00:00:05) Internet Protocol Version 4, Src: 1.1.1.2, Dst: 224.0.0.5 Open Shortest Path First OSPF Header OSPF Hello Packet Network Mask: 255.255.255.252 Hello Interval [sec]: 10 Options: 0x10, (L) LLS Data block Router Priority: 1 Router Dead Interval [sec]: 40 Designated Router: 1.1.1.2 Backup Designated Router: 1.1.1.1 Active Neighbor: 1.1.1.1 OSPF LLS Data Block Hello packet from Palo just before adjacency was lost Frame 110187: 82 bytes on wire (656 bits), 82 bytes captured (656 bits) on interface 0 Ethernet II, Src: PaloAlto_e0:40:10 (b4:0c:25:e0:40:10), Dst: IPv4mcast_05 (01:00:5e:00:00:05) Internet Protocol Version 4, Src: 1.1.1.1, Dst: 224.0.0.5 Open Shortest Path First OSPF Header OSPF Hello Packet Network Mask: 255.255.255.252 Hello Interval [sec]: 10 Options: 0x00 Router Priority: 1 Router Dead Interval [sec]: 40 Designated Router: 1.1.1.2 Backup Designated Router: 1.1.1.1 Active Neighbor: 1.1.1.2 Hello packet from 3850 when adjacency was lost Frame 110194: 90 bytes on wire (720 bits), 90 bytes captured (720 bits) on interface 0 Ethernet II, Src: Cisco_93:b1:5c (00:bf:77:93:b1:5c), Dst: IPv4mcast_05 (01:00:5e:00:00:05) Internet Protocol Version 4, Src: 1.1.1.2, Dst: 224.0.0.5 Open Shortest Path First OSPF Header OSPF Hello Packet Network Mask: 255.255.255.252 Hello Interval [sec]: 10 Options: 0x10, (L) LLS Data block Router Priority: 1 Router Dead Interval [sec]: 40 Designated Router: 1.1.1.2 Backup Designated Router: 0.0.0.0 --->missing active neighbor<--- OSPF LLS Data Block
When the Palo receives the hello from the 3850 without the Active Neighbor, the Palo brings down the adjacency. The Palo continues to send hello packets with the active neighbor set to 1.1.1.2, but the hellos received from the 3850 do not contain the active neighbor during this time. After about 9-10 minutes, the 3850 sends out a DB description packet and then starts including the Active Neighbor in the hello packets and the adjacency comes back up.
We're going to upgrade our 3850s to 16.3.8 and see if the issue goes away. This is the only pair of 3850s that are running the 16.3.6 code.
DB description packet from the 3850 just before adjacency comes back up Frame 110859: 78 bytes on wire (624 bits), 78 bytes captured (624 bits) on interface 0 Ethernet II, Src: Cisco_93:b1:5c (00:bf:77:93:b1:5c), Dst: PaloAlto_e0:40:10 (b4:0c:25:e0:40:10) Internet Protocol Version 4, Src: 1.1.1.2, Dst: 1.1.1.1 Open Shortest Path First OSPF Header Version: 2 Message Type: DB Description (2) Packet Length: 32 Source OSPF Router: 1.1.1.2 Area ID: 0.0.0.xx Checksum: 0x89eb [correct] Auth Type: Null (0) Auth Data (none): 0000000000000000 OSPF DB Description Interface MTU: 1500 Options: 0x50, O, (L) LLS Data block DB Description: 0x07, (I) Init, (M) More, (MS) Master DD Sequence: 5326 OSPF LLS Data Block Hello from 3850 just before adjacency comes back up Frame 110860: 94 bytes on wire (752 bits), 94 bytes captured (752 bits) on interface 0 Ethernet II, Src: Cisco_93:b1:5c (00:bf:77:93:b1:5c), Dst: PaloAlto_e0:40:10 (b4:0c:25:e0:40:10) Internet Protocol Version 4, Src: 1.1.1.2, Dst: 1.1.1.1 Open Shortest Path First OSPF Header Version: 2 Message Type: Hello Packet (1) Packet Length: 48 Source OSPF Router: 1.1.1.2 Area ID: 0.0.0.xx Checksum: 0xd21a [correct] Auth Type: Null (0) Auth Data (none): 0000000000000000 OSPF Hello Packet Network Mask: 255.255.255.252 Hello Interval [sec]: 10 Options: 0x10, (L) LLS Data block Router Priority: 1 Router Dead Interval [sec]: 40 Designated Router: 1.1.1.2 Backup Designated Router: 0.0.0.0 Active Neighbor: 1.1.1.1 OSPF LLS Data Block
05-02-2019 11:28 PM
Hi Stephen,
I have seen this issue before. The configuration of the command "ip ospf lls disable" on the interface facing the Palo Alto device solved the issue.
interface GigabitEthernet0/1
ip ospf lls disable
Regards,
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide