cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
12695
Views
5
Helpful
18
Replies

Palo Alto to 3850 OSPF failing

smolz
Level 4
Level 4

Recently started upgrading our 3850's to 16.3.6 and now seeing OSPF failures every 2-4 days.  Randomly the adjacency will fail after the Palo is not seeing 4 hello. Then it takes 20-30 minutes for the adjacency to come back.

 

Any else seeing this behavior?  Palos are running 7.1.10 except for one that is running 8.0.9

1 Accepted Solution

Accepted Solutions

Working with Cisco they have created the following bug CSCvk51328 and is supposed to be fixed in 16.3.7

View solution in original post

18 Replies 18

loganhreynolds
Level 1
Level 1

Hello,

 

Could you provide the OSPF configuration as well as the interface configuration for the port that connects to the Palo Alto?

Also, are you seeing packet loss across that interface?

 

 

Physical Interface:

sh interface GigabitEthernet1/0/3
GigabitEthernet1/0/3 is up, line protocol is up (connected) 
  Hardware is Gigabit Ethernet, address is 00c1.b103.3503 (bia 00c1.b103.3503)
  Description: Connection to Palo Alto e6
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 44022
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 35000 bits/sec, 13 packets/sec
  5 minute output rate 63000 bits/sec, 47 packets/sec
     5035478537 packets input, 6210967859353 bytes, 0 no buffer
     Received 547748 broadcasts (492918 multicasts)
     0 runts, 0 giants, 0 throttles 
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 492918 multicast, 0 pause input
     0 input packets with dribble condition detected
     3632703425 packets output, 2112890463418 bytes, 0 underruns
     0 output errors, 0 collisions, 2 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

Vlan Interface:

interface Vlan1000
 ip address 10.x.x.x 255.255.255.240
 ip ospf network point-to-point

OSPF:

router ospf 1
 router-id 10.x.x.x
 redistribute connected subnets
 redistribute eigrp 100 metric 200 subnets
 network 10.x.x.0 0.0.0.15 area 0

sh ip ospf:

sh ip ospf
 Routing Process "ospf 1" with ID 10.x.x.x
 Start time: 00:05:41.080, Time elapsed: 7w3d
 Supports only single TOS(TOS0) routes
 Supports opaque LSA
 Supports Link-local Signaling (LLS)
 Supports area transit capability
 Supports NSSA (compatible with RFC 3101)
 Supports Database Exchange Summary List Optimization (RFC 5243)
 Event-log enabled, Maximum number of events: 1000, Mode: cyclic
 It is an autonomous system boundary router
 Redistributing External Routes from,
    connected, includes subnets in redistribution
    eigrp with metric mapped to 200, includes subnets in redistribution
 Router is not originating router-LSAs with maximum metric
 Initial SPF schedule delay 5000 msecs
 Minimum hold time between two consecutive SPFs 10000 msecs
 Maximum wait time between two consecutive SPFs 10000 msecs
 Incremental-SPF disabled
 Minimum LSA interval 5 secs
 Minimum LSA arrival 1000 msecs
 LSA group pacing timer 240 secs
 Interface flood pacing timer 33 msecs
 Retransmission pacing timer 66 msecs
 EXCHANGE/LOADING adjacency limit: initial 300, process maximum 300
 Number of external LSA 505. Checksum Sum 0x1051AA8
 Number of opaque AS LSA 0. Checksum Sum 0x000000
 Number of DCbitless external and opaque AS LSA 1
 Number of DoNotAge external and opaque AS LSA 0
 Number of areas in this router is 1. 1 normal 0 stub 0 nssa
 Number of areas transit capable is 0
 External flood list length 0
 IETF NSF helper support enabled
 Cisco NSF helper support enabled
 Reference bandwidth unit is 100 mbps
    Area BACKBONE(0)
        Number of interfaces in this area is 1
        Area has no authentication
        SPF algorithm last executed 03:16:27.172 ago
        SPF algorithm executed 52 times
        Area ranges are
        Number of LSA 2. Checksum Sum 0x019B48
        Number of opaque link LSA 0. Checksum Sum 0x000000
        Number of DCbitless LSA 1
        Number of indication LSA 0
        Number of DoNotAge LSA 0
        Flood list length 0

 

Hi!

 

Are you running qos in the Cisco swith IF it so how is your configuration, you do have some output drops om the Interface you posted? 

Can you post show run interface GigabitEthernet1/0/3?

 

HTH

/Mohammed

 

interface GigabitEthernet1/0/3
description Connection to Palo Alto
switchport mode trunk spanning-tree portfast

 

Hi!

 

Can you post show spanning-tree interface GigabitEthernet1/0/3?

How many active vlans have you between Palo Alto to 3850?

Are you running som mls command on the 3850? If is yes post it.

 

HTH

/Mohammed

#show spanning-tree interface GigabitEthernet1/0/3

Vlan                Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
VLAN0001            Desg FWD 4         128.3    P2p 
VLAN0100            Desg FWD 4         128.3    P2p 
VLAN0101            Desg FWD 4         128.3    P2p 
VLAN0142            Desg FWD 4         128.3    P2p 
VLAN0143            Desg FWD 4         128.3    P2p 
VLAN0150            Desg FWD 4         128.3    P2p 
VLAN0160            Desg FWD 4         128.3    P2p 
VLAN0200            Desg FWD 4         128.3    P2p 
VLAN0400            Desg FWD 4         128.3    P2p 
VLAN0500            Desg FWD 4         128.3    P2p 
VLAN1000            Desg FWD 4         128.3    P2p 
VLAN1100            Desg FWD 4         128.3    P2p 
VLAN1200            Desg FWD 4         128.3    P2p 

Only 3 of the vlans are on the Palo.

 

switch#sh run | in mls
switch#

Reza Sharifi
Hall of Fame
Hall of Fame

If this issue started after the upgrade then this is most likely due to a bug in the 16.3.6 version.

interface GigabitEthernet1/0/3
description Connection to Palo Alto
switchport mode trunk spanning-tree portfast

Also, you should not configure portfast on a trunk port. 

 

no spanning-tree portfast

 

smolz
Level 4
Level 4

Any other ideas?

Hi!

 

Like  Reza Sharifi sa:

no spanning-tree portfast

 Have you tried? 

And allow only vlans that is needed to communicate to Palo only:

switchport trunk allowed vlan x,y,z

 If is this on production, please don´t forget to use this command:

switchport trunk allowd vlan add x,y,z

Otherwise it could couse spanning-tree calculation:

 

HTH 

 

/Mohammed

Lertpaiboon
Level 1
Level 1

I found the same issue with you. How can you fix it ?

Working with Cisco they have created the following bug CSCvk51328 and is supposed to be fixed in 16.3.7

Same answer from Cisco, to upgrade to 16.3.7.

The bug is listed as duplicated of following one:
CSCvj16271 Memory leak for "linux_iosd-imag" process on 16.3.6 due to IPC communication.
Reference:
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvj16271/?reffering_site=dumpcr

Stephen Buck
Level 1
Level 1

We're having a similar issue. In our case, we're running a pair of Palo 3220's in an Active/Passive HA running 8.1.6-h2 and a pair of 3850-24-XUs running 16.3.6. The Palo Alto is configured with two OSPF areas: 0 and xx which is a stub area. We're seeing OSPF adjacency going down every 12-20 hours for about 9-10 minutes each time for the xx area only. After working with Palo TAC and doing several packet captures, they determined that the Cisco 3850 stops sending hello packets with the "Active Neighbor" specified.

 

Palo Alto IP - 1.1.1.1/30

3850 IP - 1.1.1.2/30

 

Last hello packet from 3850 before adjacency was lost:

Frame 110186: 94 bytes on wire (752 bits), 94 bytes captured (752 bits) on interface 0
Ethernet II, Src: Cisco_93:b1:5c (00:bf:77:93:b1:5c), Dst: IPv4mcast_05 (01:00:5e:00:00:05)
Internet Protocol Version 4, Src: 1.1.1.2, Dst: 224.0.0.5
Open Shortest Path First
OSPF Header
OSPF Hello Packet
Network Mask: 255.255.255.252
Hello Interval [sec]: 10
Options: 0x10, (L) LLS Data block
Router Priority: 1
Router Dead Interval [sec]: 40
Designated Router: 1.1.1.2
Backup Designated Router: 1.1.1.1
Active Neighbor: 1.1.1.1
OSPF LLS Data Block

Hello packet from Palo just before adjacency was lost

Frame 110187: 82 bytes on wire (656 bits), 82 bytes captured (656 bits) on interface 0
Ethernet II, Src: PaloAlto_e0:40:10 (b4:0c:25:e0:40:10), Dst: IPv4mcast_05 (01:00:5e:00:00:05)
Internet Protocol Version 4, Src: 1.1.1.1, Dst: 224.0.0.5
Open Shortest Path First
OSPF Header
OSPF Hello Packet
Network Mask: 255.255.255.252
Hello Interval [sec]: 10
Options: 0x00
Router Priority: 1
Router Dead Interval [sec]: 40
Designated Router: 1.1.1.2
Backup Designated Router: 1.1.1.1
Active Neighbor: 1.1.1.2

Hello packet from 3850 when adjacency was lost

Frame 110194: 90 bytes on wire (720 bits), 90 bytes captured (720 bits) on interface 0
Ethernet II, Src: Cisco_93:b1:5c (00:bf:77:93:b1:5c), Dst: IPv4mcast_05 (01:00:5e:00:00:05)
Internet Protocol Version 4, Src: 1.1.1.2, Dst: 224.0.0.5
Open Shortest Path First
OSPF Header
OSPF Hello Packet
Network Mask: 255.255.255.252
Hello Interval [sec]: 10
Options: 0x10, (L) LLS Data block
Router Priority: 1
Router Dead Interval [sec]: 40
Designated Router: 1.1.1.2
Backup Designated Router: 0.0.0.0
--->missing active neighbor<---
OSPF LLS Data Block

When the Palo receives the hello from the 3850 without the Active Neighbor, the Palo brings down the adjacency. The Palo continues to send hello packets with the active neighbor set to 1.1.1.2, but the hellos received from the 3850 do not contain the active neighbor during this time. After about 9-10 minutes, the 3850 sends out a DB description packet and then starts including the Active Neighbor in the hello packets and the adjacency comes back up.

 

We're going to upgrade our 3850s to 16.3.8 and see if the issue goes away. This is the only pair of 3850s that are running the 16.3.6 code.

 

DB description packet from the 3850 just before adjacency comes back up

Frame 110859: 78 bytes on wire (624 bits), 78 bytes captured (624 bits) on interface 0
Ethernet II, Src: Cisco_93:b1:5c (00:bf:77:93:b1:5c), Dst: PaloAlto_e0:40:10 (b4:0c:25:e0:40:10)
Internet Protocol Version 4, Src: 1.1.1.2, Dst: 1.1.1.1
Open Shortest Path First
OSPF Header
Version: 2
Message Type: DB Description (2)
Packet Length: 32
Source OSPF Router: 1.1.1.2
Area ID: 0.0.0.xx
Checksum: 0x89eb [correct]
Auth Type: Null (0)
Auth Data (none): 0000000000000000
OSPF DB Description
Interface MTU: 1500
Options: 0x50, O, (L) LLS Data block
DB Description: 0x07, (I) Init, (M) More, (MS) Master
DD Sequence: 5326
OSPF LLS Data Block

Hello from 3850 just before adjacency comes back up

Frame 110860: 94 bytes on wire (752 bits), 94 bytes captured (752 bits) on interface 0
Ethernet II, Src: Cisco_93:b1:5c (00:bf:77:93:b1:5c), Dst: PaloAlto_e0:40:10 (b4:0c:25:e0:40:10)
Internet Protocol Version 4, Src: 1.1.1.2, Dst: 1.1.1.1
Open Shortest Path First
OSPF Header
Version: 2
Message Type: Hello Packet (1)
Packet Length: 48
Source OSPF Router: 1.1.1.2
Area ID: 0.0.0.xx
Checksum: 0xd21a [correct]
Auth Type: Null (0)
Auth Data (none): 0000000000000000
OSPF Hello Packet
Network Mask: 255.255.255.252
Hello Interval [sec]: 10
Options: 0x10, (L) LLS Data block
Router Priority: 1
Router Dead Interval [sec]: 40
Designated Router: 1.1.1.2
Backup Designated Router: 0.0.0.0
Active Neighbor: 1.1.1.1
OSPF LLS Data Block

Hi Stephen,

 

I have seen this issue before. The configuration of the command "ip ospf lls disable" on the interface facing the Palo Alto device solved the issue.

 

interface GigabitEthernet0/1
ip ospf lls disable

 

Regards,

 

Harold Ritter
Sr Technical Leader
CCIE 4168 (R&S, SP)
harold@cisco.com
México móvil: +52 1 55 8312 4915
Cisco México
Paseo de la Reforma 222
Piso 19
Cuauhtémoc, Juárez
Ciudad de México, 06600
México
Review Cisco Networking products for a $25 gift card