Cisco 7606 Dropping Layer 3 Sessions

[digicom]

Dear Community,

I have a Cisco 7606S with Sup 720 which is currently facing issues with layer three process.

What I am noticing is that layer three processes such as OSPF, HSRP and BGP are dropping unexpectedly.

During the time that the connections drop the router stops responding to icmp on the interfaces as well.

All of my OSPF neighbourships go down all of a sudden and come back up on their own, when the OSPF sessions go down also HSRP sessions go down, it is worth to notice that the OSPF connections are setup on different interfaces and on different cards as well so I am thinking it might be a supervisor issue but not sure where to look for clues.

Interfaces remain up during the time that my connections drop and also I have checked for error on the interface but there is none.

I also changed one of the ports where one of my OSPF connections is established but still the issue occurs. 

The connections will stay stable for several hours and there all of a sudden all my OSPF and HSRP session drop and they came back up on their own, they stay like that for several hours and then the issue happens again, it should be noted that there isnt any change performed on the network during this time. 

I am posting below the logs from the last occurence:

Sep 26 2019 21:38:07 EET1: %HSRP-5-STATECHANGE: Vlan122 Grp 122 state Standby -> Active
Sep 26 2019 21:38:08 EET1: %HSRP-5-STATECHANGE: Vlan120 Grp 120 state Standby -> Active
Sep 26 2019 21:38:21 EET1: %OSPF-5-ADJCHG: Process xxxx, Nbr xxxx on Vlan777 from FULL to DOWN, Neighbor Down: Dead timer expired
Sep 26 2019 21:38:21 EET1: %OSPF-5-ADJCHG: Process xxxx, Nbr xxxx on Vlan777 from FULL to DOWN, Neighbor Down: Dead timer expired
Sep 26 2019 21:38:22 EET1: %HSRP-5-STATECHANGE: Vlan120 Grp 120 state Active -> Speak
Sep 26 2019 21:38:28 EET1: %OSPF-5-ADJCHG: Process xxxx, Nbr xxxx on Vlan777 from FULL to DOWN, Neighbor Down: Dead timer expired
Sep 26 2019 21:38:28 EET1: %OSPF-5-ADJCHG: Process xxxx, Nbr xxxx on Vlan6 from FULL to DOWN, Neighbor Down: Dead timer expired
Sep 26 2019 21:38:28 EET1: %OSPF-5-ADJCHG: Process xxxx, Nbr xxxx on Vlan777 from FULL to DOWN, Neighbor Down: Dead timer expired
Sep 26 2019 21:38:28 EET1: %OSPF-5-ADJCHG: Process xxxx, Nbr xxxx on Vlan777 from FULL to DOWN, Neighbor Down: Dead timer expired
Sep 26 2019 21:38:29 EET1: %HSRP-5-STATECHANGE: Vlan122 Grp 122 state Active -> Speak
Sep 26 2019 21:38:39 EET1: %HSRP-5-STATECHANGE: Vlan120 Grp 120 state Speak -> Standby
Sep 26 2019 21:38:44 EET1: %HSRP-5-STATECHANGE: Vlan120 Grp 120 state Standby -> Active
Sep 26 2019 21:38:44 EET1: %HSRP-5-STATECHANGE: Vlan120 Grp 120 state Active -> Speak
Sep 26 2019 21:38:47 EET1: %HSRP-5-STATECHANGE: Vlan122 Grp 122 state Speak -> Standby
Sep 26 2019 21:38:47 EET1: %HSRP-5-STATECHANGE: Vlan122 Grp 122 state Standby -> Active
Sep 26 2019 21:38:47 EET1: %HSRP-5-STATECHANGE: Vlan122 Grp 122 state Active -> Speak
Sep 26 2019 21:39:01 EET1: %HSRP-5-STATECHANGE: Vlan120 Grp 120 state Speak -> Standby
Sep 26 2019 21:39:02 EET1: %HSRP-5-STATECHANGE: Vlan122 Grp 122 state Speak -> Standby
Sep 26 2019 21:39:15 EET1: %OSPF-5-ADJCHG: Process xxxx , Nbr xxxx on Vlan4 from FULL to DOWN, Neighbor Down: Dead timer expired
Sep 26 2019 21:39:27 EET1: %HSRP-5-STATECHANGE: Vlan120 Grp 120 state Standby -> Active
Sep 26 2019 21:39:28 EET1: %HSRP-5-STATECHANGE: Vlan120 Grp 120 state Active -> Speak
Sep 26 2019 21:39:32 EET1: %HSRP-5-STATECHANGE: Vlan122 Grp 122 state Standby -> Active
Sep 26 2019 21:39:32 EET1: %HSRP-5-STATECHANGE: Vlan122 Grp 122 state Active -> Speak
Sep 26 2019 21:39:44 EET1: %HSRP-5-STATECHANGE: Vlan120 Grp 120 state Speak -> Standby
Sep 26 2019 21:39:48 EET1: %HSRP-5-STATECHANGE: Vlan122 Grp 122 state Speak -> Standby
Sep 26 2019 21:40:14 EET1: %OSPF-5-ADJCHG: Process xxxx , Nbr xxx on Vlan777 from LOADING to FULL, Loading Done
Sep 26 2019 21:40:33 EET1: %HSRP-5-STATECHANGE: Vlan120 Grp 120 state Standby -> Active
Sep 26 2019 21:40:36 EET1: %HSRP-5-STATECHANGE: Vlan122 Grp 122 state Standby -> Active
Sep 26 2019 21:40:44 EET1: %BGP-3-NOTIFICATION: sent to neighbor xxxx 4/0 (hold time expired) 0 bytes
Sep 26 2019 21:40:44 EET1: %BGP-5-NBR_RESET: Neighbor xxxx reset (BGP Notification sent)
Sep 26 2019 21:40:54 EET1: %OSPF-5-ADJCHG: Process xxxx , Nbr xxxx on Vlan777 from FULL to DOWN, Neighbor Down: Dead timer expired
Sep 26 2019 21:40:59 EET1: %OSPF-5-ADJCHG: Process xxxx, Nbr xxx on Vlan777 from EXCHANGE to DOWN, Neighbor Down: Dead timer expired
Sep 26 2019 21:40:59 EET1: %OSPF-5-ADJCHG: Process xxxx, Nbr xxxx on Vlan777 from EXCHANGE to DOWN, Neighbor Down: Dead timer expired
Sep 26 2019 21:40:59 EET1: %OSPF-5-ADJCHG: Process xxxx, Nbr xxx on Vlan777 from EXCHANGE to DOWN, Neighbor Down: Dead timer expired
Sep 26 2019 21:40:59 EET1: %OSPF-5-ADJCHG: Process xxxx, Nbr xxx on Vlan777 from EXCHANGE to DOWN, Neighbor Down: Dead timer expired
Sep 26 2019 21:41:02 EET1: %OSPF-4-CONFLICTING_LSAID: Process xxxx area dummy area: LSA origination prevented by LSA with same LSID but a different mask
Existing Type 5 LSA: LSID x.x.x.x/23
New Destination: x.x.x.x/23
Sep 26 2019 21:41:03 EET1: %HSRP-5-STATECHANGE: Vlan120 Grp 120 state Active -> Speak
Sep 26 2019 21:41:03 EET1: %HSRP-5-STATECHANGE: Vlan122 Grp 122 state Active -> Speak
Sep 26 2019 21:41:06 EET1: %OSPF-5-ADJCHG: Process xxx, Nbr xxx on Vlan4 from LOADING to FULL, Loading Done
Sep 26 2019 21:41:08 EET1: %BGP-5-ADJCHANGE: neighbor xxxx Down BGP Notification sent
Sep 26 2019 21:41:08 EET1: %BGP_SESSION-5-ADJCHANGE: neighbor xxxx IPv4 Unicast topology base removed from session BGP Notification sent
Sep 26 2019 21:41:21 EET1: %HSRP-5-STATECHANGE: Vlan120 Grp 120 state Speak -> Standby
Sep 26 2019 21:41:22 EET1: %HSRP-5-STATECHANGE: Vlan122 Grp 122 state Speak -> Standby
Sep 26 2019 21:41:30 EET1: %HSRP-5-STATECHANGE: Vlan122 Grp 122 state Standby -> Active
Sep 26 2019 21:41:30 EET1: %HSRP-5-STATECHANGE: Vlan122 Grp 122 state Active -> Speak
Sep 26 2019 21:41:37 EET1: %HSRP-5-STATECHANGE: Vlan120 Grp 120 state Standby -> Active
Sep 26 2019 21:41:37 EET1: %HSRP-5-STATECHANGE: Vlan120 Grp 120 state Active -> Speak
Sep 26 2019 21:41:46 EET1: %HSRP-5-STATECHANGE: Vlan122 Grp 122 state Speak -> Standby
Sep 26 2019 21:41:48 EET1: %HSRP-5-STATECHANGE: Vlan122 Grp 122 state Standby -> Active
Sep 26 2019 21:41:48 EET1: %HSRP-5-STATECHANGE: Vlan122 Grp 122 state Active -> Speak
Sep 26 2019 21:41:53 EET1: %HSRP-5-STATECHANGE: Vlan120 Grp 120 state Speak -> Standby
Sep 26 2019 21:42:01 EET1: %OSPF-5-ADJCHG: Process xxxx, Nbr xxxx on Vlan777 from EXCHANGE to DOWN, Neighbor Down: Dead timer expired
Sep 26 2019 21:42:03 EET1: %OSPF-5-ADJCHG: Process xxxx, Nbr xxx on Vlan4 from FULL to DOWN, Neighbor Down: Dead timer expired
Sep 26 2019 21:42:12 EET1: %HSRP-5-STATECHANGE: Vlan120 Grp 120 state Standby -> Active
Sep 26 2019 21:42:12 EET1: %HSRP-5-STATECHANGE: Vlan120 Grp 120 state Active -> Speak
Sep 26 2019 21:42:15 EET1: %OSPF-4-CONFLICTING_LSAID: Process xxxx area dummy area: LSA origination prevented by LSA with same LSID but a different mask
Existing Type 5 LSA: LSID x.x.x.x/23
New Destination: x.x.x.x/24
Sep 26 2019 21:42:25 EET1: %OSPF-5-ADJCHG: Process 35444, Nbr xxxx on Vlan6 from LOADING to FULL, Loading Done
Sep 26 2019 21:42:27 EET1: %HSRP-5-STATECHANGE: Vlan120 Grp 120 state Speak -> Standby
Sep 26 2019 21:42:44 EET1: %HSRP-5-STATECHANGE: Vlan120 Grp 120 state Standby -> Active
Sep 26 2019 21:42:47 EET1: %HSRP-5-STATECHANGE: Vlan122 Grp 122 state Standby -> Active
Sep 26 2019 21:42:52 EET1: %HSRP-5-STATECHANGE: Vlan122 Grp 122 state Active -> Speak
Sep 26 2019 21:42:59 EET1: %HSRP-5-STATECHANGE: Vlan120 Grp 120 state Active -> Speak
Sep 26 2019 21:43:06 EET1: %BGP-5-ADJCHANGE: neighbor xxxx Up
Sep 26 2019 21:43:10 EET1: %HSRP-5-STATECHANGE: Vlan122 Grp 122 state Speak -> Standby
Sep 26 2019 21:43:15 EET1: %HSRP-5-STATECHANGE: Vlan120 Grp 120 state Speak -> Standby
Sep 26 2019 21:43:21 EET1: %OSPF-5-ADJCHG: Process xxxx, Nbr yyyy on Vlan777 from LOADING to FULL, Loading Done
Sep 26 2019 21:43:21 EET1: %OSPF-5-ADJCHG: Process xxxx, Nbr yyyy on Vlan777 from LOADING to FULL, Loading Done
Sep 26 2019 21:43:22 EET1: %OSPF-5-ADJCHG: Process xxxx, Nbr yxz on Vlan777 from LOADING to FULL, Loading Done
Sep 26 2019 21:43:23 EET1: %OSPF-5-ADJCHG: Process xxxx, Nbr xxxz on Vlan4 from LOADING to FULL, Loading Done
Sep 26 2019 21:43:23 EET1: %OSPF-5-ADJCHG: Process xxxx, Nbr xxxy on Vlan777 from LOADING to FULL, Loading Done
Sep 26 2019 21:43:24 EET1: %OSPF-5-ADJCHG: Process xxxx, Nbr xxxx on Vlan777 from LOADING to FULL, Loading Done
Sep 26 2019 22:04:57 EET1: %SYS-6-TTY_EXPIRE_TIMER: (exec timer expired, tty 1 (xxxx)), user xxx
Sep 26 2019 22:04:57 EET1: %SYS-6-LOGOUT: User xxxx has exited tty session 1(xxxx)
Sep 26 2019 22:21:42 EET1: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: xxxx] [Source: xxxx]

[localport: 22] at 22:21:42 EET1 Thu Sep 26 2019
Sep 26 2019 22:32:37 EET1: %SYS-6-TTY_EXPIRE_TIMER: (exec timer expired, tty 1 (xxxx)), user xxxx
Sep 26 2019 22:32:37 EET1: %SYS-6-LOGOUT: User xxxx has exited tty session 1(x.x.x.x)
Sep 26 2019 22:44:35 EET1: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success 

[localport: 22] at 22:44:35 EET1 Thu Sep 26 2019
border1#

Any help would be deeply appreciated as I am not sure where to focus on at this point.

Kind regards,

Markel Stefo

[marce1000]

 - FIrst of all, make sure that the involved physical connections don't have any connectivity problems.

   M.

[digicom]

Hi Marce,

First of all thanks for the response!

I have already check physical connections and everything looks fine no errors or anything on the interfaces.

Also as I have mentioned the issue happens with all peers that are connected to the router even though they are on different cards and different ports on the router.

Any other suggestion would be deeply appreciated.

Kind regards,

Markel

[marce1000]

 - Ok, make sure to have permanannt analysis-tools activated on the router such as a syslog server (enabled logging for all syslog server). 2) Same for snmp trap logging to an snmp-trap receiver. Followup for both logging-protocols at regular intervals also when everything seems normal. Check current software version being used on the 7606, make sure it is not too old w.r.t current availabel releases, if so , consider upgrading, then check if there is any improvement.

 M.

[Deepak Kumar]

Hi,

As you mentioned you had verified your Layer 1 connectivity then check for any bug in the software but same time I noticed something as:

Sep 26 2019 21:42:15 EET1: %OSPF-4-CONFLICTING_LSAID: Process xxxx area dummy area: LSA origination prevented by LSA with same LSID but a different mask
Existing Type 5 LSA: LSID x.x.x.x/23

Can you check your routing configuration? is it configured correctly? Why am I asking because your OSPF is giving an error with LSA 5 as Dummy area with a different mask? It may not a root cause but you have to verify that why is it? 

[digicom · Accepted solution]

Hi,

Just wanted to update the case, I still have not been able to pinpoint the issue but I have reduced the routing table of the router from approximately 700K to 250K routes and the issue has not appeared anymore.

Hopping not to face the issue again, and that this info might be helpful for others facing similar issue.

Regards,

Markel