Router outage mystery

steven.crutchley · ‎08-01-2012

Yesterday our customer lost internet connectivity on their lease line at about 4:30pm. After about an hour the router was rebooted and internet connectivity was restored. The customer is requesting an RFO (reason for outage).

Upon looking at the cisco logs I get the following:

*Jul 31 16:36:45.175: %ESWMRVL_FLTMG-5-NOTICE: Notice: FPGA Rev 0x71

*Jul 31 16:36:55.291: %VPN_HW-6-INFO_LOC: Crypto engine: onboard 0 State changed to: Initialized

*Jul 31 16:36:55.295: %VPN_HW-6-INFO_LOC: Crypto engine: onboard 0 State changed to: Enabled

*Jul 31 16:36:55.783: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to up

*Jul 31 16:36:55.783: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up

*Jul 31 16:36:56.707: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down

*Jul 31 16:36:56.943: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to up

*Jul 31 16:36:56.943: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to up

*Jul 31 16:36:57 GMT: %SYS-6-CLOCKUPDATE: System clock has been updated from 16:36:57 UTC Tue Jul 31 2012 to 16:36:57 GMT Tue Jul 31 2012, configured from console by console.

*Jul 31 17:36:57 BST: %SYS-6-CLOCKUPDATE: System clock has been updated from 16:36:57 GMT Tue Jul 31 2012 to 17:36:57 BST Tue Jul 31 2012, configured from console by console.

*Jul 31 17:37:05 BST: %SYS-5-CONFIG_I: Configured from memory by console

*Jul 31 17:37:07 BST: %LINK-3-UPDOWN: Interface FastEthernet0/0/0, changed state to up

*Jul 31 17:37:08 BST: %SYS-5-RESTART: System restarted --

*Jul 31 17:37:08 BST: %SNMP-5-COLDSTART: SNMP agent on host XXXXXX-XXXXXXXX is undergoing a cold start

*Jul 31 17:37:08 BST: %SSH-5-ENABLED: SSH 2.0 has been enabled

*Jul 31 17:37:08 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0/0, changed state to up

*Jul 31 17:37:08 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0/1, changed state to down

*Jul 31 17:37:08 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0/2, changed state to down

*Jul 31 17:37:08 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0/3, changed state to down

*Jul 31 17:37:08 BST: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF

*Jul 31 17:37:08 BST: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF

*Jul 31 17:37:08 BST: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF

*Jul 31 17:37:08 BST: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF

*Jul 31 17:37:09 BST: %BGP_SESSION-5-ADJCHANGE: neighbor x.x.x.x IPv4 Unicast topology base removed from session BGP Notification received

*Jul 31 17:37:09 BST: %BGP_SESSION-5-ADJCHANGE: neighbor y.y.y.y IPv4 Unicast vpn vrf INTERNET topology base removed from session BGP Notification received

*Jul 31 17:37:26 BST: %BGP-5-ADJCHANGE: neighbor x.x.x.x Up

*Jul 31 17:37:26 BST: %BGP-5-ADJCHANGE: neighbor y.y.y.y vpn vrf INTERNET Up

*Jul 31 17:37:28 BST: %TRACKING-5-STATE: 123 ip sla 1 reachability Down->Up

*Jul 31 17:37:28 BST: %TRACKING-5-STATE: 124 ip sla 2 reachability Down->Up

*Jul 31 17:37:28 BST: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 1 state Listen -> Active

*Jul 31 17:37:38 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to up

*Jul 31 17:37:44 BST: %HSRP-5-STATECHANGE: Vlan1 Grp 2 state Listen -> Active

Can anyone advise what the problem may be? I am not sure what %ESWMRVL_FLTMG-5-NOTICE means.

Thanks in advance.

Sandeep Choudhary · ‎08-01-2012

It seems the problem of memmory leak:

It can be normal to see the Dead process holding memory. The Dead process is simply a holding area for memory that was allocated by a process that was released and the memory is still being used. A good example of Dead memory usage is when you telnet into a router, make a config change then kill the telnet session. When you initually telnet into the router a Virtual Exec process is created to deal with the telnet session. While telneted in, if you go into config mode and increase the logging buffered to say 10mb then 10mb of memory will be allocated for the log. The allocation happened under the Virtual Exec process (a show mem proc would show Virtual Exec holding 10mb). When the telnet session is killed, the memory allocated by Virtual Exec is still needed, therefore the 10mb is moved to the Dead process and the Virtual Exec process is removed. Now, if the router is rebooted, the 10mb allocated for the Log will now show up under the Init process (Init allocates memory when the router is booting).

Seeing 1mb in Dead is nothing to be concerned about. If you saw 30mb or more then it may warrent a closer look. As far as the router rebooting, a show version will tell you when the last reboot occurred and the reason for it. If it was a crash, you should see the reboot reason be something like "Bus error" or "Segv". A crash should also result in a crashinfo being produced and saved to the flash.

Just contact with your device provider or Cisco and create a TAC CASE with them.

Regards

Please rate if it helps.

steven.crutchley · ‎08-01-2012

Hi Sandeep,

Thank you for your response however I do not understand your explanation.

If I may, I would like to ask a few questions:

1) In my provided output above what tells you this is a memory problem?

2) How could memory being used in dead memory cause the internet connectivity to fail? The router was not rebooted for an hour after internet connectivty was lost.

3) How likely, if at all, is this likely to recur and why?

Sorry I know the last question is a heavy one but your explanation does not really reference any of my output.

Alessio Andreoli · ‎08-01-2012

read here because it can be very useful. For your RFO just hardware failure unless it is an important customer.

https://supportforums.cisco.com/thread/2086848?tstart=0#2086848

PS: don't rate me for this post because some other colleagues were working on it

Alessio

Leo Laohoo · ‎08-01-2012

Yesterday our customer lost internet connectivity on their lease line at about 4:30pm. After about an hour the router was rebooted and internet connectivity was restored.

Honestly, I would find it hard to determine the RFO if the link was restored by rebooting the router.

I mean, the main question I would like to ask is this: What did the "sh logs" show BEFORE the router was rebooted.