2811 ROUTER - REBOOT PROBLEM

[STYLIANOS DEMETRIOU]

Hi guys,

I've implemented dual internet on my cisco 2811 router, everything is working properly and after a period of time the router gives the below message thousand of times and reboots

%SYS-2-INTSCHED: 'suspend' at level 3 -Process= "IP NAT Ager

Please see below my configuration and let me know if i have something wrong

Current configuration : 19481 bytes

!

version 12.4

service timestamps debug datetime msec

service timestamps log datetime msec

service password-encryption

!

hostname test

!

boot-start-marker

boot-end-marker

!

logging message-counter syslog

logging buffered 32568

!

aaa new-model

!

!

aaa session-id common

clock timezone EEDT 2

clock summer-time EEDT recurring last Sun Mar 3:00 last Sun Oct 4:00

!

dot11 syslog

ip source-route

ip auth-proxy max-nodata-conns 3

ip admission max-nodata-conns 3

!

!

ip cef

!

!

ip domain name neocleous.com

ip inspect tcp reassembly queue length 756

ip inspect name CBAC tcp

ip inspect name CBAC udp

ip inspect name CBAC icmp

ip inspect name CBAC ftp

ip inspect name CBAC http

ip inspect name CBAC imap

ip inspect name CBAC imap3

no ipv6 cef

!

multilink bundle-name authenticated

!

voice dsp waitstate 0

!

!

!

voice-card 0

no dspfarm

!

track 1 ip sla 1 reachability

delay down 1 up 1

!

track 2 ip sla 2 reachability

delay down 1 up 1

!

!

!

!

interface FastEthernet0/0

description PRIMETEL

bandwidth 8000

ip address [public ip] 255.255.255.240

ip access-group CBAC-OUT-NEW in

ip nat outside

ip virtual-reassembly

load-interval 30

duplex auto

speed auto

!

interface FastEthernet0/1

description INSIDE-TO-LAN

ip address 192.168.100.x 255.255.255.240

ip nat inside

ip inspect CBAC in

ip virtual-reassembly

ip policy route-map PBR

duplex full

speed 100

!

interface FastEthernet0/0/0

description CYTA

ip address [public ip 2] 255.255.255.248

ip access-group CBAC-OUT-NEW in

ip nat outside

ip virtual-reassembly

duplex full

speed 100

!

ip forward-protocol nd

ip route 0.0.0.0 0.0.0.0 [gateway public ip] track 1

ip route 0.0.0.0 0.0.0.0 [gateway public ip 2] track 2

!

!

no ip http server

no ip http secure-server

ip nat inside source route-map CYTA interface FastEthernet0/0/0 overload

ip nat inside source route-map PRIMETEL interface FastEthernet0/0 overload

ip nat inside source static 192.168.100.13 [an ip in the range of public ip 2] route-map CYTA-STATIC-NAT

ip nat inside source static 192.168.3.23 [an ip in the range of public ip 2] route-map CYTA-STATIC-NAT

ip nat inside source static 192.168.100.13 [an ip in the range of public ip] route-map PRIMETEL-STATIC-NAT

ip nat inside source static 192.168.3.23 [an ip in the range of public ip] route-map PRIMETEL-STATIC-NAT

!

ip access-list extended ALLOW-EVERYWHEREACCESS

permit ip 192.168.0.0 0.0.255.255 any

ip access-list extended ALLOW-WEB

permit tcp 192.168.0.0 0.0.255.255 any eq www

permit tcp 192.168.0.0 0.0.255.255 any eq 443

permit tcp 192.168.0.0 0.0.255.255 any eq smtp

ip access-list extended CBAC-OUT-NEW

permit ip host 192.168.20.1 any

ip access-list extended PBR-NOFILTER

permit ip 192.168.0.0 0.0.255.255 any

ip access-list extended STATIC-NAT

permit ip host 192.168.100.x any

permit ip host 192.168.3.x any

ip access-list extended WEBFILTERING

permit tcp 192.168.0.0 0.0.255.255 any eq smtp

permit ip 192.168.101.0 0.0.0.255 any

permit ip 192.168.100.0 0.0.0.255 any

deny   tcp any any eq 443

permit ip any any

!

ip sla 1

icmp-echo [gateway public ip]

timeout 500

frequency 1

ip sla schedule 1 life forever start-time now

ip sla 2

icmp-echo [gateway public ip]

timeout 500

frequency 1

ip sla schedule 2 life forever start-time now

logging 192.168.3.x

snmp-server community key RO

snmp-server location Internet Router 2

snmp-server contact x

snmp-server enable traps snmp authentication linkdown linkup coldstart warmstart

snmp-server enable traps vrrp

snmp-server enable traps ds1

snmp-server enable traps tty

snmp-server enable traps eigrp

snmp-server enable traps xgcp

snmp-server enable traps flash insertion removal change

snmp-server enable traps ds3

snmp-server enable traps envmon

snmp-server enable traps icsudsu

snmp-server enable traps isdn call-information

snmp-server enable traps isdn layer2

snmp-server enable traps isdn chan-not-avail

snmp-server enable traps isdn ietf

snmp-server enable traps ds0-busyout

snmp-server enable traps ds1-loopback

snmp-server enable traps ethernet cfm cc mep-up mep-down cross-connect loop config

snmp-server enable traps ethernet cfm crosscheck mep-missing mep-unknown service-up

snmp-server enable traps license

snmp-server enable traps disassociate

snmp-server enable traps deauthenticate

snmp-server enable traps authenticate-fail

snmp-server enable traps dot11-qos

snmp-server enable traps switch-over

snmp-server enable traps rogue-ap

snmp-server enable traps wlan-wep

snmp-server enable traps aaa_server

snmp-server enable traps atm subif

snmp-server enable traps bgp

snmp-server enable traps bulkstat collection transfer

snmp-server enable traps memory bufferpeak

snmp-server enable traps cnpd

snmp-server enable traps config-copy

snmp-server enable traps config

snmp-server enable traps config-ctid

snmp-server enable traps dial

snmp-server enable traps dsp card-status

snmp-server enable traps dsp oper-state

snmp-server enable traps entity

snmp-server enable traps fru-ctrl

snmp-server enable traps resource-policy

snmp-server enable traps event-manager

snmp-server enable traps frame-relay multilink bundle-mismatch

snmp-server enable traps frame-relay

snmp-server enable traps frame-relay subif

snmp-server enable traps hsrp

snmp-server enable traps ipmulticast

snmp-server enable traps mpls ldp

snmp-server enable traps mpls traffic-eng

snmp-server enable traps mpls fast-reroute protected

snmp-server enable traps msdp

snmp-server enable traps mvpn

snmp-server enable traps pim neighbor-change rp-mapping-change invalid-pim-message

snmp-server enable traps pppoe

snmp-server enable traps cpu threshold

snmp-server enable traps rsvp

snmp-server enable traps ipsla

snmp-server enable traps syslog

snmp-server enable traps l2tun session

snmp-server enable traps l2tun pseudowire status

snmp-server enable traps vtp

snmp-server enable traps pw vc

snmp-server enable traps firewall serverstatus

snmp-server enable traps ipmobile

snmp-server enable traps rf

snmp-server enable traps isakmp policy add

snmp-server enable traps isakmp policy delete

snmp-server enable traps isakmp tunnel start

snmp-server enable traps isakmp tunnel stop

snmp-server enable traps ipsec cryptomap add

snmp-server enable traps ipsec cryptomap delete

snmp-server enable traps ipsec cryptomap attach

snmp-server enable traps ipsec cryptomap detach

snmp-server enable traps ipsec tunnel start

snmp-server enable traps ipsec tunnel stop

snmp-server enable traps ipsec too-many-sas

snmp-server enable traps ccme

snmp-server enable traps srst

snmp-server enable traps mpls vpn

snmp-server enable traps voice

snmp-server enable traps dnis

snmp-server host 192.168.3.x version 2c key

!

!

!

!

route-map CYTA permit 10

match ip address NATacl-NEW

match interface FastEthernet0/0/0

route-map PRIMETEL permit 10

match ip address NATacl-NEW

match interface FastEthernet0/0

!

route-map PBR permit 10

match ip address ALLOW-WEB

set ip next-hop verify-availability [gateway public ip ] 1 track 10

!

route-map PBR permit 30

match ip address ALLOW-EVERYWHEREACCESS

set ip next-hop verify-availability [gateway public ip 2] 2 track 20

!

route-map CYTA-STATIC-NAT permit 10

match ip address STATIC-NAT

match interface FastEthernet0/0/0

!

route-map PRIMETEL-STATIC-NAT permit 10

match ip address STATIC-NAT

match interface FastEthernet0/0

!

!

!

control-plane

!

!

!

ccm-manager fax protocol cisco

!

mgcp fax t38 ecm

!

[Richard Burts]

I do not see any obvious issues in the configuration that you posted. So I have a few questions which may help us to give you some advice:

- when the problem occurs are there other console messages (especially anything like traceback) or is the message you refer to the only console message that is generated?

- has this router been running for a while with out issue? Or is this a fairly new router and the problem started immediately?

- am I correct in understanding that adding a second Internet was a recent change? Or were both Internet connections done at the same time?

- is this router covered by a maintenance contract?

This error message indicates a problem with the scheduler process in IOS. It is most likely a software bug. If the router is covered by a maintenance contract then opening a case with Cisco TAC is probably the best thing to do since they have better tools to diagnose the specific problem.

HTH

Rick

[STYLIANOS DEMETRIOU]

This router was kept offline us a backup (it's not a new router, we have it 4 years) and it has about the same configuration with the online router except ip sla,tracking and PBR

(We were requested to implement PBR, and have the primary internet line which is 24mbps to allow users to access the internet using www and ssl, and the other internet line to take care for any other traffic)

This router is located in the H.Q and handles around 100 of users also vpn is passing through from other remote offices.

(Total of 150 users)

During launch time when 10-20 users are working i don't see any problem and everything is working properly.

After launch time, the routers works fine for 15-30 minutes and then it reboots.

As far as i am concerned no it's not under maintenance contract.

I received a traceback please see below all output and advise

   Possible software fault. Upon reccurence,  please collect

   crashinfo, "show tech" and contact Cisco Technical Support.

--------------------------------------------------------------------

-Traceback= 0x40086950 0x43EE0BAC 0x42057CA8 0x4207BBC0 0x4207247C 0x42043FC8 0x41ECD4FC 0x41ECD680 0x41EB90F4 0x41EB9B44 0x41EB9FB4 0x41EBA070 0x41EBA2B8

$0 : 00000000, AT : 46B00000, v0 : 47150000, v1 : 00000000

a0 : 00000000, a1 : 00000000, a2 : 00000000, a3 : 47F9A8D8

t0 : 46ED0000, t1 : 3400E101, t2 : 461B0000, t3 : 00000005

t4 : 00000008, t5 : 461B0000, t6 : 0209B4A0, t7 : 0000000A

s0 : 42057CA0, s1 : 49131E88, s2 : 0B0D0B0D, s3 : 00000000

s4 : 00000000, s5 : 00000000, s6 : 00000000, s7 : 00000000

t8 : 04000000, t9 : 00000332, k0 : 3040A801, k1 : A000F000

gp : 46B0BCD0, sp : 47F9A230, s8 : 46B00000, ra : 40086950

EPC  : 40086950, ErrorEPC : BFC0900C, SREG     : 3400E103

MDLO : 00000006, MDHI     : 00000000, BadVaddr : 0B0D0AF9

DATA_START : 0x43F4AAB0

Cause 00000010 (Code 0x4): Address Error (load or instruction fetch) exception

=== Start of Crashinfo Collection (13:37:01 EEDT Tue Nov 26 2013) ===

For image:

Cisco IOS Software, 2800 Software (C2800NM-ADVIPSERVICESK9-M), Version 12.4(20)YA1, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2008 by Cisco Systems, Inc.

Compiled Fri 29-Aug-08 04:20 by prod_rel_team

[Sandeep Choudhary · Accepted solution]

HI,

May be You're hitting this bug: CSCed28542.

http://www.cisco.com/en/US/docs/ios/12_3t/release/notes/123tcavs.html

you need to Upgrade your IOS.

Regards

Dont forget to rate helpful posts.

[STYLIANOS DEMETRIOU]

I don't think  that i am hitting above mentioned bug since my IOS is 12.4(20) Y which is not included in "Known affected releases". listed in that hyperlink.

See below

Cisco IOS Software, 2800 Software (C2800NM-ADVIPSERVICESK9-M), Version 12.4(20)Y                                                                                        A1, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2008 by Cisco Systems, Inc.

Compiled Fri 29-Aug-08 04:20 by prod_rel_team

ROM: System Bootstrap, Version 12.4(13r)T, RELEASE SOFTWARE (fc1)

Which upgrade paths do i have for the current IOS, in order to exclude the possibility that it might be a software bug? i have 256 MB OF DRAM

[Richard Burts · Accepted solution]

You may not be hitting that specific bug, but the trace back indicates that you are hitting some software problem. A different version of IOS is probably your best alternative.

HTH

Rick

Sent from Cisco Technical Support iPhone App

[STYLIANOS DEMETRIOU]

After upgrading to a newer version the rebooting problem has been resolved.

However i am still facing a problem with nat translation.

I will open a new discussion please reply if you can assist.

Thanks both of you for your helpful answer

[Richard Burts]

I am glad that our suggestions helped you to achieve a solution for this problem. Thank you for using the rating system to mark this question as answered.

Opening a new thread for a different problem on the router was the right approach. I have posted some suggestions in your new thread. I hope they will be helpful.

HTH

Rick