C1900 series WAN router Issue

[sheraz butt]

Hi all,

   I had a wan router (eth0/0 public-ip/eth0/1 LAN )20/20 up/download dadicated link NAT is enabled everything was working perfect but since last 3 days every 30 min or so net gets drop or port gets hang  we just unplug wan cable and plug it again then all works normally for 30 or  20 mins and hang again.

any help for troubleshooting.

CONFIG,

interface Tunnel10
description tunnel for SER
ip unnumbered GigabitEthernet0/0
tunnel source GigabitEthernet0/0
tunnel destination 83.83.83.15
!
interface GigabitEthernet0/0
ip address 88.88.88.60 255.255.255.252
ip nat outside
ip virtual-reassembly
duplex auto
speed auto
!
interface GigabitEthernet0/1
ip address 10.1.1.1 255.255.255.0
ip nat inside
ip virtual-reassembly
duplex auto
speed auto
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
ip nat inside source list 1 interface GigabitEthernet0/0 overload
ip nat inside source static tcp 10.1.1.222 10000 interface GigabitEthernet0/0
10000
ip nat inside source static tcp 10.1.1.200 80 interface GigabitEthernet0/0 80
ip nat inside source static tcp 10.1.1.200 22 interface GigabitEthernet0/0 22
ip route 0.0.0.0 0.0.0.0 86.86.86.50
ip route 83.83.83.0 255.255.255.0 Tunnel10
!
access-list 1 permit 10.1.1.0 0.0.0.255
!
!
ERROR LOGS

*Nov 1 21:00:07.146: %SYS-5-CONFIG_I: Configured from console by vty1 (87.101.143.150)
*Nov 1 21:09:23.822: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to down
*Nov 1 21:09:24.822: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to down
*Nov 1 21:09:30.122: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel10, changed state to down
*Nov 1 21:10:27.822: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to up
*Nov 1 21:10:28.822: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to up
*Nov 1 21:10:30.158: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel10, changed state to up
*Nov 1 21:11:09.822: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to down
*Nov 1 21:11:10.194: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel10, changed state to down
*Nov 1 21:11:10.822: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to down
*Nov 1 21:11:17.822: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to up
*Nov 1 21:11:18.822: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to up
*Nov 1 21:11:20.202: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel10, changed state to up

some previous LOGS

*Nov 1 18:24:19.691: %IP_VFR-4-FRAG_TABLE_OVERFLOW: GigabitEthernet0/0: the fragment table has reached its maximum threshold 16
*Nov 1 18:24:57.631: %IP_VFR-4-FRAG_TABLE_OVERFLOW: GigabitEthernet0/0: the fragment table has reached its maximum threshold 16
*Nov 1 18:25:34.879: %IP_VFR-4-FRAG_TABLE_OVERFLOW: GigabitEthernet0/0: the fragment table has reached its maximum threshold 16
*Nov 1 18:26:08.119: %IP_VFR-4-FRAG_TABLE_OVERFLOW: GigabitEthernet0/0: the fragment table has reached its maximum threshold 16
*Nov 1 18:26:47.491: %IP_VFR-4-FRAG_TABLE_OVERFLOW: GigabitEthernet0/0: the fragment table has reached its maximum threshold 16
*Nov 1 18:27:54.671: %IP_VFR-4-FRAG_TABLE_OVERFLOW: GigabitEthernet0/0: the fragment table has reached its maximum threshold 16

Regards

[Richard Burts]

We do not have enough information to really be able to identify the problem. Based on what you have provided so far I have these observations:

- the first set of logs show interfaces (both the physical interface and the tunnel that is dependent on the physical interface) changing state. That would seem to point toward some issue with physical connectivity. But we do not know enough about the connection to go any further with what it might be.

- the second set of logs are about table overflow. This would likely be some software issue or perhaps a memory issue.

HTH

Rick

[sheraz butt]

Thank You Sir ,

    Router was working without any issue with same config and conectivity more then 6 month this just happend since last 3 days that port changing state.

i just put this command (ip virtual-reassembly max-fragments 64 max-reassemblies 1024) now and checking if this help 

IOS version is : c1900-universalk9-mz.SPA.150-1.M4.bin no one touch anything els.

please let me know what should i troublshoot.

some errors

*Nov 2 07:05:46.820: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to down
*Nov 2 07:05:47.820: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to down
*Nov 2 07:05:50.820: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to up
*Nov 2 07:05:51.820: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to up
*Nov 2 07:14:05.828: %IPC-2-INVALIDZONE: Invalid IPC Zone 0x20000000. , -Traceback= 0x212C6AD8z 0x212C6BDCz 0x2324F300z 0x2324F2E4z
*Nov 2 07:26:25.820: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to down
*Nov 2 07:26:26.820: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to down

Thanks in Advance

[paul driver]

Hello

Cisco error decoder

%IP_VFR-4-FRAG_TABLE_OVERFLOW:

%IP_VFR-4-FRAG_TABLE_OVERFLOW: [chars]: the fragment table has reached its maximum threshold [dec]

This means the number of datagrams reassembled at a time reached its maximum limit.

Recommended Action: Increase the value of max-reassemblies by entering the ip virtual-reassembly max-reassemblies command.



https://www.cisco.com/cgi-bin/Support/Errordecoder/index.cgi?action=search&index=all&locale=en&query=%25IP_VFR-4-FRAG_TABLE_OVERFLOW%3A&counter=0&paging=5&links=reference&sa=Submit

res
Paul

[sheraz butt]

Thank you sir,

   I did increase ip-virtual rbut now i am getting these errors

*Mar 1 00:00:06.531: %IOS_LICENSE_IMAGE_APPLICATION-6-LICENSE_LEVEL: Module name = c1900 Next reboot level = ipbasek9 and License = ipbasek9
*Nov 2 10:47:08.599: %IFMGR-7-NO_IFINDEX_FILE: Unable to open nvram:/ifIndex-table No such file or directory
*Nov 2 10:47:17.831: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to up
*Nov 2 10:47:17.831: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up
*Nov 2 10:47:19.015: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to up
*Nov 2 10:47:19.015: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to up
*Nov 2 10:47:20.887: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel10, changed state to down
*Nov 2 10:47:20.907: %LINEPROTO-5-UPDOWN: Line protocol on Interface NVI0, changed state to up
*Nov 2 10:47:25.571: %SYS-5-CONFIG_I: Configured from memory by console
*Nov 2 10:47:28.135: %SYS-5-RESTART: System restarted --
Cisco IOS Software, C1900 Software (C1900-UNIVERSALK9-M), Version 15.0(1)M4, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Thu 28-Oct-10 16:26 by prod_rel_team
*Nov 2 10:47:28.319: %SNMP-5-COLDSTART: SNMP agent on host ABCD is undergoing a cold start
*Nov 2 10:47:29.891: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel10, changed state to up

[Richard Burts]

It is good to know that you did increase the max reassemblies. Has the error message about frag table overflow happened again? Or has increasing the limit resolved that issue?

The log messages in your most recent post indicate a reboot on Nov 2. Was this an intentional reboot or was it the result of a crash? Sometimes the output of show version will include information that helps identify the cause of booting the router.

This message appears in your log output

*Nov 2 07:14:05.828: %IPC-2-INVALIDZONE: Invalid IPC Zone 0x20000000. , -Traceback= 0x212C6AD8z 0x212C6BDCz 0x2324F300z 0x2324F2E4z

A traceback is a sign of a software issue. I do not have access to the tools that would tell exactly what this problem is. Has it occurred before or is it a recent thing? Some of these software problems are superficial and do not impact router operation while some are significant and are cause to need to upgrade the version of code. If this router is covered under a maintenance contract I would suggest opening a case with Cisco TAC. They have the tools to identify what causes this message and whether a code upgrade is indicated.

HTH

Rick

[sheraz butt]

I did Increase the value of max-reassemblies till now didn,t  C this error %IP_VFR-4-FRAG_TABLE_OVERFLOW 

Show version saying :normal reload

About this error :%IPC-2-INVALIDZONE: Invalid IPC Zone i did some search saying that could be bug and fixed in latest version of IOS.

NOTE:One thing i came to know there was a power falcuation maybybe that could cause interface reset ??

Regards

[Richard Burts]

If there were a power fluctuation which impacted the device (like provider modem/switch) it could certainly cause the interface on the router to reset.

HTH

Rick