Buy or Renew
Buy or Renew
Cisco + Splunk: It’s a new day for your data. Learn more.
 
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
17280
Views
0
Helpful
2
Replies

High CPU Utilisation issue - IP_VFR-4-FRAG_TABLE_OVERFLOW

rashidsiddiqui
Level 1
Level 1

‎01-11-2011 09:33 PM - edited ‎03-04-2019 11:03 AM

Hello Friends,

I have a cisco 2821 router (Cisco 2821 (revision 53.51) with 247808K/14336K bytes of memory),

The CPU utilization is going high,

When interface traffic on Gi0/1 increses, CPU Utilization of device increases,

RouterL#show ip int brief
Interface                  IP-Address      OK? Method Status                Protocol
GigabitEthernet0/0         172.23.133.12   YES NVRAM  up                    up
GigabitEthernet0/1         Public IP   YES NVRAM  up                    up
FastEthernet0/0/0          10.216.111.106  YES NVRAM  up                    down
FastEthernet0/0/1          10.216.111.110  YES NVRAM  up                    up
NVI0                             172.23.133.12   YES unset  up                    up
Tunnel190                  10.216.107.98   YES NVRAM  up                    up
Tunnel191                  10.216.107.101  YES NVRAM  up                    down
RouterL#

ROUTER#show process cpu

CPU utilization for five seconds: 87%/79%; one minute: 86%; five minutes: 95%

I have checked the logs, Following is the logs observed,

ROUTER#show logging
Syslog logging: enabled (556138 messages dropped, 146 messages rate-limited,
                0 flushes, 0 overruns, xml disabled, filtering disabled)

No Active Message Discriminator.
No Inactive Message Discriminator.
    Console logging: disabled
    Monitor logging: level debugging, 120976 messages logged, xml disabled,
                     filtering disabled
    Buffer logging:  level debugging, 159547 messages logged, xml disabled,
                     filtering disabled
    Logging Exception size (4096 bytes)
    Count and timestamp logging messages: disabled
    Persistent logging: disabled
    Trap logging: level debugging, 58779 message lines logged
        Logging to 10.216.16.70  (udp port 514,  audit disabled,
              authentication disabled, encryption disabled, link up),
              58511 message lines logged,
              0 message lines rate-limited,
              0 message lines dropped-by-MD,
              xml disabled, sequence number disabled
              filtering disabled

Log Buffer (4096 bytes):
bitEthernet0/0: the fragment table has reached its maximum threshold 16
.Jan 12 05:05:40: %SEC-6-IPACCESSLOGP: list 110 denied tcp 125.17.126.10(0) -> 173.236.91.148(0), 1 packet
.Jan 12 05:05:41: %SEC-6-IPACCESSLOGDP: list 110 denied icmp 125.17.126.10 -> 173.192.153.141 (0/0), 2 packets
.Jan 12 05:05:41: %SEC-6-IPACCESSLOGDP: list 110 denied icmp 203.101.52.168 -> 173.192.153.141 (0/0), 1 packet
.Jan 12 05:05:44: %SEC-6-IPACCESSLOGDP: list 110 denied icmp 203.101.52.174 -> 196.202.246.4 (0/0), 1 packet
.Jan 12 05:05:47: %SEC-6-IPACCESSLOGDP: list 110 denied icmp 203.101.52.165 -> 196.202.246.4 (0/0), 1 packet
.Jan 12 05:05:57: %SEC-6-IPACCESSLOGDP: list 110 denied icmp 203.101.52.128 -> 190.232.139.252 (0/0), 1 packet
.Jan 12 05:05:59: %SEC-6-IPACCESSLOGDP: list 110 denied icmp 203.101.52.147 -> 190.232.139.252 (0/0), 1 packet
.Jan 12 05:06:00: %SEC-6-IPACCESSLOGDP: list 110 denied icmp 203.101.52.179 -> 190.232.139.252 (0/0), 1 packet
.Jan 12 05:06:05: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet0/0/1 (not half duplex), with NOIDA-WAN-CORE-RTR-1 GigabitEthernet1/18 (half duplex).
.Jan 12 05:06:12: %IP_VFR-4-FRAG_TABLE_OVERFLOW: GigabitEthernet0/0: the fragment table has reached its maximum threshold 16
.Jan 12 05:06:41: %SEC-6-IPACCESSLOGRL: access-list logging rate-limited or missed 295 packets
.Jan 12 05:06:44: %IP_VFR-4-FRAG_TABLE_OVERFLOW: GigabitEthernet0/0: the fragment table has reached its maximum threshold 16
.Jan 12 05:06:56: %SEC-6-IPACCESSLOGP: list 110 denied tcp 125.17.126.10(0) -> 174.122.47.26(0), 1 packet
.Jan 12 05:07:05: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet0/0/1 (not half duplex), with NOIDA-WAN-CORE-RTR-1 GigabitEthernet1/18 (half duplex).
.Jan 12 05:07:06: %SEC-6-IPACCESSLOGP: list 110 denied tcp 125.17.126.10(0) -> 216.36.248.248(0), 1 packet
.Jan 12 05:07:13: %SEC-6-IPACCESSLOGP: list 110 denied tcp 125.17.126.10(0) -> 114.27.8.196(0), 1 packet
.Jan 12 05:07:14: %IP_VFR-4-FRAG_TABLE_OVERFLOW: GigabitEthernet0/0: the fragment table has reached its maximum threshold 16
.Jan 12 05:07:39: %SEC-6-IPACCESSLOGDP: list 110 denied icmp 203.101.52.183 -> 95.83.104.193 (0/0), 1 packet
.Jan 12 05:07:41: %SEC-6-IPACCESSLOGRL: access-list logging rate-limited or missed 9 packets
.Jan 12 05:07:45: %IP_VFR-4-FRAG_TABLE_OVERFLOW: GigabitEthernet0/0: the fragment table has reached its maximum threshold 16
.Jan 12 05:08:05: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet0/0/1 (not half duplex), with NOIDA-WAN-CORE-RTR-1 GigabitEthernet1/18 (half duplex).
.Jan 12 05:08:15: %IP_VFR-4-FRAG_TABLE_OVERFLOW: GigabitEthernet0/0: the fragment table has reached its maximum threshold 16
.Jan 12 05:08:34: %SEC-6-IPACCESSLOGP: list 110 denied tcp 125.17.126.10(0) -> 212.59.148.87(0), 1 packet
.Jan 12 05:08:41: %SEC-6-IPACCESSLOGRL: access-list logging rate-limited or missed 2 packets
.Jan 12 05:08:46: %IP_VFR-4-FRAG_TABLE_OVERFLOW: GigabitEthernet0/0: the fragment table has reached its maximum threshold 16
.Jan 12 05:08:58: %SEC-6-IPACCESSLOGP: list 110 denied tcp 125.17.126.10(0) -> 114.42.110.221(0), 1 packet
.Jan 12 05:09:05: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet0/0/1 (not half duplex), with NOIDA-WAN-CORE-RTR-1 GigabitEthernet1/18 (half duplex).
.Jan 12 05:09:16: %IP_VFR-4-FRAG_TABLE_OVERFLOW: GigabitEthernet0/0: the fragment table has reached its maximum threshold 16
.Jan 12 05:09:40: %SEC-6-IPACCESSLOGRL: access-list logging rate-limited or missed 2 packets
.Jan 12 05:10:05: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet0/0/1 (not half duplex), with NOIDA-WAN-CORE-RTR-1 GigabitEthernet1/18 (half duplex).
.Jan 12 05:10:35: %SEC-6-IPACCESSLOGP: list 110 denied tcp 125.17.126.10(0) -> 122.116.206.105(0), 1 packet
.Jan 12 05:10:40: %SEC-6-IPACCESSLOGRL: access-list logging rate-limited or missed 2 packets
.Jan 12 05:11:05: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet0/0/1 (not half duplex), with NOIDA-WAN-CORE-RTR-1 GigabitEthernet1/18 (half duplex).
ROUTER#

ROUTER#
ROUTER#show ip virtual-reassembly GigabitEthernet0/0
GigabitEthernet0/0:
   Virtual Fragment Reassembly (VFR) is ENABLED...
   Concurrent reassemblies (max-reassemblies): 16
   Fragments per reassembly (max-fragments): 32
   Reassembly timeout (timeout): 3 seconds
   Drop fragments: OFF

   Current reassembly count:3
   Current fragment count:6
   Total reassembly count:219967
   Total reassembly timeout count:8265


ROUTER#

Following is the Show ip traffic details,

ROUTER#show ip traffic
IP statistics:
  Rcvd:  5595185 total, 3577131 local destination
         0 format errors, 0 checksum errors, 6358 bad hop count
         0 unknown protocol, 5 not a gateway
         0 security failures, 0 bad options, 0 with options
  Opts:  0 end, 0 nop, 0 basic security, 0 loose source route
         0 timestamp, 0 extended security, 0 record route
         0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump
         0 other
  Frags: 514890 reassembled, 55367 timeouts, 0 couldn't reassemble
         0 fragmented, 0 fragments, 0 couldn't fragment
  Bcast: 16 received, 0 sent
  Mcast: 796629 received, 87672 sent
  Sent:  317343 generated, 511135999 forwarded
  Drop:  8 encapsulation failed, 0 unresolved, 0 no adjacency
         17 no route, 17330 unicast RPF, 0 forced drop
         0 options denied
  Drop:  0 packets with source IP address zero
  Drop:  0 packets with internal loop back IP address
         0 physical broadcast

ICMP statistics:
  Rcvd: 3 format errors, 0 checksum errors, 0 redirects, 6 unreachable
        3120 echo, 6 echo reply, 0 mask requests, 0 mask replies, 0 quench
        0 parameter, 0 timestamp, 0 timestamp replies, 0 info request, 0 other
        0 irdp solicitations, 0 irdp advertisements
        0 time exceeded, 0 info replies
  Sent: 0 redirects, 0 unreachable, 20 echo, 3120 echo reply
        0 mask requests, 0 mask replies, 0 quench, 0 timestamp, 0 timestamp replies
        0 info reply, 6358 time exceeded, 0 parameter problem
        0 irdp solicitations, 0 irdp advertisements

TCP statistics:
  Rcvd: 20993 total, 1 checksum errors, 499 no port
  Sent: 20911 total

BGP statistics:
  Rcvd: 0 total, 0 opens, 0 notifications, 0 updates
        0 keepalives, 0 route-refresh, 0 unrecognized
  Sent: 0 total, 0 opens, 0 notifications, 0 updates
        0 keepalives, 0 route-refresh

IP-EIGRP statistics:
  Rcvd: 0 total
  Sent: 0 total

PIMv2 statistics: Sent/Received
  Total: 2758/2532, 0 checksum errors, 0 format errors
  Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0,  Hellos: 2582/2523
  Join/Prunes: 0/0, Asserts: 0/0, grafts: 176/0
  Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
  Queue drops: 0
  State-Refresh: 0/0

IGMP statistics: Sent/Received
  Total: 1278/1265, Format errors: 0/0, Checksum errors: 0/0
  Host Queries: 13/1265, Host Reports: 1265/0, Host Leaves: 0/0
  DVMRP: 0/0, PIM: 0/0
  Queue drops: 0

UDP statistics:
  Rcvd: 1634731 total, 25 checksum errors, 938 no port
  Sent: 143697 total, 0 forwarded broadcasts

OSPF statistics:
  Rcvd: 0 total, 0 checksum errors
        0 hello, 0 database desc, 0 link state req
        0 link state updates, 0 link state acks

  Sent: 0 total
        0 hello, 0 database desc, 0 link state req
        0 link state updates, 0 link state acks

ARP statistics:
  Rcvd: 6596 requests, 25 replies, 0 reverse, 0 other
  Sent: 73 requests, 3239 replies (5 proxy), 0 reverse
  Drop due to input queue full: 0
ROUTER#
ROUTER#

The Memory utilization is ,

ROUTER#show processes memory
Processor Pool Total:  168481024 Used:   36920964 Free:  131560060
      I/O Pool Total:   14680064 Used:    7101744 Free:    7578320

PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process

RouterL#show ip int brief
Interface                  IP-Address      OK? Method Status                Protocol
GigabitEthernet0/0         172.23.133.12   YES NVRAM  up                    up
GigabitEthernet0/1         Public IP   YES NVRAM  up                    up
FastEthernet0/0/0          10.216.111.106  YES NVRAM  up                    down
FastEthernet0/0/1          10.216.111.110  YES NVRAM  up                    up
NVI0                       172.23.133.12   YES unset  up                    up
Tunnel190                  10.216.107.98   YES NVRAM  up                    up
Tunnel191                  10.216.107.101  YES NVRAM  up                    down
RouterL#

Also Can anybody tell What is the NVI0 interface?

RouterL#show int GigabitEthernet0/0 | in rate
  Queueing strategy: fifo
  5 minute input rate 44957000 bits/sec, 17010 packets/sec
  5 minute output rate 6671000 bits/sec, 10597 packets/sec
RouterL#

RouterL#show int GigabitEthernet0/1 | in rate
  Queueing strategy: fifo
  30 second input rate 6174000 bits/sec, 11091 packets/sec
  30 second output rate 46785000 bits/sec, 17758 packets/sec
RouterL#

RouterL#show int FastEthernet0/0/1 | in rate
  Queueing strategy: fifo
  5 minute input rate 906000 bits/sec, 125 packets/sec
  5 minute output rate 1000 bits/sec, 1 packets/sec
RouterL#

Labels:
0 Helpful
2 Replies 2

Peter Paluch
Cisco Employee
Cisco Employee

‎01-16-2011 12:53 PM

Rashid,

I am not sure if this case is still active.

Just in case, a few ideas.

  1. You appear to be using GigE and Tunnel interfaces. By itself, a high load can cause the CPU to spike up as there is no hardware acceleration for packet routing in 2800 series routers. Are you performing any monitoring via SNMP? It would be very helpful to perform data collection from interfaces (packets in, packets out) along with the CPU load and to see whether they correlate in time. So far we have no clear clue as to what is causing your CPU spikes. In any case, your CPU is spending most of the time serving interrupts.
  2. Decrease the usage of the log option in your ACLs to a necessary minimum. Logging several hits on ACL may contribute to CPU load.

  3. The message %IP_VFR-4-FRAG_TABLE_OVERFLOW suggests that your router is handling lots of fragmented IP packets and is running low on the buffer that holds the data to identify the fragments. This buffer can be enlarged using the command ip virtual-reassembly max-reassemblies 512 on your Gi0/0 interface to allow 512 packets to be virtually reassembled for ACL purposes at a time. However, a more appropriate course of action would be to identify why are you receiving so many fragmented packets. Do you have any way of performing a traffic analysis on the Gi0/0 interface, perhaps using a sniffer?

Best regards,

Peter

0 Helpful

rashidsiddiqui
Level 1
Level 1
In response to Peter Paluch

‎01-16-2011 09:38 PM

Hi Peter,

We are in course of the investigation, will update as we get some improvement.

0 Helpful
Customers Also Viewed These Support Documents
Review Cisco Networking products for a $25 gift card