05-22-2013 11:03 PM - edited 03-04-2019 07:59 PM
Dear forum,
I has just setup a 2811 router to do only Internet NAT feature, but it seem to be exhausted. Few days ago the it's CPU always above 50%, and this morning it's completely 99 100%. I feel upset when console to it, so lag.
Here is some command output:
RTR-TED-0002#
RTR-TED-0002 04:08:30 AM Thursday May 23 2013 UTC
11 11
999999999999999999999009999990099999999999999999999999999999
999999999999999999999009999990099999999999999999999999999999
100 ************************************************************
90 ************************************************************
80 ************************************************************
70 ************************************************************
60 ************************************************************
50 ************************************************************
40 ************************************************************
30 ************************************************************
20 ************************************************************
10 ************************************************************
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per second (last 60 seconds)
11111
000009999999998888788787866766676654476666766644656766777768
000006169977413199970721046186575588703263055648085430031064
100 #####* *##**
90 #####**######* ** *
80 #####**######*#*##***** * * *
70 #####*#######*#########*# ******** * * **** ** *****#
60 #####*####################*##**###* ******#** ***##*######
50 ###################################**######### *############
40 ############################################################
30 ############################################################
20 ############################################################
10 ############################################################
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
11 1
86993379999344900699999098845989575344677999999 3
4997354899934690019999809757892280429196872999939
100 ** **** *** ******* * * ****
90 ** **** *** ********* * * ******
80 * ** **** *** ********* *** ********
70 **** ***** *** ********* #** * *****##**
60 **#* *#*** ****######*** *#**** *****##**
50 #*#* *#*** *****#######****#***** * ****####*
40 ###* **##*# ******#########**#*#*** **#***####* *
30 ###***###*#***##*##########*##*##*****##*###### *
20 ####**#####**##############*##*##*****######### *
10 ############################################### *
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
RTR-TED-0002#show int fa0/1
FastEthernet0/1 is up, line protocol is up
Hardware is MV96340 Ethernet, address is 0015.fa2f.d539 (bia 0015.fa2f.d539)
Internet address is 10.124.1.5/16
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 57/255, rxload 4/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 17:40:35
Input queue: 0/75/195/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1650000 bits/sec, 1658 packets/sec
5 minute output rate 22564000 bits/sec, 2205 packets/sec
48946016 packets input, 1173184643 bytes
Received 208721 broadcasts, 0 runts, 0 giants, 20 throttles
44779 input errors, 0 CRC, 0 frame, 0 overrun, 44779 ignored
0 watchdog
0 input packets with dribble condition detected
72907133 packets output, 3637260615 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
RTR-TED-0002#show int fa0/0
FastEthernet0/0 is up, line protocol is up
Hardware is MV96340 Ethernet, address is 0015.fa2f.d538 (bia 0015.fa2f.d538)
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 4/255, rxload 58/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 1d22h, output 00:00:00, output hang never
Last clearing of "show interface" counters 17:40:20
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 22753000 bits/sec, 2218 packets/sec
5 minute output rate 1700000 bits/sec, 1647 packets/sec
73338917 packets input, 4276920895 bytes
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
122824 input errors, 0 CRC, 0 frame, 0 overrun, 122824 ignored
0 watchdog
0 input packets with dribble condition detected
48756504 packets output, 1340293905 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
RTR-TED-0002#show int dialer 1
Dialer1 is up, line protocol is up (spoofing)
Hardware is Unknown
Internet address is 113.162.120.22/32
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 20000 usec,
reliability 255/255, txload 6/255, rxload 144/255
Encapsulation PPP, loopback not set
Keepalive set (10 sec)
DTR is pulsed for 1 seconds on reset
Interface is bound to Vi2
Last input never, output never, output hang never
Last clearing of "show interface" counters 2d01h
Input queue: 76/75/71454/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: weighted fair
Output queue: 0/1000/64/0 (size/max total/threshold/drops)
Conversations 0/0/256 (active/max active/max total)
Reserved Conversations 0/0 (allocated/max allocated)
Available Bandwidth 75000 kilobits/sec
5 minute input rate 56848000 bits/sec, 5388 packets/sec
5 minute output rate 2363000 bits/sec, 3583 packets/sec
242770462 packets input, 491800724 bytes
160293318 packets output, 1190208362 bytes
Bound to:
Virtual-Access2 is up, line protocol is up
Hardware is Virtual Access interface
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 20000 usec,
reliability 255/255, txload 6/255, rxload 145/255
Encapsulation PPP, LCP Open
Listen: CDPCP
Open: IPCP
PPPoE vaccess, cloned from Dialer1
Vaccess status 0x44, loopback not set
Keepalive set (10 sec)
DTR is pulsed for 5 seconds on reset
Interface is bound to Di1 (Encapsulation PPP)
Last input 00:00:00, output never, output hang never
Last clearing of "show interface" counters 1d23h
Input queue: 0/75/120991/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 56880000 bits/sec, 5391 packets/sec
5 minute output rate 2425000 bits/sec, 3723 packets/sec
242677192 packets input, 345776708 bytes, 7687 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
160316349 packets output, 1192328063 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
RTR-TED-0002#show ip traffic
IP statistics:
Rcvd: 4273236 total, 2463884 local destination
0 format errors, 0 checksum errors, 63 bad hop count
2 unknown protocol, 0 not a gateway
0 security failures, 0 bad options, 0 with options
Opts: 0 end, 0 nop, 0 basic security, 0 loose source route
0 timestamp, 0 extended security, 0 record route
0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump
0 other
Frags: 7 reassembled, 0 timeouts, 0 couldn't reassemble
35186 fragmented, 70375 fragments, 21334 couldn't fragment
Bcast: 143 received, 0 sent
Mcast: 0 received, 0 sent
Sent: 998921 generated, 419708376 forwarded
Drop: 961 encapsulation failed, 0 unresolved, 0 no adjacency
1155 no route, 0 unicast RPF, 0 forced drop
0 options denied
Drop: 0 packets with source IP address zero
Drop: 0 packets with internal loop back IP address
0 physical broadcast
ICMP statistics:
Rcvd: 87 format errors, 1 checksum errors, 0 redirects, 51566 unreachable
522 echo, 0 echo reply, 0 mask requests, 0 mask replies, 6 quench
0 parameter, 0 timestamp, 0 timestamp replies, 0 info request, 0 other
0 irdp solicitations, 0 irdp advertisements
364 time exceeded, 0 info replies
Sent: 245 redirects, 271507 unreachable, 0 echo, 522 echo reply
0 mask requests, 0 mask replies, 0 quench, 0 timestamp, 0 timestamp replies
0 info reply, 63 time exceeded, 0 parameter problem
0 irdp solicitations, 0 irdp advertisements
TCP statistics:
Rcvd: 777897 total, 73 checksum errors, 31327 no port
Sent: 726588 total
UDP statistics:
Rcvd: 1633366 total, 13 checksum errors, 1633353 no port
Sent: 0 total, 0 forwarded broadcasts
BGP statistics:
Rcvd: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh, 0 unrecognized
Sent: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh
IP-EIGRP statistics:
Rcvd: 0 total
Sent: 0 total
PIMv2 statistics: Sent/Received
Total: 0/0, 0 checksum errors, 0 format errors
Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0, Hellos: 0/0
Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0
Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
Queue drops: 0
State-Refresh: 0/0
IGMP statistics: Sent/Received
Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0
DVMRP: 0/0, PIM: 0/0
Queue drops: 0
OSPF statistics:
Rcvd: 0 total, 0 checksum errors
0 hello, 0 database desc, 0 link state req
0 link state updates, 0 link state acks
Sent: 0 total
0 hello, 0 database desc, 0 link state req
0 link state updates, 0 link state acks
ARP statistics:
Rcvd: 701648 requests, 2 replies, 0 reverse, 0 other
Sent: 745584 requests, 62 replies (0 proxy), 0 reverse
Drop due to input queue full: 0
RTR-TED-0002#show ip nat statistics
Total active translations: 0 (0 static, 0 dynamic; 0 extended)
Peak translations: 97586, occurred 01:28:59 ago
Outside interfaces:
Dialer1
Inside interfaces:
FastEthernet0/1
Hits: 421422215 Misses: 0
CEF Translated packets: 418688607, CEF Punted packets: 1882968
Expired translations: 7728354
Dynamic mappings:
-- Inside Source
[Id: 1] access-list 1 interface Dialer1 refcount 4294967259
Appl doors: 0
Normal doors: 0
Queued Packets: 0
*May 21 06:32:12.858: %IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet0/1: the fragment table has reached its maximum threshold 16
*May 21 06:32:46.030: %IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet0/1: the fragment table has reached its maximum threshold 16
*May 21 06:33:16.854: %IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet0/1: the fragment table has reached its maximum threshold 16
*May 21 06:33:48.906: %IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet0/1: the fragment table has reached its maximum threshold 16
*May 21 06:34:20.334: %IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet0/1: the fragment table has reached its maximum threshold 16
*May 21 06:37:05.106: %IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet0/1: the fragment table has reached its maximum threshold 16
*May 21 06:57:52.338: %IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet0/1: the fragment table has reached its maximum threshold 16
*May 21 06:58:56.450: %IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet0/1: the fragment table has reached its maximum threshold 16
*May 22 05:22:21.497: %IP_VFR-4-FRAG_TABLE_OVERFLOW: Dialer1: the fragment table has reached its maximum threshold 16
*May 22 05:22:55.809: %IP_VFR-4-FRAG_TABLE_OVERFLOW: Dialer1: the fragment table has reached its maximum threshold 16
*May 22 05:30:06.637: %IP_VFR-4-FRAG_TABLE_OVERFLOW: Dialer1: the fragment table has reached its maximum threshold 16
*May 22 08:35:17.740: %IP_VFR-4-FRAG_TABLE_OVERFLOW: Dialer1: the fragment table has reached its maximum threshold 16
I highlighted some info which I thinks they are clue for my problem. It seem that there are too much traffic download from Internet, and the router cannot handle.
The logging was for 2 days ago, today it has no logs buffered. Three days ago I has upgrade to OS c2800nm-advipservicesk9-mz.124-22.T5 to improve %CPU but it does not work.
Feel free to not consider the fragmented counter. Because the counter indicate the time I customized the MTU and MSS. Now the MTU is default without configuring and the MSS is 1436 for interface Dialer 1.
This company has about over 600 users. I would like to know can Router 281 handle the NAT job with such number of users? If not, which router model is good to work in this case.
Any response would be appreciated.
Thanks
Solved! Go to Solution.
05-23-2013 03:06 AM
Hello,
I agree Paolo to update IOS but I remember you that 97000 sessions for 600 users are 160 sessions per user. A browser can open 4 TCP connections simultaneously. I have bad experiences with 80K sessions in a NPE-G1 c7200 router. Please, after upgrade if the problem still exists consider tune nat timers. A higher number only can be explained by a TCP NAT timer of 24 hours. If it is not enough study you internal network because your users can be very aggressive generating TCP connections. As last alternative, consider change the PPPoE connection to your checkpoint.
Regards.
05-23-2013 09:51 AM
I decided to choose IOS c2800nm-advipservicesk9-mz.124-25g.bin. I will consider tune the timer after a test time.
I will replug the router into network at tomorrow lunch time, then report the result one day after that.
Anyway, thanks you to all quick response.
05-24-2013 03:03 AM
That is not a good choice at all. Use latest IOS.
06-16-2013 09:13 PM
Sorry, because of not updating the case for long time.
There were some other problems between me and the customer then they delayed the process of this case.
Today I start it again. I will use the OS: c2800nm-advsecurityk9-mz.151-4.M6.bin
Thanks
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide