Solved: Help 2811 router doesn't have enough power to handle Internet NAT???

tranminhc · ‎05-22-2013

Dear forum,

I has just setup a 2811 router to do only Internet NAT feature, but it seem to be exhausted. Few days ago the it's CPU always above 50%, and this morning it's completely 99 100%. I feel upset when console to it, so lag.

Here is some command output:

RTR-TED-0002#

RTR-TED-0002 04:08:30 AM Thursday May 23 2013 UTC

11 11

999999999999999999999009999990099999999999999999999999999999

100 ************************************************************

90 ************************************************************

80 ************************************************************

70 ************************************************************

60 ************************************************************

50 ************************************************************

40 ************************************************************

30 ************************************************************

20 ************************************************************

10 ************************************************************

0....5....1....1....2....2....3....3....4....4....5....5....6

0 5 0 5 0 5 0 5 0 5 0

CPU% per second (last 60 seconds)

11111

000009999999998888788787866766676654476666766644656766777768

000006169977413199970721046186575588703263055648085430031064

100 #####* *##**

90 #####**######* ** *

80 #####**######*#*##***** * * *

70 #####*#######*#########*# ******** * * **** ** *****#

60 #####*####################*##**###* ******#** ***##*######

50 ###################################**######### *############

40 ############################################################

30 ############################################################

20 ############################################################

10 ############################################################

0....5....1....1....2....2....3....3....4....4....5....5....6

0 5 0 5 0 5 0 5 0 5 0

CPU% per minute (last 60 minutes)

* = maximum CPU% # = average CPU%

11 1

86993379999344900699999098845989575344677999999 3

4997354899934690019999809757892280429196872999939

100 ** **** *** ******* * * ****

90 ** **** *** ********* * * ******

80 * ** **** *** ********* *** ********

70 **** ***** *** ********* #** * *****##**

60 **#* *#*** ****######*** *#**** *****##**

50 #*#* *#*** *****#######****#***** * ****####*

40 ###* **##*# ******#########**#*#*** **#***####* *

30 ###***###*#***##*##########*##*##*****##*###### *

20 ####**#####**##############*##*##*****######### *

10 ############################################### *

0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..

0 5 0 5 0 5 0 5 0 5 0 5 0

CPU% per hour (last 72 hours)

* = maximum CPU% # = average CPU%

RTR-TED-0002#show int fa0/1

FastEthernet0/1 is up, line protocol is up

Hardware is MV96340 Ethernet, address is 0015.fa2f.d539 (bia 0015.fa2f.d539)

Internet address is 10.124.1.5/16

MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,

reliability 255/255, txload 57/255, rxload 4/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 100Mb/s, 100BaseTX/FX

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 17:40:35

Input queue: 0/75/195/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 1650000 bits/sec, 1658 packets/sec

5 minute output rate 22564000 bits/sec, 2205 packets/sec

48946016 packets input, 1173184643 bytes

Received 208721 broadcasts, 0 runts, 0 giants, 20 throttles

44779 input errors, 0 CRC, 0 frame, 0 overrun, 44779 ignored

0 watchdog

0 input packets with dribble condition detected

72907133 packets output, 3637260615 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 unknown protocol drops

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier

0 output buffer failures, 0 output buffers swapped out

RTR-TED-0002#show int fa0/0

FastEthernet0/0 is up, line protocol is up

Hardware is MV96340 Ethernet, address is 0015.fa2f.d538 (bia 0015.fa2f.d538)

MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,

reliability 255/255, txload 4/255, rxload 58/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 100Mb/s, 100BaseTX/FX

ARP type: ARPA, ARP Timeout 04:00:00

Last input 1d22h, output 00:00:00, output hang never

Last clearing of "show interface" counters 17:40:20

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 22753000 bits/sec, 2218 packets/sec

5 minute output rate 1700000 bits/sec, 1647 packets/sec

73338917 packets input, 4276920895 bytes

Received 0 broadcasts, 0 runts, 0 giants, 0 throttles

122824 input errors, 0 CRC, 0 frame, 0 overrun, 122824 ignored

0 watchdog

0 input packets with dribble condition detected

48756504 packets output, 1340293905 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 unknown protocol drops

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier

0 output buffer failures, 0 output buffers swapped out

RTR-TED-0002#show int dialer 1

Dialer1 is up, line protocol is up (spoofing)

Hardware is Unknown

Internet address is 113.162.120.22/32

MTU 1500 bytes, BW 100000 Kbit/sec, DLY 20000 usec,

reliability 255/255, txload 6/255, rxload 144/255

Encapsulation PPP, loopback not set

Keepalive set (10 sec)

DTR is pulsed for 1 seconds on reset

Interface is bound to Vi2

Last input never, output never, output hang never

Last clearing of "show interface" counters 2d01h

Input queue: 76/75/71454/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: weighted fair

Output queue: 0/1000/64/0 (size/max total/threshold/drops)

Conversations 0/0/256 (active/max active/max total)

Reserved Conversations 0/0 (allocated/max allocated)

Available Bandwidth 75000 kilobits/sec

5 minute input rate 56848000 bits/sec, 5388 packets/sec

5 minute output rate 2363000 bits/sec, 3583 packets/sec

242770462 packets input, 491800724 bytes

160293318 packets output, 1190208362 bytes

Bound to:

Virtual-Access2 is up, line protocol is up

Hardware is Virtual Access interface

MTU 1500 bytes, BW 100000 Kbit/sec, DLY 20000 usec,

reliability 255/255, txload 6/255, rxload 145/255

Encapsulation PPP, LCP Open

Listen: CDPCP

Open: IPCP

PPPoE vaccess, cloned from Dialer1

Vaccess status 0x44, loopback not set

Keepalive set (10 sec)

DTR is pulsed for 5 seconds on reset

Interface is bound to Di1 (Encapsulation PPP)

Last input 00:00:00, output never, output hang never

Last clearing of "show interface" counters 1d23h

Input queue: 0/75/120991/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 56880000 bits/sec, 5391 packets/sec

5 minute output rate 2425000 bits/sec, 3723 packets/sec

242677192 packets input, 345776708 bytes, 7687 no buffer

Received 0 broadcasts, 0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

160316349 packets output, 1192328063 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 unknown protocol drops

0 output buffer failures, 0 output buffers swapped out

0 carrier transitions

RTR-TED-0002#show ip traffic

IP statistics:

Rcvd: 4273236 total, 2463884 local destination

0 format errors, 0 checksum errors, 63 bad hop count

2 unknown protocol, 0 not a gateway

0 security failures, 0 bad options, 0 with options

Opts: 0 end, 0 nop, 0 basic security, 0 loose source route

0 timestamp, 0 extended security, 0 record route

0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump

0 other

Frags: 7 reassembled, 0 timeouts, 0 couldn't reassemble

35186 fragmented, 70375 fragments, 21334 couldn't fragment

Bcast: 143 received, 0 sent

Mcast: 0 received, 0 sent

Sent: 998921 generated, 419708376 forwarded

Drop: 961 encapsulation failed, 0 unresolved, 0 no adjacency

1155 no route, 0 unicast RPF, 0 forced drop

0 options denied

Drop: 0 packets with source IP address zero

Drop: 0 packets with internal loop back IP address

0 physical broadcast

ICMP statistics:

Rcvd: 87 format errors, 1 checksum errors, 0 redirects, 51566 unreachable

522 echo, 0 echo reply, 0 mask requests, 0 mask replies, 6 quench

0 parameter, 0 timestamp, 0 timestamp replies, 0 info request, 0 other

0 irdp solicitations, 0 irdp advertisements

364 time exceeded, 0 info replies

Sent: 245 redirects, 271507 unreachable, 0 echo, 522 echo reply

0 mask requests, 0 mask replies, 0 quench, 0 timestamp, 0 timestamp replies

0 info reply, 63 time exceeded, 0 parameter problem

0 irdp solicitations, 0 irdp advertisements

TCP statistics:

Rcvd: 777897 total, 73 checksum errors, 31327 no port

Sent: 726588 total

UDP statistics:

Rcvd: 1633366 total, 13 checksum errors, 1633353 no port

Sent: 0 total, 0 forwarded broadcasts

BGP statistics:

Rcvd: 0 total, 0 opens, 0 notifications, 0 updates

0 keepalives, 0 route-refresh, 0 unrecognized

Sent: 0 total, 0 opens, 0 notifications, 0 updates

0 keepalives, 0 route-refresh

IP-EIGRP statistics:

Rcvd: 0 total

Sent: 0 total

PIMv2 statistics: Sent/Received

Total: 0/0, 0 checksum errors, 0 format errors

Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0, Hellos: 0/0

Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0

Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0

Queue drops: 0

State-Refresh: 0/0

IGMP statistics: Sent/Received

Total: 0/0, Format errors: 0/0, Checksum errors: 0/0

Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0

DVMRP: 0/0, PIM: 0/0

Queue drops: 0

OSPF statistics:

Rcvd: 0 total, 0 checksum errors

0 hello, 0 database desc, 0 link state req

0 link state updates, 0 link state acks

Sent: 0 total

0 hello, 0 database desc, 0 link state req

0 link state updates, 0 link state acks

ARP statistics:

Rcvd: 701648 requests, 2 replies, 0 reverse, 0 other

Sent: 745584 requests, 62 replies (0 proxy), 0 reverse

Drop due to input queue full: 0

RTR-TED-0002#show ip nat statistics

Total active translations: 0 (0 static, 0 dynamic; 0 extended)

Peak translations: 97586, occurred 01:28:59 ago

Outside interfaces:

Dialer1

Inside interfaces:

FastEthernet0/1

Hits: 421422215 Misses: 0

CEF Translated packets: 418688607, CEF Punted packets: 1882968

Expired translations: 7728354

Dynamic mappings:

-- Inside Source

[Id: 1] access-list 1 interface Dialer1 refcount 4294967259

Appl doors: 0

Normal doors: 0

Queued Packets: 0

*May 21 06:32:12.858: %IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet0/1: the fragment table has reached its maximum threshold 16

*May 21 06:32:46.030: %IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet0/1: the fragment table has reached its maximum threshold 16

*May 21 06:33:16.854: %IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet0/1: the fragment table has reached its maximum threshold 16

*May 21 06:33:48.906: %IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet0/1: the fragment table has reached its maximum threshold 16

*May 21 06:34:20.334: %IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet0/1: the fragment table has reached its maximum threshold 16

*May 21 06:37:05.106: %IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet0/1: the fragment table has reached its maximum threshold 16

*May 21 06:57:52.338: %IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet0/1: the fragment table has reached its maximum threshold 16

*May 21 06:58:56.450: %IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet0/1: the fragment table has reached its maximum threshold 16

*May 22 05:22:21.497: %IP_VFR-4-FRAG_TABLE_OVERFLOW: Dialer1: the fragment table has reached its maximum threshold 16

*May 22 05:22:55.809: %IP_VFR-4-FRAG_TABLE_OVERFLOW: Dialer1: the fragment table has reached its maximum threshold 16

*May 22 05:30:06.637: %IP_VFR-4-FRAG_TABLE_OVERFLOW: Dialer1: the fragment table has reached its maximum threshold 16

*May 22 08:35:17.740: %IP_VFR-4-FRAG_TABLE_OVERFLOW: Dialer1: the fragment table has reached its maximum threshold 16

I highlighted some info which I thinks they are clue for my problem. It seem that there are too much traffic download from Internet, and the router cannot handle.

The logging was for 2 days ago, today it has no logs buffered. Three days ago I has upgrade to OS c2800nm-advipservicesk9-mz.124-22.T5 to improve %CPU but it does not work.

Feel free to not consider the fragmented counter. Because the counter indicate the time I customized the MTU and MSS. Now the MTU is default without configuring and the MSS is 1436 for interface Dialer 1.

This company has about over 600 users. I would like to know can Router 281 handle the NAT job with such number of users? If not, which router model is good to work in this case.

Any response would be appreciated.

Thanks

Joseph W. Doherty · ‎06-17-2013

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Ok I will set MTU 1492 in dialer interface, then remove the line "ip tcp adjust-mss 1436"

Hehe cannot make check point do a PPPoE. We are going to make an HA deployment with Check Point, tomorrow. That why I must be make sure the Router is work well.

If your MTU is 1492 (for PPPoE overhead), I (and Cisco - http://www.cisco.com/en/US/tech/tk175/tk15/technologies_tech_note09186a0080093bc7.shtml#pppoemtu) recommend to use ip tcp adjust-mss 1452.

View solution in original post

antonio.guirado · ‎05-22-2013

Hello thanminhc,

It is strange that you have 97.586 peak translations.

Total active translations: 0 (0 static, 0 dynamic; 0 extended)

Peak translations: 97586, occurred 01:28:59 ago

I advice:

It not usual so many translations, so please look for some "strange" application that can produce it (p2p, virus and so on).
Use ACLs in the inside ethernet to filter protocols according your policy (only HTTP, IMAP, POP3 and so on). This is a way to reduce the translations number. Block ICMP request for example. Your CPU is limited so you must optimize it.
Tune ip nat timers for TCP translations (24 hours by default) and limit the maximum NAT entries with the following commands

ip nat translation tcp-timeout 3600

ip nat translation max-entries 30000

Notice that the values are an example. Change it until optimize your topology. Notice that if you use tcp-timeout very

low CPU can also be affected because it has to expired NAT entries and create new ones very frecuently. So you

have to tune it carefully. If you limit the max-entries new connections will be drop but at least established will work.

There are others timers but I think the most importat is tcp-timeout:

ip nat translation tcp-timeout 500

ip nat translation udp-timeout 30

ip nat translation dns-timeout 30

ip nat translation icmp-timeout 30

ip nat translation finrst-timeout 30

ip nat translation syn-timeout 30

Regards

Subeh Sharma · ‎05-22-2013

Hi,

What is the process causing the cpu. Can you paste 'show proc cpu sorted' at the time of high cpu? You can remove the fragmentation messages by configuring 'no ip virtual assembly' under dialer.

Regards,

Subeh

tranminhc · ‎05-23-2013

Thanks for replies,

Actually the command show ip nat translation is when I unpluged the router from network and replaced by another modem. I just want to show you the peak translations. The old modem work very well, I think because I can surf cisco forum without scaring the page down.

This router stand in front of a check point firewall, and I think we don't have to care about the policy.

At the time of high CPU utilization, I realized that some websites are hard to go, the pages was loading in long time, some pages cannot connect such as cisco's community. I show commands above serveral times and see the counter input drop change quickly.

You can notice that I has clear counter yesterday.

Anyway I thinks the problem is on the router. Maybe some bug in OS

tranminhc · ‎05-23-2013

Sorry, I not save the output of show process cpu sorted. But I remember that 2 service is IP input and NAT aggre are on the top with about 3 and 4% for each

antonio.guirado · ‎05-23-2013

Hello,

Are you using PPPoE in the router?. Can you post the dialer interface configuration?

Regards.

tranminhc · ‎05-23-2013

Yes of course,

!

interface Dialer1

bandwidth 100000

ip address negotiated

ip nat outside

ip virtual-reassembly

encapsulation ppp

ip tcp adjust-mss 1436

dialer pool 10

ppp authentication pap chap callin

ppp chap hostname lhdfttx

ppp chap password 0 abc123

ppp pap sent-username lhdfttx password 0 abc123

ppp ipcp dns accept

ppp ipcp route default

ppp ipcp address accept

!

interface FastEthernet0/0

no ip address

duplex auto

speed auto

pppoe enable group global

pppoe-client dial-pool-number 10

!

interface FastEthernet0/1

ip address 10.124.1.5 255.255.0.0

ip nat inside

ip virtual-reassembly

duplex auto

speed auto

!

antonio.guirado · ‎05-23-2013

Hello,

I recomend you:

interface dialer1

ip mtu 1492

!

to set the MTU to 1492 due to PPPoE header (8bytes).

You have "ip tcp adjust-mss 1436" and it is similar (for TCP connections) but it is usual to set MTU in PPPoE connections.

If you have a checkpoint, consider use the router as a brigde (rfc1483-bridging) and established the PPPoE session from

your checkpoint. So, NAT feature could be made in checkpoint.

Regards.

tranminhc · ‎05-23-2013

Hi,

Ok I will set MTU 1492 in dialer interface, then remove the line "ip tcp adjust-mss 1436"

Hehe cannot make check point do a PPPoE. We are going to make an HA deployment with Check Point, tomorrow. That why I must be make sure the Router is work well.

@Paolo: how to disable fragmentation or just remove the ip tcp adjust-mss?

Could you recommend another IOS? The router has only 256 MB RAM and 64 MB Flash. Last time, I upgrade OS, I have chosen an affordable and quite new OS (c2800nm-advipservicesk9-mz.124-22.T5)

Thanks all of you.

paolo bevilacqua · ‎05-23-2013

What you have is old and buggy. Use latest IOS.

no ip virtual-reassembly.

And it will work fine.

Joseph W. Doherty · ‎06-17-2013

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Ok I will set MTU 1492 in dialer interface, then remove the line "ip tcp adjust-mss 1436"

Hehe cannot make check point do a PPPoE. We are going to make an HA deployment with Check Point, tomorrow. That why I must be make sure the Router is work well.

If your MTU is 1492 (for PPPoE overhead), I (and Cisco - http://www.cisco.com/en/US/tech/tk175/tk15/technologies_tech_note09186a0080093bc7.shtml#pppoemtu) recommend to use ip tcp adjust-mss 1452.

tranminhc · ‎06-17-2013

Thanks to your advice,

The tool is very interesting.

You are very correct. This afternoon, I replaced the old modem with the router. First 10 mins after replacing, I realized that the Router was trying to work with all of its effort (CPU over 80%). Some websites loaded quite quickly, but never stop loading, some even cannot load, and yahoo IM cannot login. Debug ICMP, then I saw some need-defrag packet send but not reachable to its destination, I not sure whether ISP block them. I put back "ip tcp adjust-mss 1436" command in interface Dialer. It seem to work well until now. CPU average is about 40 --> 50%. Some counter info in show commands sound much better.

My final config is

interface FastEthernet0/0

no ip address

duplex auto

speed auto

pppoe enable group global

pppoe-client dial-pool-number 10

!

interface FastEthernet0/1

ip address 10.124.1.5 255.255.0.0

ip nat inside

ip virtual-reassembly in

duplex auto

speed auto

!

interface Dialer1

bandwidth 100000

ip address negotiated

ip mtu 1492

ip nat outside

ip tcp adjust-mss 1436

no ip virtual-reassembly in

encapsulation ppp

dialer pool 10

ppp authentication pap chap callin

ppp chap hostname lhdfttx

ppp chap password 0 abc123

ppp pap sent-username lhdfttx password 0 abc123

ppp ipcp dns accept

ppp ipcp route default

ppp ipcp address accept

!

ip nat translation tcp-timeout 500

I need one day more to tracking on the router process. I will consider increase the mss to 1452.

One more thing I 'm not sure is that the MTU of Checkpoint interface (behind the Router) set to 1500 (default setting). Do I need to adjust CP' MTU to 1492???

antonio.guirado · ‎06-17-2013

Hello Tranminhc,

I am glad to hear that your router is able to manage NAT entries. Although, I do not agree that the solution is the

command ip tcp adjust-mss 1436. It is true that help to solve your issue but you posted the original configuration and you already used it. Now, you have changed the IOS version and used the command "ip nat translation tcp-timeout 500". Do you think that the new good performance in only due to

" ip tcp adjust-mss 1436" commad?.

Regards.

tranminhc · ‎06-17-2013

Hi Antonio.guirado

No, I don't. I think my problem is from uncustomized configurations, and iOS bug. Therefore,from the start of posting problem, I have to upgrade iOS 2 times, no one command (no ip virtual-reassembly), put two commands (ip mtu 1492,

ip nat translation tcp-timeout 500). I did configured those commands at the first of second time of replacing the router, then I did got in some troubles I described above. The web can go smoothly, Yahoo IM can login after putting the command tcp adjust-mss. I do understand clearly the meaning of this command. However in this case, It not fully helpful at without all of your helps.

Now, Show commands like show ip traffic, show interface, show ip nat statistic, show process cpu history bring out very good counters, no drop, no ignored, no high NAT active session, no high fragment.

Anyway thanks you so much.

I learn much from you.

paolo bevilacqua · ‎05-23-2013

Update IOS, disable virtual-reassembly, and check again.