cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1943
Views
10
Helpful
18
Replies

Help 2811 router doesn't have enough power to handle Internet NAT???

tranminhc
Level 1
Level 1

Dear forum,

I has just setup a 2811 router to do only Internet NAT feature, but it seem to be exhausted. Few days ago the it's CPU always above 50%, and this morning it's completely 99 100%. I feel upset when console to it, so lag.

Here is some command output:

RTR-TED-0002#

RTR-TED-0002   04:08:30 AM Thursday May 23 2013 UTC

                         11      11                            

    999999999999999999999009999990099999999999999999999999999999

    999999999999999999999009999990099999999999999999999999999999

100 ************************************************************

90 ************************************************************

80 ************************************************************

70 ************************************************************

60 ************************************************************

50 ************************************************************

40 ************************************************************

30 ************************************************************

20 ************************************************************

10 ************************************************************

   0....5....1....1....2....2....3....3....4....4....5....5....6

             0    5    0    5    0    5    0    5    0    5    0

               CPU% per second (last 60 seconds)

    11111                                                      

    000009999999998888788787866766676654476666766644656766777768

    000006169977413199970721046186575588703263055648085430031064

100 #####* *##**                                               

90 #####**######*  ** *                                       

80 #####**######*#*##***** *      *                           *

70 #####*#######*#########*# ********   *  * ****    **  *****#

60 #####*####################*##**###*  ******#**  ***##*######

50 ###################################**######### *############

40 ############################################################

30 ############################################################

20 ############################################################

10 ############################################################

   0....5....1....1....2....2....3....3....4....4....5....5....6

             0    5    0    5    0    5    0    5    0    5    0

               CPU% per minute (last 60 minutes)

              * = maximum CPU%   # = average CPU%

                   11      1                                               

    86993379999344900699999098845989575344677999999 3                      

    4997354899934690019999809757892280429196872999939                      

100   **   ****   *** *******    *           * ****                        

90   **   ****   *** *********  * *         ******                        

80 * **   ****   *** *********  ***       ********                        

70 ****  *****   *** *********  #** *    *****##**                        

60 **#*  *#***   ****######*** *#****    *****##**                        

50 #*#*  *#***  *****#######****#***** * ****####*                        

40 ###* **##*# ******#########**#*#*** **#***####* *                      

30 ###***###*#***##*##########*##*##*****##*###### *                      

20 ####**#####**##############*##*##*****######### *                      

10 ############################################### *                      

   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..

             0    5    0    5    0    5    0    5    0    5    0    5    0 

                   CPU% per hour (last 72 hours)

                  * = maximum CPU%   # = average CPU%

RTR-TED-0002#show int fa0/1

FastEthernet0/1 is up, line protocol is up

  Hardware is MV96340 Ethernet, address is 0015.fa2f.d539 (bia 0015.fa2f.d539)

  Internet address is 10.124.1.5/16

  MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,

     reliability 255/255, txload 57/255, rxload 4/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 100Mb/s, 100BaseTX/FX

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:00:00, output 00:00:00, output hang never

  Last clearing of "show interface" counters 17:40:35

  Input queue: 0/75/195/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 1650000 bits/sec, 1658 packets/sec

  5 minute output rate 22564000 bits/sec, 2205 packets/sec

     48946016 packets input, 1173184643 bytes

     Received 208721 broadcasts, 0 runts, 0 giants, 20 throttles

     44779 input errors, 0 CRC, 0 frame, 0 overrun, 44779 ignored

     0 watchdog

     0 input packets with dribble condition detected

     72907133 packets output, 3637260615 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 unknown protocol drops

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier

     0 output buffer failures, 0 output buffers swapped out

RTR-TED-0002#show int fa0/0

FastEthernet0/0 is up, line protocol is up

  Hardware is MV96340 Ethernet, address is 0015.fa2f.d538 (bia 0015.fa2f.d538)

  MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,

     reliability 255/255, txload 4/255, rxload 58/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 100Mb/s, 100BaseTX/FX

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 1d22h, output 00:00:00, output hang never

  Last clearing of "show interface" counters 17:40:20

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 22753000 bits/sec, 2218 packets/sec

  5 minute output rate 1700000 bits/sec, 1647 packets/sec

     73338917 packets input, 4276920895 bytes

     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles

     122824 input errors, 0 CRC, 0 frame, 0 overrun, 122824 ignored

     0 watchdog

     0 input packets with dribble condition detected

     48756504 packets output, 1340293905 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 unknown protocol drops

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier

     0 output buffer failures, 0 output buffers swapped out

RTR-TED-0002#show int dialer 1

Dialer1 is up, line protocol is up (spoofing)

  Hardware is Unknown

  Internet address is 113.162.120.22/32

  MTU 1500 bytes, BW 100000 Kbit/sec, DLY 20000 usec,

     reliability 255/255, txload 6/255, rxload 144/255

  Encapsulation PPP, loopback not set

  Keepalive set (10 sec)

  DTR is pulsed for 1 seconds on reset

  Interface is bound to Vi2

  Last input never, output never, output hang never

  Last clearing of "show interface" counters 2d01h

  Input queue: 76/75/71454/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: weighted fair

  Output queue: 0/1000/64/0 (size/max total/threshold/drops)

     Conversations  0/0/256 (active/max active/max total)

     Reserved Conversations 0/0 (allocated/max allocated)

     Available Bandwidth 75000 kilobits/sec

  5 minute input rate 56848000 bits/sec, 5388 packets/sec

  5 minute output rate 2363000 bits/sec, 3583 packets/sec

     242770462 packets input, 491800724 bytes

     160293318 packets output, 1190208362 bytes

Bound to:

Virtual-Access2 is up, line protocol is up

  Hardware is Virtual Access interface

  MTU 1500 bytes, BW 100000 Kbit/sec, DLY 20000 usec,

     reliability 255/255, txload 6/255, rxload 145/255

  Encapsulation PPP, LCP Open

  Listen: CDPCP

  Open: IPCP

  PPPoE vaccess, cloned from Dialer1

  Vaccess status 0x44, loopback not set

  Keepalive set (10 sec)

  DTR is pulsed for 5 seconds on reset

  Interface is bound to Di1 (Encapsulation PPP)

  Last input 00:00:00, output never, output hang never

  Last clearing of "show interface" counters 1d23h

  Input queue: 0/75/120991/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 56880000 bits/sec, 5391 packets/sec

  5 minute output rate 2425000 bits/sec, 3723 packets/sec

     242677192 packets input, 345776708 bytes, 7687 no buffer

     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

     160316349 packets output, 1192328063 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 unknown protocol drops

     0 output buffer failures, 0 output buffers swapped out

     0 carrier transitions

RTR-TED-0002#show ip traffic

IP statistics:

  Rcvd:  4273236 total, 2463884 local destination

         0 format errors, 0 checksum errors, 63 bad hop count

         2 unknown protocol, 0 not a gateway

         0 security failures, 0 bad options, 0 with options

  Opts:  0 end, 0 nop, 0 basic security, 0 loose source route

         0 timestamp, 0 extended security, 0 record route

         0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump

         0 other

  Frags: 7 reassembled, 0 timeouts, 0 couldn't reassemble

         35186 fragmented, 70375 fragments, 21334 couldn't fragment

  Bcast: 143 received, 0 sent

  Mcast: 0 received, 0 sent

  Sent:  998921 generated, 419708376 forwarded

  Drop:  961 encapsulation failed, 0 unresolved, 0 no adjacency

         1155 no route, 0 unicast RPF, 0 forced drop

         0 options denied

  Drop:  0 packets with source IP address zero

  Drop:  0 packets with internal loop back IP address

         0 physical broadcast

ICMP statistics:

  Rcvd: 87 format errors, 1 checksum errors, 0 redirects, 51566 unreachable

        522 echo, 0 echo reply, 0 mask requests, 0 mask replies, 6 quench

        0 parameter, 0 timestamp, 0 timestamp replies, 0 info request, 0 other

        0 irdp solicitations, 0 irdp advertisements

        364 time exceeded, 0 info replies

  Sent: 245 redirects, 271507 unreachable, 0 echo, 522 echo reply

        0 mask requests, 0 mask replies, 0 quench, 0 timestamp, 0 timestamp replies

        0 info reply, 63 time exceeded, 0 parameter problem

        0 irdp solicitations, 0 irdp advertisements

TCP statistics:

  Rcvd: 777897 total, 73 checksum errors, 31327 no port

  Sent: 726588 total

UDP statistics:

  Rcvd: 1633366 total, 13 checksum errors, 1633353 no port

  Sent: 0 total, 0 forwarded broadcasts

BGP statistics:

  Rcvd: 0 total, 0 opens, 0 notifications, 0 updates

        0 keepalives, 0 route-refresh, 0 unrecognized

  Sent: 0 total, 0 opens, 0 notifications, 0 updates

        0 keepalives, 0 route-refresh

IP-EIGRP statistics:

  Rcvd: 0 total

  Sent: 0 total

PIMv2 statistics: Sent/Received

  Total: 0/0, 0 checksum errors, 0 format errors

  Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0,  Hellos: 0/0

  Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0

  Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0

  Queue drops: 0

  State-Refresh: 0/0

IGMP statistics: Sent/Received

  Total: 0/0, Format errors: 0/0, Checksum errors: 0/0

  Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0

  DVMRP: 0/0, PIM: 0/0

  Queue drops: 0

OSPF statistics:

  Rcvd: 0 total, 0 checksum errors

        0 hello, 0 database desc, 0 link state req

        0 link state updates, 0 link state acks

  Sent: 0 total

        0 hello, 0 database desc, 0 link state req

        0 link state updates, 0 link state acks

ARP statistics:

  Rcvd: 701648 requests, 2 replies, 0 reverse, 0 other

  Sent: 745584 requests, 62 replies (0 proxy), 0 reverse

  Drop due to input queue full: 0

RTR-TED-0002#show ip nat statistics

Total active translations: 0 (0 static, 0 dynamic; 0 extended)

Peak translations: 97586, occurred 01:28:59 ago

Outside interfaces:

  Dialer1

Inside interfaces:

  FastEthernet0/1

Hits: 421422215  Misses: 0

CEF Translated packets: 418688607, CEF Punted packets: 1882968

Expired translations: 7728354

Dynamic mappings:

-- Inside Source

[Id: 1] access-list 1 interface Dialer1 refcount 4294967259

Appl doors: 0

Normal doors: 0

Queued Packets: 0

*May 21 06:32:12.858: %IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet0/1: the fragment table has reached its maximum threshold 16

*May 21 06:32:46.030: %IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet0/1: the fragment table has reached its maximum threshold 16

*May 21 06:33:16.854: %IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet0/1: the fragment table has reached its maximum threshold 16

*May 21 06:33:48.906: %IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet0/1: the fragment table has reached its maximum threshold 16

*May 21 06:34:20.334: %IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet0/1: the fragment table has reached its maximum threshold 16

*May 21 06:37:05.106: %IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet0/1: the fragment table has reached its maximum threshold 16

*May 21 06:57:52.338: %IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet0/1: the fragment table has reached its maximum threshold 16

*May 21 06:58:56.450: %IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet0/1: the fragment table has reached its maximum threshold 16

*May 22 05:22:21.497: %IP_VFR-4-FRAG_TABLE_OVERFLOW: Dialer1: the fragment table has reached its maximum threshold 16

*May 22 05:22:55.809: %IP_VFR-4-FRAG_TABLE_OVERFLOW: Dialer1: the fragment table has reached its maximum threshold 16

*May 22 05:30:06.637: %IP_VFR-4-FRAG_TABLE_OVERFLOW: Dialer1: the fragment table has reached its maximum threshold 16

*May 22 08:35:17.740: %IP_VFR-4-FRAG_TABLE_OVERFLOW: Dialer1: the fragment table has reached its maximum threshold 16

I highlighted some info which I thinks they are clue for my problem. It seem that there are too much traffic download from Internet, and the router cannot handle.

The logging was for 2 days ago, today it has no logs buffered. Three days ago I has upgrade to OS c2800nm-advipservicesk9-mz.124-22.T5 to improve %CPU but it does not work.

Feel free to not consider the fragmented counter. Because the counter indicate the time I customized the MTU and MSS. Now the MTU is default without configuring and the MSS is 1436 for interface Dialer 1.

This company has about over 600 users. I would like to know can Router 281 handle the NAT job with such number of users? If not, which router model is good to work in this case.

Any response would be appreciated.

Thanks

1 Accepted Solution

Accepted Solutions

Disclaimer

The   Author of this posting offers the information contained within this   posting without consideration and with the reader's understanding that   there's no implied or expressed suitability or fitness for any purpose.   Information provided is for informational purposes only and should not   be construed as rendering professional advice of any kind. Usage of  this  posting's information is solely at reader's own risk.

Liability Disclaimer

In   no event shall Author be liable for any damages whatsoever (including,   without limitation, damages for loss of use, data or profit) arising  out  of the use or inability to use the posting's information even if  Author  has been advised of the possibility of such damage.

Posting

Ok I will set MTU 1492 in dialer interface, then remove the line "ip tcp adjust-mss 1436"

Hehe cannot make check point do a PPPoE. We are going to make an HA deployment with Check Point, tomorrow. That why I must be make sure the Router is work well.

If your MTU is 1492 (for PPPoE overhead), I (and Cisco - http://www.cisco.com/en/US/tech/tk175/tk15/technologies_tech_note09186a0080093bc7.shtml#pppoemtu) recommend to use ip tcp adjust-mss 1452.

View solution in original post

18 Replies 18

antonio.guirado
Level 3
Level 3

Hello thanminhc,

It is strange that you have 97.586 peak translations.

Total active translations: 0 (0 static, 0 dynamic; 0 extended)

Peak translations: 97586, occurred 01:28:59 ago

I advice:

  • It not usual so many translations, so please look for some "strange" application that can produce it (p2p, virus and so on).
  • Use ACLs in the inside ethernet to filter protocols according your policy (only HTTP, IMAP, POP3 and so on). This is a way to reduce the translations number. Block ICMP request for example. Your CPU is limited so you must optimize it.
  • Tune ip nat timers for TCP translations (24 hours by default) and limit the maximum NAT entries with the following commands

          ip nat translation tcp-timeout 3600

          ip nat translation max-entries 30000

     Notice that the values are an example. Change it until optimize your topology. Notice that if you use tcp-timeout very

     low CPU can also be affected because it has to expired NAT entries and create new ones very frecuently. So you

     have to tune it carefully. If you limit the max-entries new connections will be drop but at least established will work.

There are others timers but I think the most importat is tcp-timeout:

ip nat translation tcp-timeout 500

ip nat translation udp-timeout 30

ip nat translation dns-timeout 30

ip nat translation icmp-timeout 30

ip nat translation finrst-timeout 30

ip nat translation syn-timeout 30

Regards

Hi,

What is the process causing the cpu. Can you paste 'show proc cpu sorted' at the time of high cpu? You can remove the fragmentation messages by configuring 'no ip virtual assembly' under dialer.

Regards,

Subeh

Thanks for replies,

Actually the command show ip nat translation is when I unpluged the router from network and replaced by another modem. I just want to show you the peak translations. The old modem work very well, I think because I can surf cisco forum without scaring the page down.

This router stand in front of a check point firewall, and I think we don't have to care about the policy.

At the time of high CPU utilization, I realized that some websites are hard to go, the pages was loading in long time, some pages cannot connect such as cisco's community. I show commands above serveral times and see the counter input drop change quickly.

You can notice that I has clear counter yesterday.

Anyway I thinks the problem is on the router. Maybe some bug in OS

Sorry, I not save the output of show process cpu sorted. But I remember that 2 service is IP input and NAT aggre are on the top with about 3 and 4% for each

Hello,

Are you using PPPoE in the router?. Can you post the dialer interface configuration?

Regards.

Yes of course,

!

interface Dialer1

bandwidth 100000

ip address negotiated

ip nat outside

ip virtual-reassembly

encapsulation ppp

ip tcp adjust-mss 1436

dialer pool 10

ppp authentication pap chap callin

ppp chap hostname lhdfttx

ppp chap password 0 abc123

ppp pap sent-username lhdfttx password 0 abc123

ppp ipcp dns accept

ppp ipcp route default

ppp ipcp address accept

!

interface FastEthernet0/0

no ip address

duplex auto

speed auto

pppoe enable group global

pppoe-client dial-pool-number 10

!

interface FastEthernet0/1

ip address 10.124.1.5 255.255.0.0

ip nat inside

ip virtual-reassembly

duplex auto

speed auto

!

Hello,

I recomend you:

interface dialer1

ip mtu 1492

!

to set the MTU to 1492 due to PPPoE header (8bytes).

You have "ip tcp adjust-mss 1436" and it is similar (for TCP connections) but it is usual to set MTU in PPPoE connections.

If you have a checkpoint, consider use the router as a brigde (rfc1483-bridging) and established the PPPoE session from

your checkpoint. So, NAT feature could be made in checkpoint.

Regards.

Hi,

Ok I will set MTU 1492 in dialer interface, then remove the line "ip tcp adjust-mss 1436"

Hehe cannot make check point do a PPPoE. We are going to make an HA deployment with Check Point, tomorrow. That why I must be make sure the Router is work well.

@Paolo: how to disable fragmentation or just remove the ip tcp adjust-mss?

Could you recommend another IOS? The router has only 256 MB RAM and 64 MB Flash. Last time, I upgrade OS, I have chosen an affordable and quite new OS (c2800nm-advipservicesk9-mz.124-22.T5)

Thanks all of you.

What you have is old and buggy. Use latest IOS.

no ip virtual-reassembly.

And it will work fine.

Disclaimer

The   Author of this posting offers the information contained within this   posting without consideration and with the reader's understanding that   there's no implied or expressed suitability or fitness for any purpose.   Information provided is for informational purposes only and should not   be construed as rendering professional advice of any kind. Usage of  this  posting's information is solely at reader's own risk.

Liability Disclaimer

In   no event shall Author be liable for any damages whatsoever (including,   without limitation, damages for loss of use, data or profit) arising  out  of the use or inability to use the posting's information even if  Author  has been advised of the possibility of such damage.

Posting

Ok I will set MTU 1492 in dialer interface, then remove the line "ip tcp adjust-mss 1436"

Hehe cannot make check point do a PPPoE. We are going to make an HA deployment with Check Point, tomorrow. That why I must be make sure the Router is work well.

If your MTU is 1492 (for PPPoE overhead), I (and Cisco - http://www.cisco.com/en/US/tech/tk175/tk15/technologies_tech_note09186a0080093bc7.shtml#pppoemtu) recommend to use ip tcp adjust-mss 1452.

Thanks to your advice,

The tool is very interesting.

You are very correct. This afternoon, I replaced the old modem with the router. First 10 mins after replacing, I realized that the Router was trying to work with all of its effort (CPU over 80%). Some websites loaded quite quickly, but never stop loading, some even cannot load, and yahoo IM cannot login. Debug ICMP, then I saw some need-defrag packet send but not reachable to its destination, I not sure whether ISP block them. I put back "ip tcp adjust-mss 1436" command in interface Dialer. It seem to work well until now. CPU average is about 40 --> 50%. Some counter info in show commands sound much better.

My final config is

interface FastEthernet0/0

no ip address

duplex auto

speed auto

pppoe enable group global

pppoe-client dial-pool-number 10

!

interface FastEthernet0/1

ip address 10.124.1.5 255.255.0.0

ip nat inside

ip virtual-reassembly in

duplex auto

speed auto

!

interface Dialer1

bandwidth 100000

ip address negotiated

ip mtu 1492

ip nat outside

ip tcp adjust-mss 1436

no ip virtual-reassembly in

encapsulation ppp

dialer pool 10

ppp authentication pap chap callin

ppp chap hostname lhdfttx

ppp chap password 0 abc123

ppp pap sent-username lhdfttx password 0 abc123

ppp ipcp dns accept

ppp ipcp route default

ppp ipcp address accept

!

ip nat translation tcp-timeout 500

I need one day more to tracking on the router process. I will consider increase the mss to 1452.

One more thing I 'm not sure is that the MTU of Checkpoint interface (behind the Router) set to 1500 (default setting). Do I need to adjust CP' MTU to 1492???

Hello Tranminhc,

I am glad to hear that your router is able to manage NAT entries. Although, I do not agree that the solution is the

command ip tcp adjust-mss 1436. It is true that help to solve your issue but you posted the original configuration and you already used it. Now, you have changed the IOS version and used the command "ip nat translation tcp-timeout 500". Do you think that the new good performance in only due to

" ip tcp adjust-mss 1436" commad?.

Regards.

Hi Antonio.guirado

No, I don't. I think my problem is from uncustomized configurations, and iOS bug. Therefore,from the start of posting problem, I have to upgrade iOS 2 times, no one command  (no ip virtual-reassembly), put two commands (ip mtu 1492,

ip nat translation tcp-timeout 500). I did configured those commands at the first of second time of replacing the router, then I did got in some troubles I described above. The web can go smoothly, Yahoo IM can login after putting the command tcp adjust-mss. I do understand clearly the meaning of this command. However in this case, It not fully helpful at without all of your helps.

Now, Show commands like show ip traffic, show interface, show ip nat statistic, show process cpu history bring out very good counters, no drop, no ignored, no high NAT active session, no high fragment.

Anyway thanks you so much.

I learn much from you.

paolo bevilacqua
Hall of Fame
Hall of Fame

Update IOS, disable virtual-reassembly, and check again.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Innovations in Cisco Full Stack Observability - A new webinar from Cisco