Re: 2651 XM runs slow when connected to cable modem, why?

Seba · ‎12-23-2010

Hi all,

I was wondering if anyone may have some answers to my slowness issue that I have with my 2651 XM.

So, here is the set up.

Vonage device connected to Comast Cable Modem

2651 XM is connected to the Vonage device via its Fa0/0

2960 is connected to the 2651 XM to Fa0/1 on the router

Previously, i had 2621 connected in place of the 2651 XM. Same set up, just different hardware. I waped over to the 2651 XM because i wanted to run 12.4 IOS.

I configured my new 2651 XM to replace the 2621 almost the same way as the 2621 was. What I did add to the 2651 XM was the IOS Firewall. I connected the new router, all is good, i'm happy. About 20 minutes passes, and my internet browsing became very slow. CPU, RAM were definatlley not over utilzed. I poked around by could not find any reason as to why. My 2621 was still up and running so i just swapped the connections over to it, any my internet was flying again. I swaped the cable back to the 2651 XM and i again, encountered turtle speeds when browsing the net.

I tought that my IOS firewall was slowing things down, i removed it but to no effect. I tried different feature set of 12.4 IOS. Initially, i notied normal speeds but again, just minutes later, my internet speed was crawling again. I have also tried to connet the 2651 XM directly to the Comast modem, same results, slow.

I have not done any debugs as of yet. I did do some packet capture and have not noticed anything unusuall. It is a wierd issue, my 2621 flyies but my 2651 XM crawls when it comes to internet browsing.

Any ideas as to why? Below is my config that i have at the moment on the 2651 XM and show ver as well.

Thanks in advance for your comments.

version 12.3

service timestamps debug datetime msec

service timestamps log datetime msec localtime show-timezone year

service password-encryption

service sequence-numbers

!

hostname Router

!

boot-start-marker

boot system flash c2600-advipservicesk9-mz.123-26.bin

boot-end-marker

!

logging buffered 9082 debugging

no logging console

enable secret 5 //---removed--//

!

clock timezone MST -7

clock summer-time MST recurring

no network-clock-participate slot 1

no network-clock-participate wic 0

aaa new-model

!

aaa authentication attempts login 4

aaa authentication login default local enable

aaa authentication login MY_LIST none

aaa session-id common

ip subnet-zero

ip cef

!

ip dhcp excluded-address 10.10.2.1 10.10.2.21

!

ip dhcp pool HOME

network 10.10.2.0 255.255.255.0

default-router 10.10.2.1

dns-server 4.2.2.2 4.3.3.3

!

no ip domain lookup

ip inspect name FIREWALL_IN tcp

ip inspect name FIREWALL_IN udp

ip inspect name FIREWALL_IN icmp

ip inspect name FIREWALL_IN http java-list 2

ip inspect name FIREWALL_IN smtp

ip inspect name FIREWALL_IN ftp

ip inspect name FIREWALL_IN tftp

ip audit po max-events 100

!

username xxxx password 7 //--removed--//

!

controller T1 0/0

framing sf

linecode ami

!

controller T1 0/1

framing sf

linecode ami

!

controller T1 0/2

framing sf

linecode ami

!

controller T1 0/3

framing sf

linecode ami

!

interface Loopback0

ip address 10.0.0.1 255.255.255.255

ip nat inside

!

interface FastEthernet0/0

ip address 192.168.15.2 255.255.255.0

ip nat outside

ip nbar protocol-discovery

speed 100

full-duplex

!

interface FastEthernet0/1

no ip address

speed 100

full-duplex

!

interface FastEthernet0/1.10

encapsulation dot1Q 10

ip address 10.10.1.1 255.255.255.0

ip nat inside

!

interface FastEthernet0/1.20

encapsulation dot1Q 20

ip address 10.10.2.1 255.255.255.0

ip access-group ALLOW_INTERNET_ACCESS in

ip nat inside

!

interface Ethernet1/0

no ip address

ip nat outside

shutdown

full-duplex

!

ip nat inside source list ALLOW_INTERNET_ACCESS interface FastEthernet0/0 overload

ip classless

ip route 0.0.0.0 0.0.0.0 192.168.15.1

!

ip http server

no ip http secure-server

!

ip access-list extended ALLOW_INTERNET_ACCESS

permit tcp any any established log

permit udp any any log

permit udp any any eq domain log

permit tcp any any eq domain

permit icmp any any log

permit ip any any

deny ip any any log

ip access-list extended OUTSIDE_IN

permit udp host 204.34.198.40 any log

permit udp host 204.123.2.5 any log

permit icmp any any echo-reply log

deny ip any any

!

access-list 7 permit 204.34.198.40

access-list 7 permit 204.123.2.5

access-list 77 permit 10.10.1.11

access-list 77 permit 10.10.1.20

access-list 101 permit ip any any

!

line con 0

logging synchronous

login authentication MY_LIST

line aux 0

logging synchronous

login authentication MY_LIST

line vty 0 4

access-class VTY_LINES in

exec-timeout 0 0

logging synchronous

transport input telnet

transport output none

line vty 5 15

exec-timeout 0 0

logging synchronous

transport input telnet

transport output none

!

ntp clock-period 17180415

ntp source Loopback0

ntp access-group peer 7

ntp access-group serve-only 77

ntp server 204.123.2.5

ntp server 204.34.198.40

!

end

!

Router>show ver
Cisco IOS Software, C2600 Software (C2600-ADVENTERPRISEK9-M), Version 12.4(15)T14, RELEASE SOFTW
(fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Tue 17-Aug-10 05:40 by prod_rel_team

ROM: System Bootstrap, Version 12.2(8r) [cmong 8r], RELEASE SOFTWARE (fc1)

Router uptime is 22 minutes
System returned to ROM by power-on
System restarted at 22:50:40 MST Thu Dec 23 2010
System image file is "flash:c2600-adventerprisek9-mz.124-15.T14.bin"

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html you require further assistance please contact us by sending email to
export@cisco.com. 2651XM (MPC860P) processor (revision 3.0) with 253952K/8192K bytes of memory.
Processor board ID JAE0817F3KJ
M860 processor: part number 5, mask 2
1 Ethernet interface
2 FastEthernet interfaces
4 Channelized T1/PRI ports
32K bytes of NVRAM.
49152K bytes of processor board System flash (Read/Write)

If

Cisco

Configuration register is 0x2102

Router>

Message was edited by: Michael Simon

Peter Paluch · ‎12-23-2010

Sebastian,

Your config does not entirely correspond to the show version output. Obviously, the configuration is taken from 12.3 IOS while the sh ver shows a 12.4T IOS being run.

Regarding the slowdown, it would be very interesting to see the show processes cpu sorted command output in the moment of significant internet access slowdown. The CPU may not be overutilized but there may perhaps be a hint on something worth pursuing further.

I suggest removing the IP Inspect totally to debug this issue, and also, I have noticed you are using the ip nbar protocol-discovery on your Fa0/0 interface. I suggest removing that command as well unless you know very precisely that something will break without it.

Your Ethernet interfaces are forcibly put to 100 Mbit, full duplex mode. If you manually configure the speed and duplex on your router then it is strongly recommended to configure the speed and duplex manually on the adjacent device as well, otherwise, duplex mismatches may ensue. As a general rule, unless the autonegotiation is proven to not work reliably, it is not recommended to hard code the speed and duplex.

Your NAT is not configured according to Cisco's recommendations. An ACL used in NAT configuration may not be unspecific to inside local and inside global addresses - in the form of "permit ip any any". The IOS does not support ACLs in the form of "permit ip any any" in the NAT configuration. Currently, you are using the ALLOW_INTERNET_ACCESS ACL shared between the interface Fa0/1.20 and the NAT configuration, and this ACL is not specific about the internal addresses, rather, it allows any address to be translated. I suggest creating a separate ACL simply selecting the IP range of your internal networks, and using that in the NAT instead, for example:

ip access-list standard NAT

permit host 10.0.0.1

permit 10.10.1.0 0.0.0.255

permit 10.10.2.0 0.0.0.255

no ip nat inside source list ALLOW_INTERNET_ACCESS interface FastEthernet0/0 overload

ip nat inside source list NAT interface FastEthernet0/0 overload

And also, consider removing the log options from your ALLOW_INTERNET_ACCESS ACL entries. They may be hogging your resources, and there is really no point in logging the majority of your permitted packets.

Best regards,

Peter

Seba · ‎12-24-2010

Hi Peter,

I really appreciate your resonse.

I took your suggestions and implemented them on my router. I also removed the currently not needed ACLs. I have to say that no differences were noticied.

Instead of removing parts of my config i just went ahead and left what is needed for my router to operate. All of my uplinks are correctly set to 100/full.

Regarding you statement about the IOS. That is totally by bad. I just copied and pasted form the wrong file. Pretty much the same config was in place when running 12.4. The only difference was that for the IOS Firewall i was alble to select more protocols for inspeciton.

Anyway, below is my current config and show cpu pro sorted. I captured the show cpu pro sorted while accessing a website. Does the # seem a bit high for just accessing a web? Thanks a lot of your help.

Router#show run
Building configuration...

Current configuration : 2851 bytes
!
! No configuration change since last restart
!
version 12.4
service timestamps debug datetime msec
service timestamps log datetime msec localtime show-timezone year
service password-encryption
service sequence-numbers
!
hostname Router
!
boot-start-marker
boot-end-marker
!
logging buffered 9082
no logging console
enable secret 5 $1$yZhB$t3TsT.zwqriZHUnm1Ol750
!
aaa new-model
!
!
aaa authentication attempts login 4
aaa authentication login default local enable
aaa authentication login MY_LIST none
!
!
aaa session-id common
clock timezone MST -7
clock summer-time MST recurring
no network-clock-participate slot 1
no network-clock-participate wic 0
ip cef
!
!
no ip dhcp use vrf connected
ip dhcp excluded-address 10.10.2.1 10.10.2.21
!
ip dhcp pool HOME
   network 10.10.2.0 255.255.255.0
   default-router 10.10.2.1
   dns-server 4.2.2.2 4.3.3.3
!
!
no ip domain lookup
!
multilink bundle-name authenticated
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
username xxxx password 7 04480A040A2A1B
archive
log config
hidekeys
!
!
!
!
controller T1 0/0
framing sf
linecode ami
!
controller T1 0/1
framing sf
linecode ami
!
controller T1 0/2
framing sf
linecode ami
!
controller T1 0/3
framing sf
linecode ami
!
!
!
!
!
interface Loopback0
ip address 10.0.0.1 255.255.255.255
ip nat inside
ip virtual-reassembly
!
interface FastEthernet0/0
ip address 192.168.15.2 255.255.255.0
ip nat outside
ip virtual-reassembly
speed 100
full-duplex
!
interface FastEthernet0/1
no ip address
speed 100
full-duplex
!
interface FastEthernet0/1.10
encapsulation dot1Q 10
ip address 10.10.1.1 255.255.255.0
ip nat inside
ip virtual-reassembly
!
interface FastEthernet0/1.20
encapsulation dot1Q 20
ip address 10.10.2.1 255.255.255.0
ip nat inside
ip virtual-reassembly
!
interface Ethernet1/0
no ip address
ip nat outside
ip virtual-reassembly
shutdown
full-duplex
!
ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 192.168.15.1
!
!
ip http server
no ip http secure-server
ip nat inside source list NAT interface FastEthernet0/0 overload
!
ip access-list standard NAT
permit 10.0.0.1
permit 10.10.1.0 0.0.0.255
permit 10.10.2.0 0.0.0.255
!
access-list 7 permit 204.34.198.40
access-list 7 permit 204.123.2.5
access-list 77 permit 10.10.1.11
access-list 77 permit 10.10.1.20
!
!
!
!
!
control-plane
!
!
!
!
!
!
!
!
!
!
line con 0
logging synchronous
login authentication MY_LIST
line aux 0
logging synchronous
login authentication MY_LIST
line vty 0 4
access-class VTY_LINES in
exec-timeout 0 0
logging synchronous
transport input telnet
transport output none
line vty 5 15
exec-timeout 0 0
logging synchronous
transport input telnet
transport output none
!
ntp clock-period 17180428
ntp source Loopback0
ntp access-group peer 7
ntp access-group serve-only 77
ntp server 204.123.2.5
ntp server 204.34.198.40
!
end

Router#

Router#show processes cpu sorted
CPU utilization for five seconds: 17%/2%; one minute: 5%; five minutes: 3%
PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
49        8884       287      30954 12.20% 3.78% 1.97% 66 Virtual Exec
71        1110      1082       1025 1.32% 0.37% 0.20%   0 IP Input
   2          12        89        134 0.08% 0.01% 0.00%   0 Load Meter
   1          40        82        487 0.00% 0.00% 0.00%   0 Chunk Manager
   4           0         1          0 0.00% 0.00% 0.00%   0 EDDRI_MAIN
   5        1318        72      18305 0.00% 0.29% 0.22%   0 Check heaps
   6           0         1          0 0.00% 0.00% 0.00%   0 Pool Manager
   7           4         2       2000 0.00% 0.00% 0.00%   0 Timers
   8           0         1          0 0.00% 0.00% 0.00%   0 Crash writer
   9           0        16          0 0.00% 0.00% 0.00%   0 Environmental mo
10          20        37        540 0.00% 0.00% 0.00%   0 ARP Input
11          12       504         23 0.00% 0.00% 0.00%   0 ARP Background
12           0         2          0 0.00% 0.00% 0.00%   0 ATM Idle Timer
13           0         2          0 0.00% 0.00% 0.00%   0 AAA high-capacit
14           0         1          0 0.00% 0.00% 0.00%   0 AAA_SERVER_DEADT
15           0         1          0 0.00% 0.00% 0.00%   0 Policy Manager
16           0         2          0 0.00% 0.00% 0.00%   0 DDR Timers
17          12         2       6000 0.00% 0.00% 0.00%   0 Entity MIB API
18          28        20       1400 0.00% 0.00% 0.00%   0 EEM ED Syslog
19          12       124         96 0.00% 0.00% 0.00%   0 HC Counter Timer
20           4         2       2000 0.00% 0.00% 0.00%   0 Serial Backgroun
21           0         1          0 0.00% 0.00% 0.00%   0 RO Notify Timers
22           0         1          0 0.00% 0.00% 0.00%   0 RMI RM Notify Wa
23           0         2          0 0.00% 0.00% 0.00%   0 SMART
24           0       438          0 0.00% 0.00% 0.00%   0 GraphIt
25           0         2          0 0.00% 0.00% 0.00%   0 Dialer event
26           0         1          0 0.00% 0.00% 0.00%   0 SERIAL A'detect
27           0         2          0 0.00% 0.00% 0.00%   0 XML Proxy Client
28           0         1          0 0.00% 0.00% 0.00%   0 Critical Bkgnd
29          24       196        122 0.00% 0.00% 0.00%   0 Net Background
30           4         2       2000 0.00% 0.00% 0.00%   0 IDB Work
31           4         8        500 0.00% 0.00% 0.00%   0 Logger
32           0       433          0 0.00% 0.00% 0.00%   0 TTY Background
33           4       478          8 0.00% 0.00% 0.00%   0 Per-Second Jobs
34           0         4          0 0.00% 0.00% 0.00%   0 DHCPD Timer
35           0         1          0 0.00% 0.00% 0.00%   0 AggMgr Process
36           0         1          0 0.00% 0.00% 0.00%   0 Transport Port A
37           4         1       4000 0.00% 0.00% 0.00%   0 dev_device_inser
38           0         1          0 0.00% 0.00% 0.00%   0 dev_device_remov
39           0        80          0 0.00% 0.00% 0.00%   0 mxt5100
40           0         1          0 0.00% 0.00% 0.00%   0 sal_dpc_process
41           0         1          0 0.00% 0.00% 0.00%   0 ARL Table Manage
42           0         2          0 0.00% 0.00% 0.00%   0 ESWILPPM
43           0         2          0 0.00% 0.00% 0.00%   0 Eswilp Storm Con
44           0         2          0 0.00% 0.00% 0.00%   0 SM Monitor
45           8         2       4000 0.00% 0.00% 0.00%   0 VNM DSPRM MAIN
46           0         1          0 0.00% 0.00% 0.00%   0 DSPFARM DSP READ
47           0         2          0 0.00% 0.00% 0.00%   0 FLEX DNLD MAIN
48           0         1          0 0.00% 0.00% 0.00%   0 HDV background
50           0         3          0 0.00% 0.00% 0.00%   0 Net Input
51           4        89         44 0.00% 0.00% 0.00%   0 Compute load avg
52         397         9      44111 0.00% 0.10% 0.05%   0 Per-minute Jobs
53           0         1          0 0.00% 0.00% 0.00%   0 AAL2CPS TIMER_CU
54           4         1       4000 0.00% 0.00% 0.00%   0 IGMP Snooping Pr
55           0         1          0 0.00% 0.00% 0.00%   0 IGMP Snooping Re
56           0         2          0 0.00% 0.00% 0.00%   0 Call Management
57           8      1750          4 0.00% 0.00% 0.00%   0 e1t1 Framer back
58           0         2          0 0.00% 0.00% 0.00%   0 DTP Protocol
59           0         2          0 0.00% 0.00% 0.00%   0 Dot1x Mgr Proces
60           0         1          0 0.00% 0.00% 0.00%   0 MAB Framework
61           0         1          0 0.00% 0.00% 0.00%   0 EAP Framework
62           4       437          9 0.00% 0.00% 0.00%   0 PI MATM Aging Pr
63           0        45          0 0.00% 0.00% 0.00%   0 EtherChnl
64           0        16          0 0.00% 0.00% 0.00%   0 AAA Server
65           0         1          0 0.00% 0.00% 0.00%   0 AAA ACCT Proc
66          20     13716          1 0.00% 0.00% 0.00%   0 ACCT Periodic Pr
67           4         2       2000 0.00% 0.00% 0.00%   0 AAA Dictionary R
68           4        63         63 0.00% 0.00% 0.00%   0 CDP Protocol
69           4         5        800 0.00% 0.00% 0.00%   0 IP ARP Adjacency
70           8     13716          0 0.00% 0.00% 0.00%   0 IP ARP Retry Age
72           0         1          0 0.00% 0.00% 0.00%   0 ICMP event handl
73           0         1          0 0.00% 0.00% 0.00%   0 IPv6 RIB Redistr
74           8         3       2666 0.00% 0.00% 0.00%   0 MOP Protocols
75           0         3          0 0.00% 0.00% 0.00%   0 PPP Hooks
77           0         1          0 0.00% 0.00% 0.00%   0 SSS Manager
78           0        65          0 0.00% 0.00% 0.00%   0 SSS Test Client
79           0         1          0 0.00% 0.00% 0.00%   0 SSS Feature Mana
80           0      1835          0 0.00% 0.00% 0.00%   0 SSS Feature Time
81           0         2          0 0.00% 0.00% 0.00%   0 Spanning Tree
82           0         1          0 0.00% 0.00% 0.00%   0 X.25 Encaps Mana
83           0         8          0 0.00% 0.00% 0.00%   0 SSM connection m
84           4         1       4000 0.00% 0.00% 0.00%   0 AC Switch
85          48         2      24000 0.00% 0.00% 0.00%   0 EAPoUDP Process
86           0         2          0 0.00% 0.00% 0.00%   0 IP Host Track Pr
87           0         2          0 0.00% 0.00% 0.00%   0 KRB5 AAA
88           0         2          0 0.00% 0.00% 0.00%   0 PPP IP Route
89           0         2          0 0.00% 0.00% 0.00%   0 PPP IPCP
90           4        25        160 0.00% 0.00% 0.00%   0 IP Background
91           8        11        727 0.00% 0.00% 0.00%   0 IP RIB Update
92           0         1          0 0.00% 0.00% 0.00%   0 IP Traceroute
93           0         1          0 0.00% 0.00% 0.00%   0 SNMP Timers
94           0         1          0 0.00% 0.00% 0.00%   0 Socket Timers
95           0         1          0 0.00% 0.00% 0.00%   0 Asy FS Helper
96          32       667         47 0.00% 0.00% 0.00%   0 CEF process
97           8       155         51 0.00% 0.00% 0.00%   0 TCP Timer
98           8         5       1600 0.00% 0.00% 0.00%   0 TCP Protocols
99           0         1          0 0.00% 0.00% 0.00%   0 COPS
100           0         2          0 0.00% 0.00% 0.00%   0 Dot1x Supplicant
101           0         2          0 0.00% 0.00% 0.00%   0 Dot1x Supplicant
102           0         2          0 0.00% 0.00% 0.00%   0 Dot1x Supplicant
103           0         2          0 0.00% 0.00% 0.00%   0 L2MM
104           4         1       4000 0.00% 0.00% 0.00%   0 MRD
105           0         1          0 0.00% 0.00% 0.00%   0 IGMPSN
106           0         1          0 0.00% 0.00% 0.00%   0 L2X Data Daemon
107           8         2       4000 0.00% 0.00% 0.00%   0 SCTP Main Proces
108           0         1          0 0.00% 0.00% 0.00%   0 IUA Main Process
109           0       437          0 0.00% 0.00% 0.00%   0 RUDPV1 Main Proc
110           4         1       4000 0.00% 0.00% 0.00%   0 bsm_timers
111           0       469          0 0.00% 0.00% 0.00%   0 bsm_xmt_proc
112           0         1          0 0.00% 0.00% 0.00%   0 CES Client SVC R
113           4         2       2000 0.00% 0.00% 0.00%   0 Dialer Forwarder
114           0         9          0 0.00% 0.00% 0.00%   0 Adj Manager
115           0         3          0 0.00% 0.00% 0.00%   0 Flow Exporter Ti
116           0         2          0 0.00% 0.00% 0.00%   0 ATM OAM Input
117           4         2       2000 0.00% 0.00% 0.00%   0 ATM OAM TIMER
118           0         4          0 0.00% 0.00% 0.00%   0 HTTP CORE
119           0         8          0 0.00% 0.00% 0.00%   0 IP Cache Ager
120           0         1          0 0.00% 0.00% 0.00%   0 RARP Input
121           0         1          0 0.00% 0.00% 0.00%   0 IPv6 Inspect Tim
122           0         1          0 0.00% 0.00% 0.00%   0 LAPB Process
123           0         1          0 0.00% 0.00% 0.00%   0 PAD InCall
124           0         2          0 0.00% 0.00% 0.00%   0 X.25 Background
125           0         2          0 0.00% 0.00% 0.00%   0 PPP Bind
126           4         2       2000 0.00% 0.00% 0.00%   0 PPP SSS
127           0         1          0 0.00% 0.00% 0.00%   0 MQC Flow Event B
128          24      4316          5 0.00% 0.01% 0.00%   0 RBSCP Background
129           0         1          0 0.00% 0.00% 0.00%   0 VPDN call manage
130           4       850          4 0.00% 0.00% 0.00%   0 Inspect process
131           0        11          0 0.00% 0.00% 0.00%   0 DHCPD Database
132           0         2          0 0.00% 0.00% 0.00%   0 Authentication P
133           0         1          0 0.00% 0.00% 0.00%   0 Auth-proxy AAA B
134           0         1          0 0.00% 0.00% 0.00%   0 IPS Process
135           4         2       2000 0.00% 0.00% 0.00%   0 IPS Auto Update
136           0         2          0 0.00% 0.00% 0.00%   0 SDEE Management
137           0         2          0 0.00% 0.00% 0.00%   0 URL filter proc
138           0         1          0 0.00% 0.00% 0.00%   0 Crypto HW Proc
139         104         2      52000 0.00% 0.00% 0.00%   0 CCVPM_HDSPRM
140           4       165         24 0.00% 0.00% 0.00%   0 FLEX DSPRM MAIN
141           0       163          0 0.00% 0.00% 0.00%   0 FLEX DSP KEEPALI
142           0        21          0 0.00% 0.00% 0.00%   0 CRM_CALL_UPDATE_
143           0         4          0 0.00% 0.00% 0.00%   0 HDA DSPRM MAIN
145           0         2          0 0.00% 0.00% 0.00%   0 AAA Cached Serve
146          76       197        385 0.00% 0.00% 0.00%   0 ENABLE AAA
147           0         1          0 0.00% 0.00% 0.00%   0 EM Background Pr
148           0         1          0 0.00% 0.00% 0.00%   0 Key chain liveke
149           0         2          0 0.00% 0.00% 0.00%   0 LINE AAA
150           8        13        615 0.00% 0.00% 0.00%   0 LOCAL AAA
151           0         2          0 0.00% 0.00% 0.00%   0 TPLUS
152           0         2          0 0.00% 0.00% 0.00%   0 VSP_MGR
153           4         3       1333 0.00% 0.00% 0.00%   0 Crypto WUI
154           0         2          0 0.00% 0.00% 0.00%   0 Crypto Support
155           0         1          0 0.00% 0.00% 0.00%   0 IPSECv6 PS Proc
156           0         1          0 0.00% 0.00% 0.00%   0 EPM MAIN PROCESS
157           0         1          0 0.00% 0.00% 0.00%   0 CCVPM_HTSP
158           0         2          0 0.00% 0.00% 0.00%   0 VPM_MWI_BACKGROU
159           0         1          0 0.00% 0.00% 0.00%   0 CCVPM_R2
160           0         1          0 0.00% 0.00% 0.00%   0 EPHONE MWI Refre
161           0         1          0 0.00% 0.00% 0.00%   0 FB/KS Log HouseK
162           0         2          0 0.00% 0.00% 0.00%   0 EPHONE MWI BG Pr
163           4         1       4000 0.00% 0.00% 0.00%   0 Skinny HW confer
164           0        16          0 0.00% 0.00% 0.00%   0 VOICE REG BG Pro
165           0         1          0 0.00% 0.00% 0.00%   0 Presence Process
166           0         1          0 0.00% 0.00% 0.00%   0 CCSWVOICE
168           0         1          0 0.00% 0.00% 0.00%   0 http client proc
170           0         1          0 0.00% 0.00% 0.00%   0 QOS_MODULE_MAIN
171           0         1          0 0.00% 0.00% 0.00%   0 RPMS_PROC_MAIN
172           0         1          0 0.00% 0.00% 0.00%   0 VoIP AAA
173         269        25      10760 0.00% 0.00% 0.00%   0 crypto engine pr
174           8         4       2000 0.00% 0.00% 0.00%   0 Crypto CA
175           0         1          0 0.00% 0.00% 0.00%   0 Crypto PKI-CRL
176           0         1          0 0.00% 0.00% 0.00%   0 Crypto SSL
177           0         1          0 0.00% 0.00% 0.00%   0 encrypt proc
178           4         1       4000 0.00% 0.00% 0.00%   0 Crypto INT
179           4         3       1333 0.00% 0.00% 0.00%   0 Crypto IKE Dispa
180           4         3       1333 0.00% 0.00% 0.00%   0 Crypto IKMP
181          41         1      41000 0.00% 0.00% 0.00%   0 Crypto IKEv2
182           0         1          0 0.00% 0.00% 0.00%   0 IPSEC manual key
183         441        26      16961 0.00% 0.00% 0.00%   0 IPSEC key engine
184           0         1          0 0.00% 0.00% 0.00%   0 CRYPTO QoS proce
185           4         4       1000 0.00% 0.00% 0.00%   0 Crypto ACL
186           0         1          0 0.00% 0.00% 0.00%   0 Crypto PAS Proc
187           0         1          0 0.00% 0.00% 0.00%   0 Key Proc
188           0         1          0 0.00% 0.00% 0.00%   0 GDOI GM Process
189           0         1          0 0.00% 0.00% 0.00%   0 UNICAST REKEY
190           0         1          0 0.00% 0.00% 0.00%   0 UNICAST REKEY AC
191           0         2          0 0.00% 0.00% 0.00%   0 Control-plane ho
192           0         1          0 0.00% 0.00% 0.00%   0 PM Callback
193           0         1          0 0.00% 0.00% 0.00%   0 DATA Transfer Pr
194           0         1          0 0.00% 0.00% 0.00%   0 DATA Collector
195           4         4       1000 0.00% 0.00% 0.00%   0 AAA SEND STOP EV
196           0         2          0 0.00% 0.00% 0.00%   0 EEM ED Resource
197           4         3       1333 0.00% 0.00% 0.00%   0 EEM ED Track
198           0         1          0 0.00% 0.00% 0.00%   0 Syslog Traps
199           4        15        266 0.00% 0.00% 0.00%   0 Crypto cTCP proc
200           0       441          0 0.00% 0.00% 0.00%   0 trunk conditioni
201           0         1          0 0.00% 0.00% 0.00%   0 trunk conditioni
202           4         4       1000 0.00% 0.00% 0.00%   0 VLAN Manager
204           8        33        242 0.00% 0.00% 0.00%   0 EEM Server
205           4         3       1333 0.00% 0.00% 0.00%   0 EEM ED CLI
206           0         3          0 0.00% 0.00% 0.00%   0 EEM ED Counter
207           4         3       1333 0.00% 0.00% 0.00%   0 EEM ED Interface
208           0         3          0 0.00% 0.00% 0.00%   0 EEM ED IOSWD
209           4         3       1333 0.00% 0.00% 0.00%   0 EEM ED None
210           8         3       2666 0.00% 0.00% 0.00%   0 EEM ED OIR
211           0         3          0 0.00% 0.00% 0.00%   0 EEM ED SNMP
212           4        29        137 0.00% 0.00% 0.00%   0 EEM ED Timer
213           0         2          0 0.00% 0.00% 0.00%   0 EEM Policy Direc
214           8         5       1600 0.00% 0.00% 0.00%   0 Syslog
215           0         1          0 0.00% 0.00% 0.00%   0 VPDN Test
217          12        28        428 0.00% 0.00% 0.00%   0 DHCPD Receive
218           8       850          9 0.00% 0.00% 0.00%   0 IP NAT Ager
219           0         1          0 0.00% 0.00% 0.00%   0 IP NAT WLAN
220           0         1          0 0.00% 0.00% 0.00%   0 IP VFR proc
221           4        17        235 0.00% 0.00% 0.00%   0 CEF Scanner
222           0         2          0 0.00% 0.00% 0.00%   0 crypto sw pk pro
223           4       477          8 0.00% 0.00% 0.00%   0 NTP
Router#

Reza Sharifi · ‎12-24-2010

Hi,

Although the CPU numbers are a little high, but it is not that high to make the router crawl.

I think, the issue is related to the IOS version.

Here is sh CPU (3 times) and sh ver from a 2651 router. This router is running RIP, VRF, IPv6 enabled, but the IOS version is not 12.4 and is not the Crypto version.

So, I am thinking a change to a none Crypto ver of IOS may be helpful

DC#sh processes cpu sorted
CPU utilization for five seconds: 3%/0%; one minute: 0%; five minutes: 0%

DC#sh processes cpu sorted
CPU utilization for five seconds: 0%/0%; one minute: 0%; five minutes: 0%

DC#sh ver | inc bin
System image file is "flash:c2600-g4js-mz.123-12.bin"

HTH

Reza

Peter Paluch · ‎12-24-2010

Hello,

I agree absolutely with Reza - your current configuration does not contain anything obviously wrong, and in essence, your configuration is a common one. There should not be any similar issues with such a straightforward configuration. Just like Reza said, the CPU levels are a bit elevated but certainly not so high that they could indicate a CPU overload problem.

Are there absolutely no logging messages hinting at a possible cause in your show logging output? Also, when you look at the show interfaces output, are there any input or output errors logged?

I was thinking about one more possibility - the IP Virtual Reassembly feature that is activated on your interfaces by default. This feature tries to keep enough information about possible fragments of an IP packet so that, for example, ACLs that match on L4 protocol and/or ports can match all fragments, not just the first one (if an oversized IP packet carrying, say, TCP segment, is fragmented, the TCP header is logically contained in the first fragment only, thus the remaining fragments cannot match ACL entries that look for TCP header and/or ports until the Virtual Reassembly is in place). I have seen this feature to make some troubles under certain circumstances, and you may want trying to turn it off using the command no ip virtual-reassembly on all your interfaces that currently have this feature configured.

Best regards,

Peter

francisco_1 · ‎12-24-2010

Possibly a bug. You could try using another code and see if that makes a difference!

In 12.4(15)T14 there is a known bug on IPv6 but suspect it does also apply to IPv4 TCP throughput is low when TCP inspection is configured because the router drops packets due to perceived sequence number errors.

see this bug "CSCtb10776"

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCtb10776

Francisco.

Seba · ‎12-25-2010

Hey guys,

So i have tried couple of things. I took Reza's suggestion and changed to "non Crypto" IOS and my router functioned just fine. I wasn't fully satisfied with this solution, because i wanted to run IOS Firewall. So i went back again to c2600-adventerprisek9-mz.124-15.T14.bin and I did encounter the same issuea s before, slowness. I did also follow Peter's suggestion and removed the "ip virtual-reassembly but that option did not produce any changes. As far as bugs go, i could not find any that would explain my issue.

I did more research and found that IOS firewall and NAT do not go well together with IP Cef. I found it in one of Cisco's documents. I did not save this article, if i find it again, i will post the link to it.

So i had my IOS firewall, NAT and IP cef configured with the above mentioed IOS. Again, internet browsing was slow. I then removed IP cef, immiediate differecne, browsing was much faster, normal. I put IP cef back in and browsing was slow agian. As of now, IOS Firewall and NAT are configured and browsing is normal.

One thing that was a bit strange for me was that when IP Cef was enabled and IOS firewall not configured, my browsing was still slow. I was not expecting that outcome.

One other thing that i noticed with my 12.4 IOS is that i did not see any dynamic ACL entires created for my return traffic. I was able to verify that my IOS firewall functions as it should. I later found out that starting with IOS 12.3 (4)T (i did not find this on Cisco's web, i'm sure it is there somewhere) Cisco introduced feature called FAB. Instead of creating dynamic ACL entires the IOS examines the state table to determine which trafffic to allow back in.

So, for now it appears that my issue is resolved. I will attempt to find the article that talks about the restrictions for IOS Firewall, NAT and IP Cef. For now, jsut check the abbriviated output of this command, "show ip inspect statistics" Even if IP Cef is enabled, it does not show in the output.

rt-Internet# show ip inspect statistics
Packet inspection statistics [process switch:fast switch]

And below is my current config. More changes are coming to the config as well but for now, this is what i have. Thanks for all the suggestions and if you have anything else, please post it.

!
version 12.4
service timestamps debug datetime msec
service timestamps log datetime msec localtime show-timezone year
service password-encryption
service sequence-numbers
!
hostname rt-Internet
!
boot-start-marker
boot system flash c2600-entservices-mz.124-25d.bin
boot system flash c2600-adventerprisek9-mz.124-15.T14.bin
boot-end-marker
!
logging buffered 4096
no logging console
enable secret 5 $1$yZhB$t3TsT.zwqriZHUnm1Ol750
!
aaa new-model
!
!
aaa authentication attempts login 4
aaa authentication login default local enable
aaa authentication login MY_LIST none
!
!
aaa session-id common
clock timezone MST -7
clock summer-time MST recurring
no network-clock-participate slot 1
no network-clock-participate wic 0
no ip cef
!
!
no ip dhcp use vrf connected
ip dhcp excluded-address 10.10.2.1 10.10.2.21
!
ip dhcp pool HOME
   network 10.10.2.0 255.255.255.0
   default-router 10.10.2.1
   dns-server 4.2.2.2 4.3.3.3
!
!
no ip domain lookup
ip inspect name FIREWALL_OUT http
ip inspect name FIREWALL_OUT https
ip inspect name FIREWALL_OUT udp
ip inspect name FIREWALL_OUT dns
ip inspect name FIREWALL_OUT ftp
ip inspect name FIREWALL_OUT tftp
ip inspect name FIREWALL_OUT ntp
ip inspect name FIREWALL_OUT winmx
ip inspect name FIREWALL_OUT echo
ip inspect name FIREWALL_OUT smtp
ip inspect name FIREWALL_OUT realaudio
ip inspect name FIREWALL_OUT vdolive
ip inspect name FIREWALL_OUT ipsec-msft
ip inspect name FIREWALL_OUT isakmp
ip inspect name FIREWALL_OUT pptp
!
multilink bundle-name authenticated
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
username xxxx password 7 04480A040A2A1B
archive
log config
hidekeys
!
!
!
!
controller T1 0/0
framing sf
linecode ami
!
controller T1 0/1
framing sf
linecode ami
!
controller T1 0/2
framing sf
linecode ami
!
controller T1 0/3
framing sf
linecode ami
!
!
!
!
!
interface Loopback0
ip address 10.0.0.1 255.255.255.255
ip nat inside
ip virtual-reassembly
!
interface FastEthernet0/0
ip address 192.168.15.2 255.255.255.0
ip access-group DENY_OUTSIDE_IN in
ip nbar protocol-discovery
ip nat outside
ip inspect FIREWALL_OUT out
ip virtual-reassembly
speed 100
full-duplex
!
interface FastEthernet0/1
no ip address
speed 100
full-duplex
!
interface FastEthernet0/1.10
encapsulation dot1Q 10
ip address 10.10.1.1 255.255.255.0
ip access-group ALLOW_INSIDE_OUT in
ip nat inside
ip virtual-reassembly
!
interface FastEthernet0/1.20
encapsulation dot1Q 20
ip address 10.10.2.1 255.255.255.0
ip access-group ALLOW_INSIDE_OUT in
ip nat inside
ip virtual-reassembly
!
interface Ethernet1/0
no ip address
ip nat outside
ip virtual-reassembly
shutdown
full-duplex
!
ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 192.168.15.1
!
!
ip http server
no ip http secure-server
ip nat inside source list NAT interface FastEthernet0/0 overload
!
ip access-list standard NAT
permit 10.0.0.1
permit 10.10.1.0 0.0.0.255
permit 10.10.2.0 0.0.0.255
!
ip access-list extended ALLOW_INSIDE_OUT
permit tcp any any established
permit udp any any
permit gre any any
permit icmp any any echo
permit ip any any
deny   ip any any
ip access-list extended DENY_OUTSIDE_IN
permit icmp any any echo-reply
permit udp host 204.123.2.5 any
permit udp host 204.34.198.40 any
permit gre any any
deny   ip any any log
!
access-list 7 permit 204.34.198.40
access-list 7 permit 204.123.2.5
access-list 77 permit 10.10.1.11
access-list 77 permit 10.10.1.20
!
!
!
!
!
control-plane
!
!
!
!
!
!
!
!
!
!
line con 0
logging synchronous
login authentication MY_LIST
line aux 0
logging synchronous
login authentication MY_LIST
line vty 0 4
access-class VTY_LINES in
exec-timeout 0 0
logging synchronous
transport input telnet
transport output none
line vty 5 15
exec-timeout 0 0
logging synchronous
transport input telnet
transport output none
!
ntp clock-period 17180428
ntp source Loopback0
ntp access-group peer 7
ntp access-group serve-only 77
ntp server 204.123.2.5
ntp server 204.34.198.40
!
end

huajie chen · ‎12-12-2013

dear sebastian,

I have the same problem, When I disable the ip cef on cisco2651xm, it works fine. When it is enabled, the internet is very low, but the vpn is ok, and the first packet ping to the internet is always lost.

So, I have done with the problem with you method. Thank you.

My version is c2600-adventerprisek9-mz.124-15.T14.bin

Reza Sharifi · ‎12-25-2010

Sebastian

Thanks for the feedback.

So, we learned 2 things from your testing

1-Use the same IOS with Crypto and disable CEF or

2-Use a none Crypto IOS with CEF enabled

Very interesting!!

Reza

francisco_1 · ‎12-26-2010

The behaviour in your case is not normal that is why i suggested it might be a code issue. Normally some bugs are not published so might want to escalate the issue to Cisco TAC and also this issue might be a new bug so good to get Cisco TAC involved. They will recommend the best solution or workaround.

You could try ZBF which is also stateful (Zone base Firewall) Instead of creating dynamic ACL entires the IOS examines the state table to determine which trafffic to allow back in and this could work with CEF in your case cause i woundnt recommend to have CEF off!

Francisco.

Peter Paluch · ‎12-26-2010

Francisco,

I agree with you, this behavior does not make any sense and it suggests that this is a bug. Your suggestion about trying the ZBFW is worth trying - but then again, the ZBFW is an extension of the IP Inspect and if this bug is related to the IP Inspect, migrating to the ZBFW probably won't help.

What is most interesting, however, is that this bug is actually not related to the IP Inspect feature. If you read the thread closely, the original author indicates that while he indeed wanted to try the IP Inspect, during the debugging of his problem, he removed all advanced features from his configuration including the IP Inspect - and it did not help. Thus, the bug is probably not related to the IP Inspect after all.

Best regards,

Peter

Seba · ‎12-27-2010

Hey guys,

Thanks for your inputs. Based on the troubleshooting that i have done, I don't belive that if this is a bug that it is related to IP inspect. As stated by Peter, i did also encounter slow web browsing when Ip inspect was removed and ip cef enabled. I would love to open a TAC and have this investigated but this is a home router that i use for internet connectivity and studies, so i have no cotract for this device.

Just as i have mentioned before, Ip cef is disabled and my browsing is just fine. I should also mention that this slowness was only encountered when browsing the net. When i was VPNed in from home to work PCs everything worked as normal.

I think that i will put this issue to rest, at least for now.

About my new issue. I took Francisco's suggestion and gave it a shot with ZFW. All works well, except VPN. My wife uses Win 7 to connect to her work PC. So far, I was unsuccessfull getting her connected with ZFW in place. I have no issues with this VPN connection when using CBAC.

Below are my config (for ZFW only) and attached are packet captuers. You will see that i'm not getting ACK for my configuration request when using ZFW. I believe that those ACKs are send to me, they are just blocked by my config.

do appreciate your imputs.

!
version 12.4
service timestamps debug datetime msec
service timestamps log datetime msec localtime show-timezone year
service password-encryption
service sequence-numbers
!
hostname rt-Internet
!
boot-start-marker
boot system flash c2600-adventerprisek9-mz.124-15.T14.bin
boot-end-marker
!
logging buffered 4096
no logging console
enable secret 5 $1$yZhB$t3TsT.zwqriZHUnm1Ol750
!
aaa new-model
!
!
aaa authentication attempts login 4
aaa authentication login default local enable
aaa authentication login MY_LIST none
!
!
aaa session-id common
clock timezone MST -7
clock summer-time MST recurring
no network-clock-participate slot 1
no network-clock-participate wic 0
no ip cef
!
!
no ip dhcp use vrf connected
ip dhcp excluded-address 10.10.2.1 10.10.2.21
!
ip dhcp pool RESERVED_VLAN020_LOCAL_LAN
   host 10.10.2.21 255.255.255.0
   client-identifier 0100.1ec9.3ae3.cb
   default-router 10.10.2.1
   dns-server 208.67.222.222 208.67.220.220 4.2.2.2 4.2.2.3
!
ip dhcp pool VLAN020_LOCAL_LAN
   network 10.10.2.0 255.255.255.0
   default-router 10.10.2.1
   dns-server 208.67.222.222 208.67.220.220 4.2.2.2 4.2.2.3
!
!
no ip domain lookup
!
multilink bundle-name authenticated
!
!
!
!
!
username xxxx password 7 04480A040A2A1B
archive
log config
hidekeys
!
!
!
!
controller T1 0/0
framing sf
linecode ami
!
controller T1 0/1
framing sf
linecode ami
!
controller T1 0/2
framing sf
linecode ami
!
controller T1 0/3
framing sf
linecode ami
!
!
class-map type inspect match-any PERMIT_BETWEEN_ZONES
match protocol http
match protocol https
match protocol pptp
match protocol ftp
match protocol tftp
match protocol ntp
match protocol winmx
match protocol echo
match protocol smtp
match protocol realmedia
match protocol vdolive
match protocol ssh
match protocol pop3
match protocol tcp
match protocol udp
match access-group name ALLOW_GRE
!
!
policy-map type inspect INSPECT_PERMITTED_TRAFFIC
class type inspect PERMIT_BETWEEN_ZONES
inspect
class class-default
pass log
!
zone security LAN
zone security WAN
zone-pair security LAN_TO_WAN source LAN destination WAN
service-policy type inspect INSPECT_PERMITTED_TRAFFIC
!
!
interface Loopback0
ip address 10.0.0.1 255.255.255.255
ip nat inside
ip virtual-reassembly
!
interface FastEthernet0/0
ip address 192.168.15.2 255.255.255.0
ip nat outside
ip virtual-reassembly
zone-member security WAN
speed 100
full-duplex
!
interface FastEthernet0/1
ip address 10.10.2.1 255.255.255.0
ip nat inside
ip virtual-reassembly
zone-member security LAN
speed 100
full-duplex
!
interface Ethernet1/0
no ip address
ip nat outside
ip virtual-reassembly
shutdown
full-duplex
!
ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 192.168.15.1
!
!
ip http server
no ip http secure-server
ip nat inside source list NAT interface FastEthernet0/0 overload
!
ip access-list standard NAT
permit 10.0.0.1
permit 10.10.1.0 0.0.0.255
permit 10.10.2.0 0.0.0.255
!
ip access-list extended ALLOW_GRE
permit gre any any
!
logging trap debugging
logging source-interface Loopback0
logging 10.10.2.21
access-list 7 permit 204.34.198.40
access-list 7 permit 204.123.2.5
access-list 77 permit 10.10.1.11
access-list 77 permit 10.10.1.20
!
!
control-plane
!
line con 0
logging synchronous
login authentication MY_LIST
line aux 0
logging synchronous
login authentication MY_LIST
line vty 0 4
access-class VTY_LINES in
exec-timeout 0 0
logging synchronous
transport input telnet
transport output none
line vty 5 15
exec-timeout 0 0
logging synchronous
transport input telnet
transport output none
!
ntp clock-period 17180445
ntp source Loopback0
ntp access-group peer 7
ntp access-group serve-only 77
ntp server 204.123.2.5
ntp server 204.34.198.40
!
end

Seba · ‎12-28-2010

Sorry guys, but i have attached two of the same packets captures. I will attach the correct once tonight.

Seba · ‎12-28-2010

So I found a solution to my problem....

I used this command to "ip inspect log drop-pkt" to find out what is wrong with the above config (my previous post) and here is the message that was logged.

000044: Dec 28 2010 22:48:18.163 MST: %FW-6-DROP_PKT: Dropping session 155.101.167.245:50081 10.10.2.25:80 due to No
zone-pair between zones with ip ident 51521

What i needed to do is to either pass GRE traffic out and back in, or to inspect GRE traffic out and back in. It also worked when i was inspecting GRE traffic out and passing it back in, or the other way around. The bottom line was that something neded to be set, either inspect or pass for the returning GRE traffic. My understanding is that i can't inspect GRE traffic so i'm just passing it in both directions. With that being said, it was kind of weird that my VPN connection would work just fine if i specified to inspect GRE traffic. Am I correct with the statement that i can't inspect GRE traffic?

Here is my current config, relevant to the issue

class-map type inspect match-any DONT_INSPECT_PERMIT_BETWEEN_ZONES_CMAP
match access-group name ALLOW_GRE
class-map type inspect match-any PERMIT_BETWEEN_ZONES_CMAP
match protocol tcp
match protocol udp
match protocol icmp
!
!
policy-map type inspect PERMITTED_TRAFFIC_PMAP
class type inspect DONT_INSPECT_PERMIT_BETWEEN_ZONES_CMAP
pass
class type inspect PERMIT_BETWEEN_ZONES_CMAP
inspect
class class-default
policy-map type inspect ALLOW_RETURNED_TRAFFIC
class type inspect DONT_INSPECT_PERMIT_BETWEEN_ZONES_CMAP
pass
class class-default
!
zone security LAN
zone security WAN
zone-pair security LAN_TO_WAN source LAN destination WAN
service-policy type inspect PERMITTED_TRAFFIC_PMAP
zone-pair security WAN_TO_LAN source WAN destination LAN
service-policy type inspect ALLOW_RETURNED_TRAFFIC

!

ip access-list extended ALLOW_GRE
permit gre any any

!

interface FastEthernet0/0

zone-member security WAN

!

interface FastEthernet0/1.10

zone-member security LAN

!

interface FastEthernet0/1.20

zone-member security LAN