3845 router with poor performance???

sergiope · ‎06-16-2010

Hi,

We have a 3845 router installed and we are using it to route Internet traffic to the LAN, we are running around 70 MB of average traffic approximately during working hours. All the traffic flows between two Gigabit Ethernet interfaces, and the configuration is pretty plain, we are running only static routing to reach some hosts and we only have a route-map applied on the internal interface of the router (we are not running NAT in this router).

The problem we are observing is that suddenly when we access the router via telnet we are getting very slow response from the router, for example, when we write something on the CLI the characters appear really slowly on the screen, and when we try to get any output from the router it takes some time until it is displayed completely.

We have verified the CPU load and it never goes above 30% (it gets this high when we try to get a show-tech), we have no telnet sessions open (besides our own), we have tried to enable Fast-switched PBR to avoid any issues with the route-map we have applied on the Gigabit Ethernet interface, but there was no difference. The CPU sometimes shows around 4% of use and we still have this issues with the router.

The router is running the following IOS image: c3845-ipbase-mz.124-9.T6, and it has been running for 1 year, 12 weeks, 6 days, 15 hours, 0 minutes (at the time of this post).

This issue could be related with a software bug? Is there anything else we can try in order to solve this issue?.

Here is the configuration we have on the router (show tech is also attached):

version 12.4
service timestamps debug datetime msec localtime
service timestamps log datetime msec localtime
service password-encryption
!
hostname ENT_HUBII
!
boot-start-marker
boot-end-marker
!
logging buffered 4096 informational
logging console informational
logging monitor informational
enable secret 5 <removed>
enable password 7 <removed>
!
no aaa new-model
!
resource policy
!
clock timezone CDT-6 -6
clock summer-time CDT-6 recurring 1 Sun Apr 0:00 last Sun Oct 0:00
ip cef
!
!
!
!
no ip domain lookup
!
username cabrera privilege 15 password 7 <removed>
username consola privilege 15 password 7 <removed>
username casasola privilege 15 password 7 <removed>
username ccenter privilege 15 password 7 <removed>
username rsalazar privilege 15 password 7 <removed>
username galindo privilege 15 password 7 <removed>
!
!
!
!
interface GigabitEthernet0/0
ip address 192.168.254.118 255.255.255.248
duplex full
speed 100
media-type rj45
!
interface GigabitEthernet0/1
ip address 192.168.17.30 255.255.255.128
ip policy route-map GoTo
duplex full
speed 100
media-type rj45
no cdp enable
!
ip route 0.0.0.0 0.0.0.0 192.168.254.113
ip route 10.1.20.0 255.255.255.0 192.168.17.22
ip route 10.50.0.0 255.255.192.0 192.168.17.1
ip route 10.50.64.0 255.255.192.0 192.168.17.1
ip route 10.50.128.0 255.255.224.0 192.168.17.2
ip route 10.50.160.0 255.255.224.0 192.168.17.3
ip route 10.50.192.0 255.255.192.0 192.168.17.5
ip route 10.50.192.0 255.255.255.0 192.168.17.6
ip route 10.50.195.0 255.255.255.0 192.168.17.6
ip route 10.51.0.0 255.255.255.0 192.168.17.1
ip route 10.51.1.0 255.255.255.0 192.168.17.1
ip route 10.51.2.0 255.255.255.0 192.168.17.1
ip route 10.52.0.0 255.255.192.0 192.168.17.33
ip route 192.168.14.0 255.255.255.0 192.168.17.22
ip route 194.49.1.0 255.255.255.128 192.168.17.1
ip route 201.159.226.0 255.255.255.0 192.168.17.6
ip route 201.159.227.0 255.255.255.0 192.168.17.1
ip route 201.159.228.0 255.255.255.0 192.168.17.1
ip route 201.159.228.232 255.255.255.252 192.168.17.6
ip route 201.159.229.0 255.255.255.128 192.168.17.6
ip route 201.159.229.128 255.255.255.192 192.168.17.6
ip route 201.159.229.192 255.255.255.224 192.168.17.6
ip route 201.159.229.224 255.255.255.240 192.168.17.6
ip route 201.159.229.240 255.255.255.248 192.168.17.6
ip route 201.159.230.0 255.255.255.0 192.168.17.6
ip route 201.159.238.0 255.255.255.0 192.168.17.1
ip route 201.159.238.0 255.255.255.192 192.168.17.33
ip route 201.159.239.0 255.255.255.0 192.168.17.6
ip route 201.159.244.0 255.255.252.0 192.168.17.1
!
no ip http server
!
logging facility local4
logging 192.168.17.22
access-list 4 permit 192.168.17.21
access-list 4 permit 192.168.17.22
access-list 4 permit 192.168.14.213
access-list 7 permit 192.168.17.0 0.0.0.127
access-list 10 permit 201.159.238.184 0.0.0.7
access-list 31 permit 194.49.1.0 0.0.0.255
access-list 31 permit 10.51.1.32 0.0.0.15
snmp-server community <removed> RO 4
snmp-server ifindex persist
no cdp run
route-map GoTo permit 10
match ip address 10
set ip next-hop 192.168.17.26
!
route-map GoTo permit 31
match ip address 31
set ip next-hop 192.168.17.31
!
!
control-plane
!
!
line con 0
line aux 0
line vty 0 4
access-class 7 in
exec-timeout 5 0
password 7 <removed>
login local
width 120
line vty 5 15
access-class 7 in
exec-timeout 5 0
password 7 <removed>
login local
width 120
!
scheduler allocate 20000 1000
ntp server 192.168.17.22
!
end

Please let me know your thoughts about this, any help is more than welcome!.

Jerry Ye · ‎06-16-2010

I am assuming you mean 80Mbps. However, according from the data sheet, the 3845 is designed for T3/E3 line rate.

"these routers deliver multiple concurrent services up to wire-speed T3/E3 rates"

http://www.cisco.com/en/US/prod/collateral/routers/ps5855/product_data_sheet0900aecd8016a8e8.html

If you are looking for routing in and out of the GE with 80Mbps with PBR (no NAT), your might want to look into the Catalyst 3750 series switch.

HTH,

jerry

sergiope · ‎06-17-2010

Jerry,

Thanks for the information you provided. However, I was not asking for a design recommendation, we are looking for a solution for this issue, maybe a software bug or anything else, because as you can imagine this is a really weird issue and it started with any apparent reason.

Again, thanks for your information, please let me know if you have any other suggestion.

Best regards.

John Blakley · ‎06-17-2010

A couple of things that I would try is removing the PBR off of the interface and then try. Do you have the same slowness if you console into the router? Of course, I'd only remove the policy during off hours to see if it's an issue. Other than that, does the router run normally? Does the speed for all of the users seem to be okay? If that's the case, you may have a routing issue somewhere.

*Edit* - I didn't see where it's been up for a year...... did this just start happening, and what was the last change that was made to the router before seeing the problem creep up?

HTH,

John

HTH, John *** Please rate all useful posts ***

Nilesh Joshi · ‎06-17-2010

try configuration on output interpreter tool of cisco

Regards

Niljos

Nicholas Oliver · ‎07-14-2010

sergiope,

When you say you are experiencing poor performance, I assume we are talking specifically about the responsiveness to telnet, and that traffic crossing the router is fine. Is that correct? The CPU utilization and memory both appear to be fine (assuming that the attached 'show tech' was when the problem was present). When you telnet into the router, where are you telnetting from? Based on the configuration I assume it is a machine that is attached to the G0/1 segment as the access-class applied to the VTY lines would prevent other sources from being successful. When you have this delay via telnet, is the same behavior experienced via the console? Or does the console continue to be responsive during that period? What device is attached to G0/1 (I assume it is a switch) and what does its interface look like (config and 'show int')

A couple of things that I would suggest you try, in an effort to narrow down the cause of your problem:

1) Add a second subnet to access-list 7 and attempt to telnet to the device from something attached to G0/0, does the problem persist there?

2) Remove the access-class from the VTY lines temporarily and attempt to telnet from G0/0 and G0/1, does the problem persist at that point?

3) The route-map does not appear to be impacting the telnet traffic, but it wouldn't be difficult to temporarily remove it to determine if the problem persists at that point.

4) Attempt to console into the router, and telnet back to itself. Does the same lag exist when you do this, as you see when you attach to the device from a workstation?

Based on that information it should give us a better idea of where we should focus our troubleshooting.

-Nick