01-14-2016 01:56 PM - edited 03-08-2019 03:24 AM
Preamble
A couple years ago I managed to get hold of some ancient Cisco equipment, probably a pile of rubble to the eyes of professionals, but a little treasure for a young man like me that wanted to learn something about networking. To keep it short, I managed to fully recover a 2611xm and while playing with it (many days of trial & error) I noticed that one IOS feature was not really working like intended.
The issue
CEF, or to be precise, the "ip cef" command seems to put the router off its tracks. Whenever CEF is enabled the connectivity starts misbehaving, opening web pages takes considerably longer even though the bandwidth available is still the same. From my inexperienced point of view it seems like having a bad dns server, one with 500 ms latency, that's what it feels like to enable CEF on my router.
Considering my setup (below and in the *.*txt attached) is this behaviour an anomaly or is to be expected?
Some details
HW/SW:
Cisco 2611xm (C2600-ADVSECURITYK9-M), Version 12.4(15)T14, RELEASE SOFTWARE (fc2), 256MB RAM, 32MB Flash.
Topology:
Enabled Features:
dhcp server, ddns, QoS (CBWFQ), Zone-Based Policy Firewall, PPPoE client, NAT overload, NTP.
Running-config:
version 12.4
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps log datetime msec localtime show-timezone
service password-encryption
service sequence-numbers
!
hostname shield2
!
boot-start-marker
boot-end-marker
!
logging buffered 240000
no logging console
no logging monitor
enable secret 5 *snip*
enable password 7 *snip*
!
no aaa new-model
clock timezone CET 1
clock summer-time CET recurring last Sun Mar 2:00 last Sun Oct 3:00
no network-clock-participate slot 1
no network-clock-participate wic 0
no ip source-route
no ip gratuitous-arps
ip options drop
no ip cef
!
!
no ip dhcp use vrf connected
ip dhcp bootp ignore
no ip dhcp conflict logging
ip dhcp excluded-address 192.168.0.1 192.168.0.255
!
ip dhcp pool uss-local-dynamic
import all
network 192.168.0.0 255.255.254.0
domain-name *snip*
default-router 192.168.0.12
dns-server 8.8.8.8 8.8.4.4 4.2.2.2
option 26 hex 05d4
!
*snip*
!
no ip bootp server
ip domain name *snip*
*snip*
ip name-server 8.8.8.8
ip name-server 8.8.4.4
ip name-server 4.2.2.2
ip name-server 4.2.2.4
!
ip ddns update method no-ip
HTTP
add *snip*
interval maximum 2 0 0 0
!
multilink bundle-name authenticated
!
username root privilege 15 password 7 *snip*
archive
log config
hidekeys
!
ip ssh rsa keypair-name FAKE
!
class-map type inspect match-all INSIDE-TO-OUTSIDE-CLASS
match access-group name INSIDE-TO-OUTSIDE
class-map type inspect match-all OUTSIDE-TO-INSIDE-CLASS
match access-group name OUTSIDE-TO-INSIDE
class-map type inspect match-all SELF-TO-OUTSIDE-CLASS
match access-group name SELF-TO-OUTSIDE
class-map type inspect match-all OUTSIDE-TO-SELF-CLASS
match access-group name OUTSIDE-TO-SELF
!
class-map match-any Hi-Class-Outbound-Upload
match access-group name Hi-Precedence-Upload
!
class-map match-any Hi-Class-Outbound-Download
match access-group name Hi-Precedence-Download
!
policy-map Packet-Queueing-Download
class Hi-Class-Outbound-Download
priority percent 75
class class-default
fair-queue
queue-limit 8
!
policy-map Packet-Shaping-Download
class class-default
shape peak 3300000 33000
service-policy Packet-Queueing-Download
!
policy-map Packet-Queueing-Upload
class Hi-Class-Outbound-Upload
priority percent 75
class class-default
fair-queue
queue-limit 8
!
policy-map Packet-Shaping-Upload
class class-default
shape peak 175000 1750
service-policy Packet-Queueing-Upload
!
policy-map type inspect INSIDE-TO-OUTSIDE-POLICY
class type inspect INSIDE-TO-OUTSIDE-CLASS
inspect
class class-default
drop
policy-map type inspect OUTSIDE-TO-INSIDE-POLICY
class type inspect OUTSIDE-TO-INSIDE-CLASS
inspect
class class-default
drop log
policy-map type inspect SELF-TO-OUTSIDE-POLICY
class type inspect SELF-TO-OUTSIDE-CLASS
inspect
class class-default
drop
policy-map type inspect OUTSIDE-TO-SELF-POLICY
class type inspect OUTSIDE-TO-SELF-CLASS
inspect
class class-default
drop log
!
zone security INSIDE
zone security OUTSIDE
zone-pair security IN-TO-OUT source INSIDE destination OUTSIDE
service-policy type inspect INSIDE-TO-OUTSIDE-POLICY
zone-pair security OUT-TO-IN source OUTSIDE destination INSIDE
service-policy type inspect OUTSIDE-TO-INSIDE-POLICY
zone-pair security SELF-TO-OUT source self destination OUTSIDE
service-policy type inspect SELF-TO-OUTSIDE-POLICY
zone-pair security OUT-TO-SELF source OUTSIDE destination self
service-policy type inspect OUTSIDE-TO-SELF-POLICY
!
!
interface FastEthernet0/0
description LAN Interface
ip address 192.168.0.12 255.255.254.0
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat inside
no ip virtual-reassembly
zone-member security INSIDE
ip tcp adjust-mss 1452
speed auto
full-duplex
no cdp enable
no mop enabled
service-policy output Packet-Shaping-Download
!
interface FastEthernet0/1
description ADSL WAN Interface
no ip address
no ip redirects
no ip unreachables
no ip proxy-arp
no ip virtual-reassembly
duplex auto
speed auto
pppoe enable group global
pppoe-client dial-pool-number 1
no cdp enable
no mop enabled
service-policy output Packet-Shaping-Upload
!
interface Dialer1
description ADSL WAN Dialer
bandwidth 7200
ip ddns update hostname *snip*
ip ddns update no-ip
ip address negotiated
no ip redirects
no ip unreachables
no ip proxy-arp
ip mtu 1492
ip nat outside
ip virtual-reassembly
zone-member security OUTSIDE
encapsulation ppp
no ip mroute-cache
dialer pool 1
dialer-group 1
no cdp enable
ppp authentication pap callin
ppp pap sent-username *snip* password 7 *snip*
ppp ipcp route default
!
no ip forward-protocol nd
!
no ip http server
no ip http secure-server
no ip dns server
ip nat inside source list 7 interface Dialer1 overload
ip nat inside source static tcp 192.168.1.22 51613 interface Dialer1 51613
ip nat inside source static udp 192.168.1.22 51613 interface Dialer1 51613
!
ip access-list extended INSIDE-TO-OUTSIDE
permit ip 192.168.0.0 0.0.1.255 any
!
ip access-list extended OUTSIDE-TO-INSIDE
permit tcp any host 192.168.1.22 eq 51613
permit udp any host 192.168.1.22 eq 51613
!
ip access-list extended OUTSIDE-TO-SELF
deny ip any any
!
ip access-list extended SELF-TO-OUTSIDE
permit ip any any
!
ip access-list extended Hi-Precedence-Upload
remark --- High precedence connections
permit icmp any any
permit tcp any any rst syn
permit udp any any eq domain
permit tcp any any eq 2099
permit tcp any any range 5222 5223
permit udp any any range 5000 5500
!
ip access-list extended Hi-Precedence-Download
remark --- High precedence connections
permit icmp any any
permit tcp any any rst syn
permit udp any eq domain any
permit tcp any eq 2099 any
permit tcp any range 5222 5223 any
permit udp any range 5000 5500 any
!
access-list 7 permit 192.168.0.12
access-list 7 permit 192.168.1.0 0.0.0.63
dialer-list 1 protocol ip permit
snmp-server community *snip* RO
snmp-server chassis-id SHIELD2
snmp-server contact *snip*
!
no cdp run
!
control-plane
!
banner motd ^CC
*snip*
!
line con 0
exec-timeout 0 0
line aux 0
exec-timeout 0 1
no exec
transport output none
line vty 0 4
exec-timeout 120 0
password 7 *snip*
login
transport input telnet
!
no process cpu extended
no process cpu autoprofile hog
ntp clock-period 17208491
ntp server 193.204.114.232 key 0 prefer
!
end
01-18-2016 01:15 AM
Hey Nothing wrong physically with your setup , is it when you enable it globally or when you enable it per interface you face the issue ? If its globally try per interface see if its same issue
either way ran your output attached you need to change that IOS it found issue in mem which wouldn't be helping your cef issue :)
SHOW BUFFERS ANALYSIS
ERROR: Since it's last reload, this router has created or maintained a relatively large number of 'Syslog ED Pool buffers' yet still has very few free buffers. The above symptoms suggest that a buffer leak has occurred. BUFFER LEAK: When a process is finished with a buffer, the process should free the buffer. A buffer leak occurs when the code forgets to process a buffer, or forgets to free it after. It is done with the packet As a result, the buffer pool continues to grow as more and more packets are stuck in the buffers. Some routers (for example, 2600, 3600, and 4000 Series) require a minimum amount of I/O memory to support certain interface processors. Not Enough Shared Memory for the Interfaces. NOTE: (1)Some of the Public Buffer pools should be abnormally large with few free buffers. After a reload, you may see that the number of free buffers never gets close to the number of total buffers. (2)You should check the buffers on a regular basis. Some leaks are slow but others are very fast. (3)If you configure or access the router through telnet,you need to check the buffers on a regular basis via remote access (telnet) before the router hang to see in which pool is the leak. Once you see that for one pool the total number is increasing and the free number is low (the faulty pool), you need to capture a 'show buffer pool
01-19-2016 07:22 AM
Yes, when I enable it globally with the ip cef command. I then tried with the global ip cef + no ip route-cache cef to selectively disable it, but the issue persisted. Is this what you mean with "enable it per interface"?
Now, that thing about the buffers is alarming because I'm already running the latest IOS available for my platform (12.4(15)T14) so I don't really know how to fix that.
Can you please elaborate on that long error message? I'd like to know if I can do something to alleviate the problem.
Thanks.
01-19-2016 07:30 AM
Yes if you have memory leak in IOS it will effect most things on the router and things will start to act irregularly not just in cef , the fact your on the latest image you could try roll back not really recommended but what you triggered in one version of IOS to hit the bug may not be the same in previous versions. If reboots not clearing issue and still occurring I don't see any other option as the problems in the software so you need to get off it.
When you have no further memory issues I would then look at Cef and why exactly its not working , trying to troubleshoot something that relies on memory when memory is leaking will not benefit you or your time :) as cef keeps a record of the fib to work off
01-19-2016 08:29 AM
I'm gonna try with a previous IOS version then, but I have to stick to the train release, since ZBWF is not present on the mainline.
Before the downgrade I want to investigate a little more on the setting in my configuration that triggered the buffer leak. If I disable all enabled features one by one maybe I can find the culprit. Can you please tell me what parameter should I consider in the sh buffers output when searching for a buffer leak? The error message above is confusing me, it states "this router has created or maintained a relatively large number of 'Syslog ED Pool buffers' yet still has very few free buffers" but my log is reporting 118 free buffers.
Syslog ED Pool buffers, 600 bytes (total 150, permanent 150): 118 in free list (150 min, 150 max allowed) 3187 hits, 0 misses
01-19-2016 08:34 AM
I wouldn't be an expert in mem leaks whenever I come across them I replace the image as it only gets worse
This may help you though, also you can check your actual release guide for that IOS in download section it will show you any known memory issue bugs that were uncovered
As well collect the show memory outputs and use the output interpreter on the website , it dissects the mem and will provide more info , thats what provided the buffer analysis in previous post
http://www.cisco.com/c/en/us/support/web/tools-catalog.html
http://www.cisco.com/c/en/us/td/docs/ios/fundamentals/configuration/guide/15_1s/cf_15_1s_book/cf_mem-leak-detect.html
01-19-2016 08:46 AM
I'm gonna check the known bugs for my release, sadly I can't use the output interpreter, it requires a support contract it seems.
CEF not working properly is not the only problem here sadly... Thanks for your help, I'm gonna report back when I have resolved this mem leak issue and maybe CEF will work by then.
01-19-2016 10:09 AM
post what you need for the output interpreter in text file i can run it for you and post it back here
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide