running constantly at 89% Memory Untilization

[cheung2009]

Hi Expert

Please help...

I have two 6509 switches with identical hardware and IOS version. They both are running constantly 89% of memory utilization. However the CPU utilization are quite low.

Any idea what caused that, probably the IOS???

Thank

Kelvin Cheung

============================

cumbdck54rs01#sh version

Cisco Internetwork Operating System Software

IOS (tm) s72033_rp Software (s72033_rp-ENTSERVICESK9_WAN-VM), Version 12.2(18)SXF16, RELEASE SOFTWARE (fc2)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2009 by cisco Systems, Inc.

Compiled Tue 03-Mar-09 21:06 by kellythw

Image text-base: 0x01020150, data-base: 0x01021000

ROM: System Bootstrap, Version 12.2(17r)SX5, RELEASE SOFTWARE (fc1)

BOOTLDR:

cumbdck54rs01 uptime is 17 weeks, 2 days, 10 hours, 41 minutes

Time since cumbdck54rs01 switched to active is 17 weeks, 2 days, 10 hours, 41 minutes

System returned to ROM by power cycle at 06:13:02 UTC Mon Jun 15 2009 (SP by power on)

System restarted at 01:46:22 EAST Sun Apr 10 2011

System image file is "sup-bootdisk:/s72033-entservicesk9_wan-vz.122-18.SXF16.bin"

　

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

cisco WS-C6504-E (R7000) processor (revision 2.0) with 491520K/32768K bytes of memory.

Processor board ID FOX1238G4SB

SR71000 CPU at 600Mhz, Implementation 1284, Rev 1.2, 512KB L2 Cache

Last reset from s/w reset

Bridging software.

X.25 software, Version 3.0.0.

SuperLAT software (copyright 1990 by Meridian Technology Corp).

TN3270 Emulation software.

1 Virtual Ethernet/IEEE 802.3 interface

28 Gigabit Ethernet/IEEE 802.3 interfaces

8 Ten Gigabit Ethernet/IEEE 802.3 interfaces

1917K bytes of non-volatile configuration memory.

65536K bytes of Flash internal SIMM (Sector size 512K).

Configuration register is 0x2102

Patching is not available since the system is not running from an installed image. To install please use the "install file" command

cumbdck54rs01#

cumbdck54rs01#sh module

Mod Ports Card Type Model Serial No.

--- ----- -------------------------------------- ------------------ -----------

1 2 Supervisor Engine 720 (Active) WS-SUP720-3B SAL1246A8S5

2 2 Supervisor Engine 720 (Hot) WS-SUP720-3B SAL1246A8MJ

3 24 CEF720 24 port 1000mb SFP WS-X6724-SFP SAL1021P7DR

4 8 CEF720 8 port 10GE with DFC WS-X6708-10GE SAL1417GRNT

Mod MAC addresses Hw Fw Sw Status

--- ---------------------------------- ------ ------------ ------------ -------

1 000a.b86d.9dfc to 000a.b86d.9dff 5.7 8.5(2) 12.2(18)SXF1 Ok

2 001a.a22d.55c4 to 001a.a22d.55c7 5.7 8.5(2) 12.2(18)SXF1 Ok

3 0017.e041.2f00 to 0017.e041.2f17 2.3 12.2(14r)S5 12.2(18)SXF1 Ok

4 8843.e1d0.f320 to 8843.e1d0.f327 2.1 12.2(18r)S1 12.2(18)SXF1 Ok

Mod Sub-Module Model Serial Hw Status

---- --------------------------- ------------------ ----------- ------- -------

1 Policy Feature Card 3 WS-F6K-PFC3B SAL1246A69V 2.4 Ok

1 MSFC3 Daughterboard WS-SUP720 SAL1246A4QR 3.2 Ok

2 Policy Feature Card 3 WS-F6K-PFC3B SAL1246A1ZM 2.4 Ok

2 MSFC3 Daughterboard WS-SUP720 SAL1246A9ED 3.2 Ok

3 Distributed Forwarding Card WS-F6700-DFC3A SAL12448RTT 2.6 Ok

4 Distributed Forwarding Card WS-F6700-DFC3C SAL1417GHJS 1.4 Ok

Mod Online Diag Status

---- -------------------

1 Pass

2 Pass

3 Pass

4 Pass

cumbdck54rs01#

=================================

[Somasundaram Jayaraman]

Hi,

You can do the command 'show memory statistics' to see an overview of

the memory pool utilization.  If it is really at 95% as reported, then

please share the output of 'show proc mem sorted' to see what is hogging the memory.

Cheers

Somu

[cheung2009]

Dear Somu

Thank for your help.....

I think there is no such command "'show memory statistics'" and 'show proc mem sorted' in that IOS version.

Pls see the "show command" below, I hope it helps.....

cumbdck54rs01#show processes mem
System Memory : 524288K total, 469288K used, 55000K free, 1000K kernel reserved
Lowest(b)     : 52371456
PID       Text    Data   Stack  Dynamic     Total Process
1            0       0       0        0         0 kernel                       
16386     2068     100      12      184      2364 sbin/chkptd.proc             
3          604      24      12      324       964 sbin/qdelogger               
4          620      28       4      192       844 sbin/devc-pty                
5          752      32      12      112       908 sbin/devc-mistral.proc       
6          596      24      12       96       728 sbin/pipe                    
7         2220     176      32      280      2708 sbin/dumper.proc             
4104      2448     196      24      392      3060 sbin/bflash_driver.proc      
8201      2448     176      28      388      3040 sbin/pcmcia_driver.proc      
12298      592      24       4       96       716 sbin/mqueue                  
12299     2292     156      20      220      2688 sbin/flashfs_hes.proc        
12300     2164     112      16      652      2944 sbin/dfs_bootdisk.proc       
12301     2412     152      28      192      2784 sbin/ldcache.proc            
12302      716      32       8      100       856 sbin/watchdog.proc           
12303     2052     104      16      252      2424 sbin/syslogd.proc            
12304     1996      96      24      832      2948 sbin/name_svr.proc           
12305     2508     244      44      384      3180 sbin/wdsysmon.proc           
12306     2512     288      96      452      3348 sbin/sysmgr.proc             
16403     2512     288      16      220      3036 sbin/sysmgr.proc             
16404     1988     100      12      184      2284 sbin/syslog_dev.proc         
16405     2008     100      20      192      2320 sbin/itrace_exec.proc        
PID       Text    Data   Stack  Dynamic     Total Process
16406     2112     104      28      220      2464 sbin/packet.proc             
16407     2428     152      24      216      2820 sbin/installer.proc          
16408    64648   32308      84   250680    347720 sbin/ios-base                
16409     2432     136      24      200      2792 sbin/fh_fd_rf.proc           
16410     2308     120      20      196      2644 sbin/fh_fd_oir.proc          
16411     2472     140      20      196      2828 sbin/fh_fd_cli.proc          
16412     2316     112      24      356      2808 sbin/fh_metric_dir.proc      
16413     2456     124      24      200      2804 sbin/fh_fd_snmp.proc         
16414     2300     116      20      192      2628 sbin/fh_fd_none.proc         
16415     2372     132      24      200      2728 sbin/fh_fd_intf.proc         
16416     2312     136      24      200      2672 sbin/fh_fd_gold.proc         
16417     2300     116      24      196      2636 sbin/fh_fd_counter.proc      
16418     2304     116      24      196      2640 sbin/fh_fd_timer.proc        
16419     2312     116      24      196      2648 sbin/fh_fd_ioswd.proc        
16420     2636     156      36      448      3276 sbin/fh_server.proc          
16421     3776     176      28      232      4212 sbin/fh_policy_dir.proc      
16422     1984      96      24      188      2292 sbin/ipfs_daemon.proc        
16423     3500     152      52      264      3968 sbin/raw_ip.proc             
16424     3152     120      48      228      3548 sbin/inetd.proc              
16425     3756     200     116      420      4492 sbin/tcp.proc                
16426     3684     168      52      268      4172 sbin/udp.proc                
16427     9908     372      40     4596     14916 sbin/iprouting.iosproc       
16428     5492     228      36      608      6364 sbin/cdp2.iosproc            
cumbdck54rs01#

==============

cumbdck54rs01#show memory detailed all | in dynamic|free
System Memory: 524288K total, 469228K used, 55060K free, 1000K kernel reserved
    0K total, 0K text, 0K data, 0K stack, 0K dynamic
    2364K total, 2068K text, 100K data, 12K stack, 184K dynamic
    57344 heapsize, 49188 allocated, 6004 free
    964K total, 604K text, 24K data, 12K stack, 324K dynamic
    249856 heapsize, 202788 allocated, 45864 free
    844K total, 620K text, 28K data, 4K stack, 192K dynamic
    908K total, 752K text, 32K data, 12K stack, 112K dynamic
    728K total, 596K text, 24K data, 12K stack, 96K dynamic
    16384 heapsize, 5856 allocated, 9804 free
    2708K total, 2220K text, 176K data, 32K stack, 280K dynamic
    151552 heapsize, 60856 allocated, 88612 free
    3060K total, 2448K text, 196K data, 24K stack, 392K dynamic
    266240 heapsize, 246668 allocated, 17332 free
    3040K total, 2448K text, 176K data, 28K stack, 388K dynamic
    262144 heapsize, 246220 allocated, 13796 free
    716K total, 592K text, 24K data, 4K stack, 96K dynamic
    16384 heapsize, 4408 allocated, 11260 free
    2688K total, 2292K text, 156K data, 20K stack, 220K dynamic
    90112 heapsize, 64800 allocated, 23224 free
    2944K total, 2164K text, 112K data, 16K stack, 652K dynamic
    532480 heapsize, 509108 allocated, 21116 free
    2784K total, 2412K text, 152K data, 28K stack, 192K dynamic
    61440 heapsize, 55196 allocated, 4112 free
    856K total, 716K text, 32K data, 8K stack, 100K dynamic
    16384 heapsize, 1132 allocated, 14752 free
    2424K total, 2052K text, 104K data, 16K stack, 252K dynamic
    126976 heapsize, 113584 allocated, 11452 free
    2948K total, 1996K text, 96K data, 24K stack, 832K dynamic
    720896 heapsize, 633728 allocated, 42060 free
    3180K total, 2508K text, 244K data, 44K stack, 384K dynamic
    245760 heapsize, 173416 allocated, 58664 free
    3348K total, 2512K text, 288K data, 96K stack, 452K dynamic
    331776 heapsize, 265944 allocated, 60608 free
    3036K total, 2512K text, 288K data, 16K stack, 220K dynamic
    94208 heapsize, 86616 allocated, 4284 free
    2284K total, 1988K text, 100K data, 12K stack, 184K dynamic
    57344 heapsize, 46752 allocated, 8660 free
    2320K total, 2008K text, 100K data, 20K stack, 192K dynamic
    65536 heapsize, 55604 allocated, 7960 free
    2464K total, 2112K text, 104K data, 28K stack, 220K dynamic
    94208 heapsize, 78928 allocated, 12908 free
    2820K total, 2428K text, 152K data, 24K stack, 216K dynamic
    90112 heapsize, 75280 allocated, 12228 free
    347720K total, 64648K text, 32308K data, 84K stack, 250680K dynamic
    32768 heapsize, 24292 allocated, 6084 free
    2792K total, 2432K text, 136K data, 24K stack, 200K dynamic
    73728 heapsize, 62380 allocated, 9096 free
    2644K total, 2308K text, 120K data, 20K stack, 196K dynamic
    69632 heapsize, 59436 allocated, 7984 free
    2828K total, 2472K text, 140K data, 20K stack, 196K dynamic
    69632 heapsize, 60704 allocated, 6760 free
    2808K total, 2316K text, 112K data, 24K stack, 356K dynamic
    233472 heapsize, 214288 allocated, 15660 free
    2804K total, 2456K text, 124K data, 24K stack, 200K dynamic
    73728 heapsize, 63132 allocated, 8328 free
    2628K total, 2300K text, 116K data, 20K stack, 192K dynamic
    65536 heapsize, 58992 allocated, 4520 free
    2728K total, 2372K text, 132K data, 24K stack, 200K dynamic
    73728 heapsize, 62416 allocated, 9064 free
    2672K total, 2312K text, 136K data, 24K stack, 200K dynamic
    73728 heapsize, 63856 allocated, 7684 free
    2636K total, 2300K text, 116K data, 24K stack, 196K dynamic
    69632 heapsize, 61260 allocated, 6280 free
    2640K total, 2304K text, 116K data, 24K stack, 196K dynamic
    69632 heapsize, 61236 allocated, 6356 free
    2648K total, 2312K text, 116K data, 24K stack, 196K dynamic
    69632 heapsize, 61220 allocated, 6372 free
    3276K total, 2636K text, 156K data, 36K stack, 448K dynamic
    327680 heapsize, 268312 allocated, 55876 free
    4212K total, 3776K text, 176K data, 28K stack, 232K dynamic
    106496 heapsize, 95848 allocated, 7880 free
    2292K total, 1984K text, 96K data, 24K stack, 188K dynamic
    61440 heapsize, 53712 allocated, 5872 free
    3968K total, 3500K text, 152K data, 52K stack, 264K dynamic
    122880 heapsize, 114536 allocated, 4976 free
    3548K total, 3152K text, 120K data, 48K stack, 228K dynamic
    102400 heapsize, 94908 allocated, 4996 free
    4492K total, 3756K text, 200K data, 116K stack, 420K dynamic
    299008 heapsize, 286464 allocated, 7356 free
    4172K total, 3684K text, 168K data, 52K stack, 268K dynamic
    126976 heapsize, 118120 allocated, 5264 free
    16384 heapsize, 9188 allocated, 6008 free
    16384 heapsize, 7316 allocated, 8168 free
cumbdck54rs01#

[Richard Michael]

Hello Kelvin,

IOS PID 16408 seems to taking more chunks of memory, Although IOS-base seems to be a legitimate PID

somehow the acquired memory is not been released. Either it has to be a bug or this may be a transient issue which can work after a reboot of the switch

Can you share the below outputs,

1) show mem dead

2) show mem dead totals

3) show mem all totals

Also, share the trimmed version of the show run/sh log from the switch.

Thanks,

Ricky Micky

[cheung2009]

Hi Ricky

Please see the attachment. The switches has been reboot, but it does not help

[Richard Michael]

Hi Kelvin,

Thanks for the output, i have searched for bugs couldnt find anything matching my bad

I need to confirm if there is a memory leak happening in the device in order to find that out, i need to have the following command output,

show mem debug leak - All Memory leaks

show mem debug leaks chunks - Memory leaks would be shown at chunk level

show proc mem sorted - Top Guys utilizing the Memory

The debug commands that i mentioned above is CPU intensive. Once i get the debugs we can locate the PID that is doing this.

Lastly, Can you tell me if there is any recent changes done in the network after which you see the memory problem?

Thanks,

Ricky Micky

* Rate if this helps

[cheung2009]

Hi Ricky

Sorry for the late reply....The IOS does NOT have the commands you mentioned.

Please see the following command output, I hope it helps:

cumbdck54rs01#
cumbdck54rs01#show memory detailed all dead totals
System Memory: 524288K total, 474988K used, 49300K free, 1000K kernel reserved
Lowest(b)    : 46899200
Process sbin/ios-base, type IOS, PID = 16408
    352720K total, 64648K text, 32308K data, 84K stack, 255680K dynamic

                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)
Processor    9000000   261607480   259489472     2118008     2051552      524288
      I/O    5000000    33554432     4584944    28969488    28969488    28969464

Dead Proc Summary for: Processor

    PC          Total  Name
0x72236FB0        251  Virtual Exec
0x76334FB0         91  Virtual Exec
0x72367F08         77  draco-oir-process:slot 3
0x76A0DCFC         44  IP Sticky ARP subblock
0x767B10C8         44  IFINDEX hw subblock
0x74DA4C7C         44  Config Checkpointing Process
0x75578EF0         42  dai_hwsb_t
0x76052358         38  DSS-SB
0x7535DF68         37  QM-SB
0x769D5E94         36  IPC Name String
0x735B1B70         36  State Machine Instance
0x735B26C0         36  State Machine debug data
0x74CFE5A4         36  HWIF-QOS-SB
0x769AD3D8         31  IPC Port
0x72367F08         29  draco-oir-process:slot 4
0x72368E34         24  draco-oir-process:slot 3
0x74A24698         19  draco-oir-process:slot 3
0x74D813E4         15  IPC LC Port Opener
    PC          Total  Name
0x72367F08         11  draco-oir-process:slot 1
0x72367F08         11  draco-oir-process:slot 2
0x73067460          9  Config Checkpointing Process
0x7573C7F8          9  AAA MI SG NAME
0x73068788          9  Config Checkpointing Process
0x73067428          9  Config Checkpointing Process
0x75747F1C          8  AAA Server handle
0x74AECC9C          7  draco-oir-process:slot 1
0x7257B794          6  Virtual Exec
0x725877C8          5  Config Checkpointing Process
0x757AEC34          5  TTY timer block
0x73040684          4  Clock buffer
0x7257BD1C          4  Registry remot
0x763561DC          4  FIB: FIB LC
0x76356214          4  LCC Configure
0x76356228          4  LCC Configure
0x7635623C          4  LCC Configure
0x76356258          4  FIBLC failed msg
0x75146C44          4  PM port_data
0x769B2308          3  IPC Port Name
0x769AF99C          3  IPC Name
0x769AF95C          3  IPC Port
0x725A8F58          3  Config Checkpointing Process
0x722F93E8          3  AAA MLIST
    PC          Total  Name
0x75723D24          3  AAA Secrettype encrypt
0x74FC2558          2  draco-oir-process:slot 2
0x72588110          2  Config Checkpointing Process
0x74FC2558          2  draco-oir-process:slot 1
0x723BB4B4          2  draco-oir-process:slot 1
0x7554D954          2  add_port_list_to_port_hash
0x73F188EC          2  Virtual Exec
0x76334FB0          2  Config Checkpointing Process
0x74CE97E0          2  draco-oir-process:slot 2
0x769AAF34          2  IPC Port seq window
0x73FA30A8          2  small IML chun
0x723BB4B4          2  draco-oir-process:slot 2
0x72368E34          2  draco-oir-process:slot 2
0x72FE6A80          2  draco-oir-process:slot 2
0x72FE709C          2  draco-oir-process:slot 2
0x7256967C          2  draco-oir-process:slot 2
0x72368E34          2  draco-oir-process:slot 1
0x739F238C          2  TAC lp
0x75723CD4          2  AAA Secrettype
0x75723DC0          2  AAA_Secrettype pw
0x73EE1F50          2  draco-oir-process:slot 2
0x72569660          2  draco-oir-process:slot 2
0x72569690          2  draco-oir-process:slot 2
0x722F8CB0          2  AAA MLIST
    PC          Total  Name
0x75745F74          2  AAA SG HQ
0x722F9040          2  AAA MLIST
0x743EE568          1  draco-oir-process:slot 1
0x73EE1F50          1  IPC delayed init
0x72569690          1  IPC delayed init
0x72FE6A80          1  IPC delayed init
0x72569660          1  IPC delayed init
0x72FE709C          1  IPC delayed init
0x7256967C          1  IPC delayed init
0x7572D528          1  AAA Unique Id Hash Table
0x72FE894C          1  IPC delayed init
0x769BACBC          1  IPC Seat
0x769D52E4          1  IPC Thread
0x769D5320          1  IPC Thread Slot
0x74FC2558          1  draco-oir-process:slot 4
0x74FC2558          1  draco-oir-process:slot 3
0x743EE428          1  draco-oir-process:slot 2
0x7257B794          1  Config Checkpointing Process
0x72D27C78          1  IPv6 CEF default table
0x767B9578          1  draco-oir-process:slot 3
0x767B95B8          1  draco-oir-process:slot 3
0x767C6968          1  draco-oir-process:slot 3
0x74CE97E0          1  draco-oir-process:slot 3
0x743EE428          1  draco-oir-process:slot 1
    PC          Total  Name
0x74FC7C60          1  draco-oir-process:slot 1
0x753BF188          1  LCC regstration
0x74A24698          1  eobc_init_process
0x723B3728          1  eobc_init_process
0x73F13C7C          1  Config Checkpointing Process
0x7257FA44          1  registry serve
0x743EE518          1  draco-oir-process:slot 1
0x743EE4C8          1  draco-oir-process:slot 1
0x74ADF728          1  draco-oir-process:slot 3
0x73A0ECB4          1  Trange IPC small chunk
0x743EE340          1  draco-oir-process:slot 2
0x743EE5B8          1  draco-oir-process:slot 2
0x743EE518          1  draco-oir-process:slot 2
0x743EE568          1  draco-oir-process:slot 2
0x74ADF728          1  draco-oir-process:slot 1
0x73BDD0F0          1  SNMP MIB Bulk Sync
0x76334FB0          1  RF Send Bulk Sync
0x743EE608          1  draco-oir-process:slot 1
0x743EE5B8          1  draco-oir-process:slot 1
0x72D27C78          1  IPv6 CEF global table list
0x743EE518          1  draco-oir-process:slot 4
0x743EE4C8          1  draco-oir-process:slot 4
0x743EE478          1  draco-oir-process:slot 1
0x72D27C78          1  IPv6 CEF tree root
    PC          Total  Name
0x743EE340          1  draco-oir-process:slot 1
0x767B9578          1  draco-oir-process:slot 4
0x767B95B8          1  draco-oir-process:slot 4
0x767C6968          1  draco-oir-process:slot 4
0x743EE4C8          1  draco-oir-process:slot 3
0x74CE97E0          1  draco-oir-process:slot 4
0x72FE6A80          1  draco-oir-process:slot 3
0x743EE608          1  draco-oir-process:slot 2
0x743EE518          1  draco-oir-process:slot 3
0x723BB4B4          1  draco-oir-process:slot 3
0x73EE1F50          1  draco-oir-process:slot 3
0x72FE709C          1  draco-oir-process:slot 3
0x72569660          1  draco-oir-process:slot 3
0x7256967C          1  draco-oir-process:slot 3
0x72569690          1  draco-oir-process:slot 3
0x763A0A28          1  FM_VMRS
0x7639FF94          1  FM_LABEL_INFO
0x74AE80A8          1  FM_VMR_LIST
0x743EE4C8          1  draco-oir-process:slot 2
0x743EE478          1  draco-oir-process:slot 2
0x72FE6A80          1  draco-oir-process:slot 4
0x723BB4B4          1  draco-oir-process:slot 4
0x72FE709C          1  draco-oir-process:slot 4
0x7256967C          1  draco-oir-process:slot 4
    PC          Total  Name
0x73EE1F50          1  draco-oir-process:slot 4
0x72569660          1  draco-oir-process:slot 4
0x72569690          1  draco-oir-process:slot 4
0x72225ACC          1  *Init*
0x74ADF728          1  draco-oir-process:slot 4
0x74ADF728          1  draco-oir-process:slot 2
0x7572D624          1  AAA DB Chunk
0x74CDF094          1  Virtual Exec
0x75795934          1  AAA_Secrettype pw
0x7454C2AC          1  Parseinfo Bloc
0x762BCDD4          1  Virtual Exec
0x73AFD284          1  CCA CCB chunks
0x73AFD2FC          1  CCA CLA chunks
0x7304F0A8          1  Config Checkpointing Process
0x7304F3B4          1  Config Checkpointing Process
0x7452E01C          1  Virtual Exec

Dead Proc Summary for: I/O

    PC          Total  Name

Process sbin/iprouting.iosproc, type IOS, PID = 16427

                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)
Processor    1040058     4522564     2780004     1742560     1511456     1729376

Dead Proc Summary for: Processor

    PC          Total  Name
0x728EAC78        175  Init
0x7257C0B0         95  Init
0x7412819C         38  IPRTG SWSB
0x728FCF50         38  Init
0x728FCE5C         38  Init
0x73EF9F18         23  Init
0x73EF9EFC         23  Init
0x72576C44         17  Init
0x7257FA44         13  registry serve
0x73F3D09C         13  Init
0x73F69A2C          7  List Elements
0x73F188EC          5  Init
0x7292AE2C          4  Init
0x7292BDB4          4  Init
0x728F3F04          4  Init
0x728EAB08          4  Init
0x73EEB5EC          4  Init
0x7292BDD0          4  Init
    PC          Total  Name
0x7292BDE8          4  Init
0x7257C120          4  Reg Function C
0x73F69A2C          3  IDB List Eleme
0x72562ED0          3  Init
0x72562EF0          3  Init
0x72562F24          3  Init
0x73EF9334          3  Init
0x73EF93A8          3  Init
0x7257B794          2  Init
0x75B64048          2  Init
0x728F2668          2  Init
0x73F188EC          2  Router Init
0x7257C0B0          2  SNMP MIB register
0x728E5070          1  Init
0x73F9A310          1  Alignment Data
0x73F69A2C          1  IDB List Element Chunks
0x73F9CCD4          1  Packet Elements
0x728DC4A8          1  IP Address Caches
0x72588110          1  Init
0x728F25D4          1  Init
0x728FDC00          1  Init
0x728FDC1C          1  Init
0x728FDC38          1  Init
0x73F3D164          1  hwsb list0 hwsb list0 RBTree
    PC          Total  Name
0x73F3D164          1  hw idb db hw idb db RBTree
0x728FDD90          1  Init
0x728FDDAC          1  Init
0x728FDDC8          1  Init
0x73F3D164          1  swsb list0 swsb list0 RBTree
0x73F3D164          1  swsb list1 swsb list1 RBTree
0x73F3D164          1  swsb list2 swsb list2 RBTree
0x73F3D164          1  Init
0x73F3D164          1  swsb list4 swsb list4 RBTree
0x73F3D164          1  swsb list5 swsb list5 RBTree
0x73F3D164          1  swsb list6 swsb list6 RBTree
0x73F3D164          1  swsb list7 swsb list7 RBTree
0x73F3D164          1  swsb list8 swsb list8 RBTree
0x73F3D164          1  swsb list9 swsb list9 RBTree
0x73F3D164          1  sw idb db sw idb db RBTree
0x72582E9C          1  Init
0x74128D9C          1  IP RDB Chunk
0x7412F98C          1  IP single NDB entry
0x7412F9C0          1  IP subnet NDB entry
0x7412F9F4          1  NET REDIST
0x7412F81C          1  IP: Cblk-table
0x7412F4D4          1  IP: Control Block
0x7412F4F4          1  Init
0x7412FAE4          1  Init
    PC          Total  Name
0x7412FB10          1  Init
0x7412FBC0          1  IP PDB
0x7412FC4C          1  IP PDB
0x740C74C0          1  Init
0x740BECF4          1  CLNS adjacency database
0x725766D0          1  Reg Function 1
0x740C1034          1  CLNS Static PDB
0x740C1110          1  CLNS ESIS PDB
0x740C0424          1  Init
0x75FE7A38          1  Handle Mgmt Nodes
0x74157F10          1  Init
0x74157C58          1  Init
0x74157838          1  Init
0x72225ACC          1  *Init*
0x7292AD70          1  Init
0x73FA1D74          1  small IML chun
0x72576CF8          1  Reg Function 1
0x73CDB9B4          1  Init
0x73EE1F50          1  Init
0x7257C1F8          1  Reg Function C
0x728F0828          1  Init
0x7414A258          1  Router Init

Process sbin/cdp2.iosproc, type IOS, PID = 16428

                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)
Processor    1040058      426904      419920        6984        6984        1132

Dead Proc Summary for: Processor

    PC          Total  Name
0x73EF9F18         11  Init
0x73EF9EFC         11  Init
0x728EAC78          5  Init
0x73F188EC          4  Init
0x73EF9334          3  Init
0x73EF93A8          3  Init
0x7292AE2C          2  Init
0x7292BDB4          2  Init
0x7257C0B0          2  SNMP MIB register
0x72576C44          2  Init
0x73F3D09C          2  Init
0x7257FA44          2  registry serve
0x728E4E9C          2  Init
0x728F3F04          2  Init
0x73EEB5EC          2  Init
0x7292BDD0          2  Init
0x7292BDE8          2  Init
0x728E5070          1  Init
    PC          Total  Name
0x7257B794          1  Init
0x73F9A310          1  Alignment Data
0x73F69A2C          1  IDB List Element Chunks
0x73F9CCD4          1  Packet Elements
0x728FDD90          1  Init
0x728FDDAC          1  Init
0x728FDDC8          1  Init
0x73F3D164          1  swsb list0 swsb list0 RBTree
0x73F3D164          1  sw idb db sw idb db RBTree
0x73F69A2C          1  List Elements
0x72630088          1  Init
0x728F25D4          1  Init
0x72225ACC          1  *Init*
0x7292AD70          1  Init
0x73FA1D74          1  small IML chun
0x728F0828          1  Init
0x728F2668          1  Init

cumbdck54rs01#

==================================

Re: running constantly at 89% Memory Untilization

[rickyoung]

Hi

Please see the "show logg" below, the SUP card switched over last night:

000007: .Aug 12 02:50:31.877: %PFREDUN-SP-STDBY-STDBY-6-ACTIVE: Initializing as ACTIVE processor
000008: .Aug 12 02:50:32.145: %SYS-SP-STDBY-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console

debugging output.
000009: .Aug 12 02:50:32.425: %PF_ASIC-DFC4-3-ASIC_DUMP: [1:0x17E] ME_AR_KIF_TX_TX_STAT = 0x0
000010: .Aug 12 02:50:32.425: %PF_ASIC-DFC4-3-ASIC_DUMP: [1:0x1E5] ME_AR_P1MMU_FREE_PAGE_CNT = 0x5F8
000011: .Aug 12 02:50:32.425: %PF_ASIC-DFC4-3-ASIC_DUMP: [1:0x1E6] ME_AR_P1MMU_FREE_HEAD = 0x10
000012: .Aug 12 02:50:32.425: %PF_ASIC-DFC4-3-ASIC_DUMP: [1:0x1E7] ME_AR_P1MMU_FREE_TAIL = 0x7
000013: .Aug 12 02:50:32.425: %PF_ASIC-DFC4-3-ASIC_DUMP: [1:0x20A] ME_AR_P2MMU_FREE_PAGE_CNT = 0x5B8
000014: .Aug 12 02:50:32.425: %PF_ASIC-DFC4-3-ASIC_DUMP: [1:0x20B] ME_AR_P2MMU_FREE_HEAD = 0x6C
000015: .Aug 12 02:50:32.425: %PF_ASIC-DFC4-3-ASIC_DUMP: [1:0x20C] ME_AR_P2MMU_FREE_TAIL = 0x47
000016: .Aug 12 02:50:32.873: %FABRIC-SP-STDBY-5-FABRIC_MODULE_ACTIVE: The Switch Fabric Module in slot 2 became

active.
000017: .Aug 12 02:50:33.097: %C6KPWR-SP-4-PSOK: power supply 1 turned on.
000018: .Aug 12 02:50:33.097: %C6KPWR-SP-4-PSOK: power supply 2 turned on.
000019: .Aug 12 02:50:33.097: %C6KENV-SP-4-FANHPMODE: Fan-tray 1 is operating in high power mode
000020: .Aug 12 02:50:34.097: %OIR-SP-6-DOWNGRADE_EARL: Module 2 PFC installed is not identical to system PFC and

will perform at cu
rrent system operating mode.
000021: .Aug 12 02:50:34.097: %OIR-SP-6-DOWNGRADE_EARL: Module 4 DFC installed is not identical to system PFC and

will perform at cu
rrent system operating mode.
000022: .Aug 12 02:50:34.097: %OIR-SP-6-INSCARD: Card inserted in slot 2, interfaces are now online
000023: .Aug 12 02:50:34.097: %OIR-SP-6-INSCARD: Card inserted in slot 3, interfaces are now online
000024: .Aug 12 02:50:34.097: %OIR-SP-6-INSCARD: Card inserted in slot 4, interfaces are now online
000025: .Aug 12 02:50:34.000: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.4.8 on GigabitEthernet3/2 from LOADING to

FULL, Loading Done
000026: .Aug 12 02:50:34.000: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.4.7 on GigabitEthernet3/12 from LOADING to

FULL, Loading Done
000027: .Aug 12 02:50:34.000: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.4.5 on Port-channel5 from LOADING to FULL,

Loading Done
000028: .Aug 12 02:50:37.000: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.4.1 on Port-channel21 from LOADING to FULL,

Loading Done
000029: .Aug 12 02:50:40.000: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.4.17 on Port-channel4 from LOADING to FULL,

Loading Done
000030: .Aug 12 02:50:40.000: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.4.16 on Port-channel3 from LOADING to FULL,

Loading Done
000031: .Aug 12 02:50:41.000: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.4.19 on Port-channel100 from LOADING to FULL,

Loading Done
000032: .Aug 12 02:50:43.000: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.4.18 on Port-channel20 from LOADING to FULL,

Loading Done
000033: .Aug 12 02:50:47.000: %BGP-5-ADJCHANGE: neighbor 10.22.4.5 Up
000034: .Aug 12 02:51:11.000: %BGP-5-ADJCHANGE: neighbor 10.20.0.142 Up
000035: Aug 12 02:53:55.658: %PFREDUN-SP-6-ACTIVE: Standby initializing for SSO mode
000036: Aug 12 02:53:56.666: %SYS-SP-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging

output.
000037: Aug 12 02:54:03.659: %PFINIT-SP-5-CONFIG_SYNC: Sync'ing the startup configuration to the standby Router.
000038: Aug 12 02:54:51.456: %FABRIC-SP-5-CLEAR_BLOCK: Clear block option is off for the fabric in slot 1.
000039: Aug 12 02:54:51.456: %FABRIC-SP-5-FABRIC_MODULE_BACKUP: The Switch Fabric Module in slot 1 became standby
000040: Aug 12 02:54:53.080: %DIAG-SP-6-RUN_MINIMUM: Module 1: Running Minimal Diagnostics...
000041: Aug 12 02:54:55.732: %DIAG-SP-6-DIAG_OK: Module 1: Passed Online Diagnostics
000042: Aug 12 02:54:55.732: %OIR-SP-6-DOWNGRADE_EARL: Module 1 PFC installed is not identical to system PFC and

will perform at cur
rent system operating mode.
000043: Aug 12 02:54:56.936: %OIR-SP-6-INSCARD: Card inserted in slot 1, interfaces are now online
000044: Aug 12 02:55:08.648: %SYS-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging

output.
000045: Aug 12 02:55:08.648: %OIR-SP-STDBY-6-CONSOLE: Changing console ownership to route processor
000046: Aug 12 02:55:08.648: %SYS-SP-STDBY-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console

debugging output.
000047: Aug 12 02:55:08.648: %PFREDUN-SP-STDBY-6-STANDBY: Initializing for SSO mode
000048: Aug 12 02:55:08.652: %SYS-SP-STDBY-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console

debugging output.
000049: Aug 12 02:55:08.652: %SPANTREE-SP-STDBY-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
000050: Aug 12 02:55:08.652: %DIAG-SP-STDBY-6-RUN_MINIMUM: Module 1: Running Minimal Diagnostics...
000051: Aug 12 02:55:08.652: %DIAG-SP-STDBY-6-DIAG_OK: Module 1: Passed Online Diagnostics
000052: Aug 12 02:55:08.656: %SYS-SP-STDBY-5-RESTART: System restarted --
000053: Aug 12 02:55:08.656: %PFREDUN-SP-STDBY-6-STANDBY: Ready for SSO mode
000054: Aug 12 02:55:09.508: %SYS-SP-STDBY-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console

debugging output.
000055: Aug 12 02:55:52.505: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 4/0: Window did not open, LC to RP IPC is

non-operational
cumbdck54rs02#sh cdp nei
cumbdck54rs02#sh cdp neighbors
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
cumbdcd54rt07.hss.health.nsw.gov.au
                 Gig 3/12           132         R S I     CISCO3925-Gig 0/2
cumbdcd09rs19.hss.health.nsw.gov.au
                 Gig 3/19           147         R S I     WS-C3750G-Gig 1/0/50
cumbdcg54rt05.hss.health.nsw.gov.au
                 Gig 3/15           156           R       ASR1006   Gig 1/1/7
cumbdcg54rt05.hss.health.nsw.gov.au
                 Gig 3/5            153           R       ASR1006   Gig 0/1/7
cumbdck54rs17.hss.health.nsw.gov.au
                 Gig 3/18           177         R S I     WS-C6509-EGig 6/2
cumbdck54rs17.hss.health.nsw.gov.au
                 Gig 3/8            177         R S I     WS-C6509-EGig 5/2
cumbdci54rs16.hss.health.nsw.gov.au
                 Gig 3/17           158         R S I     WS-C6509-EGig 6/2
cumbdci54rs16.hss.health.nsw.gov.au
                 Gig 3/7            158         R S I     WS-C6509-EGig 5/2
cumbdck54rs01.hss.health.nsw.gov.au
                 Gig 3/14           142         R S I     WS-C6504-EGig 3/14
cumbdck54rs01.hss.health.nsw.gov.au
                 Gig 3/13           142         R S I     WS-C6504-EGig 3/13
cumbdck54rs01.hss.health.nsw.gov.au
         
Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
                 Gig 3/4            141         R S I     WS-C6504-EGig 3/4
cumbdck54rs01.hss.health.nsw.gov.au
                 Gig 3/3            135         R S I     WS-C6504-EGig 3/3
cumbdcd54rt08.hss.health.nsw.gov.au
                 Gig 3/2            142         R S I     3845      Gig 0/1
livedcc40rs02    Gig 3/24           149         R S I     WS-C6504-EGig 3/24
cumbdcd18rs19    Gig 3/16           121         R S I     WS-C6509  Gig 6/2
cumbdcd18rs19    Gig 3/11           121         R S I     WS-C6509  Gig 5/2
cumbdcd18rs18    Gig 3/10           160         R S I     WS-C6509-EGig 6/2

cumbdcd18rs18    Gig 3/6            155         R S I     WS-C6509-EGig 5/2
cumbdck54rs02#

[manju.cisco]

Hi Kelvin,

I am not aware on this issue, but i would like to say something.

Read somewhere that CEF might also be once cause, check if CEF is working properly (command show cef linecard detail ) and if required restart cef on line cards (command something like clear cef ----- )

We can see the utilization of individual linecard and determine which linecard is using much CPU/memory, i dont know about the exact commands; command like execute-on cardnumber show proc cpu ........something like that., you may try it....

This might help to give some clue on this problem.......

Pls try to figure out those commands (also post for us for our learning too including output ).

Thanks,

Manju

[Leo Laohoo]

SXF is pretty old IOS.  You sure you don't want to try upgrading to SXH5, SXJ or 15.X?

[cheung2009]

I upgraded the IOS to "s72033-ipservicesk9_wan-mz.122-33.SXH2.bin". The issues did not occur again.

Thanks for those who offered help........................

Kelvin Cheung