We have been experiencing a steady memory leak on our Cat6000 with Sup1/MSFC running hybrid CatOS/IOS. We suspect the issue is related to using the Sup1A as a VMPS server for the LAN, as the memory leak has worsened as the size of our vmps.cfg file grows (currently about 1100 lines).
The leak can be seen as an increase in the amount of memory used by "Dot1x_Dhcp" as shown by a 'show proc mem' command. Below is the output after switch has been running about 24 hours. When the Free memory drops to about 11,000,000, the switch will no longer service logins at the console or respond to vmps client switches, resulting in clients losing network connectivity. (when this occurs, Dot1xDhcp is using 20Mb or more). The leak is now so severe that a reboot (or forced failover to 2nd Sup) is required every 4-5 weeks.
We have a C6509 with Sup2/MSFC2 waiting to replace this switch, but the lack of VMPS support in native IOS is a problem - we do not have the resources to switch to dot1x at this time, as it would require replacement of about 40 edge switches (2900/3500XL) that don't support dot1x. We're hoping for a solution to this problem that will allow us to keep this switch in service another 4-6 months while we plan for dot1x and native IOS migration. Any suggestions appreciated.
Memory Used: 15141408
Free: 26421056
Total: 41562464
PID TTY Allocated Freed Holding Process
---------- ---------- ---------- ---------- ---------- ---------------
1 -2 13286272 3258112 10028160 Kernel and Idle
2 -2 192 0 192 Flash MIB Updat
3 -2 192 0 192 L2IntHdlr
4 -2 0 0 0 EobRecoveryProc
5 -2 762304 704608 57696 SynDiags
6 -2 128 0 128 GenMsgHndlr
7 -2 2169024 1065312 1103712 SynConfig
8 -2 64 0 64 TempMon
9 -2 0 0 0 System_CrashInf
10 -2 3456 0 3456 IDLE_Tasks
11 -2 320 128 192 PowerMgmt
12 -2 15716416 15716064 352 Debug Port Coun
13 -2 137568 37088 100480 SL_TASK
14 -2 1115328 185280 930048 RedundantTask
15 -2 12073728 12071232 2496 Status Poll
16 -2 64 0 64 SWPoll64bCnt
18 -2 43904 42208 1696 VMPS_Task
19 0 971904 970368 1536 HavailTask
20 -2 4224 0 4224 SyncTask
21 -2 338400 338368 32 Earl
22 -2 16608 0 16608 TablefullTask
23 -2 1109408 1101952 7456 DTP_Rx
24 -2 489440 311808 177632 EthChnlRx
25 -2 0 0 0 llcSSTPFlood
26 -2 692800 251168 441632 EthChnlConfig
27 -2 20256 18336 1920 FlexlinkRx
28 -2 102560 0 102560 SecurityRx
29 -2 55328 54016 1312 ACL
30 -2 136224 108416 27808 DhcpSnooping
31 -2 48128 28512 19616 Authenticator_S
32 -2 32 0 32 Backend_SM
33 -2 160 0 160 dot1x_rx
34 -2 775648 0 775648 Dot1x_Dhcp
35 -2 160 0 160 GrpManager
36 -2 32 0 32 Backend_Rx
37 -2 32 0 32 Radius Acct Pro
38 -2 981280 976896 4384 NetFlow
39 -2 0 0 0 L3Aging
40 -2 64 0 64 MlsHa
41 0 190464 188608 1856 SptTimer
42 0 6368 6208 160 SptBpduRx
43 0 160 0 160 SptBpduTx
44 0 0 0 0 sptTraps
45 -2 160 0 160 GL2Prot_Tunnel
46 -2 224 0 224 VtpTimer
47 0 36928 0 36928 Read Stats Task
48 0 64 0 64 QDE Task
49 0 64 0 64 BcastSuppr Task
50 -2 448 192 256 oam_tx_proc
51 -2 160 0 160 oam_rx_proc
52 -2 23282848 23282688 160 oam_monitoring_
53 -2 524352 0 524352 NvramMonitor
54 -2 160 0 160 DeviceLinkChk
55 -2 1504 0 1504 gentcp_act
56 -2 974432 948320 26112 McastRx
57 -2 185504 132000 53504 RouterConfig
58 -2 13940288 13929856 10432 cdpd
59 -2 32721568 32721344 224 cdpdtimer
60 -2 0 0 0 sptTraps
61 -2 224 0 224 ProtocolFilter
62 -2 544 0 544 MacMoveNotTask
63 -2 512 480 32 telnetd
64 -2 32 0 32 connecthelper
65 -2 32 0 32 tftpd
66 -2 6559872 6559776 96 ProtocolTimer
67 -2 64 0 64 ciscoRmonTimer
68 -2 64 0 64 ciscoUsrHistory
69 -2 96 0 96 rmonMediaIndep
70 -2 0 0 0 SnmpTraps
71 -2 0 0 0 memPoolMain
73 -2 64 0 64 RMON AlarmTimer
76 -2 32 0 32 Acct Send Bkg
77 -2 128 0 128 l2t_server
79 -2 32 0 32 SysLogTask
80 -2 160 0 160 pinggateA
81 -2 96 0 96 MacAuthProc
82 -2 96 0 96 AuthControlMgr
83 -2 32 0 32 McastIgmpTimer
84 -2 32 0 32 IGMPQuerierProc
85 -2 96 0 96 M-MLS_stats
86 -2 4416 64 4352 M-MLS_manager
87 -2 67232 15840 51392 QoSTask
88 -2 7840 7776 64 GBIC Process
89 -2 32 0 32 HPActive
90 -2 96 0 96 HPTrapMgr
143 -2 288 0 288 VtpRx
144 0 112352 57696 54656 Console
145 -2 100227136 99821664 405472 snmpdm
146 -2 19637440 19636640 800 SNMP_THREAD
147 2241991444 97056 33568 63488 telnet147
156 -2 2101504 2010944 90560 AclManager
Memory Pool Utilization
Memory Pool Type 1Min 5Min 10Min
---------------- ----- ----- -----
DRAM 55% 55% 55%
FLASH 87% 87% 87%
NVRAM 67% 67% 67%
MBUF 4% 4% 4%
CLUSTER 9% 9% 9%
MALLOC 36% 36% 36%
show ver:
WS-C6006 Software, Version NmpSW: 8.5(1)
Copyright (c) 1995-2005 by Cisco Systems
NMP S/W compiled on Oct 22 2005, 11:28:54
System Bootstrap Version: 5.3(1)
System Boot Image File is 'bootflash:cat6000-supk8.8-5-1.bin'
System Configuration register is 0x2
Hardware Version: 2.0 Model: WS-C6006 Serial #: TBA04290439
PS1 Module: WS-CAC-1300W Serial #: SNI06120229
PS2 Module: WS-CAC-1300W Serial #: ACP03280942
Mod Port Model Serial # Versions
--- ---- ------------------- ----------- --------------------------------------
1 2 WS-X6K-SUP1A-2GE SAD04260GEA Hw : 3.2
Fw : 5.3(1)
Fw1: 5.1(1)CSX
Sw : 8.5(1)
Sw1: 8.5(1)
WS-F6K-PFC SAD041701CC Hw : 1.1
Sw :
2 2 WS-X6K-SUP1A-2GE SAD04510V1L Hw : 7.0
Fw : 5.3(1)
Fw1: 5.4(2)
Sw : 8.5(1)
Sw1: 8.5(1)
WS-F6K-PFC SAD04510GPE Hw : 1.1
Sw :
3 8 WS-X6408A-GBIC SAD0429002D Hw : 1.3
Fw : 0.0
Sw : 8.5(1)
4 24 WS-X6224-100FX-MT SAD03310103 Hw : 2.0
Fw : 0.0
Sw : 8.5(1)
5 16 WS-X6516-GE-TX SAL08517EJK Hw : 2.6
Fw : 0.0
Sw : 8.5(1)
6 16 WS-X6316-GE-TX SAD0732056W Hw : 1.3
Fw : 0.0
Sw : 8.5(1)
15 1 WS-F6K-MSFC SAD04260JH6 Hw : 1.4
Fw : 12.1(26)E4
Sw : 12.1(26)E4
16 1 WS-F6K-MSFC SAD04030BBG Hw : 1.4
Fw : 12.1(26)E4
Sw : 12.1(26)E4
DRAM FLASH NVRAM
Module Total Used Free Total Used Free Total Used Free
------ ------- ------- ------- ------- ------- ------- ----- ----- -----
1 130944K 73308K 57636K 15616K 13589K 2027K 512K 346K 166K
2 130944K
Uptime is 0 day, 22 hours, 37 minutes
