3750X high memory and fragmentation

[waqas gondal]

Hi, 

Our switch is at 93% memory utilisation and CPU has regularly gone above 80%. it is preventing me from using remote logon to the switch.

I am getting this error message on my 3750X:

%Error on initialize VLAN database 1: VTP feature not yet initialized
108704495: Feb 15 16:02:52.290 pst: %SYS-2-MALLOCFAIL: Memory allocation of 989120 bytes failed from 0x25F9A74, alignment 0
Pool: Processor Free: 3164264 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "Exec", ipl= 0, pid= 195
-Traceback= 6FF140z 2764C68z 276C98Cz 35072C4z 25F9A78z 25FC32Cz 2A59C0Cz 2A59CECz 64CA94z 6473B0z 70AC38z 70DEFCz 70DF8Cz 6AB1E8z 645764z 64BABCz

This is the mem usage:

TER-S3-FVOPS-1E1#sh proc mem sorted
Processor Pool Total: 156305684 Used: 152193700 Free: 4111984
I/O Pool Total: 16777216 Used: 13279396 Free: 3497820
Driver te Pool Total: 4194304 Used: 40 Free: 4194264

PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 180764852 101069312 73138604 0 0 *Init*
309 0 3895862356 3301004848 60213116 24454224 0 Auth Manager
310 0 4209997356 3645424100 9368920 0 0 EPM MAIN PROCESS
0 0 178463204 803088616 3767028 14303535 1490354 *Dead*
93 0 5302944 1413364 2720856 34344 0 Stack Mgr Notifi
436 0 1269388 17256 1262300 0 0 EEM Server
0 0 0 0 985092 0 0 *MallocLite*
384 0 63108944 61097384 702944 10152 0 hulc running con
287 0 695532 1916 683460 0 0 IPC LC Message H
1 0 2594613476 2594059700 637320 0 0 Chunk Manager
474 0 429756012 4105509220 458728 6099660 1692 LLDP Protocol
65 0 369808 6964 372916 0 0 EEM ED Identity
30 0 391664 0 362236 0 0 IPC Seat RX Cont
201 0 2655502184 2159448684 308676 98408412 0 HQM Stack Proces
211 0 336704 51880 305032 0 0 HL2MCM
210 0 306224 31608 295188 0 0 HL2MCM
421 0 21654784 21378176 286776 100548 0 EEM ED Syslog
329 0 267752 0 262492 0 0 CEF RP Backgroun
PID TTY Allocated Freed Holding Getbufs Retbufs Process
419 0 207792 4964 212996 67468 0 EEM ED Generic
227 0 2930930804 3910739464 189968 2268 0 EAP Framework
220 0 3552660792 2056586356 179432 5812020 0 CDP Protocol
18 0 194520 258084 179036 0 40300 Entity MIB API
177 0 199736 0 159368 5076 0 HRPC stp_state_s
392 0 635112 2436 157660 199908 0 Crypto IKEv2
250 0 401472 243224 152248 0 0 mDNS
47 0 436361292 100399872 140832 99900 0 Net Background
109 0 3018092 5364 119824 1715688 0 hulc cfg mgr mas
484 0 359809224 538858188 104272 5223960 17965476 RADIUS
202 0 978445304 980689348 104120 340092 0 HRPC qos request
215 0 1890370812 1132744544 93340 1958364 0 AAA ACCT Proc
53 0 287208 243452 90968 0 0 IF-MGR control p
132 0 1717415224 1582822752 87208 552744 0 HRPC pm request
68 0 151904 22176 86560 0 0 USB Startup
387 0 436028 363560 86416 0 0 Crypto CA
477 0 71532 0 84760 0 0 IP SLAs Responde
49 0 899591860 105844 84372 513339264 39312 Logger
476 0 69884 0 80080 0 0 IP SLAs Control
277 0 71336 0 78504 0 0 Switch Backup In
313 0 99308 1136 75156 0 0 CMD HANDLER
169 0 65812 232 72800 0 0 REP Helper Proc

This is the dead processes:

TER-S3-FVOPS-1E1#show mem dead totals
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 5C34D00 156305684 152262308 4043376 2884536 945904
I/O E000000 16777216 13279716 3497500 358264 1630796
Driver te 3800000 4194304 44 4194260 4194260 4194260

Dead Proc Summary for: Processor

PC Total Count Name
0x02D03BA0 3024044 1 hulc_flash init
0x02CAE51C 222152 1 Exec
0x02D21334 100220 5 mifs sector buffers
0x0131C094 47060 5 IPC Port
0x0064D860 45920 4 SSH Process
0x032FCA04 35100 1 XOS_MEM_BASE
0x00095A90 32812 1 ACL IPC large chunk
0x00603C30 32060 5 AAA Server handle
0x01D13470 26932 7 OBFL UPTIME obfl0
0x02D240C0 25220 5 mifs cache_entry buffers
0x02D1B5E0 20044 1 mifs NOR Sector Info Pointers
0x02734CA8 20028 12 switchport
0x02F860AC 19224 272 LIM Index
0x003D2DB0 16960 13 hulc_flash init
0x02733DF4 12564 2 ID MGR
0x00EC4CF0 10004 1 EEM Shell Director
0x01366338 9924 8 IPC Port
0x000CB0C4 8092 1 AAA LOCAL SVR
PC Total Count Name
0x00095A48 5044 1 ACL IPC medium chunk
0x02734FD4 3044 1 CCA CCB chunks
0x0350AA24 2816 8 platform OBFL
0x02D21A54 2780 5 hulc_flash init
0x01CF7394 2724 9 OBFL TEMP obfl0
0x02D21400 2720 5 mifs file_map buffers
0x01D1A5BC 2200 1 OBFL POE obfl0
0x01D1A0F4 2156 1 platform OBFL
0x02D21398 2040 5 mifs file_info buffers
0x006013F8 1900 8 AAA SG HQ
0x00095A00 1828 1 ACL IPC small chunk
0x02733F8C 1324 11 SDB String
0x02D171C0 1312 19 hulc_flash init
0x02D17180 1308 19 hulc_flash init
0x02D1716C 1292 19 hulc_flash init
0x0278BC34 1228 1 Watched Semapho
0x02E48FA0 1108 4 RTTY Client Registration
0x00597D48 1064 12 AAA_Secrettype pw
0x02D23FF0 1020 15 hulc_flash init
0x02733EB8 996 11 Template Block
0x02D21A28 980 5 hulc_flash init
0x02733EE4 928 11 Template Name
0x00597CA4 880 10 AAA Secrettype
PC Total Count Name
0x02D07944 812 1 hulc_flash init
0x00669CC8 812 1 SSH Process
0x0129263C 792 9 hulc_flash init
0x01292664 792 9 hulc_flash init
0x01CFE404 772 1 platform OBFL
0x02621E30 704 4 TAC lp
0x01CFBD80 692 1 platform OBFL
0x02D03B2C 580 1 hulc_flash init
0x01D28400 524 1 platform OBFL
0x01D222B4 516 1 platform OBFL
0x01D1CFC8 508 1 platform OBFL
0x01D2D5CC 508 1 platform OBFL
0x02D1EC98 460 5 hulc_flash init
0x01D2A490 428 1 platform OBFL
0x0061D7A8 368 4 AAA NV SECRET
0x02D242DC 340 5 hulc_flash init
0x0060367C 336 4 AAA SG Server name
0x0008FE54 308 1 platform OBFL
0x0009032C 308 1 platform OBFL
0x000901D4 308 1 platform OBFL
0x00090064 308 1 platform OBFL
0x00090938 308 1 platform OBFL
0x00090780 308 1 platform OBFL
PC Total Count Name
0x00090618 308 1 platform OBFL
0x00600A40 288 1 SSH Process
0x00597D34 284 4 AAA Secrettype encrypt
0x02D1EE64 272 4 hulc_flash init
0x011F3DDC 232 1 LICENSE AGENT DEFAULT
0x032FC8CC 232 1 XOS_MEM_BASE
0x02D1B620 216 2 hulc_flash init
0x01D193C4 208 1 OBFL Infra Device info
0x02D1BA4C 160 2 hulc_flash init
0x02D1ED10 160 2 hulc_flash init
0x01247390 140 1 HTTP
0x0125E544 136 1 HTTP
0x01F8ED4C 136 1 QOS policymap name
0x005F9A94 132 1 User Passwd in csb
0x0123BCE0 128 1 HTTP
0x02D07908 112 1 hulc_flash init
0x00664DCC 104 1 Parser Mode
0x0123BA18 84 1 HTTP
0x02D03BB0 80 1 hulc_flash init
0x02D07954 80 1 hulc_flash init
0x0064439C 76 1 Parser Linkage
0x0131C158 72 1 IPC Name
0x02D16FF8 68 1 hulc_flash init
PC Total Count Name
0x02D17018 68 1 hulc_flash init
0x02D1704C 68 1 hulc_flash init
0x005E1F64 68 1 AAA MI SG NAME
0x03388A68 68 1 XOS_MEM_XDT
0x00664E44 68 1 Parser Mode Q1
0x00664E74 68 1 Parser Mode Q2
0x00EC4D88 68 1 EEM Shell Director

Any ideas what could be causing this and what I should turn off?

We do have 802.1x authentication with ISE on all the physical ports. Example port config:

interface GigabitEthernet1/0/4
description user port
switchport access vlan 232
switchport mode access
switchport voice vlan 332
ip access-group ACL-ALLOW in
srr-queue bandwidth share 10 10 60 20
priority-queue out
authentication event fail action next-method
authentication event server dead action authorize
authentication event server dead action authorize voice
authentication event server alive action reinitialize
authentication host-mode multi-domain
authentication open
authentication order dot1x mab
authentication priority dot1x mab
authentication port-control auto
authentication periodic
authentication timer reauthenticate server
authentication timer inactivity 30
authentication violation replace
mab
dot1x pae authenticator
dot1x timeout tx-period 8
spanning-tree portfast edge
spanning-tree bpduguard enable

Extended IP access list ACL-ALLOW
10 permit ip any any

Thanks,

Waqas
 

[Leo Laohoo]

Post the complete output to the command "sh version".

[waqas gondal]

TER-S3-FVOPS-1E1#sh ver
Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 15.2(4)E5, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2017 by Cisco Systems, Inc.
Compiled Mon 18-Sep-17 06:58 by prod_rel_team

ROM: Bootstrap program is C3750E boot loader
BOOTLDR: C3750E Boot Loader (C3750X-HBOOT-M) Version 12.2(53r)SE1, RELEASE SOFTWARE (fc1)

TER-S3-FVOPS-1E1 uptime is 9 weeks, 14 hours, 57 minutes
System returned to ROM by power-on
System restarted at 17:55:50 pst Thu Dec 14 2017
System image file is "flash:c3750e-universalk9-mz.152-4.E5.bin"
Last reload reason: Reload command



This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

License Level: ipbase
License Type: Permanent
Next reload license Level: ipbase

cisco WS-C3750X-48P (PowerPC405) processor (revision A0) with 262144K bytes of memory.
Processor board ID FDO1528R08F
Last reset from power-on
2 Virtual Ethernet interfaces
1 FastEthernet interface
260 Gigabit Ethernet interfaces
10 Ten Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : 44:D3:CA:79:71:00
Motherboard assembly number : 73-12553-05
Motherboard serial number : FDO15281LT9
Model revision number : A0
Motherboard revision number : C0
Model number : WS-C3750X-48PF-S
Daughterboard assembly number : 800-32727-01
Daughterboard serial number : FDO15281C1W
System serial number : FDO1528R08F
Top Assembly Part Number : 800-31324-02
Top Assembly Revision Number : C0
Version ID : V02
CLEI Code Number : COMJZ00ARB
Hardware Board Revision Number : 0x03


Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 1 54 WS-C3750X-48P 15.2(4)E5 C3750E-UNIVERSALK9-M
2 54 WS-C3750X-48P 15.2(4)E5 C3750E-UNIVERSALK9-M
3 54 WS-C3750X-48P 15.2(4)E5 C3750E-UNIVERSALK9-M
4 54 WS-C3750X-48P 15.2(4)E5 C3750E-UNIVERSALK9-M
5 54 WS-C3750X-48P 15.2(4)E5 C3750E-UNIVERSALK9-M


Switch 02
---------
Switch Uptime : 9 weeks, 14 hours, 54 minutes
Base ethernet MAC Address : 44:D3:CA:7C:A6:80
Motherboard assembly number : 73-12553-05
Motherboard serial number : FDO152818UF
Model revision number : A0
Motherboard revision number : C0
Model number : WS-C3750X-48PF-S
Daughterboard assembly number : 800-32727-01
Daughterboard serial number : FDO15281BYP
System serial number : FDO1528K05Y
Top assembly part number : 800-31324-02
Top assembly revision number : C0
Version ID : V02
CLEI Code Number : COMJZ00ARB
License Level : ipbase
License Type : Permanent
Next reboot licensing Level : ipbase


Switch 03
---------
Switch Uptime : 9 weeks, 14 hours, 54 minutes
Base ethernet MAC Address : 44:D3:CA:79:62:00
Motherboard assembly number : 73-12553-05
Motherboard serial number : FDO15281LUV
Model revision number : A0
Motherboard revision number : C0
Model number : WS-C3750X-48PF-S
Daughterboard assembly number : 800-32727-01
Daughterboard serial number : FDO15281AZA
System serial number : FDO1528R084
Top assembly part number : 800-31324-02
Top assembly revision number : C0
Version ID : V02
CLEI Code Number : COMJZ00ARB
License Level : ipbase
License Type : Permanent
Next reboot licensing Level : ipbase


Switch 04
---------
Switch Uptime : 9 weeks, 14 hours, 54 minutes
Base ethernet MAC Address : 44:D3:CA:79:5A:80
Motherboard assembly number : 73-12553-05
Motherboard serial number : FDO15281LZJ
Model revision number : A0
Motherboard revision number : C0
Model number : WS-C3750X-48PF-S
Daughterboard assembly number : 800-32727-01
Daughterboard serial number : FDO15281BYD
System serial number : FDO1528K064
Top assembly part number : 800-31324-02
Top assembly revision number : C0
Version ID : V02
CLEI Code Number : COMJZ00ARB
License Level : ipbase
License Type : Permanent
Next reboot licensing Level : ipbase


Switch 05
---------
Switch Uptime : 9 weeks, 14 hours, 54 minutes
Base ethernet MAC Address : 44:D3:CA:7A:5E:80
Motherboard assembly number : 73-12553-05
Motherboard serial number : FDO15281LMN
Model revision number : A0
Motherboard revision number : C0
Model number : WS-C3750X-48PF-S
Daughterboard assembly number : 800-32727-01
Daughterboard serial number : FDO15281BEU
System serial number : FDO1528R08G
Top assembly part number : 800-31324-02
Top assembly revision number : C0
Version ID : V02
CLEI Code Number : COMJZ00ARB
License Level : ipbase
License Type : Permanent
Next reboot licensing Level : ipbase


Configuration register is 0xF

[Leo Laohoo]

Any operational reason why this version is being used instead of, say, 12.2(55)SE train?

[waqas gondal]

Not that I am aware of but I think it was chosen because it was fairly recent. I posted the config in a reply below. There is quite a bit of config on it for ISE on all of our devices. We have this same version 15.2 running on all the other 3750X stacks in our building. This is the only switch having this issue. Unfortunately our support is expired so we can't get TAC to help with it.

[Joseph W. Doherty]

Leo is looking for version information to help determine how stable an IOS variant you're running. Some bugs are memory leaks. That might account for high memory.

Another thing to check is your TCAM usage, as using the wrong template can cause ASIC hardware processing to be done by the CPU. That might account for your high CPU.

Lastly, some IOS features will consume memory, and if too much is used, memory can be highly fragmented and cause all kinds of issues. To check for those, we would need to see your full config.

[waqas gondal]

Also here is the tcam usage:
TER-S3-FVOPS-1E1#show platform tcam usage

=============================================================================
TCAM Table
TCAM Use Width Raw Entries Num Entries Entry Size
=============================================================================
0 L2 6144
1 L2 256
2 L3 7286
3 L3 116
4 L3 180 2048 2048 32
5 L3 180 2048 32 32
6 ACL 276 2048 1948 64
=============================================================================
FRAM Entries
=============================================================================
TCAM : 0
Entry Start Entry Base Address Num Entries Bytes/Entry # Range
=============================================================================
0 0x0 0x0 6144 4 1
1 0x0 0x0 6144 4 1
2 0x0 0x0 0 0 0
3 0x0 0x0 0 0 0
4 0x0 0x0 0 0 0
5 0x0 0x0 0 0 0
6 0x0 0x0 0 0 0

TCAM : 1
Entry Start Entry Base Address Num Entries Bytes/Entry # Range
=============================================================================
0 0x0 0x0 256 4 1
1 0x0 0x0 256 4 1
2 0x0 0x0 0 0 0
3 0x0 0x0 0 0 0
4 0x0 0x0 0 0 0
5 0x0 0x0 0 0 0
6 0x0 0x0 0 0 0

TCAM : 2
Entry Start Entry Base Address Num Entries Bytes/Entry # Range
=============================================================================
0 0x0 0x0 6134 4 1
1 0x0 0x0 64 4 1
2 0x0 0x0 1024 8 1
3 0x0 0x0 64 8 1
4 0x0 0x0 0 0 0
5 0x0 0x0 0 0 0
6 0x0 0x0 0 0 0

TCAM : 3
Entry Start Entry Base Address Num Entries Bytes/Entry # Range
=============================================================================
0 0x70 0x0 10 4 1
1 0x60 0x0 10 4 1
2 0x0 0x0 48 8 1
3 0x30 0x0 48 8 1
4 0x0 0x0 0 0 0
5 0x0 0x0 0 0 0
6 0x0 0x0 0 0 0

TCAM : 4
Entry Start Entry Base Address Num Entries Bytes/Entry # Range
=============================================================================
0 0x0 0x0 2048 4 1
1 0x0 0x0 0 0 0
2 0x0 0x0 0 0 0
3 0x0 0x0 0 0 0
4 0x0 0x0 0 0 0
5 0x0 0x0 0 0 0
6 0x0 0x0 2048 0 2
Ranges:0 - 1023 , 1024 - 2047 ,

TCAM : 5
Entry Start Entry Base Address Num Entries Bytes/Entry # Range
=============================================================================
0 0x0 0x0 0 0 0
1 0x0 0x0 32 4 1
2 0x0 0x0 0 0 0
3 0x0 0x0 0 0 0
4 0x0 0x0 0 0 0
5 0x0 0x0 0 0 0
6 0x0 0x0 2048 0 2
Ranges:0 - 1023 , 1024 - 2047 ,

TCAM : 6
Entry Start Entry Base Address Num Entries Bytes/Entry # Range
=============================================================================
0 0x12 0x200 60 4 1
1 0x464 0x0 924 4 1
2 0x0 0x380 18 8 1
3 0x64 0x600 412 8 1
4 0x4E 0x300 22 8 1
5 0x200 0x400 512 8 1
6 0x0 0x0 2048 0 2
Ranges:0 - 1023 , 1024 - 2047 ,

Table Name Base Address Num Entries Bytes/Entry
=============================================================================
Equal Cost Route Table: 0x3000 128 32
Static Address Table: 0x7780 2 64
Station Table: 0x6600 8192 4
MAC Address Table: 0x4D00 6144 8
Unicast RPF Check Table: 0x0 192 32
MetPtr Table: 0x0 1058 8
Multicast Expansion Table: 0xC22 7406 8
VLAN List Table: 0x422 2048 8
L2 Learning Result Table: 0x4000 6400 4
L2 Forwarding Result Table: 0xE00 6400 4
L3 IPv4 Ucast Fwding Result Table: 0x1C00 8192 4
L3 IPv4 Mcast Fwding Ext Result Table: 0x800 1072 8
L3 IPv6 Ucast Fwding Result Table: 0x1B00 106 4
L3 IPv6 Mcast Fwding Ext Result Table: 0xC80 112 8