cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2154
Views
5
Helpful
12
Replies

Slow switch members on a Cisco 2960x Stack.

michael.wellner
Level 1
Level 1

I have a Cisco 2960x stack with 7 stack members. switch 5 and 6 in the stack is responding very slow to commands, such as the "Verify flash:xxxxxxxxxx command taking 30 minutes to an hour to complete. All the other switches in the stack respond quickly (normal). I have used the "show memory debug leak" with nothing showing up. All member are showing correct priorities and switch numbers. All stacking cables are connected correctly and port status show "OK". What troubleshooting steps should I use next? This stack is in-production and is switching with no other apparent problems. Any help is appreciated. 

 

Switch Model: WS-C2960X-48FPD-L

The current firmware is: 15.2(7)E5

BOOTLDR: C2960X Boot Loader (C2960X-HBOOT-M) Version 15.2(6r)E
If you need more info, just ask. 

Thank you

12 Replies 12

Leo Laohoo
Hall of Fame
Hall of Fame

What file is being verified?  The IOS BIN file?

Yes. I used that an example. They are slow comparatively with all commands. 


@michael.wellner wrote:

Yes. I used that an example. They are slow comparatively with all commands. 


So you are "upgrading" the IOS of the stack by copying the BIN file from one switch to another and then verifying the BIN file? 

Have you tried the recommended method:  archive download-sw tftp://<TFTP_IP_ADDRESS>/filename.tar

TAR file is required but by using this command alone, it will do the following: 

  1. Unpack the BIN file into each stack members
  2. Change the boot variable string to point to the new BIN file
  3. File verification check of the BIN file

Leo you are misunderstanding my problem. I have 2 switches out of 7 in a stack that are processing slow. I used the verify command as an example. all commands for those two switches are taking an abnormal amount of time to finish. Please reread my original post. 

 

Thanks

What is the uptime of each switch member? 

Post the complete output to the following commands:

  1. remote command 5 sh proc cpu sort | ex 0.00
  2. remote command 6 sh proc cpu sort | ex 0.00
  3. remote command 5 sh proc memory sort | ex 0.00
  4. remote command 6 sh proc memory sort | ex 0.00
  5. sh log

Leo,  here is info you requested

 SWITCH#remote command 5 sh proc cpu sort | ex 0.00
Switch : 5 :
------------

CPU utilization for five seconds: 39%/0%; one minute: 39%; five minutes: 40%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
 180    33485522     3171736      10557 22.79% 22.62% 22.60%   0 Hulc LED Process
 143      981210      126900       7732  0.59%  0.65%  0.65%   0 hpm counter proc
 197      569896      353268       1613  0.41%  0.37%  0.36%   0 HRPC qos request
 196      477193       25280      18876  0.29%  0.32%  0.30%   0 HQM Stack Proces
 216      213678      110740       1929  0.17%  0.13%  0.12%   0 CDP Protocol
  60       89348       25387       3519  0.05%  0.05%  0.05%   0 Compute load avg
 250       64349      218949        293  0.05%  0.05%  0.05%   0 HULC DAI Process
 132      111314      480731        231  0.05%  0.02%  0.04%   0 HRPC hlfm reques


 SWITCH#remote command 6 sh proc cpu sort | ex 0.00
Switch : 6 :
------------

CPU utilization for five seconds: 38%/0%; one minute: 38%; five minutes: 39%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
 180    33105514     3170955      10440 22.19% 22.24% 22.25%   0 Hulc LED Process
 197      565193      352981       1601  0.41%  0.37%  0.36%   0 HRPC qos request
 196      464452       25275      18375  0.29%  0.30%  0.29%   0 HQM Stack Proces
 143      549208      126861       4329  0.23%  0.35%  0.35%   0 hpm counter proc
 132      123549      487078        253  0.17%  0.03%  0.04%   0 HRPC hlfm reques
 107       53699      481106        111  0.05%  0.03%  0.04%   0 hrpc <- response
 216      153772       82871       1855  0.05%  0.11%  0.11%   0 CDP Protocol
  60       88543       25378       3488  0.05%  0.05%  0.05%   0 Compute load avg


 SWITCH#remote command 6 sh proc memory sort | ex 0.00
Switch : 6 :
------------
Processor Pool Total:  366118500 Used:   85644976 Free:  280473524
      I/O Pool Total:   33554432 Used:   12682708 Free:   20871724
Driver te Pool Total:    1048576 Used:         40 Free:    1048536

 PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
   0   0  179894648   97585160   76904516          0          0 *Init*
   0   0   13505448    9479008    3918296   12451527     949166 *Dead*
 104   0    5765148     863448    3783028       5076          0 Stack Mgr Notifi
 123   0    5294660    3005388    1607320      45684          0 HRPC hulc misc r
 400   0    1255608      17288    1248488          0          0 EEM Server
 271   0    2617952    2006840     887588          0          0 XDR receive
   0   0          0          0     853676          0          0 *MallocLite*
 197   0 3240851736 3240133056     779848          0          0 HRPC qos request
  71   0     369796       6964     372904          0          0 EEM ED Identity
 207   0     324560      43628     295144          0          0 HL2MCM
 206   0     297880      18788     294992          0          0 HL2MCM
 385   0     284040       5124     289244     100548          0 EEM ED Syslog
   1   0   24761880   24520860     258120          0          0 Chunk Manager
 216   0   34154784    7795276     256496      10152          0 CDP Protocol
 132   0    2895684    1581972     219480     654804          0 HRPC hlfm reques
 383   0     209404       4964     214608      67468          0 EEM ED Generic
  74   0     219256        232     201024          0          0 USB Startup
 174   0   15603488   15374692     171696          0          0 HRPC stp_state_s
 359   0     152680        600     156308       5076          0 Crypto IKEv2
 252   0    1303668    1151752     150144       5076          0 HRPC ip device t
 339   0     117216        232     130152          0          0 MRIB Process
  55   0     298648     312232      99208          0          0 IF-MGR control p
   7   0     913996    1642728      92704     630496     986931 Pool Manager
 354   0    1043624     901960      88512          0          0 Crypto CA
 261   0      71336          0      78504          0          0 Switch Backup In
 168   0      66812       3912      72748          0          0 REP Helper Proc
 113   0     245964     183288      69844          0          0 HRPC cfg_backup
 143   0   39809348        232      63812      86292          0 hpm counter proc
 187   0          0      10088      61168          0          0 HL3U PBR bkgrd p
 154   0        232       7200      61168          0          0 HACL Acl Manager
  19   0      64816      80280      59816          0      11440 Entity MIB API
 367   0      49840       5064      57608          0          0 IPSEC key engine
 211   0      49364          0      56532          0          0 ACCT Periodic Pr
 309   0      49584        232      56520          0          0 Timer Library
  49   0    1045008        204      56148          0          0 Net Background
 258   0   20398392       5276      54848     157356          0 PI MATM Aging Pr
 365   0      38608        232      51544          0          0 Crypto IKMP
 196   0 1166502404 1158771976      50992     289332          0 HQM Stack Proces
 401   0      78828      21144      50124          0          0 Call Home proces
  73   0      44124   18675716      48728          0          0 EEM ED ND
 139   0    1095200     756888      45916       5076          0 hpm main process
   3   0      19084          0      44124          0          0 LICENSE AGENT
 328   0     173696     683364      43860          0          0 IGMPSN L2MCM
 335   0     172628     673376      42660          0          0 MLDSN L2MCM
 229   0      33012          0      40180          0          0 IP ARP Retry Age
 428   0          0          0      33092          0          0 PnP-Monitor
 120   1      32944      17712      28400          0          0 HRPC hrcmd reque
  62   0   26763344      58512      26972       5076          0 Per-Second Jobs
 395   0     195980     152264      26008          0          0 SpanTree Helper
 218   0        232          0      25400          0          0 tHUB
 368   0          0          0      25168          0          0 Crypto ACL
 376   0        232        232      25168          0          0 ASP Process Crea
  37   0          0          0      25168          0          0 Crash writer
 430   0      11812        464      24516          0          0 RBM CORE
 305   0      13332          0      23500          0          0 HTTP CORE
 407   0      12588        904      21852          0          0 EEM Policy Direc
 412   0      16640       4964      21844          0          0 EEM ED CRASH
 413   0      16640       4964      21844          0          0 EEM ED DS
 418   0      16488       4964      21692          0          0 EEM ED Nf
 379   0      18588       4372      21684          0          0 DCM Core Thread
 411   0      16384       4964      21588          0          0 EEM ED Counter
 435   0     253788     233028      21528       2340          0 VLAN Manager
 133   0     302024        464      20556          0          0 HLFM address lea
 438   0      10784        484      20340          0          0 OBFL ENV obfl0
 405   0      14148       1740      19400          0          0 DCM snmp dp Thre
 184   0      26820          0      19088      15228          0 HRPC hl3u reques
 274   0       5260          0      18472          0          0 fib_fib_bfd_sb e
 103   0       5044          0      18212          0          0 Stack Mgr
 364   0       5276        380      18212          0          0 Crypto IKE Dispa
 153   0      32108       3044      17152      10152          0 HRPC acl request
 416   0      11796       4964      17000          0          0 EEM ED Interface
 417   0      11620       5124      16824          0          0 EEM ED IOSWD
 426   0      11492       4964      16696          0          0 EEM ED Timer
 424   0      11436       4964      16640          0          0 EEM ED SNMP Noti
 384   0      11436       4964      16640          0          0 EEM ED Routing
 422   0      11392       4964      16596          0          0 EEM ED SNMP
 414   0      11372       4964      16576          0          0 EEM ED Env
 415   0      11340       4964      16544          0          0 EM ED GOLD
 410   0      11340       4964      16544          0          0 EEM ED Config
 409   0      11340       4964      16544          0          0 EEM ED CLI
 420   0      11340       4964      16544          0          0 EEM ED OIR
 423   0      11340       4964      16544          0          0 EEM ED SNMP Obje
 419   0      11340       4964      16544          0          0 EEM ED None
 425   0      11340       4964      16544          0          0 EEM ED Test
 228   0       3168          0      16336          0          0 IP ARP Adjacency
 437   0      68468      66680      14828          0          0 OBFL MSG obfl0
 257   0      34788          0      14712          0          0 CEF: IPv4 proces
 301   0        888          0      14056          0          0 HTTP EPM Redirec
 357   0        888          0      14056          0          0 PKI OCSP
  53   0        788        232      13724          0          0 SXP CORE
 350   0        788        232      13724          0          0 TPLUS
 343   0        788        232      13724          0          0 LDAP process
 402   0       7012        496      13620          0          0 EEM Helper Threa
 281   0       7616        232      13512          0          0 MDFS LC Process
 403   0       8624      13284      13400          0          0 Bulkstat-Client
 227   0        216          0      13384          0          0 CEF switching ba
 273   0        216          0      13384          0          0 CEF background p
 284   0        216          0      13384          0          0 IP RIB Update
 307   0        216          0      13384          0          0 MFIB Master back
 436   0        216          0      13312          0          0 Collection proce
 351   0        216        132      13256          0          0 ADJ background
 114   0          0          0      13180          0          0 HIPC bkgrd proce
 310   0        232        232      13168          0          0 Tunnel
  88   0          0          0      13168          0          0 Notify process
 304   0          0          0      13168          0          0 TCP Protocols
 356   0          0          0      13168          0          0 Crypto PKI-Rev-P
 355   0        232        232      13168          0          0 Crypto PKI-CRL
 361   0          0          0      13168          0          0 encrypt proc
 353   0          0          0      13168          0          0 RSA background p
 286   0          0          0      13168          0          0 EPM MAIN PROCESS
 285   0        232        232      13168          0          0 Auth Manager
 348   0        232        232      13168          0          0 LOCAL AAA
 283   0          0          0      13168          0          0 Routing Topology
 363   0          0          0      13168          0          0 Crypto INT
  64   0          0          0      13168          0          0 ACT2 Crypto Engi
 358   0          0          0      13168          0          0 PKI Revocation
 298   0          0     508716      13168          0          0 EPM ACL PLUG-IN
  18   0          0          0      13168          0          0 Policy Manager
 370   0          0          0      13168          0          0 IPSec background
 378   0          0          0      13168          0          0 Test AAA Client
 303   0          0          0      13168          0          0 TCP Timer
 233   0        232        232      13168          0          0 MDFS MFIB Proces
 230   0          0          0      13168          0          0 IP Input
  41   0          0          0      13168          0          0 License IPC serv
  45   0        232        232      13168          0          0 XML Proxy Client
 124   0          0          0      13168          0          0 HVLAN main bkgrd
 185   0          0          0      13168          0          0 HIPV6 bkgrd proc
 427   0      16420      16420      13168          0          0 Syslog
 127   0          0          0      13168          0          0 HRPC vlan reques
  40   0          0          0      13168          0          0 License IPC stat
 183   0          0          0      13168          0          0 HL3U bkgrd proce
 163   0     304572     296044      13168       5076          0 HULC DOT1X Proce
 146   0          0          0      13168          0          0 HCEF ADJ Refresh
  51   0      74976      15152      13168      30456          0 Logger
  50   0          0          0      13168          0          0 IDB Work
 320   0       6164        232      13100          0          0 REP BPA/EPA Proc
 352   0     218716     212812      13072          0          0 crypto engine pr
 299   0          0          0      13040          0          0 CFC LC Test Stat
 208   0          0          0      13040          0          0 CFC stats broker
   0   0      12248     378700      12248          0          0 *Sched*
 198   0       5044          0      12212          0          0 HRPC span reques
  81   0       5276        232      12212          0          0 HLPIP Sync Proce
 374   0       5276        232      12212          0          0 VTP Trap Process
  72   0       7136       5164      12140          0          0 EEM ED MAT
   6   0       4652        232      11640          0          0 Check heaps
 393   0     122280     659940      11576          0          0 PM Callback
  31   0      11440          0      11232          0          0 IPC Seat RX Cont
 382   0       4280        240      11140          0          0 dcm_cli_provider
 129   0     132268      15168      11040      65988          0 HRPC ilp request
 270   0       3988          0      11028          0          0 CEF LC Stats
 311   0       1004       4444      10940          0          0 MVPN Mgr Process
 332   0        512        232      10448          0          0 AUTO LAG Protoco
 312   0        448        232      10384          0          0 Multicast Offloa
 279   0        216          0      10384          0          0 Tunnel FIB
 406   0       3316        240      10172          0          0 snmp dcm ma shim
  57   0          0          0      10168          0          0 IP Admission HA
 275   0          0          0      10168          0          0 IP IRDP
 287   0          0          0      10168          0          0 IPv6 RIB Cleanup
 317   0    1101100    1101100      10168          0          0 QoS stats proces
 315   0          0          0      10168          0          0 IPv6 Static Hand
 314   0        232        232      10168          0          0 static
 232   0        232        232      10168          0          0 IP ARP Track
  99   0       2812       2812      10168          0          0 Adjust Regions
 337   0          0          0      10168          0          0 MLD_SNOOP
 330   0          0          0      10168          0          0 IGMPSN
 262   0        232        232      10168          0          0 IP Host Track Pr
 288   0          0          0      10168          0          0 IPv6 RIB Event H
 366   0          0          0      10168          0          0 IPSEC manual key
 180   0        232        232      10168          0          0 Hulc LED Process
 373   0          0          0      10168          0          0 Licensing MIB pr
 381   0       2408          0       9576          0          0 dcm_cli_engine
 191   0       2392          0       9560          0          0 CoPP Policer sta
  29   0       2088          0       9256          0          0 IPC Seat Manager
 102   0       5044          0       9212          0          0 HRPC hsm request
 225   0       1892          0       9060          0          0 TRACK Main threa
 226   0       1716          0       8884          0          0 TRACK Client thr
 213   0        888          0       8056          0          0 Auth-proxy HTTP
 246   0        928        232       7864          0          0 DTP Protocol
 202   0       9872       1192       7620       5076          0 HRPC hrcli-event
 203   0   29813724   29813272       7620          0          0 HRPC rfs request
  38   0        340          0       7508          0          0 Exception contro
 259   0        532        232       7468          0          0 UDLD
 267   0        216          0       7384          0          0 XDR background p
 432   0        188          0       7384          0          0 hulc_eee_monitor
 236   0        448        232       7384          0          0 loopdetect
 277   0        448        232       7384          0          0 CEF LC IPC Backg
 241   0      41556        232       7384          0          0 Spanning Tree
 293   0        448        232       7384          0          0 Dot1x Supplicant
 392   0        216          0       7384          0          0 Online Diag EEM
 294   0        448        232       7384          0          0 Dot1x Supplicant
 295   0        448        232       7384          0          0 Dot1x Supplicant
 260   0        216          0       7384          0          0 Port-Security
 334   0        156          0       7324          0          0 Marvell wk-a Pow
 119   0      10020        932       7316       5076          0 HRPC hpsecure re
 194   0          0          0       7228          0          0 HRPC snmp reques
  66   0          0          0       7224          0          0 Transport Port A
 159   0          0          0       7212          0          0 HRPC dns-as requ
 130   0       2620          0       7208          0          0 Hulc ILP Alchemy
 344   0        232        232       7204          0          0 Crypto Support
 308   0     984832     984832       7196          0          0 NIST rng proc
 125   0        232        232       7180          0          0 HVLAN Mapped Vla
  98   0          0          0       7176          0          0 HRPC asic-stats
 193   0          0          0       7168          0          0 HRPC show_forwar
 192   0          0          0       7168          0          0 HRPC dtp request
 201   0          0          0       7168          0          0 Hulc Blue Beacon
 205   0        232        232       7168          0          0 Power RPS Proces
  42   0        232        232       7168          0          0 GraphIt
 204   0          0          0       7168          0          0 HRFS OIR Proc
 200   0          0          0       7168          0          0 Hulc Reload Mana
 209   0        232        232       7168          0          0 AAA Server
 210   0          0          0       7168          0          0 AAA ACCT Proc
  39   0          0       6652       7168          0          0 crypto sw pk pro
 212   0          0          0       7168          0          0 Webauth client
 136   0          0          0       7168          0          0 HULC PM Vector P
 214   0          0          0       7168          0          0 IP Admin SM Proc
 215   0          0          0       7168          0          0 hci usb process
 189   0          0          0       7168          0          0 HRPC obfl reques
 128   0          0          0       7168          0          0 HULC VLAN REF Ba
  36   0     525056     525056       7168          0          0 PrstVbl
 126   0        232        232       7168          0          0 Vlan shutdown Pr
 220   0          0          0       7168          0          0 DHCP Snooping
 221   0          0          0       7168          0          0 DHCP Snooping db
 222   0        232        232       7168          0          0 Dot1x Mgr Proces
 223   0          0          0       7168          0          0 EAP Framework
 224   0          0          0       7168          0          0 EAP Test
 188   0          0          0       7168          0          0 HL3U PBR n-h res
 186   0      17880          0       7168      10152          0 HRPC IPv6 Unicas
 199   0      60968      60968       7168          0          0 HRPC system post
  60   0        232        232       7168          0          0 Compute load avg
  35   0          0          0       7168          0          0 IPC Session Deta
 122   0       1160          0       7168          0          0 HULC EMAC Proces
 131   0          0          0       7168          0          0 Strider Tcam Mem
  59   0          0          0       7168          0          0 Net Input
 121   0          0          0       7168          0          0 HRPC emac reques
  61   0          0          0       7168          0          0 Per-minute Jobs
 235   0          0          0       7168          0          0 IPv6 ping proces
  46   0          0          0       7168          0          0 ARP Snoop
 237   0          0          0       7168          0          0 PPPOE IA
  58   0          0          0       7168          0          0 ICC Nego
 239   0        232        232       7168          0          0 REP Topology cha
 240   0          0          0       7168          0          0 RIB LM VALIDATE
 182   0      52992      52992       7168          0          0 hl3mm
 242   0        232        232       7168          0          0 SpanTree Msg
 181   0          0          0       7168          0          0 HRPC hl3mm reque
 118   0      97972      63500       7168          0          0 Hulc Port-Securi
 179   0          0          0       7168          0          0 HRPC hled reques
 178   0      26820          0       7168      15228          0 HRPC l2pt qnq rp
 177   0          0          0       7168          0          0 HRPC vqpc reques
 176   0       8940          0       7168       5076          0 HRPC hudld reque
 155   0          0          0       7168          0          0 HRPC aim request
 231   0          0          0       7168          0          0 ICMP event handl
  43   0          0          0       7168          0          0 client_entity_se
  34   0          0          0       7168          0          0 IPC Loadometer
 253   0          0          0       7168          0          0 HRPC ip source g
  44   0        232        232       7168          0          0 SMART
 255   0          0          0       7168          0          0 HULC Open flow S
 256   0          0          0       7168          0          0 HRPC ofsdn reque
 195   0          0          0       7168          0          0 HULC SNMP Proces
  33   0          0          0       7168          0          0 IPC Keep Alive M
 175   0        232        232       7168          0          0 S/W Bridge Proce
  47   0          0          0       7168          0          0 Dynamic ARP Insp
 249   0          0          0       7168          0          0 HRPC dai request
 173   0          0          0       7168          0          0 HRPC stp_cli req
 263   0          0          0       7168          0          0 Link State Group
 264   0          0          0       7168          0          0 MMN bkgrd proces
 172   0        232        232       7168          0          0 HSTP Sync Proces
 266   0      86076      22468       7168          0          0 VMATM Callback
 171   0        232        232       7168          0          0 Hulc Storm Contr
 268   0     368512     368712       7168          0          0 XDR mcast
 170   0          0          0       7168          0          0 HRPC sdm request
  52   0        232        232       7168          0          0 TTY Background
 152   0          0          0       7168          0          0 HACL Queue Proce
 272   0          0          0       7168          0          0 FRR Background P
 115   0          0          0       7168          0          0 IPC RTTYC Messag
 169   0          0          0       7168          0          0 HULC REP monitor
 151   0          0          0       7168          0          0 hl3mm_rp
 150   0          0          0       7168          0          0 hl3mm_mrib_rp
  48   0          0          0       7168          0          0 Critical Bkgnd
 149   0      79488      79488       7168          0          0 hl3mm_mfib_ipv6
 148   0          0          0       7168          0          0 HRPC hl3mm ipv6
 280   0          0          0       7168          0          0 IPv6 ACL RP Proc
 112   0       1232       1232       7168          0          0 HRPC hdm blockin
 282   0          0          0       7168          0          0 AAA System Acct
 167   0          0          0       7168          0          0 HRPC rep request
 110   0        500        232       7168          0          0 HULC Device Mana
 109   0        232        232       7168          0          0 apm86xxx_enet_pr
 108   0          0          0       7168          0          0 HRPC hcomp reque
 134   0        232        232       7168          0          0 HLFM aging proce
 165   0          0          0       7168          0          0 HRPC lldp reques
 289   0          0          0       7168          0          0 Critical Auth
 290   0          0          0       7168          0          0 SVM UT Process
 291   0          0          0       7168          0          0 CMD HANDLER
 292   0          0          0       7168          0          0 Socket Timers
 164   0          0          0       7168          0          0 HRPC epm vlan pl
 135   0        232        232       7168          0          0 HLFM address ret
 162   0          0          0       7168          0          0 Dot1X Msg Retry
 296   0          0          0       7168          0          0 DSensor HTTP
 297   0          0          0       7168          0          0 EPM Downlad Mana
 106   0          0          0       7168          0          0 hrpc -> request
 300   0          0          0       7168          0          0 Timer Handler Pr
 105   0          0    1332240       7168          0     756324 hrpc -> response
 302   0        232        232       7168          0          0 Flow Exporter Ti
  65   0          0          0       7168          0          0 AggMgr Process
 251   0          0          0       7168          0          0 HRPC power down
 161   0     155632      87032       7168          0          0 HRPC dot1x reque
 306   0          0          0       7168          0          0 DCT Bkgd Process
 160   0          0          0       7168          0          0 avc_dns_as_hulc
 254   0          0          0       7168          0          0 HULC IP Source g
  28   0          0          0       7168          0          0 IPC Process leve
 101   0        232        232       7168          0          0 FlexStack Hotswa
 137   0          0          0       7168          0          0 HPM Msg Retry Pr
 138   0          0          0       7168          0          0 hpm bulk_vlan_st
 313   0          0          0       7168          0          0 RARP Input
 116   0          0          0       7168          0          0 RTTYC Flush
 158   0          0          0       7168          0          0 HULC CISP Proces
 316   0          0          0       7168          0          0 MAB Framework
 157   0          0          0       7168          0          0 HRPC cdp request
 100   0        232        232       7168          0          0 Horizontal Stack
 319   0        232        232       7168          0          0 REP LSL Proc
 243   0          0          0       7168          0          0 Spanning Tree St
 321   0          0          0       7168          0          0 SNMP Timers
 322   0       8940          0       7168       5076          0 HRPC dhcp snoopi
 323   0          0          0       7168          0          0 HULC DHCP Snoopi
 324   0       8940          0       7168       5076          0 HRPC archive-dnl
 325   0       8940          0       7168       5076          0 HRPC sisf reques
 326   0          0          0       7168          0          0 HULC SISF Proces
 327   0          0          0       7168          0          0 HULC SISF Source
  27   0       7672      10388       7168          0          0 IPC Deferred Por
 329   0          0          0       7168          0          0 IGMPSN MRD
 245   0    1903564       5044       7168          0          0 802.1x switch
 331   0          0          0       7168          0          0 IGMPQR
 145   0          0          0       7168          0          0 hpm vp events ca
 333   0      60644      25860       7168       5076          0 Inline Power
 247   0          0          0       7168          0          0 HRPC IPv6 Host r
  26   0          0          0       7168          0          0 IPC Periodic Tim
 336   0          0          0       7168          0          0 MRD
 248   0          0          0       7168          0          0 IPv6 Platform Ho
 338   0        232        232       7168          0          0 AAA Cached Serve
 117   0          0          0       7168          0          0 HRPC hautosecure
 340   0          0          0       7168          0          0 HRPC hl2mcm igmp
 341   0          0          0       7168          0          0 HRPC hl2mcm mlds
 342   0        232        232       7168          0          0 ENABLE AAA
  96   0      66044        232       7168          0          0 HULC Thermal Pro
  32   0          0          0       7168          0          0 IPC Seat TX Cont
 345   0          0          0       7168          0          0 IPSECv6 PS Proc
 346   0          0          0       7168          0          0 Key chain liveke
 347   0        232        232       7168          0          0 LINE AAA
  95   0     145228     311192       7168          0          0 RedEarth Rx Mana
  93   0        232        232       7168          0          0 Stack DI Update
  92   0          0          0       7168          0          0 PSP Timer
  91   0          0          0       7168          0          0 Draught link sta
  90   0          0          0       7168          0          0 HRPC ipadm reque
  24   0          0          0       7168          0          0 IPC Session Serv
  89   0          0          0       7168          0          0 HRPC xcvr reques
  87   0          0          0       7168          0          0 HRPC actual powe
  86   0          0          0       7168          0          0 HRPC EnergyWise
  85   0          0          0       7168          0          0 HPM FEC LOAD SHA
  23   0          0          0       7168          0          0 IPC Dynamic Cach
 360   0          0          0       7168          0          0 IKEv2 AAA handle
  84   0          0          0       7168          0          0 HRPC hnetwpol re
 362   0        232        568       7168          0          0 CRYPTO MAP FREE
  83   0          0          0       7168          0          0 HULC multifs pro
  22   0          0          0       7168          0          0 IPC Mcast Pendin
  21   0          0          0       7168          0          0 IPC Event Notifi
 265   0        232        232       7168          0          0 Ethchnl
  20   0          0          0       7168          0          0 IFS Agent Manage
  82   0       5256       5256       7168          0          0 HRPC Multi-FS Sy
 369   0          0          0       7168          0          0 Crypto PAS Proc
 144   0     117740     117740       7168          0          0 HRPC pm-counters
 371   0          0          0       7168          0          0 HRPC x_setup req
 372   0        232        232       7168          0          0 REP Switch Helpe
 156   0          0          0       7168          0          0 HRPC backup inte
  54   0          0          0       7168          0          0 Cat6k NTI ICC pr
  80   0       1196       1196       7168          0          0 HRPC lpip reques
  17   0          0          0       7168          0          0 AAA_SERVER_DEADT
 377   0        232        232       7168          0          0 AAA SEND STOP EV
  79   0          0          0       7168          0          0 Hulc EEM Process
  16   0          0          0       7168          0          0 ARP Background
 380   0        232        232       7168          0          0 DHCP Security He
 269   0   74292452   74292452       7168          0          0 XDR LC PRIO Crit
  30   0          0          0       7168          0          0 IPC Check Queue
  15   0          0          0       7168          0          0 ARP Input
  78   0        232        232       7168          0          0 Porter Power Man
  14   0          0          0       7168          0          0 Image License br
 386   0          0          0       7168          0          0 Syslog Traps
 387   0        232        232       7168          0          0 FEX Logger Proce
 388   0          0          0       7168          0          0 HCD Process
 389   0          0          0       7168          0          0 HRPC cable diagn
 390   0          0          0       7168          0          0 HRPC archive_dnl
 391   0          0        324       7168          0          0 DiagCard6/-1
 111   0      55692      55692       7168          0          0 HRPC hdm non blo
  56   0          0          0       7168          0          0 IF-MGR event pro
 394   0          0          0       7168          0          0 HULC FNF
  13   0 1609745680 1609745680       7168          0          0 Licensing Auto U
 396   0        232        232       7168          0          0 dhcp snooping sw
 397   0          0          0       7168          0          0 ONEP_DPSS_SERVIC
 398   0          0          0       7168          0          0 HRPC onep reques
  12   0          0          0       7168          0          0 License Client N
  11   0          0          0       7168          0          0 HUSB Console
  77   0          0          0       7168          0          0 HRPC power_mgmt
  76   0        232        232       7168          0          0 DownWhenLooped
  75   0        232        232       7168          0          0 APM 86392 RTC
 404   0          0          0       7168          0          0 Call Home Timer
   9   0        232        232       7168          0          0 Timers
 140   0     255184     223820       7168          0          0 HPM Stack Sync P
   8   0          0          0       7168          0          0 DiscardQ Backgro
  67   0          0          0       7168          0          0 Token Daemon
  68   0          0          0       7168          0          0 HRPC pppoeia req
 276   0          0          0       7168          0          0 CEF RF HULC Conv
   5   0          0          0       7168          0          0 IPC ISSU Dispatc
   4   0          0          0       7168          0          0 Retransmission o
 278   0   26473984   26473984       7168          0          0 IP ACL XDR LC Ba
  69   0          0          0       7168          0          0 HC Counter Timer
  70   0          0          0       7168          0          0 SFF8472
 244   0          0          0       7168          0          0 avc_dns_as_clien
 219   0        232        232       7168          0          0 AAA Dictionary R
 431   0      17936          0       7168      10152          0 HRPC eee request
 429   0          0          0       7096          0          0 MFIB PROCESS IPC
 421   0          0          0       7084          0          0 IPv6 Access Cont
 349   0          0          0       7080          0          0 DiagCard4/-1
 318   0          0          0       7040          0          0 DiagCard2/-1
 375   0          0          0       7040          0          0 DiagCard5/-1
 238   0          0          0       7040          0          0 DiagCard3/-1
 399   0          0          0       7040          0          0 DiagCard7/-1
 217   0          0          0       7040          0          0 DiagCard1/-1
  25   0          0          0       7040          0          0 MDFS LC Download
 441   0          0          0       7040          0          0 OBFL I/O Buffer
 234   0        216          0       4384          0          0 ADJ NSF process
 433   0        232        232       4224          0          0 STP FAST TRANSIT
 147   0          0          0       4168          0          0 hl3md_rpfq_thrl_
 250   0   71223976          0       4168       5076          0 HULC DAI Process
 142   0        232        232       4168          0          0 HPM if_num mappi
  63   0          0          0       4168          0          0 Inode Table Dest
 166   0          0          0       4168          0          0 HRPC system mtu
  10   0          0          0       4168          0          0 WATCH_AFS
   2   0        232        232       4168          0          0 Load Meter
 434   0        232        232       4168          0          0 CSRT RAPID TRANS
                                 98285604 Total

 SWITCH#remote command 5 sh proc memory sort | ex 0.00
Switch : 5 :
------------
Processor Pool Total:  366118500 Used:   85097616 Free:  281020884
      I/O Pool Total:   33554432 Used:   12679548 Free:   20874884
Driver te Pool Total:    1048576 Used:         40 Free:    1048536

 PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
   0   0  179852280   97573752   76859448          0          0 *Init*
 104   0    7046004    1274828    3782884      67680          0 Stack Mgr Notifi
   0   0   12776952    9464168    3210828   12448479     949166 *Dead*
 123   0    5449192    3013800    1574112          0          0 HRPC hulc misc r
 400   0    1254948      16728    1248388          0          0 EEM Server
   0   0          0          0    1050416          0          0 *MallocLite*
 271   0    2652744    1980136     890808          0          0 XDR receive
 197   0 3243918072 3243199392     779848          0          0 HRPC qos request
  71   0     369796       6964     372904          0          0 EEM ED Identity
 207   0     325700      44916     295144          0          0 HL2MCM
 206   0     298680      19788     294984          0          0 HL2MCM
 385   0     284040       5124     289244     100548          0 EEM ED Syslog
   1   0   22030820   21751980     286008          0          0 Chunk Manager
 216   0   40771956    8745888     261048       5076          0 CDP Protocol
 383   0     209404       4964     214608      67468          0 EEM ED Generic
  74   0     216256        232     198024          0          0 USB Startup
 174   0   15236964   14979132     172568      10152          0 HRPC stp_state_s
 359   0     143788        620     156336          0          0 Crypto IKEv2
 252   0    1454920    1272920     153408      20304          0 HRPC ip device t
 132   0    3175132    1690108     141224     796932          0 HRPC hlfm reques
 339   0     117216        232     130152          0          0 MRIB Process
  55   0     298640     312232      99200          0          0 IF-MGR control p
 354   0     715360     639184      88628          0          0 Crypto CA
 196   0 1167264028 1159276348      83828     451764          0 HQM Stack Proces
 261   0      71336          0      78504          0          0 Switch Backup In
 168   0      66812       3912      72748          0          0 REP Helper Proc
 113   0     307060     244384      69844          0          0 HRPC cfg_backup
 143   0   35653808        232      69784     177660          0 hpm counter proc
 154   0        232       7072      61168          0          0 HACL Acl Manager
 187   0          0          0      61168          0          0 HL3U PBR bkgrd p
 141   0   40240372   39534108      60808      96444          0 HRPC pm request
  19   0      64120      77208      59096          0      10920 Entity MIB API
 139   0    4658552    3957488      57768          0          0 hpm main process
 367   0      49592       5164      57260          0          0 IPSEC key engine
 211   0      49364          0      56532          0          0 ACCT Periodic Pr
 309   0      49584        232      56520          0          0 Timer Library
 135   0      48628        232      55564          0          0 HLFM address ret
 365   0      38608        232      51544          0          0 Crypto IKMP
  49   0    1463160        996      51224          0          0 Net Background
  73   0      44124   23407772      48728          0          0 EEM ED ND
 328   0     337204     985008      47604          0          0 IGMPSN L2MCM
 335   0     327204     976864      46352          0          0 MLDSN L2MCM
   3   0      19020          0      44108          0          0 LICENSE AGENT
 229   0      33012          0      40180          0          0 IP ARP Retry Age
 129   0      77980      15168      37124      35532          0 HRPC ilp request
 428   0          0          0      33092          0          0 PnP-Monitor
 120   1      32956      17688      28436          0          0 HRPC hrcmd reque
  62   0   28096140      58544      26972       5076          0 Per-Second Jobs
 349   0     415180     293568      26092          0          0 SpanTree Helper
 218   0        232          0      25400          0          0 tHUB
 368   0          0          0      25168          0          0 Crypto ACL
 376   0        232        232      25168          0          0 ASP Process Crea
  37   0          0          0      25168          0          0 Crash writer
 430   0      11812        464      24516          0          0 RBM CORE
 305   0      13332          0      23500          0          0 HTTP CORE
 258   0   20836292        232      22068     126900          0 PI MATM Aging Pr
 407   0      12588        904      21852          0          0 EEM Policy Direc
 413   0      16640       4964      21844          0          0 EEM ED DS
 412   0      16640       4964      21844          0          0 EEM ED CRASH
 418   0      16520       4964      21724          0          0 EEM ED Nf
 379   0      18476       4304      21604          0          0 DCM Core Thread
 411   0      16384       4964      21588          0          0 EEM ED Counter
 435   0     253524     232604      21584       2340          0 VLAN Manager
 266   0     223220      72844      21088          0          0 VMATM Callback
 437   0      10744        464      20320          0          0 OBFL ENV obfl0
 133   0    1182628        464      20244          0          0 HLFM address lea
 375   0      27108          0      19716          0          0 CEF: IPv4 proces
 405   0      14108       1732      19388          0          0 DCM snmp dp Thre
 274   0       5260          0      18472          0          0 fib_fib_bfd_sb e
 364   0       5276        416      18212          0          0 Crypto IKE Dispa
 103   0       5044          0      18212          0          0 Stack Mgr
 153   0      14044       3004      17148          0          0 HRPC acl request
 416   0      11796       4964      17000          0          0 EEM ED Interface
 417   0      11620       5124      16824          0          0 EEM ED IOSWD
 426   0      11524       4964      16728          0          0 EEM ED Timer
 424   0      11436       4964      16640          0          0 EEM ED SNMP Noti
 384   0      11436       4964      16640          0          0 EEM ED Routing
 422   0      11392       4964      16596          0          0 EEM ED SNMP
 410   0      11340       4964      16544          0          0 EEM ED Config
 409   0      11340       4964      16544          0          0 EEM ED CLI
 415   0      11340       4964      16544          0          0 EM ED GOLD
 423   0      11340       4964      16544          0          0 EEM ED SNMP Obje
 419   0      11340       4964      16544          0          0 EEM ED None
 420   0      11340       4964      16544          0          0 EEM ED OIR
 414   0      11340       4964      16544          0          0 EEM ED Env
 425   0      11340       4964      16544          0          0 EEM ED Test
 393   0     273604     963808      15984          0          0 PM Callback
 408   0     388248     386420      14868          0          0 OBFL MSG obfl0
 228   0       1124          0      14292          0          0 IP ARP Adjacency
 301   0        888          0      14056          0          0 HTTP EPM Redirec
 357   0        888          0      14056          0          0 PKI OCSP
  53   0        788        232      13724          0          0 SXP CORE
 343   0        788        232      13724          0          0 LDAP process
 350   0        788        232      13724          0          0 TPLUS
 402   0       7012        496      13620          0          0 EEM Helper Threa
 281   0       7684        232      13516          0          0 MDFS LC Process
 403   0       8764      13368      13504          0          0 Bulkstat-Client
 307   0        272          0      13440          0          0 MFIB Master back
 227   0        216          0      13384          0          0 CEF switching ba
 284   0        216          0      13384          0          0 IP RIB Update
 429   0        280          0      13384          0          0 Collection proce
 273   0        216          0      13384          0          0 CEF background p
 399   0        216         68      13316          0          0 ADJ background
 361   0          0          0      13168          0          0 encrypt proc
 348   0        232        232      13168          0          0 LOCAL AAA
 358   0          0          0      13168          0          0 PKI Revocation
 310   0        232        232      13168          0          0 Tunnel
 356   0          0          0      13168          0          0 Crypto PKI-Rev-P
 355   0        232        232      13168          0          0 Crypto PKI-CRL
 363   0          0          0      13168          0          0 Crypto INT
 370   0          0          0      13168          0          0 IPSec background
  88   0          0          0      13168          0          0 Notify process
 298   0          0     504952      13168          0          0 EPM ACL PLUG-IN
 286   0          0          0      13168          0          0 EPM MAIN PROCESS
 114   0          0          0      13168          0          0 HIPC bkgrd proce
 285   0        232        232      13168          0          0 Auth Manager
 304   0          0          0      13168          0          0 TCP Protocols
 283   0          0          0      13168          0          0 Routing Topology
  18   0          0          0      13168          0          0 Policy Manager
 378   0          0          0      13168          0          0 Test AAA Client
 427   0   35780360   35780360      13168          0          0 Syslog
 353   0          0          0      13168          0          0 RSA background p
 303   0          0          0      13168          0          0 TCP Timer
  41   0          0          0      13168          0          0 License IPC serv
 124   0          0          0      13168          0          0 HVLAN main bkgrd
 233   0        232        232      13168          0          0 MDFS MFIB Proces
 230   0          0          0      13168          0          0 IP Input
 127   0          0          0      13168          0          0 HRPC vlan reques
  45   0        232        232      13168          0          0 XML Proxy Client
  64   0          0          0      13168          0          0 ACT2 Crypto Engi
 185   0          0          0      13168          0          0 HIPV6 bkgrd proc
 183   0          0          0      13168          0          0 HL3U bkgrd proce
 146   0          0          0      13168          0          0 HCEF ADJ Refresh
  40   0          0          0      13168          0          0 License IPC stat
 163   0     561584     451908      13168          0          0 HULC DOT1X Proce
  50   0          0          0      13168          0          0 IDB Work
  51   0   53974176   39320576      13168      20304          0 Logger
 320   0       6164        232      13100          0          0 REP BPA/EPA Proc
 299   0          0          0      13080          0          0 CFC LC Test Stat
  25   0          0          0      13040          0          0 CFC stats broker
 352   0     218668     212816      13020          0          0 crypto engine pr
   0   0      12248     378500      12248          0          0 *Sched*
  39   0       5044       6656      12212          0          0 crypto sw pk pro
 199   0     131996     109072      12212      10152          0 HRPC system post
 374   0       5276        232      12212          0          0 VTP Trap Process
  72   0       7136       5164      12140          0          0 EEM ED MAT
   6   0       4660        232      11648          0          0 Check heaps
  31   0      11648          0      11388          0          0 IPC Seat RX Cont
 238   0       4176          0      11216          0          0 CEF LC Stats
 382   0       4280        240      11140          0          0 dcm_cli_provider
 311   0       1004        232      10940          0          0 MVPN Mgr Process
 279   0        216          0      10384          0          0 Tunnel FIB
 312   0        448        232      10384          0          0 Multicast Offloa
 332   0        448        232      10384          0          0 AUTO LAG Protoco
 406   0       3348        240      10172          0          0 snmp dcm ma shim
 337   0          0          0      10168          0          0 MLD_SNOOP
 180   0        232        232      10168          0          0 Hulc LED Process
 232   0        232        232      10168          0          0 IP ARP Track
 317   0    1033412    1033412      10168          0          0 QoS stats proces
 330   0    2168296          0      10168          0          0 IGMPSN
 314   0        232       3964      10168          0          0 static
 315   0          0          0      10168          0          0 IPv6 Static Hand
  99   0       4632       4632      10168          0          0 Adjust Regions
  57   0          0          0      10168          0          0 IP Admission HA
 366   0          0          0      10168          0          0 IPSEC manual key
 288   0          0          0      10168          0          0 IPv6 RIB Event H
 287   0          0          0      10168          0          0 IPv6 RIB Cleanup
 373   0          0          0      10168          0          0 Licensing MIB pr
 275   0          0          0      10168          0          0 IP IRDP
 262   0        232        232      10168          0          0 IP Host Track Pr
 381   0       2408          0       9576          0          0 dcm_cli_engine
 191   0       2392          0       9560          0          0 CoPP Policer sta
 246   0      27796      19776       9504          0          0 DTP Protocol
  29   0       2088          0       9256          0          0 IPC Seat Manager
 102   0       5044          0       9212          0          0 HRPC hsm request
 225   0       1892          0       9060          0          0 TRACK Main threa
 226   0       1716          0       8884          0          0 TRACK Client thr
 213   0        888          0       8056          0          0 Auth-proxy HTTP
 333   0      83660      41336       7740          0          0 Inline Power
 202   0       2900       3112       7668          0          0 HRPC hrcli-event
 203   0   29840964   29840480       7652          0          0 HRPC rfs request
  38   0        340          0       7508          0          0 Exception contro
 259   0        532        232       7468          0          0 UDLD
 277   0        448        232       7384          0          0 CEF LC IPC Backg
 293   0        448        232       7384          0          0 Dot1x Supplicant
 432   0        188          0       7384          0          0 hulc_eee_monitor
 236   0        448        232       7384          0          0 loopdetect
 392   0        216          0       7384          0          0 Online Diag EEM
 295   0        448        232       7384          0          0 Dot1x Supplicant
 294   0        448        232       7384          0          0 Dot1x Supplicant
 241   0      46708        232       7384          0          0 Spanning Tree
 260   0        216          0       7384          0          0 Port-Security
 267   0        216          0       7384          0          0 XDR background p
 119   0       1076        872       7372          0          0 HRPC hpsecure re
 334   0        156          0       7324          0          0 Marvell wk-a Pow
 111   0      63180      63112       7236          0          0 HRPC hdm non blo
  98   0          0          0       7228          0          0 HRPC asic-stats
  16   0          0          0       7228          0          0 ARP Background
  66   0          0          0       7224          0          0 Transport Port A
 159   0          0          0       7212          0          0 HRPC dns-as requ
 344   0        232        232       7204          0          0 Crypto Support
 321   0          0          0       7196          0          0 SNMP Timers
 308   0     985356     985356       7196          0          0 NIST rng proc
 342   0        232        232       7188          0          0 ENABLE AAA
 125   0        232        232       7180          0          0 HVLAN Mapped Vla
 105   0          0    1571164       7176          0     891684 hrpc -> response
  96   0      66044        232       7176          0          0 HULC Thermal Pro
  44   0        232        232       7168          0          0 SMART
  43   0          0          0       7168          0          0 client_entity_se
 136   0          0          0       7168          0          0 HULC PM Vector P
 205   0        232        232       7168          0          0 Power RPS Proces
 209   0        232        232       7168          0          0 AAA Server
 210   0          0          0       7168          0          0 AAA ACCT Proc
 214   0          0          0       7168          0          0 IP Admin SM Proc
 212   0          0          0       7168          0          0 Webauth client
 204   0          0          0       7168          0          0 HRFS OIR Proc
 134   0        232        232       7168          0          0 HLFM aging proce
 179   0          0          0       7168          0          0 HRPC hled reques
  42   0        232        232       7168          0          0 GraphIt
 215   0          0          0       7168          0          0 hci usb process
 222   0        232        232       7168          0          0 Dot1x Mgr Proces
 223   0          0          0       7168          0          0 EAP Framework
 224   0          0          0       7168          0          0 EAP Test
 178   0          0          0       7168          0          0 HRPC l2pt qnq rp
 177   0          0          0       7168          0          0 HRPC vqpc reques
 137   0          0          0       7168          0          0 HPM Msg Retry Pr
 131   0          0          0       7168          0          0 Strider Tcam Mem
  60   0        232        232       7168          0          0 Compute load avg
 182   0     138816     138816       7168          0          0 hl3mm
 219   0        232        232       7168          0          0 AAA Dictionary R
 176   0          0          0       7168          0          0 HRPC hudld reque
 126   0        232        232       7168          0          0 Vlan shutdown Pr
  61   0          0          0       7168          0          0 Per-minute Jobs
 235   0          0          0       7168          0          0 IPv6 ping proces
 221   0          0          0       7168          0          0 DHCP Snooping db
 237   0          0          0       7168          0          0 PPPOE IA
 175   0        232        232       7168          0          0 S/W Bridge Proce
 130   0       3624          0       7168          0          0 Hulc ILP Alchemy
 240   0          0          0       7168          0          0 RIB LM VALIDATE
 128   0          0          0       7168          0          0 HULC VLAN REF Ba
 242   0        232        232       7168          0          0 SpanTree Msg
 243   0          0          0       7168          0          0 Spanning Tree St
 244   0          0          0       7168          0          0 avc_dns_as_clien
 245   0    1894472          0       7168          0          0 802.1x switch
 140   0    1294272    1125664       7168          0          0 HPM Stack Sync P
  65   0          0          0       7168          0          0 AggMgr Process
 248   0          0          0       7168          0          0 IPv6 Platform Ho
 249   0          0          0       7168          0          0 HRPC dai request
 122   0       4640          0       7168          0          0 HULC EMAC Proces
 251   0          0          0       7168          0          0 HRPC power down
 173   0          0          0       7168          0          0 HRPC stp_cli req
 253   0          0          0       7168          0          0 HRPC ip source g
 181   0          0          0       7168          0          0 HRPC hl3mm reque
 255   0          0          0       7168          0          0 HULC Open flow S
 256   0          0          0       7168          0          0 HRPC ofsdn reque
 121   0          0          0       7168          0          0 HRPC emac reques
 201   0          0          0       7168          0          0 Hulc Blue Beacon
 231   0          0          0       7168          0          0 ICMP event handl
  47   0          0          0       7168          0          0 Dynamic ARP Insp
 200   0          0          0       7168          0          0 Hulc Reload Mana
 171   0        232        232       7168          0          0 Hulc Storm Contr
 263   0          0          0       7168          0          0 Link State Group
 264   0          0          0       7168          0          0 MMN bkgrd proces
 265   0        232        232       7168          0          0 Ethchnl
  34   0          0          0       7168          0          0 IPC Loadometer
 195   0          0          0       7168          0          0 HULC SNMP Proces
 268   0     345800     346064       7168          0          0 XDR mcast
 269   0   73452820   73452820       7168          0          0 XDR LC PRIO Crit
 194   0          0          0       7168          0          0 HRPC snmp reques
 198   0          0          0       7168          0          0 HRPC span reques
 239   0        232        232       7168          0          0 REP Topology cha
 193   0          0          0       7168          0          0 HRPC show_forwar
  32   0          0          0       7168          0          0 IPC Seat TX Cont
 170   0          0          0       7168          0          0 HRPC sdm request
 276   0          0          0       7168          0          0 CEF RF HULC Conv
  46   0          0          0       7168          0          0 ARP Snoop
 278   0   26488600   26488600       7168          0          0 IP ACL XDR LC Ba
  58   0          0          0       7168          0          0 ICC Nego
 280   0          0          0       7168          0          0 IPv6 ACL RP Proc
 118   0     150336      63308       7168          0          0 Hulc Port-Securi
 254   0          0          0       7168          0          0 HULC IP Source g
 167   0          0          0       7168          0          0 HRPC rep request
 116   0          0          0       7168          0          0 RTTYC Flush
 115   0          0          0       7168          0          0 IPC RTTYC Messag
  67   0          0          0       7168          0          0 Token Daemon
  52   0        232        232       7168          0          0 TTY Background
 165   0          0          0       7168          0          0 HRPC lldp reques
 192   0          0          0       7168          0          0 HRPC dtp request
 290   0          0          0       7168          0          0 SVM UT Process
 291   0          0          0       7168          0          0 CMD HANDLER
 292   0          0          0       7168          0          0 Socket Timers
  59   0          0          0       7168          0          0 Net Input
 189   0          0          0       7168          0          0 HRPC obfl reques
 188   0          0          0       7168          0          0 HL3U PBR n-h res
  48   0          0          0       7168          0          0 Critical Bkgnd
 297   0          0          0       7168          0          0 EPM Downlad Mana
 112   0       1240       1240       7168          0          0 HRPC hdm blockin
 186   0       8940          0       7168       5076          0 HRPC IPv6 Unicas
 300   0          0          0       7168          0          0 Timer Handler Pr
 110   0       1000        464       7168          0          0 HULC Device Mana
 302   0        232        232       7168          0          0 Flow Exporter Ti
 109   0        232        232       7168          0          0 apm86xxx_enet_pr
 108   0          0          0       7168          0          0 HRPC hcomp reque
 164   0          0          0       7168          0          0 HRPC epm vlan pl
 306   0          0          0       7168          0          0 DCT Bkgd Process
 107   0     231148  217126908       7168          0     561744 hrpc <- response
  36   0     525056     525056       7168          0          0 PrstVbl
  30   0          0          0       7168          0          0 IPC Check Queue
 106   0          0          0       7168          0          0 hrpc -> request
 162   0          0          0       7168          0          0 Dot1X Msg Retry
 161   0     203692     100552       7168      10152          0 HRPC dot1x reque
 313   0          0          0       7168          0          0 RARP Input
 160   0          0          0       7168          0          0 avc_dns_as_hulc
 172   0        232        232       7168          0          0 HSTP Sync Proces
 316   0          0          0       7168          0          0 MAB Framework
 158   0       4380       3412       7168          0          0 HULC CISP Proces
  35   0          0          0       7168          0          0 IPC Session Deta
 319   0        232        232       7168          0          0 REP LSL Proc
  68   0          0          0       7168          0          0 HRPC pppoeia req
  33   0          0          0       7168          0          0 IPC Keep Alive M
 322   0          0          0       7168          0          0 HRPC dhcp snoopi
 323   0          0          0       7168          0          0 HULC DHCP Snoopi
 324   0          0          0       7168          0          0 HRPC archive-dnl
 325   0          0          0       7168          0          0 HRPC sisf reques
 326   0          0          0       7168          0          0 HULC SISF Proces
 327   0          0          0       7168          0          0 HULC SISF Source
 157   0          0          0       7168          0          0 HRPC cdp request
 329   0          0          0       7168          0          0 IGMPSN MRD
 156   0          0          0       7168          0          0 HRPC backup inte
 331   0          0          0       7168          0          0 IGMPQR
 155   0          0          0       7168          0          0 HRPC aim request
  56   0          0          0       7168          0          0 IF-MGR event pro
 138   0          0          0       7168          0          0 hpm bulk_vlan_st
  28   0          0          0       7168          0          0 IPC Process leve
 336   0          0          0       7168          0          0 MRD
  54   0          0          0       7168          0          0 Cat6k NTI ICC pr
 338   0        232        232       7168          0          0 AAA Cached Serve
  27   0       7660      10444       7168          0          0 IPC Deferred Por
 340   0          0          0       7168          0          0 HRPC hl2mcm igmp
 341   0          0          0       7168          0          0 HRPC hl2mcm mlds
 272   0          0          0       7168          0          0 FRR Background P
  69   0          0          0       7168          0          0 HC Counter Timer
 169   0          0          0       7168          0          0 HULC REP monitor
 345   0          0          0       7168          0          0 IPSECv6 PS Proc
 346   0          0          0       7168          0          0 Key chain liveke
 347   0        232        232       7168          0          0 LINE AAA
 152   0          0          0       7168          0          0 HACL Queue Proce
  26   0          0          0       7168          0          0 IPC Periodic Tim
 101   0        716        232       7168          0          0 FlexStack Hotswa
 100   0        232        232       7168          0          0 Horizontal Stack
 151   0          0          0       7168          0          0 hl3mm_rp
 184   0      17880          0       7168      10152          0 HRPC hl3u reques
 282   0          0          0       7168          0          0 AAA System Acct
  95   0     714840     823116       7168          0          0 RedEarth Rx Mana
  94   0     822796     820640       7168          0          0 RedEarth Tx Mana
  93   0        232        232       7168          0          0 Stack DI Update
  92   0          0          0       7168          0          0 PSP Timer
  24   0          0          0       7168          0          0 IPC Session Serv
 360   0          0          0       7168          0          0 IKEv2 AAA handle
  91   0          0          0       7168          0          0 Draught link sta
 362   0        232        568       7168          0          0 CRYPTO MAP FREE
  90   0          0          0       7168          0          0 HRPC ipadm reque
  23   0          0          0       7168          0          0 IPC Dynamic Cach
  22   0          0          0       7168          0          0 IPC Mcast Pendin
 150   0          0          0       7168          0          0 hl3mm_mrib_rp
  21   0          0          0       7168          0          0 IPC Event Notifi
  20   0          0          0       7168          0          0 IFS Agent Manage
 369   0          0          0       7168          0          0 Crypto PAS Proc
  89   0          0          0       7168          0          0 HRPC xcvr reques
 371   0          0          0       7168          0          0 HRPC x_setup req
 372   0        232        232       7168          0          0 REP Switch Helpe
 149   0     165312     165312       7168          0          0 hl3mm_mfib_ipv6
  87   0          0          0       7168          0          0 HRPC actual powe
  86   0          0          0       7168          0          0 HRPC EnergyWise
  17   0          0          0       7168          0          0 AAA_SERVER_DEADT
 377   0        232        232       7168          0          0 AAA SEND STOP EV
  85   0          0          0       7168          0          0 HPM FEC LOAD SHA
 289   0          0          0       7168          0          0 Critical Auth
 380   0        232        232       7168          0          0 DHCP Security He
 148   0          0          0       7168          0          0 HRPC hl3mm ipv6
 144   0     117024     117024       7168          0          0 HRPC pm-counters
  15   0          0          0       7168          0          0 ARP Input
  84   0          0          0       7168          0          0 HRPC hnetwpol re
  14   0          0          0       7168          0          0 Image License br
 386   0          0          0       7168          0          0 Syslog Traps
 387   0        232        232       7168          0          0 FEX Logger Proce
 388   0          0          0       7168          0          0 HCD Process
 389   0          0          0       7168          0          0 HRPC cable diagn
 390   0          0          0       7168          0          0 HRPC archive_dnl
 391   0          0        360       7168          0          0 DiagCard5/-1
 296   0          0          0       7168          0          0 DSensor HTTP
  83   0          0          0       7168          0          0 HULC multifs pro
 394   0          0          0       7168          0          0 HULC FNF
  13   0 1613010752 1613010752       7168          0          0 Licensing Auto U
 396   0        232        232       7168          0          0 dhcp snooping sw
 397   0          0          0       7168          0          0 ONEP_DPSS_SERVIC
 398   0          0          0       7168          0          0 HRPC onep reques
  82   0       3208       3208       7168          0          0 HRPC Multi-FS Sy
  12   0          0          0       7168          0          0 License Client N
  11   0          0          0       7168          0          0 HUSB Console
  81   0        232        232       7168          0          0 HLPIP Sync Proce
  80   0       1196       1196       7168          0          0 HRPC lpip reques
 404   0          0          0       7168          0          0 Call Home Timer
   9   0        232        232       7168          0          0 Timers
 145   0          0          0       7168          0          0 hpm vp events ca
   8   0          0          0       7168          0          0 DiscardQ Backgro
  79   0          0          0       7168          0          0 Hulc EEM Process
  78   0        232        232       7168          0          0 Porter Power Man
  77   0          0          0       7168          0          0 HRPC power_mgmt
 117   0          0          0       7168          0          0 HRPC hautosecure
   5   0          0          0       7168          0          0 IPC ISSU Dispatc
   4   0          0          0       7168          0          0 Retransmission o
  76   0        232        232       7168          0          0 DownWhenLooped
  75   0        232        232       7168          0          0 APM 86392 RTC
  70   0          0          0       7168          0          0 SFF8472
 247   0          0          0       7168          0          0 HRPC IPv6 Host r
 220   0          0          0       7168          0          0 DHCP Snooping
 431   0          0          0       7168          0          0 HRPC eee request
 421   0          0          0       7168          0          0 DiagCard6/-1
 208   0          0          0       7104          0          0 MDFS LC Download
 436   0          0          0       7084          0          0 IPv6 Access Cont
 318   0          0          0       7040          0          0 DiagCard3/-1
 257   0          0          0       7040          0          0 DiagCard1/-1
 351   0          0          0       7040          0          0 DiagCard4/-1
 270   0          0          0       7040          0          0 DiagCard2/-1
 395   0          0          0       7040          0          0 DiagCard7/-1
 438   0          0          0       7040          0          0 MFIB PROCESS IPC
 441   0          0          0       7040          0          0 OBFL I/O Buffer
 234   0        216          0       4384          0          0 ADJ NSF process
 433   0        232        232       4224          0          0 STP FAST TRANSIT
  63   0          0          0       4168          0          0 Inode Table Dest
 250   0   93472920          0       4168          0          0 HULC DAI Process
 142   0        232        232       4168          0          0 HPM if_num mappi
  10   0          0          0       4168          0          0 WATCH_AFS
 147   0          0          0       4168          0          0 hl3md_rpfq_thrl_
   2   0        232        232       4168          0          0 Load Meter
 166   0          0          0       4168          0          0 HRPC system mtu
 434   0        232        232       4168          0          0 CSRT RAPID TRANS
                                 97743564 Total

 SWITCH#sh log
Syslog logging: enabled (0 messages dropped, 1 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)

No Active Message Discriminator.



No Inactive Message Discriminator.


    Console logging: disabled
    Monitor logging: level debugging, 0 messages logged, xml disabled,
                     filtering disabled
    Buffer logging:  level debugging, 153627 messages logged, xml disabled,
                    filtering disabled
    Exception Logging: size (4096 bytes)
    Count and timestamp logging messages: disabled
    File logging: disabled
    Persistent logging: disabled

No active filter modules.

    Trap logging: level informational, 153528 message lines logged
        Logging to 10.25.8.90  (udp port 514, audit disabled,
              link up),
              152860 message lines logged,
              0 message lines rate-limited,
              0 message lines dropped-by-MD,
              xml disabled, sequence number disabled
              filtering disabled
        Logging Source-Interface:       VRF Name:

Log Buffer (4096 bytes):
POWER_INVALID: Interface Gi5/0/42: invalid power sense 51411 milliwatts current 999 mA voltage 51463 mV ( SWITCH-5)
Apr 20 10:34:28: %ILPOWER-5-SENSE_POWER_INVALID: Interface Gi5/0/44: invalid power sense 30260 milliwatts current 588 mA voltage 51463 mV ( SWITCH-5)
Apr 20 10:34:28: %ILPOWER-5-SENSE_POWER_INVALID: Interface Gi5/0/45: invalid power sense 51411 milliwatts current 999 mA voltage 51463 mV ( SWITCH-5)
Apr 20 10:34:28: %ILPOWER-5-SENSE_POWER_INVALID: Interface Gi5/0/47: invalid power sense 51411 milliwatts current 999 mA voltage 51463 mV ( SWITCH-5)
Apr 20 10:34:28: %ILPOWER-5-SENSE_POWER_INVALID: Interface Gi5/0/48: invalid power sense 30105 milliwatts current 585 mA voltage 51463 mV ( SWITCH-5)
Apr 20 10:34:31: %ILPOWER-5-SENSE_POWER_INVALID: Interface Gi5/0/41: invalid power sense 51433 milliwatts current 999 mA voltage 51485 mV ( SWITCH-5)
Apr 20 10:34:31: %ILPOWER-5-SENSE_POWER_INVALID: Interface Gi5/0/42: invalid power sense 32950 milliwatts current 640 mA voltage 51485 mV ( SWITCH-5)
Apr 20 10:34:31: %ILPOWER-5-SENSE_POWER_INVALID: Interface Gi5/0/45: invalid power sense 51433 milliwatts current 999 mA voltage 51485 mV ( SWITCH-5)
Apr 20 10:34:34: %ILPOWER-5-SENSE_POWER_INVALID: Interface Gi5/0/41: invalid power sense 51407 milliwatts current 999 mA voltage 51459 mV ( SWITCH-5)
Apr 20 10:34:34: %ILPOWER-5-SENSE_POWER_INVALID: Interface Gi5/0/42: invalid power sense 51407 milliwatts current 999 mA voltage 51459 mV ( SWITCH-5)
Apr 20 10:34:34: %ILPOWER-5-SENSE_POWER_INVALID: Interface Gi5/0/44: invalid power sense 51407 milliwatts current 999 mA voltage 51459 mV ( SWITCH-5)
Apr 20 10:34:34: %ILPOWER-5-SENSE_POWER_INVALID: Interface Gi5/0/45: invalid power sense 51407 milliwatts current 999 mA voltage 51459 mV ( SWITCH-5)
Apr 20 10:34:34: %ILPOWER-5-SENSE_POWER_INVALID: Interface Gi5/0/47: invalid power sense 51407 milliwatts current 999 mA voltage 51459 mV ( SWITCH-5)
Apr 20 10:34:34: %ILPOWER-5-SENSE_POWER_INVALID: Interface Gi5/0/48: invalid power sense 51407 milliwatts current 999 mA voltage 51459 mV ( SWITCH-5)
Apr 20 10:34:37: %ILPOWER-5-SENSE_POWER_INVALID: Interface Gi5/0/42: invalid power sense 33191 milliwatts current 645 mA voltage 51459 mV ( SWITCH-5)
Apr 20 10:34:37: %ILPOWER-5-SENSE_POWER_INVALID: Interface Gi5/0/44: invalid power sense 51407 milliwatts current 999 mA voltage 51459 mV ( SWITCH-5)
Apr 20 10:34:37: %ILPOWER-5-SENSE_POWER_INVALID: Interface Gi5/0/47: invalid power sense 51407 milliwatts current 999 mA voltage 51459 mV ( SWITCH-5)
Apr 20 10:34:37: %ILPOWER-5-SENSE_POWER_INVALID: Interface Gi5/0/48: invalid power sense 51407 milliwatts current 999 mA voltage 51459 mV ( SWITCH-5)
Apr 20 10:34:40: %ILPOWER-5-SENSE_POWER_INVALID: Interface Gi5/0/41: invalid power sense 51338 milliwatts current 999 mA voltage 51390 mV ( SWITCH-5)
Apr 20 10:34:40: %ILPOWER-5-SENSE_POWER_INVALID: Interface Gi5/0/42: invalid power sense 51338 milliwatts current 999 mA voltage 51390 mV ( SWITCH-5)
Apr 20 10:34:40: %ILPOWER-5-SENSE_POWER_INVALID: Interface Gi5/0/47: invalid power sense 51338 milliwatts current 999 mA voltage 51390 mV ( SWITCH-5)
Apr 20 10:34:43: %ILPOWER-5-SENSE_POWER_INVALID: Interface Gi5/0/47: invalid power sense 51433 milliwatts current 999 mA voltage 51485 mV ( SWITCH-5)
Apr 20 10:34:46: %ILPOWER-5-SENSE_POWER_INVALID: Interface Gi5/0/41: invalid power sense 51268 milliwatts current 999 mA voltage 51320 mV ( SWITCH-5)
Apr 20 10:34:46: %ILPOWER-5-SENSE_POWER_INVALID: Interface Gi5/0/42: invalid power sense 51268 milliwatts current 999 mA voltage 51320 mV ( SWITCH-5)
Apr 20 10:34:46: %ILPOWER-5-SENSE_POWER_INVALID: Interface Gi5/0/44: invalid power sense 51268 milliwatts current 999 mA voltage 51320 mV ( SWITCH-5)
Apr 20 10:34:46: %ILPOWER-5-SENSE_POWER_INVALID: Interface Gi5/0/47: invalid power sense 51268 milliwatts current 999 mA voltage 51320 mV ( SWITCH-5)
Apr 20 10:34:46: %ILPOWER-5-SENSE_POWER_INVALID: Interface Gi5/0/48: invalid power sense 51268 milliwatts current 999 mA voltage 51320 mV ( SWITCH-5)
 SWITCH#

 

Nothing wrong with the memory and CPU utilization.  Ideal output, actually.  

The output to the "sh logs", however, is a different story.  

What are these PED spamming the logs?  Tell us more about them?

Can I get the output to the following commands:

remote command 5 sh version
remote command 5 sh post

Leo, I don't know what you mean by PED. Here are the results of the show commands you requested.

 

Switch#remote command 5 sh ver
Switch : 5 :
------------
Cisco IOS Software, C2960X Software (C2960X-UNIVERSALK9-M), Version 15.2(7)E5, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2021 by Cisco Systems, Inc.
Compiled Mon 13-Sep-21 09:41 by mcpre

ROM: Bootstrap program is C2960X boot loader
BOOTLDR: C2960X Boot Loader (C2960X-HBOOT-M) Version 15.2(6r)E, RELEASE SOFTWARE (fc1)

HP-291-SW1-5 uptime is 3 days, 8 hours, 21 minutes
System returned to ROM by power-on
System restarted at 23:14:49 EDT Mon Apr 18 2022
System image file is "flash:c2960x-universalk9-mz.152-7.E5.bin"
Last reload reason: power-on



This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C2960X-48FPD-L (APM86XXX) processor (revision M0) with 524288K bytes of memory.
Processor board ID FOC2009Z0YZ
Last reset from power-on
2 Virtual Ethernet interfaces
1 FastEthernet interface
354 Gigabit Ethernet interfaces
24 Ten Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address       : 00:CA:E5:31:1E:80
Motherboard assembly number     : 73-16686-04
Power supply part number        : 341-0527-01
Motherboard serial number       : FOC200871TV
Power supply serial number      : DCB19516AF6
Model revision number           : M0
Motherboard revision number     : B0
Model number                    : WS-C2960X-48FPD-L
Daughterboard assembly number   : 73-14200-03
Daughterboard serial number     : FOC200846KT
System serial number            : FOC2009Z0YZ
Top Assembly Part Number        : 68-100463-01
Top Assembly Revision Number    : B0
Version ID                      : V05
CLEI Code Number                : CMMLJ00ARD
Daughterboard revision number   : A0
Hardware Board Revision Number  : 0x18

Configuration register is 0xF


Switch#remote command 5 sh post
Switch : 5 :
------------

Stored system POST messages:

Switch 5
---------

POST: MA BIST : Begin
POST: MA BIST : End, Status Passed

POST: TCAM BIST : Begin
POST: TCAM BIST : End, Status Passed

POST: Thermal, Fan Tests : Begin
POST: Thermal, Fan Tests : End, Status Passed

POST: PortASIC Stack Port Loopback Tests : Begin
POST: PortASIC Stack Port Loopback Tests : End, Status Passed

POST: PortASIC Port Loopback Tests : Begin
POST: PortASIC Port Loopback Tests : End, Status Passed

POST: EMAC Loopback Tests : Begin
POST: EMAC Loopback Tests : MAC Loopback Passed
POST: EMAC Loopback Tests : PHY Loopback Passed
POST: EMAC Loopback Tests : End, Status Passed


@michael.wellner wrote:

I don't know what you mean by PED.


Tell us more about those connected to ports Gi5/0/41, Gi5/0/42, Gi5/0/44, Gi5/0/47, Gi5/0/48.

What are they?

There are a couple of cameras, a couple of Extron devices and a Wi-Fi access point. 

hope that helps.  


@michael.wellner wrote:

There are a couple of cameras, a couple of Extron devices and a Wi-Fi access point.  


Well, the ports I have mentioned are in the logs with PoE faults.  

What happens to the CPU if these ports are disabled?  Is the response times better?

This switch is in a production environment. Shutting these ports down and testing will have to be done when i can schedule it. may take a while.  

Review Cisco Networking for a $25 gift card