Re: high memory utilization for fabric interconnect 6140

davidcheung · ‎04-09-2012

Hi all,

we have 2 fabric interconnect which runs as HA pair. One fabric interconnect of cluster can see high memory utilization 95% (the other is better and mem utilization is 70%)

Fabric-A(local-mgmt)# show processes

Cpu(s): 2.8% us, 16.4% sy, 0.8% ni, 79.4% id, 0.0% wa, 0.0% hi, 0.6% si

Mem: 3504756k total, 3373008k used, 131748k free, 119472k buffers

Swap: 0k total, 0k used, 0k free, 2240656k cached

when issue "shwo process" command, we see the following:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

4263 root 16 0 94708 7664 5152 S 11.4 0.2 3694:43 /isan/bin/pfm

4368 root 15 0 73764 4108 2712 S 5.7 0.1 3568:00 /isan/bin/fcpc

2833 root 15 0 80400 4660 3156 S 0.0 0.1 3:30.40 /isan/sbin/sysmgr -V

4258 root 16 0 2340 836 680 S 0.0 0.0 0:00.09 /isan/sbin/xinetd -syslog local7 -loop 250 -stayalive -reuse -dontfork

4260 root 15 0 86444 9856 4020 R 0.0 0.3 15:01.63 /isan/sbin/syslogd -n -m 0 -r

4261 root 16 0 72648 3092 2012 S 0.0 0.1 0:00.01 /isan/bin/sdwrapd

4262 root 17 1 1820 540 460 S 0.0 0.0 0:00.03 /usr/sbin/in.tftpd --verbosity 2 -l -c -s /bootflash

4293 root 16 0 76948 4056 2496 S 0.0 0.1 0:02.42 /isan/bin/vshd

4294 root 16 0 134m 7504 4380 S 0.0 0.2 0:04.06 /isan/bin/routing-sw/smm -f /isan/etc/routing-sw/smm_shm.cfg

4295 root 16 0 72928 3520 2188 S 0.0 0.1 0:13.20 /isan/bin/ssnmgr

4296 root 16 0 70824 2208 1612 S 0.0 0.1 0:11.28 /isan/bin/psshelper -s 219

4297 root 16 0 2324 1272 980 S 0.0 0.0 0:00.30 /isan/bin/lmgrd -l +/dev/null -z -c /isan/etc/licenses.base:/isan/etc/licenses

4298 root 16 0 71328 2852 2164 S 0.0 0.1 0:06.18 /isan/bin/licmgr

4299 root 16 0 76180 2788 2088 S 0.0 0.1 0:00.06 /isan/bin/fs_daemon

4300 root 16 0 70036 1948 1484 S 0.0 0.1 0:03.37 /isan/bin/feature_mgr

4301 root 16 0 79804 3764 2880 S 0.0 0.1 0:05.32 /isan/sbin/fcfwd -s

4302 root 16 0 69996 1640 1296 S 0.0 0.0 0:00.01 /isan/bin/confcheck

4303 root 16 0 72708 3220 2128 S 0.0 0.1 0:06.71 /isan/bin/capability

4304 root 16 0 70824 2204 1608 S 0.0 0.1 0:10.81 /isan/bin/psshelper -s 398

4312 root 16 0 2656 1552 1128 S 0.0 0.0 0:01.52 cisco -T localhost 10.1 3 -c /isan/etc/licenses.base:/isan/etc/licenses -lmgrd_port 6978 --lmgrd_start 4f4ed34c

4315 root 16 0 121m 45m 34m S 0.0 1.3 0:18.93 /isan/bin/routing-sw/clis -cli /isan/etc/routing-sw/cli

4328 root 16 0 1412 452 392 S 0.0 0.0 0:00.07 /sbin/klogd -2 -x

4345 root 16 0 97576 8008 4460 S 0.0 0.2 0:12.55 /isan/bin/zschk

4346 root 16 0 70188 1688 1324 S 0.0 0.0 0:15.33 /isan/bin/xmlma

4347 root 16 0 71032 2648 1676 S 0.0 0.1 2:05.46 /isan/bin/wwn_manager 2

4348 root 16 0 75384 6224 2748 S 0.0 0.2 0:12.57 /isan/bin/vmm

4349 root 16 0 74512 3516 2584 S 0.0 0.1 0:10.11 /isan/bin/vdc_mgr

4350 root 16 0 77580 3536 2688 S 0.0 0.1 0:06.70 /isan/bin/ttyd

4351 root 16 0 68720 1484 924 S 0.0 0.0 1:35.94 /isan/bin/sysinfo

4352 root -2 0 81296 2444 1712 S 0.0 0.1 250:18.21 /isan/bin/sensor

4354 root 16 0 71364 2488 1824 S 0.0 0.1 0:00.19 /isan/bin/res_mgr

4356 root 16 0 72924 3380 2200 S 0.0 0.1 0:00.28 /isan/bin/plugin

4357 root 16 0 82260 6520 3756 S 0.0 0.2 0:28.71 /isan/bin/pltfm_config

4358 root 16 0 68884 1396 1092 S 0.0 0.0 0:07.21 /isan/bin/plog_sup

4359 root 16 0 82936 2492 1908 S 0.0 0.1 0:01.23 /isan/bin/obfl_sup

4360 root 16 0 136m 12m 7264 S 0.0 0.4 0:16.26 /isan/bin/nppm

4361 root 15 0 91556 8868 4344 S 0.0 0.3 0:10.51 /isan/bin/routing-sw/npacl

4362 root 16 0 85040 4080 2624 S 0.0 0.1 0:12.37 /isan/bin/nohms

4363 root 16 0 72480 3108 2032 S 0.0 0.1 0:05.00 /isan/bin/mvsh

4364 root 16 0 72908 3504 2248 S 0.0 0.1 0:12.31 /isan/bin/msp

4365 root 16 0 80604 1700 1328 S 0.0 0.0 0:09.17 /isan/bin/ledmgr

4366 root 16 0 88360 12m 5192 S 0.0 0.4 0:49.82 /isan/bin/ifmgr

4367 root -2 0 144m 9276 3072 S 0.0 0.3 962:01.67 /isan/bin/gatosusd -M L -L 63

4369 root 16 0 72712 3188 2080 S 0.0 0.1 7:39.90 /isan/bin/evmc

4370 root 16 0 93732 6500 3996 S 0.0 0.2 0:12.35 /isan/bin/ethpc

4371 root 16 0 93676 15m 3904 S 0.0 0.4 0:44.88 /isan/bin/eth_dstats

4372 root 16 0 93500 11m 5148 S 0.0 0.3 0:17.21 /isan/bin/enm

4373 root 15 0 77232 3412 2608 S 0.0 0.1 0:00.06 /isan/bin/core_daemon

4374 root 16 0 70536 1920 1468 S 0.0 0.1 0:06.91 /isan/bin/bootvar

4375 root 16 0 70948 1664 1256 S 0.0 0.0 0:03.35 /isan/bin/bios_daemon

4376 root -2 0 88648 3804 1944 S 0.0 0.1 0:08.57 /isan/bin/athertonusd -M L -L 63

4377 root 16 0 81304 5124 3772 S 0.0 0.1 0:00.87 /isan/bin/ascii_cfg_server

4378 root -2 0 87732 3296 2228 S 0.0 0.1 0:25.61 /isan/bin/altos

4379 root 16 0 72436 2968 2112 S 0.0 0.1 0:19.04 /isan/bin/securityd

4380 root 16 0 81428 4748 3468 S 0.0 0.1 0:10.98 /isan/bin/cert_enrolld

4381 root 16 0 73104 3724 2472 S 0.0 0.1 0:11.49 /isan/bin/cssmgr

4382 root 16 0 73116 3500 2096 S 0.0 0.1 0:20.67 /isan/bin/aaad

4396 root 16 0 84528 9148 3768 S 0.0 0.3 0:03.90 /isan/bin/routing-sw/l3vm

4397 root 16 0 93652 14m 5080 S 0.0 0.4 0:30.20 /isan/bin/eth_span

4398 root 15 0 101m 26m 21m S 0.0 0.8 0:07.09 /isan/bin/routing-sw/u6rib

4399 root 15 0 123m 48m 41m S 0.0 1.4 1:54.94 /isan/bin/routing-sw/urib

4400 root 16 0 1424 392 340 S 0.0 0.0 0:29.69 /isan/bin/ptpd -p -b vlan4044

4403 root 16 0 112m 14m 12m S 0.0 0.4 0:12.89 /isan/bin/vms

4404 root 15 0 132m 42m 9.8m S 0.0 1.3 5:03.41 /isan/bin/fwm

4405 root 16 0 97732 7724 4956 S 0.0 0.2 0:15.27 /isan/bin/qd

4406 root 16 0 97080 7044 4200 S 0.0 0.2 1651:18 /isan/bin/stats_client

4408 root 16 0 90640 12m 5076 S 0.0 0.4 0:26.66 /isan/bin/vlan_mgr

4418 root 16 0 1480 532 428 S 0.0 0.0 0:16.88 /isan/bin/pmon -i -f /isan/etc/pmon.conf

4419 root 16 0 88316 9.8m 5768 S 0.0 0.3 0:08.57 /isan/bin/aclmgr

4420 root 16 0 98.7m 10m 6088 S 0.0 0.3 1:53.94 /isan/bin/afm

4421 root 16 0 87108 9428 6316 S 0.0 0.3 1:19.26 /isan/bin/eth_pcm

4428 root 16 0 82316 5224 3532 S 0.0 0.1 0:11.10 /isan/bin/ldapd

4430 root 15 0 144m 12m 3788 S 0.0 0.4 0:27.53 /isan/bin/routing-sw/am

4431 root 15 0 169m 33m 21m S 0.0 1.0 71:47.48 /isan/bin/snmpd -f -s udp:161 tcp:161 -e readonly -n linux_ip

4437 root 16 0 93280 14m 8784 S 0.0 0.4 0:11.40 /isan/bin/ipqosmgr

4438 root 16 0 113m 37m 7524 S 0.0 1.1 50:32.36 /isan/bin/ethpm

4450 root 26 10 78556 4468 3296 S 0.0 0.1 0:11.56 /isan/bin/callhome

4451 root 16 0 82120 5648 3976 S 0.0 0.2 1:02.47 /isan/bin/vsan_mgr

4453 root 16 0 67956 868 720 S 0.0 0.0 0:00.01 /isan/bin/core_client -t

4488 root 16 0 73020 3868 2608 S 0.0 0.1 0:14.47 /isan/bin/epp

4489 root 15 0 75116 4256 2924 S 0.0 0.1 0:25.99 /isan/bin/dom_mgr

4490 root 16 0 92564 13m 7572 S 0.0 0.4 7:49.23 /isan/bin/port_mgr

4491 root 16 0 84660 8292 3816 S 0.0 0.2 0:00.32 /isan/bin/rib

4492 root 16 0 82672 7892 5544 S 0.0 0.2 1:42.97 /isan/bin/pcm

4523 root 20 0 122m 9692 4168 S 0.0 0.3 0:11.35 /isan/bin/routing-sw/arp

4524 root 15 0 105m 11m 4960 S 0.0 0.3 0:10.91 /isan/bin/routing-sw/icmpv6

4525 root 20 0 213m 56m 7272 S 0.0 1.6 87:30.72 /isan/bin/routing-sw/netstack /isan/etc/routing-sw/pm.cfg

4568 root 16 0 119m 5444 3756 S 0.0 0.2 0:14.87 /isan/bin/radiusd

4569 root 15 0 102m 22m 19m S 0.0 0.7 12:08.24 /isan/bin/cfs

4570 root 15 0 129m 14m 6296 S 0.0 0.4 5:30.94 /isan/bin/routing-sw/igmp

4571 root 16 0 68300 1020 820 S 0.0 0.0 0:00.00 /isan/bin/routing-sw/netstack_dummy 267

4572 root 16 0 68300 1020 820 S 0.0 0.0 0:00.01 /isan/bin/routing-sw/netstack_dummy 269

4573 root 0 -20 119m 5636 3936 S 0.0 0.2 0:01.51 /isan/bin/ntp 5

4574 root 16 0 137m 6004 4092 S 0.0 0.2 0:12.23 /isan/bin/otm

4575 root 16 0 68300 1020 820 S 0.0 0.0 0:00.01 /isan/bin/routing-sw/netstack_dummy 263

4576 root 16 0 68300 1020 820 S 0.0 0.0 0:00.01 /isan/bin/routing-sw/netstack_dummy 271

4585 root 16 0 86484 10m 5812 S 0.0 0.3 0:04.91 /isan/bin/npm

4587 root 16 0 123m 6720 4748 S 0.0 0.2 0:36.47 /isan/bin/cdpd

4588 root 15 0 94184 17m 5492 S 0.0 0.5 76:51.48 /isan/bin/satmgr

4589 root 15 0 90668 14m 4832 S 0.0 0.4 0:53.38 /isan/bin/lldp

4590 root 15 0 94212 16m 11m S 0.0 0.5 3:00.82 /isan/bin/stp

4591 root 15 0 78740 4120 2972 S 0.0 0.1 0:00.03 /isan/bin/routing-sw/dcos-xinetd -syslog local7 -loop 250 -stayalive -reuse -dontfork -f /etc/dcos-xinetd.conf -pidf

4592 root 0 -20 3656 3656 2712 S 0.0 0.1 0:08.78 /isan/bin/ntpd -g -f /mnt/pss/ntp.drift -A -n

4651 root 16 0 92580 14m 5820 S 0.0 0.4 0:32.83 /isan/bin/sifmgr

4684 root 11 -5 44868 22m 15m S 0.0 0.7 8:19.72 svc_sam_controller --x /opt/db/sam.config

4685 root 21 3 278m 138m 38m S 0.0 4.1 270:35.75 svc_sam_dme --x

4689 root 18 3 175m 35m 23m S 0.0 1.0 1:43.15 svc_sam_dcosAG --x

4700 root 19 3 87264 37m 18m S 0.0 1.1 65:22.59 svc_sam_bladeAG --x

4702 root 18 3 173m 37m 25m S 0.0 1.1 0:56.74 svc_sam_portAG --x

4715 root 24 4 173m 36m 23m S 0.0 1.1 279:54.92 svc_sam_statsAG --x

4720 root 19 3 50880 23m 16m S 0.0 0.7 0:22.83 svc_sam_hostagentAG --x

4721 root 19 3 66024 29m 16m S 0.0 0.9 15:29.65 svc_sam_nicAG --x

4726 root 19 3 46836 24m 16m S 0.0 0.7 0:14.13 svc_sam_licenseAG --x

4740 root 23 3 171m 31m 21m S 0.0 0.9 0:13.11 svc_sam_extvmmAG --x

4742 root 19 3 2252 1128 924 S 0.0 0.0 5:00.53 /bin/sh /isan/bin/httpd.sh --x

4747 root 19 3 192m 56m 36m S 0.0 1.7 0:17.15 svc_sam_sessionmgrAG --x

4752 root 19 3 143m 27m 19m S 0.0 0.8 0:02.27 svc_sam_pamProxy --x vlan4044

4753 root 19 3 10156 836 632 S 0.0 0.0 0:00.00 sfcbd -c /isan/etc/dmtf/sblim-sfcb-1.2.4-100407/sfcb.cfg

4760 root 15 0 4860 2884 1108 S 0.0 0.1 3:30.41 dhcpd -f -cf /isan/etc/dhcpd.conf -wf /opt/db/sam.config -lf /isan/etc/dhcpd.leases vlan4047 vlan4044

4763 root 20 3 10156 740 552 S 0.0 0.0 0:00.00 sfcbd -c /isan/etc/dmtf/sblim-sfcb-1.2.4-100407/sfcb.cfg

4764 root 19 3 14140 4804 712 S 0.0 0.1 0:00.01 sfcbd -c /isan/etc/dmtf/sblim-sfcb-1.2.4-100407/sfcb.cfg

4765 root 21 3 44632 22m 15m S 0.0 0.7 0:11.54 svc_sam_rsdAG --x

4798 root 20 3 10188 932 720 S 0.0 0.0 0:00.00 sfcbd -c /isan/etc/dmtf/sblim-sfcb-1.2.4-100407/sfcb.cfg

5916 root 16 0 84304 8784 5060 S 0.0 0.3 0:02.31 /isan/bin/pvlan

5917 root 16 0 82728 6016 3912 S 0.0 0.2 0:31.51 /isan/bin/tacacsd

5918 root 16 0 91144 11m 5388 S 0.0 0.3 0:24.35 /isan/bin/eth_port_sec

5922 root 16 0 87476 9516 5772 S 0.0 0.3 0:33.99 /isan/bin/fcoe_mgr

5923 root 15 0 85284 9300 5116 S 0.0 0.3 4:53.83 /isan/bin/lacp

12685 root 16 0 1496 536 460 S 0.0 0.0 0:00.00 /sbin/getty -L ttyS0

14971 root 19 3 2236 676 532 S 0.0 0.0 0:00.00 /isan/bin/rotatelogs /var/sysmgr/sam_logs/error_log 2M

3481 root 15 0 6416 3044 2380 S 0.0 0.1 0:00.17 sshd:

user1@pts/5

3523 root 16 0 87808 54m 33m S 0.0 1.6 0:00.54 /isan/bin/ucssh --ucs-mgmt -p user1

4128 root 15 0 87808 53m 33m S 0.0 1.6 0:00.45 /isan/bin/ucssh --ucs-local-mgmt -t 30 -p user1

4511 root 16 0 87812 53m 33m S 0.0 1.6 0:00.00 /isan/bin/ucssh --ucs-local-mgmt -t 30 -p user1

I added all above mem percentage and the sum is about 47%. I can't find where are 48% mem utiization from?

Any help and suggestion is highly appreciated.

David

padramas · ‎04-09-2012

David,

It could be normal as Linux attempts to use most of the memory. Notice the value for buffers / cached field.

Mem: 3504756k total, 3373008k used, 131748k free, 119472k buffers

Swap: 0k total, 0k used, 0k free, 2240656k cached

You can find lot of excellent articles about linux memory management and buffers / cache mechanism.

Do we have the output from FI taken at regular intervals of time ?

Padma

davidcheung · ‎04-09-2012

The high mem utilization has been there for more 4 days. Always high. (BTW, i just logged a case with cisco TAC, Thank u very much for your help!)