Solved: Options for monitoring or retrieving system performance statistics from C-Series, EX-Series codecs

ANDREW BEEZLEY · ‎12-26-2012

Hello - I have been looking around to see if there is an option available today to retrieve system performance statistics (such as CPU Load, Memory Load, etc...) from TAA endpoint either via command line interface or via Dynamic XML Poll (http "getxml") from an endpoint. I understand that Cisco does not have an enterprise MIB available for these endpoints today and that the API is required to pull information from the codec however, I want to be able to pull CPU load and memory load from these endpoints dynamically. Is there a way to do that today?

Thank you in advance,

-Andrew

Martin Koch · ‎12-26-2012

Hello Andrew!

In general this is something I would not really worry about and be more skeptical if the

monitoring itself generate additional load = trouble.

As the systems are based on linux you have command line tools and the proc interface

you can use to get some stats out of the endpoint. I prefer looking at them if there is trouble

rather then trying to do some pro-active stuff, I think thats better handeld via the endpoints.

Example, login as root and try w or free

[Martin-Koch-EX90--ATEA-Vaas:~] $ w

00:17:02 up 14 days, 23 min, 1 user, load average: 0.00, 0.02, 0.05

USER TTY LOGIN@ IDLE JCPU PCPU WHAT

root pts/0 00:03 0.00s 0.31s 0.02s w

[Martin-Koch-EX90--ATEA-Vaas:~] $ free

total used free shared buffers cached

Mem: 510912 309832 201080 0 25768 156616

-/+ buffers/cache: 127448 383464

Swap: 0 0 0

also the proc interface is interesting:

[Martin-Koch-EX90--ATEA-Vaas:~] $ cat /proc/uptime

1210174.87 1193259.96

[Martin-Koch-EX90--ATEA-Vaas:~] $ cat /proc/loadavg

0.00 0.01 0.05 1/131 25905

[Martin-Koch-EX90--ATEA-Vaas:~] $ cat /proc/meminfo

MemTotal: 510912 kB

MemFree: 210312 kB

Buffers: 25580 kB

Cached: 156016 kB

SwapCached: 0 kB

Active: 70892 kB

Inactive: 115896 kB

...

you should be able to add an ssh key and auto login and get the information you need and

process them via a script and then use that data in your monitoring tool.

and sure you have the xstatus which you can also access via tsh or the http xmlapi, you find more info

about that in the api guide. As you wrote, there is not that much you get out via snmp.

The intention itself is right though, so maybe you could file a feature request for traps

and better pro-active monitoring, major thing I lack the most on C/TC is the lack of traffic/network/loss reports.

please rate answers!

Please remember to rate helpful responses and identify

View solution in original post

Martin Koch · ‎12-26-2012

Hello Andrew!

In general this is something I would not really worry about and be more skeptical if the

monitoring itself generate additional load = trouble.

As the systems are based on linux you have command line tools and the proc interface

you can use to get some stats out of the endpoint. I prefer looking at them if there is trouble

rather then trying to do some pro-active stuff, I think thats better handeld via the endpoints.

Example, login as root and try w or free

[Martin-Koch-EX90--ATEA-Vaas:~] $ w

00:17:02 up 14 days, 23 min, 1 user, load average: 0.00, 0.02, 0.05

USER TTY LOGIN@ IDLE JCPU PCPU WHAT

root pts/0 00:03 0.00s 0.31s 0.02s w

[Martin-Koch-EX90--ATEA-Vaas:~] $ free

total used free shared buffers cached

Mem: 510912 309832 201080 0 25768 156616

-/+ buffers/cache: 127448 383464

Swap: 0 0 0

also the proc interface is interesting:

[Martin-Koch-EX90--ATEA-Vaas:~] $ cat /proc/uptime

1210174.87 1193259.96

[Martin-Koch-EX90--ATEA-Vaas:~] $ cat /proc/loadavg

0.00 0.01 0.05 1/131 25905

[Martin-Koch-EX90--ATEA-Vaas:~] $ cat /proc/meminfo

MemTotal: 510912 kB

MemFree: 210312 kB

Buffers: 25580 kB

Cached: 156016 kB

SwapCached: 0 kB

Active: 70892 kB

Inactive: 115896 kB

...

you should be able to add an ssh key and auto login and get the information you need and

process them via a script and then use that data in your monitoring tool.

and sure you have the xstatus which you can also access via tsh or the http xmlapi, you find more info

about that in the api guide. As you wrote, there is not that much you get out via snmp.

The intention itself is right though, so maybe you could file a feature request for traps

and better pro-active monitoring, major thing I lack the most on C/TC is the lack of traffic/network/loss reports.

please rate answers!

Please remember to rate helpful responses and identify

Danny De Ridder · ‎12-27-2012

Hello,

the codecs are distributed systems, therefore looking at the PowerPC, the central nerve system of the codec running linux, only shows one processor. There are also audio and video DSPs. The commands given above all pertain to the load on the PowerPC. From the DSPs there's indeed not a lot of info one can get, at least I am not aware of any commands to check their load etc.. An additional command to check the central CPU is sys-rtos which you can run from the admin shell.

sys-rtos all

Heap statistics

---------------

Total allocated: 9113600 bytes

Allocated and in use: 8229080 bytes

Allocated but not in use: 884520 bytes

Allocated by mmap: 14307328 bytes

*********** RTOS Commands ***********

Options to this command are:

General:

heap Display heap info

help Display this menu

RTOS status:

all Display all RTOS status

ms [name] Display Mutex Status [mutex name]

qs Display Queue Status

ss [name] Display Semaphore Status [semaphore name]

ts [name] Display Task Status [task name]

CPU load measurements:

cpu adhoc Display Ad Hoc CPU Load Status

RTOS service routines:

scp pid pri Change Priority for Task

scs pid sched pri Change Scheduler for Task

src Retrieve Current Time in Ticks

srt tid Resume Task (tid = hex (0xn) format)

sst tid Suspend Task (tid = hex (0xn) format)

stt tid Terminate Task (tid = hex (0xn) format)

Mutex Address Status #Taken #Blocked

-----------------------------------------------------------

mutexListLock 0x11d56ca8 locked 135 0

TTPARAMSdbswap 0x11d56c70 unlocked 0 0

cuil_handler_regist 0x11d5c640 unlocked 10006 0

cuilAddNameToNameMa 0x11d5c6b8 unlocked 323 0

cuil_command_mutex 0x11d5c900 unlocked 9925 0

cuilSetLitrValue 0x11d5c938 unlocked 0 0

protectmutex 0x11d7da00 unlocked 36 0

TTOSmutex 0x11d71940 unlocked 0 0

taskListLock 0x11d71bb0 unlocked 37 0

semListLock 0x11d72c58 unlocked 7 0

Option key mutex 0x11d73610 unlocked 6 0

releasekey 0x11d73888 unlocked 0 0

backlightmutex 0x11d738c0 unlocked 4 0

sessionmutex 0x11d73928 unlocked 9551 0

COM mutex 0x11d73978 unlocked 8 0

ledmutex 0x11d8ab40 unlocked 7 0

uhpihostWP 0x11d75e10 unlocked 124567 0

secmodMutex 0x11dad180 unlocked 1 0

GlobalLogLock 0x11dad1e8 unlocked 343 0

MainDBmutex 0x11dad548 unlocked 32787 0

MainDBlist 0x11dad580 unlocked 111 0

StatusHandler 0x11ee8450 unlocked 21511 118

WatchdogLock 0x11f8a0a8 unlocked 47786 0

timersystem_lock 0x11f8ba50 unlocked 717432 110

watchdogclientlock 0x11f8bb18 unlocked 95571 106

pool_0 0x11fa8338 unlocked 1445803 150

pool_1 0x11fa8370 unlocked 11236064 118

pool_2 0x11fa83a8 unlocked 9224 0

pool_3 0x11fa83e0 unlocked 2546 0

pool_4 0x11fa8418 unlocked 9673 0

pool_5 0x11fa8450 unlocked 1514 0

pool_6 0x11fa8488 unlocked 415 0

pool_7 0x11fa84c0 unlocked 8 0

pool_8 0x11d73dc0 unlocked 19122 0

pool_9 0x11d73df8 unlocked 0 0

pool_10 0x11fa8600 unlocked 0 0

globalLock 0x11fb7b20 unlocked 2 0

OutRecvLock 0x11fb7b58 unlocked 8 0

socklib 0x48200890 unlocked 0 0

VidMonAgent 0x4820b720 unlocked 9 0

protectmutex 0x4820db08 unlocked 1571 0

socklib 0x11fd1700 unlocked 7578 5

comrouterRoutingMut 0x11fd1928 unlocked 0 0

TMQueueMutex 0x12302d00 unlocked 0 0

QueueMutex 0x12302e38 unlocked 3183 0

DeleteMutex 0x12302e70 unlocked 1592 0

VidMonAgent 0x123159f0 unlocked 0 0

VidMonAgent 0x123200f0 unlocked 0 0

VidMonAgent 0x1232a7f0 unlocked 0 0

VidMonAgent 0x12334f58 unlocked 0 0

VidMonAgent 0x12334f90 unlocked 0 0

VidMonAgent 0x12334fc8 unlocked 0 0

VidMonAgent 0x12360460 unlocked 0 0

VidMonAgent 0x1236ab60 unlocked 0 0

VidMonAgent 0x12375260 unlocked 0 0

VidMonAgent 0x1237f960 unlocked 0 0

VidMonAgent 0x1238a060 unlocked 0 0

TMQueueMutex 0x1238a288 unlocked 0 0

TMQueueMutex 0x12395730 unlocked 0 0

TMQueueMutex 0x12397dd0 unlocked 0 0

CallLogDBmutex 0x4989ad20 unlocked 4585 0

CallLogDBlist 0x4989ad58 unlocked 5 0

cli_com_mutex 0x1215cfc8 unlocked 9 0

cli_com_a_mutex 0x1239bbe0 unlocked 9922 0

cli_lock 0x1239c4c8 unlocked 0 0

cli_com_mutex 0x1242bc80 unlocked 10277 0

cli_com_a_mutex 0x1242bcd0 unlocked 9965 0

cli_lock 0x1242bfa8 unlocked 19840 0

cli_com_mutex 0x12408360 unlocked 16 0

cli_com_a_mutex 0x124083b0 unlocked 348 0

cli_lock 0x12408470 unlocked 4 0

CuilHandlerGateway 0x123f30e8 unlocked 3 0

VidStreamLock 0x49e271a8 unlocked 744 0

cli_com_mutex 0x1247b1e0 unlocked 0 0

cli_com_a_mutex 0x12403630 unlocked 0 0

cli_lock 0x12403668 unlocked 0 0

cli_com_mutex 0x1246ef38 unlocked 5 0

cli_com_a_mutex 0x1246ef88 unlocked 0 0

cli_lock 0x12408a00 unlocked 6 0

CuilHandlerGateway 0x1247b460 unlocked 3 0

CuilHandlerGateway 0x12438ba8 unlocked 3 0

cli_com_mutex 0x1246be48 unlocked 1 0

cli_com_a_mutex 0x1246be98 unlocked 0 0

cli_lock 0x12400090 unlocked 0 0

List info: size 84, capacity 128, additions 109, removals 25 reallocations 7

Queue Status

------------

Name Max Curr Tot QFull Watermark

reaperQ 20 0 4 0 1

persistDb 3000 0 10 0 2

persistDb 3000 0 3 0 2

tshellc 256 0 9922 0 1

Qfsmmain 1000 0 361090 0 75

tshellc 256 0 9965 0 25

Qfsmmedia 1000 0 4988243 0 12

tshellc 256 0 348 0 1

Qfsmclksync 1000 0 415091 0 6

CUILCQ 1 0 0 0 0

tshellc 256 0 0 0 0

Semaphore name Count

-------------------------

XPathParse 1

CallIdTabSema 1

fsmTrace 1

bootRequest 0

Task Type Pri TID STACK PTR %CPU %PROC TIME+ STACK

--------------------------------------------------------------------------------

applstart OT 0 2367 (nil) 0.0 0.0 0:06.348 0/0

reaperT OT 0 2426 0x48032000 0.0 0.0 0:00.001 5408/131072

tipc-dev RR 24 2466 0x48056000 0.1 1.5 0:35.073 6848/131072

UhpiRx RR 29 2508 0x4807a000 0.4 9.0 3:02.886 5440/131072

persistDb OT 0 2510 0x4809e000 0.0 0.0 0:00.020 6624/131072

WatchdogThread OT 0 2511 0x480c2000 0.0 0.1 0:01.565 5776/131072

WatchdogKickerThread OT 0 2512 0x480e6000 0.0 0.1 0:01.321 5216/131072

TimerThread RR 29 2513 0x4810a000 0.1 1.3 0:31.608 5664/131072

FsmMedia RR 24 2515 0x48152000 0.5 9.5 3:00.297 10720/131072

FsmMain OT 0 2514 0x4812e000 0.1 2.9 1:00.912 64704/131072

FsmClksync RR 24 2516 0x48176000 0.1 1.5 0:35.214 5904/131072

Socket OT 0 2582 0x4819a000 0.0 0.0 0:00.818 8752/131072

ocspc OT 0 2583 0x481be000 0.0 0.0 0:00.842 5952/131072

CurlEngine OT 0 2584 0x48c57000 0.0 0.0 0:00.091 5872/131072

TshelldT OT 0 2588 0x4949f000 0.0 0.0 0:00.012 7216/131072

snmp OT 0 2586 0x4947b000 0.0 0.1 0:02.013 5360/131072

CCP-Register OT 0 2589 0x494c3000 0.0 0.0 0:00.000 5168/131072

CCP-OutboundConn OT 0 2591 0x49602000 0.0 0.0 0:00.000 5168/131072

TshelldC OT 0 2742 0x1248d000 0.0 0.2 0:02.770 23888/131072

TshelldC OT 0 2643 0x1240a000 3.7 73.6 1:50.681 23888/131072

TshelldC OT 0 2646 0x1243a000 0.0 0.1 0:00.692 23488/131072

persistDb OT 0 2647 0x49b44000 0.0 0.0 0:00.006 6624/131072

PBSens RR 19 2718 0x4a056000 0.0 0.0 0:00.000 5568/131072

IRSens RR 19 2720 0x4a079000 0.0 0.0 0:00.001 5472/131072

TshelldC OT 0 2754 0x124b0000 0.0 0.0 0:00.002 17136/131072

TshelldC OT 0 2766 0x124d3000 0.0 0.0 0:00.390 18720/131072

TshelldC OT 0 14317 0x124f6000 0.0 0.1 0:00.342 24112/131072

--------------------------------------------------------------------------------

Cpu: 5.0%proc, 4.5%other, 90.6%idle Total time: 10:56.466

Sorry, this function is not implemented yet.

ANDREW BEEZLEY · ‎12-28-2012

Hello Danny - Thank you for your information however, I was not able to execute the sys-rtos command from the admin-cli or from root. I searched the filesystem to see if perhaps there was a script that provided this command but came up empty handed. Can you provide additional details on how to execute the utility?

Thanks in advance,

-Andrew

ANDREW BEEZLEY · ‎12-28-2012

Martin - Thank you for this information. I completely forgot about getting into the codec via root and using the procps package of utilities. Thanks for the reminder! :-) I think I will ask of Cisco for an enhancement bug to be filled for these endpoints however, I think there is a big push to get this information from the endpoints through Prime Collaboration opposed to 3rd party software. I'll give it a go. Thanks again for your response!

-Andrew