Slow installation times on publisher

[billmatthews]

Hello,

We have a two node cluster (one pub., one sub) running v10.5.2.  They are VMs (using medium OVA) running on similar Cisco Business Edition 600 hardware.  Small environment (100 devices) with all phones registered to the sub.

When I go to install a firmware or device pack to both nodes, it takes considerably longer on the publisher.  For example with cmterm-devicepack10.5.2.12014-1.cop.sgn it installed on the sub in 4 minutes.  It took 38 minutes on the pub.  Is this normal just because it's the pub?  Or is this an indication that something's wrong?

Show hardware from pub: 

Platform Type VMware

Serial Number VMware-42 

Virtual Hardware Configured

Virtual Support Supported

Processor Speed 2400

CPU Type Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz

Memory 4096 Mbytes

Object ID 1.3.6.1.4.1.9.1.1348

Unified OS Version operatingsystem.UCOS6.0.0.0-2.i386

RAID Details No RAID information is available

-----------------------------------------------------------------------

Physical device information

-----------------------------------------------------------------------

Number of Disks : 1

Hard Disk #1

Size (in GB) : 80

And from Sub:

Platform Type VMware

Serial Number VMware-56 

Virtual Hardware Configured

Virtual Support Supported

Processor Speed 2600

CPU Type Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz

Memory 4096 Mbytes

Object ID 1.3.6.1.4.1.9.1.1348

Unified OS Version operatingsystem.UCOS6.0.0.0-2.i386

RAID Details No RAID information is available

-----------------------------------------------------------------------

Physical device information

-----------------------------------------------------------------------

Number of Disks : 1

Hard Disk #1

Size (in GB) : 80

Thanks,

Bill

[Manish Gogna]

Hi Bill,

Can you check the CPU and memory utilization on Pub and Sub while doing such firmware installs or any upgrades. Also, try with a SFTP server which is in the same subnet as Pub, the difference is definitely too much between Pub and Sub so something is definitely wrong. Is there a reason for the CPU specs being diff for Pub and Sub?

HTH

Manish

[billmatthews]

Thanks for the reply.  Yes, the TFTP server is on the same subnet.

I assume the difference in the CPU is because the underlying server hardware (Cisco Business Edition 6000) was purchased at different times, so may contain slightly different CPUs?

Yes, CPU does spike on pub.  Alert shown below

 Processor load over configured threshold for configured duration of time . Configured high threshold is 90 % watchdog/1 (17 percent) uses most of the CPU.     Processor_Info:      For processor instance 1: %CPU= 90; %User= 5; %System= 4; %Nice= 0; %Idle= 9; %IOWait= 80; %softirq= 1; %irq= 0.      For processor instance _Total: %CPU= 91; %User= 5; %System= 4; %Nice= 0; %Idle= 9; %IOWait= 80; %softirq= 1; %irq= 0.      For processor instance 0: %CPU= 92; %User= 5; %System= 5; %Nice= 0; %Idle= 8; %IOWait= 80; %softirq= 1; %irq= 0.    The alert is generated on Mon May 11 08:32:48 EDT 2015 on node CUCMP.       Memory_Info: %Mem Used= 80; %VM Used= 70.     Partition_Info:   Swap: %Disk Used=50.  Active: %Disk Used=88.  Common: %Disk Used=93.      Process_Info: processes with D-State:  flush-8:0 cdragent md5sum jbd2/sda2-8 postmaster#7 dbcfs#1 jbd2/sda6-8

[Manish Gogna]

Hi Bill,

Thanks for the details. Since this is the Publisher I would suggest opening a TAC case to find the root cause for CPU spike. You may check the output of the following to get some more details about the health of your Publisher

utils diagnose test

utils core active list

HTH

Manish

[Jonathan Schulenberg]

A install duration difference between pub and sub is normal. COP installs on the publisher must update the database while installs on subscribers simply copy files into the TFTP directory. Since a device pack has to do this for every phone model I wouldn't be at all concerned about the 30 minute time delta.

TFTP operations do not have a CPU quota. You should be installing these during a maintenance window as it could impact system performance.

Lastly, the CPU spec difference is most likely just the difference between a BE7000H (M4) UCS C240 M4SX TRC#1 and a BE6000H (M3) UCS C220 M3S TRC#3. This is not a cause for alarm.

[billmatthews]

Thanks guys!  I opened a TAC case just incase.  TAC confirmed that this is normal (passed all diagnostics).  

Jonathan -- thanks for the advice re: maintenance windows.  

One question about that.  You mentioned the TFTP operation is not subject to a CPU quota.  The actual long delay and CPU spike is during the package install (after the SFTP transfer).  Is that expected as well?  

[Jonathan Schulenberg]

The firmware files are served up by a TFTP process on the server.