cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2515
Views
25
Helpful
16
Replies

CUCM 6.1(5) - replace dead subscriber server

scrye
Level 1
Level 1

HI;

I'm the pointy-haired Network Manager with some (but not enough) geeky skills. I've inherited a mess. We have (had) a 2- node CUCM 6.1(5) cluster that supports about 1000 phones. The subscriber was killed (drowned) by a water-leak. The publisher is still running, and the system is alive, users are making calls, etc.  I have access to the CUCM tools such as the Real-Time monitoring tool, which is complaining bitterly about the absent subscriber. Sady, there is no documentation on the configuration, and there were no backups of the subscriber. It is completely dead. All my Call Manager experience (not much) was from the 4.x days. The systems were upgraded by a consultant from the old 4.x install, which was originally done in 2005.

We ordered and I just racked and powered up a brand-new replacement MCS-7800 CUCM server that came with 6.1.5.10000-10 pre-installed. I powered it up to make sure the KVM was alive and was very suprprised to see it boot into Linux and run through a "pre-installed software" routine that seems to have installed CUCM. At the present moment it is at a step asking for Pre-existing Configuration Information, such as a platformConfig.xml. I've walked away while I research and ask for guidance.

I know that setting up a subscriber is complicated and don't expect this forum to walk me through the process. We have SmartNet, but I'm holding off on opening a TAC case.

It would be wonderful if there is a way to go through a minimal amount of steps to get the new server running, give it a hostname, IP etc., and then tell the existing CUCM "publisher" that the long-lost subscriber is back, but has lost it's mind and needs to be told what to do - and watch the Publisher magically push all the configs into place.

OK, now that you have stopped ROFL, any high-level guidance would be greatly appreciated! Links to Cisco config/disaster recovery guides would be great. I already have this:

http://www.cisco.com/en/US/docs/voice_ip_comm/cucm/install/6_1_1/install/cmins611-cm.html

Thanks,

Steve

16 Replies 16

Hi Steve,

It's been quite some time since I last did this but here's how I would have done it

1. Remove the sub from the server list of the publisher and then re add it

2. Simple install the subscriber (make sure that the sub is in the same exact version as the pub)

3. Use the same config as the sub (ip address, hostname etc)

4. At some point it will ask you the publisher information (ip address, hostname, security password)

5. Eventually it will start the setup and it will also perform some connectivity tests with the pub

The installation process should be pretty straightforward and the information you need to provide is pretty basic

If you have any issues feel free to post again

Hope it helps,

Christos

Thanks for the super-fast reply, Christos!

This is encouraging.

It's been a long time since I configured a subscriber. From your suggestions, it seems like Cisco might have improved things a lot since V 4.2 .

After the Pub and Sub are talking to each other, what other general configuration steps do I have to do? (aside from setting up backup!)

Do I have to do a bunch of manual stuff, or will they pretty much be ready to act as a fault-tolerant cluster?

Back in the 4.2 days, I recall that once the Subscriber was added, it took over the main functions of call setup and completion. Is that still the case?

Thanks,

Steve

Christos, our CUCM Publisher is running 6.1.3.2000-1 , according to Help/About in the RT monitoring tool. Rats. I have an install DVD that came with the new server that says 6.1(5) - this is probably the same version that is installed on the new server.

So, what to do? Upgrade the Pub, or downgrade the Sub? How hard is it to upgrade a server from 6.1.3.2000-1 to 6.1.5.10000-10 ?

You cautioned that both have to run the exact same version - are these versions different enough to cause a problem?

Thanks,

Steve

I wouldn't run two different versions of Call Manager, even if they are minor release differences.  If you have a software subscription contract, you may want to look into going to version 7.1 (or 8.0, but I'm not a fan of x.0 releases).  Although in your scenario, I'd probably want to get back to a redundant state (assuming 1 pub 1 sub deployment) as soon as possible which would mean getting my hands on a 6.1.3 disk.  If not, I would take a full backup of your pub, upgrade the pub to 6.1.5 and add install the sub.

Ok the first thing to do is get the servers on the same version. This is not optional; the new subscriber will not join the cluster if the version does not match. I recommend getting a 6.1(3a) installable disc. I suggest upgrading your cluster to a newer version at another time; get yourself and the database a stable state first! If you do not have that exact version but can find a 6.1(1) or perhaps 6.0(1) you can piece together what you need:

  1. Install the base version from the bootable DVD that you have.
  2. Download, merge, and apply the patch to the 6.1(3a) (i.e. 6.1.3.2000). You will be asked if you want to apply a patch after booting from the original DVD. Note that ISOs downloaded from CCO are not bootable for licensing reasons.

Once you have the new subscriber on 6.1(3) it will ask you to do the initial platform configuration. The values you specify (e.g. hostname, IP address, gateway, etc) need to be identical to the original subscriber. You want to try reclaiming this node's position and CTI ID in the database if possible. Note that without a valid backup from the subscriber this may not be possible. What you likely won't have is the security passphrase. This is a shared secret used to secure intra-cluster communications signaling (a.k.a. ICCS). The only way to get this is reseting it on your publisher. Note this will require a restart of the publisher to take effect.

After the Pub and Sub are talking to each other, what other general configuration steps do I have to do? (aside from setting up backup!) Do I have to do a bunch of manual stuff, or will they pretty much be ready to act as a fault-tolerant cluster?

If it assumed the original node's CTI ID and place in the database, nothing. If it doesn't, you'll need to update your CallManager Groups to ensure that the node is added and first so the phones register to it first. Phones require a reset to notice CMG changes.

After everything is online you should also ensure that the database has synchronized using the utils dbreplication status command. You can do this publisher only. What you want to see is all nodes at a two (2) eventually. Anything above three (3) indicates a failure and you should call TAC.

admin:utils dbreplication status

-------------------- utils dbreplication status --------------------

Replication status check is now running in background.

Use command 'utils dbreplication runtimestate' to check its progress

The final output will be in file cm/trace/dbl/sdi/ReplicationStatus.2010_11_15_20_07_30.out

Please use "file view activelog cm/trace/dbl/sdi/ReplicationStatus.2010_11_15_20_07_30.out " command to see the output

admin:file view activelog cm/trace/dbl/sdi/ReplicationStatus.2010_11_15_20_07_30.out

SERVER                 ID STATE    STATUS     QUEUE  CONNECTION CHANGED

-----------------------------------------------------------------------

g_manager1_ccm8_0_3_10000_8    2 Active   Local           0

g_manager2_ccm8_0_3_10000_8    3 Active   Connected       0 Oct 14 07:31:02

end of the file reached

Back in the 4.2 days, I recall that once the Subscriber was added, it took over the main functions of call setup and completion. Is that still the case?

This is best-practice but only happens if you configure the CMGs properly. The phones/gateways only speak to node(s) in their assigned CMG and it's an ordered list.

The only issue I see with what Jonathan says is that if you downgrade you loose you configuration data (unless 6.1.3 is already installed in the inactive partition, you can check with 'show version inactive' from cli). Anyway even if you would switch back to the inactive partition you would lose all the configuration that was made after the upgrade to 6.1.5

If this is not possible, you can always bring the pub to the version of the sub by installing an upgrade patch found on cisco.com

http://www.cisco.com/cisco/software/release.html?mdfid=281023410&flowid=5325&softwareid=282074295

Once this is done you can install the sub using the DVD

The version of the pub and sub should be exactly the same.

Once you install it and replication kicks in, all the configuration should be propagated to the sub's db as well. Obviously that might take some time

Regards,

Christos

Just to clarify: This person stated his publisher is currently 6.1.3.2000-1. I recommend wiping the manufacturing install of 6.1.5 on the new subscriber and installing the version that matches the publisher. I did not recommend downgrading the publisher.

Some backing for Jonathan, you should always build your Sub to match the Publisher (as far as code version) goes when rebuilding a cluster.  Is it absolutely mandatory, no.  Will you save you headache, potential problems, and downtime - absolutely.

Hailey

Please rate helpful posts!

Probably I went through it too quickly and I misunderstood it Jonathan

Apologies for that

Regards,

Christos

Well, now the fun has started

I grabbed the original 6.1(3) DVD that we used to install the original pub and sub. Booted OK, media check OK. BUT - the silly thing claims the hardware is incompatible!?! WTF? This is a brand-new server we obtained from Cisco via Smartnet replacement. It is an IBM MCS 7800 and was manufactured in May 2010. It came with 6.1(5) pre-loaded. How is it possible that a slightly older version of the software thinks brand-new hardware is not supported?!?

Steve

What is the actual model of the new/old subscriber servers? MCS-78XX-H/IY-ZZZ?

To be clear: Are you saying this server was shipped to you due to a SMARTnet case; or, you purchased it?

Hi Steve,

If you follow this link

http://www.cisco.com/en/US/prod/collateral/voicesw/ps6790/ps5748/ps378/prod_brochure0900aecd8062a4f9.html

you will see all the server models supported for every cucm version.

You will also notice that there is a separate column for CUCM 6.1(3) and 6.1(5)

If you compare those two columns you will also see that there are some servers that are only supported in 6.1(5) and NOT in 6.1(3) (MCS-7835-I3 being one of them for example)

Regards,

Christos

Hi Christos;

Yeah, I had noticed that, Thanks.

I've already contacted our Cisco sales Engineer to complain. That particular model, which as you can see on the "compatability" matirx, is the lease versatile server they sell! That Cisco touted that model as the best replacement is annoying.

I maintain that Cisco arbitrarily decided to not support 6.1(3) on that hardware. The system is certainly fast enough, and short of not finding a driver for some element of the server (something that any CLP worth his salt could do), it is baffling  that the hardware is not supported.

TAC is calling me - stand by for more details.

Thanks!

Steve

Jonathan, I can't thank you enough for taking the time to post such a detailed and thoughtful reply. (I also greatly appreciate the other responses).

I DO have the 6.1(3a) .isos, so I will proceed with the "downgrade" of the new Sub server. I'll also make one final desperate search for any info from the dead Sub, and will pester the consultant for the shared secret. My plan is to start this process today, so by tomorow things will either be going well or I will be in a panic! Please stand by.

Thanks again,

Steve