11-15-2010 11:16 AM - edited 03-16-2019 01:56 AM
HI;
I'm the pointy-haired Network Manager with some (but not enough) geeky skills. I've inherited a mess. We have (had) a 2- node CUCM 6.1(5) cluster that supports about 1000 phones. The subscriber was killed (drowned) by a water-leak. The publisher is still running, and the system is alive, users are making calls, etc. I have access to the CUCM tools such as the Real-Time monitoring tool, which is complaining bitterly about the absent subscriber. Sady, there is no documentation on the configuration, and there were no backups of the subscriber. It is completely dead. All my Call Manager experience (not much) was from the 4.x days. The systems were upgraded by a consultant from the old 4.x install, which was originally done in 2005.
We ordered and I just racked and powered up a brand-new replacement MCS-7800 CUCM server that came with 6.1.5.10000-10 pre-installed. I powered it up to make sure the KVM was alive and was very suprprised to see it boot into Linux and run through a "pre-installed software" routine that seems to have installed CUCM. At the present moment it is at a step asking for Pre-existing Configuration Information, such as a platformConfig.xml. I've walked away while I research and ask for guidance.
I know that setting up a subscriber is complicated and don't expect this forum to walk me through the process. We have SmartNet, but I'm holding off on opening a TAC case.
It would be wonderful if there is a way to go through a minimal amount of steps to get the new server running, give it a hostname, IP etc., and then tell the existing CUCM "publisher" that the long-lost subscriber is back, but has lost it's mind and needs to be told what to do - and watch the Publisher magically push all the configs into place.
OK, now that you have stopped ROFL, any high-level guidance would be greatly appreciated! Links to Cisco config/disaster recovery guides would be great. I already have this:
http://www.cisco.com/en/US/docs/voice_ip_comm/cucm/install/6_1_1/install/cmins611-cm.html
Thanks,
Steve
11-15-2010 11:34 AM
Hi Steve,
It's been quite some time since I last did this but here's how I would have done it
1. Remove the sub from the server list of the publisher and then re add it
2. Simple install the subscriber (make sure that the sub is in the same exact version as the pub)
3. Use the same config as the sub (ip address, hostname etc)
4. At some point it will ask you the publisher information (ip address, hostname, security password)
5. Eventually it will start the setup and it will also perform some connectivity tests with the pub
The installation process should be pretty straightforward and the information you need to provide is pretty basic
If you have any issues feel free to post again
Hope it helps,
Christos
11-15-2010 01:54 PM
Thanks for the super-fast reply, Christos!
This is encouraging.
It's been a long time since I configured a subscriber. From your suggestions, it seems like Cisco might have improved things a lot since V 4.2 .
After the Pub and Sub are talking to each other, what other general configuration steps do I have to do? (aside from setting up backup!)
Do I have to do a bunch of manual stuff, or will they pretty much be ready to act as a fault-tolerant cluster?
Back in the 4.2 days, I recall that once the Subscriber was added, it took over the main functions of call setup and completion. Is that still the case?
Thanks,
Steve
11-15-2010 03:02 PM
Christos, our CUCM Publisher is running 6.1.3.2000-1 , according to Help/About in the RT monitoring tool. Rats. I have an install DVD that came with the new server that says 6.1(5) - this is probably the same version that is installed on the new server.
So, what to do? Upgrade the Pub, or downgrade the Sub? How hard is it to upgrade a server from 6.1.3.2000-1 to 6.1.5.10000-10 ?
You cautioned that both have to run the exact same version - are these versions different enough to cause a problem?
Thanks,
Steve
11-15-2010 03:35 PM
I wouldn't run two different versions of Call Manager, even if they are minor release differences. If you have a software subscription contract, you may want to look into going to version 7.1 (or 8.0, but I'm not a fan of x.0 releases). Although in your scenario, I'd probably want to get back to a redundant state (assuming 1 pub 1 sub deployment) as soon as possible which would mean getting my hands on a 6.1.3 disk. If not, I would take a full backup of your pub, upgrade the pub to 6.1.5 and add install the sub.
11-15-2010 06:10 PM
Ok the first thing to do is get the servers on the same version. This is not optional; the new subscriber will not join the cluster if the version does not match. I recommend getting a 6.1(3a) installable disc. I suggest upgrading your cluster to a newer version at another time; get yourself and the database a stable state first! If you do not have that exact version but can find a 6.1(1) or perhaps 6.0(1) you can piece together what you need:
Once you have the new subscriber on 6.1(3) it will ask you to do the initial platform configuration. The values you specify (e.g. hostname, IP address, gateway, etc) need to be identical to the original subscriber. You want to try reclaiming this node's position and CTI ID in the database if possible. Note that without a valid backup from the subscriber this may not be possible. What you likely won't have is the security passphrase. This is a shared secret used to secure intra-cluster communications signaling (a.k.a. ICCS). The only way to get this is reseting it on your publisher. Note this will require a restart of the publisher to take effect.
After the Pub and Sub are talking to each other, what other general configuration steps do I have to do? (aside from setting up backup!) Do I have to do a bunch of manual stuff, or will they pretty much be ready to act as a fault-tolerant cluster?
If it assumed the original node's CTI ID and place in the database, nothing. If it doesn't, you'll need to update your CallManager Groups to ensure that the node is added and first so the phones register to it first. Phones require a reset to notice CMG changes.
After everything is online you should also ensure that the database has synchronized using the utils dbreplication status command. You can do this publisher only. What you want to see is all nodes at a two (2) eventually. Anything above three (3) indicates a failure and you should call TAC.
admin:utils dbreplication status
-------------------- utils dbreplication status --------------------
Replication status check is now running in background.
Use command 'utils dbreplication runtimestate' to check its progress
The final output will be in file cm/trace/dbl/sdi/ReplicationStatus.2010_11_15_20_07_30.out
Please use "file view activelog cm/trace/dbl/sdi/ReplicationStatus.2010_11_15_20_07_30.out " command to see the output
admin:file view activelog cm/trace/dbl/sdi/ReplicationStatus.2010_11_15_20_07_30.out
SERVER ID STATE STATUS QUEUE CONNECTION CHANGED
-----------------------------------------------------------------------
g_manager1_ccm8_0_3_10000_8 2 Active Local 0
g_manager2_ccm8_0_3_10000_8 3 Active Connected 0 Oct 14 07:31:02
end of the file reached
Back in the 4.2 days, I recall that once the Subscriber was added, it took over the main functions of call setup and completion. Is that still the case?
This is best-practice but only happens if you configure the CMGs properly. The phones/gateways only speak to node(s) in their assigned CMG and it's an ordered list.
11-15-2010 11:48 PM
The only issue I see with what Jonathan says is that if you downgrade you loose you configuration data (unless 6.1.3 is already installed in the inactive partition, you can check with 'show version inactive' from cli). Anyway even if you would switch back to the inactive partition you would lose all the configuration that was made after the upgrade to 6.1.5
If this is not possible, you can always bring the pub to the version of the sub by installing an upgrade patch found on cisco.com
http://www.cisco.com/cisco/software/release.html?mdfid=281023410&flowid=5325&softwareid=282074295
Once this is done you can install the sub using the DVD
The version of the pub and sub should be exactly the same.
Once you install it and replication kicks in, all the configuration should be propagated to the sub's db as well. Obviously that might take some time
Regards,
Christos
11-16-2010 03:32 AM
Just to clarify: This person stated his publisher is currently 6.1.3.2000-1. I recommend wiping the manufacturing install of 6.1.5 on the new subscriber and installing the version that matches the publisher. I did not recommend downgrading the publisher.
11-16-2010 06:25 AM
Some backing for Jonathan, you should always build your Sub to match the Publisher (as far as code version) goes when rebuilding a cluster. Is it absolutely mandatory, no. Will you save you headache, potential problems, and downtime - absolutely.
Hailey
Please rate helpful posts!
11-16-2010 06:39 AM
Probably I went through it too quickly and I misunderstood it Jonathan
Apologies for that
Regards,
Christos
11-16-2010 02:19 PM
Well, now the fun has started
I grabbed the original 6.1(3) DVD that we used to install the original pub and sub. Booted OK, media check OK. BUT - the silly thing claims the hardware is incompatible!?! WTF? This is a brand-new server we obtained from Cisco via Smartnet replacement. It is an IBM MCS 7800 and was manufactured in May 2010. It came with 6.1(5) pre-loaded. How is it possible that a slightly older version of the software thinks brand-new hardware is not supported?!?
Steve
11-16-2010 05:22 PM
What is the actual model of the new/old subscriber servers? MCS-78XX-H/IY-ZZZ?
To be clear: Are you saying this server was shipped to you due to a SMARTnet case; or, you purchased it?
11-17-2010 03:04 AM
Hi Steve,
If you follow this link
you will see all the server models supported for every cucm version.
You will also notice that there is a separate column for CUCM 6.1(3) and 6.1(5)
If you compare those two columns you will also see that there are some servers that are only supported in 6.1(5) and NOT in 6.1(3) (MCS-7835-I3 being one of them for example)
Regards,
Christos
11-17-2010 08:28 AM
Hi Christos;
Yeah, I had noticed that, Thanks.
I've already contacted our Cisco sales Engineer to complain. That particular model, which as you can see on the "compatability" matirx, is the lease versatile server they sell! That Cisco touted that model as the best replacement is annoying.
I maintain that Cisco arbitrarily decided to not support 6.1(3) on that hardware. The system is certainly fast enough, and short of not finding a driver for some element of the server (something that any CLP worth his salt could do), it is baffling that the hardware is not supported.
TAC is calling me - stand by for more details.
Thanks!
Steve
11-16-2010 08:34 AM
Jonathan, I can't thank you enough for taking the time to post such a detailed and thoughtful reply. (I also greatly appreciate the other responses).
I DO have the 6.1(3a) .isos, so I will proceed with the "downgrade" of the new Sub server. I'll also make one final desperate search for any info from the dead Sub, and will pester the consultant for the shared secret. My plan is to start this process today, so by tomorow things will either be going well or I will be in a panic! Please stand by.
Thanks again,
Steve
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide