Troubleshooting Failing Appliance CUCM Upgrades

Cedric Van Labeke · ‎10-20-2010

Troubleshooting Failing Appliance CUCM Upgrades

This document may help you to determine why your CUCM upgrade with CUCM 5.x, 6.x, 7.x, 8.x and above.

Most issues seen are:

- Invalid Upgrade path

- Invalid Upgrade Media

- Missing References in the Database

I. Steps to go through when you're upgrade is failing

1. Check compatibility and upgrade path

In most cases, you'll get your final answer in the install & upgrade logs. So getting these is always a good start. One other thing to verify is if your upgrade path is a supported one as Cisco TAC will always ask the exact version you're upgrading from and the exact version you're upgrading towards.

Using the following link, you can check the software compatibility and valid upgrade paths:

http://www.cisco.com/en/US/docs/voice_ip_comm/cucm/compat/ccmcompmatr.html#wp43011

2. Validate your Media

Any file upgrade file available on cisco.com or published by TAC has an MD5 Checksum in the accompanying release notes.

Always verify after download whether your checksum is ok. If not, things will go wrong for sure (although there's a possibility that it won't be that obvious that a bad install is causing issues -- think of a very frustrating situation where you are having major issues and TAC nor Development can help out further and a reinstall of the server is advised! Making sure checksums are corret when you upgrade is a must).

On Windows based systems, you can use a tool such as winmd5. On *nix based systems, you can just issue the md5 command.

3. Install/Upgrade Logs

If neither of the above steps work, you're very likely to go to the logs. If you wouldn't be able to figure out from them what's going on and are about to raise a TAC case, do provide them anyway as it's the first thing TAC will request.

II. Common issues seen in Install/Upgrade logs

While you may see some error messages around the upgrade logs, some of them could very well be benign. Focus on the bottom of the log as that's where it ended.

1. Unable to mount the local file system

If you see a message like "unable to mount the local file system" check if you're not affected by defect CSCsm55596

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCsm55596

2. Failed to exec command - cm-dbl-install L2 PostInstall

If you see a message at the end of the log file like this:

Internal Error, File:instMain.c:1403, Function: handlePhase(), Failed to exec command: "/partB/usr/local/bin/base_scripts/env.sh nice -n 19 /usr/sbin/chroot /partB /usr/local/cm/script/6.1.3.1000-16/cm-dbl-install L2 PostInstall 6.1.3.1000-16 6.1.2.1000-13 /usr/local/cm/ /partB/usr/local/cm/ /common/log/install/capture.txt"|<LVL::Critical>

This is not the real error message. It just tells us that the cm-dbl-install script has failed. To troubleshoot this, we will need DBL logs.

Using RTMT download the following logs from the inactive partition: "Cisco Database Installation Service". It will give you 2 files basically, installdb_l2.log and installdb_l2.log.err.

installdb_l2.log will cover each DB step being executed.

installdb_l2.log.err will cover each DB step being executed which has failed.

In summary, here's where you'll find the true reason it fails.

2.1 Understanding installdb_l2.log.err

Looking at these logs might be very overwhelming when you look at them the first time. They are full with errors and you don't know where to start. In fact, just start by looking at line 1 of installdb_l2.log.err and focus on that one.

An example error you can find in that log is the following one:

05/03/2009 23:18:31.857 installdb|*ERROR* Error executing "insert into EndUser (allowcticontrolflag,assocpc,building,department,enablemobilevoice,enabl emobility,facsimiletelephonenumber,firstname,fkmatrix_presence,homephone ,lastname,mailid,manager,maxdeskpickupwaittime,middlename,mobile,nicknam e,ocsprimaryuseraddress,pager,passwordreverse,pkid,remotedestinationlimi t,site,status,telephonenumber,title,tkuserlocale,tkuserprofile,uniqueide ntifier,userid) values ('T','','','','F','T','','','ad243d17-98b4-4118-8feb-5ff2e1b781ac','','T est DD','','',10000,'','','','','','','975b528e-1ff1-9133-be40-45ee9d99c761' ,4,'',1,'66','',3,1,'','user66'): [Informix][Informix ODBC Driver][Informix]Missing key in referenced table for referential constraint (informix.tk_enduser_tkuserlocale).|

While this might look cryptic at first sight, it's pretty easy to understand if you know a bit of SQL.

Basically this is an SQL Query inserting data in the Database. We are trying to insert data into a table called EndUser.

The table has the following fields:

(allowcticontrolflag,assocpc,building,department,enablemobilevoice,enabl emobility,facsimiletelephonenumber,firstname,fkmatrix_presence,homephone ,lastname,mailid,manager,maxdeskpickupwaittime,middlename,mobile,nicknam e,ocsprimaryuseraddress,pager,passwordreverse,pkid,remotedestinationlimi t,site,status,telephonenumber,title,tkuserlocale,tkuserprofile,uniqueide ntifier,userid)

And now we're simply inserting these values respecting the fields name:

('T','','','','F','T','','','ad243d17-98b4-4118-8feb-5ff2e1b781ac','','T est DD','','',10000,'','','','','','','975b528e-1ff1-9133-be40-45ee9d99c761' ,4,'',1,'66','',3,1,'','user66')

It's as simple as that.

But here, there's an error Informix is telling us about:

[Informix][Informix ODBC Driver][Informix]Missing key in referenced table for referential constraint (informix.tk_enduser_tkuserlocale)

This message tells us that we are trying to insert a value which is referencing a field in another table but in fact, that field in the other table doesn't exist yet. It even tells us which field: tkuserlocale.

05/03/2009 23:18:31.857 installdb|*ERROR* Error executing "insert into EndUser (allowcticontrolflag,assocpc,building,department,enablemobilevoice,enabl emobility,facsimiletelephonenumber,firstname,fkmatrix_presence,homephone ,lastname,mailid,manager,maxdeskpickupwaittime,middlename,mobile,nicknam e,ocsprimaryuseraddress,pager,passwordreverse,pkid,remotedestinationlimi t,site,status,telephonenumber,title,tkuserlocale,tkuserprofile,uniqueide ntifier,userid) values ('T','','','','F','T','','','ad243d17-98b4-4118-8feb-5ff2e1b781ac','','T est DD','','',10000,'','','','','','','975b528e-1ff1-9133-be40-45ee9d99c761' ,4,'',1,'66','',3,1,'','user66'): [Informix][Informix ODBC Driver][Informix]Missing key in referenced table for referential constraint (informix.tk_enduser_tkuserlocale).|

This field tkuserlocale points to field enum in table typeuserlocale. How do I know this? By checking the Database Dictionary available on cisco.com:

http://www.cisco.com/en/US/docs/voice_ip_comm/cucm/datadict/6_1_1/dd_611.pdf

(Search for Database Dictionary on cisco.com to find other CUCM versions)

So what happens is that because there was no entry 3 in the enum field in table typeuserlocale, we failed to insert this particular user into table EndUser. => Referential constraint!

The solution to fix this is to either install the locale on the server again, or remove the locale from the enduser it fails on.

Now that we fixed this one line, what about the other 200 lines?

Ok, all the other same issues (in this example "insert into EndUser failures), need to be sorted in the same way. The other errors might be a result of the first errors.

Likewise, if there's any phone configured with this locale 3, it will also fail to insert into the DB.

Remember that this is a relational Database. If inserting a PhoneButtonTemplate into the database fails because of a missing PhoneTemplateType, then any attempt to insert a phone into table Device which is trying to reference that PhoneButtonTemplate will fail as well.

So in that case you will find a bunch of errors for PhoneButtonTemplate inserts, and after those you'll find a bunch of errors pointing to the failing inserts of entries referencing the PhoneButtonTemplate (like inserts into Table Device).

Here's an overview of locales:

2 French_France

3 German_Germany

5 Russian_Russian_Federation

6 Spanish_Spain

7 Italian_Italy

8 Dutch_Netherlands

9 Norwegian_Norway

10 Portuguese_Portugal

11 Swedish_Sweden

12 Danish_Denmark

13 Japanese_Japan

14 Hungarian_Hungary

15 Polish_Poland

16 Greek_Greece

19 Chinese_Taiwan

20 Chinese_China

21 Korean_Korea_Republic

22 Finnish_Finland

23 Portuguese_Brazil

24 Chinese_Hong_Kong

25 Slovak_Slovakia

26 Czech_Czech_Republic

27 Bulgarian_Bulgaria

28 Croatian_Croatia

29 Slovenian_Slovenia

30 Romanian_Romania

32 Catalan_Spain

33 English_United_Kingdom

34 Turkish_Turkey

35 Arabic_United_Arab_Emirates

36 Arabic_Oman

37 Arabic_Saudi_Arabia

38 Arabic_Kuwait

39 Hebrew_Israel

40 Serbian_Republic_of_Serbia

41 Serbian_Republic_of_Montenegro

42 Thai_thailand

43 Vietnamese_Vietnam

44 Bahasa_Indonesia

45 Malay_Malaysia

46 Welsh_United_Kingdom

47 Arabic_Algeria

48 Arabic_Bahrain

49 Arabic_Egypt

50 Arabic_Iraq

51 Arabic_Jordan

52 Arabic_Lebanon

53 Arabic_Morocco

54 Arabic_Qatar

55 Arabic_Tunisia

56 Arabic_Yemen

2.2 Finding records with non-default locales

Any EndUser or Device with a default Locale, will have the locale field set to Null. So retrieving records which are not Null will reveal the users/devices which don't have it:

admin: run sql select userid from EndUser where tkuserlocale is not null

userid

=============

cvl_user1

martaMobility

admin:run sql select pkid,description from device where tkuserlocale is not null

pkidvvv description

==================================== ======================

1365dba4-d130-4c92-a72d-49e6551a20b6 Bruno1 1001

63344d4c-37a8-3036-a8ad-4072174f2672 SEP00215554AFE8

326921f7-3c96-4cb3-9c2a-1be43a3a982b

71575645-7e00-10e8-bd2d-1a66ba9244da 7921 Phone Cedric Test

2.3 Chile locale error

If Installdb_l2.log.err has only this:

05/28/2009 07:34:00.855 installdb|*ERROR* Error executing "Update ProductCapabilities set tkProduct=382, tkProductConfig=32, enumValue=68, moniker='COUNTRY_CHILE' where enum = 67820": [Informix][Informix ODBC Driver][Informix]Missing key in referenced table for referential constraint (informix.tk_productcapabilities_tkproduct).|
05/28/2009 07:34:00.865 installdb|*ERROR* Error executing "Update ProductCapabilities set tkProduct=383, tkProductConfig=32, enumValue=68, moniker='COUNTRY_CHILE' where enum = 67820": [Informix][Informix ODBC Driver][Informix]Missing key in referenced table for referential constraint (informix.tk_productcapabilities_tkproduct).|
05/28/2009 07:34:00.874 installdb|*ERROR* Error executing "Update ProductCapabilities set tkProduct=384, tkProductConfig=32, enumValue=68, moniker='COUNTRY_CHILE' where enum = 67820": [Informix][Informix ODBC Driver][Informix]Missing key in referenced table for referential constraint (informix.tk_productcapabilities_tkproduct).|
05/28/2009 07:34:16.466 installdb| installFull *ERROR* Prior Cancel or Error Processing installCsv (/usr/local/cm/db/csv/products)|05/28/2009 07:34:20.040 installdb| CDBInstall::ProcessRules() *ERROR* <rules> has no element content.|
05/28/2009 07:34:20.040 installdb| CDBInstall::ProcessXMLFile() *ERROR* There is a problem with the XML file. Not all display instances or rules may have been loaded. Check the log file for details.|

You're being hit by defect CSCsz30088 - CUCM with Chile locale - Upgrade fails - all the products have same value

http://http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCsz30088

III. Conclusion

Using above information, you should be able to do a very thorough analysis and troubleshooting of your failing upgrade. Fixing those relational issues can take some time and may ask some repetitive upgrade attempts if you haven't caught all issues at the first time.

If after many attempts you still have upgrade issues, and the logs don't point to anything obvious at all, please contact TAC for further assistance.

When opening the case, provide the latest and greatest set of logs you took and share what you see.

Cedric

Anonymous · ‎05-21-2011

Great article!

kusatija · ‎05-25-2011

Really very good article and information, to add to above, we need the following trace to analyze the reason for the failed upgrade, and on how to collect them via CLI:

file get install install.log

file get activelog cm/log/informix/ccm.log\*

file list inactivelog core/\*

file get install capture.txt

file get inactivelog cm/trace/dbl/sdi/installdb_l2.log

file get inactivelog cm/trace/dbl/sdi/installdb_l2.log.err

file get inactivelog cm/trace/dbl/sdi/*.sql.log

file list install in* date detail

in the list of files get the lastest, for example. file get install

install_log_061009054330.log

HTH
Kunal

Clifford McGlamry · ‎10-27-2011

Had an occurance of the "Unable to mount local file system" on an 8.5(1) upgrade. This is not affected by the listed bug id.

Fixed the issue by simply rebooting the server.