cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
720
Views
2
Helpful
10
Replies

'Unmapped Exception null' (Install/Upgrade on CUCM OS Admin page)

voip7372
Level 4
Level 4

I already have a ticket open for this but after working with TAC for a few hours, they're still not sure what the problem is and the resolution for it. I wanted to check here also and see if someone already experienced this and got it resolved. 

This happened immediately after upgrading from CUCM 11.5.1 to 14 SU3. No certificates were expired before the upgrade and of course, none are expired after the upgrade. Only one of the subscriber servers has this problem. Server has been rebooted and Tomcat certs were generated (even though they weren't expired) just to see if that would help, since other cases like this were related to expired certs. All the servers use the standard self-signed certs.

I have 5 CUCM servers in a cluster. 1 Pub and 4 Sub servers. All 5 servers were upgraded with no errors from version 11.5.1 SU11 to CUCM 14 SU3 (14.0.1.13900-155). I wanted to install the post upgrade COP file (ciscocm.postUpgradeCheck-00041.cop.sha512) to check for errors. This COP file installed with no issues on 4 of the servers, but one of the subscriber servers has an error when I go to the install/upgrade page. It won't let me install anything there. I see 'Unmapped Exception null' and the only option is to select Cancel (there's no 'Next' button). Here's the path I go to and what happens: OS Administration page > Software Upgrades > Install/Upgrade (unusually long wait, maybe over 1 minute) > Error message: 'Unmapped Exception null'

To make things more interesting (or confusing), if I go to the Publisher server and log in to the OS Admin page and then go to Software Upgrades > Install/Upgrade Cluster I am able to install this COP file in that way (I.E., using the Install/Upgrade Cluster' option from the Publisher server rather than going directly to the Subscriber server and installing it there). However, I know there are files we sometimes need to install manually/individually on each server (not using the 'upgrade cluster' option from the Pub)...so I need to get this fixed.

Have you seen anything like this yourself?...and a resolution for it?

Thanks

error.jpg

1 Accepted Solution

Accepted Solutions

Update: The upgrade still wouldn't work because port 443 was being blocked (our subscriber couldn't talk to the publisher on port 443). However, I was able to get the firewall guy to update the firewall quickly and now it's working. I did a packet capture on the subscriber with the problem and noticed a bunch of 'retransmissions' for port 443 (https) for the sub trying to contact the pub. Firewall guy confirmed it was blocked. He opened port 443 and also 8443 (Cisco suggested 8443 to be opened also) and it's working now.

By the way, a quick way to check the connectivity for a certain port from the sub to the pub is like this (I did this on the subscriber's CLI):  utils network connectivity 192.168.2.2 443   

Let's pretend 192.168.2.2 is the IP of our publisher (it's not, but let's pretend :-))...

This shows the connection to port 443 from my sub is not working/blocked (to the pub):

admin:utils network connectivity 192.168.2.2 443
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connection timed out.
Connect to the port (tcp) failed: Connection refused
Service not accessible

After the firewall guy opened port 443, it looks like this (normal/good):

admin:utils network connectivity 192.168.2.2 443
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 192.168.2.2.
Ncat: 0 bytes sent, 0 bytes received in 0.13 seconds.
Service accessible

View solution in original post

10 Replies 10

Not the same but we had  issue with a CUCM cluster that we upgraded to 14.Su3. Some users had missing device associations on one of the nodes, while the other nodes showed the device associations correctly. We could not associate the devices on the problematic node. TAC could not identify the root cause and suggested us to rebuild that node. That worked for us. You might want to try the same thing..



Response Signature


OK. Thanks. Hoping I don't have to rebuild it, but we'll see (waiting for more info from the Cisco case I have open).

Have you tried it via CLI?
GUI is not the only option to make upgrades with.

Yes, I tried CLI also (utils system upgrade initiate) and that fails as well.

I checked the CLI log and found what looks like the error message where things go wrong and sent it to Cisco for review. It looks like right after it checks the source of the file (using the source/location set on the Publisher:  Cluster Software Location), it fails.

2024-01-26 08:40:04,880 INFO [ClassExecutionThread] api.config.info - UpgradeConfig - Getting the software location configurations :
2024-01-26 08:40:04,880 DEBUG [ClassExecutionThread] api.config.debug - DomHelper - loading the config file [/usr/local/platform/conf/simpleUpgradeConfig.xml]
2024-01-26 08:40:04,881 DEBUG [ClassExecutionThread] api.config.debug - DomHelper - [UpgradeConfigSource].[ParamValue] = [remote_sftp]
2024-01-26 08:40:04,882 DEBUG [ClassExecutionThread] api.config.debug - DomHelper - [UpgradeConfigDirectory].[ParamValue] = [/misc/temp]
2024-01-26 08:40:04,882 DEBUG [ClassExecutionThread] api.config.debug - DomHelper - [UpgradeConfigServer].[ParamValue] = [192.168.1.5]
2024-01-26 08:40:04,882 DEBUG [ClassExecutionThread] api.config.debug - DomHelper - [UpgradeConfigUsername].[ParamValue] = [admin]
2024-01-26 08:40:04,882 DEBUG [ClassExecutionThread] api.config.debug - DomHelper - [UpgradeConfigPassword].[ParamValue] = [edited – I removed this text]
2024-01-26 08:40:04,882 ERROR [ClassExecutionThread] api.config.error - DomHelper - could not load sub element's value
java.lang.NullPointerException: null
at com.cisco.vos.platform.api.config.DomHelper.getFirstSubElementValue(Unknown Source) ~[platform-api.jar:?]
at com.cisco.vos.platform.api.config.UpgradeConfig.getUpgradeConfig(Unknown Source) ~[platform-api.jar:?]
at com.cisco.iptplatform.cli.cmdUtilsSystemUpgrade.checkUpgradeConfig(cmdUtilsSystemUpgrade.java:949) ~[ciscoCmd.jar:?]
at com.cisco.iptplatform.cli.cmdUtilsSystemUpgrade.loadPersistedState(cmdUtilsSystemUpgrade.java:897) ~[ciscoCmd.jar:?]
at com.cisco.iptplatform.cli.cmdUtilsSystemUpgrade.handleUpgrade(cmdUtilsSystemUpgrade.java:749) ~[ciscoCmd.jar:?]
at com.cisco.iptplatform.cli.cmdUtilsSystemUpgrade.executeCmd(cmdUtilsSystemUpgrade.java:566) ~[ciscoCmd.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_332]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_332]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_332]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_332]
at com.cisco.iptplatform.cli.cmdClassThread.run(cmdClassThread.java:209) ~[ciscoCmd.jar:?]
2024-01-26 08:40:04,884 ERROR [ClassExecutionThread] api.config.error - DomHelper - could not load sub element's value
java.lang.NullPointerException: null
at com.cisco.vos.platform.api.config.DomHelper.getFirstSubElementValue(Unknown Source) ~[platform-api.jar:?]
at com.cisco.vos.platform.api.config.UpgradeConfig.getUpgradeConfig(Unknown Source) ~[platform-api.jar:?]
at com.cisco.iptplatform.cli.cmdUtilsSystemUpgrade.checkUpgradeConfig(cmdUtilsSystemUpgrade.java:949) ~[ciscoCmd.jar:?]
at com.cisco.iptplatform.cli.cmdUtilsSystemUpgrade.loadPersistedState(cmdUtilsSystemUpgrade.java:897) ~[ciscoCmd.jar:?]
at com.cisco.iptplatform.cli.cmdUtilsSystemUpgrade.handleUpgrade(cmdUtilsSystemUpgrade.java:749) ~[ciscoCmd.jar:?]
at com.cisco.iptplatform.cli.cmdUtilsSystemUpgrade.executeCmd(cmdUtilsSystemUpgrade.java:566) ~[ciscoCmd.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_332]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_332]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_332]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_332]
at com.cisco.iptplatform.cli.cmdClassThread.run(cmdClassThread.java:209) ~[ciscoCmd.jar:?]

 

mserwon
Level 1
Level 1

We did the same upgrade 11.5 to  14 SU3 and have the same error message on the Software Installation/Upgrade page.  Was TAC able to find a fix for this?

Cisco is still working on it. I met with them three or four times so far and they had me do various tasks related to the self-signed certificates CUCM created, but so far it hasn't fixed the problem. The other three subscribers that were setup the same way do not have this problem. Only the fourth subscriber server we have is an issue. So...still waiting for a resolution. 

OK, just finished a call with Cisco and it seems our firewall in the location of this one Subscriber server (it's in a different location) is not able to get the SFTP server/password info from the Publisher server which is in another location and the most likely reason is the firewall is blocking it. We proved the problem Subscriber CAN do an install/upgrade from its own OS Admin webpage, but ONLY if I first go to the Publisher's OS Admin page, select 'Cluster Software Location', select the server with the problem and select the option for 'Use below download credentials and software location' INSTEAD OF 'Use download credentials and software location from Publisher'. After telling that server to use the manual entry of the software location settings (which is exactly the same as the username, SFTP server, etc that all the other servers use), it works. When you click 'Install/Upgrade' on the Subscriber's OS Admin page, it will try to contact the Publisher to get the SFTP server info and it looks like in my case, that traffic is not making it to the Publisher, so the only logical explanation is that the firewall is blocking it. Traffic TO that Sub from the Pub seems OK, but if that Sub tries to contact the Pub for this info, it's not working. Again, the only logical reason I can think of for that to happen is our firewall is blocking it, so I'll talk to the team about that.

In the meantime, we have an easy workaround.

Settings on the Publisher for this one Subscriber (temporary, until firewall is updated):

voip7372_0-1709750503415.png

 

 

Update: The upgrade still wouldn't work because port 443 was being blocked (our subscriber couldn't talk to the publisher on port 443). However, I was able to get the firewall guy to update the firewall quickly and now it's working. I did a packet capture on the subscriber with the problem and noticed a bunch of 'retransmissions' for port 443 (https) for the sub trying to contact the pub. Firewall guy confirmed it was blocked. He opened port 443 and also 8443 (Cisco suggested 8443 to be opened also) and it's working now.

By the way, a quick way to check the connectivity for a certain port from the sub to the pub is like this (I did this on the subscriber's CLI):  utils network connectivity 192.168.2.2 443   

Let's pretend 192.168.2.2 is the IP of our publisher (it's not, but let's pretend :-))...

This shows the connection to port 443 from my sub is not working/blocked (to the pub):

admin:utils network connectivity 192.168.2.2 443
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connection timed out.
Connect to the port (tcp) failed: Connection refused
Service not accessible

After the firewall guy opened port 443, it looks like this (normal/good):

admin:utils network connectivity 192.168.2.2 443
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 192.168.2.2.
Ncat: 0 bytes sent, 0 bytes received in 0.13 seconds.
Service accessible

I'm glad you shared your solution for folks researching this error in the future. 

Maren

Thanks and you're welcome. I know it's frustrating when people don't update their threads here with the solution when they get something fixed. Especially if it's not a common problem. The only reason this was an issue for us AFTER the upgrade from CUCM 11 to 14 is because v11 didn't behave this way (the sub never needed to contact the pub on port 443), so I was stumped trying to figure out what was happening. Cisco TAC first was focusing on certificates but of course that didn't turn out to be the issue. This server that had the problem is on a special network behind a firewall due to what we use it for, but it worked fine on CUCM 11 (so I didn't initially suspect a firewall issue...and the firewall guys get enough blame/complaints, so I didn't want to bother them too much, yet). It wasn't until Cisco TAC eventually noticed in the logs that the sub was not getting a response from the pub when I kicked off the install on the sub (for a COP file install)...so then I started asking the firewall guy about it. I also found that command to check the individual port also, once I realized what port it was trying to use (via the packet capture which showed several retransmissions for port 443 trying to get to the pub).  Had I known in the beginning to verify the sub can talk to the pub on port 443, I would have saved a LOT of time. I had 3 or 4 Webex calls with Cisco TAC about this while they checked on certificates, etc. 

Anyway, ultimately it was actually a simple fix. Live and learn