MSE haskey mismatch

[cmarva]

just noticed this one the other day, neither of our mobility engines are syncing with the WLCs.

Our setup is:

PI 2.1

all WLCs are 7.4.121.0

MSE3355 - was 7.5.102.101

vMSE - was 7.5.102.101

all was fine. We upgraded both MSEs v7.6.120.0, due to the deferral notice for 7.5.102.101. Ever since, nothing will sync. All we get is NMSP status is inactive for both MSEs to all WLCs. The message is hashkey mismatch between MSE and WLC. I have tried numerous things to no avail, like deleting the key from the WLCs. When I try to re-sync, I do see the key gets pushed from the MSE to the WLC. But in the MSE logs I do see certificate unknown errors when trying to sync. I do have a case open on this.

It really seems like it resulted from the MSE upgrade, but I didn't see anything in the release notes that caused any concern.

Has anyone seen this symptom at all? Would it really be as simple as finding and deleting the key store? Any comments are appreciated.

Thanks - chris

[cmarva]

saw the post about apache not starting after upgrading to 7.5, but I don't think that's my problem as apache is up and running.

chris

[cmarva]

am also seeing on the WLC, via command line, the following when doing a sho nmsp statis summ:

SSL Handshake failed............................. 15542

and it is incrementing. So it definitely looks like either:

/opt/mse/locsrv/ssl    or   /var/mse/certs/nss

has something corrupt. But based on a different post, I don't think it's the nss directory, as apache is running fine, so I think it is the /opt/mse/locsrc/ssl directory. But I don't want to do anything that will make things any worse.

chris

[Brian McPhillips]

Hi Guys,

Did you ever get this resolved?

I'm having a similiar issue. I have a wlc 8500 running v8.0.1, MSE is v8 and Prime is V2.1.

When i add mse on prime, MSE is added to the Auth list on the WLC automatically as a LBS-SSC. IF i change that to a SSC MSE complain about a hash key mismatch.

When i click NMSP status i get a time mismatch. i have set all 3 servers to sync to the same ntp server. The wlc is set to GMT, Prime and MSE are set to BST.  When  i click the NMSP status it says there is a time issue, but it shows that the wlc time and the MSE time are exactly the same.

Not sure what else to try apart from a MSE rebuild.

[Scott Fella]

You need to define the new hashkey. Here is a support link that guides you to obtaining that hashkey:

https://supportforums.cisco.com/discussion/11053316/mse-location-problem-wcs-map

Scott

[Brian McPhillips]

HI scott, Thanks for replying.

The hash key seems to be already defined as per below, once i added MSE it seemed to auto generate with the WLC. i have attached the screenshot error on nmsp status.

I can also see the ssl Handshake errors increasing

(Cisco Controller) >show auth-list

Authorize MIC APs against Auth-list or AAA ...... disabled
Authorize LSC APs against Auth-List ............. disabled
APs Allowed to Join
  AP with Manufacturing Installed Certificate.... yes
  AP with Self-Signed Certificate................ no
  AP with Locally Significant Certificate........ no

Mac Addr                  Cert Type    Key Hash
-----------------------   ----------   ------------------------------------------
00:0c:29:de:aa:4c         LBS-SSC      62d6c2d230f87615b1583394277f4cf59a451d96

cmd> show server-auth-info
invoke command: com.aes.server.cli.CmdGetServerAuthInfo
AesLog queue high mark: 50000
AesLog queue low mark: 500
----------------
Server Auth Info
----------------
MAC Address: 00:0c:29:de:aa:4c
SHA1 Key Hash: 62d6c2d230f87615b1583394277f4cf59a451d96
SHA2 Key Hash: 8579084679da0a14b0b07c3ca6b262d12b0a0a4ea3521668e1922d62f42ad1f6
Certificate Type: SSC

[Scott Fella]

Have you tries removing it an adding it back?

Scott

[Brian McPhillips]

Hi Scott,

yeah have removed and re added a few times. have rebooted the MSE a few times. I haven't rebooted the WLC as yet but might try that later tonight when there are no clients connected.

[Brian McPhillips]

Just to let you all know, i got it resolved. it seems MSE version 8 uses  SHA 256. So i copied that string from MSE and changed to SHA 256 on the controller and it worked straight away,

[robertschaffer]

That was also the fix for me. In the WLC GUI delete the MSE created MAC address under Security>AAA>AP Polices. You can not create an AP Authorization with SHA2(256) with the GUI. Go to the cmd line of the WLC and run.

config auth-list add sha256-lbs-ssc (MAC of the MSE in xx:yy format) SHA2 Key Hash

[cmarva]

There are some differences in your setup from mine, but I definitely had a hash mismatch and a manual copy fixed things. As for the time sync thing, I really don't know if the different timezones would cause this or not. But it seems from the screenshot that it's not even trying to establish the nmsp connection due to this. Maybe Scott or someone can chime in here regarding timezones. - chris

[cmarva]

Brian, and all,

basically, the result is that you have to manually copy the hash. FIrst, determine the hash on the MSE, and then copy it via CLI onto the controller. I did find a document that was specific to the "converged access" gear, but is applicable to all gear I guess nowadays.

The link to the document is http://www.cisco.com/c/en/us/support/docs/wireless/5700-series-wireless-lan-controllers/117477-technote-addmac-00.html, and should get you going.

Please respond whether or not this worked for you, and I can send you some other stuff from the case offline. Thanks - chris

[Johnathan Waas]

This discussion was very helpful.  We had the same problem too.  Cisco has created a Bug ID for tracking the issue:  CSCuq50069 - SHA1 key cipher not working between WLC 80 and MSE 80 CCO versions.  The Bug appears to be resolved in 8.0.132.0 code, but no workaround was mentioned.  I was able to resolve the issue by SSH'ing to each of the MSE 3365's, logging in, issuing the show server-auth-info command and copying the output to a notepad file.  Once I had the MSE's mac address and the SHA2 hash, I SSH'ed to the WLC and from the CLI I entered this command:  config auth-list add sha256-lbs-ssc <Mac address > <40bit Key> and replaced the Mac address for that of the MSE and the 40bit Key with that of the SHA2 hash.  This resolved my issue.

[janvelgado_16]

Hi Johnathan Waas,

I followed your CLI Command. And it works!

THANK YOU! - Jan

[cooperd5]

Had this issue myself when upgrading MSE from 8.0.140.9 to 8.0.150.0 with Prime Infra 3.4, WLC 8510s @8.0 and 8.3 ,  - no sync.

Several steps needed to fix it:

enable TLS v1.0 in MSE (via /opt/mse/setup/setup.sh)
delete/re-add MSE in Prime
enable R/W SNMP in WLC for Prime - then sync
Re-added MSE mac address and SHA2 key to WLC (on one WLC only, 3/4 were OK)

Hope this helps others!

Darren