Getting "Password could be wrong" error when trying to connect second FI to primary FI during initial configuration

[Tim Schroeder]

I am trying to build a 3.1(2e) UCS cluster using two UCS "Mini" FI's (6324)

I run through the initial console configuration on the first one. No problems.

I switch to the second FI and I get this:

Installer has detected the presence of a peer Fabric interconnect. This Fabric interconnect will be added to the cluster. Continue (y/n) ? y

  Enter the admin password of the peer Fabric interconnect:

    Connecting to peer Fabric interconnect... unable to connect! Password could be wrong.

    Please ensure that the authentication mode on peer Fabric interconnect is set to 'Local'

    Hit enter to try again or type 'restart' to start setup from beginning...

I have tried simple and complex passwords, and I am using local authentication. 

On the first FI that I already configured, I get this error:

ucs-pp-lab-B# 2011 Apr  1 06:32:01 ucs-pp-lab-B %$ VDC-1 %$ %DAEMON-2-SYSTEM_MSG: fatal: no matching cipher found: client aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se server aes128-ctr,aes192-ctr,aes256-ctr, - sshd[15949]

Any ideas?

[Wes Austin]

This is a brand new installation? It sounds like the UCSM versions might not be able to sync up correctly.

I would try to deploy that problem FI in standalone mode, check the UCSM/FI version and make sure they match the working FI.

Then, try to erase the configuration on the problem FI and add it back to the cluster.

connect local-mgmt

erase-configuration

[bobschalk]

Hi Wes,

Thanks for the reply, but I have already done that.  I had to go to standalone mode to install the FI firmware to ensure both cards matched.  To do so I went into standalone mode, upgraded to 3.1(2e), then wiped the configuration using the method you stated.  I did that for both cards, and then with both configurations wiped tried the cluster configuration which failed as stated above.  I then wiped card A, and started the cluster configuration with card B, but when I tried to couple card A to the configured card B I get the exact same error.

[bobschalk]

I swapped out the 6324 cards, and on one of my two 5108 chassis I was able to get the cards to pair up just fine, so I thought I was making progress.  However, on the other chassis, the cards now see eachother, but I am running into another issue.  I configured the A side just fine, but on the B side, I am getting this error:

Enter the configuration method. (console/gui) ? console

Installer has detected the presence of a peer Fabric interconnect. This Fabric interconnect will be added to the cluster. Continue (y/n) ? y

Enter the admin password of the peer Fabric interconnect:
Connecting to peer Fabric interconnect... done
Retrieving config from peer Fabric interconnect... done
/isan/bin/first-setup.core: line 1413: /isan/bin/sp_getversion.sh: No such file or directory
Peer Fabric interconnect Mgmt0 IPv4 Address: 172.19.101.3
Peer Fabric interconnect Mgmt0 IPv4 Netmask: 255.255.255.224
Cluster IPv4 address : 172.19.101.2

Peer FI is IPv4 Cluster enabled. Please Provide Local Fabric Interconnect Mgmt0 IPv4 Address

Physical Switch Mgmt0 IP address : 172.19.101.4

Apply and save the configuration (select 'no' if you want to re-enter)? (yes/no): yes
Applying configuration. Please wait.

Error while exchanging SSH keys. Rebooting in 30 secs

After that, it reboots and starts over at the configuration prompt.  Any idea on this one?

[Wes Austin]

When you booted each FI in standalone mode, did you make sure UCSM AND FI versions were matching between the two FI? I ask this because if UCSM is a 3.1.2 and FI is at 3.1.1, there will be a problem when attempting to sync them into an HA pair. 

I would recommend updating your FI individually to the latest release to move past any potential defects and then attempt to sync them.

[bobschalk]

Thank you very much Wes.  I am new to this, and the firmware guide Cisco provides for the 6324 that I was reading doesn't mention anything about a separate task for the FIs.  I rebuilt the FI in standalone mode and then did the following to activate the new image on the FI:

ucs-pp-lab-nhvb-B# scope fabric-interconnect A
ucs-pp-lab-nhvb-B /fabric-interconnect # activate firmware kernel-version 5.0(3)N2(3.12e)
ucs-pp-lab-nhvb-B /fabric-interconnect # activate firmware system-version 5.0(3)N2(3.12e)

ucs-pp-lab-nhvb-B /fabric-interconnect # commit-buffer

After about 45 minutes it came back, so I wiped the config, built the B side in standalone, did the same, and wiped it's configuration after the wait for activation.  After that I was able to build the FIs in cluster mode without any issue, and they are working together:

ucs-pp-lab-nhvb-B# show cluster state
Cluster Id: 0x74b6755c5cec11e0-0x9abe003a9c00a5c2

B: UP, SUBORDINATE
A: UP, PRIMARY

HA READY
ucs-pp-lab-nhvb-B#

Thank you very much for your help with this Wes!!

[Wes Austin]

No problem. I am glad to see it working!

[bobschalk]

Hi Wes,

One last theoretical question.  We have to be on 3.1(2e) right now to mirror the image on other existing platforms.  Let's say in a few months I have a 6324 module go bad in a platform running 3.1(2e) and we get an RMA that happens to be on new code which will result in the cypher issue.  As this chassis already has a working card in cluster mode, we wouldn't be able to install the replacement 6324 in the second slot and configure as standalone.  We also couldn't take a production chassis down to setup a standalone scenario like I had to do today, so how would we go about upgrading/downgrading the firmware on the new 6324 FI module without disrupting the remaining active module?  Would we need to have a spare 5108 chassis just lying around to do standalone FI module firmware adjustments?

[Wes Austin]

Without looking at system or audit logs to know the exact steps you took in the recent scenario, it would be hard to tell why you got into the situation you did.

When you replace a 6324 FI in the future, you would replace the physical module and re-cable. Upon POST, it will come up to the initial configuration dialog screen and would be detected by the peer. From here, a feature called firmware-autosync will automagically sync up the UCSM/FI versions and configuration between the two.

What I suspect happened in your scenario is that at some point the UCSM and FI versions were not in sync between the two FI. Like UCSM was updated, but not the system/kernel versions. That led to them not being able to copy over the images between one another and will cause the SSH mismatch error you saw. 

Even if you were to get into today's scenario again, you should be able to boot up the "replacement" FI in standalone mode and fix it the same way discussed previously. I would open a TAC case if you run into this again to have an assessment done also.

[bobschalk]

Fantastic explanation.  Thanks again for all your help Wes.