cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
3380
Views
0
Helpful
3
Replies

Recovering Fabric Interconnects Fails

MarcST1984
Level 1
Level 1

Hello all,
I hope you can help me.
I had to upgrade both FI nodes manually.
After upgrade I want to configure the FIs and I get the following error message.

I am happy about possible solutions

 

Error:


Enter the admin password of the Peer Fabric Interconnect:
Connect to Peer Fabric Interconnect.... Done
Retrieving the configuration from the Peer Fabric Interconnect.... done
The installer has detected that the peer fabric interconnect is using a different firmware version than the local fabric. Cannot join the cluster.

Local Fabric Interconnect
UCSM version: 4.1(3f)
UCSM SP version : 4.1(3)SP0
Kernel Version : 5.0(3)N2(4.13f)
System Version : 5.0(3)N2(4.13f)
System SP version:
local_model_no : UCS-FI-6248UP

Peer Fabric Interconnect
UCSM version: 4.1(3f)
UCSM SP version: 4.1(3)SP0
Kernel Version :
System Version :
System SP version:
peer_model_no : UCS-FI-6248UP


Do you want to upgrade the firmware on this fabric interconnect to the version of the peer? (y/n): y
Upgrade the firmware of the fabric interconnect....... [ Please do not press Ctrl+c while updating the firmware ]
Updating the images
Please wait for the firmware upgrade to complete.... ERROR: Copy of images failed. Possible reasons are network failure or file not found.
Please try again or configure the device as a standalone device.

 

I used this guide:

https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/ucs-manager/CLI-User-Guides/Firmware-Mgmt/4-0/b_UCSM_CLI_Firmware_Management_Guide_4-0/b_UCSM_CLI_Firmware_Management_Guide_4-0_chapter_0101.html#task_D739CF0464104D1D835106992DECE0F2

 

3 Replies 3

Kirk J
Cisco Employee
Cisco Employee

Suspect there is file system issue on partition supporting bootflash (/dev/sda3), or issue with underlying physical SSD.

Going to need a TAC case to get the dplug file to run the e2fsck.

Alternatively, bring up the FI in standalone (if it will successfully boot up), and then from local-mgmt:

connect local

local-mgmt: reboot e2fsck

 

If this corrects file system errors on stand alone, connect again to local-mgmt: erase config

Re-attempt to join the FI to the existing one.

 

Kirk...

i gonna try

MarcST1984
Level 1
Level 1

Hello,

no chance. I have performed the procedure.
It went very quickly, enclosed are the results.

 


Password:
Login incorrect

Cisco UCS 6200 Series Fabric Interconnect
login: admin
Password:
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2009, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php

 

ucsfabintm1a-mgt-A# connect local-mgmt
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2009, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php

ucsfabintm1a-mgt-A(local-mgmt)# reboot e2fsck
On reboot with the e2fsck option, the Fabric Interconnect's SSD undergoes a
thorough file system check and non-interactive repair process which could result in data loss. Before proceeding further,
you must ensure back up of the critical data including the Cisco UCS Manager database (yes/no):y
Invalid input: y
On reboot with the e2fsck option, the Fabric Interconnect's SSD undergoes a
thorough file system check and non-interactive repair process which could result in data loss. Before proceeding further,
you must ensure back up of the critical data including the Cisco UCS Manager database (yes/no):yes
nohup: ignoring input and redirecting stderr to stdout

Broadcast message from root (Tue Sep 21 11:12:06 2021):

All shells being terminated due to system /sbin/reboot

Cisco UCS 6200 Series Fabric Interconnect
ucsfabintm1a-mgt-A login: [ 2950.600173] Shutdown Ports..
[ 2950.634525] writing reset reason 9,
INIT: Sending all processes the TERM signal...
Sending all processes the KILL signal...
Saving random seed:
Syncing hardware clock to system time
Unmounting file systems: umount2: Device or resource busy
umount: /var/tmp: device is busy
umount2: Device or resource busy
umount: /var/tmp: device is busy

/var/tmp: c
Unmounting file systems (retry): umount2: Device or resource busy
umount: /var/tmp: device is busy
umount2: Device or resource busy
umount: /var/tmp: device is busy

/var/tmp: c
Unmounting file systems (retry): umount2: Device or resource busy
umount: /var/tmp: device is busy
umount2: Device or resource busy
umount: /var/tmp: device is busy

/var/tmp: c
umount2: Device or resource busy
umount: /var/tmp: device is busy
umount2: Device or resource busy
umount: /var/tmp: device is busy
mount: you must specify the filesystem type
mount: /var not mounted already, or bad option
Please stand by while rebooting the system...
[ 2994.573712] Restarting system.
[ 2994.610090] machine restart
[ 2994.643336] Resetting board (uc)

N5000 BIOS v.3.6.0, Wed 05/09/2012, 03:15 PM

 

 

 

 

 

 

 

 

 

 

 

B2
B2
Version 2.00.1201. Copyright (C) 2009 American Megatrends, Inc.
Booting kickstart image: bootflash:/installables/switch/ucs-6100-k9-kickstart.5
.0.3.N2.4.13f.bin....
...............................................................................
..............................................................Image verificatio
n OK

▒Usage: init 0123456SsQqAaBbCcUu
INIT: [ 11.875642] I2C - Mezz absent
Starting system POST.....
Executing Mod 1 1 SEEPROM Test:...done (0 seconds)
Executing Mod 1 1 GigE Port Test:....done (32 seconds)
Executing Mod 1 1 PCIE Test:.................done (0 seconds)
Mod 1 1 Post Completed Successfully
POST is completed
can't create lock file /var/lock/mtab~208: No such file or directory (use -n flag to override)
S10mount-ramfs.supnuovaca Mounting /isan 3000m
Mounted /isan
Creating /callhome..
Mounting /callhome..
Creating /callhome done.
Callhome spool file system init done.
nohup: redirecting stderr to stdout
autoneg unmodified, ignoring
autoneg unmodified, ignoring
Checking all filesystems.
.... done.
Checking NVRAM block device ... done
The startup-config won't be used until the next reboot.
.
Loading system software
Starting the smart check..
Uncompressing system image: bootflash:/installables/switch/ucs-6100-k9-system.5.0.3.N2.4.13f.bin

 

27+1 records in
27+1 records out
20480 bytes (20 kB) copied, 9.1331e-05 s, 224 MB/s
ethernet end-host mode on CA
FC end-host mode on CA
n_port virtualizer mode.
---------------------------------------------------------------
INIT: Entering runlevel: 3
touch: cannot touch `/var/lock/subsys/netfs': No such file or directory
Mounting other filesystems: mount: /dev/hd-usbslot1 is not a valid block device
[FAILED]
touch: cannot touch `/v
/isan/bin/muxif_config: fex vlan id: -f,4042
Set name-type for VLAN subsystem. Should be visible in /proc/net/vlan/config
Added VLAN with VID == 4042 to IF -:muxif:-
0000000000000000000000000
System is coming up ... Please wait ...
Changing of vsh_perm
modelNo=UCS-FI-6248UP
System is coming up ... Please wait ...
System is coming up ... Please wait ...
System is coming up ... Please wait ...
System is coming up ... Please wait ...
System is coming up ... Please wait ...
---------------------
enabled fc feature
---------------------
System is coming up ... Please wait ...
System is coming up ... Please wait ...
System is coming up ... Please wait ...
2021 Sep 21 11:18:17 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: CLIS: loading cmd files begin - clis
System is coming up ... Please wait ...
2021 Sep 21 11:18:20 %$ VDC-1 %$ Sep 21 11:18:20 %KERN-0-SYSTEM_MSG: [ 11.875642] I2C - Mezz absent - kernel
System is coming up ... Please wait ...
2021 Sep 21 11:18:27 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: CLIS: loading cmd files end - clis
2021 Sep 21 11:18:27 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: CLIS: init begin - clis
System is coming up ... Please wait ...
System is coming up ... Please wait ...
System is coming up ... Please wait ...
System is coming up ... Please wait ...
System is coming up ... Please wait ...
System is coming up ... Please wait ...
System is coming up ... Please wait ...
System is coming up ... Please wait ...
System is coming up ... Please wait ...
System is coming up ... Please wait ...
System is coming up ... Please wait ...
System is coming up ... Please wait ...
2021 Sep 21 11:19:31 %$ VDC-1 %$ %VDC_MGR-2-VDC_ONLINE: vdc 1 has come online
System is coming up ... Please wait ...
2021 Sep 21 11:19:51 ucsfabintm1a-mgt-A %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Running in PIO stats mode - carmelusd
update of ip/netmask on OOB succeeded


Cisco UCS 6200 Series Fabric Interconnect
ucsfabintm1a-mgt-A login:

Review Cisco Networking for a $25 gift card

Review Cisco Networking for a $25 gift card