01-17-2014 12:27 PM
I'm attempting to upgrade a pair of Nexus 5596's in my lab to the latest code available, and I'm getting an error on one of the switches (the other switch upgraded just fine.
Software
BIOS: version 3.6.0
loader: version N/A
kickstart: version 6.0(2)N1(2)
system: version 6.0(2)N1(2)
Power Sequencer Firmware:
Module 1: version v5.0
Microcontroller Firmware: version v1.0.0.2
SFP uC: Module 1: v1.1.0.0
QSFP uC: Module not detected
BIOS compile time: 05/09/2012
kickstart image file is: bootflash:///n5000-uk9-kickstart.6.0.2.N1.2.bin
kickstart compile time: 3/14/2013 1:00:00 [03/14/2013 08:53:55]
system image file is: bootflash:///n5000-uk9.6.0.2.N1.2.bin
system compile time: 3/14/2013 1:00:00 [03/14/2013 11:28:50]
Hardware
cisco Nexus 5596 Chassis ("O2 48X10GE/Modular Supervisor")
Intel(R) Xeon(R) CPU with 8262944 kB of memory.
Processor Board ID FOC17360JQS
Device name: Nexus5596.A2-C06
bootflash: 2007040 kB
Nexus5596.A2-C06# sh file bootflash:n5000-uk9-kickstart.6.0.2.N2.3.bin md5sum
ff2693db82a73668d0b8a07606fcfe54
Nexus5596.A2-C06# sh file bootflash:n5000-uk9.6.0.2.N2.3.bin md5sum
7cff5f41ffaee8655104af09fa8324d5
I'm getting the following error :
Nexus5596.A2-C06# sh install all impact kickstart bootflash:n5000-uk9-kickstart.6.0.2.N2.3.bin system bootflash:n5000-uk9.6.0.2.N2.3.bin
Verifying image bootflash:/n5000-uk9-kickstart.6.0.2.N2.3.bin for boot variable "kickstart".
[####################] 100% -- SUCCESS
Verifying image bootflash:/n5000-uk9.6.0.2.N2.3.bin for boot variable "system".
[####################] 100% -- SUCCESS
Verifying image type.
[####################] 100% -- SUCCESS
Extracting "system" version from image bootflash:/n5000-uk9.6.0.2.N2.3.bin.
[# ] 0% -- FAIL. Return code 0x404F0003 (SRG file not present/cannot be opened).
Pre-upgrade check failed. Return code 0x40930012 (SRG extraction failed).
Does anyone have any idea what could be causing this? I got the error on one switch, but no errors or issues at all on the other switch. I've tried restarting the switch to no avail.
01-18-2014 09:23 AM
It appears you are loading two different images:
good switch
bootflash:///n5000-uk9.6.0.2.N1.2.bin
bad switch
bootflash:/n5000-uk9.6.0.2.N2.3.bin
Can you try the good switch image on the other one?
HTH
01-18-2014 03:11 PM
All of the information posted above is from one switch. The switch is currently running 6.0.2.N1.2, and I'm trying to upgrade to 6.0.2.N2.3. The other switch has already upgraded to 6.0.2.N2.3.
I've run and re-run this on this switch many times, and the only thing that differs between this switch and the switch that succeeded is that "verifying image type" takes quite some time on the switch that upgraded properly, and it goes very quickly on the switch that is having problems.
01-18-2014 03:23 PM
There is a bug seen on this in previous releases on this behaviour. It looks like your flash disk is full with some /tmp files.
Let's check your disk space with command: " sh system internal flash "
Look at the /var/tmp, is it full ? or do you see another directory full?
Some of the temp folders get cleaned up with a reload, otherwise we can load a debug-plugin to clean the partition that has been filled.
If we dont want to load a debug plugin, You can use the hidden CLI: show system internal dir /var/tmp and to delete the unwanted files (make sure you don't delete directories). filesys delete /var/tmp/filename(s) Hope this helps.
Cheers,
-amit singh
01-18-2014 06:39 PM
Yeah, I had read about that one, but I'm not seeing it as an issue right now, unless I'm missing something. Here's the output of the disk space report :
Nexus5596.A2-C06# sh system internal flash
Mount-on 1K-blocks Used Available Use% Filesystem
/ 409600 72812 336788 18 /dev/root
/proc 0 0 0 0 proc
/post 2048 4 2044 1 none
/var 409600 72812 336788 18 none
/sys 0 0 0 0 none
/isan 1536000 765456 770544 50 none
/var/tmp 307200 100 307100 1 none
/var/sysmgr 1024000 4 1023996 1 none
/nxos/tmp 40960 32 40928 1 none
/var/sysmgr/ftp 409600 68 409532 1 none
/var/sysmgr/ftp/cores 102400 0 102400 0 none
/callhome 61440 0 61440 0 none
/dev/shm 524288 284536 239752 55 none
/volatile 153600 0 153600 0 none
/debug 20480 0 20480 0 none
/dev/mqueue 0 0 0 0 none
/debugfs 0 0 0 0 nodev
/mnt/plog 49024 1360 47664 3 /dev/mtdblock2
/mnt/cfg/0 113794 4347 103572 5 /dev/sda5
/mnt/cfg/1 113794 4347 103572 5 /dev/sda6
/var/sysmgr/startup-cfg 409600 2860 406740 1 none
/dev/pts 0 0 0 0 devpts
/mnt/pss 113802 5402 102524 6 /dev/sda4
/bootflash 1612212 562324 967988 37 /dev/sda3
/proc/bus/usb 0 0 0 0 none
01-18-2014 06:40 PM
Also, here's the contents of var/tmp
Nexus5596.A2-C06# show system internal dir /var/tmp
./ 680
../ 260
mysrg 1984
upgrade_status_file 1
mylogs 760
bootloader_ver 0
m2rib_ascii.3623 34
_clis_parse_tree_.log 2706
fcoe_mgr_init.log 15815
CSCub19397_debug 237
fcoe_mgr 0
fc-pcm-log-4225 757
fabric_start_cfg_mgr.log 761
issu_helper.log 224
stp.log.1 4316
fwm1.out 0
ntpd_1 0
ntp_client_1 0
igmp_restart.log 0
cfs_mac.log 332
radius_debug.log 232
security_stderr 77
security_stdout 0
aaa_debug.log 519
lcc.log 694
security_debug.log 647
cmp_slot_id.1 0
mvsh.3260 0
lockcisco 0
.flexlm/ 60
logs@ 20
first_setup.log 4745
boot_uptime.log 56
vsh/ 40
01-19-2014 11:28 AM
Hi Matt,
I have not tried this ever but have you tried deleting the unwanted files from the var/tmp directory above? Tried deleting few of these except the directories and see if that helps. Also, try to remove some of the files from the flash on the switch, the unwanted ones.
In case that still does not help, tried a different NX-OS version or open a TAC SR.
Cheers,
-amit singh
01-20-2014 01:35 PM
Just an update - I tested on two additional 5596's, and those both upgraded fine.
I loaded the images onto a USB drive and actually plugged in to the system (rather than SFTP'ing them), tested again and it still failed.
I reloaded the switch, renamed images to .old, uploaded new copies, tried again, still got the same error.
I reloaded the switch a third time, and lo and behold it worked flawlessly. I'm not exactly sure why it took three reloads of the swtich to get that to function, but it did. I'm thankful that this was in a lab environment, but I am a little concerned about putting this into production now. I don't really want to have to reload a switch 3 times just to get it to take an OS update.
01-20-2014 01:51 PM
Hi Matt, Many thanks for the update. I read it internally and one of the engineer suggested that by re-booting the box deletes the tmp files in the memory. I would not do that in a production environment as you said. BTW, did you try deleteting the files in tmp directory manaually if that helped at all or not?
Glad it worked for you but yes that needs to be fixed in a better way. I will try raising it with some engineers.
Cheers,
-amit singh
01-20-2014 01:56 PM
I did not remove any files from the /var/tmp directory, as I was not sure what ones were vital, and which ones could be removed. FYI this is what the flash system looks like after the upgrade.
Nexus5596.A2-C06# sh ver
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Documents: http://www.cisco.com/en/US/products/ps9372/tsd_products_support_series_home.html
Copyright (c) 2002-2013, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software are covered under the GNU Public
License. A copy of the license is available at
http://www.gnu.org/licenses/gpl.html.
Software
BIOS: version 3.6.0
loader: version N/A
kickstart: version 6.0(2)N2(3)
system: version 6.0(2)N2(3)
Power Sequencer Firmware:
Module 1: version v5.0
Microcontroller Firmware: version v1.0.0.2
SFP uC: Module 1: v1.1.0.0
QSFP uC: Module not detected
BIOS compile time: 05/09/2012
kickstart image file is: bootflash:///n5000-uk9-kickstart.6.0.2.N2.3.bin
kickstart compile time: 12/17/2013 2:00:00 [12/17/2013 12:52:59]
system image file is: bootflash:///n5000-uk9.6.0.2.N2.3.bin
system compile time: 12/17/2013 2:00:00 [12/17/2013 16:02:31]
Hardware
cisco Nexus 5596 Chassis ("O2 48X10GE/Modular Supervisor")
Intel(R) Xeon(R) CPU with 8253856 kB of memory.
Processor Board ID FOC17360JQS
Device name: Nexus5596.A2-C06
bootflash: 2007040 kB
Kernel uptime is 0 day(s), 0 hour(s), 52 minute(s), 23 second(s)
Last reset at 128479 usecs after Tue Apr 28 20:57:01 2009
Reason: Reset due to upgrade
System version: 6.0(2)N1(2)
Service:
plugin
Core Plugin, Ethernet Plugin, Fc Plugin
Nexus5596.A2-C06# sh system internal flash
Mount-on 1K-blocks Used Available Use% Filesystem
/ 409600 74520 335080 19 /dev/root
/proc 0 0 0 0 proc
/sys 0 0 0 0 none
/isan 1536000 763412 772588 50 none
/var/tmp 307200 92 307108 1 none
/var/sysmgr 1024000 24152 999848 3 none
/nxos/tmp 40960 52 40908 1 none
/var/sysmgr/ftp 409600 72 409528 1 none
/var/sysmgr/ftp/cores 102400 0 102400 0 none
/callhome 61440 0 61440 0 none
/dev/shm 524288 285464 238824 55 none
/volatile 153600 0 153600 0 none
/debug 20480 0 20480 0 none
/dev/mqueue 0 0 0 0 none
/debugfs 0 0 0 0 nodev
/mnt/plog 49024 1380 47644 3 /dev/mtdblock
2
/mnt/cfg/0 113794 4304 103615 4 /dev/sda5
/mnt/cfg/1 113794 4304 103615 4 /dev/sda6
/var/sysmgr/startup-cfg 409600 2856 406744 1 none
/dev/pts 0 0 0 0 devpts
/mnt/pss 113802 5833 102093 6 /dev/sda4
/bootflash 1612212 562348 967964 37 /dev/sda3
/usbslot1 982016 258816 723200 27 /dev/sdb1
/proc/bus/usb 0 0 0 0 none
/cgroup 0 0 0 0 vdccontrol
Nexus5596.A2-C06# sh sys internal dir /var/tmp
./ 1020
../ 260
system.bootstrap.cfg.log 0
sysmgr_startcfg_flash.lock 0
dhcp_snoop_startup_cfg.trm 0
dhcp_snoop_startup_cfg.dat 0
dhcp_snoop_startup_cfg.map 0
dhcp_snoop_startup_cfg.pss 848
upgrade_status_file 1
install_all_cli.log 244
_clis_parse_tree_.log 2706
stp.log.1 3907
ntpd_1 0
fabric_start_cfg_mgr.log 359
ntp_client_1 0
fs_upg_fspf_hs_time.txt 10
fwm1.out 0
radius_debug.log 232
cfs_mac.log 360
fcoe_mgr 0
aaa_debug.log 407
security_stderr 77
security_stdout 0
igmp_restart.log 0
fs_upg_lacp_hs_time.txt 10
CSCub19397_debug 240
security_debug.log 656
fs_upg_ctrl_up_time.txt 10
issu_helper.log 1109
lcc.log 694
fc-pcm-log-3378 829
libusd_0x30b.3343 0
libusd_0x2fd.3339 0
libusd_0x2e7.3344 0
cmp_slot_id.1 0
mvsh.3326 0
libusd_0x2a4.3314 0
libusd_0x323.3312 0
lockcisco 0
.flexlm/ 60
dhcp_snoop_running_cfg.trm 0
dhcp_snoop_running_cfg.dat 0
dhcp_snoop_running_cfg.map 0
dhcp_snoop_running_cfg.pss 848
assoc_mgr_cnv.log 366
infraconvertlibpsscheck 0
infraconvertcheck 0
logs@ 20
first_setup.log 3050
boot_uptime.log 56
vsh/ 40
01-21-2014 09:40 AM
So, I'm now upgrading the 5548's that I have in the same lab, and I'm getting the same error. I'd like to try nuking some of the files in the /var/tmp folder, but I'm trying to figure out what ones to delete. Current file structure looks like this :
Nexus5548.A-C04# sh sys int dir /var/tmp
./ 820
../ 260
mysrg 1984
upgrade_status_file 1
util_cli_history_admin 0
cfg_status.log 7
exclude 0
sysmgr_startcfg_flash.lock 0
m2rib_ascii.3996 68
_clis_parse_tree_.log 2706
fcoe_mgr_init.log 15815
CSCub19397_debug 237
fcoe_mgr 0
fc-pcm-log-4917 757
fabric_start_cfg_mgr.log 1496
issu_helper.log 224
stp.log.1 4316
fwm1.out 0
ntpd_1 0
cfs_mac.log 332
ntp_client_1 0
igmp_restart.log 0
radius_debug.log 232
security_stderr 77
security_stdout 0
security_debug.log 647
aaa_debug.log 0
libusd_0x30b.3252 0
lcc.log 694
libusd_0x2fd.3248 0
libusd_0x2e7.3253 0
mvsh.3237 0
libusd_0x2a4.3226 0
libusd_0x323.3224 0
cmp_slot_id.1 0
lockcisco 0
.flexlm/ 60
logs@ 20
first_setup.log 0
boot_uptime.log 56
vsh/ 40
01-21-2014 12:24 PM
Hi Matt,
Are you upgrading to the same NX-OS? What if you try to use a different NX-OS? Do you see the same symptoms?
Try deleting the .log files and see if this helps. Do not delete any directories.
Cheers,
-amit singh
01-21-2014 12:44 PM
I am upgrading to 6.0(2)N2(3) - only difference is that I'm upgrading from 6.0(2)N2(2) rather than 6.0(2)N1(2). I'll delete logs and see what we see.
01-21-2014 12:56 PM
Deleted all .log files, still having the exact same error. Here's the current directory listing :
Nexus5548.A-C04# sh sys int dir /var/tmp
./ 860
../ 260
mysrg 1984
mylogs 330
bootloader_ver 0
upgrade_status_file 1
util_cli_history_admin 0
cfg_status.log 0
exclude 0
sysmgr_startcfg_flash.lock 0
m2rib_ascii.3996 68
_clis_parse_tree_.log 0
fcoe_mgr_init.log 0
CSCub19397_debug 237
fcoe_mgr 0
fc-pcm-log-4917 757
fabric_start_cfg_mgr.log 0
issu_helper.log 0
stp.log.1 0
fwm1.out 0
ntpd_1 0
cfs_mac.log 0
ntp_client_1 0
igmp_restart.log 0
radius_debug.log 0
security_stderr 77
security_stdout 0
security_debug.log 0
aaa_debug.log 0
libusd_0x30b.3252 0
lcc.log 0
libusd_0x2fd.3248 0
libusd_0x2e7.3253 0
mvsh.3237 0
libusd_0x2a4.3226 0
libusd_0x323.3224 0
cmp_slot_id.1 0
lockcisco 0
.flexlm/ 60
logs@ 20
first_setup.log 0
boot_uptime.log 0
vsh/ 40
01-23-2014 08:32 AM
Just an update on this. We are working with TAC and engineering on this.
Cheers,
-amit singh
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide