Re: Nexus Upgrade 5.1.5 Pre-Upgrade check failed

alanjbrown · ‎10-10-2011

I'm looking to upgrade from 4.2.4 to 5.1.5 and after running the impact command got the following message:

show install all impact kickstart bootflash:///n7000-s1-kickstart.5.1.5.bin system bootflash:///n7000-s1-dk9.5.1.5.bin

Pre-Upgrade check failed. Return code 0x40930062 (free space in the filesystem is below threshold)

Does anyone know what this might mean? and how to resolve?

Secondly, the release document have been updated (6/10/2011) to show that there is an ISSU upgrade from 4.2.6 or 4.2.8 but no mention of 4.2.4.

Can anyone advise what's going to happen when I go from 4.2.4 to 5.1.5?

thks,

Al

alanjbrown · ‎10-11-2011

I have investigated further and found the following issue:

show system internal log install detail

check_freespace: /var/sysmgr total blocks: 262144, free blocks: 0, space being used: 100%

fsm_action_pre_verification: Free space in "/var/sysmgr" partition is below threshold: 80

check_freespace: /var/sysmgr total blocks: 262144, free blocks: 0, space being used: 100%
fsm_action_pre_verification: Free space in "/var/sysmgr" partition is below threshold: 80

show system internal flash

Mount-on 1K-blocks Used Available Use% Filesystem

/var/sysmgr 1048576 1048576 0 100 none

See

http://docwiki.cisco.com/wiki/Cisco_Nexus_7000_Series_NX-OS_Troubleshooting_Guide_--_Troubleshooting_Memory

I will look into getting this file cleared.

I have also run install check on another N7K and this shows non-disruptive upgrade from 4.2.4 to 5.1.5 however I would still like to see this confirmed.

Compatibility check is done:
Module bootable          Impact Install-type Reason
------ -------- -------------- ------------ ------
     2       yes non-disruptive       rolling
     3       yes non-disruptive       rolling
     4       yes non-disruptive       rolling
     5       yes non-disruptive         reset
     6       yes non-disruptive         reset
     7       yes non-disruptive       rolling
     8       yes non-disruptive       rolling

thks,

Al

Ruhann Du Plessis · ‎10-12-2011

Hello Alan

Always refer to the Release notes as to what supported for ISSU and what not.

The version jump you want to do will be disruptive.

Refer to the notes

If you however want to stay within ISSU, upgrade to 4.2.8 first using ISSU and then upgrade to 5.1.5 using ISSU again. That should allow hitless upgrade.

hth

alanjbrown · ‎10-13-2011

Hth,

Thanks for the feedback, although the install test says that the upgrade is non-disruptive, you and a couple of other contacts have said that the upgrade is disruptive.

The release notes don't explicitly mention 4.2.4 so the lesson here is if the release notes don't talk about your version then assume the worst and do a two stage upgrade.

We plan to upgrade in a couple of weeks and I'll post an update once finished.

Al

Ruhann Du Plessis · ‎10-13-2011

So I cannot say for certain that it would be disruptive. That does depend on the versions of code.

What is listed in the release notes, are not the only methods, but it has been test by cisco, and therefor it is safe to assume this will not cause downtime.

Look at this this way. Are you willing to risk downtime at the cost of customer experience, due to an uncertainty?

Doing a bunny hop upgrade will take you perhaps an extra 30-40 minutes. Is this really too much time for a hitless upgrade and no risk?

That is a business decision more than a technical decision.

hth

alanjbrown · ‎10-26-2011

Ruhann,

We performed the upgrade at the weekend and it was non-disruptive however we did encounter some issues. After getting to 4.2.8, we ran the install to 5.1.5 and it was shown to be a disruptive upgrade.

Compatibility check is done:

Module bootable Impact Install-type Reason

------ -------- -------------- ------------ ------

2 yes disruptive reset Current running-config is not supported by new image

3 yes disruptive reset Current running-config is not supported by new image

Additional info for this installation:
--------------------------------------

_

-----------------------------------------------------------------------------
Some features currently enabled are not available in the new image. Please use the 'show incompatibility-all system' command to obtain further information on what needs to be disabled in order to perform a non-disruptive installation.

show incompatibility-all system bootflash:///n7000-s1-dk9.5.1.5.bin

Checking incompatible configuration(s) for vdc MYNEXUS:
------------------------------------------------------------
The following configurations on active are incompatible with the system image
1) Service : vpc , Capability : CAP_FEATURE_VPC_GRACEFUL_TYPE1_CHK
Description : vPC "graceful type1 consistency-check" is configured under "vpc domain"
Capability requirement : STRICT
Disable command : Remove "graceful consistency-check" config from vPC domain

show running-config vpc all
!Time: Sat Oct 22 14:09:57 2011

version 4.2(8)
feature vpc

vpc domain 20

role priority 100

-----------------------------------------------------------------------------

vpc domain 20
role priority 100

graceful consistency-check

It appears that when we upgraded from 4.2.4 to 4.2.8, the graceful consistency-check feature had been added by default. It was diffcult to get any details as all the documents seem to suggest this feature wasn't added until 5.2.

Anyway we removed the line of config and the upgrade continued non-disruptively to 5.1.5.

Al

tneuhuber · ‎08-07-2013

hello Alan,

i have also a 100% filled /var/sysmgr, but i am not able to free some space, executing filesys delete command, i get a new error; eg:

filesys delete /var/sysmgr/work/core.3842

"Files are not supposed to be deleted from this directory"

if i look deeper in work directory, i will see a lot of core.XXXX files, thinks this are dump files about 50MB each.

sh system internal dir /var/sysmgr/work

./ 740

../ 300

core.6014 0

core.5996 0

core.5984 0

core.5927 0

core.5921 0

core.5860 0

core.5838 0

core.5836 0

core.5362 0

core.5310 0

core.5308 0

core.5306 0

core.5290 0

core.5286 0

core.5282 0

core.5280 33574912

core.5281 26292224

core.5239 55975936

core.5237 55975936

core.5221 55975936

core.5211 55975936

core.5198 55975936

core.4832 55975936

core.4830 55975936

core.4822 55975936

core.4820 55975936

core.4790 55975936

core.4788 55975936

core.4724 55877632

core.4723 55877632

core.4721 55877632

core.4722 55877632

core.4672 55881728

core.4576 55877632

core.3842 54665216

do know how to delete this files?

theres is no help on the filesys command, due to it is hidden!!!

reload has been already done, but this does not free space!

thx in advance, best regards

Thomas

alanjbrown · ‎08-08-2013

Thomas,

This was some time ago but I've been through some old mail and found this response.

"TAC have a plugin to access the kernel which they can use that clears out the files without having to reboot the switch. These are non essential files that would be cleared if the switch was rebooted anyway."

I guess your next step is to contact TAC and get them to remove the files.

Alan

S Vivek Iyer · ‎08-14-2013

Hi Alan,

I also ran into this same issue. Was checking the blog and found this one.

Error:

xxxxxxxxxx# sh install all impact kickstart m9100-s2ek9-kickstart-mz.5.0.7.bin system m9100-s2ek9-mz.5.0.7.bin

Pre-Upgrade check failed. Return code 0x40930062 (free space in the filesystem is below threshold).

check_freespace: /var/tmp total blocks: 38400, free blocks: 11038, space being used: 71%

fsm_action_pre_verification: Free space in "/var/tmp" partition is below threshold: 70

/var/tmp 153600 109448 44152 72 none

/var/sysmgr 153600 5948 147652 4 none

Was trying to list internal Dir files. but seems those commands dont support in 9124 model .

Getting error. As "Invalid Commad"

In Cisco NX-OS release 4.2(4) and later releases, use the following commands to display and delete the problem files from the CLI:

The show system internal dir full directory path command lists all the files and sizes for the specified path (hidden command).

The filesys delete full file path command deletes a specific file (hidden command).

Note:

Use caution when using this command. You cannot recover a deleted file.

Is there any other way apart from reboot , we can correct it..

Vivek

grant.maynard · ‎05-01-2014

i don't know whether you resolved your problem but i do know you can get "free space in the filesystem is below threshold" errors when attempting upgrade if the Nexus has not been rebooted since first config. This clears /var/tmp.

N5548# show system internal log install detail
<Fri Feb 27 11:53:27 2009> check_freespace: check_freespace: /var/tmp total blocks: 76800, free blocks: 29874, space being used: 61%
<Fri Feb 27 11:53:27 2009> fsm_action_pre_verification: fsm_action_pre_verification: Free space in "/var/tmp" partition is below threshold: 10

N5548# show system internal flash
Mount-on                  1K-blocks      Used   Available   Use% Filesystem
/var/tmp                     307200    187704      119496     62   none
/var/sysmgr                 1024000     20196     1003804      2   none
N5548#

S Vivek Iyer · ‎05-01-2014

Not sure what problem was in switch , reboot worked for me.

jkukunuru · ‎05-15-2019

I have received the same error on Nexus 3548s. Tried to execute " filesys delete......" command but it looks like we cannot do that on 3ks. I have read somewhere online that TAC will be able to help clearing those files but we didnt have support plan. So I did the classic trick - reload the switch. After the reload, when I installed the images, everything worked fine!