C9300 missing Flash: and not able to write config

[RichySlo]

Hii everyone,

I am currently running into an issue when I try to upload a new .bin image to one of our C9300-48P switches.

When I try to upload the image using "copy tftp: flash: " I get the error that says "permission denied"
When I check the directory using "dir flash:" it shows an empty directory. The switch is part of a stack and I tried "Dir flash-2:" it shows all the expected files, also I am able to copy the new firmware image to this directory but nothing to "flash:" I tried "flash0:" and "flash-1" but both options show an empty directory.

Another thing that I noticed is when I try to copy the running-config to the startup-config it comes up with a bunch of errors, see below:

NL-HQ-MER-CORE01#copy run startup-config
Destination filename [startup-config]?
Building configuration...
/usr/binos/conf/in.telnetd.sh: line 28: psimple.sh: No such file or directory
/us r/binos/conf/in.telnetd.sh: line 30: /usr/binos/conf/hwutil.sh: Input/output err or
/usr/binos/conf/in.telnetd.sh: line 125: hwutil_get_slot: command not found
/us r/binos/conf/in.telnetd.sh: line 135: pglobals.sh: No such file or directory
/usr /binos/conf/in.telnetd.sh: line 138: BINOS_SW_BASE: unbound variable
[OK]

Does anybody know how I might be able to fix it? I tried "write memory" as well but get the same output/result.

[marce1000]

                     - Try fsck flash:

M.

[RichySlo]

Unfortunately i get the following message when i try "fsck flash: " "could not create request: invalid argument" the line after that says "fsck of flash: completed"

Any other tips?

[marce1000]

 - Reboot the particular stack member , attach a console to it and scrutinize the boot process , then try again afterwards , note that there is a chance that it has a fatal hardware condition , this would lead then to network outage for those connections being served by it (well reboot does that anyway) , 

 M.

[RichySlo]

I currently don't have the option to reboot this particular stack member, it is part of the core stack which is not redundant.
I have been hired to improve the network at this particular company as they don't have somebody to take care of the networking part of their infrastructure. switch has an uptime of 3 years and no changes have been made since then.

[marce1000]

                  >... switch has an uptime of 3 years and no changes have been made since then.
 - That is a particular 'drawback' because then an hardware issue could have creeped in becoming visible on next restart, it would be advisable to check or do this during business downtime, the reason being that you risk having to take action during business prime time...

 M.

[pieterh]

please post output of "show boot" 

maybe the system has been booted from another filesystem than flash?
e.g.  a tftp server or maybe from USB-drive (usbflash0:)?
the normal flash (of this stack member) may be unformatted

[RichySlo]

It boots from flash:packeges.conf. i copied the output of the command below.

NL-HQ-MER-CORE01#sh boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable does not exist

Boot Variables on next reload:
BOOT variable = flash:packages.conf
Manual Boot = no
Enable Break = no
Boot Mode = DEVICE
iPXE Timeout = 0

[balaji.bandi]

what version of code running ?- show version will help you.

how many switches in the stack ? (2 or more ?) show switch give you more information

So the device is running but you can not make any changes due to that flash gone bad here looks like.

first step you need to do is, show run (copy the config out of the box as soon as possible) if show run have issue, atleast you should take backup of show startup (known good config)

since you mentioned this is critical core and stack,  i am expected all the device dual homed connect to switch 1 and switch 2 here (i am guessing) - so in this you can reload master switch, so slave will take care of master role you will not see any downtime if the network connections are dual homed.

I have been hired to improve the network at this particular company as they don't have somebody to take care of the networking part of their infrastructure. switch has an uptime of 3 years and no changes have been made since then.

how would you improve you do not have downtime or maintenance window, Technology is not magic, it has the process.

you need to explain the risk of the device, either take the bullent now with proper process, or wait for a call in the middle of the night to have more downtime (that is management decision)

i would not expect when the switch reboot - come back as expected, so prepare of worst situation, and when you get chance same time fixing flash and upgrade to 17.3.6 for fix most bugs.

good luck...!

[RichySlo]

If by version of code you mean the firmware version then it is: Cisco IOS XE Software, Version 16.12.02
The stack consists of 4 C9300 switches of which 3x 48p and 1x 24p.

Unfortunately the connected devices are not connected redundantly(its one big mess here) so reloading the master would sill result in outage of a couple device's. 

I have ordered a spare unit, as soon as it is delivered I will reload the master switch and troubleshoot the issue's that I stumble upon. If the flash remains empty and unwriteable I will replace the master.

Thanks,

[Leo Laohoo]

What firmware is this stack running on?

[RichySlo]

this is the output of "show version"

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 65 C9300-48P 16.12.02 CAT9K_IOSXE INSTALL
2 65 C9300-48P 16.12.02 CAT9K_IOSXE INSTALL
3 65 C9300-48P 16.12.02 CAT9K_IOSXE INSTALL
4 41 C9300-24P 16.12.02 CAT9K_IOSXE INSTALL

[Leo Laohoo]

What is the uptime of this stack? 

Please post the complete output to the following commands: 

• sh platform resources
• sh platform software status con brief

16.12.X is one of the buggiest train Cisco has ever given a "gold star". 

[RichySlo]

sh platform resources shows : Nothing

NL-HQ-MER-CORE01#sh platform resources
**State Acronym: H - Healthy, W - Warning, C - Critical
Resource Usage Max Warning Critical State
----------------------------------------------------------------------------------------------------

And sh platform software status control-processor brief keeps saying "Could not create request: Invalid argument" I tried to write the output to a file but I got the message "permission denied" because it wants to write it to flash: I think.

any suggestions?

[Leo Laohoo]

So be it.  The "hard way" `tis then.  

Post the complete "first page" to the output of the command "sh process memory platform sort location switch active r0".