Can't write to flash1: in a 3 switch stack

[keithsauer507]

I have a 3 switch stack at a remote office consisting of a 48 port 3750X, 24 port 3750X PoE, and a 48 port 3750v2.  Currently this stack is functioning normally and is running 12.2(55)SE3 (universal on the X's, ipservices on the 3750v2).  I want to upgrade to SE8 so that's 12.2(55)SE8 as I see in the release notes there are some vulnerabilities fixed in SSH.

For the life of me, I cannot copy the new file to flash1: which is the 24 port 3750X PoE.  This switch has 20 IP phones and 2 Cisco Access points on it and is vital to the branch office.  I have spare non poe switches but not spare poe switches (looking for approval), so go figure I am having the issue with the flash memory on this switch.

If I try to tftp the file c3750e-universalk9-mz.122-55.SE8.bin to flash1, I get Error opening flash1:/c3750e-universalk9-mz.122-55.SE8.bin (Bad file number).

I can get the file tftp'd to flash2 (the 48 port 3750X) fine, but if I try to copy it from flash2 to flash1, I get the same error (bad file number).

Cisco TAC reccomened format flash1: , so I tried that and get Error formatting flash1 (Device or resource busy).   OK so on my own I figure maybe its corrupt, lets run fsck /test flash1: and see.  NOPE can't run that either... Error fscking flash1: (Device or resource busy).   Ok now lets see if any other sessions are hung where something or someone has locked flash1... so I run systat and I am the only user on the switch stack.  Ok so no one else has a hung connection that may have had a write in process frozen.

So what should I do?  The uptime is 30 weeks, 23 hours, 3 minutes.  The last switch in the stack, the regular 3750v2 has an uptime of 15 weeks, 1 day, 20 hours, 28 minutes because it was added to the stack back then. 

Anyway the network is fine and functioning... just looking to standardize all our 3750's to the 122-55.SE8 release because it is the one IOS that we've had the best luck with.

[keithsauer507]

Cisco TAC responded an hour and 20 minutes later (glad this isn't a pressing issue!) with this:

Thank you for let me know that,

• •1-      Try to reseat the flash
• •2-      Try to copy the file

If the issue still persist then try to format the flash after reseating it

Looking forward to hear from you,

Regards,

---- 

How do you reseat the flash in a 3750X 1U switch?  I mean its not a CF card like our routers.  Does this literally mean, unrack it, take it apart and theres some slot or something inside?  I didn't think this was user accessable.

[Leo Laohoo]

Try to reseat the flash

Dumbest advice I've ever heard. 

Post the output to the command "sh switch detail".  I've got a strange feeling your stack is renumbered wrong.

[keithsauer507]

Ok here is the output of sh switch detail

Switch/Stack Mac Address : 649e.f394.ee00

                                           H/W   Current

Switch# Role   Mac Address     Priority Version State

----------------------------------------------------------

1       Member 649e.f3f1.b600     1     1       Ready

*2       Master 649e.f394.ee00     1     1       Ready

3       Member 0027.0ddb.ab00     1     0       Ready

         Stack Port Status             Neighbors

Switch# Port 1     Port 2         Port 1   Port 2

--------------------------------------------------------

1       Ok         Ok               3       2

2       Ok         Ok               1       3

3       Ok         Ok               2       1

[Leo Laohoo]

What command exactly did you use to upgrade the IOS?

[keithsauer507]

copy tftp: flash:

address entered: 10.40.1.7

filename entered: c3750e-universalk9-mz.122-55.SE8.bin

%Error opening flash1:/c3750e-universalk9-mz.122-55.SE8.bin (Bad file number)

Ok lets put it on flash2 instead and then try copying it from flash2 to flash1...

copy tftp: flash2:

address entered: 10.40.1.7

filename entered: c3750e-universalk9-mz.122-55.SE8.bin

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

sucess

Ok its on flash2, so lets copy it

copy flash2:c3750e-universalk9-mz.122-55.SE8.bin flash1:

filename? [c3750e-universalk9-mz.122-55.SE8.bin]

%Error opening flash1:/c3750e-universalk9-mz.122-55.SE8.bin (Bad file number)

I am just trying to get the bin on them because I do not do archive-sw and use a tar becasue the third switch is a different bin file (3750v2 instead of a 3750X).

So to recap

All three switches are on 12.2(55)SE3

12.2(55)SE8 is on flash2 but I did not alter it's boot variable yet.

The third switch I always do last, since its a 3750v2 it doesn't have enough space to contain two IOS versions, so once I can be sure that the new version and boot variable is set on switch 1 and 2, I delete the old IOS on switch 3 and then tftp the new one on and subsequently change it's boot variable. 

Finally I would do a wr mem and reload.

[Leo Laohoo]

copy flash2:c3750e-universalk9-mz.122-55.SE8.bin flash1:
filename? [c3750e-universalk9-mz.122-55.SE8.bin] 
%Error opening flash1:/c3750e-universalk9-mz.122-55.SE8.bin (Bad file number)

Ahhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh ...

Now I know what error message you are talking about.  

Keith, you are doing this wrong.   Go to Cisco and download the TAR file.  Once you've downloaded the TAR file make sure you compare the MD5 hash value of the file you've just downloaded vs the value found in the Cisco website.

Now to unpack the TAR file from your TFTP server to your stack, the command is "archive download-sw /dest 1 /dest 2 /dest 3 tftp://10.40.1.7/filename.tar".

The third switch I always do last, since its a 3750v2 it doesn't have enough space to contain two IOS versions, so once I can be sure that the new version and boot variable is set on switch 1 and 2, I delete the old IOS on switch 3 and then tftp the new one on and subsequently change it's boot variable.

Not possible.  Once you reboot 1 & 2, switch 3, with a different IOS will not be able to join the stack.

[keithsauer507]

Ok, I've always done it this way if the stack contains different model switches.

Does the tar file contain the "universalk9" bin file for the 3750X's in the stack and then also the "ipservicesk9" bin file for any 3750v2's in a stack? The bin for the X series is usually 2mb larger if I recall correctly.

By doing the last switch last I mean the 3rd switch being that it is a 3750v2 doesn't have enough free space to hold the current IOS and the new IOS at the same time. Since I have to delete the currently running IOS, I save those sweaty moments for last.

So for stacks of the same model, yes I've used the tar file before and it's great. But for stacks of different generations of 3750's, I've always individually copied the IOS BIN file to each switch then specify their boot variables and finally reload the stack. This way the stack comes back up fine and there's no mismatch and if doing this remotely, you can still contact all switches (ie not lose one and have to drive to the site with a console cable).

Sent from Cisco Technical Support iPhone App

[Leo Laohoo]

Ok, we'll have to do a two-step process:

1.  Download the universal and IP services TAR file from Cisco;

2.  Unpack the universal TAR file to switch 1 and switch 2 using the command "archive download-sw /dest 1 /dest 2 tftp:///filename.tar".

3.  Unpack IP Services TAR file to switch 3 using the command "archive download-sw /dest 3 tftp:///filename.tar".

[Leo Laohoo]

filename entered: c3750e-universalk9-mz.122-55.SE8.bin
%Error opening flash1:/c3750e-universalk9-mz.122-55.SE8.bin (Bad file number)

Hmmmm ... Re-reading your post.

What is switch 1?  Is it a 3750X?

Can you post the output to the command "dir flash1:"?

[keithsauer507]

Switch 1 is a 3750X 24 port PoE switch.

TAC suggested format flash. Here is that output as well as a listing of flash1...


ExSW1#format flash1:
Format operation may take a while. Continue? [confirm]
Format operation will destroy all data in "flash1:". Continue? [confirm]
%Error formatting flash1 (Device or resource busy)
ExSW1#sh flash1:

Directory of flash1:/

2 -rwx 12286 Nov 20 2013 08:02:21 -05:00 private-config.text
3 -rwx 18748 Feb 28 1993 19:01:48 -05:00 config.text.backup
4 -rwx 676 Aug 6 2013 14:08:29 -04:00 vlan.dat
5 -rwx 25554 Nov 20 2013 08:02:21 -05:00 config.text
6 -rwx 6812 Feb 28 1993 19:01:48 -05:00 private-config.text.backup
455 -rwx 12312 Jun 25 2013 11:41:01 -04:00 multiple-fs
8 drwx 512 Feb 28 1993 19:08:50 -05:00 c3750e-universalk9-mz.122-55.SE3

57671680 bytes total (23278592 bytes free)


Sent from Cisco Technical Support iPhone App

[Leo Laohoo]

%Error formatting flash1 (Device or resource busy)

I think this is an IOS bug.

You may need to reboot switch 1.

[keithsauer507]

Just switch 1?

I guess my concern is what if the flash IS corrupt and on reboot it hangs because it can't read the IOS BIN file out of memory.

Anyway we just purchased a spare 3750X POE switch, so being how vital this one is, I think I may have to wait until the spare arrives just in case.

Sent from Cisco Technical Support iPhone App

[Leo Laohoo]

Just switch 1?

Yes, because you've demonstrated that switch 2 and 3 have no problem copying the IOS into.

I guess my concern is what if the flash IS corrupt and on reboot it hangs because it can't read the IOS BIN file out of memory.

No need to worry.  You can boot an IOS on a 3750X via the USB port.

[keithsauer507]

Well after a few days of SILENCE from Cisco TAC, they got back to me again and said to reseat the flash again!  I said tell me how to reseat the flash when it is soldered onto the motherboard of a 3750X switch?

So they responded with this and they are going to replace the switch:

The error message( Error opening flash1:/c3750e-universalk9-mz.122-55.SE8.bin (Bad file number)) indicates that the flash is not visible,

This issue can be solved by the following steps :

Reseat the flash if you cannot,

Format the flash if you cannot,

Replace the device(hardware failure)

As I can see from this case that the device is unable to format the flash, the only way to solve the issue is to replace the switch 1,

I can tell you support has really gone downhilll.  So much that I am considering Extreme Networks, Arista and Broacade for an upcoming 10gbps switching project.  Another case we had with them was due to a random switch stack crash on 12.2.58SE2.  Well they had the entire show tech ouput and they still reccomended to move to 15.0.2SE4.  Well that was BAD advice.  When you have 6 3750V2's and 3 3750X's in one stack of 9, release 15 doesn't have enough memory to even run a console.  We had to break the stack and one by one go to 12.2.55SE8.  I don't know what Cisco is thinking.  How can they say oh run this release but then they see its a huge mixed breed stack with a large configuration and they don't take memory into consideration?  I even asked are you sure, and they said yes we assure you this release will solve your problems.

So between that bad advice and this bad advice, I am really discouraged at throwing more money into "SmartNet" contracts.