Firmware Image Recovery -- Upload via Xmodem ROMmon Console Serial COM port on old devices (for instance 3750v2) after "erase flash:" or file corruption

[Xerg]

Hi everyone!

I did play hard with my c 3750 v2 and have decided to put myself in a trial. The reason: I want to be prepared for unpleasant surprises that may occur in the future for any reason. I've done a similar recovery on AIR-CAP2702I-Z-K9 but it is a newer device with TFTP available in ROMMON. That's why I decided to have a test on the old and least valuable asset in my lab.

The challenge was initiated with a couple of strings:

Switch# erase flash:

Switch# reload

As expected, the switch was unable to boot up as there was no file on the flash. Instead, ROMmon mode greeted me with the following screen and strange-looking prompt:

The system has been interrupted prior to initializing the flash file system. The following

commands will initialize the flash file system and finish loading the operating system

software#

flash_init

load_helper

boot 

switch: 

So, I started my battle with the "problem" from another couple of obvious strings:

switch: flash_init

switch: load_helper

Ops... load_helper doesn't work on this device, and it didn't appear in the menu upon boot, even it mentioned in the 3750v2 CLI manual. Actually, that's not a problem. Apparently, the service was initiated already.

As I wasn't going to wait for file transfer for ages I proceeded with the following string:

switch: set BAUD 115200

This parameter going to survive during reboot, so I had to set Console (Putty) and Hyper terminal (HyperTerminal Trial) accordingly. Of course, I had to set the BAUD parameter to default 9600 baud before exiting ROMmon.

Next, I was needed for an app in order to upload the firmware image.

I couldn't fire standard HyperTerminal on my Win10 even I copied its .exe and .dll in the suitable locations. Putty tools appeared to be a bit complicated to get them working. Unfortunately, I dunno that "appropriate command on the terminal-emulation software" as the 3750 manual says, shame on me. I ended up with the utility of the same name as HyperTerminal (google hilgraeve HyperTerminal-trial). Trial version good enough for 30 days and lets to type commands in the terminal window. I did install the app and got a new connection to the COM port prepared with matching parameters: Speed = 115200 baud and Flow control = Xon/Xoff. The Putty session was terminated and HyperTerminal was initiated instead. The next string was sent:

switch: copy xmodem: flash:FIRMWARE-IMAGE-NAME.bin

Begin the Xmodem or Xmodem-1K transfer now...

CCCCCCC.

The "C" letters begun to emerge on the screen one by one letting me know that switch is waiting for firmware transmission. The timeout was long enough to locate and send a firmware file via HyperTerminal > Transfer > Send. For some reason, the real transmission speed was 41200 bps and the uploading process took about 52 minutes for a file of 13 MB in size, and ended up with a message:

File "xmodem:" successfully copied to "flash:FIRMWARE-IMAGE-NAME.bin"

Let's check:

switch: dir flash:

Directory of flash:/

2 -rwx 13013248 <date> c3750-ipservicesk9-mz.122-55.SE12.bin

2984448 bytes available (13014528 bytes used)

Everything as expected. Time to add a configuration file from backup (config.text file mapped to startup-config file on virtual NVRAM on a variety of old devices):

switch: copy xmodem: flash:config.text

Begin the Xmodem or Xmodem-1K transfer now...

CCCCCCC.

File "xmodem:" successfully copied to "flash:config.text"

Another dir flash: to ensure that things are in place.

Set BAUD-rate to default, (hence it would be possible to change the rate in Global configuration mode but wouldn't be possible to save settings, therefor upon next reboot rate gonna be 115200 again):

switch: set BAUD 9600

Then change the BAUD-rate of the terminal and boot the device with the correct image:

switch: boot flash:FIRMWARE-IMAGE-NAME.bin

As a result, I've got the system loaded and entered into Global configuration mode to finalize the settings. A new firmware image was uploaded in place of obsolete one. Moreover, the old one was deployed from a .tar archive but the new was simple .bin. It was necessary to use .bin as ROMmon mode doesn't let to deploy archives. So, there are few things to do: to deploy .tar file via TFTP, USB, etc, and/or set a correct path to the image to boot with. I was OK with the current .bin and decide to proceed with the last two options:

Switch (config)# boot system flash:FIRMWARE-IMAGE-NAME.bin

Checked it with:

Switch(config)#do show boot

BOOT path-list : flash:FIRMWARE-IMAGE-NAME.bin

Config file : flash:/config.text

Private Config file : flash:/private-config.text

Enable Break : no

Manual Boot : no

HELPER path-list :

Auto upgrade : yes

Auto upgrade path :

NVRAM/Config file

buffer size: 524288

Timeout for Config

Download: 0 seconds

Config Download

via DHCP: disabled (next boot: disabled)

Finally :

Switch# write

And

Switch# reload

Done!

----------------------------------

For recap:

Prerequisites:

1) .bin image suitable for your device flash size (actually, the smaller size the faster device will get it. Especially recommended for those who going to deploy .tar image later via TFTP. If you have the only tar. image simply extract the bin. image from it. I use 7-zip, for instance.)

2) HyperTerminal or similar soft able to upload files via Xmodem protocol. Should be installed on a machine that is connected to the device console port.

3) Appropriate console cable.

Commands:

switch: flash_init

switch: set BAUD 115200

Change your terminal speed to 115200 as well.

switch: copy xmodem: flash:FIRMWARE-IMAGE-NAME.bin

Next, you have to start .bin image file transfer from HyperTerminal or so. It will take a while. About an hour, probably.

switch: dir flash:

switch: copy xmodem: flash:config.text     ### If you have configuration backup

switch: dir flash:

switch: set BAUD 9600

Change your terminal speed to 9600 as well.

switch: boot flash:FIRMWARE-IMAGE-NAME.bin

After you've got in Global configuration mode issue this (or first, deploy a desirable image.tar over TFTP):

Switch (config)# boot system Flash:FIRMWARE-IMAGE-NAME.bin

Switch (config)# exit

Switch# show boot

Switch# write

Switch# reload

The whole process took a few hours as this experiment was spontaneous. I’ve spent time mostly on reading some not comprehensive (with corrections in replies) or simply irrelevant material on the internet and here in CC and checking tonnes of obsolete not working links, vague comments regarding Xmodem in the manuals (only one had a clear description but it was one of the last links I came across), and testing/choosing suitable working Xmodem terminal. Thus, I’ve decided to share my experience with those who have to deal with some old equipment from time to time.

All I had were console cable, Putty, and .bin image. Prior experience was based on recovering AP via TFTP in ROMMON.

HTH

Serge

[Joseph W. Doherty]

A couple of things, possibly worth knowing.

At least on the older 3750s, Cisco provided a ROMMON update that allowed booting from a USB stick.  Don't recall if you needed this update to also just copy a file, like your .BIN, from it to flash.  Possibly the 3750v2 might already have a ROMMON that supports USB.  (NB: Cisco's 3750s, at least, are very particular about what USB they will work with other than Cisco branded USBs.)

You didn't mention whether your terminal software supported Xmodem-1K, and if so, whether you selected that.  If not, the 1K variant does help to improve overall transfer rate.

In a cases like this, where you need to load via the console port, if you do have a network connection and TFTP server, and assuming the ROMMON doesn't support loading via TFTP, loading the smallest IOS image, like an older IPBase w/o any extra features, will reduce the time to get that IOS on the device.  Once such is loaded, you can boot it up and use its IOS features, TFTP, FTP, etc., to load a larger and desired IOS image.

[Xerg]

Joseph,

The c3750v2 is really old stuff with no USB ports. I wouldn't moke around if the USB port is there. No TFTP in ROMMON and only Console connection available. The article is dedicated to the recovery process of this particular kind of device. There are several old switches of that sort that still might be used as part of out-of-band management infrastructure as it is in my home lab.

Unfortunately, there is no Xmodem-1K option in the software used to console into the device. However, it's worth to be mentioned.

Regarding the FW file size, I wrote in the Prerequisites section.

Cheers

[Joseph W. Doherty]

It's certainly possible the 3750V2 series don't have USB ports.  That's why I wrote I didn't know whether the later v2s had them or not, but I do recall (?) some of the earlier 3750 and 3750G (i.e. "the older 3750s - both predate the v2s)" did have them. BTW, those older 3750s, might still be found in service, or perhaps in a "lab".

Further, I also recall most of those older 3750s didn't support the USB stick, at least for booting, without a firmware upgrade.  So, even if the device has one, it might not be directly usable.  The real point though, if you luck out, you might save yourself hours transferring an IOS via the console port.

I also recall (?), at the time the ROMMOM update was released, considering sending USB sticks, to remote branches that had low bandwidth (like 128 Kbps) WAN links and/or using them if the site reported the site's 3750, for some reason, wouldn't boot its flash IOS.

Ah, well if your terminal software doesn't support the Xmodem-1K option, it can be worthwhile to find terminal software that does.  It can speed up the transfer.  (Of course, the later Ymodem and Zmodem protocols are generally even better, but I don't recall Cisco supporting them.)

Regarding the FW size, yes you mentioned using the .BIN not the .TAR, including loading the .TAR later via TFTP, and also mentioned smaller is better.  This, of course, as you also mentioned, reduces the console cable's transfer time.  However, although many might use a smaller IOS, based on feature set (which I believe you too are recommending, although not explicitly described), that might not have been clear to all, further many might overlook dropping down to older, even much older, and smaller, IOS version.  I thought it worth expanding on this point.

Once upon a time, when the 3750 was still a recent device (the original series, even before the 3750G), I deleted one's current IOS to provide enough flash space for it's replacement.  Don't recall how I did it, but I may ended up with a 3750 that needed to be loaded via console cable.  Multi-hour process.  (At the time, I think my PC's serial port only supported up to 56 Kbps, which why the data transfer was multi-hour.)

One issue I ran into, which you covered, which I didn't comment on, was resetting the console port back to 9600.  In my case, I only tried that after successfully booting into the replaced IOS, but had a heck of a time getting it to reset to 9600 for actual booting.  It wanted to stay at the higher baud rate, which was okay if you knew that, but not okay if you assumed it was 9600.