IE-3300 not loading startup-config from SD-flash

[andreas.dahlberg]

We have a number of IE2k, IE4k and IE3x00 switches installed on our production sites. 

One of the benefits are easy replacement of faulty switches by operations personnel, without networking knowledge, by moving SD-flash card to the new switch. 

During a panic upgrade from 17.3.x to 17.12.x on a handful of our IE-3300 switches (due to a PROFINET/MRP-Ring bug), all of the switches got stuck in a boot loop. They loaded 17.12, upgraded bootloader and then restarted but could not load 17.12 and started loading 17.3 image instead, causing a bootloader downgrade, rinse and repeat.

When we did a hard reset, they booted up the new IOS image, but could not find any startup config. This triggered a restore from SD-flash, but it never recovered and instead froze up in this state:

Autoinstall: No start-up config. Triggering swap drive
Bootable image ie3x00-universalk9.17.12.04.SPA.bin and config found in sdflash:
Restore in progress...

Building configuration...
[OK]
Setting boot variable - Install

After another hard reset, the switch booted up, but with factory default startup config.

After some research, digging into the release notes, we looked at this :

I did some troubleshooting on two new switches, swapping SD-cards, but could not get any of the switches to boot from the config on the SD-card. In version 17.12.x there isn't even an option to sync from SD-flash to onboard flash which there is on at least 17.9.x.

 Is there no more easy replacement just swapping SD-card on this platform? Or, which i hope, i've done some configuration error. Anyone with ideas? Experienced the same?

[Leo Laohoo]

---

@andreas.dahlberg wrote:
upgrade from 17.3.x to 17.12.x 

---

IOS-XE router do not support direct upgrade, from 17.3.X (and earlier), to 17.12.X and requires an "intermediate" firmware upgrade to 17.9.X before proceeding to 17.12.X.  This fact, however, is undocumented.  

Please try and upgrade the IE33XX first to 17.9.X.

[Albert Mitchell]

is the IE3300 booting up in "install mode" or "bundle mode"? 
the note in the release notes is specific to "install mode".  

can you try to boot up in 'bundle mode' and see if the issue persists?

[Albert Mitchell]

for IE3200,IE3300, & IE3400, the behavior for SDFlash:, Flash: which is primary has changed.  as you've seen.

starting with IOS-XE release 17.10.1 on IE switches, the Flash: file system is now primary always regardless if a SD Card is present or not.  Starting with IOS-XE release 17.10.1, the 'sync' command only works in one direction from Flash: to SDFlash:   since, there is no reason to sync from SDFlash: to Flash: since the SDFlash: is not the primary file system.

the Day-N device replacment with SD Card still works (assuming no other issues with booting).  'Sync'  the startup config from Flash: to SDFlash:.  then for an IE3x00 in factory defaults (eg: right out of the box), when it boots it will not find startup config on Flash:, and will search teh SDFlash: for a startup config (nvram_config).  it will copy that file to Flash:, and reload.

thus the new device boots with config of older device being replaced.

[JTownshed]

Is there a reason why this command change has not been reflected in the release notes or SD card configuration document?

[Adam Black]

For me the above mentioned steps worked, I just had to wait for a few minist after the "Setting boot variable - Install" message.

[Albert Mitchell]

the "configuring SD Swap Drive" Chapter in the IE3x00 System Management book is currently under construction to update it with current IOS-XE behavior. 

[tthornbrue]

I was just misled by the IE3x00 System Management chapter on SD Swap Drive. Thankfully, I found this forum explaining what's going on. Can you provide an update on when accurate documentation will be released?

[Albert Mitchell]

the chapter on SD Swap Drive is currently being revised and is in the review process.

[lacemi]

I´m in the same case with IE3100 and 3300. I have 17.12.4 ios XE because this version dont have high vulnerabilities and recommended by CISCO TAM TEAM. I have tested the following scenario. I synchronized from the flash to the SDFlash on a working switch IE3100 by executing a "SYNC SDFLASH:". This is the only allowed command. It does not allow synchronization in the direction from sdflash to the flash as a " IE3x00 System Management chapter on SD Swap Drive" said.
Then, I took the sdcard, inserted it into a new IE3100, and powered it on. As exepected, it booted from the internal flash, meaning it had nothing configuration from the replaced switch, in others words, nothing from the old sw. So I entered in ROMMON mode and changed the BOOT var to point to sdcard:packages.conf , after rebooting, during statup process, it found sdflash:packages.conf but next point in the process it show a terrible mensaje : "Autoinstall; No start-up config. Triggering swap drive Bootable image packages.conf and config found in sdflash: Restore in progress and then Building configuration an set variable loading from de flash:packages.conf 
To make matters worse, It seens that ROMMON is automatically modified, and it resets the BOOT variable back to FLASH: because is one on the change on the start proccess.
Therefore, I had to enter ROMMON mode again and change it back, I set BOOT to SDFLASH:packages.conf and as a test I also ADDED a new variable, CONFIG_FILE=nvram_config, in this CASE ROMMON doesn´t change and continues pointing to SDFLASH. The autoinstall issue doesn´t occur but Device dont have the old configuration.
Then I conduced a new test, I made some changes to the device configuration running-config, executed a write and restarted it. It loaded the PKGs from the SDFLASH, and the configuration changes I had made were still there
So, as if it were a new device, I conducted another test. I DELETED the STARTUP-CONFIG from NVRAM to check whether, upon rebooting, the sw would automatically create the startup-config on the sdflash, as stated in CISCO official doc.
Well, it didn´t work, It couldn´t find the startup-config for all sites but It triggered the autoinstall process again, and loading .pkg the flash: again after read sdflash, and erased the ROMMON VARIABLES SDFLASH: AGAIN and put again in BOOT var = flash:packages.conf and CONFIG_FILE=(empty once again).
At this point, I decided to conduct a new test. While in enable mode, I format the sd card. I write a new changed in running-config as for example the hostname and write.  Then, I performed a sync to the SDFLASH so that everything from FLASH would be sync to the SD. I manually copied the startup-config not the nvram_config, since that one is already created when running the sync command on the SDFLASH, instead, I specifically copied startup-config from NVRAM to SDFLASH. I then restarted the sw entering ROMMON mode. I reconfigured the BOOT and CONFIG_FILE variables. To summarize the test I performed, I also dis another version of the test without and delete the CONFIG_FILE variable in other wors without setting CONFIG_FILE. 
Aftesr restarting the device again, this time it found the startup-config on the SDFLAS, thanks os this, IT DID NOT MODIFY THE ROMMON VARIABLES, NOR DID IT TRIGERRED AUTOINSTALL MESSAGE, the SW BOOTED NORMALY and KEEP the ROMMON variables in "BOOT=sdflash:packages.conf", it dint erase them BUT it booted with a BASIC CONFIGURATION, not with the canges I had previusly made , my config is missing
I conducted more test and more, but I dont want to bore you. In summary, no matter what you do, it does´nt work. Finally a complete waste of time, considering all the effort involved in configuring, setting password, reboots, wizars, waiting times, etc. So if a sw fail and you replace it with a new one, inserting the SD card that contains all the necessary information will never work.
NEVER-even if you have previously configured the replacement switch, becasue it will never load the configuration in 17.12.4
I dont know what solution this could have. Maybe switching to a different IOS version, or we´ll see what the community expert has to say about it.

[tthornbrue]

Hi lacemi,

Thanks for sharing your detailed analysis. I also experimented with a swapped SD card on a defaulted switch at an older firmware revision trying several variations, effectively following a process of elimination. What finally worked was having boot sys point to flash:<filename.bin> (bundle mode), then write mem AND sync to the sdflash. I then tried that SD card in the defaulted switch which worked.

• The fact that the switch was now quietly saving my config changes to flash instead of sdflash had me all confused. Now a sync command is always required to update boot system variable in the sdflash.
• I also confirmed the sdflash is up first in the boot order as described in the questionable swap drive documentation
• I'd read a caution note in this same documentation that using install mode (packages.conf) will NOT work for swap drive.
• System Management Configuration Guide, Cisco Catalyst IE3x00 and IE3100 Rugged, IE3400 Heavy Duty, and ESS3300 Series Switches - Configuring SD Swap Drive [Cisco Catalyst IE3200 Rugged Series] - Cisco

Results

• With no matching .bin firmware found on the “new” switch, it consults the SD card’s packages.conf (ironic since that’s install mode) to find where the .bin file is on sdflash, then boots with it.
• Then with no running config found on flash:, the switch triggers "swap drive" and looks to the sdflash for a config file.
• After this, the defaulted switch is alive with matching firmware and config from the SD card.

If this approach also works for you, please share. 

My next concern is whether to automate the sync command with every write mem. If not, the SD's backup purpose becomes outdated with all changes always being written to flash now, no matter what the switch booted up with.

[lacemi]

"My next concern is whether to automate the sync command with every write mem. If not, the SD's backup purpose becomes outdated with all changes always being written to flash now, no matter what the switch booted up with."

You can create a kron occurence backup at (X time) recurring with a policy-list and include in kron policy-list the cli sync sdflash: command.

[lacemi]

Hi TThornbrue

Thanks for replay and share yours experience, but I dont use bundle mode. I´m testing with 4 different sw, and both behave the same way, both are running the recommended version 17.12.4 free of high and medium vulnerabilities. I´m not using bundle mode, you can see I used nvram_config file and packages.conf, I used install mode because I have Cisco Catalyst Center and I know is better install mode for automatizacion or wich provides much greater versatility and other task, bundle mode is more manual, therefore, I don´t consider using Bundle mode with ios.bin
As I mentioned in my previous post, modifying the Boot variable in rommon allows the switch to load packages.conf from the sdflash, however when it looks for startup-config, it always searches in flash, is it is the first time, the file dont exist and start the switch with standar config (empty). The nvram_config file on the SDflash is completely useless. It doesn´t matter if you set it in ROMMON boot, if you dont set it, or if you rely on packages.conf, it makes no difference. 
If you  perform a normal boot, inserting the SDFLASH from the old switch into the new one, it will always boot from Flash first every single time. THE SDFlash behaves as if it doesn´t exist. For the rest of the test, you can refer to my preious comment in this thread.
 

[tthornbrue]

Hi lacemi, I was also originally trying to use install mode (boot sys sdflash:packages.conf), which is what ran me into all of these difficult behaviors you're describing. With that note about swap drive only working with bundle mode, I'm forced to change if I want swap drive to work.

[lacemi]

yes tthornbrue, thanks man!!