IE-3300 not loading startup-config from SD-flash

[andreas.dahlberg]

We have a number of IE2k, IE4k and IE3x00 switches installed on our production sites. 

One of the benefits are easy replacement of faulty switches by operations personnel, without networking knowledge, by moving SD-flash card to the new switch. 

During a panic upgrade from 17.3.x to 17.12.x on a handful of our IE-3300 switches (due to a PROFINET/MRP-Ring bug), all of the switches got stuck in a boot loop. They loaded 17.12, upgraded bootloader and then restarted but could not load 17.12 and started loading 17.3 image instead, causing a bootloader downgrade, rinse and repeat.

When we did a hard reset, they booted up the new IOS image, but could not find any startup config. This triggered a restore from SD-flash, but it never recovered and instead froze up in this state:

Autoinstall: No start-up config. Triggering swap drive
Bootable image ie3x00-universalk9.17.12.04.SPA.bin and config found in sdflash:
Restore in progress...

Building configuration...
[OK]
Setting boot variable - Install

After another hard reset, the switch booted up, but with factory default startup config.

After some research, digging into the release notes, we looked at this :

I did some troubleshooting on two new switches, swapping SD-cards, but could not get any of the switches to boot from the config on the SD-card. In version 17.12.x there isn't even an option to sync from SD-flash to onboard flash which there is on at least 17.9.x.

 Is there no more easy replacement just swapping SD-card on this platform? Or, which i hope, i've done some configuration error. Anyone with ideas? Experienced the same?

[Leo Laohoo]

---

@andreas.dahlberg wrote:
upgrade from 17.3.x to 17.12.x 

---

IOS-XE router do not support direct upgrade, from 17.3.X (and earlier), to 17.12.X and requires an "intermediate" firmware upgrade to 17.9.X before proceeding to 17.12.X.  This fact, however, is undocumented.  

Please try and upgrade the IE33XX first to 17.9.X.

[Albert Mitchell]

is the IE3300 booting up in "install mode" or "bundle mode"? 
the note in the release notes is specific to "install mode".  

can you try to boot up in 'bundle mode' and see if the issue persists?

[Albert Mitchell]

for IE3200,IE3300, & IE3400, the behavior for SDFlash:, Flash: which is primary has changed.  as you've seen.

starting with IOS-XE release 17.10.1 on IE switches, the Flash: file system is now primary always regardless if a SD Card is present or not.  Starting with IOS-XE release 17.10.1, the 'sync' command only works in one direction from Flash: to SDFlash:   since, there is no reason to sync from SDFlash: to Flash: since the SDFlash: is not the primary file system.

the Day-N device replacment with SD Card still works (assuming no other issues with booting).  'Sync'  the startup config from Flash: to SDFlash:.  then for an IE3x00 in factory defaults (eg: right out of the box), when it boots it will not find startup config on Flash:, and will search teh SDFlash: for a startup config (nvram_config).  it will copy that file to Flash:, and reload.

thus the new device boots with config of older device being replaced.

[JTownshed]

Is there a reason why this command change has not been reflected in the release notes or SD card configuration document?

[Adam Black]

For me the above mentioned steps worked, I just had to wait for a few minist after the "Setting boot variable - Install" message.

[Krystal Dephrene]

Here I am in 2025 trying to figure out how to undo this as the SDFlash being primary is essential in our environment. I absolutely cannot have this booting configs from internal flash. It has to be from SDFlash and there is no way around that.

Who can I talk to as this is going to be a problem on 330+ devices?

[Leo Laohoo]

Raise a TAC Case (because nothing will get traction without a TAC Case), add the Cisco Accounts Team into the list and, depending on how big the account it, demand for a Product Enhancement Request be raised. 

Again, depending on how big this account is, it may take several years (if not a decade) for this feature to be even considered.  If the account is specially large, maybe a year (or two).  

Good luck!

[msedghi]

@Krystal Dephrene @Leo Laohoo 
Please follow the steps:

Changes of Swap Drive procedures after 17.10
Starting with IOS-XE release 17.10.1 on IE switches, configuration file is always loaded from flash regardless of the boot media or the presence of SD card, EXCEPT a configuration file is not found in flash.
When a switch cannot locate a configuration file from flash, it tries to look up one from sdflash. If it finds one, it copies both the configuration and IOSXE image into flash. The switch will then boot from flash. 
Therefore, to maintain the same behavior of swap drive behavior pre-17.10, that a customer can simply put a SDflash into a newly open switch (or simply make sure the switch has no config on it) and then an IOSXE image and existing config could be booted from the new switch, follow the procedures as below -
1. Sync config and image from an existing switch -
If the switch is loaded with images post-17.10, do a "sync sdflash:".
As mentioned above, since the copying of config and image files has become an automatic process, there's no manual process required for synchronizing sdflash to flash, therefore after 17.10, only the "sync sdflash:" command is available.
If it's pre-17.10, do a "sync flash: sdflash:"
2a. Swap drive to a brand new switch -
Put the prepared sdflash from step 1 into a newly open switch. Since there's no configuration in the flash of the new switch, the new switch tries to lookup one from sdflash.
2b) Swap drive to a redeployed switch -
In this scenario, instead of brand new switch, customer has a used switch to be redeployed.
To make sure there's no configuration in flash, make a "wr erase". Also clear boot variables by "no boot system".
Also make sure there's enough spaces in flash. When the automatic sync happen and if the switch does not find enough spaces to copy the image, the process fail with "ERROR: Software Installation Failed: 35.
If the switch is loaded with images post-17.10,
Put the prepared sdflash from step 1 into this switch and reload. Since there's no configuration in the flash, the switch tries to lookup one from sdflash.
If the switch is loaded with images pre-17.10,
Put the prepared sdflash from step 1 into this switch, do a "sync sdflash: flash:" and reload

After the above procedures are done, the switch copies both the configuration and IOSXE image into flash. The switch will then boot from flash.

Please let me know if you need help on that

[Albert Mitchell]

the "configuring SD Swap Drive" Chapter in the IE3x00 System Management book is currently under construction to update it with current IOS-XE behavior. 

[tthornbrue]

I was just misled by the IE3x00 System Management chapter on SD Swap Drive. Thankfully, I found this forum explaining what's going on. Can you provide an update on when accurate documentation will be released?

[Albert Mitchell]

the chapter on SD Swap Drive is currently being revised and is in the review process.

[lacemi]

I´m in the same case with IE3100 and 3300. I have 17.12.4 ios XE because this version dont have high vulnerabilities and recommended by CISCO TAM TEAM. I have tested the following scenario. I synchronized from the flash to the SDFlash on a working switch IE3100 by executing a "SYNC SDFLASH:". This is the only allowed command. It does not allow synchronization in the direction from sdflash to the flash as a " IE3x00 System Management chapter on SD Swap Drive" said.
Then, I took the sdcard, inserted it into a new IE3100, and powered it on. As exepected, it booted from the internal flash, meaning it had nothing configuration from the replaced switch, in others words, nothing from the old sw. So I entered in ROMMON mode and changed the BOOT var to point to sdcard:packages.conf , after rebooting, during statup process, it found sdflash:packages.conf but next point in the process it show a terrible mensaje : "Autoinstall; No start-up config. Triggering swap drive Bootable image packages.conf and config found in sdflash: Restore in progress and then Building configuration an set variable loading from de flash:packages.conf 
To make matters worse, It seens that ROMMON is automatically modified, and it resets the BOOT variable back to FLASH: because is one on the change on the start proccess.
Therefore, I had to enter ROMMON mode again and change it back, I set BOOT to SDFLASH:packages.conf and as a test I also ADDED a new variable, CONFIG_FILE=nvram_config, in this CASE ROMMON doesn´t change and continues pointing to SDFLASH. The autoinstall issue doesn´t occur but Device dont have the old configuration.
Then I conduced a new test, I made some changes to the device configuration running-config, executed a write and restarted it. It loaded the PKGs from the SDFLASH, and the configuration changes I had made were still there
So, as if it were a new device, I conducted another test. I DELETED the STARTUP-CONFIG from NVRAM to check whether, upon rebooting, the sw would automatically create the startup-config on the sdflash, as stated in CISCO official doc.
Well, it didn´t work, It couldn´t find the startup-config for all sites but It triggered the autoinstall process again, and loading .pkg the flash: again after read sdflash, and erased the ROMMON VARIABLES SDFLASH: AGAIN and put again in BOOT var = flash:packages.conf and CONFIG_FILE=(empty once again).
At this point, I decided to conduct a new test. While in enable mode, I format the sd card. I write a new changed in running-config as for example the hostname and write.  Then, I performed a sync to the SDFLASH so that everything from FLASH would be sync to the SD. I manually copied the startup-config not the nvram_config, since that one is already created when running the sync command on the SDFLASH, instead, I specifically copied startup-config from NVRAM to SDFLASH. I then restarted the sw entering ROMMON mode. I reconfigured the BOOT and CONFIG_FILE variables. To summarize the test I performed, I also dis another version of the test without and delete the CONFIG_FILE variable in other wors without setting CONFIG_FILE. 
Aftesr restarting the device again, this time it found the startup-config on the SDFLAS, thanks os this, IT DID NOT MODIFY THE ROMMON VARIABLES, NOR DID IT TRIGERRED AUTOINSTALL MESSAGE, the SW BOOTED NORMALY and KEEP the ROMMON variables in "BOOT=sdflash:packages.conf", it dint erase them BUT it booted with a BASIC CONFIGURATION, not with the canges I had previusly made , my config is missing
I conducted more test and more, but I dont want to bore you. In summary, no matter what you do, it does´nt work. Finally a complete waste of time, considering all the effort involved in configuring, setting password, reboots, wizars, waiting times, etc. So if a sw fail and you replace it with a new one, inserting the SD card that contains all the necessary information will never work.
NEVER-even if you have previously configured the replacement switch, becasue it will never load the configuration in 17.12.4
I dont know what solution this could have. Maybe switching to a different IOS version, or we´ll see what the community expert has to say about it.

[tthornbrue]

Hi lacemi,

Thanks for sharing your detailed analysis. I also experimented with a swapped SD card on a defaulted switch at an older firmware revision trying several variations, effectively following a process of elimination. What finally worked was having boot sys point to flash:<filename.bin> (bundle mode), then write mem AND sync to the sdflash. I then tried that SD card in the defaulted switch which worked.

• The fact that the switch was now quietly saving my config changes to flash instead of sdflash had me all confused. Now a sync command is always required to update boot system variable in the sdflash.
• I also confirmed the sdflash is up first in the boot order as described in the questionable swap drive documentation
• I'd read a caution note in this same documentation that using install mode (packages.conf) will NOT work for swap drive.
• System Management Configuration Guide, Cisco Catalyst IE3x00 and IE3100 Rugged, IE3400 Heavy Duty, and ESS3300 Series Switches - Configuring SD Swap Drive [Cisco Catalyst IE3200 Rugged Series] - Cisco

Results

• With no matching .bin firmware found on the “new” switch, it consults the SD card’s packages.conf (ironic since that’s install mode) to find where the .bin file is on sdflash, then boots with it.
• Then with no running config found on flash:, the switch triggers "swap drive" and looks to the sdflash for a config file.
• After this, the defaulted switch is alive with matching firmware and config from the SD card.

If this approach also works for you, please share. 

My next concern is whether to automate the sync command with every write mem. If not, the SD's backup purpose becomes outdated with all changes always being written to flash now, no matter what the switch booted up with.

[lacemi]

"My next concern is whether to automate the sync command with every write mem. If not, the SD's backup purpose becomes outdated with all changes always being written to flash now, no matter what the switch booted up with."

You can create a kron occurence backup at (X time) recurring with a policy-list and include in kron policy-list the cli sync sdflash: command.