4431 Router Booted into SD WAN Mode When I Updated the Code

[benweber]

Has anyone else seen this?  I've been upgrading a bunch of ISR routers for a client from a 16 code to 17.3.5.  Pretty standard: upload the code to flash, verify it, change the boot variable, and set it to reboot over night.  Done it a thousand times.

This one particular 4431 just didn't come back the next morning.  (Thank God for backup VPN circuits -no one but IT support knew it was down.)  I assumed it had just hung during the reboot, which happens sometimes.  But when I got on site it was asking for a username and password, but wouldn't take the programmed ones.  I figured out that it was in controller/SD-WAN mode.  I was able to get it back into normal mode but had to then restore from a backup of the config as the conversion to SD WAN mode had deleted it.  I've since rebooted the router to verify that it's now staying in normal (non-SD WAN) mode.

Has anyone seen this?  Is there some command I can enter to make sure the next router I upgrade knows not to do this?

Thanks,

Ben

[benweber]

Quick correction.  I just checked my notes.  I was only going from 17.3.3 to 17.3.5.  So not a major upgrade.

[Leo Laohoo]

Post the complete output to the command "sh boot" and "sh romvar".

[Richard Burts]

Ben

Could you provide the image name of the image you are currently running and the image name that you used in the problematic upgrade? My guess is that in the failed upgrade that you used an image intended for SD-WAN and not the usual router image.

[benweber]

It's the isr4400-universalk9.17.03.05.SPA.bin code.

I had that same thought.  But I went and looked at the naming convention of the SD WAN codes and they are sufficiently different that there's no way I wouldn't have noticed, especially since I had run a verification on the code the night before the reboot.

This is the boot variable that was set before the reboot:

boot system flash bootflash:/isr4400-universalk9.17.03.05.SPA.bin

I always save my before/after configs and that's from the final config file for that router the afternoon before it rebooted.

[Leo Laohoo]

---

@benweber wrote:

DEVICE_MANAGED_MODE = autonomous

---

That is odd.  This output is telling me the router should be booting into IOS-XE (non-SD-WAN) mode.

[benweber]

I confirmed through a colleague that the newer ISR codes like this one have the SD WAN code embeded in them so it's possible to boot into controller mode with this software version.

[benweber]

Sure:

<router>#sho boot
BOOT variable = bootflash:/isr4400-universalk9.17.03.05.SPA.bin,1;
CONFIG_FILE variable does not exist
BOOTLDR variable does not exist
Configuration register is 0x2102

Standby not ready to show bootvar

<router>#sho romvar
ROMMON variables:
PS1 = rommon ! >
ETHER_SPEED_MODE = 4
THRPUT =
LICENSE_SUITE =
MCP_STARTUP_TRACEFLAGS = 00000000:00000000
LICENSE_BOOT_LEVEL = securityk9,securityk9:esg;
DEVICE_MANAGED_MODE = autonomous
BOOT = bootflash:/isr4400-universalk9.17.03.05.SPA.bin,1;
RET_2_RTS = 12:32:42 EST Fri Mar 31 2023
BSI = 0
RET_2_RCALTS =
RANDOM_NUM = 474422917

[CyrusKain]

OMG, this JUST happened to me (I know it's over a year later).  Absolutely routine 4451-X upgrade (although I went from an older code to a recent code, 16.12.5 to 17.09.05) and the router didn't recover, and its redundant neighbor showed the name Router in CDP off the connecting interface.  Went out on site, and it had done the same thing:  wanted username/PW for SD-WAN mode.  Wiped out my config and everything (fortunately I brought recent running-configs with me in a notepad so I recovered quickly) but as I'm not SD-WAN savvy, I had no clue what was going on.  All my attempts at password recovery using confreg in ROMMON failed - nothing would bypass.  Why in the world would would this be allowed to happen?  I'm definitely open ears for anyone who can better explain the 'how' and 'why' this happened(s)..