cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
89804
Views
26
Helpful
38
Comments
xthuijs
Cisco Employee
Cisco Employee

 

Introduction

This document shows you how to reset a lost password or when you have locked yourself out due to a problematic AAA configuration.

 

Core Issue

Because IOS-XR is substantially different in the way config files are managed, the standard trick of conf-reg 0x2142 will not work for IOS-XR.

You can lock yourself out if you are configuring aaa authentication to tacacs with no local fall back, if the tacacs server is unavailable there

is no way for you to get in.

 

eg:

aaa authentication login default groupt tacacs

 

Also this procedure is good when you have forgotten the password to your super user in IOS-XR to manage your machine.

 

Resolution

 

The following step through guide can be tried, the details of each step are listed below with more explanation:

 

  • •1)       Fixing AAA configuration errors  
    • •a.       On the standby RP/RSP from the CONSOLE port hit the ESC key and type ‘ksh’ without quotes and hit ENTER  
      • i.      Login with a local username and password
      • ii.      If this fails get the standby RP/RSP into ROMMON
      • iii.      Bypass KSH authentication with AUX_AUTHEN_LEVEL=0 and boot
      • iv.      Try step 1a again or use the AUX port and go to step 1b
    • •b.       View and edit the configuration from KSH  
      • i.      Save the configuration to harddisk with ‘nvgen -c -l 1 -t 1 -o 1  > harddisk:/backupconfig.txt’
      • ii.      Edit out the bad AAA statements with ‘nano –e /harddisk:/backupconfig.txt’
    • •c.        Try to roll back the configuration with ‘config_rollback –n 0x1’
    • •d.       Bypass AAA and enter exec mode with ‘/pkg/bin/exec –a’
    • •e.        Attempt to use show commands or change the configuration 
      • i.      If this fails reload all RP/RSP ROMMON
      • ii.      On the standby card set IOX_CONFIG_FILE=/harddisk:/backupconfig.txt or use ‘boot <image> -a <bogus_config>’ and boot
      • iii.      Also follows step 2g if you saw issues in 1a
      • iv.      If nothing above worked then this is the only option
  • •2)       Fixing a lost local username/password  
    • •a.       Get the standby RP/RSP into ROMMON  
      • i.      Bypass KSH authentication with AUX_AUTHEN_LEVEL=0 and boot
    • •b.       View the admin configuration with ‘nvgen –b /admin/cfg’
    • •c.        Save the admin configuration to the harddisk and edit out any and all users if you need other portions of this file
    • •d.       Bypass AAA and enter exec mode with ‘/pkg/bin/exec –a’
    • •e.        Attempt to use show commands or change the configuration
    • •f.        If this fails reload all RP/RSP to ROMMON
    • •g.        Set confreg 0x142 or IOX_ADMIN_CONFIG_FILE=/harddisk:/backupconfig.txt on the standby card or ‘boot <image> -o <bogus_config>’ and boot  
      • i.      Note that this does not ignore the exec configuration and will not help if the issue is AAA related
    • •h.       Enter a new username and password when prompted
  • •3)       Fixing both issues 
    • •a.       If you do not know a local login or cannot use the KSH method to recover the configuration then both the IOX_CONFIG_FILE and IOX_ADMIN_CONFIG_FILE will need to be pointed towards non-existent files. Both the admin and exec configurations will be cleared by this method
  • •4)       Make sure to remove any ROMMON variables which were change
  • 5)  XR-VM Username/Password reset procesdure using Sysadmin VM

 

There are 2 steps to this process.

1) Override the BASE running configuration

     When you configure the problematic AAA statement sample as above.

2) Override the admin configuration that stores local usernames and passwords

     When you don't remember any of the local usernames/passwords you have defined locally.

Overriding the Base configuration in XR:

 

Step 1

In rommon set the following variable:

 

rommon> IOX_CONFIG_FILE=/harddisk:/no-config

 

the file no-config is just a non existent file, you can give any name here really.

 

Note: This ROMMON variable will persist and needs to be removed after password recovery. Check the 'clean up' section.

 

 

Step 2

And issue 'sync', this will make the change persistent in the rommon config vars.

 

rommon> sync

 

Step 3

Issue 'i' or 'reset' and when the rsp is booting up, it should ignore  the config file, since there's no config file found on /harddisk:  called no-config

 

rommon> reset

or

rommon> i

 

 

Overriding the ADMIN configuration in XR:

In Admin configuration we store all the local usernames and passwords.

Step 1

 

Similarly you can do the same thing for admin config:

 

IOX_ADMIN_CONFIG_FILE=/disk0:/none

 

You should get prompted for root user/pass and will have a blank config on the box.

You need to load your config and do your modification.

 

Note: This ROMMON variable will persist and needs to be removed after password recovery. Check the 'clean up' section.

Step 2 and 3

are the same as for the base xr config file.

 

Second Option

 

Another way of recoveryof the password is to enable the following again in rommon:

 

rommon> AUX_AUTHEN_LEVEL=0

 

Which will allow the aux port to drop to ksh upon the RSP bootup with no prompt for login.

 

At the prompt you can either type:

 

/pkg/bin/exec -a

 

Which will give you a router prompt: Or simply

 

# Config

 

Which drops you into EXEC config mode.

Example

# uname -a

QNX node0_RSP0_CPU0 6.4.0 2009/12/10-13:43:22PST asr9k ppcbe

# config

RP/0/RSP0/CPU0:RO-A(config)#exit

#

# /pkg/bin/exec -a

RP/0/RSP0/CPU0:RO-A#

RP/0/RSP0/CPU0:RO-A#

RP/0/RSP0/CPU0:RO-A#exit

#

Clean up

Make sure that after you're done with your changes, in case you made the rommon vars persistent, you may want to unset

the variables to get back to the normal files that are used.

 

rommon> unset IOX_ADMIN_CONFIG_FILE

rommon> unset IOX_CONFIG_FILE

rommon> sync

 

All set!

 

If you forget the cleanup, you might see these lines:

 

RP/0/RSP1/CPU0:Oct 28 07:18:37.141 : locald_DSC[301]: %SECURITY-LOCALD-3-LWA_ADD_FAIL : Failed to add the username admin to lightweight authentication password database: No such file or directory

 

Another way to clear the variable:

more nvram:/classic-rommon-var  location 0/RSP1/CPU0

run iox_on 0/RSP1/CPU0 nvram_rommonvar IOX_CONFIG_FILE ""

Step 5


XR-VM Username/Password reset procedure using Sysadmin VM

 

Note: This is not a process to hack router but user need sysadmin username / password for the accessibility of box by bypass XR credentials.

Steps to perform this activity:

1. login to router : I was having console access to the box.

bgl-xdm-009:112> telnet 10.67.30.20 2037
Trying 10.67.30.20...
Connected to 10.67.30.20.
Escape character is '^]'.


User Access Verification

Password:
Password OK

2. Pass interrupts "ctrl + o" to toggle to sysadmin

sysadmin-vm:0_RP0#
*** IDLE TIMEOUT ***

System Admin Username:

3. enter sysadmin username and password

System Admin Username: xxxxx

Password:

xxxxx connected from 127.0.0.1 using console on sysadmin-vm:0_RP0
sysadmin-vm:0_RP0#


4. with this login you can access "sysadmin- VM prompt"

sysadmin-vm:0_RP0#

5. From Sysadmin VM to access XR - VM perform following action

i. list sdr ips
sysadmin-vm:0_RP0# show sdr
Tue Apr 3 05:12:47.110 UTC

SDR: default-sdr
Location IP Address Status Boot Count Time Started
-----------------------------------------------------------------------------
0/RP0/VM1 192.0.0.4 RUNNING 1 03/01/2019 02:10:28
0/RP0/VM2 192.0.0.6 RUNNING 1 03/01/2019 02:10:56

ii. ssh SDR VM1 address

[sysadmin-vm:0_RP0:~]$ssh 192.0.0.4

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
<>
Last login: Fri Mar 1 02:14:39 2019 from 192.0.0.4

iii. Now you are at XR-VM and to disable credential pass "ctrl + a" interrupt

[xr-vm_node0_RP0_CPU0:~]$exec -a

6. After disabling credential you can access XR-VM

RP/0/RP0/CPU0:customer2#

at this stage you can create/ delete / modify user credentials to access router at XR-VM directly.

P/0/RP0/CPU0:customer2#show running-config
Tue Apr 3 05:13:02.445 UTC
Building configuration...
!! IOS XR Configuration version = 6.3.3
!! Last configuration change at Sun Feb 17 23:31:51 2019 by ZTP
!
hostname customer2 >>>
username asd
group root-lr
group cisco-support
secret 5 $1$rdaY$qJt7aNcc8uFKqhP/rK11V1
!

Related Information

It has been seen that sometimes a system autonomously enters password recovery mode. This is identified with:

“enter root-system username”

 

This is due to a ddts known as CSCth03923

You end up providing what you think is a known username and password combination and it failes to get you in.

The solution is simple, just enter a fake username/password that you know for sure has not been configured yet and you're in!

 

 

 

Xander Thuijs - CCIE #6775

Sr Tech Lead ASR9000

Comments
thushar362
Level 1
Level 1

Hi Xander,

              Thanks for your precious response. If I copy the steps and paste into my ASR router, will the telnet be activated ? Let me know if missed any mandatory steps because I didn't yet configure the same in XR .

xthuijs
Cisco Employee
Cisco Employee

you are missing the telnet ipv4 server, that is far more important then the line template (which is optional).

this is the minimum configuration to enable telnet:

telnet vrf default ipv4 server max-servers 4

vty-pool default 0 4 line-template default

xander

thushar362
Level 1
Level 1

Xander, thanks a lot. I want to create a vty for around 50 numbers and want to limit the maximum number of inbound connections as around 7 and maximum outbound connections as 25. Let me know if any more corrections required.

thushar362
Level 1
Level 1

Hi Xander,

                I have added the  above steps into my router. But I am not getting the expected result. Is there any mistakes in my above configuration ? This is my first experience on ASR. Please help me, I am waiting for your response.

xthuijs
Cisco Employee
Cisco Employee

Tushar, I don't have a crystal bowl so I can't really tell why it is not working for your case.

There are 2 steps very important here. that is the config register for pw recovery and the deviation of the admin and iox config files to boot an empty config and bypass any potential AAA and local user directives.

If that doesn't work, then it would be best to capture the logging, and document the steps you took and open a TAC case for additional support.

xander

thushar362
Level 1
Level 1

Xander,

                  Thanks for your reply. We have solved the problem. Still we want to redirect the traffic coming from some particular ip address(sources)  into some other destination. I planned to use class map along with policy map. But in policy map, there is no "next hop" option. Which method is the best to redirect the traffic ?. Along with that we want to apply the policy or condition on some interfaces only.

xthuijs
Cisco Employee
Cisco Employee

That functionality you're after is ABF (access list based forwarding). It is a "regular" ACL with a next hop option in any vrf you like.

Just one comment, this question has nothing to do with the article above. Moving forward, would want to recommend to raise "new" questions via the right forum so everyone can chime in in case I can't respond.

regards

xander

---

Xander Thuijs CCIE #6775

Principal Engineer ASR9000

manuv1984
Level 1
Level 1

Hi Xander,

                    You have mentioned in the above comments as "main purpose of line template is for console". But in most of the configurations I have seen this with telnet configuration. Above you have mentioned the step

"telnet vrf default ipv4 server max-servers 4" , here 4 means number of inbound connections(maximum number of  incomming connections to the router). If so  where can we configure maximum number of outbound connections? Along with that for simply enabling telnet, can't we use "telnet server" instead of the above step ?

Garry Peirce
Level 1
Level 1

re: the clean up stage, is the resetting of these variables possible from the IOS-XR CLI or only through rommon?

Ex. If I've made them persistant via 'sync' and then booted into image, do I have to return to rommon to unset the config file variable?

xthuijs
Cisco Employee
Cisco Employee

you can do it out of admin config also:

RP/0/RSP0/CPU0:A9K-BNG#admin config-register ?

  <0x0-0xffff>       a value for the config register

  boot-mode          set the boot mode characteristics

  console-baud       set the console baud rate

  console-break-key  set the console break key

  password-recovery  set the password recovery mode

Garry Peirce
Level 1
Level 1

ah - I was looking for something in more IOS-XR speak rather than the IOS method ;-)

I currently show a config-reg of 0x2102 which I would think should boot the current config.

However, I'm also seeing this on reload.

%MGBL-CONFIG-6-STARTUP_ALTERNATE : Configuration Manager can not find any configuration to apply from the alternate source '/harddisk:/no-config' . Default configuration will be applied.

booting to rommon I see that IOX_CONFIG_FILE remains set to something that does not exist.

I can clear this from within rommon, but I thought there may be a way from CLI - not sure config-reg can modify this variable (?).


xthuijs
Cisco Employee
Cisco Employee

there is no XR command to unset rommon variables other then the config register,

so you'd need to go back to rommon and "UNSET" the IOX_CONFIG_FILE variable to have the system use the default

which is sysdb that is the actual "start up" configuration.

xander

Hi Xander, 

   I believe step 3.a in the resolution should read:

     •a.       If you do not know a  local login or cannot use the KSH method to recover the configuration  then both the IOX_CONFIG_FILE and IOX_ADMIN_CONFIG_FILE 

   Instead of the current, where it states to change the ADMIN_CONFIG file twice:

  If you do not know a  local login or cannot use the KSH method to recover the configuration  then both the IOX_ADMIN_CONFIG_FILE and IOX_ADMIN_CONFIG_FILE

Other than that thank you for the detailed instructions.

Thanks,

Alex

xthuijs
Cisco Employee
Cisco Employee

Thanks Alex, good catch, yup that is what I meant! revised!

xander

mirkolobina
Level 1
Level 1

hi xander, tnx very much for this guide. we have a problem of tacacs reach. here with an asr 9010. for this reason we want to bypass base but not admin (which in our case contains only root pwd). i would fix the exact steps as follows. please comment/adjust.

1) reboot asr and via console CTRL-C to access ROMMON
2) in ROMMON: IOX_CONFIG_FILE=/harddisk:/no-config
3) in ROMMON: reset (this will reboot asr)
4) the asr will boot with the admin config untouched but without base config

now several questions:

1) the most important thing: after the asr reboots, let say i want to load a config. does the command "rollback configuration last/to" work? and also the command "sh configuration rollback changes *"?
2) what is the aim of sync (step 2) and can we avoid it?
3) where is phisically saved the base config in xr?

tnx in advance mirko


 

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Quick Links