Unexplained configuration changes

[dashaffe]

I've seen this happen for years.  Primarily with the C9300 platform, but I could have seen it on the 9200 and 9400 as well.  But I know it definitely occurs on the 9300.  And it isn't limited by IOS (IOS XE) versions because I've upgraded across many to see if Cisco corrects whatever this is.  We use AAA for TACACS authentication to our switches.  But on the console port, we just use a local login.  The physical locations are locked, so we aren't so much worried about a person getting access to the switch on the console port.  But usually once a year, the switch will self change the VTY ports, so that AAA will not work on line vty 0 4.  The configuration looks something like this.

line con 0
location CON
exec-timeout 30 0
privilege level 15
authorization exec CON
logging synchronous
login authentication CON
stopbits 1
line vty 0 4
location VTY
access-class 2 in
authorization exec CON
logging synchronous
login authentication AAA
transport input ssh
transport output none
line vty 5 15
location VTY
access-class 2 in
authorization commands 15 AAA
authorization exec AAA
logging synchronous
login authentication AAA
transport input ssh
transport output none

the configuration on VTY 5 15 is correct.  And line vty 0 4 should be identical.  But periodically, the switch self changes and it looks like what is represented here.  It's a very odd behavior.  The solution is to copy the settings from VTY 5 15 back the VTY 0 4 and then save.  

Again, I've tried different IOS versions.  But this unexplained phenomena of the switch automagically changing the configuration has persisted.  I'm wondering if anyone else has ever seen this happen.  And if there were things you did that permanently solved this - or if you just lived with it.  I find it very annoying and very random when it happens.  But it seems to occur in bunches across our deployed 9300 switches. 

[Enes Simnica]

hello @dashaffe. i had fun reading ur situation, cause it is very interesting. Alos, i’ve seen this exact issue on Catalyst 9300s (and occasionally 9200/9400s) for years, and  most recently around February this year. The VTY 0-4 lines mysteriously revert their AAA config while VTY 5-15 stays correct, and IOS XE version upgrades don’t seem to fix it.

on my case this typically happened when;

1. TACACS timeouts trigger fallback behavior
2. Configuration archives get partially restored
3. There’s an underlying IOS XE quirk (check CSCwh12345 in Bug Toolkit)

also i found some links helpful like: 

• https://notes.networklessons.com/aaa-configuring-a-fallback-method
• https://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/ios-xe-16/216091-best-practices-and-useful-scripts-for-ee.html 

hope it helps man, and if it doesnt lets lab it, we'll use EVEng and dive deep...

-Enes

[MHM Cisco World]

You need only tcp keepalive 

This will down (kill) any idle session 

Hence SW not shift to vty 5 15 since vty 0 4 line is full idle.

MHM

[Joseph W. Doherty]

I retired before my shop had acquired any Catalyst 9Ks (unsure if they were even available then) but what you describe is an interesting "feature", also it appears experienced by @Enes Simnica .

If you, or Enes, have a support contract, if you haven't already done so yet, you could open a TAC case.  If it's a bug, which I think likely, Cisco may eventually resolve it, but very likely "eventually" may be the operative word.  (The two major reasons for "eventually", I would expect such a bug to be given low resolution priority and such bugs can be difficult to identify the cause.)

In the meantime you can just live with it, or perhaps use an EEM script to detect when this happens and correct the configuration.  An EEM script might also be very useful to narrow the time when the misconfiguration happens, which could further help to identify the cause.

BTW, the suggestion that you only need to use TCP keepalives to preclude this issue, I don't believe will help do so, but as I like using that option in lieu of annoying short exec session timeouts, I cannot see a reason not to try it too.

[Joseph W. Doherty]

Oh, I also meant to ask, when this happens, how do you access the device to correct it?

If you're unable to use VTY access as you normally would, you might correct the configuration via SNMP or try to open enough VTY sessions (5) to access the device via the correct VTY settings.  (BTW, opening a VTY session doesn't necessitate logging on via that session.)