Some 3850 switch ports are automatically reverting changes?

[Mogwai]

I've found 4 switchports on two switches in our environment that are automatically reverting changes that are made to them. In this case, I'm attempting to simply change the assigned VLAN of the switchport and then bounce the port.  When I look at the port configuration, I see different settings for it in two places... The current running config of the switchport is below:

interface GigabitEthernet1/0/13
 description Box Access Point
 switchport access vlan 440
 switchport mode access
 switchport voice vlan 2
 switchport port-security maximum 50
 switchport port-security aging time 1
 switchport port-security aging type inactivity
 switchport port-security
 ipv6 traffic-filter DENY-IPV6 in
 storm-control broadcast level bps 2m 1m
 storm-control action shutdown
 storm-control action trap
 spanning-tree portfast
 spanning-tree bpduguard enable
 spanning-tree guard root

The config of the same port from the "show interface status" is below:

Gi1/0/13  Access Point       connected    924        a-full a-1000 10/100/1000BaseTX

The VLAN it is now supposed to be on is 440, but after a bounce, it always reverts to VLAN 924 instead. When I attempt the change again & then check the logging of the port, I see the following, but don't see any references that show an explanation, other than the port going back down on it's own, changing the port description & VLAN, & then coming back up. 

Dec 10 07:34:57.024 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/13, changed state to down
Dec 10 07:34:58.033 EST: %LINK-5-CHANGED: Interface GigabitEthernet1/0/13, changed state to administratively down
Dec 10 07:35:02.370 EST: %ILPOWER-7-DETECT: Interface Gi1/0/13: Power Device detected: IEEE PD
Dec 10 07:35:03.227 EST: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/13, changed state to down
Dec 10 07:35:03.369 EST: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/13: Power granted
Dec 10 07:35:07.557 EST: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/13, changed state to up
Dec 10 07:35:08.557 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/13, changed state to up
Dec 10 07:35:11.856 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/13, changed state to down
Dec 10 07:35:12.861 EST: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/13, changed state to down
Dec 10 07:35:15.454 EST: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/13, changed state to up
Dec 10 07:35:16.454 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/13, changed state to up

Any thoughts on what might be causing this? I've also found this same issue on one other switch in our environment. 100's of other access points have moved without any of these issues in the past. 

Things I've checked:

Performed a reboot of the switch. 

Performed a reload of the switch. 

Checked to make sure confreg is not set to 0x2142. 

Both switches have the correct VLAN's trunked to them & the VLAN is working on other ports on the same switches. 

Running & startup configs are updated.

[Mark Elsen]

  - @Mogwai     Are you saving the configuration when making changes to port settings ?
                         Also look into : https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9300-series-switches/216850-configuration-register-equivalent-clis-i.html

  M.

[Mogwai]

Yes, the config has been saved multiple times after applying the changes.

[Mark Elsen]

 - @Mogwai  Check the logs on the switch when unusual events happen ,

 M.

[Mogwai]

I have, those were already posted in the original post.

[Mark Elsen]

  - @Mogwai   As far as rebooting is concerned ; boot into rommon and issue the command :
                       switch # show romvar     
                       Look at the line: SWITCH_IGNORE_STARTUP_CFG   and set the value to 0 (if needed)

  M.

[pieterh]

1) do you have some function like smartport macro's enabled ?
the switch might dynamically detect an access point is connected and apply corresponding macro?

2) do you have a authentication-server (radius) policy that dynamically configures the port for vlan 924 on detecting of  the access point?  
    (or maybe to a client connected to this access point? where vlan 924 is assigned to the client, not the access point)

3) do you have any other management server that may modify this port configuration

[Mogwai]

1) do you have some function like smartport macro's enabled ? > No, nothing of that nature was ever configured. 
the switch might dynamically detect an access point is connected and apply corresponding macro?

2) do you have a authentication-server (radius) policy that dynamically configures the port for vlan 924 on detecting of  the access point?  > Yes, we do have RAD server, but none have polices that would tie to VLAN's for AP's. Just used for SSID's. 
    (or maybe to a client connected to this access point? where vlan 924 is assigned to the client, not the access point)

3) do you have any other management server that may modify this port configuration > Not at this time. 

[Cristian Matei]

Hi,

   Weird and interesting at the same time. If "service config" would be enabled, such scenarios would happen at boot time which is not your case. If "service scripting" is enabled, there could be local configurations (EEM, TCL) of which outcome would be the one you're seeing, or there may be SNMP doing it.

   Best would be if you could attach the entire switch configuration, after you sanitise it and hide sensitive information like users, passwords, IP addresses, etc. This way we have better chances to get to the root cause. Before proceeding this way, assuming you have the luxury to do so, you could enable configuration change notification and logging via below mentioned command set, afterwards perform config changes, do a shut / no shut on the port, validate changes are gone, and look at the logger feature to see if you see the config rollback commands being parsed:

archive
 log config
  logging enable
  logging size 500
  notify syslog

See from where and which user has done any config changes via command:

show archive log config all 

Thanks,

Cristian. 

[Mogwai]

Thanks for the tips, I did attach a clean copy of the config for you to review when you have a moment. I did find the 924 template, but that tempate wasn't applied at the interface, so I'm not sure what else might be causing it?

Also, I enabled the logging & then attempted to update the interface config again, but had the same result. 

Clean-switch0#show archive log config all
 idx   sess           user@line      Logged command
    1     1     user@vty0     |  logging enable 
    2     1     user@vty0     |  logging size 500
    3     1     user@vty0     |configure 
    4     1     user@vty0     |interface GigabitEthernet1/0/13 
    5     1     user@vty0     | switchport access vlan 443
    6     1     user@vty0     | shutdown 
    7     1     user@vty0     | no shutdown

The below were the only output in the logging after the interface swapped back.  

[Cristian Matei]

Hi,

   Very interesting and completely forgot about this functionality. You have "autoconf enable", which is a feature supposed to simplify and automate configuration, including port configuration based on attached device on the port, via templates. At the same time, this functionality, when enabled, overrides any manual changes configuration at the port level, which is what you're seeing. 

    Can you attach the output of following commands:

show template interface binding all
show template binding target GigabitEthernet1/0/13

 Thanks,

Cristian.

[Leo Laohoo]

1.  Look for an EEM or KRON script.

2.  Command:  show history all

3.  Command:  sh run | i Last configuration change|NVRAM config

[paul driver · Accepted solution]

Hello

Try the following:
int gig1/0/13
access-session inherit disable autoconf
switcport acces vlan 440
exit
wr mem
reload

[Mogwai]

Late reply, as it's been a busy week, but this did it!  Thanks!