Weird TACACS issue after upgrading to IOS 15 - periods in username replaced with underscores

[viawest.net]

I ran into a weird issue tonight when upgrading some 4948E switches. I upgraded 4 to 15.1(2)SG - cat4500e-ipbasek9-mz.151-2.SG.bin.

2 of them directly connect to firewalls so management stayed on SVIs but the other two were going to be migrated to use VRFs. I made the VRF change including AAA on the one of them and couldn't login to TACACS. Then noticed I couldn't login to the other one either, even though it was still using the SVI.

Checking the debug logs on my TACACS server I noticed that my username "john.smith" was being sent as "john_smith" so it failed auth. We have an LDAP user account for "rancid" and that was fine. So for some reason these two switches were changing the underscore to a period in the username, causing the LDAP authentication to fail.

I checked the configs for these two 4948E's against other switches, same model, same software version. Made the AAA/VRF configs exactly the same - still had this issue. The four I was upgrading tonight do point to a different TACACS server, but only minor things changed in the config, like the key and accounting file.

Rembering how NX-OS will complain about invalid operator strings, and not knowing what else to do, I changed this line in my tac_plus.conf file:

optional shell:roles = "\"operations network network-admin\""

to this:

optional shell:roles = "\"network-admin\""

And viola! It fixed the issue.

Anyone know how that would've caused the issue? That same line is in my other TACACS server. Same switch model, some software, same config. Only difference is IP addresses used and network layout. In the issue tonight, it was a private IP environment and on same subnet as the TACACS server but I don't see how that would have any effect.

Also I did debugs on the switch, and the switch itself was sending the underscore in the username.

[Javier Henderson]

Can you please collect the console output after enabling "debug tacacs" and "debug aaa authentication" and "debug aaa authorization" and post the results here?

Javier Henderson

Cisco Systems

[viawest.net]

I will see about reverting that tacacs config statement and getting that debug output. Also, I made a typo in my initial post. The *period* in the username was being replaced with an underscore. Here are some tacacs debug logs I have right now showing the issue happening.

Fri Jan 24 23:06:51 2014 [19139]: login query for 'john.smith' tty1 from 172.16.1.200 accepted

Fri Jan 24 23:06:51 2014 [19150]: authorization query for 'john.smith' tty1 from 172.16.1.200 accepted

Fri Jan 24 23:13:45 2014 [20053]: login query for 'john.smith' tty0 from 172.16.1.200 accepted

Fri Jan 24 23:13:54 2014 [20122]: enable query for 'john.smith' tty0 from 172.16.1.200 accepted

Fri Jan 24 23:21:12 2014 [21117]: login query for 'john_smith' tty1 from 172.16.1.200 rejected

Fri Jan 24 23:21:17 2014 [21130]: login query for 'john_smith' tty1 from 172.16.1.200 rejected

Fri Jan 24 23:34:10 2014 [22948]: login query for 'john.smith' tty0 from 172.16.1.210 accepted

Fri Jan 24 23:34:13 2014 [22976]: enable query for 'john.smith' tty0 from 172.16.1.210 accepted

Fri Jan 24 23:34:47 2014 [23049]: login query for 'john_smith' tty2 from 172.16.1.200 rejected

Fri Jan 24 23:35:40 2014 [23070]: login query for 'john_smith' tty2 from 172.16.1.200 rejected

Fri Jan 24 23:36:21 2014 [23271]: login query for 'john_smith' tty1 from 172.16.1.210 rejected

Fri Jan 24 23:36:39 2014 [23325]: login query for 'john.smith' tty2 from 172.16.1.210 accepted

Fri Jan 24 23:36:39 2014 [23336]: authorization query for 'john.smith' tty2 from 172.16.1.210 accepted

Fri Jan 24 23:38:16 2014 [23540]: login query for 'john_smith' tty1 from 172.16.1.200 rejected

Fri Jan 24 23:39:44 2014 [23749]: login query for 'john_smith' tty1 from 172.16.1.200 rejected

Sat Jan 25 00:06:44 2014 [27449]: login query for 'john.smith' tty0 from 172.16.1.210 accepted

Sat Jan 25 00:06:53 2014 [27504]: enable query for 'john.smith' tty0 from 172.16.1.210 accepted

Sat Jan 25 00:34:57 2014 [31581]: login query for 'john_smith' tty1 from 172.16.1.210 rejected

Sat Jan 25 00:52:18 2014 [1510]: login query for 'john_smith' tty1 from 172.16.1.200 rejected

Sat Jan 25 00:52:26 2014 [1525]: login query for 'john_smith' tty1 from 172.16.1.200 rejected

Sat Jan 25 00:59:42 2014 [2617]: login query for 'john_smith' tty2 from 172.16.1.200 rejected

Sat Jan 25 01:05:25 2014 [3499]: login query for 'john.smith' 0 from 172.16.1.220 accepted

Sat Jan 25 01:05:26 2014 [3502]: authorization query for 'john.smith' 0 from 172.16.1.220 accepted

Sat Jan 25 01:07:24 2014 [3790]: login query for 'john.smith' tty2 from 172.16.1.200 accepted

Sat Jan 25 01:07:24 2014 [3801]: authorization query for 'john.smith' tty2 from 172.16.1.200 accepted

Sat Jan 25 01:09:22 2014 [4071]: login query for 'john.smith' 0 from 172.16.1.220 accepted

Sat Jan 25 01:09:23 2014 [4078]: authorization query for 'john.smith' 0 from 172.16.1.220 accepted