After Sup1 to Sup2 upgrade on 9509, strange behavior for TACACS+ and DIR BOOTFLASH:

kbyrd · ‎09-05-2010

I upgraded a 9509 from dual sup1s running 3.3(1c) to dual sup2s running 3.3(1c). I later upgraded to 4.2(7a). we did this in order to get support for a new DS-X9248-96K9 module.

After replacing the sup1s with sup2s, everything appeared fine with two exceptions:

1) when we telnet'd, we would fail AAA/tacacs authentication. Console using local username is fine,

MDS Switch
login: validuser
Password:
Login incorrect

Digging deeper, there were no configuration changes for AAA nor tacacs. We see failed attempts in the ACS server log. However:

C9509-DC-1# test aaa server tacacs+ 10.1.14.8 validuser validpassword
user has been authenticated
C9509-DC-1# test aaa server tacacs+ 10.1.14.8 validuser invalidpassword
user has failed authentication
...with the corresponding passed and failed logs in the ACS server.

Also, show tacacs-server statistics shows all passed attempts.

C9509-DC-1# show tacacs-server statistics 10.1.14.8
Server is not monitored

Authentication Statistics
        failed transactions: 0
        sucessfull transactions: 46
        requests sent: 46
        requests timed out: 0
        responses with no matching requests: 0
        responses not processed: 0
        responses containing errors: 0

Authorization Statistics
        failed transactions: 0
        sucessfull transactions: 14
        requests sent: 14
        requests timed out: 0
        responses with no matching requests: 0
        responses not processed: 0
        responses containing errors: 0

Accounting Statistics
        failed transactions: 0
        sucessfull transactions: 0
        requests sent: 0
        requests timed out: 0
        responses with no matching requests: 0
        responses not processed: 0
        responses containing errors: 0

Any Ideas:

2) DIR BOOTFLASH://SUP-LOCAL/ works, but DIR BOOTFLASH://SUP-STANDBY/ does not.

From the standby module, DIR BOOTFLASH://SUP-LOCAL/ works fine.

C9509-DC-1# dir bootflash://sup-local/
      16384     Dec 16 21:04:10 1916 lost+found/
   16604160     Jan 04 16:20:36 1917 m9500-sf2ek9-kickstart-mz.3.3.1c.bin
   21764608     Jan 04 16:23:54 1917 m9500-sf2ek9-kickstart-mz.4.2.7a.bin
   78718938     Jan 04 16:21:15 1917 m9500-sf2ek9-mz.3.3.1c.bin
103575233     Jan 04 16:25:46 1917 m9500-sf2ek9-mz.4.2.7a.bin
       2168     Jan 04 16:55:24 1917 mts.log

Usage for bootflash://sup-local
283611136 bytes used
619552768 bytes free
903163904 bytes total
C9509-DC-1# dir bootflash://sup-standby/
Input/output error
C9509-DC-1# attach module 6
Attaching to module 6 ...
To exit type 'exit', to abort type '$.'
---snip out the copywrite info----

C9509-DC-1(standby)# dir bootflash:
        315     Jan 04 16:28:16 1917 MDS20071129112452322.lic
      49152     Jan 04 16:46:30 1917 lost+found/
   16604160     Jan 04 15:51:05 1917 m9500-sf2ek9-kickstart-mz.3.3.1c.bin
   21764608     Jan 04 16:23:54 1917 m9500-sf2ek9-kickstart-mz.4.2.7a.bin
   78718938     Jan 04 15:49:28 1917 m9500-sf2ek9-mz.3.3.1c.bin
103575233     Jan 04 16:25:46 1917 m9500-sf2ek9-mz.4.2.7a.bin
       1484     Jan 26 17:11:39 1970 mts.log

Usage for bootflash://sup-local
283648000 bytes used
619515904 bytes free
903163904 bytes total

Any ideas?

Thanks.

bfeeny · ‎09-06-2010

Just to clarify, are you saying you see failed attempts in the TACACS log when you enter valid passwords? Or are you just saying you see failed attempts when you purposely fail authentication?

kbyrd · ‎09-06-2010

In the ACS server's passed and failed authentication logs, I see the corresponding Pass and Fail when I do the TEST AAA command from the MDS command line.

When I actually try to telnet and login, I see failed attempts in the ACS server log. That's what is so strange. The reason is either invalid password and one time I saw "user locked out" from the local ACS directory. Then I would run the test command from the command line and all works normaly.

Thanks for your follow-up...

bfeeny · ‎09-06-2010

By any chance do you have "aaa authentication login mschap enable" configured?

kbyrd · ‎09-06-2010

I do not:

aaa group server tacacs+ VTYACS
aaa group server radius radius
aaa authentication login default group VTYACS local
aaa authentication login console local
aaa authentication login error-enable

bfeeny · ‎09-06-2010

Strange indeed, but what does your TACACS group look like vs. what you are typing when you do the test aaa command?

kbyrd · ‎09-06-2010

This is the test:

C9509-DC-1# test aaa group VTYACS username goodpassword
user has been authenticated
C9509-DC-1# test aaa group VTYACS username badpassword
user has failed authentication
C9509-DC-1#

tacacs-server host 10.1.14.8 key 7 "glxJq09"
tacacs-server host 10.32.14.8 key 7 "glxJq09"
aaa group server tacacs+ VTYACS
server 10.1.14.8
server 10.32.14.8