IronPort 8.5.6 Cluster RADIUS Authentication - One Node works; the other doesn't!

[Ian Cowley]

Two Ironport Nodes with User Authentication configured under Cluster Management mode (the only way).and Submitted

RADIUS provided by 3-node Cisco ISE v1.2, authenticating against Microsoft AD.  Both nodes defined as Network Devices.

One machine works fine.

One machine communicates with ISE but is sending the incorrect password information and authentication fails.

ISE logs

5400 Authentication failed

24408 User authentication against Active Directory failed since user has entered the wrong password

How can the 2 machines behave differently when sharing the same user configuration!

[Ian Cowley]

RADIUS 101 Failure!!!

Turned out that RADIUS Shared Secret had a typo for one of the nodes.

But working with Cisco TAC introduced us to a very useful CLI command

'tail'

Currently configured logs:
    Log Name            Log Type                      Retrieval           Interval
 ---------------------------------------------------------------------------------
 1. amp                 AMP Engine Logs               Manual Download     None
 2. amparchive          AMP Archive                   Manual Download     None
 3. antispam            Anti-Spam Logs                Manual Download     None
 4. antivirus           Anti-Virus Logs               Manual Download     None
 5. asarchive           Anti-Spam Archive             Manual Download     None
 6. authentication      Authentication Logs           Manual Download     None
 7. avarchive           Anti-Virus Archive            Manual Download     None
 8. bounces             Bounce Logs                   Manual Download     None
 9. cli_logs            CLI Audit Logs                Manual Download     None
10. error_logs          IronPort Text Mail Logs       Manual Download     None
11. euqgui_logs         Spam Quarantine GUI Logs      Manual Download     None
12. ftpd_logs           FTP Server Logs               Manual Download     None
13. gui_logs            HTTP Logs                     Manual Download     None
14. mail_logs           IronPort Text Mail Logs       Manual Download     None
15. reportd_logs        Reporting Logs                Manual Download     None
16. reportqueryd_logs   Reporting Query Logs          Manual Download     None
17. scanning            Scanning Logs                 Manual Download     None
18. sntpd_logs          NTP logs                      Manual Download     None
19. status              Status Logs                   Manual Download     None
20. system_logs         System Logs                   Manual Download     None
21. trackerd_logs       Tracking Logs                 Manual Download     None
22. updater_logs        Updater Logs                  Manual Download     None
23. upgrade_logs        Upgrade Logs                  Manual Download     None
Enter the number of the log you wish to tail.
[]> 6

Press Ctrl-C to stop.
Fri Aug  1 10:11:01 2014 Info: User ***** from 10.72.160.10 failed authentication.
Fri Aug  1 10:20:21 2014 Info: Trying RADIUS server 172.16.18.9
Fri Aug  1 10:20:21 2014 Warning: Service RADIUS communication error (No valid responses from server (Incorrect shared secret))
Fri Aug  1 10:20:21 2014 Info: User ***** from 10.72.160.10 failed authentication.

Even ISE logs weren't to specific though in hindsight they do show that it was a possibility