cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
726
Views
5
Helpful
8
Replies
Beginner

Cisco ISE - all switch ports blocked after disconnection and re-connection

Hi Everyone (long time reader first time poster),

 

I have a Cisco IE4000 (actually a Rockwell Stratix 5400 OEM switch but they are hardware & IOS identical for purpose of this discussion) setup with RADIUS and TrustSec connections to an ISE server (running as a VM on my laptop). Current versions of ISE 2.6.0.156 and IOS 15.2(7)E are running.

 

On first connection to a TrustSec port everything is fine it connects normally and all the authentications pass and it runs  indefinitely. TrustSec ports work OK all is good.

 

If I disconnect the laptop (ie. ISE) and reconnect to TrustSec port it will drop the connection no matter which port. 

 

If I initially connect to a non-TrustSec port then connects for approx 1 minute before dropping the connections and same behavior as above on all ports. Again the RADIUS logs up until disconnection appear all green for the VM's (also running a Windows 2012 Server VM for ISE Web Interface), Host MAC Addresses etc. & Windows Authentication for the host NIC adapter (being bridged to the VM's) is disabled.

 

Being short of a console cable & Serial port I haven't yet been able to look at syslog to see what is going on inside the switch, unfortunately since ISE is disconnected I don't get any further updates in the RADIUS logs to tell me what's going on either.

 

I do plan to get a console cable and get access (need to find an older laptop with serial port !) but in the interim if anyone has any idea's what might be going on please let me know !

 

Thanks,


Adrian

 

PS. Have attached the switch config file. 

1 ACCEPTED SOLUTION

Accepted Solutions
Beginner

Re: Cisco ISE - all switch ports blocked after disconnection and re-connection

Thanks everyone for the suggestions.

 

I tried them to no avail however I reverted to an earlier snapshot of my ISE server and things are working again (without changing the switch config) so there must be something I've done since that has created an issue in ISE - though it still doesn't explain why when I connect to ports on the switch with no authentication settings I still cannot reach the switch.


Yes I am using authentication host-mode multi-auth but as mentioned everything actually authenticates OK the first time I boot up the switch all greens - it's only after disconnecting and re-connecting that it won't even attempt to connect it's blocked immediately upon connection.

 

I did get on the console and look at syslog and it tells me that the 4 MAC addresses (representing the VM's & host machine) cannot authenticate (this is on the re-connection of course) but again that's not an ISE configuration or policy issue it's simply that the switch cannot reach ISE to authenticate after the re-cojnnection as the port is immediately blocked (even though interface and VLAN's are Up). 

 

What is curious about that is the fact the switch does remember the MAC addresses that were previously authenticated - is there a table or way of showing what previous authentications are stored in the switch and/or clearing them ? 

 

I'll do some comparisons between the original ISE setup and the newer non-working one and try to find the root cause. 

8 REPLIES 8
Cisco Employee

Re: Cisco ISE - all switch ports blocked after disconnection and re-connection

If you can provide more information about the ISE policy, I may be able to provide better answer. But, it sounds like you have hypervisor running on the PC. When using VM hypervisor with bridged networking, each MAC address will need to authenticate when multi-auth mode is used. If using ACL for authorization make sure host OS and each VM running gets ACL assigned.

Rising star

Re: Cisco ISE - all switch ports blocked after disconnection and re-connection

I agree with @howon. Adding additional nice-to-knows:
authentication host-mode multi-auth; As mentioned by howon it will authenticate each mac;
authentication host-mode multi-host; not recommended, but this will authenticate one mac and every other mac will be good after the one mac auth.
If you are running multiple connections and multiple physical hosts and you do vMotion or another technology that is similar you will want to enable this switch side:
authentication mac-move permit; basically tells the switch to allow VM macs to move between the physical server links;
HTH!
Cisco Employee

Re: Cisco ISE - all switch ports blocked after disconnection and re-connection

Make sure to check the Cisco ISE Secure Wired Access Prescriptive Deployment Guide

Http://cs.co/ise-guides
Beginner

Re: Cisco ISE - all switch ports blocked after disconnection and re-connection

Thanks everyone for the suggestions.

 

I tried them to no avail however I reverted to an earlier snapshot of my ISE server and things are working again (without changing the switch config) so there must be something I've done since that has created an issue in ISE - though it still doesn't explain why when I connect to ports on the switch with no authentication settings I still cannot reach the switch.


Yes I am using authentication host-mode multi-auth but as mentioned everything actually authenticates OK the first time I boot up the switch all greens - it's only after disconnecting and re-connecting that it won't even attempt to connect it's blocked immediately upon connection.

 

I did get on the console and look at syslog and it tells me that the 4 MAC addresses (representing the VM's & host machine) cannot authenticate (this is on the re-connection of course) but again that's not an ISE configuration or policy issue it's simply that the switch cannot reach ISE to authenticate after the re-cojnnection as the port is immediately blocked (even though interface and VLAN's are Up). 

 

What is curious about that is the fact the switch does remember the MAC addresses that were previously authenticated - is there a table or way of showing what previous authentications are stored in the switch and/or clearing them ? 

 

I'll do some comparisons between the original ISE setup and the newer non-working one and try to find the root cause. 

Highlighted
Cisco Employee

Re: Cisco ISE - all switch ports blocked after disconnection and re-connection

'show authentication session interface' command shows the sessions and MAC addresses tied to the session. However, as noted by Mike 'authentication mac-move permit' will allow the specific MAC to move to different interface.

Beginner

Re: Cisco ISE - all switch ports blocked after disconnection and re-connection

I have added 'authentication mac-move permit' to no avail however it should be noted again that the issue happens even if I disconnect and reconnect on the same port. 

 

When the issue occurs I do a 'show authentication session interface' and it shows the 4 learned MAC addresses as NOAUTH but again there is something more fundamental going on where the switch won't forward packets at all hence no attempt to authenticate is made or registered (the RADIUS log on ISE server is empty). I tried 'clear authentication session' via console but to no avail they still show up with NOAUTH when I reconnect. 


I should also that contrary to my original post this blocking also happens immediately on ports with no additional config or authentication settings - in that case a 'show authentication session interface' shows no sessions but the blocking still occurs.


I can reproduce / solve it consistently by reverting to a 'Good' snapshot of ISE vs. 'Bad'  snapshot of ISE to which I have saved the .tar.grp Backups of the Config & Ops along with local CA's.

 

Does anyone know of a tool or simple way to compare these backups and identify any differences - or is it a case of manually going through the ISE interface section by section ? The latter is going to be quite time consuming not least for having to run both 'Good' and 'Bad' snapshot VM's simultaneously by cloning etc. 

Cisco Employee

Re: Cisco ISE - all switch ports blocked after disconnection and re-connection

No tool that I know of to compare ISE policies. You could try exporting the policies out at ISE admin web UI > Administration > System > Backup & Restore > Policy Export.

I just noticed that your setup is not a norm, because the same set of the switch interface and PC network connection are used by ISE as well as the test client. Thus, the switch is likely considering the RADIUS is dead for some time.

I would suggest to get another wired adapter on your machine so that ISE may use its own connection to a switch interface not doing authentication, and the test client has its own as well to connect to a 802.1X enforced switch port, if you are unable to find another machine to serve as the test client.

Beginner

Re: Cisco ISE - all switch ports blocked after disconnection and re-connection

hslai,

 

Yes I had thought about the sharing of interface and did try connecting ISE with a separate standalone NIC but to no avail and it also doesn't really explain why the 'good' configuration works. I did try to look at the order of authentication for the 4 MAC addresses all coming out of the interface (ISE, 2 x Windows VMs + Windows Host) on console to see if that had an effect but I'm convinced the switch simply will not forward any packets in the 'bad' config - again this is based on zero activity in ISE RADIUS log and also being unable to connect on an un-configured interface. 

 

That said the last time I tried using separate NIC for ISE was before I had identified with certainty my 'good' and 'bad' ISE configurations - so I will try again testing with both now I have a baseline to work from.

 

Thanks also for the tip on Policy Export I will look at that as well.