ISE 2.4 has several long-lived TCP sessions between ISE nodes

[Nadav]

Hi everyone,

I have a dispered ISE deployment (each persona is on a dedicated node).

It seems like several of the inter-node traffic are in fact long-lived TCP connections. This means the TCP connection may last hours and is never reset by either client or server. Such traffic includes TCP 9300, TCP 1521, TCP 443, TCP 12001. 

Firewalls such as Checkpoint GAIA, Fortinet etc. keep track of all existing TCP connections and usually age them out after a certain amount of time. This is mostly as a security practice and to also keep the TCP session table manageable. Once the TCP connection is aged out, the firewall will drop the traffic without sending a RST/FIN flag to either side so that they can reestablish the connection. This may appear as TCP traffic is dropped because the "first" packet doesn't include the SYN flag.

1) Since it is likely that some ISE nodes will have to traverse a firewall to reach one another, the aging of these TCP connections is a problem. It requires restarting the ISE process or waiting some unknown span of time for the session to recognize that the traffic is being dropped. I've seen this happen constantly for each of the aforementioned protocols. 

2) This is the only application I recall with such long lived TCP connections which aren't able to quickly reconcile aging out of the session. It seems like an outlier with enteprise software and I hope it will be addressed in future versions.

3) Is there any way to work around this issue without having to allow traffic between these nodes to not be aged out? 

I'd be happy to hear the community's thoughts on this and if anyone else has experienced this issue.

[hslai]

As it a common practice, ISE services need work with firewalls. You could be hitting this bug -- CSCve31569.

[Nadav]

I have insufficient permissions to view that link, any chance you can send a screenshot?

[hslai]

I updated CSCve31569 and it would take usually a day or two to be reviewed by Cisco before it becomes visible externally.

Symptom:

Every two weeks once and normal work Live Logs page become very slow, stuck in "Loading Data" state.

TCP 1521 keeps going out of state with the firewalls (which then drop the packet because the connection is stale). It takes about 20 minutes for ISE to re-establish the connection during which the PAN node is almost unusable (incredibly slow, pages not loading, etc).
Conditions:
Firewalls between Admin and Mnt nodes.
Workaround:
restart or reboot of PAN server . Moving the Admin and Mnt node to same VLAN.

[hslai · Accepted solution]

It turns out that ISE 2.3+ should already have the fix for CSCve31569 and the out-of-state connections are not actually used. Thus, it's best to engage Cisco TAC if you are seeing an issue with ISE services.

[Nadav]

It's curious that they mention only TCP 1521, when several sockets in fact suffer from firewall aging out. 

[hslai]

Right. TCP 1521 is usually the most problematic, and served as an example that should not prevent ISE services from working properly due to state-ful fire-walling.