Cisco ISE 3.1 - Nodes trying to communicate on 169.254.169.254

[milos_p]

Hi,

I have two nodes Cisco ISE 3.1 patch3 deployment.

I am seeing lot of communication from both ISE nodes towards IP 169.254.169.254, that is eventually getting blocked on the firewall interface.

I cannot find any information related to this IP and Cisco ISE.

Is this some kind of normal inter-nodes communication?

Regards,

Milos

[balaji.bandi]

 169.254.169.254  - This is Link Local Address - not that aware ISE use until is there any bugs around 3.1 version

[milos_p]

Hi,

It's quite often, every few minutes there is communication from both ISE nodes towards 169.254.169.254.

[ahollifield]

Could be the docker processes within ISE, see here: https://community.cisco.com/t5/network-access-control/cni-podman1-amp-cni-podman2-on-ise-version-3-1-patch-3/m-p/4628250#M575340

[milos_p]

Hi,

I see, most probably you are right and it's related to docker IPs.

Let's see if someone else has some other explanation :-).

Regards,

Milos

[Damien Miller]

As Adam mentioned and linked to in Greg's post, 169.254.x.x are addresses used by the docker containers internal to the ISE node. You'll see many in use if you run a "tech netstat" on the CLI for many of the functions we know and love, COA, RADIUS, Portals etc. 

What's odd to me is that you're saying you are seeing this traffic at a firewall, the internal docker traffic should stay that way. 

So what might be happening is ISE is resolving something via DNS to the 169.254.169.254 address, then sending requests. I did a google of "169.254.169.254" because it's an odd APIPA address, I found that AWS, Azure, GCP, and others use this specific IP for certain metadata lookups. Where are your nodes hosted? 

[milos_p]

Hi Damien,

Thanks a lot for your reply!

Yes, I am seeing this traffic on the firewall interface, which serves as a default gateway for ISE.

My nodes are deployed locally on in our infra on vmware.

I checked tech netstat and I can see 169.254.2.1 and 169.254.4.1, no sign of 169.254.169.254 or any other APIPA address.

Just to point again, traffic log is showing that source is real IP addresses of ISE servers (not APIPA) and destination is 169.254.169.254 on TCP port 80.

You know what, now that I noticed that communication is HTTP (TCP/80), and sourced from ISE servers, this could be CRL check from some certificate.

[cisco.smj]

I was modifying an ACL and started seeing blocks to this address from my ISE admin node.  Since this is HTTP traffic I decided to spin up a little web server to see the payload.  I have attached a snippet from Wireshark.  ISE seems to be looking for the instance identity information. 

[Vittoriusly]

169.254.169.254 is the IP address associated with AWS/Azure Instance metadata services when you install a VM on those environments.

[RedOne]

Hello, I see the same IP (169.254.169.254) blocked in our FWs.

From ISE nodes http to 169.254.169.254

 ISE 3.1 patch 5

did you have your answer?

[cisco.smj]

It has to do with the instance identity information used for cloud deployments.

[poongarg · Accepted solution]

There is a defect already filed for this issue:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwc45154

Check the Known Limitations for Using Cisco ISE on AWS section of Cisco ISE Installation Guide 3.1.

https://www.cisco.com/c/en/us/td/docs/security/ise/3-1/install_guide/b_ise_InstallationGuide31/m_ISEaaS.html#known-limitations-for-using-cisco-ise-on-aws

"In Cisco ISE Release 3.1 Patch 3, Cisco ISE sends traffic to AWS Cloud through IP address 169.254.169.254 to obtain the instance details. This is to check if it is a cloud instance and can be ignored in on-prem deployments"

HTH