Cisco Identity Service Engine (ISE) in AWS cloud - Page 2

adamscottmaster2013 · ‎01-06-2023

I am currently running Cisco ISE version 3.0 (running on SNS-3655) and version 3.1 (running on SNS-3415). There is a push to move this to AWS cloud in order to reduce the data center footprint. Anyone running Cisco ISE in AWS cloud can share your experience here? Both the Pros and Cons. TIA

Greg Gibbs · ‎02-01-2023

The only thing I can think of is that the keypair didn't attach correctly or something happened with the build. I have two AWS 3.1p1 instances in my lab and both work fine with my public key.

ssh -i cisco-ise-lab.pem admin@172.32.32.24
Failed to log in 0 time(s)
ise31aws1/admin#

ssh -i cisco-ise-lab.pem admin@172.32.32.132
Failed to log in 0 time(s)
ise31aws2/admin#

You might need to try rebuilding the 3.1p1 instance. If you built it using the AMI with user_data, you might try building it using the CloudFormation template instead to mitigate any potential issues with the manual user_data file.

adamscottmaster2013 · ‎02-03-2023

@Greg Gibbs: I've tried to rebuilt the 3.1p1 instance multiple times without much access with the exact same method I did with 3.2. I am using the Cisco market place AMI. I do have a TAC case opened with Cisco and it said there are issues with the 3.1p1 according to Cisco internal bug document. I guess it is not yet stable in AWS, no?

Greg Gibbs · ‎02-05-2023

The 'issues with 3.1p1' could also be due to the many bugs that have been fixed in the 5 patches that have been released for 3.1 so far.

I have a customer that I've worked with to deploy a global ISE cluster with 3.1 (patch 5) on AWS using Terraform. They have 10 nodes in the us-east-1 region and 4 nodes in the ap-se2 region. We did not experience SSH issues with any of those nodes.

As @hslai mentioned, it takes time for the SSH login to be available. From the time the node build starts, I would give it a good 30 minutes before trying to access it via SSH or GUI.

hslai · ‎02-04-2023

@adamscottmaster2013 Likely you need wait longer.

ISE 3.1 on AWS takes ~ 25 minutes after the instance first started before we may use SSH to login. ISE 3.2 reduces this to ~ 15 minutes.

If that is not the case, please share the SR number so I may review the case notes.

adamscottmaster2013 · ‎02-07-2023

I waited 75 minutes after the instance is launched and attempted to ssh into the ISE 3.1 p1 and still without success.

Greg Gibbs · ‎02-07-2023

Are you able to access the GUI after that time? If not, there is most likely something going wrong with the build and it is not completing.

When building directly from the AMI, you should confirm that you are using only the key-value pairs and meet all the guidelines for the User Data content as defined here:
Deploy Cisco Identity Services Engine Natively on Cloud Platforms

If you have not tried building using the CloudFormation Template option, you might try that to see if it works better.

adamscottmaster2013 · ‎02-07-2023

@Greg Gibbs: As mentioned before, I am using THE SAME METHOD to build ISE 3.2 as I did with ISE 3.1 patch-1. ISE 3.2 has no issues but 3.1 p1 does not work. I couldn't access the GUI. Yes, I am using the same key pairs for both ISE 3.1p1 as I did with ISE 3.2. I launched both of them in the same AWS VPC and the same subnet. Only 3.2 works.

Why should I need to use CloudFormation Template option when the other option is also supported by Cisco as clearly stated in the documentation? I opened a TAC case with Cisco but the TAC doesn't seem to understand ISE in AWS. He sent me back some documentation for me to try but it is the same documentation that I attached to the ticket when I first opened the ticket. Doesn't seem like Cisco TAC engineer understand much about ISE in AWS.

Greg Gibbs · ‎02-07-2023

I was suggesting the CF Template as it at least automates the creation of the User Data in case there is something in that content that is failing with 3.1 but not with 3.2 for some reason.

I've spun up/down ISE 3.1p1 instances in AWS (ap-southeast-2 region) using Terraform 20+ times in the past 6 months (including 2 days ago) and have not run into any issues with the build completing or the SSH/GUI interfaces not being accessible after 30+ minutes.

adamscottmaster2013 · ‎02-07-2023

The problem with the CF template is that it doesn't let me specify m5.8xlarge instance, only m5.4xlarge is available. I want to build the ISE instance with m5.8xlarge.

I would like to mimick the SNS-3695 with the m5.8xlarge but apparently with CF template, I only see m5.4xlarge. I guess I can change the EC2 instance after deploying but it should be available for me at the start, right?

Greg Gibbs · ‎02-07-2023

I'm not sure why it does not have the m5.8xlarge instance size. However, there is a simple option to add it in the template.

On the Create Stack page, if you use the pre-selected template and click on 'View in Designer' you can edit the template in either JSON or YAML. After editing the template, you can create the stack and it will include the instance option.

The only section you need to update is (in YAML):

 AllowedValues:
- c5.4xlarge
- m5.4xlarge
- c5.9xlarge
- t3.xlarge
- m5.8xlarge

hslai · ‎02-11-2023

@adamscottmaster2013 Apparently, m5.8xlarge might have some compatibility issue with ISE 3.1. Please use it with ISE 3.2 only or at least for now.

If your region allows serial console connections, then try connecting to the serial console of your ISE 3.1 Patch 1 instance. Initially you would see the prompt as p2 login: (where p2 is the hostname used to build the AMI). After the instance started for 8 minutes or so, you should see a line with bonding in it

... [   59.453172] bonding: unable to delete non-existent bond0

Then, if you hit enter/carriage-return, the prompt should change p2 to the hostname you specified in the user data file. If the prompt is not changed after ~ 20 minutes uptime, then it means some user data is no good.

If the prompt is changed with your specified hostname, login as the username admin and the password specified in the user data. Then, you may use this CLI command to check whether the SSH key gets copied:

show logging system ade/ADE.log | include authorized

Below is a sample output:

tt-31p1/admin# show logging system ade/ADE.log | i authorized
2023-02-12T04:00:06.244781+00:00 tt-31p1 root: info:[application:operation:cpmcontrol.sh] authorized_keys file created
2023-02-12T04:00:06.248533+00:00 tt-31p1 root: info:[application:operation:cpmcontrol.sh] adding the public key to the authorized_keys
--
2023-02-12T04:05:13.641055+00:00 tt-31p1 ADE-SERVICE[1497]: [50706]:[info] utils: vsh_pipe.c[117] [admin]: pattern after filtering dashes: authorized

adamscottmaster2013 · ‎02-12-2023

@hslai: is this issue resolved after I upgrade ISE 3.1 to patch-5?

hslai · ‎02-12-2023

@adamscottmaster2013 wrote:

@hslai: is this issue resolved after I upgrade ISE 3.1 to patch-5?

@adamscottmaster2013 No, we are not publishing new AMIs for subsequent ISE 3.1 patch releases.

I tried one instance of ISE 3.1 Patch 1 with m5.8xlarge and it worked. In fact, that is how I got those outputs. If possible, please ask TAC to include me in your next WebEx meeting.

adamscottmaster2013 · ‎02-14-2023

@hslai: Thank you. So this is a known issue with ISE 3.1 P-1, correct?

I can't use ISE 3.2 at the moment because my environment is 3.0 and 3.1 and I prefer to stick with 3.1-P5 in AWS. It seems to me it is a stupid idea by Cisco not to publish new AMI's for subsequent ISE 3.1 path releases. All other vendors are not doing what Cisco is doing in AWS.

ahollifield · ‎02-14-2023

FWIW Aruba ClearPass doesn't push new versions of their AMI when a patch is released. They only publish the base ".0" release and then you are free to install patches through the normal process (same as Cisco ISE).