ISE 3.1 auto install on VMware with iso hangs

[praestans]

Hello, I'm working on a fully automated install of ISE 3.1 in VMware, and installing via the Cisco provided full installation iso. I'm having an issue where the auto installation starts as expected, but then hangs at the 3rd line (Probing EDD...). I haven't been able to get this to progress past this point with esxi 6.5, 6.7, or 7.0, and I've tested in multiple environments, and have allowed up to 20 hours for the install to complete with no success. If I manually select option 1 on the initial screen (instead of allow the 150 seconds to elapse for the auto install to start) the install runs fine. I'm attaching a screen shot of where the install gets stuck. Consequently, the ISE 3.1 ZTP guides show this same stopping point in several screenshots, with no explanation of how to get it to progress past this point except in Hyper-V, which I can't use.

I'm hoping I am just missing a setting on the VM provisioning or something, but so far I haven't found it. Any insight anyone can provide would be really helpful.

Thanks

[ammahend]

This is not direct answer to your question, but see this link to make sure you are following the right process.

https://community.cisco.com/t5/security-knowledge-base/ise-zero-touch-provisioning-ztp/ta-p/4541606/jump-to/first-unread-message#toc-hId-1597011557

[praestans]

Thanks for the reply @ammahend. I have a request on that thread for help as well. I'm following the process he outlines with an exception, and that may be the issue but I can't find any info on it. Instead of using the .img file for the setup configuration, I'm using an advanced configuration parameter in the VM settings as outlined in this document: https://www.cisco.com/c/en/us/support/docs/security/identity-services-engine/217536-install-and-setup-ise-3-1-using-zero-tou.html#anc11

The setup portion proceeds smoothly, once the installation completes, which only occurs if I select option 1 manually, as the auto install hangs every time. I haven't found anything stating that the auto install will only proceed if a .img file is mounted to a second CD/DVD drive. But I also haven't found anyone that has used this method for ZTP with ISE.

[praestans]

Well I believe I found the answer and unfortunately it looks like Cisco decided to require the setup config in the .img format to be mounted to the secondary CD/DVD drive, which essentially eliminates the possibility of using the process described in the actual ZTP document for vm advanced configuration parameter. Or at least ISE is not recognizing that method. I inspected the serial.out log and found the following:

***** Auto install configured
***** The ZTP configuration image is missing or improper. Automatic installation flow exited.
***** Power off and attach the proper ZTP configuration image or choose manual boot to proceed.
[ 49.174738] reboot: System halted

So, if anyone in the BU is reading this, can you please remove the configuration image requirement for the auto install, so the process in your document will work? In the environment I'm developing for, I don't have access to a linux box to dynamically create the img per deployment, so I need to either use the vm advanced configuration parameter option, or find a way to create the img using powershell. If anyone can point me to that, I'd greatly appreciate it.

Thanks

[Greg Gibbs]

Is there a reason you are using the ISO rather then using the OVA? Using the OVA is much faster and I've found the method of using the OVA with the VM Data option for ZTP to work consistently. I've used this method in my lab multiple times.
Installing ISE using OVA and VM Data 

[praestans]

@Greg Gibbs thanks for the reply. There are a couple reasons I am using (and prefer) the ISO over the OVA.

First is that the ISO is much more flexible when building nodes, particularly in a lab environment where resources are scarce. If I had found any documentation (there may be one somewhere, I just didn't find it) saying that the .img config was required for auto install to work with ISO it would have saved me the headache of trying to figure out why it wasn’t working with ISO and switch to the OVA. Now that I figured it out it's not a big deal and I can test the OVA, but it was frustrating in the process.

Second, is that the OVA is almost twice the size of the ISO. The install package will be delivered to the customer (full suite, not just ISE) who will have to do the install while the network is air-gapped, so keeping the size low is helpful. In fact, since the ISO is required to recover lost/expired admin password, the size required to be delivered may possibly be ISO plus OVA.

So, I prefer the ISO. I'm going to test out the OVA, and will reply here if it's all good, but would greatly appreciate the option to use ISO and VM Data. Thanks again for your reply, its greatly appreciated.

[Greg Gibbs]

The VM Data option does also work with the ISO, but you need to ensure that the VM is built with the same specs that are automatically configured by the OVA (storage size can be customised as long as you meet the requirements), including all of the CPU/Memory Reservations. I tested the same VM Data method using the recently re-spun ISO (ise-3.1.0.518b.SPA.x86_64.iso) and it worked as expected. When building 3.1 using the ISO, you need to ensure you're using the RHEL7x64 Guest OS Version and verify that the Boot Options Firmware setting is BIOS.

[praestans]

Thanks @Greg Gibbs.

I was able to get the OVA working, and I was really glad to read that the ISO + VM Data works. Unfortunately, I’ve been trying it all day with no success. Not sure what I’m missing because it sounds like you’ve had no trouble with it.

I ensured I’m using the exact resource allocations as the OVA builds (I checked them side by side) in both the eval configuration (CPU 4, no reservation, Memory 16Gb, no reservation, and HDD 300Gb, and 6 Network adapters), and Small configuration (CPU 16, 16k MHz reservation, Memory 32Gb, 32Gb reservation, and HDD 300Gb, and 6 Network adapters), with boot to BIOS, and I used the same base64 encoded config that worked in the OVA. I reproduced this in both esxi 6.7 and 7.0, and I’m using the new copy of the ISO (ise-3.1.0.518b.SPA.x86_64.iso) but get the same error I posted above in every scenario.

I did notice that the OVA actually installs with RHEL8x64 when I was comparing side by side, so I tested that too, but no joy.

If there’s anything else you can see that I’ve missed, please let me know. I’m sure there’s got to be one small thing I’m missing.

[Greg Gibbs]

Hi @praestans 

You are correct that the OVA builds the VM with the RHEL8x64 guest type. The RHEL8 boot options default to EFI, however, so when using that setting you must manually change the setting to BIOS.

I did some additional testing in my lab and found that, in order for the ISO + VM Data method to work, I have to select option 1 (Keyboard/Monitor). If I simply hit <enter> it appears to default to serial and look for a mounted image. In the serial output, I can see the ZTP errors and the installation halts.

When using the Keyboard/Monitor option, there is no serial output until after all the packages install and the initial reboot to the setup wizard happens. I do not see any serial output related to ZTP, so you only know it works when you see the setup content automatically populated on the vmware console (or waiting ~30minutes for the install to complete and connecting via SSH/GUI).

[praestans]

Hi @Greg Gibbs , sorry for the late response, I took a much needed vacation last week.

I did notice that the RHEL8 VM defaulted to EFI in the boot options, and changed it to BIOS.

So after reading through your additional testing, I’m thinking I may not have been clear on my initial description.

I too have had success with ISO + VM Data, if, and only if, I manually select option 1 at the boot options screen. I’m attempting to accomplish a Zero Touch install and config process, so requiring a person to manually select option 1 + [ENTER] defeats the purpose.

My goal is to require no human interaction after the VM is booted (the boot will actually be automated as well using PowerCLI). Once booted, the VM will sit there with no human interaction, and allow 150 seconds to elapse so the automatic installation kicks off (this is what I may not have been clear on, ie the Auto Install kicking off after 150 seconds). The installation should complete, and the VM Data configuration will automatically complete the initial setup wizard.

Currently, the automatic installation after 150 kicks off as expected, but then fails with the following error (from the serial output):

***** Auto install configured
***** The ZTP configuration image is missing or improper. Automatic installation flow exited.
***** Power off and attach the proper ZTP configuration image or choose manual boot to proceed.
[ 50.483692] reboot: System halted

So, when using the ISO + VM Data + Automatic Install, the "ZTP configuration image" is not being found.

By using the exact same setup and changing only the install source type (OVA instead of ISO) it works.

Would you be willing to test the scenario I laid out above (ISO + VM Data + Automatic Install)?

At this point I just need a sanity check. If you get the same result, then I guess the question is: should it work with OVA but not with ISO?

BTW, I did test the above on ISE 3.2 that was release yesterday with the same result. Thanks again for engaging on this with me.

[praestans]

@Greg Gibbsas soon as I hit reply on that last message something hit me that you wrote. You wrote that if you hit <enter> it appears to default to serial and look for a mounted image, which is what it must be doing when the auto install kicks off. So I compared the serial.out from when I selected option 1 to one that the auto install started and sure enough I see at the top of the auto install copy all the serial settings (9600 baud, etc...). So, that must be why the ISO + VM Data + Auto Install isn't working. It's looking for the config file on the serial interface instead of the virtual CD/DVD drive.

I'm guessing they made serial the default for CIMC use on physical boxes????

I'm going to play around with the settings and see if there's a way to get the VM Data option recognized with the ISO + Auto Install. This is out of my wheelhouse though so I may just have to find another way to automate the install. I'll reply here one way are another for anyone else who might be interested.

[Greg Gibbs · Accepted solution]

@praestans, I tested the same scenario (ISO + VM Data + Automatic Install) with the re-spin ISO and get the same error output to the virtual serial console.

23:31:09 Running pre-installation scripts
***** Auto install configured
***** The ZTP configuration image is missing or improper. Automatic installation flow exited.
***** Power off and attach the proper ZTP configuration image or choose manual boot to proceed.
[ 41.928220] reboot: System halted

All of the documentation I've seen for ZTP only discusses VM Data when used with the OVA, so I suspect this combination is not supported. Manually selecting Option 1 at the boot menu must somehow trick the system into checking for the VM Data.

[praestans]

@Greg Gibbsthanks again for checking into this. I think that's final.

I did try to find another way to make it work, but to no avail.

[praestans]

For anyone who may be trying to use this method also, I did eventually find a solution.
This is for automated install of ISE 3.X using ISO + VM Data method on VMware, which allows config via API, and in my case I was able to achieve 90% of the ISE config for my client via scripts.

When using ISO + VM Data, if the installer allows the 150 seconds to elapse for ZTP, ISE does not use the VM Data setup parameters attached to the VM Advanced configs, which is why my scenario was failing. In this case, the installer has to enter option 1 and press Enter to kick off the install and use the VM Data setup parameters attached to the VM Advanced configs.

Using PowerCLI in Windows Powershell created by William Lam of VMware (https://williamlam.com/2017/09/automating-vm-keystrokes-using-the-vsphere-api-powercli.html) I was able to send the "1" and "Enter" keystrokes to the VM Guest and launch the ISO + VM Data install.

Also, here's bonus that isn't listed in the ISE API guide included with ISE. It's on the DevNet page, but kind of hard to find.
If you need use the API to check if the API is enabled on ISE, run a get to:
"https://[ISE FQDN or IP]:443/admin/API/apiService/get"
If it's not enabled and you want it to be, run a post to:
"https://[ISE FQDN or IP]:443/admin/API/apiService/update"

[Arne Bier]

I'm still not sure why the ZTP autostart function does not support the 2nd CDROM option (where my .img file is attached), when it doesn't find the VM Advanced parameters (which I didn't configure). There should be support for both options in my opinion - pressing 1 during BIOS boot works for me as a temporary workaround, but it's not ideal. I am building ISE 3.2 using ZTP in the lab, and using OVA is not an option for me.

@Greg Gibbs - do you know if there is an enhancement request for this?