CSCwq14246 - ISE internal disk check - ISE VM read/write issues

[FieldSanitation]

Greetings!  I am not familiar with ISE and am trying to gather information on how best troubleshoot an issue.

I'm fairly new to an enterprise environment that recently expanded into some remote sites using ISE on VMs and upgraded to 3.2

0500 EST on wednesdays we are seeing a 1.5GB spike of traffic going to or from the MNT and the endpoint posture assessment nodes.  I am trying to verify that this IS the read/write test or is there something else that runs once per week at 0500 that would send that much data inside of a 2-4 minute window?  Most other days of the week those nodes don't get past 750MB.

trying to better understand what goes on inside the ISEDAILYCRON.SH (found online) that may or may not be associated with that read/write test that happens found in this post:  

Triggered alerts after upgrading to 3.1 - Cisco Community

ISE continues to test I/O Read and Write Performance on a 3Hr interval (0, 3, 6, 12, 18 and 21 - 9 is skipped due to the CSCvx44981 VM IO Performance Checks not done at 09:00) and ISE has a Daily Task (isedailycron.sh) that takes the Average Disk I/O Performance Measurements gather for the  day and triggers an Alarm if the average is poor than the threshold.

(thanks @Marcelo Morais  for posting that tidbit.)

Is anyone familiar with challenges associated with that disk check on VM's that may have shared storage backends?

cheers!

[Marcelo Morais]

Hi @FieldSanitation ,

 the isedailycron.sh is a Shell Script that performs daily cleanup called from System Scheduler.

 It is made of many subroutines, I will describe the ones I remember:

• Check the DB Availability, checking if the ISE Database is running.
• Perform ISE daily DB Cleanup
• Check VM IO Performance Specs (write / read) that are updated once an hour from isehourlycron.sh.
• Perform a daily Purge Report Data.
• Delete the logwatch files older than 10 days, to solve the: CSCvc74307 ISE /var/cache/logwatch temp files are not removed
• Compress existing messages.x files in the system, to solve the: CSCvu45697 Compress messages.x files in the system

You are specifically having issues on Wednesday at 05:00 AM EST, am I correct ?

Hope this helps !!!

[FieldSanitation]

---

@Marcelo Morais wrote:

You are specifically having issues on Wednesday at 05:00 AM EST, am I correct ?

---

Yes, sir.  Trying to get as much information as possible on how much network traffic and how dense it is for about 1-3 minutes.  I really appreciate the insight!  Trying to piece it together as I'm not sure it's publicly available information.

it feels like there's a spike of iSCSI traffic causing some denial of service symptoms - pretty sure that is the read/write test and maybe the log purging...  but hoping for other peoples experience to hear about other things as well.