CSCwk06817 - Known OS Bug causing swap to increase on ISE nodes

[amunozmo]

Linear Swap Memory growth on two ISE nodes cluster having all personas on 3715 appliances. High swap and RAM memory usage, memory leaks in jsvc and Java admin processes, the critical point here for this discussion is that this issue is still happening on ISE version 3.3 patch 6 when the known fixed Release is 3.3 patch 4 according to this bug https://bst.cisco.com/bugsearch/bug/CSCwk06817 

How is possible to still have linear Swap Memory growth on ISE nodes when the defect was fixed on ISE 3.3p4? TAC confirmed evidence of defect CSCwo82042 (LSD MAC handling) in logs but there is no evidence of what is causing swap memory leak and growth. 

[Marcelo Morais]

Hi @MatthewA ,

 at Administration > System > Settings > Profiling, are your MFC Profiling and AI Rules also disabled ?

 If the the answer is No, could you please disable it and check if there is any improvement ?

Hope this helps !!!

[MatthewA]

Good morning Marcelo!

Thanks so much for the response! I have disabled MFC Profiling and AI Rules (didn't realize it was on and don't use it anyway).  No immediate change in tech top but I will be working on it all day today and can watch.  I will enable SXP again later to see if SWAP starts to spiral again. 

[Marcelo Morais]

@MatthewA ,

 excellent ...

 MFC Profiling and AI Rules had issues In ISE 3.3 P2, which were solved in ISE 3.3 P4.

 It's worth a shot to disable it and test it (even if you are on ISE 3.3 P7).

Best regards

[MatthewA]

So far it doesn't seem to be part of the problem.  No change in memory usage outside of a small bump (couple hundred mbs on average).  I re-enabled SXP on one of my servers to see if SWAP continued to spiral and immediately saw my available mem tank and SWAP start rising.

I appreciate checking that though! This has made expanding into TrustSec very difficult.

*for time context that tech top was taken roughly 15min after enabling SXP and has risen an additional 150mbs in typing this so I am re-disabling SXP*

Extra notes that might help others:

TAC and I originally believed the bug to be related to Log Analytics, disabling this stabalized the environment until enabling SXP caused the same behavior.

While typical SWAP usage will delete and release, the SWAP increase seen from enabling these 2 services does not recover on its own and require a restart to clean out.

Disabling/Enabling SXP provides an instant response on SWAP usage (within a minute to see effects)

[Marcelo Morais]

Hi @MatthewA ,

 you said that " ... Disabling/Enabling SXP provides an instant response on SWAP usage (within a minute to see effects) ... ".

 This issue occurs even if you do not have any SXP Configuration (at Work Centers > Trustsec > SXP Devices) ?

Best regards

[MatthewA]

Sorry for the delayed response!

Hey Marcelo,

I currently have 2 devices configured in the SXP devices, SWAP increases with SXP enabled on the switch and disabled from tests I have done so far.  I meet with the TAC engineer again tomorrow so I will delete the devices afterwards to see if it has an impact!

[Marcelo Morais]

Hi @MatthewA ,

 during the last couple of days, I "Enable SXP Service" (no configuration added) in two scenarios:

• 100K+ AuthC per Day
• 1Mi+ AuthC per Day,

both using VMs compatible with SNS 3755 & SNS 3795 ... the MiB Swap remains the same.

I will try to adding a couple of Devices to check if any changes occur.

Regards

[Marcelo Morais]

Hi @MatthewA ,

 during my testing, an ise application stop freed most of the MiB Swap: Used, could you try this in your environment ?

Note 1: a reload solves the MiB Swap: Used. temporarily.

Note 2: I will continue testing ...

Regards