CSCwk06817 - Known OS Bug causing swap to increase on ISE nodes

[amunozmo]

Linear Swap Memory growth on two ISE nodes cluster having all personas on 3715 appliances. High swap and RAM memory usage, memory leaks in jsvc and Java admin processes, the critical point here for this discussion is that this issue is still happening on ISE version 3.3 patch 6 when the known fixed Release is 3.3 patch 4 according to this bug https://bst.cisco.com/bugsearch/bug/CSCwk06817 

How is possible to still have linear Swap Memory growth on ISE nodes when the defect was fixed on ISE 3.3p4? TAC confirmed evidence of defect CSCwo82042 (LSD MAC handling) in logs but there is no evidence of what is causing swap memory leak and growth. 

[Marcelo Morais]

Hi @MatthewA ,

 at Administration > System > Settings > Profiling, are your MFC Profiling and AI Rules also disabled ?

 If the the answer is No, could you please disable it and check if there is any improvement ?

Hope this helps !!!

[MatthewA]

Good morning Marcelo!

Thanks so much for the response! I have disabled MFC Profiling and AI Rules (didn't realize it was on and don't use it anyway).  No immediate change in tech top but I will be working on it all day today and can watch.  I will enable SXP again later to see if SWAP starts to spiral again. 

[Marcelo Morais]

@MatthewA ,

 excellent ...

 MFC Profiling and AI Rules had issues In ISE 3.3 P2, which were solved in ISE 3.3 P4.

 It's worth a shot to disable it and test it (even if you are on ISE 3.3 P7).

Best regards

[MatthewA]

So far it doesn't seem to be part of the problem.  No change in memory usage outside of a small bump (couple hundred mbs on average).  I re-enabled SXP on one of my servers to see if SWAP continued to spiral and immediately saw my available mem tank and SWAP start rising.

I appreciate checking that though! This has made expanding into TrustSec very difficult.

*for time context that tech top was taken roughly 15min after enabling SXP and has risen an additional 150mbs in typing this so I am re-disabling SXP*

Extra notes that might help others:

TAC and I originally believed the bug to be related to Log Analytics, disabling this stabalized the environment until enabling SXP caused the same behavior.

While typical SWAP usage will delete and release, the SWAP increase seen from enabling these 2 services does not recover on its own and require a restart to clean out.

Disabling/Enabling SXP provides an instant response on SWAP usage (within a minute to see effects)