IDS Signature S140 Hung IDS-4235 and IDS-4215 (AGAIN!)

[lbhoang]

I fear applying signature updates to our Cisco IDS' since each update is like playing Russian roulette. I just had an IDS-4235 RMA'd because it was resetting on its own several times a day. So, when I got the replacement I updated with signature S91, applied 4.1(4e) patch then the latest signature update S137. That ran for about a week then S138 came out. I went through the standard CLI ftp upgrade but on the IDS-4235 that just got RMA'd the upgrade process hung. When I try to open another SSH session it would report "Error: Cannot communicate with system processes. Please contact your system administrator." I can telnet in and reset the IDS but after it comes back up it's hosed and I can't manage it at all. I had to go through the whole reimaging, repatching and restoring config process. So, when S139 came out I was very very hestitant to upgrade and it's a good thing I put it off since S140 came out the same day. Today, I had the courage to do the upgrade but, unsurprisingly, S140 hung an IDS-4235 and also IDS-4215. Sigh...

I find out there's a 4.1(4f) patch but after reading the release notes I don't see any mention of fixing this upgrade hanging problem. It would be nice if customers are notified of patches in addition to signature updates.

On a different note, does anyone have any experience with www.sourcefire.com commercial Snort IDS appliance?

[lbhoang]

Correction: I originally said when the IDS is hung during the upgrade process I can't SSH but can telnet. I was wrong. Telnet also returns the same message "Error: Cannot communicate with system processes. Please contact your system administrator." If these two doesn't finish upgrading by tomorrow morning I'm going to have to call our Cisco account manager for help since I get no response/fix whenever I contact Cisco TAC.

[craiwill]

If you are not running the 'f' patch on your sensors, 4.1.4(f), you should download and install that patch. It fixes some out-of-memory on upgrade issues that are most likely the cause of your problem.

See http://www.cisco.com/cgi-bin/tablebuild.pl/ids-patches for patch information

[lbhoang]

Craig, thanks for the suggestion on applying 4.1.4(f) patch. That's the impression I get scouring this forum.

I just checked on the IDS' and they're still in lala land so my morning will be preoccupied with restoring these. Assuming that I'm able to, should I attempt to downgrade and reapply signature S140? Or, should I play it safe and reimage the two IDS'?

I used to go through a ritual of disabling the span port for the sensing interface to stop traffic, resetting the IDS to clear memory, apply the signature update, reset and reenable span port. This was about a year ago when we first converted from SunOS 3.x to Linux 4.x and starting having the issue with the IDS appliances locking us out of the management interfaces (telnet/SSH/https). I think I may have to revisit that ritual. It'll take an additional 15 minutes per IDS but it'll, hopefully, prevent downtime and save me the work of having to reimage these appliances.

[craiwill]

I'm not sure if I understand your question, if the sensor is responding you may want to apply the patch without restoring the sensor. Restoring the sensor is essentially the same as a re-image only the sensor’s network configuration files are not overwritten. You may need to restore the sensor in order to apply the patch. Unless the sensor is oversubscribed this patch should solve all memory issues.