topic Both IPS 4260 is down - Application Failed Critical in Network Security

Both IPS 4260 is down - Application Failed Critical

Rene Rolsted — Sun, 10 Mar 2019 12:30:31 GMT

Tonight was both my 4260 IPS appliance boxes, down.

Health Status for Failed Applications Red

Both IPS appliance have both interfaces down.

Gi eth 2/0 and Gi Eth 2/1 running in Inline-Vlan-Pair

When I reboot one of the boxe, both interface go up, but after 1-2 min they go down.

Now I have put both boxes in bypass mode and then both of their interfaces are up

I suspect that it's the new signature version 6.01 which is to blame.

Is there anyone who has experienced this?

UDVIK01_AWG_IPS# sh ver

Application Partition:

Cisco Intrusion Prevention System, Version 7.0(4)E4

Host:

Realm Keys key1.0

Signature Definition:

Signature Update S601.0 2011-10-10

OS Version: 2.4.30-IDS-smp-bigphys

Platform: IPS-4260-K9

Serial Number:

Licensed, expires: 10-Nov-2011 UTC

Sensor up-time is 2:39.

Using 1892966400 out of 4100345856 bytes of available memory (46% usage)

system is using 17.4M out of 38.5M bytes of available disk space (45% usage)

application-data is using 47.5M out of 166.8M bytes of available disk space (30% usage)

boot is using 41.6M out of 68.6M bytes of available disk space (64% usage)

application-log is using 494.0M out of 513.0M bytes of available disk space (96% usage)

MainApp B-BEAU_704_2010_JUL_21_15_57_7_0_3_29 (Ipsbuild) 2010-07-21T15:59:36-0500 Running

AnalysisEngine B-BEAU_704_2010_JUL_21_15_57_7_0_3_29 (Ipsbuild) 2010-07-21T15:59:36-0500 Running

CollaborationApp B-BEAU_704_2010_JUL_21_15_57_7_0_3_29 (Ipsbuild) 2010-07-21T15:59:36-0500 Running

CLI B-BEAU_704_2010_JUL_21_15_57_7_0_3_29 (Ipsbuild) 2010-07-21T15:59:36-0500

Upgrade History:

* IPS-sig-S600-req-E4 21:15:10 UTC Wed Oct 05 2011

IPS-sig-S601-req-E4.pkg 21:15:10 UTC Tue Oct 11 2011

Recovery Partition Version 1.1 - 7.0(4)E4

Host Certificate Valid from: 01-Nov-2010 to 01-Nov-2012

UDVIK01_AWG_IPS# sh health

Overall Health Status Red

Health Status for Failed Applications Red

Health Status for Signature Updates Green

Health Status for License Key Expiration Yellow

Health Status for Running in Bypass Mode Red

Health Status for Interfaces Being Down Green

Health Status for the Inspection Load Green

Health Status for the Time Since Last Event Retrieval Green

Health Status for the Number of Missed Packets Green

Health Status for the Memory Usage Green

Health Status for Global Correlation Green

Health Status for Network Participation Green

Security Status for Virtual Sensor IDS-UDVIK Green

Security Status for Virtual Sensor IPS-Udvik Green

Security Status for Virtual Sensor vs0 Green

UDVIK02_AWG_IPS# sh ver

Application Partition:

Cisco Intrusion Prevention System, Version 7.0(4)E4

Host:

Realm Keys key1.0

Signature Definition:

Signature Update S601.0 2011-10-10

OS Version: 2.4.30-IDS-smp-bigphys

Platform: IPS-4260-K9

Serial Number:

Licensed, expires: 10-Nov-2011 UTC

Sensor up-time is 3:47.

Using 1893654528 out of 4100345856 bytes of available memory (46% usage)

system is using 17.4M out of 38.5M bytes of available disk space (45% usage)

application-data is using 47.6M out of 166.8M bytes of available disk space (30% usage)

boot is using 41.6M out of 68.6M bytes of available disk space (64% usage)

application-log is using 494.0M out of 513.0M bytes of available disk space (96% usage)

MainApp B-BEAU_704_2010_JUL_21_15_57_7_0_3_29 (Ipsbuild) 2010-07-21T15:59:36-0500 Running

AnalysisEngine B-BEAU_704_2010_JUL_21_15_57_7_0_3_29 (Ipsbuild) 2010-07-21T15:59:36-0500 Running

CollaborationApp B-BEAU_704_2010_JUL_21_15_57_7_0_3_29 (Ipsbuild) 2010-07-21T15:59:36-0500 Running

CLI B-BEAU_704_2010_JUL_21_15_57_7_0_3_29 (Ipsbuild) 2010-07-21T15:59:36-0500

Upgrade History:

* IPS-sig-S600-req-E4 21:45:10 UTC Wed Oct 05 2011

IPS-sig-S601-req-E4.pkg 21:45:10 UTC Tue Oct 11 2011

Recovery Partition Version 1.1 - 7.0(4)E4

Host Certificate Valid from: 01-Nov-2010 to 01-Nov-2012

UDVIK02_AWG_IPS# sh health

Overall Health Status Red

Health Status for Failed Applications Red

Health Status for Signature Updates Green

Health Status for License Key Expiration Yellow

Health Status for Running in Bypass Mode Red

Health Status for Interfaces Being Down Green

Health Status for the Inspection Load Green

Health Status for the Time Since Last Event Retrieval Green

Health Status for the Number of Missed Packets Green

Health Status for the Memory Usage Green

Health Status for Global Correlation Green

Health Status for Network Participation Green

Security Status for Virtual Sensor IDS-Udvik Green

Security Status for Virtual Sensor IPS-Udvik Green

Security Status for Virtual Sensor vs0 Green

Both IPS 4260 is down - Application Failed Critical

whaydenscana — Wed, 12 Oct 2011 12:50:33 GMT

I am having the same problem. I so far have 4 IPS down. They all crashed after 601 signature update. If you hadn't updated your signatures don't and turn off auto update.

I have two open TAC cases about this issue.

Also upgrading/downgrading isn't working right now....just freezes.....

this is bad.

Both IPS 4260 is down - Application Failed Critical

mik.gustafsson — Wed, 12 Oct 2011 13:10:42 GMT

Reset worked for me. Im on my last appliance now, 6 in total, and all are up and running again.

Both IPS 4260 is down - Application Failed Critical

Rene Rolsted — Wed, 12 Oct 2011 13:13:12 GMT

Hi William,

Okay, I'm waiting for Cisco comes up with a solution.

I have contacted Cisco Denmark with the error.

I had set myself to do a downgrade of the signature, but I wait.

thanks for your reply

Both IPS 4260 is down - Application Failed Critical

Rene Rolsted — Wed, 12 Oct 2011 13:16:24 GMT

Hi Mikael,

I did a reset in CLI this morning

but this is not working for me

Both IPS 4260 is down - Application Failed Critical

mik.gustafsson — Wed, 12 Oct 2011 13:42:13 GMT

Hi Rene,

All but one did come back to normal after a reset.

So not a 100% solution unfortunately.

Both IPS 4260 is down - Application Failed Critical

hartkl5277 — Wed, 12 Oct 2011 14:55:16 GMT

Having the same issue with S601, reset did not work on either unit. Glad I only updated in the non-production environment. Hope a fix comes out soon.

Both IPS 4260 is down - Application Failed Critical

whaydenscana — Wed, 12 Oct 2011 15:21:58 GMT

Cisco respone..it is a bug.

Hi William,

I have got a response from the collaboration team. It seems that there is a bug , we have the following response from the developers

“Due to a bug in IPS software versions prior to 7.0.6, it is not advised to apply signature updates past S601 without having upgraded to 7.0.6 first.If a customer has applied S601 with no problems, there is no issue beyond having to upgrade to 7.0.6 to apply updates S602 and later. If you have not applied S601 yet, it is advised to upgrade to 7.0.6 before applying update S602 and later.”

I am still finding some more details and would get back to you.

I am currently downgrading and having success.

Downgrade from console

....unplug monitor port (prevent engine from running) .... reset IPS.....downgrade from command line to 600....go to gui turn off auto update....plug ports back in.

Downgrade from ssh

shutdown monitor port (prevent engine from running) .... reset IPS.....downgrade from command line to 600....go to gui turn off auto update....plug ports back in.

Thanks,

Will

Both IPS 4260 is down - Application Failed Critical

rrfield — Wed, 12 Oct 2011 15:43:36 GMT

In our environment this is limited to our single 4260 sensor. We have multiple SSM-10 modules in ASA's and they are all functioning properly.

Is this the case for others as well?

Both IPS 4260 is down - Application Failed Critical

whaydenscana — Wed, 12 Oct 2011 16:34:10 GMT

I am only working with 4240s here. I am not familar with the modules in the ASA.

Both IPS 4260 is down - Application Failed Critical

hoylea — Wed, 12 Oct 2011 17:23:35 GMT

Same issue here, on both sensors and modules, though not on EVERY device.

Both IPS 4260 is down - Application Failed Critical

mulhollandm — Wed, 12 Oct 2011 20:00:40 GMT

folks

aip-ssm-20 same problem

if i rest it runs ok for a day or so then stops logging though curiously when i reset the ips module my asa cluster fails over

didn't expect that

i'm running in promiscuous mode but i think i'll upgrade

currently running

Cisco Intrusion Prevention System, Version 7.0(2)E4

Host:
    Realm Keys          key1.0
Signature Definition:
    Signature Update    S601.0                   2011-10-10

Upgrade History:

* IPS-sig-S600-req-E4 17:37:23 UTC Wed Oct 05 2011
IPS-sig-S601-req-E4.pkg 18:37:09 UTC Tue Oct 11 2011

i'll update when i upgrade and get a signature update

Both IPS 4260 is down - Application Failed Critical

ABDUL MAJID KHAN — Sat, 12 Nov 2011 12:47:29 GMT

Folks,

This is a known BUG with ID (CSCti79423) in the version 7.0 (4) E4 that will be caused once the global correlation is configured. It is fixed in 7.0 (6) E4.

For details find the following URL;

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCti79423

Workaround:

Disable global correlation:

conf t

service global-correlation

global-correlation-inspection off

exit and accept changes

Restart the IDSM-2