Re: Signature Update S601 Pulled From Cisco.com

nicksmi · ‎10-12-2011

Due to a bug in IPS software versions prior to 7.0.6, it is not advised to apply signature updates past S601 without having upgraded to 7.0.6 first. This issue is being tracked as CSCtn23051. If a customer has applied S601 without problems, there is no issue beyond having to upgrade to 7.0.6 to apply updates S602 and later. If you have not applied S601 yet, it is advised to upgrade to 7.0.6 before applying update S602 and later.

Thank you for your patience and understanding in this matter.

Nicholas Smith

Cisco IPS Signature Team

hartkl5277 · ‎10-12-2011

Is there a fix for the systems that have the S601 applied and not responding?

nicksmi · ‎10-12-2011

Upgrade to 7.0(6) or 7.1. Update 6.2.4 also resolves the issue.

hoyleanderson · ‎10-12-2011

This is unacceptable. I've attempted to upgrade 4 devices to 7.0.6 today and three have failed.

You released signature update that only works on the latest code

...which has only been released for a month

... which automatically applies to all devices in the field

... without any ability to remove short of a reimage

... without sufficient (any?) testing on other versions

What if you released a 602 sig file that is actually 600? Would that roll back? If so, do that and release 601 as 603 when it's actually been tested!

EDIT: I have found that rebooting before attempting the upgrade seems to increase the chance of success.

nicksmi · ‎10-12-2011

Mr. Anderson,

We apologize for the problems you have been experiencing with our sensors. The root cause is a memory segmentation problem that is fixed in the 7.0(6) software release and cannot be reliably fixed via a signature update. Signature update S601 merely brought us to the threshold where the problem manifests itself more frequently. Our updates are tested and due to the nature of this problem its occurrence is fairly random and we did not observe it. We apologize for the problems this has caused and are reviewing our internal processes to avoid a reoccurrence.

We recomend that customers be at the latest signature update level to provide the max protection, and it is important to be at the latest code level as well to ensure reliable and effective functionality.

At this time, we are asking you to open a TAC case to document your upgrade woes and resolve them.

http://www.cisco.com/cisco/web/support/index.html

Thank you for your patience in this matter.

Nicholas Smith

Cisco IPS Signature Team

whaydenscana · ‎10-12-2011

Hoyle,

I had the same problem, this is what I did to get it downgraded.

---------

Downgrade from console---

....unplug monitor port (prevent engine from running) .... reset IPS.....downgrade from command line to 600....go to gui turn off auto update....plug ports back in.

Downgrade from ssh---

shutdown monitor port (prevent engine from running) .... reset IPS.....downgrade from command line to 600....go to gui turn off auto update....plug ports back in.

At this point you should be able initiate the system upgrade.

Thanks,

Will

Costin Vilcu · ‎10-13-2011

Nick, you said "Update 6.2.4 also resolves the issue".

Does that mean that also the 6.0.x, 6.1.x and 6.2.[1,2,3] are affected by this bug?

Thank you,

C.

nicksmi · ‎10-13-2011

I belive that is correct. We reccomend that customers be at the latest sig update level to provide the max protection, and it is important to be at the latest code level as well to ensure reliable and effective functionality.

twilcox · ‎10-13-2011

What is the reccommended course of action for those of us that have AIP-SSM's inside our firewalls?

Question: should we wait until 602 is released? and will the automatic upgrade work for them i.e. will they automatically update to 602?

I cannot upgrade any of our AIP-SSM's currently running 7.0.4 (E4) to the current release via the GUI either with ASDM or with IEV. This is pretty bad.

I would like to avoid downtime since they are inside redundant fw pairs. Upgrades of the AIP-SSM typically require a FW failover when upgrading, so waiting for a sig update to 602 is preferable if it will automatically update. Otherwise I need to know if this will not work in order to schedule downtime / change control to get this fixed.

From a customer support standpoint, if this was a know issue which it evidently was ... Cisco needs to tell customers beforehand to upgrade to the most recent code rev.

The fact that a sig update can kill off the IPS inside a FW is pretty lame. It also shows the lack of regression testing in your software process. You need to fix that but I guess since you guys outsource so much to foreign countries, I'm not surprised when stuff like this happens. I've learned how to suffer being a Cisco customer.

nicksmi · ‎10-13-2011

Upgrading to S602 will not solve the problem.

Signature Update S601 aggravated a bug (CSCtn23051) existing in some older versions of the sensor software.

To resolve this issue, upgrade the sensor software to version 6.2(4) (released June 2011) or 7.0(5) (released May 2011) or greater (7.0(6) (released Sept 2011) is recommended). Sensors running 7.1(x) software are not affected.

A signature update will not resolve the problem. While the problem may not immediately manifest itself with future signature updates, unless the sensor is updated, the problem can occur, depending on the sensor configuration, the traffic being inspected, and the state of the memory.

For the best protection it is critical to apply the appropriate service packs to the sensor to ensure reliable and accurate operation.

hartkl5277 · ‎10-13-2011

Is the issue with a specific signature being loaded or something else in the package? I was thinking that if is just a signature that you could use the command line to retire the signature so that it is not loaded and then reset the system. Any thoughts or feedback? Seems like S602 is taking a long time to get released, going through a better QA process I hope.

nicksmi · ‎10-13-2011

The root issue is with the sensor software (CSCtn23051) and was addressed a number of months ago in the 7.0(5) SP. We reccomend upgrading to 7.0(6) due to other issues with 7.0(5).

S601 happens to fairly reliably reproduce the memory condition that triggers the bug. The only way to reliably fix the problem is to upgrade the sensor software.

hoyleanderson · ‎10-13-2011

Nicholas,

Thank you for taking the time to respond to this thread.

H

nicksmi · ‎10-13-2011

UPDATE:

Signature Update S601 aggravated a bug (CSCtn23051) existing in some older versions of the sensor software.

To resolve this issue, upgrade the sensor software to version 6.2(4) (released June 2011) or 7.0(5) (released May 2011) or greater (7.0(6) (released Sept 2011) is recommended). Sensors running 7.1(x) software are not affected.

A signature update will not resolve the problem. While the problem may not immediately manifest itself with future signature updates, unless the sensor is updated, the problem can occur, depending on the sensor configuration, the traffic being inspected, and the state of the memory.

For the best protection it is critical to apply the appropriate service packs to the sensor to ensure reliable and accurate operation.

yantiscompany · ‎10-14-2011

Wow, what a $#%# up. Glad I had my module set to bypass, or I would have been super pissed.

Here is what worked for me without any down time, since I'm a little lost without my precious GUI:

Putty into your ASA, and do "sh module", and you should see your ips module as 1. Run command "hw-module module 1 reset" to reset your ips module. I tried running the update without restarting the module, and it just got hung up until I restarted it. Download the upgrade 7.0(6) .pkg from cisco. After module is backup, telnet to the module. Run upgrade as stated in this article "http://www.cisco.com/en/US/docs/security/ips/6.2/configuration/guide/cli/cli_system_images.html#wp1142504 ". After module is backup you should be good to go. Module takes a while to restart, so be patient.

Not a pro, so please correct me if any of the above is wrong.