Updates From MC Not Taking On NIDS

ryan.brennan · ‎01-04-2005

We have 2 4235 NIDS devices that are run by a Security Monitor VMS server. Up until version S128 we were able to upgrade both devices through the MC, however this function has stopped working. The subsequent updates (S129 thru S135) appear to have worked, and even running a version report on the MC shows both sensors at S135. However when you manually telnet into each of the two sensors and do a show ver, both sensors are still back on the old S128 code. Any suggestions as to what has failed in the interim, or what service/process could have stopped/failed to stop updates between the Sec Mon and the sensors from happening?

scothrel · ‎01-04-2005

Could you post what version of MC you are running?

Thnx

SC

jason.santamaria · ‎01-05-2005

I have exactly the same symptom. I have one 4235 one 4210 that I have been trying to get updated via IDS MC. While they appear to update to the latest signatures in the MC, telnetting to them or accessing them via IDM shows they are stuck on S123. I have a 4235 in a remote location that seems will show being updated to S91, but has no updates installed when I telnet to it or accessing it via IDM.

IDS MC Versions below:

Apache 1.3.27 12-26-2003 11:02:16 2 ENABLED

Auto Update Server 1.1 11-30-2004 10:58:03 none ENABLED

Client Application Manager 3.0 11-19-2003 16:17:51 none ENABLED

CWCS SQL Components 7.1.3 11-19-2003 16:17:51 none ENABLED

CiscoWorks Common Services with SP2 2.2 12-26-2003 11:02:16 2 ENABLED

Cisco Common Services Help 1.1 11-19-2003 16:17:51 none ENABLED

CWCS Foundation 2.2 12-26-2003 11:02:16 2 ENABLED

CWCS java2 engine 1.2 11-19-2003 16:17:51 none ENABLED

CWCS Web Desktop 2.2 12-26-2003 11:02:16 2 ENABLED

CWCS Utilities 1.1 12-26-2003 11:02:16 1 ENABLED

Database package 4.2 11-19-2003 16:17:52 none ENABLED

CiscoWorks Process Management package 3.5 11-19-2003 16:17:52 none ENABLED

CWCS Event Distribution System 3.2 11-19-2003 16:17:52 none ENABLED

Event Services Software 2.0 12-26-2003 11:02:16 1 ENABLED

Argent Grid classes 1.29 11-19-2003 16:17:52 none ENABLED

IDS MC/Security Monitor Common Framework 1.2 11-30-2004 11:23:49 1 ENABLED

IDS MC 1.2 11-30-2004 11:23:49 1 ENABLED

Security Monitor 1.2 11-30-2004 11:23:49 1 ENABLED

IpSecPole 1.22 11-19-2003 16:17:52 none ENABLED

Java SDK 1.3.1 12-26-2003 11:02:17 1 ENABLED

Jscape widget classes 1.1 11-19-2003 16:17:52 none ENABLED

JChart package 4.0.0.J 4.0 11-19-2003 16:17:52 none ENABLED

JDOM 1.0.7 11-19-2003 16:17:52 none ENABLED

Sun JRE Standard Extensions 1.0 11-19-2003 16:17:52 none ENABLED

Objectspace JGL classes 3.1 11-19-2003 16:17:52 none ENABLED

Jscape powersearch classes 1.1 11-19-2003 16:17:52 none ENABLED

Java Runtime Environment 1.2.2 2.2 11-19-2003 16:17:52 none ENABLED

Job and Resource Management Services 2.1 11-19-2003 16:17:52 none ENABLED

JRUN Servlet Engine 2.3.3 11-19-2003 16:17:52 none ENABLED

Log4j 1.01.03 11-19-2003 16:17:52 none ENABLED

LotusXSL for Java classes 0.16 11-19-2003 16:17:52 3 ENABLED

Application Administration Server 1.1 12-26-2003 11:02:17 2 ENABLED

CWCS Core 1.1 12-26-2003 11:02:17 2 ENABLED

NMCS Network Management Common Services 2.2 12-26-2003 11:02:17 1 ENABLED

nsdb 1.43 11-30-2004 09:30:59 none ENABLED

Perl package 5.00502.1 11-19-2003 16:17:52 none ENABLED

Management Center for Firewalls 1.1 11-30-2004 10:25:51 3 ENABLED

Java Plug-in 1.4.1_02 1.4 12-26-2003 11:02:17 1 ENABLED

Cisco Secure Post Office 1.0196 11-30-2004 09:30:59 none ENABLED

CWCS Help 2.2 12-26-2003 11:02:17 1 ENABLED

Java SNMP 2.6 11-19-2003 16:17:52 none ENABLED

Secure Shell Services 2.2 12-26-2003 11:02:17 1 ENABLED

Java Runtime Environment 1.3.1 11-19-2003 16:17:52 none ENABLED

Syslog, TFTP and RSH services 2.2 11-19-2003 16:17:52 none ENABLED

Sun JFC (Swing) Components 1.1 11-19-2003 16:17:52 none ENABLED

TomCat 3.3 12-26-2003 11:02:17 2 ENABLED

VisiBroker Orb 4.1 11-19-2003 16:17:52 none ENABLED

Web Server package 3.4 12-26-2003 11:02:17 1 ENABLED

Xalan 2.2 11-19-2003 16:17:52 none ENABLED

Xerces 1.5.1 11-19-2003 16:17:52 none ENABLED

IBM XML parser for Java classes 2.0.11 11-19-2003 16:17:52 none ENABLED

RunTime System package 3.2.2 12-26-2003 11:02:17 2 ENABLED

scothrel · ‎01-07-2005

Jason, see my reply further down in the thread. Same suggestions apply. Also, wrt the sensor showing S91. S91 was installed by the update to 4.1.4 from 4.1.3, the full package name is 4.1.4S91. So you get S91 included.

jason.santamaria · ‎01-05-2005

I have exactly the same symptom. I have one 4235 one 4210 that I have been trying to get updated via IDS MC. While they appear to update to the latest signatures in the MC, telnetting to them or accessing them via IDM shows they are stuck on S123. I have a 4235 in a remote location that will show being updated to S91, but has no updates installed when I telnet to it or accessing it via IDM.

IDS MC Versions below:

Apache 1.3.27 12-26-2003 11:02:16 2 ENABLED

Auto Update Server 1.1 11-30-2004 10:58:03 none ENABLED

Client Application Manager 3.0 11-19-2003 16:17:51 none ENABLED

CWCS SQL Components 7.1.3 11-19-2003 16:17:51 none ENABLED

CiscoWorks Common Services with SP2 2.2 12-26-2003 11:02:16 2 ENABLED

Cisco Common Services Help 1.1 11-19-2003 16:17:51 none ENABLED

CWCS Foundation 2.2 12-26-2003 11:02:16 2 ENABLED

CWCS java2 engine 1.2 11-19-2003 16:17:51 none ENABLED

CWCS Web Desktop 2.2 12-26-2003 11:02:16 2 ENABLED

CWCS Utilities 1.1 12-26-2003 11:02:16 1 ENABLED

Database package 4.2 11-19-2003 16:17:52 none ENABLED

CiscoWorks Process Management package 3.5 11-19-2003 16:17:52 none ENABLED

CWCS Event Distribution System 3.2 11-19-2003 16:17:52 none ENABLED

Event Services Software 2.0 12-26-2003 11:02:16 1 ENABLED

Argent Grid classes 1.29 11-19-2003 16:17:52 none ENABLED

IDS MC/Security Monitor Common Framework 1.2 11-30-2004 11:23:49 1 ENABLED

IDS MC 1.2 11-30-2004 11:23:49 1 ENABLED

Security Monitor 1.2 11-30-2004 11:23:49 1 ENABLED

IpSecPole 1.22 11-19-2003 16:17:52 none ENABLED

Java SDK 1.3.1 12-26-2003 11:02:17 1 ENABLED

Jscape widget classes 1.1 11-19-2003 16:17:52 none ENABLED

JChart package 4.0.0.J 4.0 11-19-2003 16:17:52 none ENABLED

JDOM 1.0.7 11-19-2003 16:17:52 none ENABLED

Sun JRE Standard Extensions 1.0 11-19-2003 16:17:52 none ENABLED

Objectspace JGL classes 3.1 11-19-2003 16:17:52 none ENABLED

Jscape powersearch classes 1.1 11-19-2003 16:17:52 none ENABLED

Java Runtime Environment 1.2.2 2.2 11-19-2003 16:17:52 none ENABLED

Job and Resource Management Services 2.1 11-19-2003 16:17:52 none ENABLED

JRUN Servlet Engine 2.3.3 11-19-2003 16:17:52 none ENABLED

Log4j 1.01.03 11-19-2003 16:17:52 none ENABLED

LotusXSL for Java classes 0.16 11-19-2003 16:17:52 3 ENABLED

Application Administration Server 1.1 12-26-2003 11:02:17 2 ENABLED

CWCS Core 1.1 12-26-2003 11:02:17 2 ENABLED

NMCS Network Management Common Services 2.2 12-26-2003 11:02:17 1 ENABLED

nsdb 1.43 11-30-2004 09:30:59 none ENABLED

Perl package 5.00502.1 11-19-2003 16:17:52 none ENABLED

Management Center for Firewalls 1.1 11-30-2004 10:25:51 3 ENABLED

Java Plug-in 1.4.1_02 1.4 12-26-2003 11:02:17 1 ENABLED

Cisco Secure Post Office 1.0196 11-30-2004 09:30:59 none ENABLED

CWCS Help 2.2 12-26-2003 11:02:17 1 ENABLED

Java SNMP 2.6 11-19-2003 16:17:52 none ENABLED

Secure Shell Services 2.2 12-26-2003 11:02:17 1 ENABLED

Java Runtime Environment 1.3.1 11-19-2003 16:17:52 none ENABLED

Syslog, TFTP and RSH services 2.2 11-19-2003 16:17:52 none ENABLED

Sun JFC (Swing) Components 1.1 11-19-2003 16:17:52 none ENABLED

TomCat 3.3 12-26-2003 11:02:17 2 ENABLED

VisiBroker Orb 4.1 11-19-2003 16:17:52 none ENABLED

Web Server package 3.4 12-26-2003 11:02:17 1 ENABLED

Xalan 2.2 11-19-2003 16:17:52 none ENABLED

Xerces 1.5.1 11-19-2003 16:17:52 none ENABLED

IBM XML parser for Java classes 2.0.11 11-19-2003 16:17:52 none ENABLED

RunTime System package 3.2.2 12-26-2003 11:02:17 2 ENABLED

ryan.brennan · ‎01-05-2005

I'm running version 1.2.

Thx.

scothrel · ‎01-05-2005

We have people looking into this from the sensor side and I will also forward the case to the MC folks in case they know something about MC issues. Could the folks on this thread let me know what version of the sensor software they have running?

We are also rechecking the last couple of signature updates to ensure compatibility with MC. This will take us a couple of days, as we had just moved our testing system to the MC 2.0 version.

Scott Cothrell

ryan.brennan · ‎01-05-2005

Per the 'show ver' on one of our sensors:

Cisco Systems Intrusion Detection Sensor, Version 4.1(4)S128

OS Version 2.4.18-5smpbigphys

Platform: IDS-4235

The S128 code is the last time we were to successfully push out the upgrade to our 2 sensors from the MC. We've installed S129 thru S136 (today) and it doesn not go through. When you run a sensor version report from the MC it says that both sensors are up to date. But it is my understanding that when this report is run it does not query the device to find it's version. It simply checks it's own database to see what sensors it has updated, and what release they were updated to.

scothrel · ‎01-07-2005

Ryan,

We re-verified signatures updates from S126 to S136 with no errors on MC ver. 1.2.3. After talking with the sensor and MC folks, we have two suggestions.

First, you can upgrade to 4.1.4f patch level (see http://www.cisco.com/cgi-bin/tablebuild.pl/ids-patches for information). The patch contains modifications to lower memory usage during updates. It is possible that sensors with lower total memory (4210 for example) and/or a lot of signatures enabled can run out of memory during a signature update.

Second, the MC folks suggest that "to troubleshoot the problem [from the MC side] they'll probably have to enable debugging to look at the CLI logs, so the best thing for these customers is to open a TAC case, so that a TAC engineer can walk them through the debug process..."

jason.santamaria · ‎01-07-2005

I have opened a TAC Case, and sent in the mc debugs for analysis. Waiting to hear back on Monday.

ryan.brennan · ‎01-07-2005

Thanks to Jason and Scott for the efforts thus far. I'll wait to see what comes of the TAC case. I'm upgrading to the 4.1(4)f and see what happens. Thanks again.

twilcox · ‎01-18-2005

Scott,

any news on this; we've had a TAC case open for about a month now and so far no results. Is the course of action to upgrade to MC 2.01 right now?

We are on MC 1.2.3 and can't do any upgrades; all our patches are currently. What happpened? Please advise.

Best regards,

Tom

ryan.brennan · ‎01-18-2005

I tried the new certificate route that was suggested here in this thread, as well as elsewhere. This did not work.

When querying the sensor it detects the version it's on (S128), however when you go to update it to the latest sig (S137) it responds as if you've already upgraded it and will not proceed.

Any idea on how to roll back an MC's database that houses the signature version table of the sensors it knows of?

scothrel · ‎01-18-2005

Tom,

Sorry to say that I come at this from the sensor side, MC is not a product I know much about. From what I have heard, upgrading to 2.01 is a good idea, as this looks to be an MC issue. It is my understanding that 2.01 does a better job of keeping the MC in sync with the sensor.

Brainstorming this: You might try setting up a 2.01 MC in parallel with your normal system and importing one of your "problem children sensors" into it to see what you get.

I'll have to reiterate, though, that I'm not an MC guy and I have no idea if what I proposed will screw things up more than they already are. YMMV.

I would also suggest that everyone reading this post, who has a similar situation, should open a TAC case. A flood of similar complaints should clue the TAC into the problem being more than a unique case.

Scott

twilcox · ‎01-07-2005

We are having the exact same problem. But I think ours broke around S132 sig. update.

Queries from IDSMC to the NIDS work but pushes to the NIDS for new sig. updates don't work. Security Monitor is fine. The IDSMC actually recognizes the new sig updates if I do a query which ordinarily won't work if the MC itself hasn't been updated.

So I it's a bit baffling to say the least.