cidserver (new IDS 3.1 feature) not answering on 443

stbob · ‎05-14-2002

Upgraded to IDS 3.1 today, seems to have gone well, except I'm trying to use the new cidserver (IDM) feature. ps -ef shows it's running, I can start it but not stop it (cidServer start works but cidServer stop doesn't kill the server, (apparently because it's issuing a kill -TERM instead of a kill -9?).

Even while cidserver -d is running (shown by ps -ef) netstat -a does not show port 443 among its listening ports, and I cannot browse to it. The docs I found seem to imply that it's configured and enabled by default, but is there something else I need to do to make it function?

ED CRAIG · ‎05-14-2002

same here..

I also "re-imaged" the sensor,

clean install of 3.1 but still no answer on 443.

todriscoll · ‎05-15-2002

I don't know if this will help but I had exactly the same problems with my first attempt to go to 3.1. What I discovered was that the first download of the SP image from CCO (IDSk9-sp-3.1-1-S22.bin) wasn't the correct size (should be about 107MB but it was only about 80MB after the download completed). This was most likely a problem with the download rather than a problem with the binary posted on CCO. It allowed me to install the "incorrect" image but the sensor hung on the creation of the web certificates and never started up 443 either. I imagine those 30MB or so of data are kind of important :-). I just downloaded a fresh image (which was then the correct size), installed it and things have worked great since. After a clean installation of the 3.1.1 SP, port 443 does appear as "listening" with no additional configuration needed.

This was on a 4210 but I doubt the model matters...

Hope that helps!

stbob · ‎05-15-2002

I thought that might have been my problem but I reinstalled a fresh tested version (correct size, correct sum) and still have no 443.

todriscoll · ‎05-15-2002

Thats definitely strange. As long as you see option "11" when in sysconfig-sensor (for IDS Device Manager) and its enabled, which is the default anyway, I would expect 443 to be listening. Maybe one of the Cisco guys has other things to try...

marcabal · ‎05-15-2002

We are trying to track down this issue.

We did not run across it in either internal testing or Beta.

A first guess is that there may be something in your configuration that the webserver isn't accounting for.

Can you tar up your /usr/nr/etc directory and send it to me as well as:

1) output you receive when executing "cidServer start"

2) output of "cidServer version"

3) output of "ps -ef | grep web"

4) output of yor netstat command withouth the 443 port open

marco

marcabal · ‎05-15-2002

Good News/Bad News

We were able to replicate the problem at one of our internal sites.

What I found is that the 3.1 installation never completed.

The final stages of the 3.1 installation will prepare all of the files necessary for running the web server. So if the installation ended prematurely then those changes were never completed.

How to tell if 3.1 installation had completed:

Look at the bottom of the output.log file in the /us/nr/sp-update directory.

The bottom of the file should read:

ids-postpatch: IDSk9-sp-3.1-1-S22.bin has been successfully installed.

Warning! Your system will begin shutdown in 30 seconds!!!

Press to quit if you do not wish to reboot!

..............................

Shutting down now!

What if I now know that 3.1 did not install correctly?

You can try to uninstall 3.1 (the installation may have completed enough to allow an uninstall).

If the uninstall works then verify that you have at least 300000KB in /usr prior to re-installing 3.1(1).

If the uninstall doesn't work, you will need to re-image from CD.

Marco

stbob · ‎05-16-2002

Still no love for me. netstat, output.log, ps -ef output below.

# netstat -a

UDP: IPv4

Local Address Remote Address State

-------------------- -------------------- -------

*.syslog Idle

*.45000 Idle

*.514 Idle

*.* Unbound

TCP: IPv4

Local Address Remote Address Swind Send-Q Rwind Recv-Q State

-------------------- -------------------- ----- ------ ----- ------ -------

*.* *.* 0 0 24576 0 IDLE

*.ftp *.* 0 0 24576 0 LISTEN

*.telnet *.* 0 0 24576 0 LISTEN

*.22 *.* 0 0 24576 0 LISTEN

*.* *.* 0 0 24576 0 IDLE

sensor.32859 admin-4.potomacnet.com.telnet 3548 0 24820 1 ESTABLISHED

sensor.32863 admin-3.potomacnet.com.telnet 2987 0 24820 0 ESTABLISHED

sensor.telnet admin-10.potomacnet.com.61504 64230 1 24820 0 ESTABLISHED

*.* *.* 0 0 24576 0 IDLE

TCP: IPv6

Local Address Remote Address Swind Send-Q Rwind Recv-Q State If

--------------------------------- --------------------------------- ----- ------ ----- ------ ----------- -----

*.* *.* 0 0 24576 0 IDLE

*.ftp *.* 0 0 24576 0 LISTEN

*.telnet *.* 0 0 24576 0 LISTEN

*.22 *.* 0 0 24576 0 LISTEN

# tail output.log

Adding signature: SigOfStringMatch 0 to /usr/nr/etc/packetd.conf.

Adding signature: SigOfTcpPacket 0 to /usr/nr/etc/packetd.conf.

Adding signature: SigOfUdpPacket 0 to /usr/nr/etc/packetd.conf.

ids-postpatch: IDSk9-sp-3.1-1-S22.bin has been successfully installed.

Warning! Your system will begin shutdown in 30 seconds!!!

Press to quit if you do not wish to reboot!

..............................

Shutting down now!

# ps -ef | grep cid

root 15188 1 0 16:25:57 ? 0:09 /usr/nr/idsRoot/bin/cidwebserver -d

root 27070 27034 0 11:50:51 pts/1 0:00 grep cid

brenden · ‎05-16-2002

One of my sensors updated just fine.

I did notice that the Patches on that sensor didn't error out with Error Code 5.

I decided to try one more sensor to upgrade. This sensor showed a lot of error code 5's -- and I cannot access the IDS Device Manager from the website...

marcabal · ‎05-16-2002

On the one sensor we have been able to diagnose this problem, the installation of 3.1 never completed.

If you are experiencing this problem of hte web server not responding then please try the tips listed below.

If the web server still does not respond then:

As root, please run /usr/nr/idsRoot/bin/cidDump and send me the log - /usr/nr/idsRoot/htdocs/private/cidDump.html

And the contents of the /usr/nr/sp-update/output.log file.

Troubleshooting tips:

1) Run "cidServer version" as user root

# cidServer version

cidwebserver v33 (Release) 02/04/26-01:32

cidwebserver (27394) is running.

2) Run sysconfig-sensor option 11 to ensure IDM is enabled

IDS Device Manager

Current Mode: Enabled

1 - Disable

x - Exit

Selection:

3) Attempt to telnet to the sensor from the same machine that the web browser is running from

Telnet and web connections are both restricted by option 5 of sysconfig-sensor above.

4) Be sure the user is using a supported web browser:

http://www.cisco.com/univercd/cc/td/doc/product/iaabu/csids/csids8/13876_01.htm#xtocid5

5) Be sure that the web browser is configured to accept cookies:

http://www.cisco.com/univercd/cc/td/doc/product/iaabu/csids/csids8/13876_01.htm#40768

6) Be sure the user is typing "https://sensoripaddress" the "s" on the end of the http is very important.

7) In a worst case you can log in as root and execute:

snoop -d port 443

And see if the connection is being established.

It could be that a firewall or router may be blocking either 443 or port 80 traffic to the sensor, in which case the user would need to change their firewall or router config.

8) You could also try the following:

a) Login as root

b) cidServer stop

c) cd /usr/nr/idsRoot/etc

d) cp cidwebserver.conf cidwebserver.conf.bak

e) vi cidwebserver.conf

f) within vi change the "ports=443" to 80

g) cidServer start

h) Now try to connect to the sensor using https://ipaddress

9) Execute cat /usr/nr/sp-update/output.log

You should see the following lines if the installation completed successfully:

ids-postpatch: IDSk9-sp-3.1-1-S22.bin has been successfully installed.

Warning! Your system will begin shutdown in 30 seconds!!!

Press to quit if you do not wish to reboot!

..............................

Shutting down now!

brenden · ‎05-16-2002

marco:

Did all of that, All results were there..

However I looked at the output log of the working sensor (for webserver) vs. the non-working sensor (webserver).

The working sensor with the open 443 port had no errors for the Solaris patches.

The non-working sensor did..: Look at this log:

ids-postpatch: Adjusting and updating OS files

Patch cluster install script for IDS Patch Update v2.8.3

Determining if sufficient save space exists...

Sufficient save space exists, continuing...

Installing patches located in /usr/nr/33-update/SEN31SP/install/u

Using patch_order file for patch installation sequence

112397-02 already installed.

108988-09 already installed.

108876-12 already installed.

Installing 110284-05...

Installation of 110284-05 failed. Return code 5.

110663-07 already installed.

Installing 110952-02...

Installation of 110952-02 failed. Return code 5.

109899-05 already installed.

Installing 110899-04...

Installation of 110899-04 failed. Return code 5.

110902-01 already installed.

Installing 110459-02...

Installation of 110459-02 failed. Return code 5.

110076-01 already installed.

Installing 109092-05...

Installation of 109092-05 failed. Return code 5.

111233-01 already installed.

111235-01 already installed.

111112-03 already installed.

111334-01 already installed.

Installing 110402-03...

Installation of 110402-03 failed. Return code 5.

Installing 109327-07...

Installation of 109327-07 failed. Return code 5.

111326-01 already installed.

110323-01 already installed.

110701-01 already installed.

109325-04 already installed.

Installing 110946-05...

Installation of 110946-05 failed. Return code 5.

111505-01 already installed.

Installing 110917-02...

Installation of 110917-02 failed. Return code 5.

111607-02 already installed.

111070-01 already installed.

111827-01 already installed.

Installing 110616-04...

Installation of 110616-04 failed. Return code 25.

Installing 109148-15...

Installation of 109148-15 failed. Return code 5.

111875-05 already installed.

Installing 109278-02...

Installation of 109278-02 failed. Return code 5.

Installing 111660-06...

Installation of 111660-06 failed. Return code 5.

Installing 110904-04...

Installation of 110904-04 failed. Return code 5.

Installing 109668-04...

Installation of 109668-04 failed. Return code 5.

110958-02 already installed.

112219-01 already installed.

108902-04 already installed.

111086-02 already installed.

112326-01 already installed.

110669-03 already installed.

112255-01 already installed.

112460-01 already installed.

Installing 112238-02...

Installation of 112238-02 failed. Return code 5.

Installing 111294-03...

Installation of 111294-03 failed. Return code 5.

Installing 111307-03...

Installation of 111307-03 failed. Return code 5.

111311-01 already installed.

Installing 111328-04...

Installation of 111328-04 failed. Return code 25.

Installing 111099-01...

Installation of 111099-01 failed. Return code 5.

110403-04 already installed.

108969-05 already installed.

108976-06 already installed.

Installing 108986-03...

Installation of 108986-03 failed. Return code 25.

Installing 110935-07...

Installation of 110935-07 failed. Return code 5.

Installing 108529-14...

Installation of 108529-14 failed. Return code 25.

Installing 108726-07...

Installation of 108726-07 failed. Return code 25.

108728-14 already installed.

Installing 108990-02...

Installation of 108990-02 failed. Return code 5.

Installing 108828-21...

Installation of 108828-21 failed. Return code 25.

Installing 108994-07...

Installation of 108994-07 failed. Return code 25.

Installing 108998-03...

Installation of 108998-03 failed. Return code 5.

Installing 109239-02...

Installation of 109239-02 failed. Return code 25.

112139-01 already installed.

Installing 109319-27...

Installation of 109319-27 failed. Return code 25.

Installing 109008-07...

Installation of 109008-07 failed. Return code 25.

The following patches were not able to be installed:

110284-05

110952-02

110899-04

110459-02

109092-05

110402-03

109327-07

110946-05

110917-02

110616-04

109148-15

109278-02

111660-06

110904-04

109668-04

112238-02

111294-03

111307-03

111328-04

111099-01

108986-03

110935-07

108529-14

108726-07

108990-02

108828-21

108994-07

108998-03

109239-02

109319-27

109008-07

For more installation messages refer to the installation logfile:

/var/sadm/install_data/IDS_Patch_Update_v2.8.3_log

Use '/usr/bin/showrev -p' to verify installed patch-ids.

Error 5 according to Sun Docs is:

Error Code 5:

The -p parameter must be a directory. $uPATCHDIR is not a directory.

Explanation and recommended action: You selected the -p option and supplied a path that is not a valid directory. Reinvoke install_mu (or backout_mu) with a valid path to the -p option.

and Error 25 is:

Error Code 25:

The -f and -D options are mutually exclusive.

Explanation and recommended action: The -f option instructs install_mu to skip the dry-run disk space calculation phase. The -D option requests that only the dry-run calculations be made. Choose one option, but not both.

Let me know what engineers think.. >?

Brenden

marcabal · ‎05-16-2002

It is looking like certain patches not being installed properly are the cause of some of the web server issues being seen. Several pathces were added in the release which are required for the web server to function properly.

If you don't see the Installation Completed message then most likely several patches were not installed, but even if you are seeing the Installation Completed message it looks like some of the patches still weren't installed.

We are wokring on determining the cause for the patch install errors and trying to remedy the problem.

stbob · ‎05-21-2002

FYI, the 3.1 sp 23 that was released over the weekend solved this problem for me.