topic Re: IPS 4255 in software bypass in Network Security

IPS 4255 in software bypass

dragnia_s — Sun, 10 Mar 2019 11:23:42 GMT

Hi,

I have an IPS 4255 it is used to inspect traffic to the internet. it is used after the firewall. The internet traffic is around 40 Mbps.

When i inspect trafic the procesor is around 90 percent and the inspection load is 30 percent. after about a day, the virtual sensor is in bypass.

The sensor is suposed to manage 500 Mbps.

Any ideeas what is causing this?

Thank you!

Here is the output from the log i think it helps:

Messages, like this one, in the category - SensorApp status timed out - were logged 397 times in the last 3600 seconds.

13Nov2008 05:47:26.084 3600.961 monitor[386] Monitor/W last update time=1226479551133094000

Messages, like this one, in the category - SensorApp status timed out - were logged 397 times in the last 3600 seconds.

13Nov2008 06:20:59.210 2013.126 mainApp[325] Cid/E errSystemError AppManager::ApplicationEntry::updateProcessStatus - Application "AnalysisEngine "terminated prematurely

================================= END OF FILE ==================================

Re: IPS 4255 in software bypass

attmidsteam — Fri, 28 Nov 2008 15:00:45 GMT

Please provide a 'sh ver' but if this sensor is running 6.0.5(E3) or 6.1.1(E3) you may be encountering the E3 runtime bug (CSCsv66660). It commonly occurs when traffic slows (like at night).. to confirm this, login as a service account, navigate to /usr/cids/idsRoot/core/sensorApp and paste the contents of core.txt (if there is one). The first line will look like this:

0x0826128a +/usr/cids/idsRoot/bin/sensorApp(_ZN3Cid6Sensor6Common18AutoExpireHashTreeIyLj17EE6expireEPNS0_4Proc13TimeProcessor9TimeEventEPvPNS0_11CsidsBufferE+0xda) [0x826128a];

There is no _official_ patch at this time but if you contact TAC, you can probably get a devel binary of sensorApp to use until the real patch comes out.

Re: IPS 4255 in software bypass

dragnia_s — Fri, 28 Nov 2008 15:47:21 GMT

Hi,

Thanks for the answer, the bug fits the problem but I haven't got a file in the /usr/cids/idsRoot/core/sensorApp. could it be deleted after restart?

this problem existed when it was using 6.1.1 E2, I thought upgrading to E3 would fix the problem.

Do you know how should the CPU/inspection load look? Because this sensors CPU is at 90 percent when the inspection load is just 30.

Thanks

Re: IPS 4255 in software bypass

attmidsteam — Fri, 28 Nov 2008 16:00:50 GMT

I'm running all 6.0 but I don't think 6.1 will delete a core file. There can be sensorApp crashes without a core, how often does this crash? For me, the E3 runtime bug would kill a sensor almost daily; note that this problem appears to have been introduced in E3. I have various other bugs affecting E2 (including a config time crash), SMB-A crashes, and low-end sensors crashing due to out of memory. Most of those will produce a core dump however.

Your issue sounds similar but I would have expected a core dump. Run 'ls -lR /usr/cids/idsRoot/core' to see if any of the other engines have produced one. If you run 'vi /usr/cids/idsRoot/log/main.log', then search (/) for '===='. This will take you to the end of the log, paste the last 5-10 lines. If this sensor is crashing often enough, I would get a TAC opened on it as well.

Re: IPS 4255 in software bypass

dragnia_s — Fri, 28 Nov 2008 16:37:41 GMT

This is the firs output:bash-2.05b#

-bash-2.05b# ls -lR /usr/cids/idsRoot/core/

/usr/cids/idsRoot/core/:

drwxrwx--- 2 cisco cids 1024 Nov 11 10:29 cidcli

drwxrwx--- 2 cids cids 1024 Nov 11 10:27 cidsInit

drwxrwx--- 2 cids cids 1024 Nov 11 10:27 mainApp

drwxrwx--- 2 cids cids 1024 Nov 26 07:44 sensorApp

/usr/cids/idsRoot/core/cidcli:

/usr/cids/idsRoot/core/cidsInit:

/usr/cids/idsRoot/core/mainApp:

/usr/cids/idsRoot/core/sensorApp:

-bash-2.05b#

This is the log I saw after it crashed :

13Nov2008 03:47:24.203 3600.911 monitor[386] Monitor/W last update time=1226479551133094000

Messages, like this one, in the category - SensorApp status timed out - were logged 397 times in the last 3600 seconds.

13Nov2008 04:47:25.123 3600.920 monitor[386] Monitor/W last update time=1226479551133094000

Messages, like this one, in the category - SensorApp status timed out - were logged 397 times in the last 3600 seconds.

13Nov2008 05:47:26.084 3600.961 monitor[386] Monitor/W last update time=1226479551133094000

Messages, like this one, in the category - SensorApp status timed out - were logged 397 times in the last 3600 seconds.

13Nov2008 06:20:59.210 2013.126 mainApp[325] Cid/E errSystemError AppManager::ApplicationEntry::updateProcessStatus - Application "AnalysisEngine "terminated prematurely

13Nov2008 07:42:46.590 4907.380 cidwebserver[380] Cid/W errTransport WebSession::sessionTask(7) socket exception: socket error [3,32]

Messages, like this one, in the category - TCP socket failure - were logged 1 times in the last 0 seconds.

13Nov2008 09:58:23.329 8136.739 -cidcli[22279] Cid/W errWarning Session was closed by the remote end.

13Nov2008 10:36:10.332 2267.003 logApp[340] Cid/W Disabled status event logging for status category: controlTransaction

13Nov2008 12:36:10.704 7200.372 -cidcli[10641] Cid/D doCtrlTrans - After execute buffer contents . http://www.cisco.com/cids/idiom" schemaVersion="2.00">http://www.cisco.com/cids/idconf">

13Nov2008 12:36:10.704 0.000 -cidcli[10641] Cid/D doCtrlTrans - After constructing myRspDocPtr

13Nov2008 12:36:10.704 0.000 -cidcli[10641] Cid/D doCtrlTrans - exiting

13Nov2008 10:36:11.076 79200.372 cidwebserver[374] Cid/D Subscription completed reading 0 events

13Nov2008 10:36:11.076 0.000 cidwebserver[374] Cid/D RequestorInfo::reset(cids,system)

13Nov2008 10:36:11.077 0.001 cidwebserver[374] Cid/D session (instance 1) inactive

13Nov2008 10:36:11.086 0.009 cidwebserver[12150] Cid/D WebSession (instance 12) Client 172.30.16.105:2977, GET /cgi-bin/sdee-server?subscriptionId=sub-1-8e8d9e1a&confirm=yes&maxNbrOfEvents=150&timeout=5&sessionId=497005d1735401

Re: IPS 4255 in software bypass

attmidsteam — Fri, 28 Nov 2008 16:42:58 GMT

looks like sensorApp was having issues before the actual crash, I would look at the last 20 lines before this one:

Application "AnalysisEngine "terminated prematurely

Re: IPS 4255 in software bypass

dragnia_s — Fri, 28 Nov 2008 16:56:31 GMT

I've attached the log, i think just the previous 20 lines wold not help.

Thanks!

Re: IPS 4255 in software bypass

attmidsteam — Fri, 28 Nov 2008 17:05:36 GMT

This issue looks different.. your sensorApp was basically dead for 10-11 hours instead of a quick crash. If this happens frequently, I would open a case but I don't think it is the normal E3 runtime issue since you don't have a core (I saw a core on every one of mine that had died - dozens of sensors). I would start watching your sensors closer since they could be silently failing like this.

It stopped working here (the bypass is their watchdog process):

12Nov2008 08:45:52.127 81263.792 interface[362] Cid/W errWarning Inline data bypass has started.

12Nov2008 08:46:53.076 60.949 monitor[386] Monitor/W last update time=1226479551133094000

Messages, like this one, in the category - SensorApp status timed out - were logged 1 times in the last 0 seconds.

And then finally crashed here:

13Nov2008 06:20:59.210 2013.126 mainApp[325] Cid/E errSystemError AppManager::ApplicationEntry::updateProcessStatus - Application "AnalysisEngine "terminated prematurely

Re: IPS 4255 in software bypass

dragnia_s — Wed, 17 Dec 2008 09:13:03 GMT

Hi,

I began checking the satus of the processes when the sensor app was stopped.

I began checking the status of the processes in the /etc/init.d directory, and when i hit one of them the sensor replied:file system corrupt, rebooting...

So i reimaged the appliance hoping that reformatting the file system will fix the problem.

The issue reapered but the log file looks diffrent. The analysisEngine terminates right after bypass.

And a core file was created.

I guess the "certificate_unknown" problem from the log file appears because the sensor changed it's certificate and one of the clients try to connect with IME but another oppinion will be helpfull

Some Log output:

14Dec2008 14:08:26.562 1057.830 cidwebserver[21884] tls/W errWarning received fatal alert: certificate_unknown

Messages, like this one, in the category - receipt of TLS fatal alert message - were logged 1795 times in the last 3601 seconds.

14Dec2008 14:08:26.563 0.001 cidwebserver[392] tls/W errTransport WebSession::sessionTask TLS connection exception: handshake incomplete.

Messages, like this one, in the category - TLS connection failure - were logged 1795 times in the last 3601 seconds.

14Dec2008 16:08:43.156 7216.593 -cidcli[21387] Cid/W errWarning Session was closed by the remote end.

14Dec2008 14:42:35.043 81231.887 interface[378] Cid/W errWarning Inline data bypass has started.

14Dec2008 14:42:35.640 0.597 mainApp[339] Cid/E errSystemError AppManager::ApplicationEntry::updateProcessStatus - Application "AnalysisEngine "terminated prematurely

14Dec2008 15:08:27.761 1552.121 cidwebserver[29551] tls/W errWarning received fatal alert: certificate_unknown

Core file:

-bash-2.05b# cd /usr/cids/idsRoot/core/sensorApp/

-bash-2.05b# ll

-rw-rw-rw- 1 cids cids 2055 Dec 14 16:42 core.txt

-bash-2.05b# more core.txt

Application thread 918 received trap: 11

--------------------------------------------------------------

eax 0x0000fbce 64462

ebx 0x4039300c 1077489676

edx 0x40800008 1082130440

ecx 0x408067b0 1082156976

edi 0x6a0d6c00 1779264512

esi 0x0053fbce 5503950

eip 0x4038fe15 1077476885

ebp 0xb59ff184 -1247809148

esp 0xb59ff11c -1247809252

cs 0x00000023 35

es 0x0000002b 43

ds 0xc010002b -1072693205

gs 0x0000002f 47

fs 0x00000000 0

ss 0x0000002b 43

efl 0x00010246 66118

uesp 0xb59ff11c -1247809252

trapno 0x0000000e 14

err 0x00000004 4

--------------------------------------------------------------

0x0x4038fe15 +/usr/cids/idsRoot/lib/libhoard.so(free+0x125) [0x4038fe15];

0x0x4054d8f0 +/lib/libc.so.6 [0x4054d8f0];

0x0x404ca4a1 +/lib/libstdc++.so.6(_ZdlPv+0x21) [0x404ca4a1];

0x0x836d4af +/usr/cids/idsRoot/bin/sensorApp(_ZN3Cid6Sensor3Eng15InspectorStringD0Ev+0x3f) [0x836d4af];

0x0x815a670 +/usr/cids/idsRoot/bin/sensorApp(_ZN3Cid6Sensor6Common11ListObjTmplINS1_9BasicNodeEED0Ev+0x80) [0x815a670];

0x0x82efba8 +/usr/cids/idsRoot/bin/sensorApp(_ZN3Cid6Sensor4Proc9BiDirDataD0Ev+0x58) [0x82efba8];

0x0x82f8e9e +/usr/cids/idsRoot/bin/sensorApp(_ZN3Cid6Sensor4Proc6TcpTcbD0Ev+0x47e) [0x82f8e9e];

0x0x82fc21c +/usr/cids/idsRoot/bin/sensorApp(_ZN3Cid6Sensor4Proc6TcpTcb6expireEPNS1_13TimeProcessor9TimeEventEPvPNS0_11CsidsBufferE+0x6c) [0x82fc21c];

0x0x82fe489 +/usr/cids/idsRoot/bin/sensorApp(_ZN3Cid6Sensor4Proc13TimeProcessor10updateTimeERNS0_11CsidsBufferE+0x5d9) [0x82fe489];

0x0x8163359 +/usr/cids/idsRoot/bin/sensorApp(_ZN3Cid6Sensor12CidsEnetStub7produceERKNS0_4Proc9ProcessorE+0x17e9) [0x8163359];

0x0x81a26dd +/usr/cids/idsRoot/bin/sensorApp(_ZN3Cid6Sensor9SensorApp13startProducerEPNS_2Mt12ThreadedTaskEPv+0xad) [0x81a26dd];

0x0x4008598a +/usr/cids/idsRoot/lib/libcidcore.002.041.so(_ZN3Cid2Mt12ThreadedTask11threadStartEPv+0x93a) [0x4008598a];

0x0x4001b004 +/lib/libpthread.so.0 [0x4001b004];

0x0x405fc87a +/lib/libc.so.6(clone+0x3a) [0x405fc87a];