Introduction:
This document gives the explanation about Event Store Wrapping.
From IPSpedia
Jump to: navigation, search
Description:
Event store wrapping is really a symptom caused by another root cause, typically a chatty signature or as a result of an Event Action Override This symptom is often referred to as "hyper alerting", defined by excessive writing of events to the sensor's eventstore. These events may include non-signature alerts as well, i.e., fatal, error, warning, or status events. Note that much of this article material is pulled from the Event Action Overrides writeup, as the symptom and fix is generally the same but is duplicated here for ease of finding the information.
Impact:
The main issue here is typically the frequency of writes to the eventstore. Excessive eventstore writes is typically associated with high CPU and general unresponsiveness of the sensor via management access tools (cidcli, IDM).
A note about SNMP Trap actions and EAOs:
Please note that a signature that has only the single action of request-snmp-trap also generates an alert event that is written to the eventstore, so excessive firing of SNMP Trap action can also trigger the same problems seen with excessive produce alert actions.
A note about Produce Alert actions (or, any other action that causes eventstore writes -- request-snmp-trap or one of the log- actions) and Normalizer engine signatures:
This applies to all 1200 - 1330 range signature IDs. Except during brief troubleshooting scenarios, customers should be cautioned against using EAOs for the Normalizer engine signatures. This can be particularly problematic in either 1) highly fragmented IP scenarios (due to the 1200-range signatures) or 2) heavily out-of-order (ooo) TCP scenarios (1300-range signatures). An EAO that causes a write to the eventstore for every ooo TCP packet (as an example) can cause resource issues for the same reasons already discussed.
A note about EAOs with Risk Rating of 0 - 100:
In general, this is a very bad idea. The reason being, Meta component signatures will often fire for seemingly benign (and common) types of traffic. Meta signatures will look for a combination of 2 or more meta component signatures to trigger before the parent Meta signature will fire an alert. Meta component signatures, by default, have no actions associated with them, and this is on purpose for the reason that they will often frequently match on common traffic. Meta component signaures will have a default base risk rating of 15. To exclude catching these signature matches in an EAO, we would recommend not going lower than 25 risk rating when doing an EAO (i.e., no lower than RR of 25 - 100).
Relevant bugs:
There is a bug, CSCsq51334 (currently in U state), which is associated with heavy ip logging. The error message associated with this issue is:
14Sep2010 17:45:34.739 554.913 sensorApp[747] Cid/W errWarn IpLogProcessor::addIpLog: Ran out of file descriptors
There may also be web sessions executing iplog-server.
Commands (cidcli & service account):
Via the cidcli, you can issue the command "show statistics virtual-sensor" and look for the inspection load percentage, like so:
sensor# show statistics virtual-sensor | inc Load
Processing Load Percentage = 100
A very high load percentage (90 or above) may indicate that the sensor is experiencing a hyper alerting scenario. To further confirm that this possibility is occurring, see "Logs" section below.
Logs:
The main indicator of hyper alerting will be rapid eventstore wrapping, as seen in the main.log file. Example:
25Jan2010 05:13:08.326 19.897 sensorApp[18316] IdsEventStore/W errWarning - the event store wrapped around [IdsEventStore::writeEvent(), index = 19530]
25Jan2010 05:32:05.751 85.031 sensorApp[18316] IdsEventStore/W errWarning - the event store wrapped around [IdsEventStore::writeEvent(), index = 19529]
25Jan2010 05:50:45.442 4.989 sensorApp[18316] IdsEventStore/W errWarning - the event store wrapped around [IdsEventStore::writeEvent(), index = 19530]
25Jan2010 06:08:59.281 70.143 sensorApp[18316] IdsEventStore/W errWarning - the event store wrapped around [IdsEventStore::writeEvent(), index = 19529]
25Jan2010 06:25:40.923 34.562 sensorApp[18316] IdsEventStore/W errWarning - the event store wrapped around [IdsEventStore::writeEvent(), index = 19531]
A general rule of thumb is that a sensor wrapping eventstore more frequently than once per hour may indicate a problem. There's many variables to consider, including overall performance capability of the platform. In some scenarios, the wrapping is truly and obviously excessive, sometimes wrapping multiple times within a minute.
Workarounds/fixes/tuning recommendations:
This will depend on what type of event/traffic/action is causing the hyper alerting problem (i.e., produce alert, ip logging, normalizer signatures, meta component signature). If it is simply a chatty signature, we would recommend writing an event action filter (EAF) if the customer determines this to be false positive events. For IP Logging, we would generally recommend to either not do this with an EAO, or to do so with caution and to understand the risks. For normalizer signatures and meta component signatures, these signatures should not have an alert action except during temporary troubleshooting scenarios.
Hope this document is informative and i want to thank you for viewing.
