topic Re: We were having lots of issues in Network Security

Drain of unprocessed events from Connection Events - Appear on my Sourcefire

pongsiri_chu — Tue, 12 Mar 2019 13:14:01 GMT

Hi all,

Have your ever got error "Drain of unprocessed events from Connection Events" on your device? I look on my FMC and it show this error on my sourcefire module. I tried to search this error on internet but didn't found any issue like this. Please help

noted : I check disk free space, both are fine.

If the FP module is

Claudiu Cismaru — Mon, 02 Jan 2017 18:46:11 GMT

If the FP module is configured to send the events to FMC, then you could have:

- pretty slow connection between the module and FMC and a high number of generated connection events, which are lagged behind

- hung process on FMC which receive the events. There were some bugs related on this.

What's the sensor and FMC version?

Hi Claudiu,

pongsiri_chu — Thu, 05 Jan 2017 06:00:32 GMT

Hi Claudiu,

Thanks for reply FMC version is 6.1.0.1

Do you use file policies?

Claudiu Cismaru — Fri, 06 Jan 2017 09:59:14 GMT

Do you use file policies?

Yes just 1 policy for torrent

pongsiri_chu — Wed, 11 Jan 2017 03:14:32 GMT

Yes just 1 policy for torrent file.

Did you ever get this

jwornstaff — Thu, 27 Apr 2017 11:38:13 GMT

Did you ever get this resolved? I getting same error after upgrading from 6.0.x to 6.1.x.

Was this issue resolved? we

fescobar48 — Wed, 03 May 2017 15:34:05 GMT

Was this issue resolved? we have the same issue, we are on 6.1.0. We open a TACs, but cisco hasn't provided a solution yet. Thinking of going to 6.2.x cause of many more other non related issues.

I had multiple issues

jwornstaff — Wed, 03 May 2017 17:17:07 GMT

I had multiple issues including the one you mentioned when upgrading from 6.0 to 6.1 that I lost track of most of the fixes and not sure which one thing fixed the issue.

But here are several things I did:

I patched all the way up to 6.1.02-57 for both FMC and my SFR sensor module, then I applied a Hotfix of 6.1.3-1 and then the next day there was another patch of 6.1.0.3-57 which I applied.

I also cleaned up/deleted all the old upgrade patches, VDBs, rules, etc, and purged excess data, logs, and backups.

Using the following link for example:

https://www.speaknetworks.com/cisco-firepower-high-disk-space-utilization/

There is also a bug note of this problem but I don't think I implement the workaround, I belive the patching and purging of files did the trick.

Here is link to bug.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuz86604/?referring_site=bugquickviewredir

We were having lots of issues

ewhite — Wed, 03 May 2017 17:24:09 GMT

We were having lots of issues, including the one above. TAC advised us to install 6.2.0. After the 6.2.0 install we have not had the "Drain of unprocessed events from Connection Events". Try the 6.2.0 updates.

thanks jwornstaff!

fescobar48 — Thu, 04 May 2017 14:11:42 GMT

thanks jwornstaff!

We implemented the workaround that was in bug (CSCuz86604) and that stop the alerting.

Soon we will be upgrading to 6.2 and hopefully the many other bugs we have.

Re: We were having lots of issues

dan.letkeman — Tue, 26 Sep 2017 23:04:18 GMT

I have 6.2.1 installed and I still get the messages.

Re: We were having lots of issues

mikael.lahtela — Tue, 17 Oct 2017 08:02:47 GMT

Hi,

I have this issue on 6.2.2

br, Mikael

Re: We were having lots of issues

workforcesoftware — Tue, 23 Jan 2018 15:25:09 GMT

We're running the latest version(6.2.2.1) and still get these issues. We notice the most issues with our Firepower 2110 Active/Failover pair.

Re: We were having lots of issues

Daniel Mansilla — Tue, 20 Feb 2018 17:15:27 GMT

I also have 6.2.2.1 and i am receiving the error, someone has fixed this¿?

Re: Drain of unprocessed events from Connection Events - Appear on my Sourcefire

argrullo — Tue, 20 Feb 2018 17:49:48 GMT

Hello everyone. This is a common error. Please read below. If the below does not solve your issue, please open a TAC SR for further investigation. The messages is version independent, this is not a bug on itself.

One of the main reason is oversubscription/ the amount of logging being done in the FMC. If you set up logging, we recommend to do it at "End of Connection", not both. The end will also catch the beginning timestamp.

+ Why are we seeing this message?

When a silo is being drained on a consistent basis before the events can be sent to the Defense Center, you will get a Health Alert for Frequent Drain of X Events (X is whatever type of events are being drained). This is generally due to a communications error/issue between the sensor and Defense Center which prevents the events from being sent. These events will be queued to be sent to the Defense Center once communication is re-established. This alert doesn't necessarily means the events were unprocessed. Whether they were processed or not can be verified by checking the dismanager.log file.

+ How is this determined?

Depending on the amount of diskspace available on the system, the The High and Low Water Mark will be assigned for each sileo. The HWM and LWM for each of these categories can be found in the /var/log/diskmanager.log file . The output for a Drain of Events will show several related pieces of information. We will look at a conn_events drain as an example:

conn_events, 1430229196,0,0,0,0,0,0,7761331964,129355537,0,7

In this instance, the HWM is 7761331964 bytes and the LWM is 129355537 bytes. The last piece of this output is possibly the most important. This section shows how many unprocessed files were deleted, 7 in this case. Seeing this can indicate that the Defense Center is oversubscribed and that connection event files are being deleted faster than the Defense Center can process.

+ What events were drained?

Run the following command on the sensor CLI

> expert
$ sudo su
# grep -v '0,0' /var/log/diskmanager.log |less

+ How may of the files were drined on this sensor?

Total Silo change:

+ How do I check the silo?

The diskmanager process manages the various file types using silos. This silo information can be seen from the CLISH by running the "show disk-manager" command.

Here is some sample output:
> show disk-manager
Silo                                    Used        Minimum     Maximum
Temporary Files                         142.472 MB 145.368 MB 581.474 MB
Action Queue Results                    0 KB        145.368 MB 581.474 MB
Connection Events                       306.195 MB 1.420 GB    8.518 GB
User Identity Events                    0 KB        145.368 MB 581.474 MB
UI Caches                               184 KB      436.105 MB 872.211 MB
Backups                                 0 KB        1.136 GB    2.839 GB
Updates                                 26 KB       1.704 GB    4.259 GB
Other Detection Engine                  0 KB        872.211 MB 1.704 GB
Performance Statistics                  3.017 MB    290.736 MB 3.407 GB
Other Events                            2.126 MB    581.474 MB 1.136 GB
IP Reputation & URL Filtering           0 KB        726.843 MB 1.420 GB
Archives & Cores & File Logs            1.315 GB    978.961 MB    5.678 GB
RNA Events                              0 KB        1.136 GB    4.543 GB
File Capture                            0 KB        2.839 GB    5.678 GB
IPS Events                              229 KB      3.407 GB    8.518 GB

As you can see, this command shows the Silo, how much space is used, and is another way to see the LWM and HWM.

+ What should I do when files are being drained on the sensor?

Check disk space.

# df -h

Check tunnel status. There should be no error in the output.

# sftunnel_status.pl |less

Reset tunnel

# manage_procs.pl

**************** Configuration Utility **************

1   Reconfigure Correlator
2   Reconfigure and flush Correlator
3   Restart Comm. channel
4   Update routes
5   Reset all routes
6   Validate Network
0   Exit

**************************************************************
Enter choice: 5
Enter choice: 4
Enter choice: 3
Enter choice: 0

Re: Drain of unprocessed events from Connection Events - Appear on my Sourcefire

Daniel Mansilla — Tue, 20 Feb 2018 18:08:32 GMT

Thank you for the explanation, I will affect the services with that procedure?

Re: Drain of unprocessed events from Connection Events - Appear on my Sourcefire

argrullo — Tue, 20 Feb 2018 18:23:42 GMT

It would not affect services since what is being restarted is part of the management channel, not the Detection Engine.

Re: I had multiple issues

tonypearce1 — Fri, 14 Dec 2018 07:58:01 GMT

I had this error and so I began to delete the updates and VDBs that were applied previously and no longer required and during this the error cleared.

Re: We were having lots of issues

Ahmed Noaman — Sun, 16 Dec 2018 07:52:32 GMT

I have 6.2.2 installed and I still get the messages.

Re: We were having lots of issues

tonypearce1 — Mon, 17 Dec 2018 00:13:13 GMT

Go to the latest and then retest. I'm on 6.2.3.7 I believe