Hello everyone. This is a common error. Please read below. If the below does not solve your issue, please open a TAC SR for further investigation. The messages is version independent, this is not a bug on itself.
One of the main reason is oversubscription/ the amount of logging being done in the FMC. If you set up logging, we recommend to do it at "End of Connection", not both. The end will also catch the beginning timestamp.
+ Why are we seeing this message?
When a silo is being drained on a consistent basis before the events can be sent to the Defense Center, you will get a Health Alert for Frequent Drain of X Events (X is whatever type of events are being drained). This is generally due to a communications error/issue between the sensor and Defense Center which prevents the events from being sent. These events will be queued to be sent to the Defense Center once communication is re-established. This alert doesn't necessarily means the events were unprocessed. Whether they were processed or not can be verified by checking the dismanager.log file.
+ How is this determined?
Depending on the amount of diskspace available on the system, the The High and Low Water Mark will be assigned for each sileo. The HWM and LWM for each of these categories can be found in the /var/log/diskmanager.log file . The output for a Drain of Events will show several related pieces of information. We will look at a conn_events drain as an example:
conn_events, 1430229196,0,0,0,0,0,0,7761331964,129355537,0,7
In this instance, the HWM is 7761331964 bytes and the LWM is 129355537 bytes. The last piece of this output is possibly the most important. This section shows how many unprocessed files were deleted, 7 in this case. Seeing this can indicate that the Defense Center is oversubscribed and that connection event files are being deleted faster than the Defense Center can process.
+ What events were drained?
Run the following command on the sensor CLI
> expert
$ sudo su
# grep -v '0,0' /var/log/diskmanager.log |less
+ How may of the files were drined on this sensor?
Total Silo change:
+ How do I check the silo?
The diskmanager process manages the various file types using silos. This silo information can be seen from the CLISH by running the "show disk-manager" command.
Here is some sample output:
> show disk-manager
Silo Used Minimum Maximum
Temporary Files 142.472 MB 145.368 MB 581.474 MB
Action Queue Results 0 KB 145.368 MB 581.474 MB
Connection Events 306.195 MB 1.420 GB 8.518 GB
User Identity Events 0 KB 145.368 MB 581.474 MB
UI Caches 184 KB 436.105 MB 872.211 MB
Backups 0 KB 1.136 GB 2.839 GB
Updates 26 KB 1.704 GB 4.259 GB
Other Detection Engine 0 KB 872.211 MB 1.704 GB
Performance Statistics 3.017 MB 290.736 MB 3.407 GB
Other Events 2.126 MB 581.474 MB 1.136 GB
IP Reputation & URL Filtering 0 KB 726.843 MB 1.420 GB
Archives & Cores & File Logs 1.315 GB 978.961 MB 5.678 GB
RNA Events 0 KB 1.136 GB 4.543 GB
File Capture 0 KB 2.839 GB 5.678 GB
IPS Events 229 KB 3.407 GB 8.518 GB
As you can see, this command shows the Silo, how much space is used, and is another way to see the LWM and HWM.
+ What should I do when files are being drained on the sensor?
- Check disk space.
# df -h
- Check tunnel status. There should be no error in the output.
# sftunnel_status.pl |less
- Reset tunnel
# manage_procs.pl
**************** Configuration Utility **************
1 Reconfigure Correlator
2 Reconfigure and flush Correlator
3 Restart Comm. channel
4 Update routes
5 Reset all routes
6 Validate Network
0 Exit
**************************************************************
Enter choice: 5
Enter choice: 4
Enter choice: 3
Enter choice: 0
