We are currently planning a Cisco Prime installation in an effort to replace a sprawl of tools currently. One of the crucial pieces of our environment is syslog, and even though we are a small shop, we currently generate roughly 22million log entries daily. Looking at the syslog limits in Prime, it appears that this will be not even close to enough (and it also seems to be overkill, considering that our device counts line up with Prime Express). Anyone else have suggestions or willing to share how they've approached logging in their environments?
Just to update, it's actually ~52million. It's true we are seeing 80% of these logs being generated from two ASA's and 94% of the log severity is Level6 - Informational. Being a government agency in oil/gas, we do have some regulatory and compliance issues to keep in mind (although I don't know if Informational is required, or just enabled as a CYA measure).
I did look into Splunk, but based on our volume it seems astronomically expensive (unless I am reading it wrong). Do you have feedback on Splunk Licensing?
Absent a hard requirement to log Level 6 from your firewalls I'd back it down to level 4. That's the Cisco TAC recommendation too (unless you're actively troubleshooting). I do the same for other device syslogs, even excluding certain noisy events that are never actionable.
Splunk is known for being "reassuringly expensive". So you probably read it right.
One adage I repeat often (maybe I even coined it - it's been a while) is "When requirements are free the demand is infinite." Also, if you're only keeping syslogs for the sake of keeping them then why bother. The tool without a process and people who follow it as part of their core tasks is just about useless. I once managed a NOC where I measure out vendor on their ability to address and reduce the volume of log messages (by fixing root cause - not by turning them off).
I advocate a responsible level of syslog collection. For firewalls, if you must collect high volume then use a tool that's purpose built like Splunk or Cisco Security Manager. The latter has nice retention tools that can back up historical data to remote storage. CSM is a lot less expensive than Splunk too. Roadmap wise its on the way out in favor of a Cisco solution that integrates it into FirPOWER Management Center but it's stillactively developed.