Ok understood.
First off, I'd forget the Netflow at the start. Once you have a working environment (which can be a much harder task than it seems), any of the special sauce will be easier after the fact.
So starting with capturing syslog is the best first goat rodeo.
That link is incredibly OLD and I would not follow it.
I have some notes I will post that I use for pushing out a ELK box but first, let me articulate "my" design.
I have 30'ish locations and 3 data centers.
Each site (data center included) has a dedicated syslog-ng box. Every device, at each site, points its syslog that IP address.
From there, syslog-ng pushes the sources up to Logstash (installed on the same box). All of the ELK stuff is Java and it is a pain/security risk giving those apps root access to ports below 1024. Plus syslog-ng is incredibly fast and power and functions as a one to many for me. The syslog-ng instance also pipes off a copy to a central Cacti instance for email alerting based on RegEx.
Once the log has been parsed by Logstash, it is configured to push out to each Data Center cluster of 10 Elasticsearch servers. I have a client node, setup that hooks into the cluster for the visualization of all the parsed data.
Bonus: All our windows servers have the Elastic winlogbeat client on them who point to the site local Logstash instance. Which then puts that data in the same pipeline into the data center clusters.
Most of that sounds simple and it is but is incredibly scalable. Since my design and deployment last year, we added 4 sites and 6 unique sources that were a breeze to add into the path. If I have a problem, I can re-route the output pre-parse with syslog-ng or post-parse with Logstash.
Hopefully that all makes sense.
Here are my basic install notes:
I use Ubuntu 16.04
- Remove rsyslog(installed by default)
- Install syslog-ng
- Configure syslog-ng to listen on whatever port(s) you need.
From there the fun begins, installed Elasticsearch.
- Install Java 8 (latest)
- Install Elasticsearch (the DB portion of ELK)
Those 4 commands and you are setup. There are a LOT of tweaks that you will want to do. I'll be happy to help you with those but worry about that after you are up and running. You will need to systemctl or update-rc.d to make sure the service starts at boot.
Your Elasticsearch config file is [ elasticsearch.yml ].
- Next, install Kibana.
- sudo apt-get update && sudo apt-get install kibana
You will need to systemctl or update-rc.d to make sure the service starts at boot.
Your Kibana config file is [ kibana.yml ]
- Next, install NGINX.
- sudo apt-get -y install nginx
Nginx is a reverse proxy, long story, good security measure.
- Finally, install Logstash.
- sudo apt-get update && sudo apt-get install logstash
Your Logstash config file is [ logstash.yml ]. You will need to systemctl or update-rc.d to make sure the service starts at boot.
Hopefully this helps you. I am glad to help however I can as getting this up and running has been incredibly eye opening for me as to all the data that can be gleaned. If you need help, you can post here or PM. I can talk on the phone or stand a WebEx that we can hop on.
