My basic configuration.
rsyslog –> Logstash –> Elasticsearch –> Kibana with GeoIP
At this time I do not plan to proxy Kibana with Nginx. The goal here is to keep the setup and configuration as simple as possible.
Versions:
CentOS 7
Logstash 1.4.2
Kibana 4.0.2
ElasticSearch 1.5.0
This is the fourth time I have set this up and each time I have fought through different errors. My goal is to more thoroughly document the process so that I can recreate it easily the next time.
1. Turn off SE Linux. I fought app armor on Ubuntu doing one of these installs and I am not going to fight SE Linux. Reboot.
shutdown -r now
Check if selinux is on or off.
sestatus
2. Turn off the firewall.
systemctl stop firewalld systemctl disable firewalld systemctl status firewalld
3. Update.
yum update yum upgrade
4. Configure rsyslog to accept remote logging and name the file appropriately. This is the way we have named our log files and we have a number of scripts that work with them, so they are not changing.
cat /etc/rsyslog.conf | grep -v ^# | egrep -v ^$ $ModLoad imuxsock # provides support for local system logging (e.g. via logger command) $ModLoad imjournal # provides access to the systemd journal $ModLoad imklog # reads kernel messages (the same are read from journald) $ModLoad immark # provides --MARK-- message capability $ModLoad imudp $UDPServerRun 514 $ModLoad imtcp $InputTCPServerRun 514 $WorkDirectory /var/lib/rsyslog $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat $IncludeConfig /etc/rsyslog.d/*.conf $OmitLocalLogging on $IMJournalStateFile imjournal.state *.info;mail.none;authpriv.none;cron.none /var/log/messages authpriv.* /var/log/secure mail.* -/var/log/maillog cron.* /var/log/cron *.emerg :omusrmsg:* uucp,news.crit /var/log/spooler local7.* /var/log/boot.log $template DynaFile,"/var/log/remote-%fromhost-ip%.log" *.* -?DynaFile
The last two lines are the ones that count.
$template DynaFile,"/var/log/remote-%fromhost-ip%.log" *.* -?DynaFile
Restart rsyslog.
systemctl restart rsyslog
5. Install Java and some other tools you might need later.
yum install -y java-1.7.0-openjdk lsof wget rubygems
6. Install the GPG for the ElasticSearch repository and install it but don’t start it yet.
Add the Elasticsearch repository.
cat /etc/yum.repos.d/elasticsearch.repo [elasticsearch-1.5] name=Elasticsearch repository for 1.5.x packages baseurl=http://packages.elasticsearch.org/elasticsearch/1.5/centos gpgcheck=1 gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch enabled=1
And install.
yum update yum -y install elasticsearch
7. Change your Elasticsearch configuration to suit your needs. I had trouble with Elasticsearch and Logstash only listening on IPv6. As a result I had to explicitly set the IP address as seen below. We also have another Elasticsearch cluster and I don’t want the two talking so I turned off multicast discovery. This is not a problem if you edit your configs first, but I had a terrible time because they were named the same.
cat /etc/elasticsearch/elasticsearch.yml | egrep -v "^#|^$" cluster.name: logserver node.name: "elk" node.master: true node.data: true node.max_local_storage_nodes: 1 network.bind_host: 172.22.225.76 network.host: 172.22.225.76 discovery.zen.ping.multicast.enabled: false
If you have problems with ElasticSearch not listening on IPv4.
In the file /usr/share/elasticsearch/bin/elasticsearch.in.sh
You see the lines:
# Force the JVM to use IPv4 stack
if [ "x$ES_USE_IPV4" != "x" ]; then
JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"
fi
So in the /etc/sysconfig/elasticsearch file add:
# Tell ES to use IPv4 ES_USE_IPV4=true
Here is my entire syconfig file for Elasticsearch.
cat /etc/sysconfig/elasticsearch | egrep -v "^#|^$" ES_HOME=/usr/share/elasticsearch MAX_OPEN_FILES=65535 MAX_MAP_COUNT=262144 LOG_DIR=/var/log/elasticsearch DATA_DIR=/var/lib/elasticsearch WORK_DIR=/tmp/elasticsearch CONF_DIR=/etc/elasticsearch CONF_FILE=/etc/elasticsearch/elasticsearch.yml ES_USER=elasticsearch ES_USE_IPV4=true
These are good examples from the Elasitcsearch configuration file:
# Use the Cluster Health API [http://localhost:9200/_cluster/health], the # Node Info API [http://localhost:9200/_nodes] or GUI tools # such as <http://www.elasticsearch.org/overview/marvel/ # http://github.com/karmi/elasticsearch-paramedic # http://github.com/lukas-vlcek/bigdesk and # http://mobz.github.com/elasticsearch-head to inspect the cluster state.
8. Start Elasticsearch.
systemctl enable elasticsearch systemctl start elasticsearch
9. Test it.
curl http://172.22.225.76:9200
{
"status" : 200,
"name" : "elk",
"cluster_name" : "logserver",
"version" : {
"number" : "1.5.1",
"build_hash" : "5e38401bc4e4388537a615569ac60925788e1cf4",
"build_timestamp" : "2015-04-09T13:41:35Z",
"build_snapshot" : false,
"lucene_version" : "4.10.4"
},
"tagline" : "You Know, for Search"
}
Other testing examples, some will not work now as we have not finished the Logstash configuration.
curl -XGET http://172.22.225.76:9200/_cluster/health curl http://172.22.225.76:9200 curl http://172.22.225.76:9200/_status?pretty=true curl http://172.22.225.76:9200/_search?q=type:syslog&pretty=true
If you need to debug Elasticsearch here is the strace command I used.
strace java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.5.0.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch -Djava.net.preferIPv6Addresses=false
10. LogStash installation, first enable the repository.
cat /etc/yum.repos.d/logstash.repo [logstash-1.5] name=logstash repository for 1.5.x packages baseurl=http://packages.elasticsearch.org/logstash/1.5/centos gpgcheck=1 gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch enabled=1
Then install Logstash.
yum install -y logstash
I also had some trouble getting Logstash to run properly. It was missing a directory and only listening on IPv6. The first thing I has do was a make specific directory.
mkdir /var/run/nscd
The second thing I had to do was add a Java option in the sysconfig directory.
cat /etc/sysconfig/logstash | egrep -v "^#|^$" LS_JAVA_OPTS="-Djava.net.preferIPv4Stack=true"
If you have trouble with logstash, here is the strace command I ran to debug it.
strace -o /tmp/strace.log -fe trace=network /opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf
Below is the error where I had to create the directory to fix.
21790 socket(PF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
21790 connect(3, {sa_family=AF_LOCAL, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
21790 socket(PF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
21790 connect(3, {sa_family=AF_LOCAL, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
Here is my very simple logstash.conf file.
input {
syslog {
debug => true
type => syslog
port => 5140
}
}
filter {
}
output {
stdout { }
elasticsearch {
cluster => "logserver"
host => "172.22.225.76"
}
}
Turn on logstash and start it.
chkconfig logstash on /etc/init.d/logstash start
I pointed a firewall at this ELK server, and now you should see messages coming into Elasticsearch from Logstash.
curl http://172.22.225.76:9200/_search?q=type:syslog&pretty=true
11. Install Kibana
Download the gzipped tar file.
wget https://download.elasticsearch.org/kibana/kibana/kibana-4.0.2-linux-x64.tar.gz
Extract it.
tar -xvzf kibana-4.0.2-linux-x64.tar.gz
Make a directory for Kibana and copy the unzipped contents into it.
mkdir /opt/kibana cp -R kibana-4.0.2-linux-x64/* /opt/kibana/
Here is my basic Kibana configuration file.
cat /opt/kibana/config/kibana.yml | egrep -v "^#|^$" port: 5601 host: "172.22.225.76" elasticsearch_url: "http://172.22.225.76:9200" elasticsearch_preserve_host: true kibana_index: ".kibana" default_app_id: "discover" request_timeout: 300000 shard_timeout: 0 verify_ssl: true bundled_plugin_ids: - plugins/dashboard/index - plugins/discover/index - plugins/doc/index - plugins/kibana/index - plugins/markdown_vis/index - plugins/metric_vis/index - plugins/settings/index - plugins/table_vis/index - plugins/vis_types/index - plugins/visualize/index
Set it up ready to run.
gem install pleaserun /usr/local/bin/pleaserun --platform systemd --install /opt/kibana/bin/kibana
Enable Kibana and start it.
systemctl enable kibana systemctl start kibana
Go and check it out by pointing your browser to:
http://elk.chainringcircus.org:5601.
12. Install GeoLite
Here is the website for more information:
http://dev.maxmind.com/geoip/legacy/geolite/
Download the city database.
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
I put the database in my /opt/logstash directory but there may be other directories that make more sense.
gunzip GeoLiteCity.dat.gz mv GeoLiteCity.dat /opt/logstash
13. Here is my /etc/logstash/conf.d/asa.conf configuration file. You will notice that I tag each interface as inside_out or outside_in. That is so that we can track who we are blocking from the inside as well as who we are blocking from the outside. That also lets us create a map for each type of traffic for visualization purposes.
cat /etc/logstash/conf.d/asa.conf
input {
file {
path => ["/var/log/remote-192.168.2.250.log"]
sincedb_path => "/var/log/logstash/since.db"
start_position => "beginning"
type => "asa"
}
file {
path => ["/var/log/remote-192.168.2.254.log"]
sincedb_path => "/var/log/logstash/since.db"
start_position => "beginning"
type => "asa"
}
}
# begin filter block
filter {
if [type] == "asa" { # begin ASA block
grok {
match => ["message", "%{CISCO_TAGGED_SYSLOG} %{GREEDYDATA:cisco_message}"]
}
# Parse the syslog severity and facility
syslog_pri { }
# Parse the date from the "timestamp" field to the "@timestamp" field
date {
match => ["timestamp",
"MMM dd HH:mm:ss",
"MMM d HH:mm:ss",
"MMM dd yyyy HH:mm:ss",
"MMM d yyyy HH:mm:ss"
]
timezone => "America/New_York"
}
# Clean up redundant fields if parsing was successful
if "_grokparsefailure" not in [tags] {
mutate {
rename => ["cisco_message", "message"]
remove_field => ["timestamp"]
}
}
# Extract fields from the each of the detailed message types
# The patterns provided below are included in Logstash since 1.2.0
grok {
match => [
"message", "%{CISCOFW106001}",
"message", "%{CISCOFW106006_106007_106010}",
"message", "%{CISCOFW106014}",
"message", "%{CISCOFW106015}",
"message", "%{CISCOFW106021}",
"message", "%{CISCOFW106023}",
"message", "%{CISCOFW106100}",
"message", "%{CISCOFW110002}",
"message", "%{CISCOFW302010}",
"message", "%{CISCOFW302013_302014_302015_302016}",
"message", "%{CISCOFW302020_302021}",
"message", "%{CISCOFW305011}",
"message", "%{CISCOFW313001_313004_313008}",
"message", "%{CISCOFW313005}",
"message", "%{CISCOFW402117}",
"message", "%{CISCOFW402119}",
"message", "%{CISCOFW419001}",
"message", "%{CISCOFW419002}",
"message", "%{CISCOFW500004}",
"message", "%{CISCOFW602303_602304}",
"message", "%{CISCOFW710001_710002_710003_710005_710006}",
"message", "%{CISCOFW713172}",
"message", "%{CISCOFW733100}"
]
}
# GeoIP for the ASA
# Source
if [src_interface] == "Outside" {
geoip {
source => "src_ip"
target => "geoip"
database =>"/opt/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float" ]
add_tag => [ 'outside_in' ]
}
} # end source block
# Destination
##if [src_interface] !~ "(Inside)|(Bypass)" {
if [src_interface] == "Inside" {
geoip {
source => "dst_ip"
target => "geoip"
database =>"/opt/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float" ]
add_tag => [ 'inside_out' ]
}
}
} # end ASA
} # end filter block
output {
stdout { }
elasticsearch {
cluster => "logserver"
host => "172.22.225.76"
}
}
14. Some final thoughts and hints.
Here are some commands I used for debugging purposes.
lsof -i4 lsof -i6 netstat -l curl -XGET http://172.22.225.76:9200/_cluster/health
Last is a helper script I created.
cat /usr/local/bin/elkstack-restart #!/bin/bash systemctl restart elasticsearch systemctl restart logstash systemctl restart kibana
Sources in no particular order:
http://www.networkassassin.com/elk-stack-for-network-operations-reloaded/
https://community.ulyaoth.net/threads/how-to-create-a-logstash-geoip-based-dashboard-in-kibana-3.29/
http://kartar.net/2014/09/when-logstash-and-syslog-go-wrong/
http://blog.domb.net/?p=367
http://grokdebug.herokuapp.com/patterns#
http://blog.stevenmeyer.co.uk/2014/06/add-configuration-test-to-logstash-service-configtest.html
http://www.itzgeek.com/how-tos/linux/centos-how-tos/how-do-i-disable-ipv6-on-centos-7-rhel-7.html#axzz3WjuY2oLh
http://slacklabs.be/2012/04/02/force-Elastic-search-on-ipv4-debian/
ChainRingCircus 