My basic configuration.
rsyslog –> Logstash –> Elasticsearch –> Kibana with GeoIP
At this time I do not plan to proxy Kibana with Nginx. The goal here is to keep the setup and configuration as simple as possible.
Versions:
CentOS 7
Logstash 1.4.2
Kibana 4.0.2
ElasticSearch 1.5.0
This is the fourth time I have set this up and each time I have fought through different errors. My goal is to more thoroughly document the process so that I can recreate it easily the next time.
1. Turn off SE Linux. I fought app armor on Ubuntu doing one of these installs and I am not going to fight SE Linux. Reboot.
[code]
shutdown -r now
[/code]
Check if selinux is on or off.
[code]
sestatus
[/code]
2. Turn off the firewall.
[code]
systemctl stop firewalld
systemctl disable firewalld
systemctl status firewalld
[/code]
3. Update.
[code]
yum update
yum upgrade
[/code]
4. Configure rsyslog to accept remote logging and name the file appropriately. This is the way we have named our log files and we have a number of scripts that work with them, so they are not changing.
[code]
cat /etc/rsyslog.conf | grep -v ^# | egrep -v ^$
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
$ModLoad imklog # reads kernel messages (the same are read from journald)
$ModLoad immark # provides –MARK– message capability
$ModLoad imudp
$UDPServerRun 514
$ModLoad imtcp
$InputTCPServerRun 514
$WorkDirectory /var/lib/rsyslog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
$OmitLocalLogging on
$IMJournalStateFile imjournal.state
*.info;mail.none;authpriv.none;cron.none /var/log/messages
authpriv.* /var/log/secure
mail.* -/var/log/maillog
cron.* /var/log/cron
*.emerg :omusrmsg:*
uucp,news.crit /var/log/spooler
local7.* /var/log/boot.log
$template DynaFile,"/var/log/remote-%fromhost-ip%.log"
*.* -?DynaFile
[/code]
The last two lines are the ones that count.
[code]
$template DynaFile,"/var/log/remote-%fromhost-ip%.log"
*.* -?DynaFile
[/code]
Restart rsyslog.
[code]
systemctl restart rsyslog
[/code]
5. Install Java and some other tools you might need later.
[code]
yum install -y java-1.7.0-openjdk lsof wget rubygems
[/code]
6. Install the GPG for the ElasticSearch repository and install it but don’t start it yet.
Add the Elasticsearch repository.
[code]
cat /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-1.5]
name=Elasticsearch repository for 1.5.x packages
baseurl=http://packages.elasticsearch.org/elasticsearch/1.5/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1
[/code]
And install.
[code]
yum update
yum -y install elasticsearch
[/code]
7. Change your Elasticsearch configuration to suit your needs. I had trouble with Elasticsearch and Logstash only listening on IPv6. As a result I had to explicitly set the IP address as seen below. We also have another Elasticsearch cluster and I don’t want the two talking so I turned off multicast discovery. This is not a problem if you edit your configs first, but I had a terrible time because they were named the same.
[code]
cat /etc/elasticsearch/elasticsearch.yml | egrep -v "^#|^$"
cluster.name: logserver
node.name: "elk"
node.master: true
node.data: true
node.max_local_storage_nodes: 1
network.bind_host: 172.22.225.76
network.host: 172.22.225.76
discovery.zen.ping.multicast.enabled: false
[/code]
If you have problems with ElasticSearch not listening on IPv4.
In the file /usr/share/elasticsearch/bin/elasticsearch.in.sh
You see the lines:
[code]
# Force the JVM to use IPv4 stack
if [ "x$ES_USE_IPV4" != "x" ]; then
JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"
fi
[/code]
So in the /etc/sysconfig/elasticsearch file add:
[code]
# Tell ES to use IPv4
ES_USE_IPV4=true
[/code]
Here is my entire syconfig file for Elasticsearch.
[code]
cat /etc/sysconfig/elasticsearch | egrep -v "^#|^$"
ES_HOME=/usr/share/elasticsearch
MAX_OPEN_FILES=65535
MAX_MAP_COUNT=262144
LOG_DIR=/var/log/elasticsearch
DATA_DIR=/var/lib/elasticsearch
WORK_DIR=/tmp/elasticsearch
CONF_DIR=/etc/elasticsearch
CONF_FILE=/etc/elasticsearch/elasticsearch.yml
ES_USER=elasticsearch
ES_USE_IPV4=true
[/code]
These are good examples from the Elasitcsearch configuration file:
[code]
# Use the Cluster Health API [http://localhost:9200/_cluster/health], the
# Node Info API [http://localhost:9200/_nodes] or GUI tools
# such as <http://www.elasticsearch.org/overview/marvel/
# http://github.com/karmi/elasticsearch-paramedic
# http://github.com/lukas-vlcek/bigdesk and
# http://mobz.github.com/elasticsearch-head to inspect the cluster state.
[/code]
8. Start Elasticsearch.
[code]
systemctl enable elasticsearch
systemctl start elasticsearch
[/code]
9. Test it.
[code]
curl http://172.22.225.76:9200
{
"status" : 200,
"name" : "elk",
"cluster_name" : "logserver",
"version" : {
"number" : "1.5.1",
"build_hash" : "5e38401bc4e4388537a615569ac60925788e1cf4",
"build_timestamp" : "2015-04-09T13:41:35Z",
"build_snapshot" : false,
"lucene_version" : "4.10.4"
},
"tagline" : "You Know, for Search"
}
[/code]
Other testing examples, some will not work now as we have not finished the Logstash configuration.
[code]
curl -XGET http://172.22.225.76:9200/_cluster/health
curl http://172.22.225.76:9200
curl http://172.22.225.76:9200/_status?pretty=true
curl http://172.22.225.76:9200/_search?q=type:syslog&pretty=true
[/code]
If you need to debug Elasticsearch here is the strace command I used.
[code]
strace java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.5.0.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch -Djava.net.preferIPv6Addresses=false
[/code]
10. LogStash installation, first enable the repository.
[code]
cat /etc/yum.repos.d/logstash.repo
[logstash-1.5]
name=logstash repository for 1.5.x packages
baseurl=http://packages.elasticsearch.org/logstash/1.5/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1
[/code]
Then install Logstash.
[code]
yum install -y logstash
[/code]
I also had some trouble getting Logstash to run properly. It was missing a directory and only listening on IPv6. The first thing I has do was a make specific directory.
[code]
mkdir /var/run/nscd
[/code]
The second thing I had to do was add a Java option in the sysconfig directory.
[code]
cat /etc/sysconfig/logstash | egrep -v "^#|^$"
LS_JAVA_OPTS="-Djava.net.preferIPv4Stack=true"
[/code]
If you have trouble with logstash, here is the strace command I ran to debug it.
[code]
strace -o /tmp/strace.log -fe trace=network /opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf
[/code]
Below is the error where I had to create the directory to fix.
[code]
21790 socket(PF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
21790 connect(3, {sa_family=AF_LOCAL, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
21790 socket(PF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
21790 connect(3, {sa_family=AF_LOCAL, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
[/code]
Here is my very simple logstash.conf file.
[code]
input {
syslog {
debug => true
type => syslog
port => 5140
}
}
filter {
}
output {
stdout { }
elasticsearch {
cluster => "logserver"
host => "172.22.225.76"
}
}
[/code]
Turn on logstash and start it.
[code]
chkconfig logstash on
/etc/init.d/logstash start
[/code]
I pointed a firewall at this ELK server, and now you should see messages coming into Elasticsearch from Logstash.
[code]
curl http://172.22.225.76:9200/_search?q=type:syslog&pretty=true
[/code]
11. Install Kibana
Download the gzipped tar file.
[code]
wget https://download.elasticsearch.org/kibana/kibana/kibana-4.0.2-linux-x64.tar.gz
[/code]
Extract it.
[code]
tar -xvzf kibana-4.0.2-linux-x64.tar.gz
[/code]
Make a directory for Kibana and copy the unzipped contents into it.
[code]
mkdir /opt/kibana
cp -R kibana-4.0.2-linux-x64/* /opt/kibana/
[/code]
Here is my basic Kibana configuration file.
[code]
cat /opt/kibana/config/kibana.yml | egrep -v "^#|^$"
port: 5601
host: "172.22.225.76"
elasticsearch_url: "http://172.22.225.76:9200"
elasticsearch_preserve_host: true
kibana_index: ".kibana"
default_app_id: "discover"
request_timeout: 300000
shard_timeout: 0
verify_ssl: true
bundled_plugin_ids:
– plugins/dashboard/index
– plugins/discover/index
– plugins/doc/index
– plugins/kibana/index
– plugins/markdown_vis/index
– plugins/metric_vis/index
– plugins/settings/index
– plugins/table_vis/index
– plugins/vis_types/index
– plugins/visualize/index
[/code]
Set it up ready to run.
[code]
gem install pleaserun
/usr/local/bin/pleaserun –platform systemd –install /opt/kibana/bin/kibana
[/code]
Enable Kibana and start it.
[code]
systemctl enable kibana
systemctl start kibana
[/code]
Go and check it out by pointing your browser to:
[code]
http://elk.chainringcircus.org:5601.
[/code]
12. Install GeoLite
Here is the website for more information:
[code]
http://dev.maxmind.com/geoip/legacy/geolite/
[/code]
Download the city database.
[code]
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
[/code]
I put the database in my /opt/logstash directory but there may be other directories that make more sense.
[code]
gunzip GeoLiteCity.dat.gz
mv GeoLiteCity.dat /opt/logstash
[/code]
13. Here is my /etc/logstash/conf.d/asa.conf configuration file. You will notice that I tag each interface as inside_out or outside_in. That is so that we can track who we are blocking from the inside as well as who we are blocking from the outside. That also lets us create a map for each type of traffic for visualization purposes.
[code]
cat /etc/logstash/conf.d/asa.conf
input {
file {
path => ["/var/log/remote-192.168.2.250.log"]
sincedb_path => "/var/log/logstash/since.db"
start_position => "beginning"
type => "asa"
}
file {
path => ["/var/log/remote-192.168.2.254.log"]
sincedb_path => "/var/log/logstash/since.db"
start_position => "beginning"
type => "asa"
}
}
# begin filter block
filter {
if [type] == "asa" { # begin ASA block
grok {
match => ["message", "%{CISCO_TAGGED_SYSLOG} %{GREEDYDATA:cisco_message}"]
}
# Parse the syslog severity and facility
syslog_pri { }
# Parse the date from the "timestamp" field to the "@timestamp" field
date {
match => ["timestamp",
"MMM dd HH:mm:ss",
"MMM d HH:mm:ss",
"MMM dd yyyy HH:mm:ss",
"MMM d yyyy HH:mm:ss"
]
timezone => "America/New_York"
}
# Clean up redundant fields if parsing was successful
if "_grokparsefailure" not in [tags] {
mutate {
rename => ["cisco_message", "message"]
remove_field => ["timestamp"]
}
}
# Extract fields from the each of the detailed message types
# The patterns provided below are included in Logstash since 1.2.0
grok {
match => [
"message", "%{CISCOFW106001}",
"message", "%{CISCOFW106006_106007_106010}",
"message", "%{CISCOFW106014}",
"message", "%{CISCOFW106015}",
"message", "%{CISCOFW106021}",
"message", "%{CISCOFW106023}",
"message", "%{CISCOFW106100}",
"message", "%{CISCOFW110002}",
"message", "%{CISCOFW302010}",
"message", "%{CISCOFW302013_302014_302015_302016}",
"message", "%{CISCOFW302020_302021}",
"message", "%{CISCOFW305011}",
"message", "%{CISCOFW313001_313004_313008}",
"message", "%{CISCOFW313005}",
"message", "%{CISCOFW402117}",
"message", "%{CISCOFW402119}",
"message", "%{CISCOFW419001}",
"message", "%{CISCOFW419002}",
"message", "%{CISCOFW500004}",
"message", "%{CISCOFW602303_602304}",
"message", "%{CISCOFW710001_710002_710003_710005_710006}",
"message", "%{CISCOFW713172}",
"message", "%{CISCOFW733100}"
]
}
# GeoIP for the ASA
# Source
if [src_interface] == "Outside" {
geoip {
source => "src_ip"
target => "geoip"
database =>"/opt/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float" ]
add_tag => [ ‘outside_in’ ]
}
} # end source block
# Destination
##if [src_interface] !~ "(Inside)|(Bypass)" {
if [src_interface] == "Inside" {
geoip {
source => "dst_ip"
target => "geoip"
database =>"/opt/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float" ]
add_tag => [ ‘inside_out’ ]
}
}
} # end ASA
} # end filter block
output {
stdout { }
elasticsearch {
cluster => "logserver"
host => "172.22.225.76"
}
}
[/code]
14. Some final thoughts and hints.
Here are some commands I used for debugging purposes.
[code]
lsof -i4
lsof -i6
netstat -l
curl -XGET http://172.22.225.76:9200/_cluster/health
[/code]
Last is a helper script I created.
[code]
cat /usr/local/bin/elkstack-restart
#!/bin/bash
systemctl restart elasticsearch
systemctl restart logstash
systemctl restart kibana
[/code]
Sources in no particular order:
http://www.networkassassin.com/elk-stack-for-network-operations-reloaded/
https://community.ulyaoth.net/threads/how-to-create-a-logstash-geoip-based-dashboard-in-kibana-3.29/
http://kartar.net/2014/09/when-logstash-and-syslog-go-wrong/
http://blog.domb.net/?p=367
http://grokdebug.herokuapp.com/patterns#
http://blog.stevenmeyer.co.uk/2014/06/add-configuration-test-to-logstash-service-configtest.html
How To Disable IPv6 On CentOS 7 / RHEL 7
http://slacklabs.be/2012/04/02/force-Elastic-search-on-ipv4-debian/
