At the Circus we have been having random reboots and are struggling to figure out the culprit as no one can seem to find any errors in the logs. In order to help narrow the scope I wrote this script to help us hunt down the culprit.
The script pings a number of infrastructure services such as gateways, ESXi hosts and VMs. I run it every minute from one of the linux servers.
#!/bin/bash
# 2017-11-13
# Jud Bishop
# This script pings a number of infrastructure address to try and figure out
# what is going wrong with infrastructure rebooting.
#
# Just run this command to follow the ping sequence:
# grep ping-test /var/log/messages
readonly SCRIPT_NAME=$(basename $0)
log() {
echo "$@"
logger -p user.notice -t $SCRIPT_NAME "$@"
}
err() {
echo "$@" >&2
logger -p user.error -t $SCRIPT_NAME "$@"
}
for I in `cat /usr/local/bin/ping-test-servers.txt`
do
OUT="$(ping -c1 -W1 $I | egrep 'bytes\ from|loss')"
log "${OUT}"
done
The file /usr/local/bin/ping-test-servers.txt is just a list of IP addresses or server names.
192.168.1.1 192.168.1.100 dns.chainringcircus.org vm.chainringcircus.org physical.chainringcircus.org
Here is the crontab entry, I run it once a minute.
[linuxserver]# crontab -l # m h dom mon dow command * * * * * /usr/local/bin/ping-test.sh
Here is a sample of the output in the log files.
Nov 13 13:53:01 ns1 ping-test.sh: 64 bytes from dns.chainringcircus.org (192.168.1.1): icmp_seq=1 ttl=127 time=0.233 ms#0121 packets transmitted, 1 received, 0% packet loss, time 0ms Nov 13 13:53:01 ns1 ping-test.sh: 64 bytes from vm.chainringcircus.org (192.168.1.2): icmp_seq=1 ttl=128 time=0.243 ms#0121 packets transmitted, 1 received, 0% packet loss, time 0ms Nov 13 13:53:01 ns1 ping-test.sh: 64 bytes from physical.chainringcircus.org (192.168.1.3): icmp_seq=1 ttl=127 time=0.208 ms#0121 packets transmitted, 1 received, 0% packet loss, time 0ms
ChainRingCircus 