Setting up Splunk with Syslog-ng and a FIFO

Assumptions ¶
* You have a supported Splunk platform and root access to the server.
* You are installing Splunk 3.3.x or higher.
The facts at abc company ¶
What is a FIFO? ¶
A FIFO is an old programming term that stands for First In First Out. A more accurate name for what you're setting up is a Named Pipe. Basically, what it is, is a special file on your UNIX system that receives input from some program and sends that information to another program. You can think of it as a stack. One program puts things on the stack, and another program takes stuff off the stack, but it starts with the first item that went on.
Why Syslog-ng? ¶
Well basically because the standard syslog daemon sucks. Seriously, the standard syslog daemon that comes with your typical UNIX is very limited. Syslog-ng is very flexible and includes support for many things that the standard syslog daemon does not, including, but not limited to: multiple sources and destinations, extensive filtering including regex support, the ability to read and write to pipes, and much more. It's also extremely efficient on system resources.
Syslog-ng installation ¶
Depending on your platform, the procedure for installing syslog-ng may vary widely. On Linux platforms, you will most likely be using RPM or Debs. On Solaris, it's quite likely that Sun Freeware will have a package that will work for you.
Examples for common platforms ¶
redhat variant# yum install syslog-ng

debian variant# apt-get install syslog-ng

gentoo# emerge syslog-ng

Run this command to install syslog-ng to abc company's servers:
server# yum install syslog-ng

To have the latest version, see Appendix F - Upgrading Syslog-ng.
Syslog deactivate ¶
Whistle syslog and syslog-ng are programs that have same purposes, please deactivate syslog before using the syslog-ng:
/etc/rc.d/init.d/syslog stop
chmod 644 /etc/rc.d/init.d/syslog
chkconfig syslog off
rm -f /etc/logrotate.d/syslog

Creating the FIFO ¶
Select a place on your filesystem for the FIFO to live. It doesn't matter much where as the FIFO will only buffer a limited amount of lines that will consume an insignificant amount of resources. I would suggest /var as that is where most sockets and pipes on the system are located by default. I gave mine a subdirectory called syslog-ng for the sake of tidiness.
Example command to create the FIFO: ¶
server# mkfifo /opt/syslog_fifo

Remember this location as you will need to put it in your syslog-ng and splunk config files.
Configuring syslog-ng ¶
In most installs of syslog-ng, your main configuration file will be located in /usr/local/etc/syslog-ng.conf (or /etc/syslog-ng.conf). At a minimum, you will need to set up a source that accepts remote connections [assuming that you wish to send logs from multiple systems to be included in your Splunk index], and a destination for your FIFO. It would also be wise to log to a location on the filesystem as well. Here's the first bit:
source remote {
udp();
};

destination splunk {
pipe("/opt/syslog_fifo");
};

log {
source(remote);
destination(splunk);
};

That will be sufficient to send the logs only to your named pipe. However, I would recommend that you also set up syslog-ng to log to a location on the local filesystem like so:
destination hosts {
file("/opt/hosts/$HOST/messages"
owner(root) group(logs) perm(0640) dir_perm(0750) create_dirs(yes));
};

log {
source(remote);
destination(hosts);
};

Note that the $HOST moniker is a magic variable in syslog-ng that is replaced by the hostname of the system that is sending the logs. It's quite handy. The create_dirs bit automatically creates the subdirectories if they do not exist.
See Apendix A for the initial configuration of syslog-ng at abc company.
Configuring Splunk ¶
For this portion, I will assume that Splunk was installed in it's default location of /opt. If you have installed in a different location, adjust the paths accordingly.
inputs.conf (3.x.x) ¶
Splunk Init script ¶
As root, you run:
splunk enable boot-start

This will create an init script (or other configuration change) appropriate for your operating system.
Reference:
http://www.splunk.com/doc/latest/installation/ConfigSystemStartup
http://www.splunk.com/doc/latest/installation/RunningNonRoot
Making a secondary syslog-ng server ¶
Create another instance of syslog-ng reduces the risk of losting log files.
Installing a secondary server is not different to installing the primary, except for the configuration file:

Primary server
Secondary server
* Destination directive:
d_local
file("/opt/syslog-ng/192.168.XX.YY/$YEAR/$MONTH/$DAY/$FACILITY.log" owner("root") group("root") perm(0640) dir_perm(0750) create_dirs(yes));
file("/opt/syslog-ng/$YEAR/$MONTH/$DAY/192.168.XX.YY/$FACILITY.log" owner("root") group("root") perm(0640) dir_perm(0750) create_dirs(yes));
* Destination directive:
d_separatedbyhosts
file("/opt/syslog-ng/$HOST/$YEAR/$MONTH/$DAY/$FACILITY.log" owner("root") group("root") perm(0640) dir_perm(0750) create_dirs(yes));
file("/opt/syslog-ng/$YEAR/$MONTH/$DAY/$HOST/$FACILITY.log" owner("root") group("root") perm(0640) dir_perm(0750) create_dirs(yes));
Forward to Splunk
Yes
Currently No
Forward the local log files
N/A
Forward from the secondary server to the primary server.
Debugging FIFOs ¶
If you are having problems with your FIFO, the first thing that you should do is to verify that it it getting data put into it by syslog-ng. I have written a small tool called Piper that can help you do this. Download Piper and run the following command:
./piper.pl -d debug.log /opt/syslog_fifo

Then check your debug.log to make sure that it contains data. If it does contain data, then syslog-ng and the FIFO that you created are working properly and you should re-examine your Splunk config. If it looks correct, contact Splunk support for assistance. If your debug.log does not contain any data, then something is most likely wrong with your syslog-ng config. First try restarting syslog-ng to make sure that your changes have applied. If that doesn't do the trick, examine your syslog-ng configuration closely and make sure that you didn't miss any semicolons or other necessary syntax.
Apendix A – The configuration of syslog-ng for the primary syslog-ng server at abc company ¶

Apendix B – daily crond script to compress log file ¶
[root@syslog ~]# cat /etc/cron.daily/syslog-ng
#!/bin/sh

/usr/bin/find /opt/syslog-ng ! -name "*bz2" -type f ! -path "*`/bin/date +%Y/%m/%d`*" -exec /usr/bin/bzip2 {} \;

Caution.
Make sure that this script can be executable.
Apendix B.1 – logrotate script of local syslog files ¶
[root@syslog ~]# cat /etc/logrotate.d/syslog-ng
/var/log/messages /var/log/secure /var/log/maillog /var/log/spooler /var/log/boot.log /var/log/cron /var/log/kern /opt/syslog-ng-internal/syslog-ng.log {
monthly
rotate 12
compress
postrotate
/etc/rc.d/init.d/syslog-ng reload 2>/dev/null
endscript
}

Caution.
Remember always disable the logrotate script /etc/logrotate.d/syslog associated with syslog program by commenting all commands or removing this file.
Apendix B.2 – logrotate script of Splunk log files ¶
By default, Splunk does not rotate some log files. So this script below will rotate logs that have too many lines.
[root@syslog ~]# touch /etc/logrotate.d/splunk
[root@syslog ~]# vim /etc/logrotate.d/splunk
[root@syslog ~]# cat /etc/logrotate.d/splunk
/opt/splunk/var/log/splunk/splunkd.log /opt/splunk/var/log/splunk/audit.log /opt/splunk/var/log/splunk/splunklogger.log /opt/splunk/var/log/splunk/python.log /opt/splunk/var/log/splunk/license_audit.log /opt/splunk/var/log/splunk/web_service.log {
missingok
monthly
rotate 12
create 0600 splunk splunk
}

Apendix C – the configuration of syslogd on Linux servers ¶
System messages on Linux server are logged to syslog-ng server by syslogd daemon. The main configuration file of syslogd is located at /etc/syslog.conf.
Here is a sample of syslog.conf file on Linux servers at abc company:
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages

# The authpriv file has restricted access.
authpriv.* /var/log/secure

# Log all the mail messages in one place.
mail.* -/var/log/maillog

# Log cron stuff
cron.* /var/log/cron

# Everybody gets emergency messages
*.emerg *

# Save boot messages also to boot.log
local7.* /var/log/boot.log

#
# log messages to syslog-ng servers
*.* @192.168.XX.YY
*.* @192.168.XX.YY

Important:
- Do not use syslogd on syslog-ng server. This will prevent syslog-ng from receiving log messages.
- Verify that the SSH daemon use LogFacilitate? = authpriv & LogLevel? = VERBOSE.
Apendix D – logging Windows applications using Datagram SyslogAgent? ¶
Unfortunately all Windows OSes does not support syslog protocol. They are only have Event Viewer, which stores all messages created by themselves. The event tool also receives messages from some applications supported to do this. To make the syslog server collect messages from Windows-based clients, we (DongA) use SyslogAgent from Datagram. This software is licenced under GNU GPL.
Download this software with the URI: http://download.eab.com.vn/pub/windows/server-installation/syslogagent_setup.zip. Read the setup-guide.txt on the zip file to install this software.
Merge these registries below to configure:
Windows Registry Editor Version 5.00

# Configure Syslog Agent parameters

[HKEY_LOCAL_MACHINE\SOFTWARE\Datagram]

[HKEY_LOCAL_MACHINE\SOFTWARE\Datagram\SyslogAgent]
"UsePingBeforeSend"=hex:00
"Syslog"="192.168.XX.YY"
"SendToPort"=dword:00000202
"ForwardToMirror"=hex:01
"Syslog1"="192.168.XX.YY"
"SendToBackupPort"=dword:00000202
"ForwardEventLogs"=hex:01
"ForwardApplicationLogs"=hex:01
"EventLogPollInterval"=dword:00000002
"LastRun"=dword:00000001
"EventIDFilterList"="1,1000,1001,17,4321,6013,7001,7035,7036,7040,4786,515,528,610,560"
"TCPDelivery"=hex:00
"CarrigeReturnReplacementCharInASCII"=dword:00000020
"LineFeedReplacementCharInASCII"=dword:00000020

[HKEY_LOCAL_MACHINE\SOFTWARE\Datagram\SyslogAgent\Application]
"Information"=dword:00000001
"Information Priority"=dword:000000be
"Warning"=dword:00000001
"Warning Priority"=dword:000000bc
"Error"=dword:00000001
"Error Priority"=dword:000000bb
"Audit Success"=dword:00000001
"Audit Success Priority"=dword:000000be
"Audit Failure"=dword:00000001
"Audit Failure Priority"=dword:000000bd

[HKEY_LOCAL_MACHINE\SOFTWARE\Datagram\SyslogAgent\ApplicationLogs]

[HKEY_LOCAL_MACHINE\SOFTWARE\Datagram\SyslogAgent\Security]
"Information"=dword:00000001
"Information Priority"=dword:00000026
"Warning"=dword:00000001
"Warning Priority"=dword:00000024
"Error"=dword:00000001
"Error Priority"=dword:00000023
"Audit Success"=dword:00000001
"Audit Success Priority"=dword:00000026
"Audit Failure"=dword:00000001
"Audit Failure Priority"=dword:00000025

[HKEY_LOCAL_MACHINE\SOFTWARE\Datagram\SyslogAgent\System]
"Information"=dword:00000001
"Information Priority"=dword:0000001e
"Warning"=dword:00000001
"Warning Priority"=dword:0000001c
"Error"=dword:00000001
"Error Priority"=dword:0000001b
"Audit Success"=dword:00000001
"Audit Success Priority"=dword:0000001e
"Audit Failure"=dword:00000001
"Audit Failure Priority"=dword:0000001d


# make the Syslog Agent service start automatically

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Syslog Agent]
"DisplayName"="Syslog Agent"
"Description"="Forwards Event logs to Syslog Server"
"Type"=dword:00000010
"Start"=dword:00000002
"ErrorControl"=dword:00000001
"DependOnService"=hex(7):45,00,76,00,65,00,6e,00,74,00,4c,00,6f,00,67,00,00,00,\
00,00
"DependOnGroup"=hex(7):00,00

To filter out some messages, fill the EventIDFilterList parameter with event Ids, e.g.:
"EventIDFilterList"="1,1000,1001,17,4321,6013,7001,7035,7036,7040,4786,515,528,610,560"

To enable mirror delivery, make sure the clients have these registries below:
Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Datagram\SyslogAgent]
"UsePingBeforeSend"=hex:00
"ForwardToMirror"=hex:01
"Syslog1"="192.168.2.23"
"SendToBackupPort"=dword:00000202

Apendix E – Upgrading Syslog-ng ¶
Stop the 'monit' service
/etc/init.d/monit stop

Stop syslog-ng
/etc/init.d/syslog-ng stop