Centralize Your System Logging Using Syslog

System logs are a critical part of system and network maintenance. They are, in fact, one of the key components for managing just about ANY system related incident. Unfortunately these logs are completely inaccessible or untrustable if a system reaches an unbootable state, suffers a catastrophic disk failure, or is compromised by an attacker. For these scenarios, and for general ease of maintenance, one of the best moves that you can make toward converting a group of machines into a manageable network is to set up remote Syslog services.

Syslog Reporting Basics

Syslog, defined by RFC 3164 and RFC 5424, is a unix logging system which has been one of the mainstays of datacenter computing since their origination. This utility/protocol set is simply an effective method of setting custom log level and log location. Beyond that, there is not a tremendous amount of description necessary beyond the switches used in configuration. Syslog specifies three types of options that can be used when configuring the daemon. understanding each of theses is critical to understanding syslog.

Syslog Utilities

Utilities are processes, applications, or services which are configured to write to syslog. These utilities, while all that are used, were proposed as guidelines for application developers and are not specified on great detail. A given application may not log at the same level as another, depending on how it was written. These utilities are currently supported:

auth – Login/Logout authentication messages
cron – Scheduler Messages and updates
daemon – Resident Daemon service messages
kern – Kernel level messages
lpr – Printer Services Messages
mail – Messages from Mail Services
user – User-initiated processes and applications
local0local7 – User-defined
syslog – Syslog process status messages
– All Utilities

Syslog Levels

Log levels indicate the severity of the reported incident,and are, again, recommended as a guidline. The following levels are defined:

0 – Emergency (emerg)
1 – Alerts (alert)
2 – Critical (crit)
3 – Errors (err)
4 – Warnings (warn)
5 – Notification (notice)
6 – Information (info)
7 – Debug (debug)
* – All

Note – Most Posix systems expect the recognized short description in configuration files (shown in parenthesis), not the number.

Syslog Actions

Once you have determined what to log and at what level, syslog will need to know where to log it. The options are listed below:

filename – Write the message to the specified file on the local machine – (you CAN specify a tty device here)
@hostname or @ipaddress – Forward the message to the specified remote loghost
user1,user2,… – Write the message to consoles of users named in list, if the user is currently logged-in
* – Write the message to all logged-in users

Note – Although not listed as an action officially, most Posix syslog apps can also log to a pipe.

Syslog Installation and Configuration

Now that we know the basics, it is time to move on to installation. Syslog is a long time Unix favorite, and is included in pretty much every Posix Operating system. In the case that syslog has not been installed by default, the best option is to use the version included in your install media or repository. In most Linux distributions, the syslog service Daemon is actually named sysklogd. Using my normal Debian example, the install would be as shown below:

apt-get install sysklogd

There is nothing else required for basic installation.

Enable Remote Syslog on the Server

Basic configuration is simple. syslog, by default, only listens locally. we can change that very easily. we will set syslog to listen to remtoe syslog submissions by editing /etc/init.d/sysklogd . We will need to find the line that reads

SYSLOGD =””

and change it to read

SYSLOGD=”-rm 0″

For reference, the r enables Remote logging and the m 0 disables the annoying –MARK– messages that are placed in the log by default..

Configuring Linux/Posix Clients

Configuring Remote Syslog Submission on Posix clients is actually very simple. There is only a single file edit on the client to configure the default settings. Syslog uses the following format for configuration files. Note the reminder that convention state that you double tab the whitespace in text config files.

utility.level action

with that stated, The log settings in the test client are best set initially at everything. We can tune this in the lab later if we want. The simple rule here is that you cannot add it later. By default you should log more liberally on the syslog server until you are confident in what you can filter out. For that, let’s set the default log server settings by adding the following line to /etc/syslog.conf:

*.* @log-server-IP-address

The only thing left is to restart syslogd. It is worth noting that BSD requires a Sighup to restart syslog.

Note – While hostname is acceptable for @logserver in the configuration file, it would be wise to log by IP address.

Installing Windows Syslog Clients

Intersect Alliance has an interesting set of utilities, including Snare Agent for Wndows which is a configurable agent or server for syslog data. Configuration for windows logging is beyond the scope of this article, but it seemed worth noting that it is available if you insist on running windows.

Configuring Logging Options on the Server

Configuring the Syslog Server follows the same format as the clients, but in this case we want to expand on the available options. Configuration uses the same utility.level<TAB><TAB>action format.

The log settings in the test client are best set initially at everything. We can tune this in the lab later if we want. The simple rule here is that you cannot add it later. By default you should log more liberally on the syslog server until you are confident in what you can filter out. For that, let’s set the default log server settings by adding the following line to /etc/syslog.conf:

kern.crit /var/lor/syslogs/kernel-critical.log
daemon.crit /var/lor/syslogs/daemon-crit.log
syslog.crit /var/lor/syslogs/syslog-crit.log
mail,news.=crit /var/lor/syslogs/mail-err.log
auth.* /dev/tty1

Make sure to create the logfiles you have specified, if they do not exist already, and chown them (chown 640 is a good choice).

Restart syslogd and test. You should start seeing log data writing to your logs.

Log Rotation

Logrotate handles automation of log rotation to ease management of logs. If your system is not already running logrotate, you will want to install it. There is no discussion there. Logrotate is included in all modern Posix Operating Systems and is install (debian again) simply from the repositories:

apt-get install logrotate

Once installed, configuration is done via the configuration file located in /etc/logrotate.conf and can be completed in a matter of minutes. This is an example of the logrotate.conf file from a default debian install.

# see “man logrotate” for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp, or btmp — we’ll rotate them here
/var/log/wtmp {
missingok
monthly
create 0664 root utmp
rotate 1
}

/var/log/btmp {
missingok
monthly
create 0660 root utmp
rotate 1
}

# system-specific logs may be configured here

We will want to add entries for your logs to the bottom of this conf file to ease management of the Syslogs, and to extend log retention periods. The additional fields are configured as shown below:

/var/log/syslogs/kernel-crit.log {
  daily
  missingok
  rotate 30
  create 640 root root
}

This can be explained simplest as shown below:

/location/filename {
daily -sets rotation frequency
missingok – specifies that the file should be created if it does not exist
rotate 30 – specifies the number of rotations to keep
compress – specifies that rotated files should be compressed to save space
delaycompress – specifies that the first rotation should not be compressed
create 640 root root – Specifies the permissions that should be set on the logfiles
sharedscripts ——————————————————————————————
/etc/init.d/<service> restart — specifies any script that shoulf be run after rotation
endscript ———————————————————————————————–
}

Note that I have left a few of the common options out of the recommendation. Once you have all of this syslog data, you will probably want to find unique ways to scan it. This will be easier at first if you do not have to uncompress the files first.

Last Recommendations

There are a few parting recommendations for a production Syslog server. These are optional, but will make maintaining the system MUCH easier.

  • Create a separate partition for syslog data, and mount it to your log directory (the examples above figure for a mount of /var/log/syslogs)
  • Remove ANY unsecure access method. My recommendation would be to limit access to ssh using key-based authentication.
  • Once you are comfortable with what you are loggin Consider following up on the syslog server by eliminating uneeded duplicates using utilies.!level<tab><tab>/default/logpath
  • Once you are confident that you are logging all you need from the clients, consider reducing the logs sent over the network using the same rules as the server, replacing the actions with the @IPaddress action. This will reduce unwanted syslog traffic considerably.

Keep in mind that, while a Syslog server will significantly reduce the time spent checking logs, no two sites will use the same Syslog setup. You will need to do some tweaking to get this exactly where you want it.