Shipping with Rsyslog

• Rsyslog over TLS
• Rsyslog with SELinux
• Automatic configuration
• Manual configuration

Shipping with Rsyslog

Most Unix systems these days come with pre-installed rsyslog, which is a great lightweight service to consolidate logs.

You can configure rsyslog to monitor a single log file or directory and ship them over to Layerlog over TLS. For directories, all the first level files are monitored.

Configuration

Before you begin, you’ll need:

• Sudo access
• Rsyslog version 5.8.0 and above
• Outgoing TCP traffic to destination port 5001 allowed
• A common linux distribution

Install the rsyslog-gnutls add-on

On your Debian or Ubuntu Linux machine, run:

sudo apt-get install rsyslog-gnutls

For distributions based on Red Hat, use rpm or yum in place of apt-get.

Download the Layerlog public certificate to your credentials server

For HTTPS shipping, download the Layerlog public certificate to your certificate authority folder.

sudo curl https://raw.githubusercontent.com/logzio/public-certificates/master/AAACertificateServices.crt --create-dirs -o /etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt

Configure rsyslog file spooling

To ship a log to Layerlog, SSH to your Linux server, and run the command below. The code verifies the working directory exists. For an Ubuntu server, it will set the proper permissions.

sudo mkdir -v /var/spool/rsyslog 
if [ "$(lsb_release -ds | grep Ubuntu)" != "" ]; then
    sudo chown -R syslog:adm /var/spool/rsyslog
fi

Create a new configuration file for Layerlog

sudo vim /etc/rsyslog.d/21-logzio-sample.conf

Add the additional configuration:

#   -------------------------------------------------------
#        File Logging Directives for Layerlog
#   -------------------------------------------------------

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/spool/rsyslog

# File access file:
$InputFileName <<PATH_TO_FILE>>
$InputFileTag TYPE:
$InputFileStateFile stat-TYPE
$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

$DefaultNetstreamDriverCAFile /etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt
$ActionSendStreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer *.layerlog.com

$template logzFormatFileTagName,"[<<LOG-SHIPPING-TOKEN>>] <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [type=<<TYPE>>] %msg%\n"
if $programname == 'TYPE' then @@<<LISTENER-HOST>>:5001;logzFormatFileTagName
if $programname == 'TYPE' then ~

Replace the placeholders to match your specifics. (They are indicated by the double angle brackets << >>):

• Replace <<LOG-SHIPPING-TOKEN>> with the token of the account you want to ship to.

• Replace <<LISTENER-HOST>> with the host for your region. For example, listener.layerlog.com if your account is hosted on AWS US East, or listener-nl.layerlog.com if hosted on Azure West Europe.
• <<PATH_TO_FILE>>: Path to your file or directory.
• <<TYPE>>: Declare your log type for parsing purposes. Layerlog applies default parsing pipelines to the following list of built-in log types. If you declare another type, contact support for assistance with custom parsing. Can’t contain spaces.

Restart rsyslog

sudo service rsyslog restart

Check Layerlog for your logs

Give your logs some time to get from your system to ours, and then open Kibana.

If you still don’t see your logs, see our rsyslog troubleshooting guide.

Shipping with Rsyslog and SELinux

Security-Enhanced Linux (SELinux) is a security architecture for Linux based systems that allows administrators to have more control over who can access the system.

In systems where SELinux is enabled, rsyslog is one of the system processes that SELinux protects. One of the ways SELinux protects the service is by allowing it to only send logs using the standard port, which is 514 UDP. To be able to ship logs to Layerlog, you’ll need to modify the current SELinux policy to allow shipping logs using the non-standard port 5000 TCP.

Modify the SELinux policy to allow shipping logs to Layerlog

Before you begin, you’ll need:

• Sudo access
• Rsyslog version 5.8.0 and above
• Outgoing TCP traffic to destination port 5000 allowed
• A common linux distribution

The commands presented here are for Red Hat/Fedora-based distributions. Depending on which distribution you’re using, you may need to modify the commands slightly.

Ensure that logs are not being sent to Layerlog from your system

Execute the command below to check if your system sends logs to Layerlog.

The following defaults are assumed:

• Log location - /var/log/
• Log type - syslog

curl -sLO https://github.com/logzio/logzio-shipper/raw/master/dist/logzio-rsyslog.tar.gz \
  && tar xzf logzio-rsyslog.tar.gz \
  && sudo rsyslog/install.sh -t linux -a "<<LOG-SHIPPING-TOKEN>>" -l "<<LISTENER-HOST>>"

Replace <<LOG-SHIPPING-TOKEN>> with the token of the account you want to ship to.

Replace <<LISTENER-HOST>> with the host for your region. For example, listener.layerlog.com if your account is hosted on AWS US East, or listener-nl.layerlog.com if hosted on Azure West Europe.

If you do not see logs on your Kibana dashboard, proceed to the next step.

Check the audit logs for incoming logs

Navigate to the directory or to your audit logs (by default, this is /var/log/audit/audit.log) and check for AVC records containing dest=5000. If you see them, this means that SELinux is blocking the logs export over port TCP 5000.

Ensure that logs are sent to Layerlog when SELinux is disabled

Temporarily disable SELinux by running this command:

$ sudo setenforce 0

Wait a few minutes and check your Kibana dashboard. If you see the logs, it means that the problem is caused by SELinux and using port 5000 TCP to send logs.

Enable SELinux

At this point, switch SELinux back on to ensure that the system remains protected.

To do this, run:

$ sudo setenforce 1

Install the SELinux policy core utilities package

To enable modifications to the SELinux policy, add the package policycoreutils, which includes the SELinux command semanage.

If you already have this package installed, proceed to the next step. Otherwise, run:

$ sudo dnf install policycoreutils

Add port 5000 TCP to the SELinux policy

Execute the following command to add port 5000 TCP to the SELinux policy:

$ sudo semanage port -m -t syslogd_port_t -p tcp 5000

Check your dashboard for logs

Give your logs some time to get from your system to ours, and then open Kibana.

If you still don’t see your logs, see log shipping troubleshooting.

Shipping with Rsyslog

Most Unix systems these days come with pre-installed rsyslog, which is a great lightweight service to consolidate logs.

You can configure rsyslog to monitor a single log file or directory, and ship them over to Layerlog. For directories, all the first level files are monitored.

Automatic configuration

Before you begin, you’ll need:

• Sudo access
• Rsyslog version 5.8.0 and above
• Outgoing TCP traffic to destination port 5000 allowed
• A common linux distribution

Configure your rsyslog daemon

If your are monitoring plain-text logs

Run the following to configure your rsyslog daemon to monitor a log file or directory.

curl -sLO https://github.com/logzio/logzio-shipper/raw/master/dist/logzio-rsyslog.tar.gz && tar xzf logzio-rsyslog.tar.gz && sudo rsyslog/install.sh -t file -a "<<LOG-SHIPPING-TOKEN>>" -l "<<LISTENER-HOST>>" --filepath "<<PATH_TO_FILE>>" -tag "<<TYPE>>" 

If you are monitoring JSON logs

Run the following command to configure your rsyslog daemon to monitor JSON log files. Each log should be formatted as a single JSON line that ends with a newline character.

curl -sLO https://github.com/logzio/logzio-shipper/raw/master/dist/logzio-rsyslog.tar.gz && tar xzf logzio-rsyslog.tar.gz && sudo rsyslog/install.sh -t file -a "<<LOG-SHIPPING-TOKEN>>" -l "<<LISTENER-HOST>>" --filepath "<<PATH_TO_FILE>>" -tag "<<TYPE>>" -c json

Replace the placeholders to match your specifics. (They are indicated by the double angle brackets << >>):

• Replace <<LOG-SHIPPING-TOKEN>> with the token of the account you want to ship to.

• Replace <<LISTENER-HOST>> with the host for your region. For example, listener.layerlog.com if your account is hosted on AWS US East, or listener-nl.layerlog.com if hosted on Azure West Europe.
• <<PATH_TO_FILE>>: Path to your file or directory.
• <<TYPE>>: Declare your log type for parsing purposes. Layerlog applies default parsing pipelines to the following list of built-in log types. If you declare another type, contact support for assistance with custom parsing. Can’t contain spaces.

Shipping with Rsyslog

Most Unix systems these days come with pre-installed rsyslog, which is a great lightweight service to consolidate logs.

You can configure rsyslog to monitor a single log file or directory, and ship them over to Layerlog. For directories, all the first level files are monitored.

Manual configuration

Before you begin, you’ll need:

• Sudo access
• Rsyslog version 5.8.0 and above
• Outgoing TCP traffic to destination port 5000 allowed
• A common linux distribution

Configure rsyslog file spooling

To ship a log to Layerlog, SSH to your Linux server, copy the code snippet below to your terminal window and execute it. The code verifies that the working directory exists. For an Ubuntu server, it sets the proper permissions.

sudo mkdir -v /var/spool/rsyslog
if [ "$(lsb_release -ds | grep Ubuntu)" != "" ]; then
    sudo chown -R syslog:adm /var/spool/rsyslog
fi

Create a new configuration file for Layerlog

sudo vim /etc/rsyslog.d/21-logzio-sample.conf

Add the additional configuration

#   -------------------------------------------------------
#        File Logging Directives for Layerlog
#   -------------------------------------------------------

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/spool/rsyslog

# File access file:
$InputFileName <<PATH_TO_FILE>>
$InputFileTag TYPE:
$InputFileStateFile stat-TYPE
$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

$template logzFormatFileTagName,"[<<LOG-SHIPPING-TOKEN>>] <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [type=TYPE] %msg%\n"
if $programname == 'TYPE' then @@<<LISTENER-HOST>>:5000;logzFormatFileTagName
if $programname == 'TYPE' then ~

Replace the placeholders to match your specifics. (They are indicated by the double angle brackets << >>):

• Replace <<LOG-SHIPPING-TOKEN>> with the token of the account you want to ship to.

• Replace <<LISTENER-HOST>> with the host for your region. For example, listener.layerlog.com if your account is hosted on AWS US East, or listener-nl.layerlog.com if hosted on Azure West Europe.
• <<PATH_TO_FILE>>: Path to your file or directory.
• <<TYPE>>: Declare your log type for parsing purposes. Layerlog applies default parsing pipelines to the following list of built-in log types. If you declare another type, contact support for assistance with custom parsing. Can’t contain spaces.

Restart rsyslog

After editing and saving the file, execute the following command:

sudo service rsyslog restart

Check Layerlog for your logs

Give your logs some time to get from your system to ours, and then open Kibana.

If you still don’t see your logs, see our rsyslog troubleshooting guide.