Using Logstash with Zabbix

Logstash is a tool for receiving, processing and outputing logs and other data streams.
Our focus here is how output logs to Zabbix with Logstash Zabbix output plugin on specific events.

Installation

Plugin is not installed by default so you must install them. It’s easy to install it with commands:

gotanbl:~>bin/logstash-plugin install logstash-output-zabbix

Logstash bin files are in /usr/share/logstash/bin folder.
You could check installed plugin with:

gotanbl:~>bin/logstash-plugin list

Description

The Zabbix output is used to send item data (key/value pairs) to a Zabbix server. The event @timestamp will automatically be associated with the Zabbix item data.

Be careful not to flood your Zabbix server with too many events per second.

Zabbix output configuration options

multi_value
Use the multi_value directive to send multiple key/value pairs. This can be thought of as an array, like:
[ zabbix_key1, zabbix_value1, zabbix_key2, zabbix_value2, … zabbix_keyN, zabbix_valueN ]
Where zabbix_key1 is an instance of zabbix_key, and zabbix_value1 is an instance of zabbix_value. If the field referenced by any zabbix_key or zabbix_value does not exist, that entry will be ignored.
This directive cannot be used in conjunction with the single-value directives zabbix_key and zabbix_value.

timeout
The number of seconds to wait before giving up on a connection to the Zabbix server. This number should be very small, otherwise delays in delivery of other outputs could result.

zabbix_host
The field name which holds the Zabbix host name. This can be a sub-field of the @metadata field.

zabbix_key
A single field name which holds the value you intend to use as the Zabbix item key. This can be a sub-field of the @metadata field. This directive will be ignored if using multi_value.
zabbix_key is required if not using multi_value.

zabbix_server_host
The IP or resolvable hostname where the Zabbix server is running.

zabbix_server_port
The port on which the Zabbix server is running(default is 10051).

zabbix_value
The field name which holds the value you want to send. This directive will be ignored if using multi_value.
Default value is message(whole message).

Use case

I want to know which user is blocked on webmail of my mail server, after 5 unsuccessfull attempts to login.
To monitor this I use little program, fail2ban. Log from program is located at /var/log/fail2ban.log and exported to Logstash for further processing. For exporting use one of this: rsyslog, syslog-ng, filebeat, fluentd …

Setup Logstash

Logstash has three pipelines: input, filter and output for processing data.
Input
I export logs with rsyslog(no need to install additional packages) and input pipeline is:

input {
  tcp {
    port => 4156
    type => syslog
    tags => [ "fail2ban" ]
  }
}

Filter
After receive logs, next step is to filter it. I want to analyze only logs that contain Ban, Unban or Found word. Other logs wil be dropped.

filter {
    if "fail2ban" in [tags] {
        if [message] =~ /^.*(Ban|Unban|Found).*$/ {
            grok {
                patterns_dir => [ "/etc/logstash/conf.d/patterns/" ]
                match => { "message" => "<%{INT:sourceid}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{HOSTNAME:hostname} %{FAIL2BAN}" }
                remove_field => [ "sourceid", "syslog_timestamp", "message" ]
            }
            if [fail2ban_jail] == "roundcube-iredmail" and [banned_ip] =~ /.*$/ {
                elasticsearch {
                    hosts => ["elk:9200", "elk2:9200", "elk3:9200"]
                    index => "mail-%{+YYYY.MM}"
                    query_template => "/etc/logstash/conf.d/fail2ban_user.json"
                    fields => { "roundcube_user" => "blocked_user" }
                }
                mutate {
                    add_tag => [ "zabbix-webmail-blocked-user" ]
                    add_field => { "[@metadata][fail2banip_key]" => "bannedip" }
                    add_field => { "[@metadata][fail2banuser_key]" => "banneduser" }
                }
            }
        }
        else {
            drop {}
        }
    }
}

I use conditional statement that select messages with field [fail2ban-jail] that catch unsuccessfull login(roundcube-iredmail) and field [banned_ip] that contains from which address user try to login.
Because fail2ban doesn’t contain information about user, section start with elasticsearch(predifined Logstash filter) find which user is trying regarding ip address, field [banned_ip].
We want to export this information to Zabbix, so we need field for zabbix_key with values that we use for creating items.

add_field => { "[@metadata][fail2banip_key]" => "bannedip" }
add_field => { "[@metadata][fail2banuser_key]" => "banneduser" }

zabbix_key: bannedip and banneduser must be zabbix key value at Zabbix side when creating items

Also, add tag for this event for simplier output processing.

Output

if "zabbix-webmail-blocked-user" in [tags] {
    zabbix {
      zabbix_server_host => "IP_ADDRESS_OF_ZABBIX_SERVER"
      zabbix_host => "FIELD_THAT_CONTAIN_HOSTNAME"
      multi_value => ["[@metadata][fail2banip_key]", "[banned_ip]", "[@metadata][fail2banuser_key]", "[blocked_user]"]
    }
  }

For multi_value, order is:

• field that contain information about zabbix_key([@metadata][fail2banip_key] -> value bannedip)
• field that contain information about zabbix_value for first zabbix_key ([bannedip] -> value ip address of user)
• field that contain information about second zabbix_key([@metadata][fail2banuser_key] -> value banneduser)
• field that contain information about zabbix_value for second zabbix_key ([banned] -> value name of the user)

If some key or value is missing, processing is ignored for that

Setup Zabbix Server

We need to configure items on Zabbix side that could receive events from Logstash.

Item must be of type Zabbix trapper

When events occur, we have information at Latest Data.