Centralized Fail2Ban

Fail2Ban is one of the most widely used systems to stop attacks on Linux servers by scanning log files and blocking IP addresses in the firewall from the results. However, Fail2Ban is designed to run locally on a single server, but can easily be expanded to a centralized Fail2Ban system where all servers in a group can share a common list of IP addresses that need to be blocked.

Instead of being reactive, you get a proactive system, which is very useful as the same attacker often attacks many servers.

A centralized Fail2Ban system is easy to build for the handy. What we need is a centrally located database, something that triggers an update of the database, and something that retrieves data and blocks in the firewall. In a safe way in all parts.

The example described here uses an RHEL 8 server (which should also work with all clones based on RHEL) and a MySQL database, but the example should not be a major problem to “translate” to other systems. What is described for Fail2Ban should be general for all Linux distributions.

NOTE! What is described here is not a complete key-in-hand system, you need to handle all security around this yourself and also adapt sample code etc. for your specific environment and your specific needs. If you have SELinux enabled (which you should definitely have in your production systems) then you probably need to make adjustments there to make everything work.

Ps. We are experts in safety, contact us when you need help!

1. Create a database (only on the master server)

Install MySQL according to taste and liking, create database, table, users and scheduler as below, change name and password according to what you want to use:

CREATE DATABASE fail2ban;

CREATE TABLE `fail2ban` (
  `id` bigint(20) UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
  `hostname` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
  `created` datetime NOT NULL,
  `name` text COLLATE utf8_unicode_ci NOT NULL,
  `protocol` varchar(16) COLLATE utf8_unicode_ci NOT NULL,
  `port` varchar(32) COLLATE utf8_unicode_ci NOT NULL,
  `ip` varchar(64) COLLATE utf8_unicode_ci NOT NULL,
  `client` varchar(64) COLLATE utf8_unicode_ci NOT NULL,
  INDEX `hostname` (`hostname`,`ip`),
  INDEX `client` (`client`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;

CREATE USER 'fail2ban'@'localhost' IDENTIFIED BY 'fail2ban';
GRANT ALL PRIVILEGES ON fail2ban.* TO 'fail2ban'@'localhost';
CREATE USER 'fail2ban'@'%' IDENTIFIED BY 'fail2ban';
GRANT ALL PRIVILEGES ON fail2ban.* TO 'fail2ban'@'%';
FLUSH PRIVILEGES;

SET GLOBAL event_scheduler = ON;

SHOW VARIABLES WHERE VARIABLE_NAME = 'event_scheduler';

CREATE EVENT `cleaning` ON SCHEDULE EVERY 1 DAY ON COMPLETION PRESERVE ENABLE
  DO
  DELETE FROM fail2ban WHERE `created` < NOW() - INTERVAL 1 MONTH;

The scheduler is used to keep the database reasonably tidy by deleting old unnecessary posts that are saved for eg statistics or function check, change according to your own needs.

2. Update FirewallD

Make sure not to be accidentally locked out by creating a whitelisted zone with your IP(s) that you use to log in to the systems. Do this on all servers that are to be included in the Fail2Ban group.

firewall-cmd --permanent --new-zone=000-trusted
firewall-cmd --set-target=ACCEPT --zone=000-trusted --permanent
firewall-cmd --permanent --zone=000-trusted --add-source=123.123.123.123
firewall-cmd --reload
firewall-cmd --zone=000-trusted --list-all

Open a special port to receive updates and data downloads. Only on the master server.

firewall-cmd --permanent --add-port=49153/tcp
firewall-cmd --reload

3. Create a virtual website (only on the master server)

Updates and retrieval of data take place via a special website that listens to port 49153. It does not matter which web server or domain address is used as long as it is secured with a certificate and supports dynamic content. In this example we use PHP.

IMPORTANT! You should also create an access list in the site configuration to allow only servers in the Fail2Ban group (including the master server) and keep the rest of the world out. Do this in Apache and Nginx.

index.php

Receives updates. To protect the blocking of your own IP addresses, each server, including the master host, is added to the $ whitelisted [] array. This array must be updated if servers are added or removed from the group. We only allow POST calls for security reasons.

<?php
ini_set('display_errors', 1);
ini_set('display_startup_errors', 1);
error_reporting(E_ALL);

// block registration array of own hosts used in group
$whitelisted[] = '123.123.123.123';	// host1.my.domain

$client = (!empty ($_SERVER['REMOTE_ADDR'])) ? addslashes($_SERVER['REMOTE_ADDR']) : '?';
$name = (!empty($_POST['name'])) ? addslashes($_POST['name']) : die('no valid input');				// jail name
$protocol = (!empty($_POST['protocol'])) ? addslashes($_POST['protocol']) : die('no valid input');	// blocked service
$port = (!empty($_POST['port'])) ? addslashes($_POST['port']) : die('no valid input');				// blocked port
$ip = (!empty($_POST['ip'])) ? addslashes($_POST['ip']) : die('no valid input');					// blocked IPv4
$hostname = (!empty($_POST['hostname'])) ? addslashes($_POST['hostname']) : die('no valid input');	// blocked at host

if (in_array($ip, $whitelisted)) { die('whitelisted ip'); }

$conn = new mysqli('localhost', 'fail2ban', 'fail2ban', 'fail2ban');
if ($conn->connect_error) {
  die("Connection failed: " . $conn->connect_error);
}

$query = 'INSERT INTO `fail2ban` set hostname="'.$hostname.'", name="'.$name.'", protocol="'.$protocol.'", port="'.$port.'", ip="'.$ip.'", client="'.$client.'", created=NOW()';
$result = $conn->query($query);

$conn->close();

die('ok');

?>

fetchip.php

Returns a text list with recently registered IP addresses, one per new line. We allow both GET and POST calls to allow easy testing via browsers, but POST should be used for security reasons by the system. Change all $ _REQUEST to $ _POST in the script if you do not need browser access.

<?php
ini_set('display_errors', 1);
ini_set('display_startup_errors', 1);
error_reporting(E_ALL);

// only return what other hosts have reported, caller own reports are already blocked in caller firewall
if (!empty($_REQUEST['exclude']) && preg_match("/^[a-z0-9\.]+$/i", $_REQUEST['exclude'])) {
	$exclude = $_REQUEST['exclude'];
} else {
	die('exclude is required');
}

// filter by reported time to avoid dublettes, timespan should be the same as used in cron to fetch this list, default 1 minute
$intval = (!empty($_REQUEST['intval']) && is_numeric($_REQUEST['intval'])) ? $_REQUEST['intval'] : 60;

$ips = '';

$conn = new mysqli('localhost', 'fail2ban', 'fail2ban', 'fail2ban');
if ($conn->connect_error) { die("Connection failed: " . $conn->connect_error); }
$query = 'SELECT DISTINCT ip FROM `fail2ban` WHERE hostname != "'.$exclude.'" AND created > NOW() - INTERVAL '.$intval.' SECOND;';
$result = $conn->query($query);

if ($result->num_rows > 0) {
	while($row = $result->fetch_assoc()) {
		$ips .= $row["ip"]."\n";
	}
}

$conn->close();

header('Content-Type:text/plain');
header('Content-Transfer-Encoding: binary');
header('Cache-Control: must-revalidate');
header('Pragma: public');
die($ips);

?>

IMPORTANT! Test everything, thoroughly, so that you are absolutely sure that everything works as intended!

4. Let Fail2Ban make updates (all servers in the group)

Create a new action in the “./fail2ban/action.d” folder and add banaction to the jail configuration.

central_fail2ban.conf

# Fail2Ban configuration file
#
# Author: Erik Schütten <erik@scab.ax>
#
# Send banned IP and attributes to central fail2ban database
# Require "cf2b_host" var as database host full URL set in jail.local
#
# action: central_fail2ban[cf2b_host="%(cf2b_host)s", port="%(port)s", protocol="%(protocol)s"]
#

[Definition]

norestored = 1
actionstart = 
actionstop =
actioncheck =
actionban = curl -s -d "name=f2b-<name>&protocol=<protocol>&port=<port>&ip=<ip>&hostname=<fq-hostname>" -X POST <cf2b_host>
actionunban =

[Init]
port = 1:65535
protocol = tcp

jail.local

...
cf2b_host  = https://master.my.tld:49153
...
action_mw = %(banaction)s[name=%(__name__)s, ...
            %(mta)s-whois[name=%(__name__)s, ...
            central_fail2ban[cf2b_host="%(cf2b_host)s", port="%(port)s", protocol="%(protocol)s"]

Restart the fail2ban service. Wait with the following steps until you are completely sure that Fail2Ban submits the correct data to the database.

5. Retrieve data and block IP (all servers in the group)

In the last step, we will retrieve data and block all returned IP addresses in the local firewall. We do this by running a Bash script in Cron.

fetchip_list.sh

NOTE! We also use a whitelist here so as not to accidentally block any IP addresses in the Fail2Ban server group. The whitelist should be a plain text file with each IP address on its own line. We use grepcidr in the script for this which should be installed.

We also log every call.

This script should be run regularly via Cron and takes some input parameters if you do not want to use default settings. Note that all input parameters must be specified if any parameters need to be changed.

<exclude_host> <fetch_interval> <ban_time>

<exclude_host> = the server script is running on to not get its own posts, such as host1.my.tld
<fetch_interval> = download interval in seconds, same as specified in Cron, default 60 seconds
<ban_time> = time setting in seconds for the block, default 86400 seconds (24h)

The script must be executable (chmod u + x /path/to/script/…)

#!/bin/bash
#
# Script made by erik@scab.ax (https://scab.ax), use at your own risk
#

# paths to honeypot whitelist and log
WHITELIST_PATH=/path/to/whitelist
LOGPATH=/var/log/fetchip_list.log

# (calling) host to exclude
if [ -z "$1" ]; then
	echo "No host supplied"
	exit 0
fi

# time range in seconds to fetch new items, should be same as cron interval for this script, default 1 minute
if [ -z "$2" ]; then
		SEARCH_RANGE=60
	else
		SEARCH_RANGE=$2
fi

# time range in seconds to block item, default 24 hours
if [ -z "$3" ]; then
		RULE_TIMEOUT=86400
	else
		RULE_TIMEOUT=$3
fi

SEARCH_RANGE=$(( $SEARCH_RANGE + 1 ))
HOST_IP=`hostname -I`

echo "$(date -Iseconds) INFO $1/fetchip.php.php?exclude=$HOSTNAME&intval=$SEARCH_RANGE call initiated" >> $LOGPATH

curl -s -d "exclude=$HOSTNAME&intval=$SEARCH_RANGE" -X POST "$1/fetchip.php.php" | while read line ; do

	if [ $HOST_IP = $line ]; then
		echo "$(date -Iseconds) INFO $HOST_IP = $line, no action" >> $LOGPATH
		continue
	fi

	if [ "$WHITELIST_PATH" ] && [ -f "$WHITELIST_PATH" ]; then
		if grepcidr -f "$WHITELIST_PATH" <(echo "$line") >/dev/null; then
			echo "$(date -Iseconds) INFO $line is whitelisted, no action" >> $LOGPATH
			continue
		fi
	fi

	firewall-cmd --zone=public --timeout=$RULE_TIMEOUT --add-rich-rule="rule family='ipv4' source address='$line' reject type='icmp-port-unreachable'"
	echo "$(date -Iseconds) BANNED $RULE_TIMEOUT IPv4 $line" >> $LOGPATH

done

exit 0

Create a new blank log file

touch /var/log/fetchip_list.log

Rotate the log files with /etc/logrotate.d

fetchip_list

/var/log/fetchip_list.log {
    missingok
    notifempty
    weekly
    rotate 4
    compress
    delaycompress
}

Add the script to Cron, we run every minute, all params are there to clarify it all.

...
* * * * * /path/to/fetchip_list.sh https://master.my.tld:49153 60 86400 >/dev/null 2>&1
...

Done! Test, test, test before you roll it out in production. Good luck!