blacklist

Introduction

Since a few month programs spread which try to login to a SSH or a FTP server. SSH and FTP servers get hundreds or thousands login attempts daily. To get rid of those bruteforce attempts I wrote a small python script which I call blacklist.py.

Motivation

I don't want to use PAM's cracklib module. Allthough it would grant a max. security it just lacks user-friendliness. Using SSH Key Pairs would entrust the responsability to the user (he would most certainly not use any password and save his private key who knows where).

I tried iptables' –hitcount option too (read this article on how to use it) - but it just doesn't work properly. I get unknown delays using that method for IP addresses not exeeding a specified limit.

I gave up sending providers extracts from my logs (although this would be fighting the cause and not just the effect). If they do anything I guess they just make sure the person in question doesn't get the same IP for some time.

Quick Overview

Per default, blacklist.py blocks an ip for 10 minutes if more than 6 login failures from the same ip occured within 10 minutes. Using iptables it blocks only ssh connections - meaning ftp, mail, web and any other connections from the attackers ip to your server are still possible.

Since version 0.4.4 you may now check for FTP login attempts too. SSH and FTP login attempts are counted seperately.

It will mail you a similar message:

From: blacklist@yourdomain.org
To: ssh@yourdomain.org
Subject: [yourhostname]: Too many login failures from 220.245.172.219 on port 22
Blocking out 1 IP(s):<br>
220.245.172.219 on port 22

or in case of several blocked out IPs during one run:

From: blacklist@yourdomain.org
To: ssh@yourdomain.org
Subject: [yourhostname]: Too many login failures from multiple IPs
Blocking out 5001 IP(s):<br>
10.243.244.152 port 22
10.239.34.24 port 22
10.72.136.23 port 21
10.57.151.40 port 21
...

It writes a blocking and unblocking messages to /var/log/blacklist.log:

 06.01.2006 21:37:19: Blocking 220.245.172.219 on port 21
 06.01.2006 21:47:19: Remove blocking 220.245.172.219 on port 22

A blocked out IP will see the following message trying to log in to your box again:

ssh: connect to host blinkeye.ch port 22: Connection refused

or

ftp: connect to host blinkeye.ch port 21: Connection refused

List of past login attempts (IPs)

Execute the following command (providing failed logins are written to /var/log/auth.log) to get an overview of past login failures:

# g# grep Failed /var/log/auth.log | grep -o '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort | uniq -c
...
497   82.140.87.203
256   82.211.37.4
26    82.231.80.34
1429  82.233.3.135
15    217.160.128.168
129   217.160.128.235
197   217.160.242.150
239   217.172.184.31
...

List of past login attempts (usernames)

If you use syslog-ng you get a list of tried login names with

# grep "Failed" /var/log/auth.log | sed "s/.*for\( invalid user\)*\(.*\)\(from.*\)/\2/" | sort | uniq -c
...
31     roo
31     rooot
2601   root
31     rooti
42     rootkit
...

What you need

  • Python: The programming language of blacklist.py
  • syslog-ng: The system logger
  • logtail: Used to read new lines of a previously read file
  • iptables: The GNU/Linux packet filtering tool
  • MTA: Some kind of a MTA to get a mail notification [Optional]

System logger

Of course you may use any system logger you want. You probably have to adjust the regex used in blacklist.py though. Read the Gentoo Security Handbook how to configure syslog-ng to log authentications to /var/log/auth.log. In short, add

destination authlog { file("/var/log/auth.log"); };
destination authlog { file("/var/log/auth.log"); };
filter f_authpriv { facility(auth, authpriv); };
log { source(src); filter(f_authpriv); destination(authlog); };

to /etc/syslog-ng/syslog-ng.conf. This will log all authentication messages to /var/log/auth.log.

Iptables

You have to configure your kernel for iptables support. Iptables-tutorial and/or Gentoo Server Firewall Guide and/or Gentoo Security Handbook will help you configure your kernel. You won't need any specific knowledge about iptables to use this script - but it's probably a good time to get up-to-date with GNU/Linux's packet filtering anyway. This iptables howto looks like a good starting point.

Root access

I strongly suggest disabling root access in /etc/ssh/sshd_config:

PermitRootLogin no

(don't forget to restart the SSH server).

Gentoo users

Portage is programmed in Python, so, you got Python installed already. If you don't know which system logger you use execute

# rc-status | grep log

To get syslog-ng, logtail and iptables execute

# emerge app-admin/syslog-ng
# emerge app-admin/logsentry
# emerge net-firewall/iptables
# rc-update add syslog-ng default

How to customise

Root login tries

blacklist.py will block out an ip instantly if it tries to login with root. If you don't like that remove

# no tolerance for a root login attempt
if ( match[ 0 ] == "root" ):
   entry[ 1 ] += PERMITTED_LOGIN_FAILURES

from the script.

LOG_INPUT

This is the logfile where blacklist.py reads authentication failures from.

LOG_OUTPUT

This is the file blacklist.py writes it's taken actions to.

PID_FILE

The path to the file the PID is written to.

LOGTAIL

The path to logtail.

WHOAMI

The path to whoami.

PERMITTED_LOGIN_FAILURES

Any login failure from the same ip above this threshold results in getting blocked out.

BLOCKING_PERIOD

The time in seconds to block an ip after PERMITTED_LOGIN_FAILURES failures. Keep in mind that this period increases with the number of attacks during a run. For every hundred login failure an additional second is added.

SUSPECTING_PERIOD

The time in seconds to count a login failure as a possible attack. If SUSPECTING_PERIOD passed since the last login failure and the IP is not blocked it gets removed.

CHECK_INTERVALL

The time in seconds when the script checks if it should unblock or remove an IP. So, in the worst case an IP gets blocked (BLOCKING_PERIOD + CHECK_INTERVALL) seconds.

SLEEP_PERIOD

The time in seconds the main thread sleeps before rescanning log entries.

Keep in mind that an ip may succeed in trying to login more than PERMITTED_LOGIN_FAILURES. This is because the main thread only runs every SLEEP_PERIOD seconds. Decrease SLEEP_PERIOD to your liking.

DATE_FORMAT

The date format used for LOG_OUTPUT.

SSH_PORT

The port your SSH server listens to.

FTP_PORT

THe port your FTP server listens to.

SENDMAIL

The path to sendmail

MAIL_SENDER

The “From: ” address for the mail.

MAIL_RECEIVER

Whom the mail is sent to.

SSH_REGEX

This is the regex to catch ssh login failures. It is a comma seperated list (meaning you could just add other regexs). Please use the Test_mail_notification to verify your regexs.

It catches following similar logging entries:

Jan  2 21:48:05 blinkeye sshd[4529]: Failed password for invalid user sato from 61.172.192.3 port 54177 ssh2
Jan  2 21:48:05 blinkeye sshd[4529]: Failed password for invalid user sato from ::ffff:61.172.192.3 port 54177 ssh2
Oct 21 18:52:01 blinkeye sshd[31286]: Failed password for root from 152.149.148.115 port 36667 ssh2
Oct 21 18:52:01 blinkeye sshd[31286]: Failed password for root from ::ffff:152.149.148.115 port 36667 ssh2
Sep 18 05:08:06 blinkeye sshd[3971]: Failed keyboard-interactive/pam for root from 152.149.148.115 port 44896 ssh2
Sep 18 05:08:06 blinkeye sshd[3971]: Failed keyboard-interactive/pam for root from ::ffff:152.149.148.115 port 44896 ssh2
Mar 7 02:37:29 blinkeye sshd[8559]: User root from 61.178.20.170 not allowed because not listed in AllowUsers 
Mar 7 02:37:29 blinkeye sshd[8559]: User root from ::ffff:61.178.20.170 not allowed because not listed in AllowUsers 
Mar 7 12:57:11 blinkeye sshd[14334]: User admin from 205.241.227.14 not allowed because not listed in AllowUsers
Mar 7 12:57:11 blinkeye sshd[14334]: User admin from ::ffff:205.241.227.14 not allowed because not listed in AllowUsers
Mar 7 14:27:16 blinkeye sshd[15213]: User apache from 218.71.137.69 not allowed because not listed in AllowUsers 
Mar 7 14:27:16 blinkeye sshd[15213]: User apache from ::ffff:218.71.137.69 not allowed because not listed in AllowUsers

FTP_REGEX

This is the regex to catch ftp login failures (currently created for vsftpd). It is a comma seperated list (meaning you could just add other regexs). Please use the Test_mail_notification to verify your regexs.

It catches following similar logging entries:

Oct  3 19:35:41 blinkeye ftp(pam_unix)[8746]: authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=206.222.29.194
Oct  3 19:35:41 blinkeye ftp(pam_unix)[8746]: authentication failure; logname= uid=0 euid=0 tty= ruser= ::ffff:rhost=206.222.29.194
Oct  3 19:35:43 blinkeye ftp(pam_unix)[8746]: authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=206.222.29.194  user=root
Oct  3 19:35:43 blinkeye ftp(pam_unix)[8746]: authentication failure; logname= uid=0 euid=0 tty= ruser= ::ffff:rhost=206.222.29.194  user=root

Advanced Configuration

Following are some adjustement targeted to the advanced user. If you don't understand what I'm talking about don't worry about these settings.

Don't give him a clue

If you don't want your attacker to know what's going on you might want to surpress

ssh: connect to host blinkeye.ch port 22: Connection refused

In that case change

system_command( IPTABLES + " --insert " + CUSTOM_CHAIN + " --source " + self.threadParameter + \
" --protocol tcp --dport " + str( SSH_PORT ) + " --jump REJECT" )

to

system_command( IPTABLES + " --insert " + CUSTOM_CHAIN + " --source " + self.threadParameter + \
" --protocol tcp --dport " + str( SSH_PORT ) + " --jump DROP" )

and

system_command( IPTABLES + " --delete " + CUSTOM_CHAIN + " --source " + self.threadParameter + \
" --protocol tcp --dport " + str( SSH_PORT ) + " --jump REJECT" )

to

system_command( IPTABLES + " --delete " + CUSTOM_CHAIN + " --source " + self.threadParameter + \
" --protocol tcp --dport " + str( SSH_PORT ) + " --jump DROP" )

(replace REJECT with DROP). If you DROP his IP he will just get a hanging session and eventually a connection timeout.

Advanced iptables configuration

If you use iptables to secure your server you might have rules you want to take precedence before the CUSTOM_CHAIN of blacklist.py. In this case, use

# iptables -L -n

to define the index (starting at 1) and change

system_command( IPTABLES + " --insert INPUT --jump " + CUSTOM_CHAIN )
<code>

to

<code>
system_command( IPTABLES + " --insert INPUT index --jump " + CUSTOM_CHAIN ) #replace index with your desired insert location

Per default an established connection from the attackers IP (meaning someone who logged in successfully earlier) will hang after that IP gets blocked out. If you have a RELATED,ESTABLISHED rule in your INPUT CHAIN like

# iptables -L -n

Chain INPUT (policy DROP)
target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED

you could set the insert command to

system_command( IPTABLES + " --insert INPUT 2 --jump " + CUSTOM_CHAIN )

which would add blacklist.py's CUSTOM_CHAIN after the RELATED,ESTABLISHED rule

Chain BLACKLIST (1 references)
target     prot opt source               destination <br>
Chain INPUT (policy DROP)
Target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
BLACKLIST  all  --  0.0.0.0/0            0.0.0.0/0

and blocking an IP will not affect any earlier successfully made connections.

The script

#!/usr/bin/python
 
# blacklist.py is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# blacklist.py is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# Copyright: Reto Glauser aka blinkeye
# Mailto: blacklist at blinkeye dot ch
# Homepage: http://blinkeye.ch
# Forum post: http://forums.gentoo.org/viewtopic-t-421706.html
# Date: 2006-03-16
# Version 0.4.5
 
import re;
import commands;
import thread;
import threading;
import sys;
from os import access, popen, R_OK, W_OK, X_OK;
from time import sleep, strftime, time;
from string import find;
from syslog import *;
import errno;
import os;
 
LOG_INPUT = "/var/log/auth.log"
LOG_OUTPUT = "/var/log/blacklist.log"
PID_FILE = "/var/run/blacklist.pid"
LOGTAIL = "/usr/sbin/logtail"
WHOAMI = "/usr/bin/whoami"
IPTABLES = "/sbin/iptables"
CUSTOM_CHAIN = "BLACKLIST"
PERMITTED_LOGIN_FAILURES = 6
BLOCKING_PERIOD = 600 #seconds
SUSPECTING_PERIOD = 600 #seconds
SLEEP_PERIOD = 30 #seconds
CHECK_INTERVALL = 300 #seconds
DATE_FORMAT = "%d.%m.%Y %X" # e.g.: 02.01.2006 23:49:12
SSH_PORT = 22
FTP_PORT = 21
SENDMAIL = "/usr/sbin/sendmail"
MAIL_SENDER = "blacklist@yourdomain"
MAIL_RECEIVER = "ssh@yourdomain"
 
 
SSH_REGEX =    [ 
               r"Failed (?:none|password|keyboard-interactive/pam) for (?:invalid user )*(?P<user>.*) from (?:::ffff:)*(?P<ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})",
               r"Invalid user (?P<user>.*) from (?:::ffff:)*(?P<ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"
            ]
# SSH_REGEX catches following similar entries:
# Jan  2 21:48:05 blinkeye sshd[4529]: Failed password for invalid user sato from 61.172.192.3 port 54177 ssh2
# Jan  2 21:48:05 blinkeye sshd[4529]: Failed password for invalid user sato from ::ffff:61.172.192.3 port 54177 ssh2
# Oct 21 18:52:01 blinkeye sshd[31286]: Failed password for root from 152.149.148.115 port 36667 ssh2
# Oct 21 18:52:01 blinkeye sshd[31286]: Failed password for root from ::ffff:152.149.148.115 port 36667 ssh2
# Sep 18 05:08:06 blinkeye sshd[3971]: Failed keyboard-interactive/pam for root from 152.149.148.115 port 44896 ssh2
# Sep 18 05:08:06 blinkeye sshd[3971]: Failed keyboard-interactive/pam for root from ::ffff:152.149.148.115 port 44896 ssh2         
 
 
FTP_REGEX = [
            r"ftp(?:.*) authentication failure(?:.*) rhost=(?:::ffff:)*(?P<ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) (?: user=)*(?P<user>.*)"
            ]
# FTP_REGEX catches following similar entries:
# Oct  3 19:35:41 blinkeye ftp(pam_unix)[8746]: authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=206.222.29.194
# Oct  3 19:35:43 blinkeye ftp(pam_unix)[8746]: authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=206.222.29.194  user=root
 
 
# Use
# grep Failed /var/log/auth.log | grep -o '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort | uniq -c
# to get a statistic of ip's from past login failures
 
# Use
# grep "Failed" /var/log/auth.log | sed "s/.*for\( invalid user\)*\(.*\)\(from.*\)/\2/" | sort | uniq -c
# to get a statistic of login names from past login failures
 
 
# Wrapper function for commands
def system_command( string_command ):
   return_value = [ ]
   return_value = ( commands.getstatusoutput( string_command ) )
   if not return_value[ 0 ] == 0:
      raise IOError( return_value[ 1 ] )
   return return_value[ 1 ]
 
 
# block ip for the duration of time
def block( ip, time, port ):
   try:
      system_command( IPTABLES + " --new-chain " + CUSTOM_CHAIN )
      system_command( IPTABLES + " --insert INPUT --jump " + CUSTOM_CHAIN )
   except:
      None     
   system_command( IPTABLES + " --insert " + CUSTOM_CHAIN + " --source " + ip + " --protocol tcp --dport " + str( port ) + " --jump REJECT" )
   LOG_OUTPUT.write( strftime( DATE_FORMAT ) + ": Blocking " + ip + " for " + str( time ) + " seconds\n" )
   LOG_OUTPUT.flush()
 
 
# unblock IP
def unblock( ip, port ):
   system_command( IPTABLES + " --delete " + CUSTOM_CHAIN + " --source " + ip + " --protocol tcp --dport " + str( port ) + " --jump REJECT" )
   LOG_OUTPUT.write( strftime( DATE_FORMAT ) + ": Remove blocking " + ip + "\n" )
   LOG_OUTPUT.flush()
 
 
# mail list of IP blocked during this run
def mail( mail_list ):
   count = len( mail_list )
   if count == 0:
      return
   ip = str( mail_list[ 0 ][ 0 ] ) + " on port " + str( mail_list[ 0 ][ 3 ] ) + "\n"
   mail_list.remove( mail_list[ 0 ] )
   p = popen( "%s -t" % SENDMAIL, "w" )
   p.write( "From: " + MAIL_SENDER + "\n" )
   p.write( "To: " + MAIL_RECEIVER + "\n" )
   if count == 1:
      p.write( "Subject: [" + system_command( 'hostname --long' ) + "]: Too many login failures from " + ip + "\n\n" )
   else:
      p.write( "Subject:  [" + system_command( 'hostname --long' ) + "]: Too many login failures from multiple IPs\n\n" )
      for entry in mail_list[ : ]:
         ip += entry[ 0 ] + " on port " + str( entry[ 3 ] ) + "\n"
         mail_list.remove( entry )
   p.write( "Blocking out " + str( count ) + " IP(s):\n\n" + ip + "\n" )
   if p.close() != None:
      LOG_OUTPUT.write( strftime( DATE_FORMAT ) + ": Unable to send a mail. Check your SENDMAIL configuration.\n" )
      LOG_OUTPUT.flush()
 
 
# append IPs from regex_matches to ip_list and increase counter of login failures for IPs
def create_stat( regex_matches, ip_list, ip_list_blocked, delay, port ):
   current_time = time() + delay
   for match in regex_matches:
      entry = ip_list.get( match.group( 'ip' ) )
      if entry == None:
         ip_list[ match.group( 'ip' ) ] = [ match.group( 'ip' ), 1, current_time, port ] # [ [ ip ],[ counter for catched login failures ],[ time of last login failure ], [ port ] ]
         entry = ip_list.get( match.group( 'ip' ) )
      else:
         entry[ 1 ] += 1
      # no tolerance for a root login attempt
      if ( match.group( 'user' ) == "root" ):
         entry[ 1 ] += PERMITTED_LOGIN_FAILURES
   for key in ip_list.keys()[ : ]:
      if ip_list.get( key )[ 1 ] > PERMITTED_LOGIN_FAILURES and ip_list.get( key)[ 0 ] not in ip_list_blocked:
         ip_list_blocked.insert( 0, ip_list.get( key ) )
         del ip_list[ key ]
         block( ip_list_blocked[ 0 ][ 0 ], BLOCKING_PERIOD + delay, port )
         mail_list.insert( 0, ip_list_blocked[ 0 ] )
 
 
# Someone must do the work
def scan():
   global countdown
 
   try:
      new_log_entries = system_command( LOGTAIL + " -f " + LOG_INPUT )
   except:
      new_log_entries  =  system_command( LOGTAIL + " " + LOG_INPUT )
 
   for i in range( 0, len( SSH_REGEX ) ):
      re_ssh = re.compile( SSH_REGEX[ i ] )
      regex_matches = re_ssh.finditer( new_log_entries )
      create_stat( regex_matches, ssh_list, ssh_list_blocked, len( re_ssh.findall( new_log_entries ) )/100, SSH_PORT )
 
   for i in range( 0, len( FTP_REGEX ) ):
      re_ftp = re.compile( FTP_REGEX[ i ] )
      regex_matches = re_ftp.finditer( new_log_entries )
      create_stat( regex_matches, ftp_list, ftp_list_blocked, len( re_ftp.findall( new_log_entries ) )/100, FTP_PORT )
 
   # +++ mail section +++ #
   mail( mail_list )
   # +++ mail section +++ #
 
   if countdown <= 0:
      countdown = CHECK_INTERVALL
      current_time = time()
      for entry in ssh_list_blocked[ : ]:
         if( current_time - entry[ 2 ] ) > BLOCKING_PERIOD:
            unblock( entry[ 0 ], SSH_PORT )
            ssh_list_blocked.remove( entry )
      for key in ssh_list.keys():
         if( current_time - ssh_list.get( key )[ 2 ] ) > SUSPECTING_PERIOD:
            del ssh_list[ key ]
 
      for entry in ftp_list_blocked[ : ]:
         if( current_time - entry[ 2 ] ) > BLOCKING_PERIOD:
            unblock( entry[ 0 ], FTP_PORT )
            ftp_list_blocked.remove( entry )
      for key in ftp_list.keys():
         if( current_time - ftp_list.get( key )[ 2 ] ) > SUSPECTING_PERIOD:
            del ftp_list[ key ]           
 
# Check if there's another instance running
def handlepid():
   try:
      pidfile = os.fdopen( os.open( PID_FILE, os.O_WRONLY | os.O_CREAT | os.O_EXCL ), 'w' )
   except OSError:
      try:
         pid = int( open( PID_FILE ).read() )
      except IOError:
         sys.exit( "Error opening pidfile %s" %PID_FILE )
      try:
         os.kill( pid, 0 )
      except OSError, why:
         if why.errno == errno.ESRCH:
            print "Removing stale pidfile %s with pid %d\n" %( PID_FILE, pid )
            os.remove( PID_FILE )
            if os.path.exists( PID_FILE ):
               sys.exit( "Cannot remove pidfile." )
            else:
               return handlepid()
      sys.exit( "\nAnother blacklist daemon is running with pid %d" %pid )
   pidfile.write( "%d\n" %os.getpid() )
   pidfile.flush()
   pidfile.close()   
   cleanup();
 
 
if not system_command( "whoami" ) == "root":
   raise IOError, "This script must be run as root"
if not access( LOG_INPUT, R_OK ):
   raise IOError, LOG_INPUT + " is not readable"
if not access( LOGTAIL, X_OK ):
   raise IOError, LOGTAIL + " is not executable"
if not access( WHOAMI, X_OK ):
   raise IOError, WHOAMI + " is not executable"
if not access( IPTABLES, X_OK ):
   raise IOError, IPTABLES + " is not executable"
# +++ mail section +++ #
if not access(  SENDMAIL, X_OK ):
   raise IOError, SENDMAIL + " is not executable"
# +++ mail section +++ #
 
 
# test modus
if len( sys.argv ) == 2:
   print( "\n* Entering test mode" )
 
   for i in range( 0, len( SSH_REGEX ) ):
      re_ssh = re.compile( SSH_REGEX[ i ] )
      if not len( re_ssh.findall( sys.argv[ 1 ] ) ):
         print( "* SSH_REGEX[ " + str( i ) + " ]: No match found"  )
      else:
         regex_matches = re_ssh.finditer( sys.argv[ 1 ] )
         for match in regex_matches:
            print( "* SSH_REGEX[ " + str( i ) + " ]: Caught ip \"" + str( match.group( 'ip' ) ) + " and username \"" + str( match.group( 'user' ) ) + "\"" )
 
   for i in range( 0, len( FTP_REGEX ) ):
      re_ftp = re.compile( FTP_REGEX[ i ] )
      if not len( re_ftp.findall( sys.argv[ 1 ] ) ):
         print( "* FTP_REGEX[ " + str( i ) + " ]: No match found" )
      else:
         regex_matches = re_ftp.finditer( sys.argv[ 1 ] )
         for match in regex_matches:
            print( "* FTP_REGEX[ " + str( i ) + " ]: Caught ip \"" + str( match.group( 'ip' ) ) + "\" and username \"" + str( match.group( 'user' ) ) + "\"" )
 
   p = popen( "%s -t" % SENDMAIL, "w" )
   p.write( "From: " + MAIL_SENDER + "\n" )
   p.write( "To: " + MAIL_RECEIVER + "\n" )
   p.write( "Subject: blacklist.py ist testing your sendmail configuration\n\n" )
   p.write( "A test mail from blacklist.py\n" )
   if p.close() != None:
      print( "* ERROR sending a mail. Check your SENDMAIL configuration.\n" )
   else:
      print( "* SUCCESS: Sending mail from " + MAIL_SENDER + " to " + MAIL_RECEIVER )
   sys.exit(0)
 
 
# Cleanup
def cleanup():
   try:
      system_command( IPTABLES + " --delete INPUT -j " + CUSTOM_CHAIN )
      system_command( IPTABLES + " --flush " + CUSTOM_CHAIN )
      system_command( IPTABLES + " --delete-chain " + CUSTOM_CHAIN )
   except:
      None
 
 
# Call cleanup() before exiting the script
sys.exitfunc = cleanup
handlepid()
LOG_OUTPUT = file( LOG_OUTPUT, "a")
 
ssh_list = { }
ssh_list_blocked = [ ]
 
ftp_list = { }
ftp_list_blocked = [ ]
 
mail_list = [ ]
countdown = CHECK_INTERVALL
cleanup() # remove old entries
 
while 1:
   scan()
   sleep( SLEEP_PERIOD )
   countdown -= SLEEP_PERIOD

Run it

Either copy and paste the script from above (keep in mind that Python does not tolerate any whitespace deviation) or:

and make it executable

  1. chmod +x blacklist.py

Important: Execute the following command to set logtail up-to-date or you'll block out IP addresses for BLOCKING_PERIOD which lay way in the past (must be done only once):

# logtail /var/log/auth.log

Important: If you don't have a MTA you must comment all lines (several occurences) between the tag

# +++ mail section +++ #

You won't get an email when IPs are blocked, but blacklist.py will work nevertheless.

Run it:

# ./blacklist.py &

Login to another box and ssh to your server using wrong passwords.

Watching the magic

# tail -f /var/log/blacklist.log

If an IP Address is added to blacklist.log verify that it actually is getting blocked:

# iptables -L -n

Chain BLACKLIST (1 references)
target     prot opt source               destination
REJECT     tcp  --  220.245.172.219       0.0.0.0/0           tcp dpt:22 reject-with icmp-port-unreachable <br>
Chain INPUT (policy DROP)
target     prot opt source               destination
BLACKLIST  all  --  0.0.0.0/0            0.0.0.0/0
...

Watching iptables

You can actually watch iptables change over time (instead of using repeatedly 'iptables -L -n'):

# watch -d 'iptables -L -n'

This will execute 'iptables -L -n' every 2 seconds and update your screen.

Test mode / Test regex / Test mail notification

You can test if blacklist.py catches your log lines. This comes in handy if you want to adjust or try new regexs or just want to be sure. Execute blacklist.py and pass the loglines (as many as you'd like) within quotes:

# ./blacklist.py "Jan  2 21:48:05 blinkeye sshd[4529]: Failed password for invalid user sato from 61.172.192.3 port 54177 ssh2
Jan  2 21:48:05 blinkeye sshd[4529]: Failed password for invalid user sato from ::ffff:61.172.192.3 port 54177 ssh2
Oct 21 18:52:01 blinkeye sshd[31286]: Failed password for root from 152.149.148.115 port 36667 ssh2
Oct 21 18:52:01 blinkeye sshd[31286]: Failed password for root from ::ffff:152.149.148.115 port 36667 ssh2
Sep 18 05:08:06 blinkeye sshd[3971]: Failed keyboard-interactive/pam for root from 152.149.148.115 port 44896 ssh2
Sep 18 05:08:06 blinkeye sshd[3971]: Failed keyboard-interactive/pam for root from ::ffff:152.149.148.115 port 44896 ssh2
Oct  3 19:35:41 blinkeye ftp(pam_unix)[8746]: authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=206.222.29.194
Oct  3 19:35:41 blinkeye ftp(pam_unix)[8746]: authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=::ffff:206.222.29.194
Oct  3 19:35:43 blinkeye ftp(pam_unix)[8746]: authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=206.222.29.194  user=root
Oct  3 19:35:43 blinkeye ftp(pam_unix)[8746]: authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=::ffff:206.222.29.194  user=root"

* Entering test mode
* SSH_REGEX[ 0 ]: Caught ip "61.172.192.3 and username "sato"
* SSH_REGEX[ 0 ]: Caught ip "61.172.192.3 and username "sato"
* SSH_REGEX[ 0 ]: Caught ip "152.149.148.115 and username "root"
* SSH_REGEX[ 0 ]: Caught ip "152.149.148.115 and username "root"
* SSH_REGEX[ 0 ]: Caught ip "152.149.148.115 and username "root"
* SSH_REGEX[ 0 ]: Caught ip "152.149.148.115 and username "root"
* SSH_REGEX[ 1 ]: No match found
* FTP_REGEX[ 0 ]: Caught ip "206.222.29.194" and username ""
* FTP_REGEX[ 0 ]: Caught ip "206.222.29.194" and username ""
* FTP_REGEX[ 0 ]: Caught ip "206.222.29.194" and username "root"
* FTP_REGEX[ 0 ]: Caught ip "206.222.29.194" and username "root"
* SUCCESS: Sending mail from blacklist@yourdomain to ssh@yourdomain

For every regex in SSH_REGEX and FTP_REGEX it will try to match an ip and username. The ip catch is mandatory - but it does need to catch a username.

Server Reboot / Cleaning up

To run blacklist.py automatically upon reboot add blacklist.py to a runlevel (creating an init script) or even better add it directly to your firewall/iptables script. The script will remove any blocked IP and the CUSTOM_CHAIN every time it is restarted. In case you kill blacklist.py you may want to clean up your iptables' rules. The following will remove anything added by the script (only necessary if you kill it):

# /sbin/iptables --delete INPUT -j CUSTOM_CHAIN 
# /sbin/iptables --flush CUSTOM_CHAIN
# /sbin/iptables --delete-chain CUSTOM_CHAIN

Replace CUSTOM_CHAIN with the used CHAIN name (default 'BLACKLIST' ).

UPDATE: You may safely reset your iptable rules while running blacklist.py. It will (re)add it's needed rules automatically when blocking the next IP.

DoS / Performance / Stress Test

Let's see how the script works under a simulated Denial of Service attack. We don't want to use a defense mechanism which would indirectly create a DoS under heavy load.

Scenario

Your SSH Server has had a lot of activity during the last SLEEP_PERIOD, namely

  • 500000 (half a million) new log entries
  • 50 MB of loglines to parse
  • 375000 IP occurences
  • 125000 regex matches
  • 5000 different IPs
  • Reocurring failed login entries are scattered throughout the 500000 log lines to simulate a scheduled attack

Create the logfile

#!/usr/bin/python
from random import randint;
from time import strftime;
 
DATE_FORMAT = "%d.%m.%Y %X" # e.g.: 02.01.2006 23:49:12
 
# stress test / DoS simulation. Creates random private A-Class IPs from 10.0.0.0 to 10.254.254.254
def DoS():
   DoS_list = [ ]
   user_list = [ "root", "admin", "guest", "test", "sql", "web", "www", "apache", "mysql", "webmail", "mail", "sysadmin", "tomcat", "webmaster", "hostmaster" ]
   for j in range( 0, 5000 ):
      DoS_list.append( "10." + str( randint( 1, 244 ) ) + "." + str( randint( 1, 244 ) ) + "." + str( randint( 1, 244 ) ) + " port " + str( randint( 1024, 65535) ) )
   for k in range( 0, 125000 ):
      user = user_list[ randint(0, 14) ]
      ip = DoS_list[ randint( 0, 4999 ) ]
      print( str( strftime( "%b %d %X" ) ) + " x40 sshd[" + str( randint( 1024, 65535 ) ) + "]: Failed keyboard-interactive/pam for invalid user " + user + " from " + ip + " ssh2" )
      print( str( strftime( "%b %d %X" ) ) + " x40 sshd[" + str( randint( 1024, 65535 ) ) + "]: check pass; user unknown" )
      print( str( strftime( "%b %d %X" ) ) + " x40 sshd[" + str( randint( 1024, 65535 ) ) + "]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=" + ip )
      print( str( strftime( "%b %d %X" ) ) + " x40 sshd[" + str( randint( 1024, 65535 ) ) + "]: error: PAM: Authentication failure for " + user + " from " + ip )
 
DoS()

This script will print randomized log entries ( don't worry, these IP addresses are all private - just make sure you don't use an IP address from 10.X.X.X yourself ).

Pentium M 1700 Mhz, 1024 MB RAM

This is the time for each run (SLEEP_PERIOD is 30 seconds):

  • Run took: 66.2886929512 seconds
  • Run took: 0.0120289325714 seconds
  • Run took: 0.0120761394501 seconds

  • Run took: 0.0120210647583 seconds
  • Run took: 64.545445919 seconds

Pentium M 1200 Mhz, 512 MB RAM

This is the time for each run (SLEEP_PERIOD is 30 seconds):

  • Run took: 105.404068947 seconds
  • Run took: 0.0309278964996 seconds
  • Run took: 0.0309059619904 seconds

  • Run took: 98.3297541142 seconds
  • Run took: 0.0316820144653 seconds

Pentium M 222 Mhz, 1024 RAM

This is the time for each run (SLEEP_PERIOD is 30 seconds):

  • Run took: 586.678808212 seconds
  • Run took: 0.0743868350983 seconds
  • Run took: 0.0695021152496 seconds

  • Run took: 0.0639469623566 seconds
  • Run took: 654.533771992 seconds
  • Run took: 0.0641090869904 seconds

Conclusion

  • Regex-ing the 500000 entries takes 3 seconds.
  • Reading, parsing and blocking the 5000 IPs took a little more than 1 minute
  • The 5000 IPs were blocked out for 40 minutes (BLOCKING_PERIOD plus the additional dynamic calculated time - in this case another 20 minutes).
  • Subsequent runs take ~0.010 (10 miliseconds)
  • Every CHECK_INTERVALL seconds it takes ~0.020 ( 20 miliseconds) to check if IPs should be removed or unblocked
  • Removing the 5000 IPs took a little more than 1 minute
  • Running detailed statistics revelead that the iptables command takes most of the time. That's why adding and removing the IPs take about the same time.

I didn't notice a peformance loss connecting via SSH to those boxes having over 5000 iptables rules (based on personal impression). So, the script passed the test.

Note: Results taken from the Pentium M 1700 Mhz runs

Contact / Issues

In case you run into problems you might want to check out the Gentoo Forum post. Feel free to mailto:blacklist@blinkeye.ch mail me with comments, modifications or suggestions.

In case you use a different logger and the script doesn't work for you provide me a detailed log and information.

Changelog

2006-01-06

  • Adding Howto.

2006-01-11

  • Adding section Advanced_Configuration
  • Adding section Watching_iptables
  • Adding cleanup function
  • Adding system_command function to catch run time errors
  • Adding custom iptables chain for easy distinction and use
  • Changing iptables example output. Thanks to Magic919 for pointing out the issue

2006-01-14

2006-01-19

  • Completely rewrote the core
  • Removing thread logic due to limitation and ressource usage
  • Changing datastructure of main list from an array to a hash map to speed up searches
  • Adding CHECK_INTERVALL variable
  • Adding dynamical BLOCKING_PERIOD (for every hundred login failure an additional second is added)
  • Modified mailing feature so you get one mail per each run (if any IP were blocked) instead of a mail per IP to prevent mail flood/ressource usage upon a DoS
  • Modifying Howto
  • Thorough testing

2006-03-03

  • Added new function handlepid() to check if an instance is already running (thanks to Erik J.)
  • Added try/except block to handle the issue if iptables get flushed while the script is running
  • Added try/except block to handle the different logtail versions
  • Fixed an issue where wrong entries would be written to the LOG_OUTPUT (modifying the hash table while iterating through it without making a copy)
  • Minor speed improvement
  • Added forum link to the header

2006-03-15

  • Added new functionality to permit several regexs at once
  • Added ftp regex
  • Changed subject of the status mail to contain the hostname
  • The mail now contains the port an IP is blocked on
  • Correctly blocking out invalid users/login tries if the ssh daemon is configured with the “AllowUsers” variable
  • Enhanced the test modus to support several loglines

2006-03-16

  • Fixed ftp regex (vsftp)

2006-03-22

  • Updated Wiki

2006-03-29

  • Updated Wiki

2009-04-13

  • Moved content from Mediawiki to Dokuwiki

Sources/Further Information

 
projects/blacklist.txt · Last modified: 2009-04-13 13:00 by blinkeye
 
Recent changes RSS feed Creative Commons License Powered by GNU/Linux Powered by Gentoo Powered by Apache Powered by XCache Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki