= blacklist = == Introduction == Since a few month programs spread which try to login to a [[http://openssh.org|SSH]] or a FTP server. SSH and FTP servers get hundreds or thousands login attempts daily. To get rid of those bruteforce attempts I wrote a small python script which I call //blacklist.py//. ==== Motivation ==== I don't want to use [[http://www.kernel.org/pub/linux/libs/pam/modules.html|PAM's cracklib module]]. Allthough it would grant a max. security it just lacks user-friendliness. Using [[http://www.openssh.com|SSH Key Pairs]] would entrust the responsability to the user (he would most certainly not use any password and save his private key who knows where). I tried iptables' --hitcount option too (read [[http://www.debian-administration.org/articles/187|this article]] on how to use it) - but it just doesn't work properly. I get unknown delays using that method for IP addresses not exeeding a specified limit. I gave up sending providers extracts from my logs (although this would be fighting the cause and not just the effect). If they do anything I guess they just make sure the person in question doesn't get the same IP for some time. ==== Quick Overview ==== Per default, //blacklist.py// blocks an ip for 10 minutes if more than 6 login failures from the same ip occured within 10 minutes. Using iptables it blocks only ssh connections - meaning ftp, mail, web and any other connections from the attackers ip to your server are still possible. Since version 0.4.4 you may now check for FTP login attempts too. SSH and FTP login attempts are counted seperately. It will mail you a similar message: From: blacklist@yourdomain.org To: ssh@yourdomain.org Subject: [yourhostname]: Too many login failures from 220.245.172.219 on port 22 Blocking out 1 IP(s):
220.245.172.219 on port 22 or in case of several blocked out IPs during one run: From: blacklist@yourdomain.org To: ssh@yourdomain.org Subject: [yourhostname]: Too many login failures from multiple IPs Blocking out 5001 IP(s):
10.243.244.152 port 22 10.239.34.24 port 22 10.72.136.23 port 21 10.57.151.40 port 21 ... It writes a blocking and unblocking messages to /var/log/blacklist.log: 06.01.2006 21:37:19: Blocking 220.245.172.219 on port 21 06.01.2006 21:47:19: Remove blocking 220.245.172.219 on port 22 A blocked out IP will see the following message trying to log in to your box again: ssh: connect to host blinkeye.ch port 22: Connection refused or ftp: connect to host blinkeye.ch port 21: Connection refused ==== List of past login attempts (IPs) ==== Execute the following command (providing failed logins are written to /var/log/auth.log) to get an overview of past login failures: # g# grep Failed /var/log/auth.log | grep -o '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort | uniq -c ... 497 82.140.87.203 256 82.211.37.4 26 82.231.80.34 1429 82.233.3.135 15 217.160.128.168 129 217.160.128.235 197 217.160.242.150 239 217.172.184.31 ... ==== List of past login attempts (usernames) ==== If you use syslog-ng you get a list of tried login names with # grep "Failed" /var/log/auth.log | sed "s/.*for\( invalid user\)*\(.*\)\(from.*\)/\2/" | sort | uniq -c ... 31 roo 31 rooot 2601 root 31 rooti 42 rootkit ... == What you need == * [[http://www.python.org|Python]]: The programming language of //blacklist.py// * [[http://www.balabit.com/products/syslog_ng|syslog-ng]]: The system logger * [[http://sourceforge.net/projects/sentrytools|logtail]]: Used to read new lines of a previously read file * [[http://www.netfilter.org/|iptables]]: The GNU/Linux packet filtering tool * [[http://en.wikipedia.org/wiki/Mail_transfer_agent|MTA]]: Some kind of a MTA to get a mail notification **[Optional]** === System logger === Of course you may use any system logger you want. You probably have to adjust the [[http://en.wikipedia.org/wiki/Regex|regex]] used in //blacklist.py// though. Read the [[http://www.gentoo.org/doc/en/security/security-handbook.xml?part=1&chap=3#doc_chap4|Gentoo Security Handbook]] how to configure syslog-ng to log authentications to /var/log/auth.log. In short, add destination authlog { file("/var/log/auth.log"); }; destination authlog { file("/var/log/auth.log"); }; filter f_authpriv { facility(auth, authpriv); }; log { source(src); filter(f_authpriv); destination(authlog); }; to /etc/syslog-ng/syslog-ng.conf. This will log all authentication messages to /var/log/auth.log. === Iptables === You have to configure your kernel for iptables support. [[http://iptables-tutorial.frozentux.net/iptables-tutorial.html#PREPARATIONS| Iptables-tutorial]] and/or [[http://www.gentoo.org/proj/en/infrastructure/firewall/server-firewall.xml|Gentoo Server Firewall Guide]] and/or [[http://www.gentoo.org/doc/en/security/security-handbook.xml?full=1#book_part1_chap12|Gentoo Security Handbook]] will help you configure your kernel. You won't need any specific knowledge about iptables to use this script - but it's probably a good time to get up-to-date with GNU/Linux's packet filtering anyway. This [[http://forums.gentoo.org/viewtopic.php?t=159133&highlight=iptables+howto| iptables howto]] looks like a good starting point. === Root access === I strongly suggest disabling root access in /etc/ssh/sshd_config: PermitRootLogin no (don't forget to restart the SSH server). === Gentoo users === Portage is programmed in Python, so, you got Python installed already. If you don't know which system logger you use execute # rc-status | grep log To get syslog-ng, logtail and iptables execute # emerge app-admin/syslog-ng # emerge app-admin/logsentry # emerge net-firewall/iptables # rc-update add syslog-ng default == How to customise == ==== Root login tries ==== //blacklist.py// will block out an ip instantly if it tries to login with root. If you don't like that remove # no tolerance for a root login attempt if ( match[ 0 ] == "root" ): entry[ 1 ] += PERMITTED_LOGIN_FAILURES from the script. ==== LOG_INPUT ==== This is the logfile where //blacklist.py// reads authentication failures from. ==== LOG_OUTPUT ==== This is the file //blacklist.py// writes it's taken actions to. ==== PID_FILE ==== The path to the file the PID is written to. ==== LOGTAIL ==== The path to logtail. ==== WHOAMI ==== The path to whoami. ==== PERMITTED_LOGIN_FAILURES ==== Any login failure from the same ip above this threshold results in getting blocked out. ==== BLOCKING_PERIOD ==== The time in seconds to block an ip after PERMITTED_LOGIN_FAILURES failures. Keep in mind that this period increases with the number of attacks during a run. For every hundred login failure an additional second is added. ==== SUSPECTING_PERIOD ==== The time in seconds to count a login failure as a possible attack. If SUSPECTING_PERIOD passed since the last login failure and the IP is not blocked it gets removed. ==== CHECK_INTERVALL ==== The time in seconds when the script checks if it should unblock or remove an IP. So, in the worst case an IP gets blocked (BLOCKING_PERIOD + CHECK_INTERVALL) seconds. ==== SLEEP_PERIOD ==== The time in seconds the main thread sleeps before rescanning log entries. **Keep in mind that an ip may succeed in trying to login more than PERMITTED_LOGIN_FAILURES. This is because the main thread only runs every SLEEP_PERIOD seconds. Decrease SLEEP_PERIOD to your liking**. ==== DATE_FORMAT ==== The date format used for LOG_OUTPUT. ==== SSH_PORT ==== The port your SSH server listens to. ==== FTP_PORT ==== THe port your FTP server listens to. ==== SENDMAIL ==== The path to sendmail ==== MAIL_SENDER ==== The "From: " address for the mail. ==== MAIL_RECEIVER ==== Whom the mail is sent to. ==== SSH_REGEX ==== This is the regex to catch ssh login failures. It is a comma seperated list (meaning you could just add other regexs). Please use the [[blacklist#Test_mail_notification]] to verify your regexs. It catches following similar logging entries: Jan 2 21:48:05 blinkeye sshd[4529]: Failed password for invalid user sato from 61.172.192.3 port 54177 ssh2 Jan 2 21:48:05 blinkeye sshd[4529]: Failed password for invalid user sato from ::ffff:61.172.192.3 port 54177 ssh2 Oct 21 18:52:01 blinkeye sshd[31286]: Failed password for root from 152.149.148.115 port 36667 ssh2 Oct 21 18:52:01 blinkeye sshd[31286]: Failed password for root from ::ffff:152.149.148.115 port 36667 ssh2 Sep 18 05:08:06 blinkeye sshd[3971]: Failed keyboard-interactive/pam for root from 152.149.148.115 port 44896 ssh2 Sep 18 05:08:06 blinkeye sshd[3971]: Failed keyboard-interactive/pam for root from ::ffff:152.149.148.115 port 44896 ssh2 Mar 7 02:37:29 blinkeye sshd[8559]: User root from 61.178.20.170 not allowed because not listed in AllowUsers Mar 7 02:37:29 blinkeye sshd[8559]: User root from ::ffff:61.178.20.170 not allowed because not listed in AllowUsers Mar 7 12:57:11 blinkeye sshd[14334]: User admin from 205.241.227.14 not allowed because not listed in AllowUsers Mar 7 12:57:11 blinkeye sshd[14334]: User admin from ::ffff:205.241.227.14 not allowed because not listed in AllowUsers Mar 7 14:27:16 blinkeye sshd[15213]: User apache from 218.71.137.69 not allowed because not listed in AllowUsers Mar 7 14:27:16 blinkeye sshd[15213]: User apache from ::ffff:218.71.137.69 not allowed because not listed in AllowUsers ==== FTP_REGEX ==== This is the regex to catch ftp login failures (currently created for vsftpd). It is a comma seperated list (meaning you could just add other regexs). Please use the [[blacklist#Test_mail_notification]] to verify your regexs. It catches following similar logging entries: Oct 3 19:35:41 blinkeye ftp(pam_unix)[8746]: authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=206.222.29.194 Oct 3 19:35:41 blinkeye ftp(pam_unix)[8746]: authentication failure; logname= uid=0 euid=0 tty= ruser= ::ffff:rhost=206.222.29.194 Oct 3 19:35:43 blinkeye ftp(pam_unix)[8746]: authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=206.222.29.194 user=root Oct 3 19:35:43 blinkeye ftp(pam_unix)[8746]: authentication failure; logname= uid=0 euid=0 tty= ruser= ::ffff:rhost=206.222.29.194 user=root === Advanced Configuration === Following are some adjustement targeted to the advanced user. If you don't understand what I'm talking about don't worry about these settings. ==== Don't give him a clue ==== If you don't want your attacker to know what's going on you might want to surpress ssh: connect to host blinkeye.ch port 22: Connection refused In that case change system_command( IPTABLES + " --insert " + CUSTOM_CHAIN + " --source " + self.threadParameter + \ " --protocol tcp --dport " + str( SSH_PORT ) + " --jump REJECT" ) to system_command( IPTABLES + " --insert " + CUSTOM_CHAIN + " --source " + self.threadParameter + \ " --protocol tcp --dport " + str( SSH_PORT ) + " --jump DROP" ) and system_command( IPTABLES + " --delete " + CUSTOM_CHAIN + " --source " + self.threadParameter + \ " --protocol tcp --dport " + str( SSH_PORT ) + " --jump REJECT" ) to system_command( IPTABLES + " --delete " + CUSTOM_CHAIN + " --source " + self.threadParameter + \ " --protocol tcp --dport " + str( SSH_PORT ) + " --jump DROP" ) (replace REJECT with DROP). If you DROP his IP he will just get a hanging session and eventually a connection timeout. ==== Advanced iptables configuration ==== If you use iptables to secure your server you might have rules you want to take precedence before the CUSTOM_CHAIN of //blacklist.py//. In this case, use # iptables -L -n to define the index (starting at 1) and change system_command( IPTABLES + " --insert INPUT --jump " + CUSTOM_CHAIN ) to system_command( IPTABLES + " --insert INPUT index --jump " + CUSTOM_CHAIN ) #replace index with your desired insert location Per default an established connection from the attackers IP (meaning someone who logged in successfully earlier) will hang after that IP gets blocked out. If you have a RELATED,ESTABLISHED rule in your INPUT CHAIN like # iptables -L -n Chain INPUT (policy DROP) target prot opt source destination ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED you could set the insert command to system_command( IPTABLES + " --insert INPUT 2 --jump " + CUSTOM_CHAIN ) which would add //blacklist.py//'s CUSTOM_CHAIN after the RELATED,ESTABLISHED rule Chain BLACKLIST (1 references) target prot opt source destination
Chain INPUT (policy DROP) Target prot opt source destination ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED BLACKLIST all -- 0.0.0.0/0 0.0.0.0/0 and blocking an IP will not affect any earlier successfully made connections. == The script == #!/usr/bin/python # blacklist.py is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # blacklist.py is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # Copyright: Reto Glauser aka blinkeye # Mailto: blacklist at blinkeye dot ch # Homepage: http://blinkeye.ch # Forum post: http://forums.gentoo.org/viewtopic-t-421706.html # Date: 2006-03-16 # Version 0.4.5 import re; import commands; import thread; import threading; import sys; from os import access, popen, R_OK, W_OK, X_OK; from time import sleep, strftime, time; from string import find; from syslog import *; import errno; import os; LOG_INPUT = "/var/log/auth.log" LOG_OUTPUT = "/var/log/blacklist.log" PID_FILE = "/var/run/blacklist.pid" LOGTAIL = "/usr/sbin/logtail" WHOAMI = "/usr/bin/whoami" IPTABLES = "/sbin/iptables" CUSTOM_CHAIN = "BLACKLIST" PERMITTED_LOGIN_FAILURES = 6 BLOCKING_PERIOD = 600 #seconds SUSPECTING_PERIOD = 600 #seconds SLEEP_PERIOD = 30 #seconds CHECK_INTERVALL = 300 #seconds DATE_FORMAT = "%d.%m.%Y %X" # e.g.: 02.01.2006 23:49:12 SSH_PORT = 22 FTP_PORT = 21 SENDMAIL = "/usr/sbin/sendmail" MAIL_SENDER = "blacklist@yourdomain" MAIL_RECEIVER = "ssh@yourdomain" SSH_REGEX = [ r"Failed (?:none|password|keyboard-interactive/pam) for (?:invalid user )*(?P.*) from (?:::ffff:)*(?P\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})", r"Invalid user (?P.*) from (?:::ffff:)*(?P\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})" ] # SSH_REGEX catches following similar entries: # Jan 2 21:48:05 blinkeye sshd[4529]: Failed password for invalid user sato from 61.172.192.3 port 54177 ssh2 # Jan 2 21:48:05 blinkeye sshd[4529]: Failed password for invalid user sato from ::ffff:61.172.192.3 port 54177 ssh2 # Oct 21 18:52:01 blinkeye sshd[31286]: Failed password for root from 152.149.148.115 port 36667 ssh2 # Oct 21 18:52:01 blinkeye sshd[31286]: Failed password for root from ::ffff:152.149.148.115 port 36667 ssh2 # Sep 18 05:08:06 blinkeye sshd[3971]: Failed keyboard-interactive/pam for root from 152.149.148.115 port 44896 ssh2 # Sep 18 05:08:06 blinkeye sshd[3971]: Failed keyboard-interactive/pam for root from ::ffff:152.149.148.115 port 44896 ssh2 FTP_REGEX = [ r"ftp(?:.*) authentication failure(?:.*) rhost=(?:::ffff:)*(?P\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) (?: user=)*(?P.*)" ] # FTP_REGEX catches following similar entries: # Oct 3 19:35:41 blinkeye ftp(pam_unix)[8746]: authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=206.222.29.194 # Oct 3 19:35:43 blinkeye ftp(pam_unix)[8746]: authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=206.222.29.194 user=root # Use # grep Failed /var/log/auth.log | grep -o '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort | uniq -c # to get a statistic of ip's from past login failures # Use # grep "Failed" /var/log/auth.log | sed "s/.*for\( invalid user\)*\(.*\)\(from.*\)/\2/" | sort | uniq -c # to get a statistic of login names from past login failures # Wrapper function for commands def system_command( string_command ): return_value = [ ] return_value = ( commands.getstatusoutput( string_command ) ) if not return_value[ 0 ] == 0: raise IOError( return_value[ 1 ] ) return return_value[ 1 ] # block ip for the duration of time def block( ip, time, port ): try: system_command( IPTABLES + " --new-chain " + CUSTOM_CHAIN ) system_command( IPTABLES + " --insert INPUT --jump " + CUSTOM_CHAIN ) except: None system_command( IPTABLES + " --insert " + CUSTOM_CHAIN + " --source " + ip + " --protocol tcp --dport " + str( port ) + " --jump REJECT" ) LOG_OUTPUT.write( strftime( DATE_FORMAT ) + ": Blocking " + ip + " for " + str( time ) + " seconds\n" ) LOG_OUTPUT.flush() # unblock IP def unblock( ip, port ): system_command( IPTABLES + " --delete " + CUSTOM_CHAIN + " --source " + ip + " --protocol tcp --dport " + str( port ) + " --jump REJECT" ) LOG_OUTPUT.write( strftime( DATE_FORMAT ) + ": Remove blocking " + ip + "\n" ) LOG_OUTPUT.flush() # mail list of IP blocked during this run def mail( mail_list ): count = len( mail_list ) if count == 0: return ip = str( mail_list[ 0 ][ 0 ] ) + " on port " + str( mail_list[ 0 ][ 3 ] ) + "\n" mail_list.remove( mail_list[ 0 ] ) p = popen( "%s -t" % SENDMAIL, "w" ) p.write( "From: " + MAIL_SENDER + "\n" ) p.write( "To: " + MAIL_RECEIVER + "\n" ) if count == 1: p.write( "Subject: [" + system_command( 'hostname --long' ) + "]: Too many login failures from " + ip + "\n\n" ) else: p.write( "Subject: [" + system_command( 'hostname --long' ) + "]: Too many login failures from multiple IPs\n\n" ) for entry in mail_list[ : ]: ip += entry[ 0 ] + " on port " + str( entry[ 3 ] ) + "\n" mail_list.remove( entry ) p.write( "Blocking out " + str( count ) + " IP(s):\n\n" + ip + "\n" ) if p.close() != None: LOG_OUTPUT.write( strftime( DATE_FORMAT ) + ": Unable to send a mail. Check your SENDMAIL configuration.\n" ) LOG_OUTPUT.flush() # append IPs from regex_matches to ip_list and increase counter of login failures for IPs def create_stat( regex_matches, ip_list, ip_list_blocked, delay, port ): current_time = time() + delay for match in regex_matches: entry = ip_list.get( match.group( 'ip' ) ) if entry == None: ip_list[ match.group( 'ip' ) ] = [ match.group( 'ip' ), 1, current_time, port ] # [ [ ip ],[ counter for catched login failures ],[ time of last login failure ], [ port ] ] entry = ip_list.get( match.group( 'ip' ) ) else: entry[ 1 ] += 1 # no tolerance for a root login attempt if ( match.group( 'user' ) == "root" ): entry[ 1 ] += PERMITTED_LOGIN_FAILURES for key in ip_list.keys()[ : ]: if ip_list.get( key )[ 1 ] > PERMITTED_LOGIN_FAILURES and ip_list.get( key)[ 0 ] not in ip_list_blocked: ip_list_blocked.insert( 0, ip_list.get( key ) ) del ip_list[ key ] block( ip_list_blocked[ 0 ][ 0 ], BLOCKING_PERIOD + delay, port ) mail_list.insert( 0, ip_list_blocked[ 0 ] ) # Someone must do the work def scan(): global countdown try: new_log_entries = system_command( LOGTAIL + " -f " + LOG_INPUT ) except: new_log_entries = system_command( LOGTAIL + " " + LOG_INPUT ) for i in range( 0, len( SSH_REGEX ) ): re_ssh = re.compile( SSH_REGEX[ i ] ) regex_matches = re_ssh.finditer( new_log_entries ) create_stat( regex_matches, ssh_list, ssh_list_blocked, len( re_ssh.findall( new_log_entries ) )/100, SSH_PORT ) for i in range( 0, len( FTP_REGEX ) ): re_ftp = re.compile( FTP_REGEX[ i ] ) regex_matches = re_ftp.finditer( new_log_entries ) create_stat( regex_matches, ftp_list, ftp_list_blocked, len( re_ftp.findall( new_log_entries ) )/100, FTP_PORT ) # +++ mail section +++ # mail( mail_list ) # +++ mail section +++ # if countdown <= 0: countdown = CHECK_INTERVALL current_time = time() for entry in ssh_list_blocked[ : ]: if( current_time - entry[ 2 ] ) > BLOCKING_PERIOD: unblock( entry[ 0 ], SSH_PORT ) ssh_list_blocked.remove( entry ) for key in ssh_list.keys(): if( current_time - ssh_list.get( key )[ 2 ] ) > SUSPECTING_PERIOD: del ssh_list[ key ] for entry in ftp_list_blocked[ : ]: if( current_time - entry[ 2 ] ) > BLOCKING_PERIOD: unblock( entry[ 0 ], FTP_PORT ) ftp_list_blocked.remove( entry ) for key in ftp_list.keys(): if( current_time - ftp_list.get( key )[ 2 ] ) > SUSPECTING_PERIOD: del ftp_list[ key ] # Check if there's another instance running def handlepid(): try: pidfile = os.fdopen( os.open( PID_FILE, os.O_WRONLY | os.O_CREAT | os.O_EXCL ), 'w' ) except OSError: try: pid = int( open( PID_FILE ).read() ) except IOError: sys.exit( "Error opening pidfile %s" %PID_FILE ) try: os.kill( pid, 0 ) except OSError, why: if why.errno == errno.ESRCH: print "Removing stale pidfile %s with pid %d\n" %( PID_FILE, pid ) os.remove( PID_FILE ) if os.path.exists( PID_FILE ): sys.exit( "Cannot remove pidfile." ) else: return handlepid() sys.exit( "\nAnother blacklist daemon is running with pid %d" %pid ) pidfile.write( "%d\n" %os.getpid() ) pidfile.flush() pidfile.close() cleanup(); if not system_command( "whoami" ) == "root": raise IOError, "This script must be run as root" if not access( LOG_INPUT, R_OK ): raise IOError, LOG_INPUT + " is not readable" if not access( LOGTAIL, X_OK ): raise IOError, LOGTAIL + " is not executable" if not access( WHOAMI, X_OK ): raise IOError, WHOAMI + " is not executable" if not access( IPTABLES, X_OK ): raise IOError, IPTABLES + " is not executable" # +++ mail section +++ # if not access( SENDMAIL, X_OK ): raise IOError, SENDMAIL + " is not executable" # +++ mail section +++ # # test modus if len( sys.argv ) == 2: print( "\n* Entering test mode" ) for i in range( 0, len( SSH_REGEX ) ): re_ssh = re.compile( SSH_REGEX[ i ] ) if not len( re_ssh.findall( sys.argv[ 1 ] ) ): print( "* SSH_REGEX[ " + str( i ) + " ]: No match found" ) else: regex_matches = re_ssh.finditer( sys.argv[ 1 ] ) for match in regex_matches: print( "* SSH_REGEX[ " + str( i ) + " ]: Caught ip \"" + str( match.group( 'ip' ) ) + " and username \"" + str( match.group( 'user' ) ) + "\"" ) for i in range( 0, len( FTP_REGEX ) ): re_ftp = re.compile( FTP_REGEX[ i ] ) if not len( re_ftp.findall( sys.argv[ 1 ] ) ): print( "* FTP_REGEX[ " + str( i ) + " ]: No match found" ) else: regex_matches = re_ftp.finditer( sys.argv[ 1 ] ) for match in regex_matches: print( "* FTP_REGEX[ " + str( i ) + " ]: Caught ip \"" + str( match.group( 'ip' ) ) + "\" and username \"" + str( match.group( 'user' ) ) + "\"" ) p = popen( "%s -t" % SENDMAIL, "w" ) p.write( "From: " + MAIL_SENDER + "\n" ) p.write( "To: " + MAIL_RECEIVER + "\n" ) p.write( "Subject: blacklist.py ist testing your sendmail configuration\n\n" ) p.write( "A test mail from blacklist.py\n" ) if p.close() != None: print( "* ERROR sending a mail. Check your SENDMAIL configuration.\n" ) else: print( "* SUCCESS: Sending mail from " + MAIL_SENDER + " to " + MAIL_RECEIVER ) sys.exit(0) # Cleanup def cleanup(): try: system_command( IPTABLES + " --delete INPUT -j " + CUSTOM_CHAIN ) system_command( IPTABLES + " --flush " + CUSTOM_CHAIN ) system_command( IPTABLES + " --delete-chain " + CUSTOM_CHAIN ) except: None # Call cleanup() before exiting the script sys.exitfunc = cleanup handlepid() LOG_OUTPUT = file( LOG_OUTPUT, "a") ssh_list = { } ssh_list_blocked = [ ] ftp_list = { } ftp_list_blocked = [ ] mail_list = [ ] countdown = CHECK_INTERVALL cleanup() # remove old entries while 1: scan() sleep( SLEEP_PERIOD ) countdown -= SLEEP_PERIOD == Run it == Either copy and paste the script from above (keep in mind that Python does not tolerate any whitespace deviation) or: # wget ftp://blinkeye.ch/public/blacklist.py and make it executable # chmod +x blacklist.py **Important**: Execute the following command to set logtail up-to-date or you'll block out IP addresses for BLOCKING_PERIOD which lay way in the past (must be done only once): # logtail /var/log/auth.log **Important**: If you don't have a MTA you must comment all lines (several occurences) between the tag # +++ mail section +++ # You won't get an email when IPs are blocked, but //blacklist.py// will work nevertheless. Run it: # ./blacklist.py & Login to another box and ssh to your server using wrong passwords. === Watching the magic === # tail -f /var/log/blacklist.log If an IP Address is added to blacklist.log verify that it actually is getting blocked: # iptables -L -n Chain BLACKLIST (1 references) target prot opt source destination REJECT tcp -- 220.245.172.219 0.0.0.0/0 tcp dpt:22 reject-with icmp-port-unreachable
Chain INPUT (policy DROP) target prot opt source destination BLACKLIST all -- 0.0.0.0/0 0.0.0.0/0 ... === Watching iptables === You can actually watch iptables change over time (instead of using repeatedly 'iptables -L -n'): # watch -d 'iptables -L -n' This will execute 'iptables -L -n' every 2 seconds and update your screen. === Test mode / Test regex / Test mail notification === You can test if //blacklist.py// catches your log lines. This comes in handy if you want to adjust or try new regexs or just want to be sure. Execute //blacklist.py// and pass the loglines (as many as you'd like) within quotes: # ./blacklist.py "Jan 2 21:48:05 blinkeye sshd[4529]: Failed password for invalid user sato from 61.172.192.3 port 54177 ssh2 Jan 2 21:48:05 blinkeye sshd[4529]: Failed password for invalid user sato from ::ffff:61.172.192.3 port 54177 ssh2 Oct 21 18:52:01 blinkeye sshd[31286]: Failed password for root from 152.149.148.115 port 36667 ssh2 Oct 21 18:52:01 blinkeye sshd[31286]: Failed password for root from ::ffff:152.149.148.115 port 36667 ssh2 Sep 18 05:08:06 blinkeye sshd[3971]: Failed keyboard-interactive/pam for root from 152.149.148.115 port 44896 ssh2 Sep 18 05:08:06 blinkeye sshd[3971]: Failed keyboard-interactive/pam for root from ::ffff:152.149.148.115 port 44896 ssh2 Oct 3 19:35:41 blinkeye ftp(pam_unix)[8746]: authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=206.222.29.194 Oct 3 19:35:41 blinkeye ftp(pam_unix)[8746]: authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=::ffff:206.222.29.194 Oct 3 19:35:43 blinkeye ftp(pam_unix)[8746]: authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=206.222.29.194 user=root Oct 3 19:35:43 blinkeye ftp(pam_unix)[8746]: authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=::ffff:206.222.29.194 user=root" * Entering test mode * SSH_REGEX[ 0 ]: Caught ip "61.172.192.3 and username "sato" * SSH_REGEX[ 0 ]: Caught ip "61.172.192.3 and username "sato" * SSH_REGEX[ 0 ]: Caught ip "152.149.148.115 and username "root" * SSH_REGEX[ 0 ]: Caught ip "152.149.148.115 and username "root" * SSH_REGEX[ 0 ]: Caught ip "152.149.148.115 and username "root" * SSH_REGEX[ 0 ]: Caught ip "152.149.148.115 and username "root" * SSH_REGEX[ 1 ]: No match found * FTP_REGEX[ 0 ]: Caught ip "206.222.29.194" and username "" * FTP_REGEX[ 0 ]: Caught ip "206.222.29.194" and username "" * FTP_REGEX[ 0 ]: Caught ip "206.222.29.194" and username "root" * FTP_REGEX[ 0 ]: Caught ip "206.222.29.194" and username "root" * SUCCESS: Sending mail from blacklist@yourdomain to ssh@yourdomain For every regex in SSH_REGEX and FTP_REGEX it will try to match an ip and username. The ip catch is mandatory - but it does need to catch a username. == Server Reboot / Cleaning up == To run //blacklist.py// automatically upon reboot add //blacklist.py// to a runlevel (creating an init script) or even better add it directly to your firewall/iptables script. The script will remove any blocked IP and the CUSTOM_CHAIN every time it is restarted. In case you kill //blacklist.py// you may want to clean up your iptables' rules. The following will remove anything added by the script (only necessary if you kill it): # /sbin/iptables --delete INPUT -j CUSTOM_CHAIN # /sbin/iptables --flush CUSTOM_CHAIN # /sbin/iptables --delete-chain CUSTOM_CHAIN Replace CUSTOM_CHAIN with the used CHAIN name (default 'BLACKLIST' ). **UPDATE: You may safely reset your iptable rules while running //blacklist.py//. It will (re)add it's needed rules automatically when blocking the next IP.** == DoS / Performance / Stress Test == Let's see how the script works under a simulated [[http://en.wikipedia.org/wiki/Ddos|Denial of Service]] attack. We don't want to use a defense mechanism which would indirectly create a DoS under heavy load. === Scenario === Your SSH Server has had a lot of activity during the last SLEEP_PERIOD, namely * 500000 (half a million) new log entries * 50 MB of loglines to parse * 375000 IP occurences * 125000 regex matches * 5000 different IPs * Reocurring failed login entries are scattered throughout the 500000 log lines to simulate a scheduled attack ==== Create the logfile ==== #!/usr/bin/python from random import randint; from time import strftime; DATE_FORMAT = "%d.%m.%Y %X" # e.g.: 02.01.2006 23:49:12 # stress test / DoS simulation. Creates random private A-Class IPs from 10.0.0.0 to 10.254.254.254 def DoS(): DoS_list = [ ] user_list = [ "root", "admin", "guest", "test", "sql", "web", "www", "apache", "mysql", "webmail", "mail", "sysadmin", "tomcat", "webmaster", "hostmaster" ] for j in range( 0, 5000 ): DoS_list.append( "10." + str( randint( 1, 244 ) ) + "." + str( randint( 1, 244 ) ) + "." + str( randint( 1, 244 ) ) + " port " + str( randint( 1024, 65535) ) ) for k in range( 0, 125000 ): user = user_list[ randint(0, 14) ] ip = DoS_list[ randint( 0, 4999 ) ] print( str( strftime( "%b %d %X" ) ) + " x40 sshd[" + str( randint( 1024, 65535 ) ) + "]: Failed keyboard-interactive/pam for invalid user " + user + " from " + ip + " ssh2" ) print( str( strftime( "%b %d %X" ) ) + " x40 sshd[" + str( randint( 1024, 65535 ) ) + "]: check pass; user unknown" ) print( str( strftime( "%b %d %X" ) ) + " x40 sshd[" + str( randint( 1024, 65535 ) ) + "]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=" + ip ) print( str( strftime( "%b %d %X" ) ) + " x40 sshd[" + str( randint( 1024, 65535 ) ) + "]: error: PAM: Authentication failure for " + user + " from " + ip ) DoS() This script will print randomized log entries ( don't worry, these IP addresses are all private - just make sure you don't use an IP address from 10.X.X.X yourself ). ==== Pentium M 1700 Mhz, 1024 MB RAM ==== This is the time for each run (SLEEP_PERIOD is 30 seconds): * Run took: 66.2886929512 seconds * Run took: 0.0120289325714 seconds * Run took: 0.0120761394501 seconds ... * Run took: 0.0120210647583 seconds * Run took: 64.545445919 seconds ... ==== Pentium M 1200 Mhz, 512 MB RAM ==== This is the time for each run (SLEEP_PERIOD is 30 seconds): * Run took: 105.404068947 seconds * Run took: 0.0309278964996 seconds * Run took: 0.0309059619904 seconds ... * Run took: 98.3297541142 seconds * Run took: 0.0316820144653 seconds ... ==== Pentium M 222 Mhz, 1024 RAM ==== This is the time for each run (SLEEP_PERIOD is 30 seconds): * Run took: 586.678808212 seconds * Run took: 0.0743868350983 seconds * Run took: 0.0695021152496 seconds ... * Run took: 0.0639469623566 seconds * Run took: 654.533771992 seconds * Run took: 0.0641090869904 seconds ... === Conclusion === * Regex-ing the 500000 entries takes 3 seconds. * Reading, parsing and blocking the 5000 IPs took a little more than 1 minute * The 5000 IPs were blocked out for 40 minutes (BLOCKING_PERIOD plus the additional dynamic calculated time - in this case another 20 minutes). * Subsequent runs take ~0.010 (10 miliseconds) * Every CHECK_INTERVALL seconds it takes ~0.020 ( 20 miliseconds) to check if IPs should be removed or unblocked * Removing the 5000 IPs took a little more than 1 minute * Running detailed statistics revelead that the iptables command takes most of the time. That's why adding and removing the IPs take about the same time. I didn't notice a peformance loss connecting via SSH to those boxes having over 5000 iptables rules (based on personal impression). So, the script passed the test. Note: Results taken from the Pentium M 1700 Mhz runs == Contact / Issues == In case you run into problems you might want to check out the [[http://forums.gentoo.org/viewtopic-t-421706.html|Gentoo Forum post]]. Feel free to [[mailto:blacklist@blinkeye.ch mail]] me with comments, modifications or suggestions. In case you use a different logger and the script doesn't work for you provide me a detailed log and information. == Changelog == **2006-01-06 ** * Adding Howto. **2006-01-11** * Adding section [[blacklist#Advanced_Configuration]] * Adding section [[blacklist#Watching_iptables]] * Adding cleanup function * Adding system_command function to catch run time errors * Adding custom iptables chain for easy distinction and use * Changing iptables example output. Thanks to Magic919 for pointing out the issue **2006-01-14** * Adding test modus * Adding section [[blacklist#test_mode_test_regex_test_mail_notification|Test mode / Test regex / Test mail notification]] * Adding variable SUSPECTING_PERIOD to remove failed non-attack logins **2006-01-19** * Completely rewrote the core * Removing thread logic due to limitation and ressource usage * Changing datastructure of main list from an array to a hash map to speed up searches * Adding [[blacklist#CHECK_INTERVALL | CHECK_INTERVALL]] variable * Adding dynamical [[blacklist#BLOCKING_PERIOD | BLOCKING_PERIOD]] (for every hundred login failure an additional second is added) * Modified mailing feature so you get one mail per each run (if any IP were blocked) instead of a mail per IP to prevent mail flood/ressource usage upon a DoS * Adding [[blacklist#test_mode_test_regex_test_mail_notification]] * Modifying Howto * Thorough testing **2006-03-03** * Added new function handlepid() to check if an instance is already running (thanks to Erik J.) * Added try/except block to handle the issue if iptables get flushed while the script is running * Added try/except block to handle the different logtail versions * Fixed an issue where wrong entries would be written to the LOG_OUTPUT (modifying the hash table while iterating through it without making a copy) * Minor speed improvement * Added forum link to the header **2006-03-15** * Added new functionality to permit several regexs at once * Added ftp regex * Changed subject of the status mail to contain the hostname * The mail now contains the port an IP is blocked on * Correctly blocking out invalid users/login tries if the ssh daemon is configured with the "AllowUsers" variable * Enhanced the test modus to support several loglines **2006-03-16** * Fixed ftp regex (vsftp) **2006-03-22** * Updated Wiki **2006-03-29** * Updated Wiki **2009-04-13** * Moved content from Mediawiki to Dokuwiki == Sources/Further Information == [[http://www.gentoo.org/proj/en/infrastructure/firewall/server-firewall.xml|Server Firewall (Gentoo Doc)]] [[http://www.gentoo.org/doc/en/articles/dynamic-iptables-firewalls.xml|Dynamic Iptables (Gentoo Doc)]] [[http://iptables-tutorial.frozentux.net/chunkyhtml|A complete Iptables Tutorial]] [[http://staff.washington.edu/dittrich/misc/ddos|Distributed Denial of Service (DDoS) Attacks/tools]]