| Message ID | 20200427143123.6378-1-ipfr@tfitzgeorge.me.uk |
|---|---|
| Headers |
Return-Path: <development-bounces@lists.ipfire.org> Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384 client-signature ECDSA (P-384) client-digest SHA384) (Client CN "mail01.haj.ipfire.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 499nKh5GTDz3xQy for <patchwork@web04.haj.ipfire.org>; Mon, 27 Apr 2020 14:32:08 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail02.haj.ipfire.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 499nKf6wk3zXb; Mon, 27 Apr 2020 14:32:06 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 499nKf4hbNz2yVN; Mon, 27 Apr 2020 14:32:06 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384 client-signature ECDSA (P-384) client-digest SHA384) (Client CN "mail01.haj.ipfire.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 499nKc5ftSz2xnp for <development@lists.ipfire.org>; Mon, 27 Apr 2020 14:32:04 +0000 (UTC) Received: from smtp.hosts.co.uk (smtp.hosts.co.uk [85.233.160.19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (Client did not present a certificate) by mail01.ipfire.org (Postfix) with ESMTPS id 499nKZ63cszg3 for <development@lists.ipfire.org>; Mon, 27 Apr 2020 14:32:02 +0000 (UTC) Received: from [95.149.142.196] (helo=aragorn.hosts.co.uk.tfitzgeorge.me.uk) by smtp.hosts.co.uk with esmtpa (Exim) (envelope-from <ipfr@tfitzgeorge.me.uk>) id 1jT4nb-0003Z9-6T; Mon, 27 Apr 2020 15:31:56 +0100 From: Tim FitzGeorge <ipfr@tfitzgeorge.me.uk> To: development@lists.ipfire.org Subject: [PATCH v2 0/8] ipblacklist: IP Address Blacklists Date: Mon, 27 Apr 2020 15:31:15 +0100 Message-Id: <20200427143123.6378-1-ipfr@tfitzgeorge.me.uk> X-Mailer: git-send-email 2.16.4 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=lists.ipfire.org; s=202003rsa; t=1587997923; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc; bh=mw/y6OgFT/Km6IuFuVzWKaca22Vbl86YVckmxcOi0bs=; b=YNdUEwR5o8XyikuHjLholTJyBiej7sizeZknMH4AQGgE+Obd/eX6TYWCcNP/H1MYko1gqH uBREgb8wfcYuL85buvh+W8qVJiGH9kpc8DGSsQzRR2tiJiZEfYxTY2amPU4ht6QdeD+AMk CYk+6ZTysVsQgTdtwdLGAEnRNkD8BEmKOWl47qWcBfLxP/9AyEk9gdKhIisJGLQRu8K6R0 XkFsCDEUN+WSnvrvShyT+YbWqQd0zut72cJGGFHXAIf4whlITrRrSzFn0GoQkxLl7tYUrA ar0YajEBJ0JcVSANiu9hEfPxkWdrLA2oyxmdUJlb1heEEgJCv2d5piR6JVbizw== ARC-Seal: i=1; s=202003rsa; d=lists.ipfire.org; t=1587997923; a=rsa-sha256; cv=none; b=f4hQ/r0jja4OCqXnmW0jC9XoFCTJoz2gEgJCPF2FWFSfo3gLcumZu/GPAt5N7FBUIF7PYc Zyrw/Ny/sGo/ODn9F3HMzSB35IHmdp7fgsPdfQ9meKEiGKi4ezs0jUvlaeVl0FOgvoLXX3 f3/b71FfVNf0LX/QO+VKjcqv26ZJPdxe2A6nyJ5vnq9pBSO7yIrc8y9M8gmCcAwbFRzBS1 YOjDOOsLP71cbw3T2IJGJQ5I9GGqzdOzPc85dC61LNpp9O8h6z3GAZpLkkSa5Vn0xzXZQ2 /1bR3MlGiUwSnakVNOLlg9ofhUYwMoLLdsphy0N4T+uXGr+Z9PNzA2WBvnoNQg== ARC-Authentication-Results: i=1; mail01.ipfire.org; dkim=none; spf=pass (mail01.ipfire.org: domain of ipfr@tfitzgeorge.me.uk designates 85.233.160.19 as permitted sender) smtp.mailfrom=ipfr@tfitzgeorge.me.uk Authentication-Results: mail01.ipfire.org; dkim=none; dmarc=none; spf=pass (mail01.ipfire.org: domain of ipfr@tfitzgeorge.me.uk designates 85.233.160.19 as permitted sender) smtp.mailfrom=ipfr@tfitzgeorge.me.uk X-Rspamd-Queue-Id: 499nKZ63cszg3 X-Spamd-Result: default: False [-3.30 / 11.00]; RCVD_TLS_LAST(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[85.233.160.19:from]; R_SPF_ALLOW(-0.20)[+ip4:85.233.160.0/27]; MIME_GOOD(-0.10)[text/plain]; ARC_SIGNED(0.00)[i=1]; DMARC_NA(0.00)[tfitzgeorge.me.uk]; RECEIVED_SPAMHAUS_PBL(0.00)[95.149.142.196:received]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[]; RCPT_COUNT_TWO(0.00)[2]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM(-0.93)[-0.935]; IP_REPUTATION_SPAM(0.01)[asn: 8622(0.00), country: GB(0.01), ip: 85.233.160.19(0.00)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8622, ipnet:85.233.160.0/19, country:GB]; RCVD_COUNT_TWO(0.00)[2]; BAYES_HAM(-2.96)[99.84%]; RCVD_IN_DNSWL_LOW(-0.10)[85.233.160.19:from] X-Rspamd-Server: mail01.haj.ipfire.org X-BeenThere: development@lists.ipfire.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: IPFire development talk <development.lists.ipfire.org> List-Unsubscribe: <https://lists.ipfire.org/mailman/options/development>, <mailto:development-request@lists.ipfire.org?subject=unsubscribe> List-Archive: <http://lists.ipfire.org/pipermail/development/> List-Post: <mailto:development@lists.ipfire.org> List-Help: <mailto:development-request@lists.ipfire.org?subject=help> List-Subscribe: <https://lists.ipfire.org/mailman/listinfo/development>, <mailto:development-request@lists.ipfire.org?subject=subscribe> Errors-To: development-bounces@lists.ipfire.org Sender: "Development" <development-bounces@lists.ipfire.org> |
| Series |
ipblacklist: IP Address Blacklists
|
|
Message
Tim FitzGeorge
April 27, 2020, 2:31 p.m. UTC
Implements downloading of IP address blacklists and implementing them as IPSets. A separate IPSet is used for each blacklist; this simplifies handling of overlaps between different lists. Traffic to or from the red0/ppp0 interface is checked against the IPSets. The check is placed before the IPS check as the IPSet check is much lighter on CPU use which means that overall CPU use is reduced. The available lists are defined in a separate file. A WUI page allows the desired lists to be enabled. A minimum update check interval is defined for each blacklist in the definition file. Changes since Version 1: - Changed Dshield download URL to preferred address. - Removed Abuse.ch blacklist (discontinued). - Removed Talos Malicious blacklist (not for production use). - Added Feodo recommended blacklist. - Added blocklist.de all blacklist. - Updated ignored messages in logwatch. - Modified sources file 'rate' to allow unit to be specified. - Updated sources file 'disable' to allow list to be specified. - Removed autoblacklist. - Added WUI log pages. - Removed status from settings WUI page. Tim FitzGeorge (8): ipblacklist: Main script ipblacklist: WUI Settings page ipblacklist: WUI Log page ipblacklist: WUI Log details page ipblacklist: WUI menus, language file etc ipblacklist: Ancillary files ipblacklist: Modifications to system ipblacklist: Build infrastructure config/backup/backup.pl | 1 + config/backup/include | 2 + config/ipblacklist/sources | 138 ++ config/logwatch/ipblacklist | 105 ++ config/logwatch/ipblacklist.conf | 34 + config/menu/50-firewall.menu | 5 + config/menu/70-log.menu | 5 + config/rootfiles/common/aarch64/stage2 | 1 + config/rootfiles/common/configroot | 2 + config/rootfiles/common/ipblacklist-sources | 1 + config/rootfiles/common/logwatch | 2 + config/rootfiles/common/misc-progs | 2 + config/rootfiles/common/stage2 | 1 + config/rootfiles/common/web-user-interface | 3 + config/rootfiles/common/x86_64/stage2 | 1 + html/cgi-bin/ipblacklist.cgi | 463 +++++++ html/cgi-bin/logs.cgi/ipblacklists.dat | 363 +++++ html/cgi-bin/logs.cgi/log.dat | 2 + html/cgi-bin/logs.cgi/showrequestfromblacklist.dat | 415 ++++++ langs/en/cgi-bin/en.pl | 27 +- lfs/configroot | 4 +- lfs/ipblacklist-sources | 53 + lfs/logwatch | 2 + make.sh | 1 + src/initscripts/system/firewall | 12 + src/misc-progs/Makefile | 2 +- src/misc-progs/getipsetstat.c | 25 + src/misc-progs/ipblacklistctrl.c | 48 + src/scripts/ipblacklist | 1382 ++++++++++++++++++++ 29 files changed, 3098 insertions(+), 4 deletions(-) create mode 100644 config/ipblacklist/sources create mode 100644 config/logwatch/ipblacklist create mode 100644 config/logwatch/ipblacklist.conf create mode 100644 config/rootfiles/common/ipblacklist-sources create mode 100644 html/cgi-bin/ipblacklist.cgi create mode 100755 html/cgi-bin/logs.cgi/ipblacklists.dat create mode 100755 html/cgi-bin/logs.cgi/showrequestfromblacklist.dat create mode 100644 lfs/ipblacklist-sources create mode 100644 src/misc-progs/getipsetstat.c create mode 100644 src/misc-progs/ipblacklistctrl.c create mode 100755 src/scripts/ipblacklist
Comments
Hi Tim, This has now been sitting in my inbox for almost a month. Nobody else has commented on it. I tried a couple of times to review this, but I do not even know where to start. This patch is just too large. First of all, although it is split across multiple patches, it is one large patch. It is not split by changes that belong together, but just one patch per file. I have no idea what has changed since I last time looked at the code. It has been months since the previous patchset and I do not remember each individual line. You Git repository also does not have any changes any more. The branch has been reset. Are you able to send a diff with the changes since the first patchset? How can we finally bring this into the distribution? Best, -Michael > On 27 Apr 2020, at 15:31, Tim FitzGeorge <ipfr@tfitzgeorge.me.uk> wrote: > > Implements downloading of IP address blacklists and implementing > them as IPSets. A separate IPSet is used for each blacklist; this > simplifies handling of overlaps between different lists. Traffic > to or from the red0/ppp0 interface is checked against the IPSets. > The check is placed before the IPS check as the IPSet check is > much lighter on CPU use which means that overall CPU use is > reduced. > > The available lists are defined in a separate file. A WUI page > allows the desired lists to be enabled. A minimum update check > interval is defined for each blacklist in the definition file. > > Changes since Version 1: > > - Changed Dshield download URL to preferred address. > - Removed Abuse.ch blacklist (discontinued). > - Removed Talos Malicious blacklist (not for production use). > - Added Feodo recommended blacklist. > - Added blocklist.de all blacklist. > - Updated ignored messages in logwatch. > - Modified sources file 'rate' to allow unit to be specified. > - Updated sources file 'disable' to allow list to be specified. > - Removed autoblacklist. > - Added WUI log pages. > - Removed status from settings WUI page. > > Tim FitzGeorge (8): > ipblacklist: Main script > ipblacklist: WUI Settings page > ipblacklist: WUI Log page > ipblacklist: WUI Log details page > ipblacklist: WUI menus, language file etc > ipblacklist: Ancillary files > ipblacklist: Modifications to system > ipblacklist: Build infrastructure > > config/backup/backup.pl | 1 + > config/backup/include | 2 + > config/ipblacklist/sources | 138 ++ > config/logwatch/ipblacklist | 105 ++ > config/logwatch/ipblacklist.conf | 34 + > config/menu/50-firewall.menu | 5 + > config/menu/70-log.menu | 5 + > config/rootfiles/common/aarch64/stage2 | 1 + > config/rootfiles/common/configroot | 2 + > config/rootfiles/common/ipblacklist-sources | 1 + > config/rootfiles/common/logwatch | 2 + > config/rootfiles/common/misc-progs | 2 + > config/rootfiles/common/stage2 | 1 + > config/rootfiles/common/web-user-interface | 3 + > config/rootfiles/common/x86_64/stage2 | 1 + > html/cgi-bin/ipblacklist.cgi | 463 +++++++ > html/cgi-bin/logs.cgi/ipblacklists.dat | 363 +++++ > html/cgi-bin/logs.cgi/log.dat | 2 + > html/cgi-bin/logs.cgi/showrequestfromblacklist.dat | 415 ++++++ > langs/en/cgi-bin/en.pl | 27 +- > lfs/configroot | 4 +- > lfs/ipblacklist-sources | 53 + > lfs/logwatch | 2 + > make.sh | 1 + > src/initscripts/system/firewall | 12 + > src/misc-progs/Makefile | 2 +- > src/misc-progs/getipsetstat.c | 25 + > src/misc-progs/ipblacklistctrl.c | 48 + > src/scripts/ipblacklist | 1382 ++++++++++++++++++++ > 29 files changed, 3098 insertions(+), 4 deletions(-) > create mode 100644 config/ipblacklist/sources > create mode 100644 config/logwatch/ipblacklist > create mode 100644 config/logwatch/ipblacklist.conf > create mode 100644 config/rootfiles/common/ipblacklist-sources > create mode 100644 html/cgi-bin/ipblacklist.cgi > create mode 100755 html/cgi-bin/logs.cgi/ipblacklists.dat > create mode 100755 html/cgi-bin/logs.cgi/showrequestfromblacklist.dat > create mode 100644 lfs/ipblacklist-sources > create mode 100644 src/misc-progs/getipsetstat.c > create mode 100644 src/misc-progs/ipblacklistctrl.c > create mode 100755 src/scripts/ipblacklist > > -- > 2.16.4 >
Hi Michael, The diff between the V1 and V2 patches is at the end of this email. I've not included html/cgi-bin/logs.cgi/ipblacklists.dat html/cgi-bin/logs.cgi/showrequestfromblacklist.dat (V2 patches 0003 and 0004) as they're completely new at V2 - they're modifications of the other similar log files. There are a lot of lines here, but a lot of them are deletions due to moving the status etc. into the logging. I hope the is OK. Tim On 16/05/2020 10:40, Michael Tremer wrote: > Hi Tim, > > This has now been sitting in my inbox for almost a month. Nobody else has commented on it. > > I tried a couple of times to review this, but I do not even know where to start. This patch is just too large. > > First of all, although it is split across multiple patches, it is one large patch. It is not split by changes that belong together, but just one patch per file. > > I have no idea what has changed since I last time looked at the code. It has been months since the previous patchset and I do not remember each individual line. > > You Git repository also does not have any changes any more. The branch has been reset. > > Are you able to send a diff with the changes since the first patchset? > > How can we finally bring this into the distribution? > > Best, > -Michael > >> On 27 Apr 2020, at 15:31, Tim FitzGeorge <ipfr@tfitzgeorge.me.uk> wrote: >> >> Implements downloading of IP address blacklists and implementing >> them as IPSets. A separate IPSet is used for each blacklist; this >> simplifies handling of overlaps between different lists. Traffic >> to or from the red0/ppp0 interface is checked against the IPSets. >> The check is placed before the IPS check as the IPSet check is >> much lighter on CPU use which means that overall CPU use is >> reduced. >> >> The available lists are defined in a separate file. A WUI page >> allows the desired lists to be enabled. A minimum update check >> interval is defined for each blacklist in the definition file. >> >> Changes since Version 1: >> >> - Changed Dshield download URL to preferred address. >> - Removed Abuse.ch blacklist (discontinued). >> - Removed Talos Malicious blacklist (not for production use). >> - Added Feodo recommended blacklist. >> - Added blocklist.de all blacklist. >> - Updated ignored messages in logwatch. >> - Modified sources file 'rate' to allow unit to be specified. >> - Updated sources file 'disable' to allow list to be specified. >> - Removed autoblacklist. >> - Added WUI log pages. >> - Removed status from settings WUI page. >> >> Tim FitzGeorge (8): >> ipblacklist: Main script >> ipblacklist: WUI Settings page >> ipblacklist: WUI Log page >> ipblacklist: WUI Log details page >> ipblacklist: WUI menus, language file etc >> ipblacklist: Ancillary files >> ipblacklist: Modifications to system >> ipblacklist: Build infrastructure >> >> config/backup/backup.pl | 1 + >> config/backup/include | 2 + >> config/ipblacklist/sources | 138 ++ >> config/logwatch/ipblacklist | 105 ++ >> config/logwatch/ipblacklist.conf | 34 + >> config/menu/50-firewall.menu | 5 + >> config/menu/70-log.menu | 5 + >> config/rootfiles/common/aarch64/stage2 | 1 + >> config/rootfiles/common/configroot | 2 + >> config/rootfiles/common/ipblacklist-sources | 1 + >> config/rootfiles/common/logwatch | 2 + >> config/rootfiles/common/misc-progs | 2 + >> config/rootfiles/common/stage2 | 1 + >> config/rootfiles/common/web-user-interface | 3 + >> config/rootfiles/common/x86_64/stage2 | 1 + >> html/cgi-bin/ipblacklist.cgi | 463 +++++++ >> html/cgi-bin/logs.cgi/ipblacklists.dat | 363 +++++ >> html/cgi-bin/logs.cgi/log.dat | 2 + >> html/cgi-bin/logs.cgi/showrequestfromblacklist.dat | 415 ++++++ >> langs/en/cgi-bin/en.pl | 27 +- >> lfs/configroot | 4 +- >> lfs/ipblacklist-sources | 53 + >> lfs/logwatch | 2 + >> make.sh | 1 + >> src/initscripts/system/firewall | 12 + >> src/misc-progs/Makefile | 2 +- >> src/misc-progs/getipsetstat.c | 25 + >> src/misc-progs/ipblacklistctrl.c | 48 + >> src/scripts/ipblacklist | 1382 ++++++++++++++++++++ >> 29 files changed, 3098 insertions(+), 4 deletions(-) >> create mode 100644 config/ipblacklist/sources >> create mode 100644 config/logwatch/ipblacklist >> create mode 100644 config/logwatch/ipblacklist.conf >> create mode 100644 config/rootfiles/common/ipblacklist-sources >> create mode 100644 html/cgi-bin/ipblacklist.cgi >> create mode 100755 html/cgi-bin/logs.cgi/ipblacklists.dat >> create mode 100755 html/cgi-bin/logs.cgi/showrequestfromblacklist.dat >> create mode 100644 lfs/ipblacklist-sources >> create mode 100644 src/misc-progs/getipsetstat.c >> create mode 100644 src/misc-progs/ipblacklistctrl.c >> create mode 100755 src/scripts/ipblacklist >> >> -- >> 2.16.4 >> > diff --git a/config/firewall/firewall-policy b/config/firewall/firewall-policy index 1198d120f..21165e933 100755 --- a/config/firewall/firewall-policy +++ b/config/firewall/firewall-policy @@ -22,7 +22,6 @@ eval $(/usr/local/bin/readhash /var/ipfire/ethernet/settings) eval $(/usr/local/bin/readhash /var/ipfire/firewall/settings) eval $(/usr/local/bin/readhash /var/ipfire/optionsfw/settings) -eval $(/usr/local/bin/readhash /var/ipfire/ipblacklist/settings) function iptables() { /sbin/iptables --wait "$@" @@ -98,10 +97,6 @@ case "${HAVE_OPENVPN},${POLICY}" in ;; esac -if [ "${AUTOBLACKLIST}" = "on" ]; then - iptables -A POLICYIN -i ${IFACE} -m hashlimit --hashlimit-mode srcip --hashlimit-above ${BLOCK_THRESHOLD}/hour --hashlimit-name AUTOBLACKLIST -j SET --add-set AUTOBLACKLIST src -fi - case "${FWPOLICY2}" in REJECT) if [ "${DROPINPUT}" = "on" ]; then diff --git a/config/ipblacklist/sources b/config/ipblacklist/sources index ab991e12a..3cfa7f7d4 100644 --- a/config/ipblacklist/sources +++ b/config/ipblacklist/sources @@ -13,139 +13,126 @@ # # # The fields are: # # # -# name The blacklist's full name # -# url URL of the file containing the list # -# info URL giving information about the source # -# parser The parser function used to extract IP addresses from the # -# downloaded list # -# method Method used to download updates. # -# rate Minimum number of hours between checks for updates # -# safe 'yes' if the list is unlikely to contain addresses that can be # -# used for legitimate traffic, or 'no' otherwise # -# disable Name of another list to disable if this one is enabled. Used # -# when the other list is a subset of this one. # +# name The blacklist's full name # +# url URL of the file containing the list # +# info URL giving information about the source # +# parser The parser function used to extract IP addresses from the # +# downloaded list # +# rate Minimum period between checks for updates. Can be specified in # +# days (d), hours (h) or minutes (m) # +# category Used for documentation on the WUI. Can be one of the following # +# 'application' Potentially unwanted applications # +# 'attacker' Generic source of malicious packets # +# 'c and c' Malware Command and Control source # +# 'composite' Composite of other lists # +# 'invalid' Invalid addresses on the public internet # +# 'scanner' Port scanner that is not initself malicious # +# disable Name of another list to disable if this one is enabled. Used # +# when the other list is a subset of this one. # # # -# The info and safe fields are purely for documentation. # -# # -# Note that the Emerging Threats blacklist is a composite list containing # -# addresses from some of the other lists. It is unnecessary to enable # -# this list if the other lists are enabled. # +# The info and category fields are purely for documentation. # # # ############################################################################ %sources = ( 'EMERGING_FWRULE' => { 'name' => 'Emerging Threats Blocklist', 'url' => 'https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt', 'info' => 'https://doc.emergingthreats.net/bin/view/Main/EmergingFirewallRules', - 'parser' => 'text-with-hash-comments', - 'method' => 'check-header-time', - 'rate' => 1, - 'safe' => 'no' }, + 'parser' => 'ip-or-net-list', + 'rate' => '1h', + 'category' => 'composite', + 'disable' => ['FEODO_RECOMMENDED', 'FEODO_IP', 'FEODO_AGGRESIVE', 'SPAMHAUS_DROP', 'DSHIELD'] }, 'EMERGING_COMPROMISED' => { 'name' => 'Emerging Threats Compromised IPs', 'url' => 'https://rules.emergingthreats.net/blockrules/compromised-ips.txt', 'info' => 'https://doc.emergingthreats.net/bin/view/Main/CompromisedHost', - 'parser' => 'text-with-hash-comments', - 'method' => 'check-header-time', - 'rate' => 1, - 'safe' => 'no' }, + 'parser' => 'ip-or-net-list', + 'rate' => '1h', + 'category' => 'attacker' }, 'SPAMHAUS_DROP' => { 'name' => "Spamhaus Don't Route or Peer List", 'url' => 'https://www.spamhaus.org/drop/drop.txt', 'info' => 'https://www.spamhaus.org/drop/', - 'parser' => 'text-with-semicolon-comments', - 'method' => 'check-header-time', - 'rate' => 12, - 'safe' => 'yes' }, + 'parser' => 'ip-or-net-list', + 'rate' => '12h', + 'category' => 'reputation' }, 'SPAMHAUS_EDROP' => { 'name' => "Spamhaus Extended Don't Route or Peer List", 'url' => 'https://www.spamhaus.org/drop/edrop.txt', 'info' => 'https://www.spamhaus.org/drop/', - 'parser' => 'text-with-semicolon-comments', - 'method' => 'check-header-time', - 'rate' => 1, - 'safe' => 'no' }, + 'parser' => 'ip-or-net-list', + 'rate' => '1h', + 'category' => 'reputation' }, 'DSHIELD' => { 'name' => 'Dshield.org Recommended Block List', 'url' => 'https://www.dshield.org/block.txt', 'info' => 'https://dshield.org/', 'parser' => 'dshield', - 'method' => 'check-header-time', - 'rate' => 2, - 'safe' => 'no' }, + 'rate' => '1h', + 'category' => 'attacker' }, + 'FEODO_RECOMMENDED'=> {'name' => 'Feodo Trojan IP Blocklist (Recommended)', + 'url' => 'https://feodotracker.abuse.ch/downloads/ipblocklist_recommended.txt', + 'info' => 'https://feodotracker.abuse.ch/blocklist', + 'parser' => 'ip-or-net-list', + 'rate' => '5m', + 'category' => 'c and c' }, 'FEODO_IP' => { 'name' => 'Feodo Trojan IP Blocklist', 'url' => 'https://feodotracker.abuse.ch/downloads/ipblocklist.txt', 'info' => 'https://feodotracker.abuse.ch/blocklist', - 'parser' => 'text-with-hash-comments', - 'method' => 'check-header-time', - 'rate' => 1, - 'safe' => 'no' }, + 'parser' => 'ip-or-net-list', + 'rate' => '5m', + 'category' => 'c and c', + 'disable' => 'FEODO_RECOMMENDED' }, 'FEODO_AGGRESIVE' => { 'name' => 'Feodo Trojan IP Blocklist (Aggresive)', 'url' => 'https://feodotracker.abuse.ch/downloads/ipblocklist_aggressive.txt', 'info' => 'https://feodotracker.abuse.ch/blocklist', - 'parser' => 'text-with-hash-comments', - 'method' => 'check-header-time', - 'rate' => 1, - 'safe' => 'no', - 'disable' => 'FEODO_IP' }, - 'ABUSE_CH' => { 'name' => 'Abuse.ch Ransomware C&C Blocklist', - 'url' => 'https://ransomwaretracker.abuse.ch/downloads/RW_IPBL.txt', - 'info' => 'https://ransomwaretracker.abuse.ch/blocklist/', - 'parser' => 'text-with-hash-comments', - 'method' => 'check-header-time', - 'rate' => 1, - 'safe' => 'no' }, + 'parser' => 'ip-or-net-list', + 'rate' => '5m', + 'category' => 'c and c', + 'disable' => ['FEODO_IP', 'FEODO_RECOMMENDED'] }, 'CIARMY' => { 'name' => 'The CINS Army List', 'url' => 'https://cinsscore.com/list/ci-badguys.txt', 'info' => 'https://cinsscore.com/#list', - 'parser' => 'text-with-hash-comments', - 'method' => 'check-header-time', - 'rate' => 1, - 'safe' => 'no' }, + 'parser' => 'ip-or-net-list', + 'rate' => '15m', + 'category' => 'reputation' }, 'TOR_ALL' => { 'name' => 'Known TOR Nodes', 'url' => 'https://www.dan.me.uk/torlist', 'info' => 'https://www.dan.me.uk/tornodes', - 'parser' => 'text-with-hash-comments', - 'method' => 'wget', - 'rate' => 1, - 'safe' => 'no', + 'parser' => 'ip-or-net-list', + 'rate' => '1h', + 'category' => 'application', 'disable' => 'TOR_EXIT' }, 'TOR_EXIT' => { 'name' => 'Known TOR Exit Nodes', 'url' => 'https://www.dan.me.uk/torlist/?exit', 'info' => 'https://www.dan.me.uk/tornodes', - 'parser' => 'text-with-hash-comments', - 'method' => 'wget', - 'rate' => 1, - 'safe' => 'no' }, - 'TALOS_MALICIOUS' => { 'name' => 'Talos Malicious hosts list', - 'url' => 'https://www.talosintelligence.com/documents/ip-blacklist', - 'info' => 'https://www.talosintelligence.com/reputation', - 'parser' => 'text-with-hash-comments', - 'method' => 'wget', - 'rate' => 24, - 'safe' => 'no' }, + 'parser' => 'ip-or-net-list',, + 'rate' => '1h', + 'category' => 'application' }, 'ALIENVAULT' => { 'name' => 'AlienVault IP Reputation database', 'url' => 'https://reputation.alienvault.com/reputation.generic', 'info' => 'https://www.alienvault.com/resource-center/videos/what-is-ip-domain-reputation', - 'parser' => 'text-with-hash-comments', - 'method' => 'check-header-time', - 'rate' => 1, - 'safe' => 'no' }, - 'BOGON' => { 'name' => 'Bogus address list (Martian)', + 'parser' => 'ip-or-net-list', + 'rate' => '1h', + 'category' => 'reputation' }, + 'BOGON' => { 'name' => 'Bogus address list (Martian)', 'url' => 'https://www.team-cymru.org/Services/Bogons/bogon-bn-agg.txt', 'info' => 'https://www.team-cymru.com/bogon-reference.html', - 'parser' => 'text-with-hash-comments', - 'method' => 'check-header-time', - 'rate' => 24, - 'safe' => 'yes' }, - 'BOGON_FULL' => { 'name' => 'Full Bogus Address List', + 'parser' => 'ip-or-net-list', + 'rate' => '1d', + 'category' => 'invalid' }, + 'BOGON_FULL' => { 'name' => 'Full Bogus Address List', 'url' => 'https://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt', 'info' => 'https://www.team-cymru.com/bogon-reference.html', - 'parser' => 'text-with-hash-comments', - 'method' => 'check-header-time', - 'rate' => 24, - 'safe' => 'yes', + 'parser' => 'ip-or-net-list', + 'rate' => '4h', + 'category' => 'invalid', 'disable' => 'BOGON' }, - 'SHODAN' => { 'name' => 'ISC Shodan scanner blacklist', + 'SHODAN' => { 'name' => 'ISC Shodan scanner blacklist', 'url' => 'https://isc.sans.edu/api/threatlist/shodan?tab', 'info' => 'https://isc.sans.edu', - 'parser' => 'text-with-hash-comments', - 'method' => 'wget', - 'rate' => 24, - 'safe' => 'no' } + 'parser' => 'ip-or-net-list', + 'rate' => '1d', + 'category' => 'scanner' }, + 'BLOCKLIST_DE' => { 'name' => 'Blocklist.de all attacks list', + 'url' => 'https://lists.blocklist.de/lists/all.txt', + 'info' => 'https://www.blocklist.de', + 'parser' => 'ip-or-net-list', + 'rate' => '30m', + 'category' => 'attacker' } ); diff --git a/config/logwatch/ipblacklist b/config/logwatch/ipblacklist index 0fadc6250..6d6c46188 100644 --- a/config/logwatch/ipblacklist +++ b/config/logwatch/ipblacklist @@ -49,24 +49,26 @@ while (defined(my $ThisLine = <STDIN>)) my $text = $2; - if ($text =~ m/Finished updating (\w+) blacklist with (\d+) changes/) + if ($text =~ m/Updated (\w+) blacklist with (\d+) changes/) { $Updates{$1}{updates}++; $Updates{$1}{changes} += $2; } - elsif ($text !~ m/Starting IP Blacklists/ and - $text !~ m/Starting IP Blacklist processing/ and - $text !~ m/Updating \w+ blacklist/ and - $text !~ m/Stopping IP Blacklists/ and - $text !~ m/Deleting IP Blacklists/ and - $text !~ m/Completed IP Blacklist update/ and - $text !~ m/Finished IP Blacklist processing/ and - $text !~ m/Blacklist \w+ Modification times/ and - $text !~ m/Create IPTables chains for blacklist/ and - $text !~ m/Delete IPTables chains for blacklist/ and - $text !~ m/Checking modification time for blacklist/ and - $text !~ m/Restoring blacklist / and - $text !~ m/Downloading blacklist/ ) + elsif ($text !~ m/Starting IP Blacklists/ and + $text !~ m/Starting IP Blacklist processing/ and + $text !~ m/Stopping IP Blacklists/ and + $text !~ m/Deleting IP Blacklists/ and + $text !~ m/Finished IP Blacklist processing/ and + $text !~ m/Create IPTables chains for blacklist/ and + $text !~ m/Delete IPTables chains for blacklist/ and + $text !~ m/Add IP Address Blacklist update to crontab/ and + $text !~ m/Enable IP Address Blacklist update in crontab/ and + $text !~ m/Disable IP Address Blacklist updates/ and + $text !~ m/Restoring blacklist / and + $text !~ m/Blacklist \w+ changed type/ and + $text !~ m/Blacklist \w+ changed size/ and + $text !~ m/Enabling IP Blacklist logging/ and + $text !~ m/Disabling IP Blacklist logging/ ) { $Errors{$text}++; } diff --git a/config/menu/50-firewall.menu b/config/menu/50-firewall.menu index cd82bfaa3..3cfcde835 100644 --- a/config/menu/50-firewall.menu +++ b/config/menu/50-firewall.menu @@ -21,7 +21,7 @@ 'title' => "$Lang::tr{'intrusion detection system'}", 'enabled' => 1, }; - $subfirewall->{'45.ipblacklist'} = {'caption' => $Lang::tr{'ipblacklist'}, + $subfirewall->{'45.ipblacklist'} = {'caption' => $Lang::tr{'ipblacklist'}, 'uri' => '/cgi-bin/ipblacklist.cgi', 'title' => "$Lang::tr{'ipblacklist'}", 'enabled' => 1, diff --git a/config/menu/70-log.menu b/config/menu/70-log.menu index 2fa0e426e..c597de60a 100644 --- a/config/menu/70-log.menu +++ b/config/menu/70-log.menu @@ -43,12 +43,16 @@ 'title' => "$Lang::tr{'ids logs'}", 'enabled' => 1 }; + $sublogs->{'55.ipblacklist'} = {'caption' => $Lang::tr{'ipblacklist logs'}, + 'uri' => '/cgi-bin/logs.cgi/ipblacklists.dat', + 'title' => "$Lang::tr{'ipblacklist logs'}", + 'enabled' => 1 + }; $sublogs->{'55.ovpnclients'} = { 'caption' => $Lang::tr{'ovpn rw connection log'}, 'uri' => '/cgi-bin/logs.cgi/ovpnclients.dat', 'title' => "$Lang::tr{'ovpn rw connection log'}", 'enabled' => 1, - }; $sublogs->{'60.urlfilter'} = { 'caption' => $Lang::tr{'urlfilter logs'}, 'uri' => '/cgi-bin/logs.cgi/urlfilter.dat', diff --git a/config/rootfiles/common/web-user-interface b/config/rootfiles/common/web-user-interface index ea31a943a..37a62d357 100644 --- a/config/rootfiles/common/web-user-interface +++ b/config/rootfiles/common/web-user-interface @@ -45,9 +45,11 @@ srv/web/ipfire/cgi-bin/logs.cgi/firewalllogcountry.dat srv/web/ipfire/cgi-bin/logs.cgi/firewalllogip.dat srv/web/ipfire/cgi-bin/logs.cgi/firewalllogport.dat srv/web/ipfire/cgi-bin/logs.cgi/ids.dat +srv/web/ipfire/cgi-bin/logs.cgi/ipblacklists.dat srv/web/ipfire/cgi-bin/logs.cgi/log.dat srv/web/ipfire/cgi-bin/logs.cgi/ovpnclients.dat srv/web/ipfire/cgi-bin/logs.cgi/proxylog.dat +srv/web/ipfire/cgi-bin/logs.cgi/showrequestfromblacklist.dat srv/web/ipfire/cgi-bin/logs.cgi/showrequestfromcountry.dat srv/web/ipfire/cgi-bin/logs.cgi/showrequestfromip.dat srv/web/ipfire/cgi-bin/logs.cgi/showrequestfromport.dat diff --git a/html/cgi-bin/ipblacklist.cgi b/html/cgi-bin/ipblacklist.cgi index b2ccf7b3f..28b42edf2 100644 --- a/html/cgi-bin/ipblacklist.cgi +++ b/html/cgi-bin/ipblacklist.cgi @@ -17,13 +17,13 @@ # You should have received a copy of the GNU General Public License # # along with this program. If not, see <http://www.gnu.org/licenses/>. # # # -# Copyright (C) 2018 - 2019 The IPFire Team # +# Copyright (C) 2018 - 2020 The IPFire Team # # # ############################################################################### use strict; use CGI qw/:standard/; -#enable only the following on debugging purpose +# enable the following only for debugging purposes #use warnings; #use CGI::Carp 'fatalsToBrowser'; use Sort::Naturally; @@ -34,41 +34,40 @@ require "${General::swroot}/lang.pl"; require "${General::swroot}/header.pl"; ############################################################################### -# Initialize variables and hashes +# Configuration variables ############################################################################### my $settings = "${General::swroot}/ipblacklist/settings"; -my $modified = "${General::swroot}/ipblacklist/modified"; my $sources = "${General::swroot}/ipblacklist/sources"; my $getipstat = '/usr/local/bin/getipstat'; my $getipsetstat = '/usr/local/bin/getipsetstat'; my $control = '/usr/local/bin/ipblacklistctrl'; my $lockfile = '/var/run/ipblacklist.pid'; -my $autoblacklist = 'AUTOBLACKLIST'; -my %cgiparams = ('ACTION' => '', 'AUTOACTION' => ''); +my %cgiparams = ('ACTION' => ''); + +############################################################################### +# Variables +############################################################################### + my $errormessage = ''; my $updating = 0; my %mainsettings; my %color; -my %modified; my %sources; my %stats; -my %autoblock_addresses; + +# Default settings - normally overwritten by settings file my %settings = ( 'DEBUG' => 0, 'LOGGING' => 'on', - 'RATE' => 24, - 'ENABLE' => 'off', - 'BLOCK_THRESHOLD' => 10, - 'BLOCK_PERIOD' => 3600, - $autoblacklist => 'off' ); + 'ENABLE' => 'off' ); + +# Read all parameters -# Read all parameters for site Header::getcgihash( \%cgiparams); General::readhash( "${General::swroot}/main/settings", \%mainsettings ); General::readhash( "/srv/web/ipfire/html/themes/".$mainsettings{'THEME'}."/include/colors.txt", \%color ); General::readhash( $settings, \%settings ) if (-r $settings); -General::readhash( $modified, \%modified) if (-r $modified); eval qx|/bin/cat $sources| if (-r $sources); # Show Headers @@ -79,22 +78,11 @@ Header::showhttpheaders(); if ($cgiparams{'ACTION'} eq "$Lang::tr{'save'}") { - #Save Button on configsite + # Save Button my %new_settings = ( 'ENABLE' => 'off', - 'RATE' => 24, 'LOGGING' => 'off', - 'DEBUG' => 0, - 'BLOCK_THRESHOLD' => $settings{'BLOCK_THRESHOLD'} || 10, - 'BLOCK_PERIOD' => $settings{'BLOCK_PERIOD'} || 3600, - $autoblacklist => $settings{$autoblacklist} ); - - $errormessage .= "$Lang::tr{'ipblacklist invalid check rate'}<br>" if (($cgiparams{'RATE'} !~ m/^\d+$/) or - ($cgiparams{'RATE'} < 1) or - ($cgiparams{'RATE'} > 1000)); - - $new_settings{'RATE'} = $cgiparams{'RATE'}; - delete $cgiparams{'RATE'}; + 'DEBUG' => 0 ); foreach my $item ('LOGGING', 'ENABLE', keys %sources) { @@ -107,16 +95,32 @@ if ($cgiparams{'ACTION'} eq "$Lang::tr{'save'}") foreach my $list (keys %sources) { - if (exists $settings{$list} and - $settings{$list} eq 'on' and - exists $sources{$list}{'override'} and - $settings{$sources{$list}{'override'}} eq 'on') + if (exists $new_settings{$list} and + $new_settings{$list} eq 'on' and + exists $sources{$list}{'disable'}) { - $settings{$sources{$list}{'override'}} = 'off'; + my @disable; + + if ('ARRAY' eq ref $sources{$list}{'disable'}) + { + @disable = @{ $sources{$list}{'disable'} }; + } + else + { + @disable = ( $sources{$list}{'disable'} ); + } - $updating = 1; - $errormessage .= "$Lang::tr{'ipblacklist disable pre'} $sources{$list}{'override'} " . - "$Lang::tr{'ipblacklist disable mid'} $list $Lang::tr{'ipblacklist disable post'}<br>\n"; + foreach my $disable (@disable) + { + if ($new_settings{$disable} eq 'on') + { + $new_settings{$disable} = 'off'; + + $updating = 1; + $errormessage .= "$Lang::tr{'ipblacklist disable pre'} $disable " . + "$Lang::tr{'ipblacklist disable mid'} $list $Lang::tr{'ipblacklist disable post'}<br>\n"; + } + } } } @@ -140,7 +144,6 @@ if ($cgiparams{'ACTION'} eq "$Lang::tr{'save'}") } else { - $settings{$autoblacklist} = 'off'; system( "$control disable" ); } @@ -160,51 +163,11 @@ if ($cgiparams{'ACTION'} eq "$Lang::tr{'save'}") if ($updating) { system( "$control update &" ); - get_ipset_stats(); show_running(); exit 0; } } } -elsif ($cgiparams{'AUTOACTION'} eq "$Lang::tr{'save'}") -{ - $updating = 1 if ($settings{$autoblacklist} eq 'on' and not exists $cgiparams{$autoblacklist}); - $updating = 1 if ($settings{$autoblacklist} eq 'off' and exists $cgiparams{$autoblacklist}); - - $settings{$autoblacklist} = (exists $cgiparams{$autoblacklist}) ? 'on' : 'off'; - $settings{'BLOCK_THRESHOLD'} = $cgiparams{'BLOCK_THRESHOLD'}; - $settings{'BLOCK_PERIOD'} = $cgiparams{'BLOCK_PERIOD'} ; - - if (($cgiparams{'BLOCK_THRESHOLD'} !~ m/^\d+$/) or - ($cgiparams{'BLOCK_THRESHOLD'} < 1) or - ($cgiparams{'BLOCK_THRESHOLD'} > 1000000)) - { - $errormessage .= "$Lang::tr{'ipblacklist invalid threshold'}: $cgiparams{'BLOCK_THRESHOLD'}<br>"; - } - - if (($cgiparams{'BLOCK_PERIOD'} !~ m/^\d+$/) or - ($cgiparams{'BLOCK_PERIOD'} < 1) or - ($cgiparams{'BLOCK_PERIOD'} > 86400)) - { - $errormessage .= "$Lang::tr{'ipblacklist invalid block time'}: $cgiparams{'BLOCK_PERIOD'}<br>"; - } - - if ($errormessage) - { - $updating = 0; - } - else - { - General::writehash($settings, \%settings); - system( "$control autoblacklist-update" ) if ($updating); - } -} -elsif ($cgiparams{'AUTOACTION'} eq "$Lang::tr{'unblock all'}") -{ - system( "$control autoblacklist-clear" ); -} - -get_ipset_stats(); if (is_running()) { @@ -212,17 +175,12 @@ if (is_running()) exit 0; } -# Get blacklist statistics - -get_iptables_stats(); - # Show site Header::openpage($Lang::tr{'ipblacklist'}, 1, ''); Header::openbigbox('100%', 'left'); -error(); -showstatus() if ($settings{ENABLE} eq 'on'); +error() if ($errormessage); configsite(); @@ -299,10 +257,6 @@ END <tr> <td style='width:24em'>$Lang::tr{'ipblacklist log'}</td> <td><input type='checkbox' name="LOGGING" id="LOGGING"$enable></td> - <td style='width:24em'>$Lang::tr{'ipblacklist check rate'}</td> - <td> - <input type='number' name='RATE' min='1' max='1000' maxlength='7' pattern='\\d+' value='$settings{'RATE'}'> - </td> </tr> </table> <br><br> @@ -311,7 +265,7 @@ END <tr> <th align='left'>$Lang::tr{'ipblacklist id'}</th> <th align='left'>$Lang::tr{'ipblacklist name'}</th> - <th align='center'>$Lang::tr{'ipblacklist safe'}</th> + <th align='left'>$Lang::tr{'ipblacklist category'}</th> <th align='center'>$Lang::tr{'ipblacklist enable'}</th> </tr> END @@ -322,10 +276,10 @@ END foreach my $list (sort keys %sources) { - my $name = escapeHTML( $sources{$list}{'name'} ); - my $safe = $Lang::tr{$sources{$list}{safe}}; - $enable = ''; - my $col = ($lines++ % 2) ? "bgcolor='$color{'color20'}'" : "bgcolor='$color{'color22'}'"; + my $name = escapeHTML( $sources{$list}{'name'} ); + my $category = $Lang::tr{"ipblacklist category $sources{$list}{'category'}"}; + $enable = ''; + my $col = ($lines++ % 2) ? "bgcolor='$color{'color20'}'" : "bgcolor='$color{'color22'}'"; $enable = ' checked' if (exists $settings{$list} and $settings{$list} eq 'on'); @@ -346,7 +300,7 @@ END print <<END; </td> <td>$name</td> - <td align='center'>$safe</td> + <td>$category</td> <td align='center'><input type='checkbox' name="$list" id="$list"$enable></td> </tr>\n END @@ -356,165 +310,18 @@ END print <<END; </table> - <p>$Lang::tr{'ipblacklist safe note'}</p> </div> <table style='width:100%;'> <tr> <td colspan='3' display:inline align='right'><input type='submit' name='ACTION' value='$Lang::tr{'save'}'></td> </tr> </table> - <div class='sources'> - <br> -END - - $enable = $settings{$autoblacklist} eq 'on' ? ' checked' : ''; - - print <<END; - <br><br> - <h2>$Lang::tr{'ipblacklist auto list'}</h2> - <table style='width:100%' border='0'> - <tr> - <td> - $Lang::tr{'ipblacklist autoblacklist enable'} - </td> - <td> - <input type='checkbox' name="$autoblacklist" id="$autoblacklist"$enable> - </td> - <td> - - </td> - </tr> - <tr> - <td> - $Lang::tr{'ipblacklist autoblacklist threshold'} - </td> - <td> - <input type='number' name='BLOCK_THRESHOLD' min='1' max='1000000' maxlength='7' pattern='\\d+' value='$settings{BLOCK_THRESHOLD}'> - </input> - </td> - </tr> - <tr> - <td> - $Lang::tr{'ipblacklist autoblacklist block time'} - </td> - <td> - <input type='number' name='BLOCK_PERIOD' min='1' max='86400' maxlength='7' pattern='\\d+' value='$settings{BLOCK_PERIOD}'> - </input> - </td> - </tr> - </table> - <table style='width:100%;'> - <tr> - <td colspan='3' display:inline align='right'><input type='submit' name='AUTOACTION' value='$Lang::tr{'save'}'></td> - </tr> - </table> - </form> - </div> END Header::closebox(); } -#------------------------------------------------------------------------------ -# sub showstatus() -# -# Displays current blacklist status -#------------------------------------------------------------------------------ - -sub showstatus -{ - Header::openbox('100%', 'center', $Lang::tr{'status'}); - - print <<END; - <table width='100%' cellspacing='1'> - <tr> - <th align='left'>$Lang::tr{'ipblacklist id'}</th> - <th align='right'>$Lang::tr{'ipblacklist entries'}</th> - <th align='right'>$Lang::tr{'ipblacklist pkts in'}</th> - <th align='right'>$Lang::tr{'ipblacklist bytes in'}</th> - <th align='right'>$Lang::tr{'ipblacklist pkts out'}</th> - <th align='right'>$Lang::tr{'ipblacklist bytes out'}</th> - <th align='center'>$Lang::tr{'ipblacklist updated'}</th> - </tr> -END - - # Iterate through the list of sources - - foreach my $list ($autoblacklist, sort keys %sources) - { - next unless ($settings{$list} eq 'on'); - - my $size = ' '; - my $pkts_in = ' '; - my $bytes_in = ' '; - my $pkts_out = ' '; - my $bytes_out = ' '; - my $updated = ' '; - - if (exists $stats{$list}) - { - ($pkts_in, $bytes_in) = @{ $stats{$list}{IPBLACKLISTREDIN} } if (exists $stats{$list}{IPBLACKLISTREDIN}); - ($pkts_out, $bytes_out) = @{ $stats{$list}{IPBLACKLISTREDOUT} } if (exists $stats{$list}{IPBLACKLISTREDOUT}); - $size = $stats{$list}{size} if (exists $stats{$list}{size}); - } - - if (exists $modified{$list} and $modified{$list} > 0) - { - $updated = localtime( $modified{$list} ); - } - - print <<END; - <tr> - <td>$list</td> - <td align='right'>$size</td> - <td align='right'>$pkts_in</td> - <td align='right'>$bytes_in</td> - <td align='right'>$pkts_out</td> - <td align='right'>$bytes_out</td> - <td align='center'>$updated</td> - </tr>\n -END - - } - - print <<END; - </table> -END - - if ($settings{$autoblacklist} eq 'on') - { - print <<END; - <br><br> - <h2>$Lang::tr{'ipblacklist auto list'}</h2> - <table width='60%' cellspacing='1'> - <tr> - <th align='left'>$Lang::tr{'ip address'}</th> - <th align='center'>$Lang::tr{'ipblacklist block time remaining'}</th> - </tr> -END - - foreach my $address (nsort keys %autoblock_addresses ) - { - print "<tr><td>$address</td><td align='center'>$autoblock_addresses{$address}</td></tr>\n"; - } - - print <<END; - </table> - <table width='60%' border='0'> - <form method='post' action='$ENV{'SCRIPT_NAME'}'> - <tr> - <td align='right' width='15%'><input type='submit' name='ACTION' value='$Lang::tr{'unblock all'}'></td> - </tr> - </form> - </table> -END - } - - Header::closebox(); -} - - #------------------------------------------------------------------------------ # sub get_ipset_stats() # @@ -527,8 +334,6 @@ sub get_ipset_stats system( $getipsetstat ); - # Get the number of entries in each IP set - if (-r '/var/tmp/ipsets.txt') { open STATS, '<', '/var/tmp/ipsets.txt' or die "Can't open IP Sets stats file: $!"; @@ -551,66 +356,6 @@ sub get_ipset_stats unlink( '/var/tmp/ipsets.txt' ); } - - # Get the IP addresses in the autoblacklist - - if (-r '/var/tmp/autoblacklist.txt') - { - open HASHTABLE, '<', '/var/tmp/autoblacklist.txt' or die "Can't open autoblacklist address file: $!"; - - # Iterate through the blocked addresses - - foreach my $line (<HASHTABLE>) - { - next unless ($line =~ m/(\d+\.\d+\.\d+\.\d+) timeout (\d+)/); - - $autoblock_addresses{$1} = format_time( $2 ); - } - - close HASHTABLE; - - unlink( '/var/tmp/autoblacklist.txt' ); - } - -} - - -#------------------------------------------------------------------------------ -# sub get_iptables_stats() -# -# Gets information on the number of packets and bytes rejected by each -# blacklist -#------------------------------------------------------------------------------ - -sub get_iptables_stats -{ - system( $getipstat ); - unlink( '/var/tmp/iptablesmangle.txt' ); - unlink( '/var/tmp/iptablesnat.txt' ); - - return unless (-r '/var/tmp/iptables.txt'); - - open STATS, '<', '/var/tmp/iptables.txt' or die "Can't open IP Tables stats file: $!"; - - my $table = 'Unknown'; - - foreach my $line (<STATS>) - { - if ($line =~ m/^Chain (\w+)/) - { - $table = $1; - next; - } - - next unless ($line =~ m/_BLOCK/); - - my ($pkts, $bytes, $chain) = $line =~ m/^\s*(\d+\w?)\s+(\d+\w?)\s+(\w+)_BLOCK/; - $stats{$chain}{$table} = [ $pkts, $bytes ]; - } - - close STATS; - - unlink( '/var/tmp/iptables.txt' ); } @@ -637,7 +382,8 @@ sub is_running #------------------------------------------------------------------------------ # sub show_running # -# Displayed when update is running +# Displayed when update is running. +# Shows a 'working' message plus some information about the IPSets. #------------------------------------------------------------------------------ sub show_running @@ -662,12 +408,7 @@ sub show_running <tr><th>$Lang::tr{'ipblacklist id'}</th><th>$Lang::tr{'ipblacklist entries'}</th></tr> END - foreach my $name (keys %sources) - { - $stats{$name}{'size'} = ' ' if (not exists ($stats{$name}) and - exists $settings{$name} and - $settings{$name} eq 'on'); - } + get_ipset_stats(); foreach my $name (sort keys %stats) { @@ -693,13 +434,10 @@ END sub error { - if ($errormessage) - { - Header::openbox('100%', 'left', $Lang::tr{'error messages'}); - print "<class name='base'>$errormessage\n"; - print " </class>\n"; - Header::closebox(); - } + Header::openbox('100%', 'left', $Lang::tr{'error messages'}); + print "<class name='base'>$errormessage\n"; + print " </class>\n"; + Header::closebox(); } diff --git a/langs/en/cgi-bin/en.pl b/langs/en/cgi-bin/en.pl index b10fbdbf2..ab590cc1b 100644 --- a/langs/en/cgi-bin/en.pl +++ b/langs/en/cgi-bin/en.pl @@ -1,4 +1,4 @@ -%tr = ( +%tr = ( %tr, '24 hours' => '24 Hours', @@ -1538,37 +1538,31 @@ 'ip alias changed' => 'External IP alias changed', 'ip alias removed' => 'External IP alias removed', 'ip info' => 'IP Information', -'ipblacklist auto list' => 'Automatic blacklist', -'ipblacklist autoblacklist enable' => 'Enable automatically updating local blacklist', -'ipblacklist autoblacklist threshold' => 'Threshold (packets/hour)', -'ipblacklist autoblacklist block time' => 'Block time (seconds)', +'ipblacklist' => 'IP Address Blacklists', 'ipblacklist blacklist settings' => 'Blacklist settings', -'ipblacklist bytes in' => 'bytes in', -'ipblacklist bytes out' => 'bytes out', -'ipblacklist check rate' => 'Update check rate (hours)', -'ipblacklist day' => 'day', -'ipblacklist disable pre' => 'Disabling', +'ipblacklist category' => 'Category', +'ipblacklist category application' => 'Application', +'ipblacklist category attacker' => 'Attacker', +'ipblacklist category c and c' => 'Malware C&C', +'ipblacklist category composite' => 'Composite', +'ipblacklist category invalid' => 'Invalid Address', +'ipblacklist category reputation' => 'Reputation', +'ipblacklist category scanner' => 'Scanner', 'ipblacklist disable mid' => 'because it is included in', -'ipblacklist disable port' => '', +'ipblacklist disable post' => '', +'ipblacklist disable pre' => 'Disabling', 'ipblacklist enable' => 'Enable', 'ipblacklist entries' => 'Entries', -'ipblacklist hour' => 'hour', +'ipblacklist hits' => 'Total number of blacklist hits for', 'ipblacklist id' => 'Blacklist', -'ipblacklist invalid block time' => 'Invalid automatic blacklist block time', -'ipblacklist invalid check rate' => 'Invalid update check rate', -'ipblacklist invalid threshold' => 'Invalid automatic blacklist threshold', +'ipblacklist input' => 'Packets Dropped In', +'ipblacklist log list' => 'Firewall log (blacklist)', 'ipblacklist log' => 'Log dropped packets', +'ipblacklist logs' => 'IP Address Blacklist Logs', 'ipblacklist name' => 'Name', -'ipblacklist pkts in' => 'pkts in', -'ipblacklist pkts out' => 'pkts out', -'ipblacklist safe note' => 'Note: safe blacklists block addresses that only generate malicious traffic and therefore will not block any wanted sites.', -'ipblacklist safe' => 'Safe', -'ipblacklist sixhour' => 'six hours', -'ipblacklist updated' => 'Last updated', +'ipblacklist output' => 'Packets Dropped Out', 'ipblacklist use ipblacklists' => 'Enable IP Blacklists', -'ipblacklist week' => 'week', 'ipblacklist working' => 'Updating IP address blacklists...', -'ipblacklist' => 'IP Address Blacklists', 'ipfire has now rebooted' => 'IPFire is rebooting now.', 'ipfire has now shutdown' => 'IPFire is shutting down now.', 'ipfire side' => 'IPFire side:', diff --git a/lfs/configroot b/lfs/configroot index ba87debb1..90b90eb3c 100644 --- a/lfs/configroot +++ b/lfs/configroot @@ -65,7 +65,7 @@ $(TARGET) : captive/settings captive/agb.txt captive/clients captive/voucher_out certs/index.txt certs/index.txt.attr ddns/config ddns/settings ddns/ipcache dhcp/settings \ dhcp/fixleases dhcp/advoptions dhcp/dhcpd.conf.local dns/settings dns/servers dnsforward/config ethernet/aliases ethernet/settings ethernet/known_nics ethernet/scanned_nics \ ethernet/wireless extrahd/scan extrahd/devices extrahd/partitions extrahd/settings firewall/settings firewall/config firewall/geoipblock firewall/input firewall/outgoing \ - fwhosts/customnetworks fwhosts/customhosts fwhosts/customgroups fwhosts/customservicegrp fwhosts/customgeoipgrp fwlogs/ipsettings fwlogs/portsettings ipblacklist/settings\ + fwhosts/customnetworks fwhosts/customhosts fwhosts/customgroups fwhosts/customservicegrp fwhosts/customgeoipgrp fwlogs/ipsettings fwlogs/portsettings ipblacklist/settings \ isdn/settings mac/settings main/hosts main/routing main/security main/settings optionsfw/settings \ ovpn/ccd.conf ovpn/ccdroute ovpn/ccdroute2 pakfire/settings portfw/config ppp/settings-1 ppp/settings-2 ppp/settings-3 ppp/settings-4 \ ppp/settings-5 ppp/settings proxy/settings proxy/squid.conf proxy/advanced/settings proxy/advanced/cre/enable remote/settings qos/settings qos/classes qos/subclasses qos/level7config qos/portconfig \ diff --git a/make.sh b/make.sh index 3d111dbab..25b8b1d59 100755 --- a/make.sh +++ b/make.sh @@ -1665,7 +1665,7 @@ buildpackages() { export LOGFILE echo "... see detailed log in _build.*.log files" >> $LOGFILE - + # Generating list of packages used print_line "Generating packages list from logs" rm -f $BASEDIR/doc/packages-list @@ -1680,7 +1680,7 @@ buildpackages() { rm -f $BASEDIR/doc/packages-list # packages-list.txt is ready to be displayed for wiki page print_status DONE - + # Update changelog cd $BASEDIR [ -z $GIT_TAG ] || LAST_TAG=$GIT_TAG @@ -1755,7 +1755,7 @@ while [ $# -gt 0 ]; do done # See what we're supposed to do -case "$1" in +case "$1" in build) START_TIME=$(now) @@ -1794,7 +1794,7 @@ build) print_build_stage "Building packages" buildpackages - + print_build_stage "Checking Logfiles for new Files" cd $BASEDIR @@ -1859,7 +1859,7 @@ downloadsrc) FINISHED=0 cd $BASEDIR/lfs for c in `seq $MAX_RETRIES`; do - if (( FINISHED==1 )); then + if (( FINISHED==1 )); then break fi FINISHED=1 diff --git a/src/initscripts/system/firewall b/src/initscripts/system/firewall index ebb73062d..08c434440 100644 --- a/src/initscripts/system/firewall +++ b/src/initscripts/system/firewall @@ -180,17 +180,7 @@ iptables_init() { iptables -A FORWARD -j P2PBLOCK iptables -A OUTPUT -j P2PBLOCK - # IP Address Blacklist chains - iptables -N IPBLACKLISTIN - iptables -N IPBLACKLISTOUT - iptables -N IPBLACKLISTREDIN - iptables -N IPBLACKLISTREDOUT - iptables -A INPUT ! -p icmp -j IPBLACKLISTIN - iptables -A FORWARD ! -p icmp -j IPBLACKLISTIN - iptables -A FORWARD ! -p icmp -j IPBLACKLISTOUT - iptables -A OUTPUT ! -p icmp -j IPBLACKLISTOUT - - # Guardian (IPS) chains + # IPS (Guardian) chains iptables -N GUARDIAN iptables -A INPUT -j GUARDIAN iptables -A FORWARD -j GUARDIAN @@ -206,7 +196,15 @@ iptables_init() { iptables -A FORWARD -i tun+ -j OVPNBLOCK iptables -A FORWARD -o tun+ -j OVPNBLOCK - # IPS (Suricata) chains + # IP Address Blacklist chains + iptables -N BLACKLISTIN + iptables -N BLACKLISTOUT + iptables -A INPUT ! -p icmp -j BLACKLISTIN + iptables -A FORWARD ! -p icmp -j BLACKLISTIN + iptables -A FORWARD ! -p icmp -j BLACKLISTOUT + iptables -A OUTPUT ! -p icmp -j BLACKLISTOUT + + # IPS (suricata) chains iptables -N IPS_INPUT iptables -N IPS_FORWARD iptables -N IPS_OUTPUT @@ -420,8 +418,6 @@ iptables_red_up() { iptables -F REDINPUT iptables -F REDFORWARD iptables -t nat -F REDNAT - iptables -F IPBLACKLISTIN - iptables -F IPBLACKLISTOUT # PPPoE / PPTP Device if [ "$IFACE" != "" ]; then @@ -479,10 +475,6 @@ iptables_red_up() { iptables -t nat -A REDNAT -s "${network}" -o "${IFACE}" -j RETURN done - # IP Address Blacklists - iptables -A IPBLACKLISTIN -i $IFACE -j IPBLACKLISTREDIN - iptables -A IPBLACKLISTOUT -o $IFACE -j IPBLACKLISTREDOUT - # Masquerade everything else iptables -t nat -A REDNAT -o $IFACE -j MASQUERADE fi diff --git a/src/misc-progs/getipsetstat.c b/src/misc-progs/getipsetstat.c index aee79542a..781bfc55b 100644 --- a/src/misc-progs/getipsetstat.c +++ b/src/misc-progs/getipsetstat.c @@ -21,8 +21,5 @@ int main(void) safe_system("/usr/sbin/ipset list -t -f /var/tmp/ipsets.txt"); safe_system("chown nobody:nobody /var/tmp/ipsets.txt"); - safe_system("/usr/sbin/ipset list AUTOBLACKLIST -q -f /var/tmp/autoblacklist.txt"); - safe_system("chown -f nobody:nobody /var/tmp/autoblacklist.txt"); - return 0; } diff --git a/src/misc-progs/ipblacklistctrl.c b/src/misc-progs/ipblacklistctrl.c index 506fa2f46..7536b1e97 100644 --- a/src/misc-progs/ipblacklistctrl.c +++ b/src/misc-progs/ipblacklistctrl.c @@ -21,7 +21,7 @@ int main(int argc, char *argv[]) { if (argc < 2) { fprintf(stderr, "\nNo argument given.\n" "ipblacklistctrl (update|restore|log-on|log-off|" - "enable|disable|autoblacklist-update|autoblacklist-clear)\n\n"); + "enable|disable)\n\n"); exit(1); } @@ -37,14 +37,10 @@ int main(int argc, char *argv[]) { safe_system("/usr/local/bin/ipblacklist enable >/dev/null 2>&1 &"); } else if (strcmp(argv[1], "disable") == 0) { safe_system("/usr/local/bin/ipblacklist disable >/dev/null 2>&1 &"); - } else if (strcmp(argv[1], "autoblacklist-update") == 0) { - safe_system("/usr/local/bin/ipblacklist autoblacklist-update >/dev/null 2>&1 &"); - } else if (strcmp(argv[1], "autoblacklist-clear") == 0) { - safe_system("/usr/local/bin/ipblacklist autoblacklist-clear >/dev/null 2>&1 &"); } else { fprintf(stderr, "\nBad argument given.\n" "ipblacklistctrl (update|restore|log-on|log-off|" - "enable|disable|autoblacklist-update|autoblacklist-clear)\n\n"); + "enable|disable)\n\n"); exit(1); } diff --git a/src/scripts/ipblacklist b/src/scripts/ipblacklist index b3f8048d9..6f950214c 100755 --- a/src/scripts/ipblacklist +++ b/src/scripts/ipblacklist @@ -18,25 +18,26 @@ # along with IPFire; if not, write to the Free Software # # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # # # -# Copyright (C) 2018 - 2019 The IPFire team # +# Copyright (C) 2018 - 2020 The IPFire team # # # ############################################################################ # # -# This script use a file containing blacklist details in # +# This script uses a file containing blacklist details in # # /var/ipfire/ipblacklist/sources as well as # # /var/ipfire/ipblacklistsettings containing an enable/disable flag for # # each source. # # # -# Two IPTables chains are used, IPBLACKLISTREDIN and IPBLACKLISTREDOUT, # -# which are inserted into the main INPUT, OUTPUT and FORWARD chains. # +# Two IPTables chains are used: BLACKLISTIN and BLACKLISTOUT are inserted # +# inserted into the main INPUT, OUTPUT and FORWARD chains; they capture # +# packets other than for the ICMP protocol. # # # # For each blacklist that is loaded, a chain is created to optionally log # # and then to drop matching packets. An IPSet is created containing the # # addresses or networks blocked by the blacklist, and then rules are added # -# to the IPBLACKLISTREDIN and IPBLACKLISTREDOUT chains to jump to this # -# chain if appropriate packet list matches in the set. # +# to the BLACKLISTIN and BLACKLISTOUT chains to jump to this chain if a # +# packet list matches the set. # # # -# When checking for updates, the modification time is read for each source # +# When checking for updates, the modification time is used for each source # # and if necessary the list is downloaded. The downloaded list is # # compared to the existing IPSet contents and entries created or deleted # # as necessary. # @@ -44,7 +45,7 @@ ############################################################################ use strict; -use warnings; +#use warnings; use Carp; use Sys::Syslog qw(:standard :macros); @@ -76,23 +77,29 @@ my $lockfile = "/var/run/ipblacklist.pid"; my $proxy_settings = "${General::swroot}/proxy/settings"; my $red_setting = "/var/ipfire/red/iface"; my $detailed_log = "$tmpdir/ipblacklist_log.txt"; -my $autoblacklist = 'AUTOBLACKLIST'; +my $active = "/var/ipfire/red/active"; -my %parsers = ( 'text-with-hash-comments' => \&parse_text_with_hash_comments, - 'text-with-semicolon-comments' => \&parse_text_with_semicolon_comments, - 'dshield' => \&parse_dshield ); +# Other configuration items + +my $margin = 30; # Scheduling allowance for run time etc in seconds +my $count = 30; # Maximum time to wait for another instance (300s) +my $max_dl_fails = 3; # Ignore check rate limit for this number of failures +my $max_size_fraction = 0.7; # Maximum fill fraction of IPSet before enlarging. +my $min_ipset_entries = 1024; # The minimum size of an IPSet. +my $max_dl_bytes = 10_485_760; # Maximum number of bytes to download. +my %parsers = ( 'ip-or-net-list' => \&parse_ip_or_net_list, + 'dshield' => \&parse_dshield ); ############################################################################ # Default settings # Should be overwritten by reading settings files ############################################################################ -my %sources = ( ); +my %sources = ( ); -my %settings = ( 'DEBUG' => 0, - 'LOGGING' => 'on', - 'RATE' => 24, - 'ENABLE' => 'off' ); +my %settings = ( 'DEBUG' => 0, + 'LOGGING' => 'on', + 'ENABLE' => 'off' ); my %proxy_settings = ( 'UPSTREAM_PROXY' => '' ); # No Proxy in use @@ -101,13 +108,9 @@ my %proxy_settings = ( 'UPSTREAM_PROXY' => '' ); # No Proxy in use ############################################################################ sub abort( $ ); -sub autoblacklist_update(); -sub autoblacklist_clear(); -sub create_autoblacklist(); sub create_list( $ ); sub create_ipset( $$$ ); sub debug( $$ ); -sub delete_autoblacklist(); sub delete_list( $ ); sub disable_logging(); sub disable_updates(); @@ -121,15 +124,15 @@ sub download_wget( $$$ ); sub enable_logging(); sub enable_updates(); sub get_ipsets(); +sub get_rate_seconds( $ ); sub iptables( $ ); sub ipset( $ ); sub stop_ipset(); sub is_connected(); sub log_message( $$ ); sub parse_dshield( $ ); -sub parse_text_with_hash_comments( $ ); -sub parse_text_with_semicolon_comments( $ ); -sub read_ipset( $$$ ); +sub parse_ip_or_net_list( $ ); +sub read_ipset( $$$$ ); sub update_list( $$$ ); ############################################################################ @@ -144,25 +147,18 @@ my $ipset_running = 0; # Set to 1 if IPSet process is running my %status; # Status information my %checked; # Time blacklists last changed my %modified; # Time blacklists last modified -my $red_iface; # Name of red interface -my $hours = 3600; # One hour in seconds -my $margin = 600; # Allowance for run time etc -my $count = 30; # Maximum time to wait for another instance (300s) -my @wget_status = ( 'Success', 'Error', 'Parse Error', 'File I/O Error', - 'Network Error', 'SSL Verification Error', - 'Authentication Error', 'Protocol Error', 'Server Error' ); - +my $red_iface; # The name of the red interface ############################################################################ # Synchronise runs ############################################################################ # This script can be triggered either by cron or the WUI. If another -# instance is running, wait for it to finish. +# instance is running, wait for it to finish or timeout. while (-r $lockfile and $count > 0) { - open LOCKFILE, '<', $lockfile or die "Can't open lockfile"; + open LOCKFILE, '<', $lockfile or (abort "Can't open lockfile", last); my $pid = <LOCKFILE>; close LOCKFILE; @@ -176,7 +172,7 @@ while (-r $lockfile and $count > 0) # Create pid file before starting main processing -open LOCKFILE, '>', '/var/run/ipblacklist.pid' or die "Can't open PID file: $!"; +open LOCKFILE, '>', '/var/run/ipblacklist.pid' or abort "Can't open PID file: $!"; print LOCKFILE "$$\n"; close LOCKFILE; @@ -205,16 +201,12 @@ if (-r $sources) eval qx|/bin/cat $sources|; } -# Find out the red interface name - if (-r $red_setting) { - open IN, '<', $red_setting or die "Can't open red interface name file: $!"; - - $red_iface = <IN>; + open REDIF, '<', $red_setting or (abort "Can't open red interface file", exit); + $red_iface = <REDIF>; chomp $red_iface; - - close IN; + close REDIF; } if (@ARGV) @@ -235,7 +227,7 @@ if (@ARGV) { # Called during system startup. # Restore saved blacklists. - # Don't do an update since that takes too long. + # Don't do an update since can take too long. do_start() if ($settings{'ENABLE'} eq 'on'); } @@ -300,25 +292,13 @@ if (@ARGV) disable_updates(); do_delete(); } - elsif ('autoblacklist-update' =~ m/^$cmd/i) - { - # Updates AUTOBLACKLIST options - - autoblacklist_update(); - } - elsif ('autoblacklist-clear' =~ m/^$cmd/i) - { - # Clears AUTOBLACKLIST contents - - autoblacklist_clear(); - } else { - print "Usage: $0 [update|start|stop|restart|log-on|log-off|enable|disable|autoblacklist-update|autoblacklist-clear]\n"; + print "Usage: $0 [update|start|stop|restart|log-on|log-off|enable|disable]\n"; } } } -elsif ($settings{'ENABLE'} eq 'on') +elsif ($settings{'ENABLE'} eq 'on') # Default action if none specified { do_update(); } @@ -353,12 +333,9 @@ sub do_stop() log_message LOG_NOTICE, "Stopping IP Blacklists"; - foreach my $list ( $autoblacklist, sort keys %sources ) + foreach my $list ( sort keys %sources ) { - if (exists $chains{$list}) - { - delete_list( $list ); - } + delete_list( $list ) if (exists $chains{$list}); } } @@ -375,19 +352,17 @@ sub do_start() foreach my $list ( sort keys %sources ) { - if (-e "$savedir/$list.conf") + delete_list( $list ) if (exists $chains{$list}); # Make sure OK to start + + if ((-e "$savedir/$list.conf") and ($red_iface)) { log_message LOG_INFO, "Restoring blacklist $list"; - system( "$ipset restore -f $savedir/$list.conf" ); + system( "$ipset restore -f $savedir/$list.conf" ); # Can't use the ipset + # function to do this create_list( $list ); } } - - if ($settings{$autoblacklist} eq 'on') - { - create_autoblacklist(); - } } @@ -418,10 +393,8 @@ sub do_delete() } } - if ($settings{$autoblacklist} eq 'on') - { - delete_autoblacklist(); - } + %modified = (); + $update_status = 1; } @@ -430,14 +403,14 @@ sub do_delete() # # Updates all the blacklists. # Creates or deletes the blacklist firewall rules as necessary and checks for -# updates to the blacklists. +# updates to the blacklists. Each blacklist has its own minimum elapsed time +# between updates, which is specified in the sources file, so the time of each +# check is stored. #------------------------------------------------------------------------------ sub do_update() { - return unless (is_connected()); - - my $type = 'hash:ip'; + return unless ($red_iface); # Get the list of current ipsets @@ -447,38 +420,39 @@ sub do_update() debug 1, "Checking blacklist sources"; + LIST: foreach my $list ( sort keys %sources ) { my @new_blacklist = (); my $name = $sources{$list}{'name'}; - my $rate = $sources{$list}{'rate'}; my $last_checked = $checked{$list} || 0; + my $failures = $checked{"${list}_failures"} || 0; my $enabled = 0; - if (exists $modified{$list}) - { - # Limit the check rate to the minimum defined in the WUI, unless we're - # creating the list - - $rate = $settings{'RATE'} if ($settings{'RATE'} > $rate); - } - if (exists $settings{$list}) { $enabled = $settings{$list} eq 'on'; } - debug 1, "Checking blacklist source: $name"; - - if ($enabled) + if ($enabled and is_connected()) { + debug 1, "Checking blacklist source: $name"; + + # Calculate the per list rate + + my $rate = get_rate_seconds( $sources{$list}{'rate'} ); + # Has enough time passed since the last time we checked the list? + # Ignore the limit if the last download failed - if (($last_checked + $rate * $hours) < (time() + $margin)) + if (($last_checked + $rate) < (time() + $margin) or + ($failures > 0 and $failures < $max_dl_fails)) { + my $type = 'hash:ip'; + download_list( $list, \@new_blacklist, \$type ); - next unless (@new_blacklist); + next LIST unless (@new_blacklist); if (not exists $chains{$list}) { @@ -504,76 +478,81 @@ sub do_update() unlink "$savedir/$list.conf" if (-e "$savedir/$list.conf"); - delete $modified{$list} if (exists $modified{$list}); + delete $modified{$list} if (exists $modified{$list}); + delete $checked{"${list}_failures"} if (exists $checked{"${list}_failures"}); $update_status = 1; } } - # Check for any deleted lists + # Check for any lists that don't exist any more - foreach my $list (keys %sources) + foreach my $list (keys %modified) { - if (not exists $sources{$list}) - { - delete_list( $list ); + next if (exists $sources{$list}); - # Delete the save file + delete_list( $list ); - unlink "$savedir/$list.conf" if (-e "$savedir/$list.conf"); + # Delete the save file - # Delete from the status + unlink "$savedir/$list.conf" if (-e "$savedir/$list.conf"); - delete $modified{$list} if (exists $modified{$list}); - delete $checked{$list} if (exists $checked{$list}); - $update_status = 1; - } - } + # Delete from the status - if ($settings{$autoblacklist} eq 'on') - { - create_autoblacklist() if (not exists $chains{$autoblacklist}); + delete $modified{$list} if (exists $modified{$list}); + $update_status = 1; } - else + + foreach my $list (keys %checked) { - delete_autoblacklist() if (exists $chains{$autoblacklist}); - } + next if ($list =~ m/_failures/); + next if (exists $sources{$list}); - log_message LOG_INFO, "Completed IP Blacklist update"; + delete $checked{$list}; + delete $checked{"${list}_failures"}; + delete $settings{$list} if (exists $settings{$list}); + $update_status = 1; + } } #------------------------------------------------------------------------------ -# sub autoblacklist_update() +# sub get_rate_seconds( text ) +# +# Converts a check rate into seconds. A sanity check is made on the coverted +# value. # -# Updates the settings for the AUTOBLACKLIST +# Parameters: +# text The value to convert in the form nnnu, where nnn is a number and u +# is either m (minutes), h (hours) or d (days). Hours is assumed if +# not specified and everything after the first letter is ignored. #------------------------------------------------------------------------------ -sub autoblacklist_update() +sub get_rate_seconds( $ ) { - # Get the list of current ipsets - - get_ipsets(); + my ($text) = @_; - # Delete the existing AUTOBLACKLIST, if it currently exists. + my ($value, $unit) = (uc $text) =~ m/(\d+)([DHM]?)/; - delete_autoblacklist() if (exists $chains{$autoblacklist}); - - # Re-create the AUTOBLACKLIST with the correct parameters. - - create_autoblacklist() if ($settings{$autoblacklist} eq 'on'); -} + if ($unit eq 'D') # Days + { + $value *= 60 * 60 * 24; + } + elsif ($unit eq 'M') # Minutes + { + $value *= 60; + } + else # Everything else - assume hours + { + $value *= 60 * 60; + } + # Sanity check - limit to range 5 min .. 1 week -#------------------------------------------------------------------------------ -# sub autoblacklist_clear() -# -# Clears the contents of the AUTOBLACKLIST -#------------------------------------------------------------------------------ + # d h m s + $value = 5 * 60 if ($value < 5 * 60); + $value = 7 * 24 * 60 * 60 if ($value > 7 * 24 * 60 * 60); -sub autoblacklist_clear() -{ - log_message LOG_INFO, "Flush Automatic blacklist"; - ipset( "flush $autoblacklist" ); + return $value; } @@ -587,7 +566,7 @@ sub autoblacklist_clear() sub is_connected() { - return (-e "${General::swroot}/red/active"); + return (-e $active); } @@ -595,7 +574,16 @@ sub is_connected() # sub create_list( list ) # # Creates a new IPTables chain for a blacklist source. -# The set must be created before calling this function. +# The set must be created before calling this function. Two rules are added to +# the chain: +# (optional) 1 Log the packet +# 2 Drop the packet +# +# The log rule is only added when logging is enabled by the WUI. +# +# Rules are then added to the BLACKLISTIN and BLACKLISTOUT chains that check +# the packet's IP address against the IPSet and then jump to the newly created +# chain. # # Parameters: # list The name of the blacklist @@ -609,56 +597,24 @@ sub create_list( $ ) # Create new chain in filter table - iptables( " -N ${list}_BLOCK" ) == 0 or - ( abort "Could not create IPTables chain ${list}_BLOCK", return ); + iptables( "-N ${list}_DROP" ) == 0 or + ( abort "Could not create IPTables chain ${list}_DROP", return ); # Add the logging and drop rules if ($settings{'LOGGING'} eq 'on') { - iptables( "-A ${list}_BLOCK -j LOG -m limit --limit 10/second --log-prefix 'DROP_$list'" ) == 0 or + iptables( "-A ${list}_DROP -j LOG -m limit --limit 10/second --log-prefix 'BLKLST_$list'" ) == 0 or ( abort "Could not create IPTables chain $list LOG rule", return ); } - iptables( "-A ${list}_BLOCK -j DROP" ) == 0 or + iptables( "-A ${list}_DROP -j DROP" ) == 0 or ( abort "Could not create IPTables chain $list drop rule", return ); # Add the rules to check against the set - iptables( "-A IPBLACKLISTREDIN -p ALL -m set --match-set $list src -j ${list}_BLOCK" ); - iptables( "-A IPBLACKLISTREDOUT -p ALL -m set --match-set $list dst -j ${list}_BLOCK" ); -} - - -#------------------------------------------------------------------------------ -# sub create_autoblacklist() -# -# Creates a new IPTables chain for the AUTOBLACKLIST. This also creates the -# IPSet with the correct timeout. -#------------------------------------------------------------------------------ - -sub create_autoblacklist() -{ - return unless ($red_iface); # Can't add rule to policy unless this is set - - # Create the set for the AUTOBLACKLIST - - ipset( "create $autoblacklist hash:ip timeout $settings{BLOCK_PERIOD}" ); - - # Create new chain in filter table - - create_list( $autoblacklist ); - - # For the AUTOBLACKLIST there are extra rules to reset the timeout on the - # blockled addresses - - iptables( "-I ${autoblacklist}_BLOCK -m set --match-set $autoblacklist src -j SET --add-set $autoblacklist src --exist" ); - iptables( "-I ${autoblacklist}_BLOCK -m set --match-set $autoblacklist dst -j SET --add-set $autoblacklist dst --exist" ); - - # For the AUTOBLACKLIST there is an extra rule to add an entry to the list - # of blocked addresses. This is added to the input policy chain. - - iptables( "-I POLICYIN 1 -i $red_iface -m hashlimit --hashlimit-mode srcip --hashlimit-above $settings{BLOCK_THRESHOLD}/hour --hashlimit-name $autoblacklist -j SET --add-set $autoblacklist src" ); + iptables( "-A BLACKLISTIN -p ALL -i $red_iface -m set --match-set $list src -j ${list}_DROP" ); + iptables( "-A BLACKLISTOUT -p ALL -o $red_iface -m set --match-set $list dst -j ${list}_DROP" ); } @@ -680,60 +636,25 @@ sub delete_list( $ ) # Remove the blacklist chains from the main INPUT and OUTPUT chains - iptables( "-D IPBLACKLISTREDIN -p ALL -m set --match-set $list src -j ${list}_BLOCK" ) == 0 or - log_message LOG_ERR, "Could not remove IPSet $list from IPBLACKLISTREDIN chain"; - - iptables( "-D IPBLACKLISTREDOUT -p ALL -m set --match-set $list dst -j ${list}_BLOCK" ) == 0 or - log_message LOG_ERR, "Could not remove IPSet $list from IPBLACKLISTREDOUT chain"; + iptables( "-D BLACKLISTIN -p ALL -i $red_iface -m set --match-set $list src -j ${list}_DROP" ); + iptables( "-D BLACKLISTOUT -p ALL -o $red_iface -m set --match-set $list dst -j ${list}_DROP" ); # Flush and delete the chain - iptables( "-F ${list}_BLOCK" ) == 0 or - log_message LOG_ERR, "Could not flush IPTables chain ${list}_BLOCK"; - - iptables( "-X ${list}_BLOCK" ) == 0 or - log_message LOG_ERR, "Could not delete IPTables chain ${list}_BLOCK"; + iptables( "-F ${list}_DROP" ); + iptables( "-X ${list}_DROP" ); # Flush and delete the set ipset( "flush $list" ); - ipset( "destroy $list" ); } #------------------------------------------------------------------------------ -# sub delete_autoblacklist() +# sub download_list( list, ref_list, ref_type ) # -# Deletes the autoblacklist IPTables chain when it is disabled. Also flushes -# and destroys the IPSet. -#------------------------------------------------------------------------------ - -sub delete_autoblacklist() -{ - # For the AUTOBLACKLIST there is an extra rule to remove - - unless ($red_iface) - { - iptables( "-D POLICYIN -i $red_iface -m hashlimit --hashlimit-mode srcip --hashlimit-above $settings{BLOCK_THRESHOLD}/hour --hashlimit-name $autoblacklist -j SET --add-set $autoblacklist src" ); - } - - # Now do a normal delete - - delete_list( $autoblacklist ); -} - - -#------------------------------------------------------------------------------ -# sub download_list( chain, ref_list, ref_type ) -# -# Updates the IP Addresses for a blacklist. Depending on the blacklist one of -# two methods are used: -# -# - For some lists the header is downloaded and the modification date checked. -# If newer than the existing list, the update is downloaded. -# - For other lists this is not supported,so the whole file has to be -# downloaded regardless. +# Downloads the IP Addresses for a blacklist. # # Once downloaded the list is parsed to get the IP addresses and/or networks. # @@ -747,8 +668,8 @@ sub download_list( $$$ ) { my ($list, $new_blacklist, $type) = @_; - $checked{$list} = time(); - $update_status = 1; + $checked{$list} = time(); # Record that the list has been checked + $update_status = 1; # Check the parser for the blacklist @@ -758,23 +679,18 @@ sub download_list( $$$ ) return; } - if ($sources{$list}{'method'} eq 'check-header-time') - { - download_check_header_time( $list, $new_blacklist, $type ); - } - else - { - download_wget( $list, $new_blacklist, $type ); - } + # Add alternative download mechanisms here + + download_check_header_time( $list, $new_blacklist, $type ); } #------------------------------------------------------------------------------ -# sub download_check_header_time( chain, ref_list, ref_type ) +# sub download_check_header_time( list, ref_list, ref_type ) # -# Updates the IP Addresses for a blacklist. The header is downloaded and the -# modification date checked. If newer than the existing list, the update is -# downloaded. +# Updates the IP Addresses for a blacklist. The If-Modified-Since header is +# specified in the request so that only updated lists are downloaded (providing +# that the server supports this functionality). # # Once downloaded the list is parsed to get the IP addresses and/or networks. # @@ -782,6 +698,9 @@ sub download_list( $$$ ) # list The name of the blacklist # ref_list A reference to an array to store the downloaded blacklist # ref_type A reference to store the type of the blacklist +# +# Returns: +# The list type: 'hash:ip' or 'hash:net' #------------------------------------------------------------------------------ sub download_check_header_time( $$$ ) @@ -794,12 +713,12 @@ sub download_check_header_time( $$$ ) my $parser = $parsers{ $sources{$list}{'parser'} }; - log_message LOG_INFO, "Checking modification time for blacklist $list update with LWP"; + debug 1, "Checking for blacklist $list updates with LWP"; # Create a user agent for downloading the blacklist - # Limit the download size for safety (10 MiB) + # Limit the download size for safety - my $ua = LWP::UserAgent->new( max_size => 10485760 ); + my $ua = LWP::UserAgent->new( max_size => $max_dl_bytes ); # Get the Proxy settings @@ -807,55 +726,42 @@ sub download_check_header_time( $$$ ) { if ($proxy_settings{'UPSTREAM_USER'}) { - $ua->proxy("http" => "http://$proxy_settings{'UPSTREAM_USER'}:$proxy_settings{'UPSTREAM_PASSWORD'}\@$proxy_settings{'UPSTREAM_PROXY'}/"); - $ua->proxy("https" => "http://$proxy_settings{'UPSTREAM_USER'}:$proxy_settings{'UPSTREAM_PASSWORD'}\@$proxy_settings{'UPSTREAM_PROXY'}/"); + $ua->proxy( [["http", "https"] => "http://$proxy_settings{'UPSTREAM_USER'}:$proxy_settings{'UPSTREAM_PASSWORD'}\@$proxy_settings{'UPSTREAM_PROXY'}/"] ); } else { - $ua->proxy("http" => "http://$proxy_settings{'UPSTREAM_PROXY'}/"); - $ua->proxy("https" => "http://$proxy_settings{'UPSTREAM_PROXY'}/"); + $ua->proxy( [["http", "https"] => "http://$proxy_settings{'UPSTREAM_PROXY'}/"] ); } } - # Get the blacklist modification time from the internet - - my $request = HTTP::Request->new( HEAD => $sources{$list}{'url'} ); - - my $response = $ua->request( $request ); + # Get the last modified time - if (not $response->is_success) - { - log_message LOG_WARNING, "Failed to download $list header $sources{$list}{'url'}: ". $response->status_line; - - return; - } - - # Has the blacklist been modified since we last read it? - - if (exists $modified{$list} and $modified{$list} >= $response->last_modified) - { - # We've already got this version of the blacklist - - debug 1, "Blacklist $list not modified"; - return; - } - - debug 1, "Blacklist $list Modification times: old " . $modified{$list} . ", new " . $response->last_modified if (exists $modified{$list}); - log_message LOG_INFO, "Downloading blacklist $list with LWP"; + my $modified = gmtime( $modified{$list} || 0 ); # Download the blacklist - $request = HTTP::Request->new( GET => $sources{$list}{'url'} ); - $response = $ua->request($request); + my $response = $ua->get( $sources{$list}{'url'}, 'If-Modified-Since' => $modified ); if (not $response->is_success) { + if ($response->code == 304) + { + # Not an error - list has not been modified + debug 1, "Blacklist $list not modified"; + + return; + } + log_message LOG_WARNING, "Failed to download $list blacklist $sources{$list}{'url'}: ". $response->status_line; + $checked{"${list}_failures"}++; return; } - $modified{$list} = $response->last_modified; + $modified{$list} = $response->last_modified; + $checked{"${list}_failures"} = 0; + + # Parse the downloaded list, checking if it's a list of addresses or nets foreach my $line (split /[\r\n]+/, $response->content) { @@ -865,108 +771,12 @@ sub download_check_header_time( $$$ ) next unless ($address and $address =~ m/\d+\.\d+\.\d+\.\d+/); - if ($address =~ m|/\d+|) - { - $found_net = 1; - } - else + if ($address =~ m|/32|) { + $address =~ s|/32||; $found_ip = 1; } - - push @{ $new_blacklist }, $address; - } - - if ($found_net and $found_ip) - { - # Convert mixed address and network set to all network - - foreach my $address (@{ $new_blacklist }) - { - $address .= '/32' unless ($address =~ m|/\d+|); - } - - $found_ip = 0; - } - - $$type = $found_net ? 'hash:net' : 'hash:ip'; -} - - -#------------------------------------------------------------------------------ -# sub download_wget( chain, ref_list, ref_type ) -# -# Updates the IP Addresses for a blacklist. The whole file is download with -# wget and then the modification time compared with the stored modification -# time. If the update is newer then the downloaded list is parsed. -# -# Once downloaded the list is parsed to get the IP addresses and/or networks. -# -# Parameters: -# list The name of the blacklist -# ref_list A reference to an array to store the downloaded blacklist -# ref_type A reference to store the type of the blacklist -#------------------------------------------------------------------------------ - -sub download_wget( $$$ ) -{ - my ($list, $new_blacklist, $type) = @_; - my $wget_proxy = ''; - my $found_ip = 0; - my $found_net = 0; - - my $parser = $parsers{ $sources{$list}{'parser'} }; - - log_message LOG_INFO, "Downloading blacklist $list update with wget"; - - # Get the Proxy settings - - if ($proxy_settings{'UPSTREAM_PROXY'}) - { - if ($proxy_settings{'UPSTREAM_USER'}) - { - $wget_proxy = "--proxy=on --proxy-user=$proxy_settings{'UPSTREAM_USER'} --proxy-passwd=$proxy_settings{'UPSTREAM_PASSWORD'} -e http_proxy=http://$proxy_settings{'UPSTREAM_PROXY'}/"; - } - else - { - $wget_proxy = "--proxy=on -e http_proxy=http://$proxy_settings{'UPSTREAM_PROXY'}/"; - } - } - - my $retv = system( "wget $wget_proxy --no-show-progress -o $detailed_log -O $tmpdir/ipblacklist_$list $sources{$list}{'url'}" ); - - if ($retv != 0) - { - my $error = $wget_status[ $retv/256 ]; - log_message LOG_WARNING, "Failed to download $list blacklist $sources{$list}{'url'}: $error"; - return; - } - - my @file_info = stat( "$tmpdir/ipblacklist_$list" ); - - if (exists $modified{$list} and $modified{$list} >= $file_info[9]) - { - # We've already got this version of the blocklist - - debug 1, "Blacklist $list not modified"; - unlink "$tmpdir/ipblacklist_$list"; - return; - } - - open LIST, '<', "$tmpdir/ipblacklist_$list" or (abort "Can't open downloaded blacklist for $list: $!", return); - - $modified{$list} = $file_info[9]; - - foreach my $line (<LIST>) - { - chomp $line; - - my $address = &$parser( $line ); - - next unless ($address); - next unless ($address =~ m|\d+\.\d+\.\d+\.\d+|); - - if ($address =~ m|/\d+|) + elsif ($address =~ m|/\d+|) { $found_net = 1; } @@ -978,13 +788,9 @@ sub download_wget( $$$ ) push @{ $new_blacklist }, $address; } - close LIST; - - unlink "$tmpdir/ipblacklist_$list"; - if ($found_net and $found_ip) { - # Convert mixed address and network set to all network + # Convert mixed addresses and networks to all networks foreach my $address (@{ $new_blacklist }) { @@ -999,38 +805,49 @@ sub download_wget( $$$ ) #------------------------------------------------------------------------------ -# sub read_ipset( list, old, type ) +# sub read_ipset( list, old, type, maxelem ) # -# Reads the existing contents of the set +# Reads the existing contents and type of the set. # # Parameters: -# chain The name of the blacklist -# old Reference to array to contain blacklist -# type Reference to type +# list The name of the blacklist +# old Reference to array to contain blacklist +# type Reference to type +# maxelem Reference to maximum number of elements #------------------------------------------------------------------------------ -sub read_ipset( $$$ ) +sub read_ipset( $$$$ ) { - my ($list, $old, $type) = @_; - my $found_net = 0; - my $found_ip = 0; + my ($list, $old, $type, $maxelem) = @_; + my $found_net = 0; + my $found_ip = 0; debug 2, "Reading existing ipset for blacklist $list"; foreach my $line (qx/$ipset list $list/) { + if ($line =~ m|Header:.*maxelem (\d+)|) + { + $$maxelem = $1; + next; + } + next unless ($line =~ m|(\d+\.\d+\.\d+\.\d+(?:/\d+)?)|); my $address = $1; - if (($address =~ m|/\d+$|) and ($address !~ m|/32$|)) + if ($address =~ m|/32|) + { + $found_ip = 1; + $address =~ s|/32$||; + } + elsif ($address =~ m|/\d+$|) { $found_net = 1; } else { $found_ip = 1; - $address =~ s|/32$||; } $$old{$address} = 1; @@ -1038,7 +855,7 @@ sub read_ipset( $$$ ) if ($found_ip and $found_net) { - # Convert mixed address and network set to all network + # Convert mixed addresses and networks to all networks my @ads_list = keys %{ $old }; @@ -1059,12 +876,13 @@ sub read_ipset( $$$ ) #------------------------------------------------------------------------------ -# sub update_list( chain, new, new_type ) +# sub update_list( list, new, new_type ) # # Updates the IP Addresses for a blacklist # # The new list is compared to the existing list and new entries added or old -# entries deleted as necessary. +# entries deleted as necessary. If the list type ('hash:ip' or 'hash:net') has +# changed then the IPSet is deleted and re-created with the new type. # # Parameters: # list The name of the blacklist @@ -1078,49 +896,63 @@ sub update_list( $$$ ) my %old; my $old_type; my $changes = 0; + my $maxelem = 0; - debug 2, "Checking for $list blacklist update from $sources{$list}{'url'}"; + debug 1, "Checking for $list blacklist update from $sources{$list}{'url'}"; - log_message LOG_INFO, "Updating $list blacklist"; + if (exists $chains{$list} ) + { + my $recreate_ipset = 0; - read_ipset( $list, \%old, \$old_type ); + read_ipset( $list, \%old, \$old_type, \$maxelem ); - # Check the IPSet type hasn't changed + # Check the IPSet type hasn't changed - if ($new_type ne $old_type) - { - # Change the IPSet type. This requires removing references to it first. - # We could delete and then create the chain, but doing it like this keeps - # the statistics. + if ($new_type ne $old_type) + { + log_message LOG_NOTICE, "Blacklist $list changed type from $old_type to $new_type"; + $recreate_ipset = 1; + } - log_message LOG_NOTICE, "Blacklist $list changed type from $old_type to $new_type"; + if ($max_size_fraction * $maxelem < scalar @{ $new } ) + { + log_message LOG_NOTICE, "Blacklist $list changed size from $maxelem"; + $recreate_ipset = 1; + } + + if ($recreate_ipset) + { + # Change the IPSet type and/or size. This requires removing references + # to it first. We could delete and then create the chain, but doing it + # like this keeps the statistics. - # Remove the IPSet from the IPTables chains + # Remove the IPSet from the IPTables chains - iptables( "-D 'IPBLACKLISTREDIN' -p ALL -m set --match-set $list src -j ${list}_BLOCK" ) == 0 or - log_message LOG_ERR, "Could not remove ${list} from IPBLACKLISTREDIN chain"; + iptables( "-D 'BLACKLISTIN' -p ALL -i $red_iface -m set --match-set $list src -j ${list}_DROP" ) == 0 or + log_message LOG_ERR, "Could not remove ${list} from BLACKLISTIN chain"; - iptables( "-D 'IPBLACKLISTREDOUT' -p ALL -m set --match-set $list dst -j ${list}_BLOCK" ) == 0 or - log_message LOG_ERR, "Could not remove ${list} from IPBLACKLISTREDOUT chain"; + iptables( "-D 'BLACKLISTOUT' -p ALL -o $red_iface -m set --match-set $list dst -j ${list}_DROP" ) == 0 or + log_message LOG_ERR, "Could not remove ${list} from BLACKLISTOUT chain"; - # Flush and delete the old set + # Flush and delete the old set - ipset( "flush $list" ); - ipset( "destroy $list" ); + ipset( "flush $list" ); + ipset( "destroy $list" ); - %old = (); + %old = (); # Since we've deleted the old set it can't have any entries. - # Create the new ipset + # Create the new ipset - create_ipset( $list, $new_type, scalar @{ $new } ); + create_ipset( $list, $new_type, scalar @{ $new } ); - # Add the rules to check against the set + # Add the rules to check against the set - iptables( "-A 'IPBLACKLISTREDIN' -p ALL -m set --match-set $list src -j ${list}_BLOCK" ) == 0 or - log_message LOG_ERR, "Could not add IPSet $list to IPBLACKLISTREDIN chain"; + iptables( "-A 'BLACKLISTIN' -p ALL -i $red_iface -m set --match-set $list src -j ${list}_DROP" ) == 0 or + log_message LOG_ERR, "Could not add IPSet $list to BLACKLISTIN chain"; - iptables( "-A 'IPBLACKLISTREDOUT' -p ALL -m set --match-set $list dst -j ${list}_BLOCK" ) == 0 or - log_message LOG_ERR, "Could not add IPSet $list to IPBLACKLISTREDOUT chain"; + iptables( "-A 'BLACKLISTOUT' -p ALL -o $red_iface -m set --match-set $list dst -j ${list}_DROP" ) == 0 or + log_message LOG_ERR, "Could not add IPSet $list to BLACKLISTOUT chain"; + } } # Process the blacklist @@ -1131,11 +963,11 @@ sub update_list( $$$ ) if (exists $old{$address}) { - delete $old{$address}; # Not new + delete $old{$address}; # Not new - don't delete from chain later } else { - ipset( "add $list $address -exist" ); + ipset( "add $list $address -exist" ); # New - add it $changes++; } @@ -1156,14 +988,13 @@ sub update_list( $$$ ) debug 3, "Delete old net $address from blacklist $list"; } - - log_message LOG_INFO, "Finished updating $list blacklist with $changes changes"; + log_message LOG_INFO, "Updated $list blacklist with $changes changes"; # Save the blacklist for the next reboot mkdir "$savedir" unless (-d "$savedir" ); - ipset( "save $list -file $savedir/$list.conf" ); + ipset( "save $list -file $savedir/$list.conf" ) if ($changes > 0); stop_ipset(); } @@ -1195,28 +1026,29 @@ sub get_ipsets( ) #------------------------------------------------------------------------------ -# sub create_ipset( name, type, size ) +# sub create_ipset( list, type, size ) # -# Creates a new IPSet. The current and maximum size of the set are determined -# by taking the next power of two greater than the numer of entries, subject to -# a minimum size. This allows for future expansion. +# Creates a new IPSet. The current size of the set is determined by taking the +# next power of two greater than the number of entries; the maximum size is set +# to double this, subject to a minimum size. This allows for future expansion. # # Parameters: -# name The name of the blacklist +# list The name of the blacklist # type The type of the blacklist (hash:ip or hash:net) # size The number of entries in the lsit #------------------------------------------------------------------------------ sub create_ipset( $$$ ) { - my ($name, $type, $size) = @_; + my ($list, $type, $size) = @_; my $hashsize = 1; $hashsize <<= 1 while ($hashsize < $size); - my $maxsize = ($hashsize < 16384) ? 32768 : $hashsize * 2; + my $maxsize = $hashsize * 2; + $maxsize = $min_ipset_entries if ($maxsize < $min_ipset_entries); # Create the new ipset - ipset( "create $name $type hashsize $hashsize maxelem $maxsize" ); + ipset( "create $list $type hashsize $hashsize maxelem $maxsize" ); stop_ipset(); # Need to do this to action the IPSet commands } @@ -1225,6 +1057,7 @@ sub create_ipset( $$$ ) # sub enable_logging() # # Enable logging of packets dropped by IP Blacklist rules. +# This adds a rule to log the packet to each lists' IPTables chain. #------------------------------------------------------------------------------ sub enable_logging() @@ -1237,7 +1070,7 @@ sub enable_logging() { if (exists $chains{$list}) { - iptables( "-I ${list}_BLOCK 1 -j LOG -m limit --limit 10/second --log-prefix 'DROP_$list'" ); + iptables( "-I ${list}_DROP 1 -j LOG -m limit --limit 10/second --log-prefix 'BLKLST_$list'" ); } } } @@ -1247,6 +1080,7 @@ sub enable_logging() # sub disable_logging() # # Disable logging of packets dropped by IP Blacklist rules. +# This deletes a rule to log the packet from each lists' IPTables chain. #------------------------------------------------------------------------------ sub disable_logging() @@ -1259,7 +1093,7 @@ sub disable_logging() { if (exists $chains{$list}) { - iptables( "-D ${list}_BLOCK -j LOG -m limit --limit 10/second --log-prefix 'DROP_$list'" ); + iptables( "-D ${list}_DROP -j LOG -m limit --limit 10/second --log-prefix 'BLKLST_$list'" ); } } } @@ -1269,9 +1103,14 @@ sub disable_logging() # sub enable_updates() # # Adds a command to the fcrontab to run the update hourly. +# If there is a command already in the fcrontab to do this it will be +# uncommented, otherwise a new line is added. +# # The update is executed at an offset from the hour so that all the users don't # try to download the updates at exactly the same time - the blacklists are -# provided free, so it's good manners to spread the load on the servers. +# provided free, so it's good manners to spread the load on the servers. The +# offset is initialised to a random number that avoids running on the hour +# (when a lot of other things happen), and every fifteen minutes thereafter. #------------------------------------------------------------------------------ sub enable_updates() @@ -1285,9 +1124,9 @@ sub enable_updates() { if ($line =~ m|/usr/local/bin/ipblacklist|) { - return if ($line !~ m/^#/); # Already enabled + return if ($line !~ m/^#/); # Already enabled - do nothing - # Found - uncomment the line + # Already in fcrontab - uncomment the line $line =~ s/^#+//; $found = 1; @@ -1298,13 +1137,20 @@ sub enable_updates() if (not $found) { - # Add a new entry + # Add a new entry to fcrontab - my $start = int( rand(50) ) + 5; + my $start = int( rand(13) ) + 1; + + my $times = $start; + + for (my $offset = $times+15 ; $offset < 60 ; $offset += 15) + { + $times .= ",$offset"; + } push @lines, "\n"; push @lines, "# IP Blacklist update\n"; - push @lines, "\%hourly,nice(1),random,serial $start /usr/local/bin/ipblacklist\n"; + push @lines, "\%nice(1) $times * * * * /usr/local/bin/ipblacklist\n"; log_message LOG_INFO, "Add IP Address Blacklist update to crontab"; } @@ -1329,9 +1175,9 @@ sub disable_updates() { if ($line =~ m|/usr/local/bin/ipblacklist|) { - return if ($line =~ m/^#/); # Already disabled + return if ($line =~ m/^#/); # Already disabled - do nothing - # Found - comment the line + # In fcrontab - comment the line $line =~ s/^#*/#/; $found = 1; @@ -1349,9 +1195,10 @@ sub disable_updates() #------------------------------------------------------------------------------ -# sub parse_text_with_hash_comments( line ) +# sub parse_ip_or_net_list( line ) # -# Parses an input line removing comments. +# Parses an input line, looking for lines starting with an IP Address or +# Network specification. # # Parameters: # line The line to parse @@ -1360,41 +1207,11 @@ sub disable_updates() # Either an IP Address or a null string #------------------------------------------------------------------------------ -sub parse_text_with_hash_comments( $ ) +sub parse_ip_or_net_list( $ ) { my ($line) = @_; - return "" if ($line =~ m/^\s*#/); - - $line =~ s/#.*$//; - - $line =~ m|(\d+\.\d+\.\d+\.\d+(?:/\d+)?)|; - - return $1; -} - - -#------------------------------------------------------------------------------ -# sub parse_text_with_semicolon_comments( line ) -# -# Parses an input line removing comments. -# -# Parameters: -# line The line to parse -# -# Returns: -# Either and IP Address or a null string -#------------------------------------------------------------------------------ - -sub parse_text_with_semicolon_comments( $ ) -{ - my ($line) = @_; - - return "" if ($line =~ m/^\s*;/); - - $line =~ s/;.*$//; - - $line =~ m|(\d+\.\d+\.\d+\.\d+(?:/\d+)?)|; + $line =~ m|^(\d+\.\d+\.\d+\.\d+(?:/\d+)?)|; return $1; } @@ -1424,6 +1241,7 @@ sub parse_dshield( $ ) $line =~ s/#.*$//; + # |Start addrs | |End Addrs | |Mask $line =~ m|(\d+\.\d+\.\d+\.\d+(?:/\d+)?)\s+\d+\.\d+\.\d+\.\d+(?:/\d+)?\s+(\d+)|; return unless ($1); @@ -1436,8 +1254,8 @@ sub parse_dshield( $ ) #------------------------------------------------------------------------------ # sub iptables( cmd ) # -# Executes an IPTables command, waiting for the lock to ensure only one change -# is made at a time. +# Executes an IPTables command, waiting for the internal lock to ensure only +# one change is made at a time. # # Parameters: # cmd The command to execute @@ -1462,6 +1280,11 @@ sub iptables( $ ) # new process for each command. The sub-process is started if it's not already # running. # +# Note that the pipe is buffered so commands are not necessarily executed +# immediately. Use ipset_stop() to force commands to be executed. This should +# be done before relying on anything that the ipset commands do, for example +# before referencing the IPSet in an IPTables command. +# # Parameters: # cmd The command to execute #------------------------------------------------------------------------------ @@ -1485,6 +1308,7 @@ sub ipset( $ ) # sub stop_ipset( ) # # Stops the ipset sub-process. +# This causes any pending ipset commands to be executed. #------------------------------------------------------------------------------ sub stop_ipset( ) @@ -1500,7 +1324,7 @@ sub stop_ipset( ) #------------------------------------------------------------------------------ # sub abort( message, parameters... ) # -# Aborts the current activity, printing out an error message. +# Used when aborting the current activity, printing out an error message. # # Parameters: # message Message to be printed