From patchwork Sat Sep 25 13:31:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Peter_M=C3=BCller?= X-Patchwork-Id: 4738 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384 client-signature ECDSA (P-384) client-digest SHA384) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4HGqYp3v5Jz3x1J for ; Sat, 25 Sep 2021 13:31:42 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384 client-signature ECDSA (P-384) client-digest SHA384) (Client CN "mail02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4HGqYn6JZjzQK; Sat, 25 Sep 2021 13:31:41 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4HGqYn1zswz2yCW; Sat, 25 Sep 2021 13:31:41 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4HGqYl5RRSz2xcn for ; Sat, 25 Sep 2021 13:31:39 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4HGqYk1cWnzQK for ; Sat, 25 Sep 2021 13:31:37 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1632576698; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=NwjZymAOm+N4z8Lq1+Va7++K1TrjNj95+eWyKoSShSk=; b=H4+lR4lmrcgTSAGelpMush8gMqDsJwSHD53YBuBxKa9SQnZVewQYTucmrd/6SnmhGAOITp yvvEgPEZN38R8yAg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1632576698; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=NwjZymAOm+N4z8Lq1+Va7++K1TrjNj95+eWyKoSShSk=; b=bXV18sKvNiCae9ORqmn9HGyqQdq3TiQoM/WBYK3h67aFH8p8G3/lkRwzljYhX87m06BeYX eBtMQ24GX7z9P4BTIe5f0bOQorTqkqk+Xgny7Uu5mxg5DQ3EBFSk9HDvoEsxLiEqNNNTV4 aiJ4fX+kzKRSS6EjnSAB28SRf4l797HdabSN6QrpJg4/lctHjowav0ChgNNT0s/0yY4wfk tO5k31tJqWWNMATKgC4OxcZZ7ggo1ldSlc9PtaNNMf7SvjGfguhNivc90EiZADrO9W70fF UPjzzWF0yOnMm8M/VhZ+CGetGhlFSHnuNr+/YXgcLdOwNK3lMPMGywAoIR7sjw== To: "IPFire: Location" From: =?utf-8?q?Peter_M=C3=BCller?= Subject: [RFC PATCH] location-importer.in: Add Spamhaus DROP lists Message-ID: <07e4fb71-b852-216d-0ea3-534f428eaa9b@ipfire.org> Date: Sat, 25 Sep 2021 15:31:33 +0200 MIME-Version: 1.0 Content-Language: en-US X-BeenThere: location@lists.ipfire.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: location-bounces@lists.ipfire.org Sender: "Location" A while ago, it was discussed whether or not libloc should become an "opinionated database", i. e. including any information on a network's reputation. In general, this idea was dismissed as libloc is neither intended nor suitable for such tasks, and we do not want to make (political?) decisions like these for various reasons. All we do is to provide a useful location database in a neutral way, and leave it up to our users on how to react on certain results. However, there is a problematic area. Take AS55303 as an example: We _know_ this is to be a dirty network, tampering with RIR data and hijacking IP space, and strongly recommend against processing any connection originating from or directed to it. Since it appears to be loaded with proxies used by miscreants for abusive purposes, all we can do at the time of writing is to flag it as "anonymous proxy", but we lack possibility of telling our users something like "this is not a safe area". The very same goes for known bulletproof ISPs, IP hijackers, and so forth. This patch therefore suggests to populate the "is_drop" flag introduced in libloc 0.9.8 (albeit currently unused in production) with the contents of Spamhaus' DROP lists (https://www.spamhaus.org/drop/), to have at least the baddest of the bad covered. The very same lists are, in fact, included in popular IPS rulesets as well - a decent amount of IPFire users is therefore likely to have them already enabled, but in a very costly way. It is not planned to go further, partly because there is no other feed publicly available, which would come with the same intention, volatility, and FP rate. Signed-off-by: Peter Müller --- src/python/location-importer.in | 104 ++++++++++++++++++++++++++++++++ 1 file changed, 104 insertions(+) diff --git a/src/python/location-importer.in b/src/python/location-importer.in index da058d3..0f06465 100644 --- a/src/python/location-importer.in +++ b/src/python/location-importer.in @@ -1059,6 +1059,9 @@ class CLI(object): # network allocation lists in a machine-readable format... self._update_overrides_for_aws() + # Update overrides for Spamhaus DROP feeds... + self._update_overrides_for_spamhaus_drop() + for file in ns.files: log.info("Reading %s..." % file) @@ -1243,6 +1246,107 @@ class CLI(object): ) + def _update_overrides_for_spamhaus_drop(self): + downloader = location.importer.Downloader() + + ip_urls = [ + "https://www.spamhaus.org/drop/drop.txt", + "https://www.spamhaus.org/drop/edrop.txt", + "https://www.spamhaus.org/drop/dropv6.txt" + ] + + asn_urls = [ + "https://www.spamhaus.org/drop/asndrop.txt" + ] + + for url in ip_urls: + try: + with downloader.request(url, return_blocks=False) as f: + fcontent = f.body.readlines() + except Exception as e: + log.error("Unable to download Spamhaus DROP URL %s: %s" % (url, e)) + return + + # Iterate through every line, filter comments and add remaining networks to + # the override table in case they are valid... + with self.db.transaction(): + for sline in fcontent: + + # XXX: Is there a better way to do this? + sline = sline.decode("utf-8") + + # Comments start with a semicolon, followed by a space... + if sline.startswith("; "): + continue + + # Extract network and ignore anything afterwards... + try: + network = ipaddress.ip_network(sline.split()[0], strict=False) + except ValueError: + log.error("Unable to parse line: %s" % sline) + continue + + # Sanitize parsed networks... + if not self._check_parsed_network(network): + log.warning("Skipping bogus network found in Spamhaus DROP URL %s: %s" % \ + (url, network)) + continue + + # Conduct SQL statement... + self.db.execute(""" + INSERT INTO network_overrides( + network, + source, + is_drop + ) VALUES (%s, %s, %s) + ON CONFLICT (network) DO NOTHING""", + "%s" % network, + "Spamhaus DROP lists", + True + ) + + for url in asn_urls: + try: + with downloader.request(url, return_blocks=False) as f: + fcontent = f.body.readlines() + except Exception as e: + log.error("Unable to download Spamhaus DROP URL %s: %s" % (url, e)) + return + + # Iterate through every line, filter comments and add remaining ASNs to + # the override table in case they are valid... + with self.db.transaction(): + for sline in fcontent: + + # XXX: Is there a better way to do this? + sline = sline.decode("utf-8") + + # Comments start with a semicolon, followed by a space... + if sline.startswith("; "): + continue + + # Extract ASN and ignore anything afterwards... + asn = int(sline.split()[0].strip("AS")) + + # Sanitize parsed ASNs... + if not ((1 <= asn and asn <= 23455) or (23457 <= asn and asn <= 64495) or (131072 <= asn and asn <= 4199999999)): + log.warning("Skipping bogus ASN found in Spamhaus DROP URL %s: %s" % \ + (url, asn)) + continue + + # Conduct SQL statement... + self.db.execute(""" + INSERT INTO autnum_overrides( + number, + source, + is_drop + ) VALUES (%s, %s, %s) + ON CONFLICT (number) DO NOTHING""", + "%s" % asn, + "Spamhaus ASN-DROP list", + True + ) + @staticmethod def _parse_bool(block, key): val = block.get(key)