From patchwork Mon Nov 1 18:24:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Peter_M=C3=BCller?= X-Patchwork-Id: 4826 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4HjhJR2T6hz3wdP for ; Mon, 1 Nov 2021 18:24:23 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384 client-signature ECDSA (P-384) client-digest SHA384) (Client CN "mail02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4HjhJQ5jqvzTm; Mon, 1 Nov 2021 18:24:22 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4HjhJQ2wGfz2xR7; Mon, 1 Nov 2021 18:24:22 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4HjhJP0Gbsz2xfN for ; Mon, 1 Nov 2021 18:24:21 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4HjhJM5xTwzB0 for ; Mon, 1 Nov 2021 18:24:19 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1635791060; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=1v2TETuohUt8Rc+87YTX7LaL9Xzi6bWj71UL5WSvlpg=; b=P+MzaBfVEcTl4hllTbBBlgIQSq7nwykqk7q6WpYJQe8dR2+M4h9wO7IHCxgO1kRZSvTq09 NKSZ7o59hdl2PnBQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1635791060; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=1v2TETuohUt8Rc+87YTX7LaL9Xzi6bWj71UL5WSvlpg=; b=XeNiUvc2paFSue4oAjHBfQWkm+BHMKDhWsiIUxFE/qSRZ9QlXPeEeB4Ty3WdCe0CbIuMmI SPJKK1fxoQ0ifxlN5SCucxLqqpSW9Jd/Jsv5GyMn3/e+Vr39n80wwT/KBA3hxlYvl9Clvb S/bP7NDB0Rx/i9KDdqHJysFUGwOxtCS1INA/NlWz0V1BWEisffPwBM2cpVB0oa4qrD4Lpc 1LLVr8n4HND+l7E1LOOcC3FQo0AYIG3OupQvwFM53cQ8T/6BjZ7YEjPIeCECs8fTt7x4D4 MrDK2QxtrZrjK+viN05lfDYa8RjD57OerZqo80aRH1vb6E5aA24KnNURKSxtEQ== To: "IPFire: Location" From: =?utf-8?q?Peter_M=C3=BCller?= Subject: [PATCH v4 1/2] location-importer: Introduce auxiliary function to sanitise ASNs Message-ID: <50907932-7f1f-e104-c409-736c4e4db029@ipfire.org> Date: Mon, 1 Nov 2021 19:24:14 +0100 MIME-Version: 1.0 Content-Language: en-US X-BeenThere: location@lists.ipfire.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: location-bounces@lists.ipfire.org Sender: "Location" The third version of this patch does this in an even more Pythonic way. Signed-off-by: Peter Müller --- src/python/location-importer.in | 25 +++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/src/python/location-importer.in b/src/python/location-importer.in index da058d3..ede8a75 100644 --- a/src/python/location-importer.in +++ b/src/python/location-importer.in @@ -37,6 +37,14 @@ from location.i18n import _ log = logging.getLogger("location.importer") log.propagate = 1 +# Define constants +VALID_ASN_RANGES = ( + (1, 23455), + (23457, 64495), + (131072, 4199999999), +) + + class CLI(object): def parse_cli(self): parser = argparse.ArgumentParser( @@ -574,6 +582,19 @@ class CLI(object): # be suitable for libloc consumption... return True + def _check_parsed_asn(self, asn): + """ + Assistive function to filter Autonomous System Numbers not being suitable + for adding to our database. Returns False in such cases, and True otherwise. + """ + + for start, end in VALID_ASN_RANGES: + if start <= asn and end >= asn: + return True + + log.info("Supplied ASN %s out of publicly routable ASN ranges" % asn) + return False + def _parse_block(self, block, source_key, validcountries = None): # Get first line to find out what type of block this is line = block[0] @@ -829,8 +850,8 @@ class CLI(object): log.debug("Skipping ARIN AS names line not containing an integer for ASN") continue - if not ((1 <= asn and asn <= 23455) or (23457 <= asn and asn <= 64495) or (131072 <= asn and asn <= 4199999999)): - log.debug("Skipping ARIN AS names line not containing a valid ASN: %s" % asn) + # Filter invalid ASNs... + if not self._check_parsed_asn(asn): continue # Skip any AS name that appears to be a placeholder for a different RIR or entity... From patchwork Mon Nov 1 18:24:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Peter_M=C3=BCller?= X-Patchwork-Id: 4827 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4HjhJn53R1z3wdP for ; Mon, 1 Nov 2021 18:24:41 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4HjhJn2vHqzB0; Mon, 1 Nov 2021 18:24:41 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4HjhJn3LKFz2xR7; Mon, 1 Nov 2021 18:24:41 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4HjhJm2xYDz2xR7 for ; Mon, 1 Nov 2021 18:24:40 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4HjhJl2HJzzB0 for ; Mon, 1 Nov 2021 18:24:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1635791079; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=K/njz8MQXdIkBKQMX3Ty5VuGvNbHbxVUcHtKZYRML6Q=; b=gqi09vakOi3SYzZqXl4HKlYYGGp9J3W1hh+WmWBZ3BvryoYZugjwIm1J81Zt2ERHoco9RL X3TMBgGPSyuVEeEWg2z/XJP5YMOkWrUn+avZZfeD7RKc5jMF4Qgw1jrOE4LzfOEKbr2r10 VMMRLYstT2OOenBHtcPYceuvRcazbDW3OpTChY4OPPBGZklWN5p0FZxLoCqz5macXnJjOJ fAPcLcgs6CxTexh0AYGQZ/K1yaVFti4AL28SRz9APXM4QH4zblNrL9cE0GGdmlOFuJ3rVE PeLxO/l+sOsUCh9CPijPfaeWSwcVyBP6cSFUesDN0LPxKPKsQ4F9x5mbTdTomA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1635791079; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=K/njz8MQXdIkBKQMX3Ty5VuGvNbHbxVUcHtKZYRML6Q=; b=NHzK5ehyPlpU0IiLpFQDTmn+SYPfycyt8nHLIdCH4esdBvTakyRyCNjJviwpH49UhJUeuS et98kV0WB2V8uyAg== Subject: [PATCH v4 2/2] location-importer.in: Add Spamhaus DROP lists To: location@lists.ipfire.org References: <50907932-7f1f-e104-c409-736c4e4db029@ipfire.org> From: =?utf-8?q?Peter_M=C3=BCller?= Message-ID: <06633ea1-1f77-29f1-6378-597f8075a327@ipfire.org> Date: Mon, 1 Nov 2021 19:24:37 +0100 MIME-Version: 1.0 In-Reply-To: <50907932-7f1f-e104-c409-736c4e4db029@ipfire.org> Content-Language: en-US X-BeenThere: location@lists.ipfire.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: location-bounces@lists.ipfire.org Sender: "Location" A while ago, it was discussed whether or not libloc should become an "opinionated database", i. e. including any information on a network's reputation. In general, this idea was dismissed as libloc is neither intended nor suitable for such tasks, and we do not want to make (political?) decisions like these for various reasons. All we do is to provide a useful location database in a neutral way, and leave it up to our users on how to react on certain results. However, there is a problematic area. Take AS55303 as an example: We _know_ this is to be a dirty network, tampering with RIR data and hijacking IP space, and strongly recommend against processing any connection originating from or directed to it. Since it appears to be loaded with proxies used by miscreants for abusive purposes, all we can do at the time of writing is to flag it as "anonymous proxy", but we lack possibility of telling our users something like "this is not a safe area". The very same goes for known bulletproof ISPs, IP hijackers, and so forth. This patch therefore suggests to populate the "is_drop" flag introduced in libloc 0.9.8 (albeit currently unused in production) with the contents of Spamhaus' DROP lists (https://www.spamhaus.org/drop/), to have at least the baddest of the bad covered. The very same lists are, in fact, included in popular IPS rulesets as well - a decent amount of IPFire users is therefore likely to have them already enabled, but in a very costly way. It is not planned to go further, partly because there is no other feed publicly available, which would come with the same intention, volatility, and FP rate. The third version of this patch makes use of an auxiliary function to sanitise ASNs, hence avoiding boilerplate code, and treats any line starting with a semicolon as a comment, which should be sufficient. Further, extracting ASNs from the ASN-DROP feed is done in a more clear way, avoiding code snippets hard to read. Signed-off-by: Peter Müller --- src/python/location-importer.in | 110 ++++++++++++++++++++++++++++++++ 1 file changed, 110 insertions(+) diff --git a/src/python/location-importer.in b/src/python/location-importer.in index ede8a75..b791b4d 100644 --- a/src/python/location-importer.in +++ b/src/python/location-importer.in @@ -1080,6 +1080,9 @@ class CLI(object): # network allocation lists in a machine-readable format... self._update_overrides_for_aws() + # Update overrides for Spamhaus DROP feeds... + self._update_overrides_for_spamhaus_drop() + for file in ns.files: log.info("Reading %s..." % file) @@ -1264,6 +1267,113 @@ class CLI(object): ) + def _update_overrides_for_spamhaus_drop(self): + downloader = location.importer.Downloader() + + ip_urls = [ + "https://www.spamhaus.org/drop/drop.txt", + "https://www.spamhaus.org/drop/edrop.txt", + "https://www.spamhaus.org/drop/dropv6.txt" + ] + + asn_urls = [ + "https://www.spamhaus.org/drop/asndrop.txt" + ] + + for url in ip_urls: + try: + with downloader.request(url, return_blocks=False) as f: + fcontent = f.body.readlines() + except Exception as e: + log.error("Unable to download Spamhaus DROP URL %s: %s" % (url, e)) + return + + # Iterate through every line, filter comments and add remaining networks to + # the override table in case they are valid... + with self.db.transaction(): + for sline in fcontent: + + # The response is assumed to be encoded in UTF-8... + sline = sline.decode("utf-8") + + # Comments start with a semicolon... + if sline.startswith(";"): + continue + + # Extract network and ignore anything afterwards... + try: + network = ipaddress.ip_network(sline.split()[0], strict=False) + except ValueError: + log.error("Unable to parse line: %s" % sline) + continue + + # Sanitize parsed networks... + if not self._check_parsed_network(network): + log.warning("Skipping bogus network found in Spamhaus DROP URL %s: %s" % \ + (url, network)) + continue + + # Conduct SQL statement... + self.db.execute(""" + INSERT INTO network_overrides( + network, + source, + is_drop + ) VALUES (%s, %s, %s) + ON CONFLICT (network) DO NOTHING""", + "%s" % network, + "Spamhaus DROP lists", + True + ) + + for url in asn_urls: + try: + with downloader.request(url, return_blocks=False) as f: + fcontent = f.body.readlines() + except Exception as e: + log.error("Unable to download Spamhaus DROP URL %s: %s" % (url, e)) + return + + # Iterate through every line, filter comments and add remaining ASNs to + # the override table in case they are valid... + with self.db.transaction(): + for sline in fcontent: + + # The response is assumed to be encoded in UTF-8... + sline = sline.decode("utf-8") + + # Comments start with a semicolon... + if sline.startswith(";"): + continue + + # Throw away anything after the first space... + sline = sline.split()[0] + + # ... strip the "AS" prefix from it ... + sline = sline.strip("AS") + + # ... and convert it into an integer. Voila. + asn = int(sline) + + # Filter invalid ASNs... + if not self._check_parsed_asn(asn): + log.warning("Skipping bogus ASN found in Spamhaus DROP URL %s: %s" % \ + (url, asn)) + continue + + # Conduct SQL statement... + self.db.execute(""" + INSERT INTO autnum_overrides( + number, + source, + is_drop + ) VALUES (%s, %s, %s) + ON CONFLICT (number) DO NOTHING""", + "%s" % asn, + "Spamhaus ASN-DROP list", + True + ) + @staticmethod def _parse_bool(block, key): val = block.get(key)