From patchwork Mon Sep 26 18:26:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Peter_M=C3=BCller?= X-Patchwork-Id: 6012 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4Mbrnc1qbsz3wc4 for ; Mon, 26 Sep 2022 18:27:00 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4Mbrnb4RHvz2BD; Mon, 26 Sep 2022 18:26:59 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4Mbrnb4PFRz2xX7; Mon, 26 Sep 2022 18:26:59 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4MbrnZ6W88z2xGV for ; Mon, 26 Sep 2022 18:26:58 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4MbrnY1LsPzDc for ; Mon, 26 Sep 2022 18:26:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1664216818; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pUCMqtR4UkFSuJOvRSeBOuTNBKbJpadaJRTALq839E8=; b=UOkYzkkOkR/D49wuCHpTqS7/+dHXYHlxlJeSeuKzaADKoIJ6L72X1hUj+s9bHheGYcWGcb g0mlOgyoKp5Z/YqnEVVwcMPQGcMyddrLA1tE5hEUAtkpegJj8ERscgoNGTJ+I/s8DuXEDD 64WzWnD8fnQM7XEFdo1o38h+7SHkY/ByEELvRNxRXyU/4CLvPqdjZkpkvBYxCgcZ20JRuj saQlbx0qYhMAZQaqsydu4q/hXUC/6On3F8FFOMjg1KuyhyFec+x+lEf5JRHT82oBWDzjK1 oQF7SskNLm6PIJ9g5V2eGIJXshvyK5fHVICk6QdHjonh+/4W5mOYt+JB/tlOsQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1664216818; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pUCMqtR4UkFSuJOvRSeBOuTNBKbJpadaJRTALq839E8=; b=y4ig1mp+VeqahlX2lohIX6Pi+62CkNGIlu9RQ8VmyC9cNQEOlLbMxntfpQMGScq4Iwiu/b XGPNcQYKSYv0A2BQ== Message-ID: Date: Mon, 26 Sep 2022 18:26:44 +0000 MIME-Version: 1.0 Subject: [PATCH v2] location-importer.in: Conduct sanity checks per DROP list Content-Language: en-US To: location@lists.ipfire.org References: <87682839-dc30-ac0e-6bb1-ec705bdc793e@ipfire.org> From: =?utf-8?q?Peter_M=C3=BCller?= In-Reply-To: <87682839-dc30-ac0e-6bb1-ec705bdc793e@ipfire.org> X-BeenThere: location@lists.ipfire.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: location-bounces@lists.ipfire.org Sender: "Location" Previously, the lack of distinction between different DROP lists caused only the last one to be persisted. The second version of this patch incorporates suggestions from Michael on the first version. Tested-by: Peter Müller Signed-off-by: Peter Müller --- src/scripts/location-importer.in | 74 +++++++++++++++++++------------- 1 file changed, 44 insertions(+), 30 deletions(-) diff --git a/src/scripts/location-importer.in b/src/scripts/location-importer.in index 8d47497..d405eb2 100644 --- a/src/scripts/location-importer.in +++ b/src/scripts/location-importer.in @@ -1427,37 +1427,37 @@ class CLI(object): def _update_overrides_for_spamhaus_drop(self): downloader = location.importer.Downloader() - ip_urls = [ - "https://www.spamhaus.org/drop/drop.txt", - "https://www.spamhaus.org/drop/edrop.txt", - "https://www.spamhaus.org/drop/dropv6.txt" + ip_lists = [ + ("SPAMHAUS-DROP", "https://www.spamhaus.org/drop/drop.txt"), + ("SPAMHAUS-EDROP", "https://www.spamhaus.org/drop/edrop.txt"), + ("SPAMHAUS-DROPV6", "https://www.spamhaus.org/drop/dropv6.txt") ] - asn_urls = [ - "https://www.spamhaus.org/drop/asndrop.txt" + asn_lists = [ + ("SPAMHAUS-ASNDROP", "https://www.spamhaus.org/drop/asndrop.txt") ] - for url in ip_urls: - # Fetch IP list + for name, url in ip_lists: + # Fetch IP list from given URL f = downloader.retrieve(url) # Split into lines fcontent = f.readlines() - # Conduct a very basic sanity check to rule out CDN issues causing bogus DROP - # downloads. - if len(fcontent) > 10: - self.db.execute(""" - DELETE FROM autnum_overrides WHERE source = 'Spamhaus ASN-DROP list'; - DELETE FROM network_overrides WHERE source = 'Spamhaus DROP lists'; - """) - else: - log.error("Spamhaus DROP URL %s returned likely bogus file, ignored" % url) - continue - - # Iterate through every line, filter comments and add remaining networks to - # the override table in case they are valid... with self.db.transaction(): + # Conduct a very basic sanity check to rule out CDN issues causing bogus DROP + # downloads. + if len(fcontent) > 10: + self.db.execute(""" + DELETE FROM network_overrides WHERE source = '%s'; + """ % name, + ) + else: + log.error("%s (%s) returned likely bogus file, ignored" % (name, url)) + continue + + # Iterate through every line, filter comments and add remaining networks to + # the override table in case they are valid... for sline in fcontent: # The response is assumed to be encoded in UTF-8... sline = sline.decode("utf-8") @@ -1475,8 +1475,8 @@ class CLI(object): # Sanitize parsed networks... if not self._check_parsed_network(network): - log.warning("Skipping bogus network found in Spamhaus DROP URL %s: %s" % \ - (url, network)) + log.warning("Skipping bogus network found in %s (%s): %s" % \ + (name, url, network)) continue # Conduct SQL statement... @@ -1488,17 +1488,31 @@ class CLI(object): ) VALUES (%s, %s, %s) ON CONFLICT (network) DO UPDATE SET is_drop = True""", "%s" % network, - "Spamhaus DROP lists", + name, True ) - for url in asn_urls: + for name, url in asn_lists: # Fetch URL f = downloader.retrieve(url) - # Iterate through every line, filter comments and add remaining ASNs to - # the override table in case they are valid... + # Split into lines + fcontent = f.readlines() + with self.db.transaction(): + # Conduct a very basic sanity check to rule out CDN issues causing bogus DROP + # downloads. + if len(fcontent) > 10: + self.db.execute(""" + DELETE FROM autnum_overrides WHERE source = '%s'; + """ % name, + ) + else: + log.error("%s (%s) returned likely bogus file, ignored" % (name, url)) + continue + + # Iterate through every line, filter comments and add remaining ASNs to + # the override table in case they are valid... for sline in f.readlines(): # The response is assumed to be encoded in UTF-8... sline = sline.decode("utf-8") @@ -1518,8 +1532,8 @@ class CLI(object): # Filter invalid ASNs... if not self._check_parsed_asn(asn): - log.warning("Skipping bogus ASN found in Spamhaus DROP URL %s: %s" % \ - (url, asn)) + log.warning("Skipping bogus ASN found in %s (%s): %s" % \ + (name, url, asn)) continue # Conduct SQL statement... @@ -1531,7 +1545,7 @@ class CLI(object): ) VALUES (%s, %s, %s) ON CONFLICT (number) DO UPDATE SET is_drop = True""", "%s" % asn, - "Spamhaus ASN-DROP list", + name, True )