From patchwork Sun Jun 5 10:04:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Peter_M=C3=BCller?= X-Patchwork-Id: 5645 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4LGC0W315Mz3xqn for ; Sun, 5 Jun 2022 10:04:59 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4LGC0V4p0nzlX; Sun, 5 Jun 2022 10:04:58 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4LGC0V3rZbz2xxK; Sun, 5 Jun 2022 10:04:58 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4LGC0V10gCz2xLF for ; Sun, 5 Jun 2022 10:04:58 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4LGC0T1GbXzlX for ; Sun, 5 Jun 2022 10:04:56 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1654423498; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=j+XpFzRLylttGULSJyIG5+kc1+ya0ZPTD9dAHiSVV+c=; b=HE60M3iOH7lVmuzGR/0lnW0cG/rBVVj6YKoMj12PV30o7JQJn4Sr1YQ2TNiTDR+siYCUHD VAN3Zmrnl59icyAQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1654423498; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=j+XpFzRLylttGULSJyIG5+kc1+ya0ZPTD9dAHiSVV+c=; b=fNGSyrUGJt4u4dlwdkVF4f9/8oDDrNKEJJBrOEc2eiVrBwlbQ/ZZ+CpZ3x501ulwf+MXNJ 4cPha0Zhtw8nmwVyEbgAthH0QBIyAs90vtProjI/tNUATYvI7kl3PjAnXLoHp/EECGe8to FSHMY9UuwiMEmLc0HjP09lwv8+Ofd0Fw19bIwSvlTI2HViFB2MkyAisiy402JxO49YmWVD 5u8enk6y5PqiDu0p8xKC5V7PE4lwscKxCDNfxb0Th7sorFO8efUNcl3JYvUp4ApU/j4+hl u4HV6ZRrQ47WChSunRO9XgK3msRB9KnEnreFxo+2WaM1qCU3GmB75LsNPxVi+Q== Message-ID: <1c84d2fc-c061-80eb-4624-288c263b78bb@ipfire.org> Date: Sun, 5 Jun 2022 10:04:50 +0000 MIME-Version: 1.0 Content-Language: en-US To: "IPFire: Location" From: =?utf-8?q?Peter_M=C3=BCller?= Subject: [PATCH] location-importer: Only delete override data if we are sure to have a valid replacement X-BeenThere: location@lists.ipfire.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: location-bounces@lists.ipfire.org Sender: "Location" The current way of truncating all override data straight away leaves us with no data at all, should a source turn out to be unreachable or returning bogus files (yes, Cloudflare, I _am_ looking at you). It is therefore better to only delete data we know to have a valid replacement for, rather than just dropping the source altogether. Signed-off-by: Peter Müller --- src/scripts/location-importer.in | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/src/scripts/location-importer.in b/src/scripts/location-importer.in index bee9186..bde92ce 100644 --- a/src/scripts/location-importer.in +++ b/src/scripts/location-importer.in @@ -1168,10 +1168,11 @@ class CLI(object): def handle_update_overrides(self, ns): with self.db.transaction(): - # Drop all data that we have + # Only drop manually created overrides, as we can be reasonably sure to have them, + # and preserve the rest. If appropriate, it is deleted by correspondent functions. self.db.execute(""" - TRUNCATE TABLE autnum_overrides; - TRUNCATE TABLE network_overrides; + DELETE FROM autnum_overrides WHERE source = 'manual'; + DELETE FROM network_overrides WHERE source = 'manual'; """) # Update overrides for various cloud providers big enough to publish their own IP @@ -1267,6 +1268,11 @@ class CLI(object): log.error("unable to preprocess Amazon AWS IP ranges: %s" % e) return + # At this point, we can assume the downloaded file to be valid + self.db.execute(""" + DELETE FROM network_overrides WHERE source = 'Amazon AWS IP feed'; + """) + # XXX: Set up a dictionary for mapping a region name to a country. Unfortunately, # there seems to be no machine-readable version available of this other than # https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html @@ -1387,6 +1393,16 @@ class CLI(object): log.error("Unable to download Spamhaus DROP URL %s: %s" % (url, e)) return + # Conduct a very basic sanity check to rule out CDN issues causing bogus DROP + # downloads. + if len(fcontent) > 10: + self.db.execute(""" + DELETE FROM autnum_overrides WHERE source = 'Spamhaus ASN-DROP list'; + DELETE FROM network_overrides WHERE source = 'Spamhaus DROP lists'; + """) + else: + log.error("Spamhaus DROP URL %s returned likely bogus file, ignored" % url) + # Iterate through every line, filter comments and add remaining networks to # the override table in case they are valid... with self.db.transaction():