From patchwork Sat Nov 4 14:04:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Peter_M=C3=BCller?= X-Patchwork-Id: 7302 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384 client-signature ECDSA (secp384r1) client-digest SHA384) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4SMzrc6HvSz3wtG for ; Sat, 4 Nov 2023 14:04:48 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384 client-signature ECDSA (secp384r1) client-digest SHA384) (Client CN "mail02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4SMzrb6J2Kz4m; Sat, 4 Nov 2023 14:04:47 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4SMzrb4rlSz2xgv; Sat, 4 Nov 2023 14:04:47 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384 client-signature ECDSA (secp384r1) client-digest SHA384) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4SMzrY1m0vz2xgv for ; Sat, 4 Nov 2023 14:04:45 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4SMzrW3c40z4m for ; Sat, 4 Nov 2023 14:04:43 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1699106684; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=HqbPpveX0tNJXY1Gg5m0Za6SYcdRGSytwyOhRx2TVQY=; b=i1RjpfG57lGTZsSQmhI7a4QqhbH+HIy2k+A+w8DTmp1raFAj3gy/gkMbMVJ7s59EHWSqCo +moh+jGEn+216mAw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1699106684; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=HqbPpveX0tNJXY1Gg5m0Za6SYcdRGSytwyOhRx2TVQY=; b=NOFtylpAYJttwwKiVdSXuKh0IAaLufzVum+VO0VR1GLCvmL6jJMzOrrojiAA7L5WpRb6MM KnH45URi8pTIg/NX2PeNKznC/i0+edznmQAWXQbUGVAQGHCxT+7T3WP1sMN8+xK2O5U7jA qHCHYrhcQQyFnLYHa2CwExrmy2xvQ0dDrskkwGlyXaex6eskp7FgTQkgYCbHQc3PuLMhG2 RJ4/PYkXFHT0o21VgnhCMZFR29AV5bVR4qiHqSL62fpg7mejp4KC6LI2ip4tHErQXOhE2G VwVrYx3iys+Ctt2r95Mcz3KdiFlFCeb9roMP+3Jf0GWl6K9QpgUj/qoA0a7nyw== Message-ID: Date: Sat, 4 Nov 2023 14:04:00 +0000 MIME-Version: 1.0 To: "IPFire: Location" From: =?utf-8?q?Peter_M=C3=BCller?= Subject: [PATCH] location-importer: Fix Spamhaus ASN-DROP parsing Message-ID-Hash: DA5B2BP47JPBO63UTXOW3TTFBB7I3ACR X-Message-ID-Hash: DA5B2BP47JPBO63UTXOW3TTFBB7I3ACR X-MailFrom: peter.mueller@ipfire.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.8 Precedence: list List-Id: "IPFire Location development/database maintainance talk." Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: The format of this list has changed, from a plain text file with a customer schema to JSON. Adjust our routines accordingly to make use of this list again. Signed-off-by: Peter Müller Tested-by: Peter Müller --- src/scripts/location-importer.in | 21 +++++++-------------- 1 file changed, 7 insertions(+), 14 deletions(-) diff --git a/src/scripts/location-importer.in b/src/scripts/location-importer.in index 28a4f6c..8b0c676 100644 --- a/src/scripts/location-importer.in +++ b/src/scripts/location-importer.in @@ -3,7 +3,7 @@ # # # libloc - A library to determine the location of someone on the Internet # # # -# Copyright (C) 2020-2022 IPFire Development Team # +# Copyright (C) 2020-2023 IPFire Development Team # # # # This library is free software; you can redistribute it and/or # # modify it under the terms of the GNU Lesser General Public # @@ -1686,7 +1686,7 @@ class CLI(object): ] asn_lists = [ - ("SPAMHAUS-ASNDROP", "https://www.spamhaus.org/drop/asndrop.txt") + ("SPAMHAUS-ASNDROP", "https://www.spamhaus.org/drop/asndrop.json") ] for name, url in ip_lists: @@ -1759,23 +1759,16 @@ class CLI(object): # Iterate through every line, filter comments and add remaining ASNs to # the override table in case they are valid... - for sline in f.readlines(): + for sline in fcontent: # The response is assumed to be encoded in UTF-8... sline = sline.decode("utf-8") - # Comments start with a semicolon... - if sline.startswith(";"): + # Load every line as a JSON object and try to obtain an ASN from it... + try: + asn = json.loads(sline)["asn"] + except KeyError: continue - # Throw away anything after the first space... - sline = sline.split()[0] - - # ... strip the "AS" prefix from it ... - sline = sline.strip("AS") - - # ... and convert it into an integer. Voila. - asn = int(sline) - # Filter invalid ASNs... if not self._check_parsed_asn(asn): log.warning("Skipping bogus ASN found in %s (%s): %s" % \