From patchwork Sat Feb 17 22:31:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Peter_M=C3=BCller?= X-Patchwork-Id: 7554 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384 client-signature ECDSA (secp384r1) client-digest SHA384) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4Tck6v5Lttz3wlw for ; Sat, 17 Feb 2024 22:31:35 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384 client-signature ECDSA (secp384r1) client-digest SHA384) (Client CN "mail02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4Tck6s2synzqR; Sat, 17 Feb 2024 22:31:33 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4Tck6s1W39z2xcR; Sat, 17 Feb 2024 22:31:33 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) client-signature ECDSA (secp384r1)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4Tck6p57Wnz2xcR for ; Sat, 17 Feb 2024 22:31:30 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4Tck6m4t1NzqR for ; Sat, 17 Feb 2024 22:31:28 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1708209089; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=HMC/EPbxENYdFVKgpP0CtxtD/T41fpIws95cUF4jc/s=; b=yXLgdPdtcSz/3VEiGXzY0pivvNH9etpOCSRtmJJbj5hPHxgAAcx2q5kWq89xjnshErG+Ye qAHv7gUDEPRq+yCw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1708209089; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=HMC/EPbxENYdFVKgpP0CtxtD/T41fpIws95cUF4jc/s=; b=kut6Ao4Ohmsbl3rX4CY1HobSS1FLMMV4h1+ufL9Cjsh1FSpbjZCd2WT5FoGY4OwYlU0uv/ Z90sMKjbBzcwJlwoJ0BFMqo7gD1y1egKn7fY6Yy2JoJZO5SQStigFzd/e5Vfy7F7tXAx21 eXWD7t51/7lL/rXvtH6ex6F7SW5CRXiqImE9yPqaksKJ2hK+kC1LM4L2Rh8bt6S2HrgnFg m8iKJ7usVY/K26l8A/GzbsdK3DFNFAsZSOTo329IYtQEzAiG7ZfdHuQYxlmpMvu/S2hiX/ 6H/46SKiRx6SNQRHXwxKHdlVMUlwRUQQHedx3tPuL9tgfyy8KuNmb2enELVbwQ== Message-ID: <4add526f-913d-4f16-ac80-7642ff9800e0@ipfire.org> Date: Sat, 17 Feb 2024 22:31:00 +0000 MIME-Version: 1.0 To: "IPFire: Location" From: =?utf-8?q?Peter_M=C3=BCller?= Subject: [PATCH v2] location-importer: Fix Spamhaus ASN-DROP parsing Message-ID-Hash: K5K4AHRBLDL6BCTGUDLNJ4JHZPSHFXNJ X-Message-ID-Hash: K5K4AHRBLDL6BCTGUDLNJ4JHZPSHFXNJ X-MailFrom: peter.mueller@ipfire.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.8 Precedence: list List-Id: "IPFire Location development/database maintainance talk." Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: The format of this list has changed, from a plain text file with a customer schema to JSON. Adjust our routines accordingly to make use of this list again. The second version of this patch incorporates Michael's feedback on the first version, and adds AS names to the autnums table in case they are not there already, which closes some gaps on rogue ASNs in the LACNIC area. Signed-off-by: Peter Müller Tested-by: Peter Müller --- src/scripts/location-importer.in | 46 ++++++++++++++++++++++++-------- 1 file changed, 35 insertions(+), 11 deletions(-) diff --git a/src/scripts/location-importer.in b/src/scripts/location-importer.in index 28a4f6c..ac7249d 100644 --- a/src/scripts/location-importer.in +++ b/src/scripts/location-importer.in @@ -3,7 +3,7 @@ # # # libloc - A library to determine the location of someone on the Internet # # # -# Copyright (C) 2020-2022 IPFire Development Team # +# Copyright (C) 2020-2024 IPFire Development Team # # # # This library is free software; you can redistribute it and/or # # modify it under the terms of the GNU Lesser General Public # @@ -1686,7 +1686,7 @@ class CLI(object): ] asn_lists = [ - ("SPAMHAUS-ASNDROP", "https://www.spamhaus.org/drop/asndrop.txt") + ("SPAMHAUS-ASNDROP", "https://www.spamhaus.org/drop/asndrop.json") ] for name, url in ip_lists: @@ -1759,22 +1759,32 @@ class CLI(object): # Iterate through every line, filter comments and add remaining ASNs to # the override table in case they are valid... - for sline in f.readlines(): + for sline in fcontent: # The response is assumed to be encoded in UTF-8... sline = sline.decode("utf-8") - # Comments start with a semicolon... - if sline.startswith(";"): + # Load every line as a JSON object and try to obtain an ASN from it... + try: + lineobj = json.loads(sline) + except json.decoder.JSONDecodeError: + log.error("Unable to parse line as a JSON object: %s" % sline) continue - # Throw away anything after the first space... - sline = sline.split()[0] + # Skip line contiaining file metadata + try: + type = lineobj["type"] - # ... strip the "AS" prefix from it ... - sline = sline.strip("AS") + if type == "metadata": + continue + except KeyError: + pass - # ... and convert it into an integer. Voila. - asn = int(sline) + try: + asn = lineobj["asn"] + as_name = lineobj["asname"] + except KeyError: + log.warning("Unable to extract necessary information from line: %s" % sline) + continue # Filter invalid ASNs... if not self._check_parsed_asn(asn): @@ -1795,6 +1805,20 @@ class CLI(object): True ) + # In case we do not have an name for this AS already, update + # autnums table accordingly + self.db.execute(""" + INSERT INTO autnums( + number, + name, + source + ) VALUES (%s, %s, %s) + ON CONFLICT (number) DO NOTHING""", + "%s" % asn, + as_name, + name + ) + @staticmethod def _parse_bool(block, key): val = block.get(key)