From patchwork Mon Mar 29 20:24:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Peter_M=C3=BCller?= X-Patchwork-Id: 3997 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4F8PFT5lQ6z3wrV for ; Mon, 29 Mar 2021 20:24:45 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4F8PFT49glzld; Mon, 29 Mar 2021 20:24:45 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4F8PFT40Sjz2xSn; Mon, 29 Mar 2021 20:24:45 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4F8PFS1PRSz2xTN for ; Mon, 29 Mar 2021 20:24:44 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4F8PFQ0T2Zzld for ; Mon, 29 Mar 2021 20:24:41 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1617049483; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=z4MhFljFrXganzoyT+wpMfi2CzCAI59SaO59ZSBqjsU=; b=6YM7VfjfLOIqhrpwbD5iMcB/iuV805OkvBD0/PV485DNKlB1KhxetnBal0nHVOEqjsj9Ma /CPJqGY4y//ed+BA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1617049483; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=z4MhFljFrXganzoyT+wpMfi2CzCAI59SaO59ZSBqjsU=; b=plqcH3+oYtS9G3DPOG3s3NnJ22/FwygPZ+Ph5uAM6YP1H8WD1n85Bd7mzwFHgevfiVCLqG PBpsNnKcwxuqS7SFXoWb8qtNBDqAg20ochYdp6GjlBYCE3dp5LOkqZJestyeeEbVRe1Z54 bdSw9OqJQlMhKGh6IMHotIQ83wSgshfHNwz8gDtF7hZIBwuWLWRGbujqCrA3CKs4dB+0rL VXQAXgUGvnhaO5wyaBan4A7Ov7sOq7x9Ws5LJ9vPgejZg4UU5GCffCICu97IYQSchhW3DA RXxEcHaCKYxv++nSHMYZtZQnhzINXV8YZ+ppQW0Ez7ox6dU0Tj+5png+YrRxnQ== To: "IPFire: Location" From: =?utf-8?q?Peter_M=C3=BCller?= Subject: [PATCH] location-importer.in: process unaligned IP ranges in RIR data files correctly Message-ID: <8d42901f-e1be-285f-2060-639218e2b694@ipfire.org> Date: Mon, 29 Mar 2021 22:24:36 +0200 MIME-Version: 1.0 Content-Language: en-US X-BeenThere: location@lists.ipfire.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: location-bounces@lists.ipfire.org Sender: "Location" The IP range given in an inetnum object apparently not necessarily matches distinct subnet boundaries. As a result, the current attempt to calculate its CIDR mask resulted in faulty subnets not covering the entire IP range. This patch leaves the task of enumerating subnets to the ipaddress module itself, which handles things much more robust. Since the output may contain of several subnets, a list for the inetnum key is necessary as well as a loop over them when conducting the SQL statements. Fixes: #12595 Cc: Michael Tremer Signed-off-by: Peter Müller --- src/python/location-importer.in | 31 +++++++++++-------------------- 1 file changed, 11 insertions(+), 20 deletions(-) diff --git a/src/python/location-importer.in b/src/python/location-importer.in index 2506925..e2f201b 100644 --- a/src/python/location-importer.in +++ b/src/python/location-importer.in @@ -3,7 +3,7 @@ # # # libloc - A library to determine the location of someone on the Internet # # # -# Copyright (C) 2020 IPFire Development Team # +# Copyright (C) 2020-2021 IPFire Development Team # # # # This library is free software; you can redistribute it and/or # # modify it under the terms of the GNU Lesser General Public # @@ -604,18 +604,10 @@ class CLI(object): log.warning("Could not parse line: %s" % line) return - # Set prefix to default - prefix = 32 - - # Count number of addresses in this subnet - num_addresses = int(end_address) - int(start_address) - if num_addresses: - prefix -= math.log(num_addresses, 2) - - inetnum["inetnum"] = "%s/%.0f" % (start_address, prefix) + inetnum["inetnum"] = list(ipaddress.summarize_address_range(start_address, end_address)) elif key == "inet6num": - inetnum[key] = val + inetnum[key] = [ipaddress.ip_network(val, strict=False)] elif key == "country": inetnum[key] = val.upper() @@ -630,15 +622,14 @@ class CLI(object): (inetnum.get("inet6num") or inetnum.get("inetnum"))) return - network = ipaddress.ip_network(inetnum.get("inet6num") or inetnum.get("inetnum"), strict=False) - - if not self._check_parsed_network(network): - return - - self.db.execute("INSERT INTO _rirdata(network, country) \ - VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country = excluded.country", - "%s" % network, inetnum.get("country"), - ) + # Iterate through all networks enumerated from above, check them for plausibility and insert + # them into the database, if _check_parsed_network() succeeded + for single_network in inetnum.get("inet6num") or inetnum.get("inetnum"): + if self._check_parsed_network(single_network): + self.db.execute("INSERT INTO _rirdata(network, country) \ + VALUES(%s, %s) ON CONFLICT (network) DO UPDATE SET country = excluded.country", + "%s" % single_network, inetnum.get("country"), + ) def _parse_org_block(self, block): org = {}