From patchwork Sat May 15 11:57:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Peter_M=C3=BCller?= X-Patchwork-Id: 4276 Return-Path: Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4Fj3m53R90z3wbf for ; Sat, 15 May 2021 11:57:09 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384 client-signature ECDSA (P-384) client-digest SHA384) (Client CN "mail02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4Fj3m50cLMzjl; Sat, 15 May 2021 11:57:09 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4Fj3m50CfZz2xPx; Sat, 15 May 2021 11:57:09 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4Fj3m36vntz2xPx for ; Sat, 15 May 2021 11:57:07 +0000 (UTC) Received: from location02.haj.ipfire.org (location02.haj.ipfire.org [172.28.1.170]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384 client-signature ECDSA (P-384) client-digest SHA384) (Client CN "location02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4Fj3m31fNHzpJ; Sat, 15 May 2021 11:57:07 +0000 (UTC) Received: by location02.haj.ipfire.org (Postfix, from userid 0) id 4Fj3m30p99zyXN; Sat, 15 May 2021 11:57:07 +0000 (UTC) From: =?utf-8?q?Peter_M=C3=BCller?= To: location@lists.ipfire.org Subject: [PATCH 2/2] importer.py: add source information for RIR data feeds Date: Sat, 15 May 2021 11:57:05 +0000 Message-Id: <20210515115705.9794-2-peter.mueller@ipfire.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210515115705.9794-1-peter.mueller@ipfire.org> References: <20210515115705.9794-1-peter.mueller@ipfire.org> MIME-Version: 1.0 X-BeenThere: location@lists.ipfire.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: location-bounces@lists.ipfire.org Sender: "Location" Signed-off-by: Peter Müller --- src/python/importer.py | 66 +++++++++++++++++++++++++++--------------- 1 file changed, 42 insertions(+), 24 deletions(-) diff --git a/src/python/importer.py b/src/python/importer.py index 5f46bc3..4c8406c 100644 --- a/src/python/importer.py +++ b/src/python/importer.py @@ -25,50 +25,68 @@ import urllib.request log = logging.getLogger("location.importer") log.propagate = 1 -WHOIS_SOURCES = ( +WHOIS_SOURCES = { # African Network Information Centre - "https://ftp.afrinic.net/pub/pub/dbase/afrinic.db.gz", + "AFRINIC": [ + "https://ftp.afrinic.net/pub/pub/dbase/afrinic.db.gz" + ], # Asia Pacific Network Information Centre - "https://ftp.apnic.net/apnic/whois/apnic.db.inet6num.gz", - "https://ftp.apnic.net/apnic/whois/apnic.db.inetnum.gz", - #"https://ftp.apnic.net/apnic/whois/apnic.db.route6.gz", - #"https://ftp.apnic.net/apnic/whois/apnic.db.route.gz", - "https://ftp.apnic.net/apnic/whois/apnic.db.aut-num.gz", - "https://ftp.apnic.net/apnic/whois/apnic.db.organisation.gz", + "APNIC": [ + "https://ftp.apnic.net/apnic/whois/apnic.db.inet6num.gz", + "https://ftp.apnic.net/apnic/whois/apnic.db.inetnum.gz", + #"https://ftp.apnic.net/apnic/whois/apnic.db.route6.gz", + #"https://ftp.apnic.net/apnic/whois/apnic.db.route.gz", + "https://ftp.apnic.net/apnic/whois/apnic.db.aut-num.gz", + "https://ftp.apnic.net/apnic/whois/apnic.db.organisation.gz" + ], # American Registry for Internet Numbers # XXX there is nothing useful for us in here - #"https://ftp.arin.net/pub/rr/arin.db", + # ARIN: [ + # "https://ftp.arin.net/pub/rr/arin.db" + # ], # Latin America and Caribbean Network Information Centre # XXX ??? # Réseaux IP Européens - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inet6num.gz", - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inetnum.gz", - #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route6.gz", - #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route.gz", - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.aut-num.gz", - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.organisation.gz", -) - -EXTENDED_SOURCES = ( + "RIPE": [ + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inet6num.gz", + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inetnum.gz", + #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route6.gz", + #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route.gz", + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.aut-num.gz", + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.organisation.gz" + ], +} + +EXTENDED_SOURCES = { # African Network Information Centre - #"https://ftp.afrinic.net/pub/stats/afrinic/delegated-afrinic-extended-latest", + # "ARIN": [ + # "https://ftp.afrinic.net/pub/stats/afrinic/delegated-afrinic-extended-latest" + # ], # Asia Pacific Network Information Centre - #"https://ftp.apnic.net/apnic/stats/apnic/delegated-apnic-extended-latest", + # "APNIC": [ + # "https://ftp.apnic.net/apnic/stats/apnic/delegated-apnic-extended-latest" + # ], # American Registry for Internet Numbers - "https://ftp.arin.net/pub/stats/arin/delegated-arin-extended-latest", + "ARIN": [ + "https://ftp.arin.net/pub/stats/arin/delegated-arin-extended-latest" + ], # Latin America and Caribbean Network Information Centre - "https://ftp.lacnic.net/pub/stats/lacnic/delegated-lacnic-extended-latest", + "LACNIC": [ + "https://ftp.lacnic.net/pub/stats/lacnic/delegated-lacnic-extended-latest" + ], # Réseaux IP Européens - #"https://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest", -) + # "RIPE": [ + # "https://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest" + # ], +} class Downloader(object): def __init__(self):