| Message ID | 20210522203352.22746-2-peter.mueller@ipfire.org |
|---|---|
| State | Accepted |
| Headers |
Return-Path: <location-bounces@lists.ipfire.org> Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384 client-signature ECDSA (P-384) client-digest SHA384) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4FnZv82xTYz3wbk for <patchwork@web04.haj.ipfire.org>; Sat, 22 May 2021 20:33:56 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384 client-signature ECDSA (P-384) client-digest SHA384) (Client CN "mail02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4FnZv75D4xzjT; Sat, 22 May 2021 20:33:55 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4FnZv74tZnz2xbB; Sat, 22 May 2021 20:33:55 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384 client-signature ECDSA (P-384) client-digest SHA384) (Client CN "mail01.haj.ipfire.org", Issuer "R3" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4FnZv636nZz2y0t for <location@lists.ipfire.org>; Sat, 22 May 2021 20:33:54 +0000 (UTC) Received: from location02.haj.ipfire.org (location02.haj.ipfire.org [172.28.1.170]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) client-signature ECDSA (P-384)) (Client CN "location02.haj.ipfire.org", Issuer "R3" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4FnZv5491qz1T6; Sat, 22 May 2021 20:33:53 +0000 (UTC) Received: by location02.haj.ipfire.org (Postfix, from userid 0) id 4FnZv53CyLz13Rf; Sat, 22 May 2021 20:33:53 +0000 (UTC) From: =?utf-8?q?Peter_M=C3=BCller?= <peter.mueller@ipfire.org> To: location@lists.ipfire.org Subject: [PATCH v2 2/2] importer.py: add source information for RIR data feeds Date: Sat, 22 May 2021 20:33:52 +0000 Message-Id: <20210522203352.22746-2-peter.mueller@ipfire.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210522203352.22746-1-peter.mueller@ipfire.org> References: <20210522203352.22746-1-peter.mueller@ipfire.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: location@lists.ipfire.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: <location.lists.ipfire.org> List-Unsubscribe: <https://lists.ipfire.org/mailman/options/location>, <mailto:location-request@lists.ipfire.org?subject=unsubscribe> List-Archive: <http://lists.ipfire.org/pipermail/location/> List-Post: <mailto:location@lists.ipfire.org> List-Help: <mailto:location-request@lists.ipfire.org?subject=help> List-Subscribe: <https://lists.ipfire.org/mailman/listinfo/location>, <mailto:location-request@lists.ipfire.org?subject=subscribe> Errors-To: location-bounces@lists.ipfire.org Sender: "Location" <location-bounces@lists.ipfire.org> |
| Series |
[v2,1/2] location-importer.in: keep track of sources for networks, ASNs, and organisations
|
|
Commit Message
Peter Müller
May 22, 2021, 8:33 p.m. UTC
Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
---
src/python/importer.py | 66 +++++++++++++++++++++++++++---------------
1 file changed, 42 insertions(+), 24 deletions(-)
Comments
Hallo, This probably should have been in the first patch because it won’t work without this. Best, -Michael > On 22 May 2021, at 21:33, Peter Müller <peter.mueller@ipfire.org> wrote: > > Signed-off-by: Peter Müller <peter.mueller@ipfire.org> > --- > src/python/importer.py | 66 +++++++++++++++++++++++++++--------------- > 1 file changed, 42 insertions(+), 24 deletions(-) > > diff --git a/src/python/importer.py b/src/python/importer.py > index 5f46bc3..4c8406c 100644 > --- a/src/python/importer.py > +++ b/src/python/importer.py > @@ -25,50 +25,68 @@ import urllib.request > log = logging.getLogger("location.importer") > log.propagate = 1 > > -WHOIS_SOURCES = ( > +WHOIS_SOURCES = { > # African Network Information Centre > - "https://ftp.afrinic.net/pub/pub/dbase/afrinic.db.gz", > + "AFRINIC": [ > + "https://ftp.afrinic.net/pub/pub/dbase/afrinic.db.gz" > + ], > > # Asia Pacific Network Information Centre > - "https://ftp.apnic.net/apnic/whois/apnic.db.inet6num.gz", > - "https://ftp.apnic.net/apnic/whois/apnic.db.inetnum.gz", > - #"https://ftp.apnic.net/apnic/whois/apnic.db.route6.gz", > - #"https://ftp.apnic.net/apnic/whois/apnic.db.route.gz", > - "https://ftp.apnic.net/apnic/whois/apnic.db.aut-num.gz", > - "https://ftp.apnic.net/apnic/whois/apnic.db.organisation.gz", > + "APNIC": [ > + "https://ftp.apnic.net/apnic/whois/apnic.db.inet6num.gz", > + "https://ftp.apnic.net/apnic/whois/apnic.db.inetnum.gz", > + #"https://ftp.apnic.net/apnic/whois/apnic.db.route6.gz", > + #"https://ftp.apnic.net/apnic/whois/apnic.db.route.gz", > + "https://ftp.apnic.net/apnic/whois/apnic.db.aut-num.gz", > + "https://ftp.apnic.net/apnic/whois/apnic.db.organisation.gz" > + ], > > # American Registry for Internet Numbers > # XXX there is nothing useful for us in here > - #"https://ftp.arin.net/pub/rr/arin.db", > + # ARIN: [ > + # "https://ftp.arin.net/pub/rr/arin.db" > + # ], > > # Latin America and Caribbean Network Information Centre > # XXX ??? > > # Réseaux IP Européens > - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inet6num.gz", > - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inetnum.gz", > - #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route6.gz", > - #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route.gz", > - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.aut-num.gz", > - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.organisation.gz", > -) > - > -EXTENDED_SOURCES = ( > + "RIPE": [ > + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inet6num.gz", > + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inetnum.gz", > + #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route6.gz", > + #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route.gz", > + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.aut-num.gz", > + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.organisation.gz" > + ], > +} > + > +EXTENDED_SOURCES = { > # African Network Information Centre > - #"https://ftp.afrinic.net/pub/stats/afrinic/delegated-afrinic-extended-latest", > + # "ARIN": [ > + # "https://ftp.afrinic.net/pub/stats/afrinic/delegated-afrinic-extended-latest" > + # ], > > # Asia Pacific Network Information Centre > - #"https://ftp.apnic.net/apnic/stats/apnic/delegated-apnic-extended-latest", > + # "APNIC": [ > + # "https://ftp.apnic.net/apnic/stats/apnic/delegated-apnic-extended-latest" > + # ], > > # American Registry for Internet Numbers > - "https://ftp.arin.net/pub/stats/arin/delegated-arin-extended-latest", > + "ARIN": [ > + "https://ftp.arin.net/pub/stats/arin/delegated-arin-extended-latest" > + ], > > # Latin America and Caribbean Network Information Centre > - "https://ftp.lacnic.net/pub/stats/lacnic/delegated-lacnic-extended-latest", > + "LACNIC": [ > + "https://ftp.lacnic.net/pub/stats/lacnic/delegated-lacnic-extended-latest" > + ], > > # Réseaux IP Européens > - #"https://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest", > -) > + # "RIPE": [ > + # "https://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest" > + # ], > +} > > class Downloader(object): > def __init__(self): > -- > 2.20.1 >
Hello Michael, thanks for your reply. I am not sure if I understood you correctly. What am I missing in which patch(set)? Thanks, and best regards, Peter Müller > Hallo, > > This probably should have been in the first patch because it won’t work without this. > > Best, > -Michael > >> On 22 May 2021, at 21:33, Peter Müller <peter.mueller@ipfire.org> wrote: >> >> Signed-off-by: Peter Müller <peter.mueller@ipfire.org> >> --- >> src/python/importer.py | 66 +++++++++++++++++++++++++++--------------- >> 1 file changed, 42 insertions(+), 24 deletions(-) >> >> diff --git a/src/python/importer.py b/src/python/importer.py >> index 5f46bc3..4c8406c 100644 >> --- a/src/python/importer.py >> +++ b/src/python/importer.py >> @@ -25,50 +25,68 @@ import urllib.request >> log = logging.getLogger("location.importer") >> log.propagate = 1 >> >> -WHOIS_SOURCES = ( >> +WHOIS_SOURCES = { >> # African Network Information Centre >> - "https://ftp.afrinic.net/pub/pub/dbase/afrinic.db.gz", >> + "AFRINIC": [ >> + "https://ftp.afrinic.net/pub/pub/dbase/afrinic.db.gz" >> + ], >> >> # Asia Pacific Network Information Centre >> - "https://ftp.apnic.net/apnic/whois/apnic.db.inet6num.gz", >> - "https://ftp.apnic.net/apnic/whois/apnic.db.inetnum.gz", >> - #"https://ftp.apnic.net/apnic/whois/apnic.db.route6.gz", >> - #"https://ftp.apnic.net/apnic/whois/apnic.db.route.gz", >> - "https://ftp.apnic.net/apnic/whois/apnic.db.aut-num.gz", >> - "https://ftp.apnic.net/apnic/whois/apnic.db.organisation.gz", >> + "APNIC": [ >> + "https://ftp.apnic.net/apnic/whois/apnic.db.inet6num.gz", >> + "https://ftp.apnic.net/apnic/whois/apnic.db.inetnum.gz", >> + #"https://ftp.apnic.net/apnic/whois/apnic.db.route6.gz", >> + #"https://ftp.apnic.net/apnic/whois/apnic.db.route.gz", >> + "https://ftp.apnic.net/apnic/whois/apnic.db.aut-num.gz", >> + "https://ftp.apnic.net/apnic/whois/apnic.db.organisation.gz" >> + ], >> >> # American Registry for Internet Numbers >> # XXX there is nothing useful for us in here >> - #"https://ftp.arin.net/pub/rr/arin.db", >> + # ARIN: [ >> + # "https://ftp.arin.net/pub/rr/arin.db" >> + # ], >> >> # Latin America and Caribbean Network Information Centre >> # XXX ??? >> >> # Réseaux IP Européens >> - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inet6num.gz", >> - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inetnum.gz", >> - #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route6.gz", >> - #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route.gz", >> - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.aut-num.gz", >> - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.organisation.gz", >> -) >> - >> -EXTENDED_SOURCES = ( >> + "RIPE": [ >> + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inet6num.gz", >> + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inetnum.gz", >> + #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route6.gz", >> + #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route.gz", >> + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.aut-num.gz", >> + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.organisation.gz" >> + ], >> +} >> + >> +EXTENDED_SOURCES = { >> # African Network Information Centre >> - #"https://ftp.afrinic.net/pub/stats/afrinic/delegated-afrinic-extended-latest", >> + # "ARIN": [ >> + # "https://ftp.afrinic.net/pub/stats/afrinic/delegated-afrinic-extended-latest" >> + # ], >> >> # Asia Pacific Network Information Centre >> - #"https://ftp.apnic.net/apnic/stats/apnic/delegated-apnic-extended-latest", >> + # "APNIC": [ >> + # "https://ftp.apnic.net/apnic/stats/apnic/delegated-apnic-extended-latest" >> + # ], >> >> # American Registry for Internet Numbers >> - "https://ftp.arin.net/pub/stats/arin/delegated-arin-extended-latest", >> + "ARIN": [ >> + "https://ftp.arin.net/pub/stats/arin/delegated-arin-extended-latest" >> + ], >> >> # Latin America and Caribbean Network Information Centre >> - "https://ftp.lacnic.net/pub/stats/lacnic/delegated-lacnic-extended-latest", >> + "LACNIC": [ >> + "https://ftp.lacnic.net/pub/stats/lacnic/delegated-lacnic-extended-latest" >> + ], >> >> # Réseaux IP Européens >> - #"https://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest", >> -) >> + # "RIPE": [ >> + # "https://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest" >> + # ], >> +} >> >> class Downloader(object): >> def __init__(self): >> -- >> 2.20.1 >> >
Nothing. It is merged. I am just saying that you split one change into two, but merging only one patch breaks the code - and that shouldn’t happen. -Michael > On 26 May 2021, at 19:11, Peter Müller <peter.mueller@ipfire.org> wrote: > > Hello Michael, > > thanks for your reply. > > I am not sure if I understood you correctly. What am I missing in which patch(set)? > > Thanks, and best regards, > Peter Müller > > >> Hallo, >> >> This probably should have been in the first patch because it won’t work without this. >> >> Best, >> -Michael >> >>> On 22 May 2021, at 21:33, Peter Müller <peter.mueller@ipfire.org> wrote: >>> >>> Signed-off-by: Peter Müller <peter.mueller@ipfire.org> >>> --- >>> src/python/importer.py | 66 +++++++++++++++++++++++++++--------------- >>> 1 file changed, 42 insertions(+), 24 deletions(-) >>> >>> diff --git a/src/python/importer.py b/src/python/importer.py >>> index 5f46bc3..4c8406c 100644 >>> --- a/src/python/importer.py >>> +++ b/src/python/importer.py >>> @@ -25,50 +25,68 @@ import urllib.request >>> log = logging.getLogger("location.importer") >>> log.propagate = 1 >>> >>> -WHOIS_SOURCES = ( >>> +WHOIS_SOURCES = { >>> # African Network Information Centre >>> - "https://ftp.afrinic.net/pub/pub/dbase/afrinic.db.gz", >>> + "AFRINIC": [ >>> + "https://ftp.afrinic.net/pub/pub/dbase/afrinic.db.gz" >>> + ], >>> >>> # Asia Pacific Network Information Centre >>> - "https://ftp.apnic.net/apnic/whois/apnic.db.inet6num.gz", >>> - "https://ftp.apnic.net/apnic/whois/apnic.db.inetnum.gz", >>> - #"https://ftp.apnic.net/apnic/whois/apnic.db.route6.gz", >>> - #"https://ftp.apnic.net/apnic/whois/apnic.db.route.gz", >>> - "https://ftp.apnic.net/apnic/whois/apnic.db.aut-num.gz", >>> - "https://ftp.apnic.net/apnic/whois/apnic.db.organisation.gz", >>> + "APNIC": [ >>> + "https://ftp.apnic.net/apnic/whois/apnic.db.inet6num.gz", >>> + "https://ftp.apnic.net/apnic/whois/apnic.db.inetnum.gz", >>> + #"https://ftp.apnic.net/apnic/whois/apnic.db.route6.gz", >>> + #"https://ftp.apnic.net/apnic/whois/apnic.db.route.gz", >>> + "https://ftp.apnic.net/apnic/whois/apnic.db.aut-num.gz", >>> + "https://ftp.apnic.net/apnic/whois/apnic.db.organisation.gz" >>> + ], >>> >>> # American Registry for Internet Numbers >>> # XXX there is nothing useful for us in here >>> - #"https://ftp.arin.net/pub/rr/arin.db", >>> + # ARIN: [ >>> + # "https://ftp.arin.net/pub/rr/arin.db" >>> + # ], >>> >>> # Latin America and Caribbean Network Information Centre >>> # XXX ??? >>> >>> # Réseaux IP Européens >>> - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inet6num.gz", >>> - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inetnum.gz", >>> - #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route6.gz", >>> - #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route.gz", >>> - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.aut-num.gz", >>> - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.organisation.gz", >>> -) >>> - >>> -EXTENDED_SOURCES = ( >>> + "RIPE": [ >>> + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inet6num.gz", >>> + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inetnum.gz", >>> + #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route6.gz", >>> + #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route.gz", >>> + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.aut-num.gz", >>> + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.organisation.gz" >>> + ], >>> +} >>> + >>> +EXTENDED_SOURCES = { >>> # African Network Information Centre >>> - #"https://ftp.afrinic.net/pub/stats/afrinic/delegated-afrinic-extended-latest", >>> + # "ARIN": [ >>> + # "https://ftp.afrinic.net/pub/stats/afrinic/delegated-afrinic-extended-latest" >>> + # ], >>> >>> # Asia Pacific Network Information Centre >>> - #"https://ftp.apnic.net/apnic/stats/apnic/delegated-apnic-extended-latest", >>> + # "APNIC": [ >>> + # "https://ftp.apnic.net/apnic/stats/apnic/delegated-apnic-extended-latest" >>> + # ], >>> >>> # American Registry for Internet Numbers >>> - "https://ftp.arin.net/pub/stats/arin/delegated-arin-extended-latest", >>> + "ARIN": [ >>> + "https://ftp.arin.net/pub/stats/arin/delegated-arin-extended-latest" >>> + ], >>> >>> # Latin America and Caribbean Network Information Centre >>> - "https://ftp.lacnic.net/pub/stats/lacnic/delegated-lacnic-extended-latest", >>> + "LACNIC": [ >>> + "https://ftp.lacnic.net/pub/stats/lacnic/delegated-lacnic-extended-latest" >>> + ], >>> >>> # Réseaux IP Européens >>> - #"https://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest", >>> -) >>> + # "RIPE": [ >>> + # "https://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest" >>> + # ], >>> +} >>> >>> class Downloader(object): >>> def __init__(self): >>> -- >>> 2.20.1 >>> >>
Hello Michael, thanks for your reply. Actually, I have never looked at patchsets being able to work in part - to me, they always seemed to be atomic. Good to know this is wrong, I guess... :-) Thanks, and best regards, Peter Müller > Nothing. It is merged. > > I am just saying that you split one change into two, but merging only one patch breaks the code - and that shouldn’t happen. > > -Michael > >> On 26 May 2021, at 19:11, Peter Müller <peter.mueller@ipfire.org> wrote: >> >> Hello Michael, >> >> thanks for your reply. >> >> I am not sure if I understood you correctly. What am I missing in which patch(set)? >> >> Thanks, and best regards, >> Peter Müller >> >> >>> Hallo, >>> >>> This probably should have been in the first patch because it won’t work without this. >>> >>> Best, >>> -Michael >>> >>>> On 22 May 2021, at 21:33, Peter Müller <peter.mueller@ipfire.org> wrote: >>>> >>>> Signed-off-by: Peter Müller <peter.mueller@ipfire.org> >>>> --- >>>> src/python/importer.py | 66 +++++++++++++++++++++++++++--------------- >>>> 1 file changed, 42 insertions(+), 24 deletions(-) >>>> >>>> diff --git a/src/python/importer.py b/src/python/importer.py >>>> index 5f46bc3..4c8406c 100644 >>>> --- a/src/python/importer.py >>>> +++ b/src/python/importer.py >>>> @@ -25,50 +25,68 @@ import urllib.request >>>> log = logging.getLogger("location.importer") >>>> log.propagate = 1 >>>> >>>> -WHOIS_SOURCES = ( >>>> +WHOIS_SOURCES = { >>>> # African Network Information Centre >>>> - "https://ftp.afrinic.net/pub/pub/dbase/afrinic.db.gz", >>>> + "AFRINIC": [ >>>> + "https://ftp.afrinic.net/pub/pub/dbase/afrinic.db.gz" >>>> + ], >>>> >>>> # Asia Pacific Network Information Centre >>>> - "https://ftp.apnic.net/apnic/whois/apnic.db.inet6num.gz", >>>> - "https://ftp.apnic.net/apnic/whois/apnic.db.inetnum.gz", >>>> - #"https://ftp.apnic.net/apnic/whois/apnic.db.route6.gz", >>>> - #"https://ftp.apnic.net/apnic/whois/apnic.db.route.gz", >>>> - "https://ftp.apnic.net/apnic/whois/apnic.db.aut-num.gz", >>>> - "https://ftp.apnic.net/apnic/whois/apnic.db.organisation.gz", >>>> + "APNIC": [ >>>> + "https://ftp.apnic.net/apnic/whois/apnic.db.inet6num.gz", >>>> + "https://ftp.apnic.net/apnic/whois/apnic.db.inetnum.gz", >>>> + #"https://ftp.apnic.net/apnic/whois/apnic.db.route6.gz", >>>> + #"https://ftp.apnic.net/apnic/whois/apnic.db.route.gz", >>>> + "https://ftp.apnic.net/apnic/whois/apnic.db.aut-num.gz", >>>> + "https://ftp.apnic.net/apnic/whois/apnic.db.organisation.gz" >>>> + ], >>>> >>>> # American Registry for Internet Numbers >>>> # XXX there is nothing useful for us in here >>>> - #"https://ftp.arin.net/pub/rr/arin.db", >>>> + # ARIN: [ >>>> + # "https://ftp.arin.net/pub/rr/arin.db" >>>> + # ], >>>> >>>> # Latin America and Caribbean Network Information Centre >>>> # XXX ??? >>>> >>>> # Réseaux IP Européens >>>> - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inet6num.gz", >>>> - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inetnum.gz", >>>> - #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route6.gz", >>>> - #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route.gz", >>>> - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.aut-num.gz", >>>> - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.organisation.gz", >>>> -) >>>> - >>>> -EXTENDED_SOURCES = ( >>>> + "RIPE": [ >>>> + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inet6num.gz", >>>> + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inetnum.gz", >>>> + #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route6.gz", >>>> + #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route.gz", >>>> + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.aut-num.gz", >>>> + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.organisation.gz" >>>> + ], >>>> +} >>>> + >>>> +EXTENDED_SOURCES = { >>>> # African Network Information Centre >>>> - #"https://ftp.afrinic.net/pub/stats/afrinic/delegated-afrinic-extended-latest", >>>> + # "ARIN": [ >>>> + # "https://ftp.afrinic.net/pub/stats/afrinic/delegated-afrinic-extended-latest" >>>> + # ], >>>> >>>> # Asia Pacific Network Information Centre >>>> - #"https://ftp.apnic.net/apnic/stats/apnic/delegated-apnic-extended-latest", >>>> + # "APNIC": [ >>>> + # "https://ftp.apnic.net/apnic/stats/apnic/delegated-apnic-extended-latest" >>>> + # ], >>>> >>>> # American Registry for Internet Numbers >>>> - "https://ftp.arin.net/pub/stats/arin/delegated-arin-extended-latest", >>>> + "ARIN": [ >>>> + "https://ftp.arin.net/pub/stats/arin/delegated-arin-extended-latest" >>>> + ], >>>> >>>> # Latin America and Caribbean Network Information Centre >>>> - "https://ftp.lacnic.net/pub/stats/lacnic/delegated-lacnic-extended-latest", >>>> + "LACNIC": [ >>>> + "https://ftp.lacnic.net/pub/stats/lacnic/delegated-lacnic-extended-latest" >>>> + ], >>>> >>>> # Réseaux IP Européens >>>> - #"https://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest", >>>> -) >>>> + # "RIPE": [ >>>> + # "https://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest" >>>> + # ], >>>> +} >>>> >>>> class Downloader(object): >>>> def __init__(self): >>>> -- >>>> 2.20.1 >>>> >>> >
Hello, > On 30 May 2021, at 09:08, Peter Müller <peter.mueller@ipfire.org> wrote: > > Hello Michael, > > thanks for your reply. > > Actually, I have never looked at patchsets being able to work in part - to me, they always seemed to be atomic. > Good to know this is wrong, I guess... :-) Sometimes it is difficult to do this, but it would be good to have them at least compile and run so that “git bisect” can be used. -Michael > Thanks, and best regards, > Peter Müller > >> Nothing. It is merged. >> >> I am just saying that you split one change into two, but merging only one patch breaks the code - and that shouldn’t happen. >> >> -Michael >> >>> On 26 May 2021, at 19:11, Peter Müller <peter.mueller@ipfire.org> wrote: >>> >>> Hello Michael, >>> >>> thanks for your reply. >>> >>> I am not sure if I understood you correctly. What am I missing in which patch(set)? >>> >>> Thanks, and best regards, >>> Peter Müller >>> >>> >>>> Hallo, >>>> >>>> This probably should have been in the first patch because it won’t work without this. >>>> >>>> Best, >>>> -Michael >>>> >>>>> On 22 May 2021, at 21:33, Peter Müller <peter.mueller@ipfire.org> wrote: >>>>> >>>>> Signed-off-by: Peter Müller <peter.mueller@ipfire.org> >>>>> --- >>>>> src/python/importer.py | 66 +++++++++++++++++++++++++++--------------- >>>>> 1 file changed, 42 insertions(+), 24 deletions(-) >>>>> >>>>> diff --git a/src/python/importer.py b/src/python/importer.py >>>>> index 5f46bc3..4c8406c 100644 >>>>> --- a/src/python/importer.py >>>>> +++ b/src/python/importer.py >>>>> @@ -25,50 +25,68 @@ import urllib.request >>>>> log = logging.getLogger("location.importer") >>>>> log.propagate = 1 >>>>> >>>>> -WHOIS_SOURCES = ( >>>>> +WHOIS_SOURCES = { >>>>> # African Network Information Centre >>>>> - "https://ftp.afrinic.net/pub/pub/dbase/afrinic.db.gz", >>>>> + "AFRINIC": [ >>>>> + "https://ftp.afrinic.net/pub/pub/dbase/afrinic.db.gz" >>>>> + ], >>>>> >>>>> # Asia Pacific Network Information Centre >>>>> - "https://ftp.apnic.net/apnic/whois/apnic.db.inet6num.gz", >>>>> - "https://ftp.apnic.net/apnic/whois/apnic.db.inetnum.gz", >>>>> - #"https://ftp.apnic.net/apnic/whois/apnic.db.route6.gz", >>>>> - #"https://ftp.apnic.net/apnic/whois/apnic.db.route.gz", >>>>> - "https://ftp.apnic.net/apnic/whois/apnic.db.aut-num.gz", >>>>> - "https://ftp.apnic.net/apnic/whois/apnic.db.organisation.gz", >>>>> + "APNIC": [ >>>>> + "https://ftp.apnic.net/apnic/whois/apnic.db.inet6num.gz", >>>>> + "https://ftp.apnic.net/apnic/whois/apnic.db.inetnum.gz", >>>>> + #"https://ftp.apnic.net/apnic/whois/apnic.db.route6.gz", >>>>> + #"https://ftp.apnic.net/apnic/whois/apnic.db.route.gz", >>>>> + "https://ftp.apnic.net/apnic/whois/apnic.db.aut-num.gz", >>>>> + "https://ftp.apnic.net/apnic/whois/apnic.db.organisation.gz" >>>>> + ], >>>>> >>>>> # American Registry for Internet Numbers >>>>> # XXX there is nothing useful for us in here >>>>> - #"https://ftp.arin.net/pub/rr/arin.db", >>>>> + # ARIN: [ >>>>> + # "https://ftp.arin.net/pub/rr/arin.db" >>>>> + # ], >>>>> >>>>> # Latin America and Caribbean Network Information Centre >>>>> # XXX ??? >>>>> >>>>> # Réseaux IP Européens >>>>> - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inet6num.gz", >>>>> - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inetnum.gz", >>>>> - #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route6.gz", >>>>> - #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route.gz", >>>>> - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.aut-num.gz", >>>>> - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.organisation.gz", >>>>> -) >>>>> - >>>>> -EXTENDED_SOURCES = ( >>>>> + "RIPE": [ >>>>> + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inet6num.gz", >>>>> + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inetnum.gz", >>>>> + #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route6.gz", >>>>> + #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route.gz", >>>>> + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.aut-num.gz", >>>>> + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.organisation.gz" >>>>> + ], >>>>> +} >>>>> + >>>>> +EXTENDED_SOURCES = { >>>>> # African Network Information Centre >>>>> - #"https://ftp.afrinic.net/pub/stats/afrinic/delegated-afrinic-extended-latest", >>>>> + # "ARIN": [ >>>>> + # "https://ftp.afrinic.net/pub/stats/afrinic/delegated-afrinic-extended-latest" >>>>> + # ], >>>>> >>>>> # Asia Pacific Network Information Centre >>>>> - #"https://ftp.apnic.net/apnic/stats/apnic/delegated-apnic-extended-latest", >>>>> + # "APNIC": [ >>>>> + # "https://ftp.apnic.net/apnic/stats/apnic/delegated-apnic-extended-latest" >>>>> + # ], >>>>> >>>>> # American Registry for Internet Numbers >>>>> - "https://ftp.arin.net/pub/stats/arin/delegated-arin-extended-latest", >>>>> + "ARIN": [ >>>>> + "https://ftp.arin.net/pub/stats/arin/delegated-arin-extended-latest" >>>>> + ], >>>>> >>>>> # Latin America and Caribbean Network Information Centre >>>>> - "https://ftp.lacnic.net/pub/stats/lacnic/delegated-lacnic-extended-latest", >>>>> + "LACNIC": [ >>>>> + "https://ftp.lacnic.net/pub/stats/lacnic/delegated-lacnic-extended-latest" >>>>> + ], >>>>> >>>>> # Réseaux IP Européens >>>>> - #"https://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest", >>>>> -) >>>>> + # "RIPE": [ >>>>> + # "https://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest" >>>>> + # ], >>>>> +} >>>>> >>>>> class Downloader(object): >>>>> def __init__(self): >>>>> -- >>>>> 2.20.1 >>>>> >>>> >>
diff --git a/src/python/importer.py b/src/python/importer.py index 5f46bc3..4c8406c 100644 --- a/src/python/importer.py +++ b/src/python/importer.py @@ -25,50 +25,68 @@ import urllib.request log = logging.getLogger("location.importer") log.propagate = 1 -WHOIS_SOURCES = ( +WHOIS_SOURCES = { # African Network Information Centre - "https://ftp.afrinic.net/pub/pub/dbase/afrinic.db.gz", + "AFRINIC": [ + "https://ftp.afrinic.net/pub/pub/dbase/afrinic.db.gz" + ], # Asia Pacific Network Information Centre - "https://ftp.apnic.net/apnic/whois/apnic.db.inet6num.gz", - "https://ftp.apnic.net/apnic/whois/apnic.db.inetnum.gz", - #"https://ftp.apnic.net/apnic/whois/apnic.db.route6.gz", - #"https://ftp.apnic.net/apnic/whois/apnic.db.route.gz", - "https://ftp.apnic.net/apnic/whois/apnic.db.aut-num.gz", - "https://ftp.apnic.net/apnic/whois/apnic.db.organisation.gz", + "APNIC": [ + "https://ftp.apnic.net/apnic/whois/apnic.db.inet6num.gz", + "https://ftp.apnic.net/apnic/whois/apnic.db.inetnum.gz", + #"https://ftp.apnic.net/apnic/whois/apnic.db.route6.gz", + #"https://ftp.apnic.net/apnic/whois/apnic.db.route.gz", + "https://ftp.apnic.net/apnic/whois/apnic.db.aut-num.gz", + "https://ftp.apnic.net/apnic/whois/apnic.db.organisation.gz" + ], # American Registry for Internet Numbers # XXX there is nothing useful for us in here - #"https://ftp.arin.net/pub/rr/arin.db", + # ARIN: [ + # "https://ftp.arin.net/pub/rr/arin.db" + # ], # Latin America and Caribbean Network Information Centre # XXX ??? # Réseaux IP Européens - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inet6num.gz", - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inetnum.gz", - #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route6.gz", - #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route.gz", - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.aut-num.gz", - "https://ftp.ripe.net/ripe/dbase/split/ripe.db.organisation.gz", -) - -EXTENDED_SOURCES = ( + "RIPE": [ + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inet6num.gz", + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.inetnum.gz", + #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route6.gz", + #"https://ftp.ripe.net/ripe/dbase/split/ripe.db.route.gz", + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.aut-num.gz", + "https://ftp.ripe.net/ripe/dbase/split/ripe.db.organisation.gz" + ], +} + +EXTENDED_SOURCES = { # African Network Information Centre - #"https://ftp.afrinic.net/pub/stats/afrinic/delegated-afrinic-extended-latest", + # "ARIN": [ + # "https://ftp.afrinic.net/pub/stats/afrinic/delegated-afrinic-extended-latest" + # ], # Asia Pacific Network Information Centre - #"https://ftp.apnic.net/apnic/stats/apnic/delegated-apnic-extended-latest", + # "APNIC": [ + # "https://ftp.apnic.net/apnic/stats/apnic/delegated-apnic-extended-latest" + # ], # American Registry for Internet Numbers - "https://ftp.arin.net/pub/stats/arin/delegated-arin-extended-latest", + "ARIN": [ + "https://ftp.arin.net/pub/stats/arin/delegated-arin-extended-latest" + ], # Latin America and Caribbean Network Information Centre - "https://ftp.lacnic.net/pub/stats/lacnic/delegated-lacnic-extended-latest", + "LACNIC": [ + "https://ftp.lacnic.net/pub/stats/lacnic/delegated-lacnic-extended-latest" + ], # Réseaux IP Européens - #"https://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest", -) + # "RIPE": [ + # "https://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest" + # ], +} class Downloader(object): def __init__(self):