location-importer.in: treat AQ and BV as invalid countries

Message ID dd49ba06-895b-6e0f-edee-462019f47b06@ipfire.org
State New
Headers show
Series
  • location-importer.in: treat AQ and BV as invalid countries
Related show

Commit Message

Peter Müller April 1, 2021, 7:57 p.m. UTC
Both the Bouvet Island (BV) and Antarctica (AQ) are unpopulated at the
time of writing. Network owners/operators putting these countries into
their RIR data objects are either completely braindead or doing so for
hostile reasons.

While we might correct these locations to something useful by manually
creating overrides for them, the rationale behind this patch is not to
let these countries appear on productive systems in the first place, as
we know they _cannot_ be true.

Therefore, this patch skips any network object that has either AQ or BV
country code set.

See also: https://lists.ipfire.org/pipermail/location/2020-October/000199.html

Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
---
 src/python/location-importer.in | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Comments

Michael Tremer April 7, 2021, 8:49 p.m. UTC | #1
Hello,

* would we not want this to be configurable in countries.txt?

* The list should probably be a tuple.

-Michael

> On 1 Apr 2021, at 20:57, Peter Müller <peter.mueller@ipfire.org> wrote:
> 
> Both the Bouvet Island (BV) and Antarctica (AQ) are unpopulated at the
> time of writing. Network owners/operators putting these countries into
> their RIR data objects are either completely braindead or doing so for
> hostile reasons.
> 
> While we might correct these locations to something useful by manually
> creating overrides for them, the rationale behind this patch is not to
> let these countries appear on productive systems in the first place, as
> we know they _cannot_ be true.
> 
> Therefore, this patch skips any network object that has either AQ or BV
> country code set.
> 
> See also: https://lists.ipfire.org/pipermail/location/2020-October/000199.html
> 
> Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
> ---
> src/python/location-importer.in | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/src/python/location-importer.in b/src/python/location-importer.in
> index 1e08458..ac678dc 100644
> --- a/src/python/location-importer.in
> +++ b/src/python/location-importer.in
> @@ -624,7 +624,7 @@ class CLI(object):
> 			return
> 
> 		# Skip objects with unknown country codes
> -		if validcountries and inetnum.get("country") not in validcountries:
> +		if validcountries and (inetnum.get("country") not in validcountries or inetnum.get("country") in ["AQ", "BV"]):
> 			log.warning("Skipping network with bogus country '%s': %s" % \
> 				(inetnum.get("country"), inetnum.get("inet6num") or inetnum.get("inetnum")))
> 			return
> -- 
> 2.26.2
Peter Müller April 10, 2021, 12:32 p.m. UTC | #2
Hello Michael,

thanks for your reply.

Yes, having this configurable in countries.txt would be nice indeed. Do you propose a certain syntax for this?

And yes, a tuple is a better idea here. I will wait for your reply and submit a second version of this patch then.

Thanks, and best regards,
Peter Müller


> Hello,
> 
> * would we not want this to be configurable in countries.txt?
> 
> * The list should probably be a tuple.
> 
> -Michael
> 
>> On 1 Apr 2021, at 20:57, Peter Müller <peter.mueller@ipfire.org> wrote:
>>
>> Both the Bouvet Island (BV) and Antarctica (AQ) are unpopulated at the
>> time of writing. Network owners/operators putting these countries into
>> their RIR data objects are either completely braindead or doing so for
>> hostile reasons.
>>
>> While we might correct these locations to something useful by manually
>> creating overrides for them, the rationale behind this patch is not to
>> let these countries appear on productive systems in the first place, as
>> we know they _cannot_ be true.
>>
>> Therefore, this patch skips any network object that has either AQ or BV
>> country code set.
>>
>> See also: https://lists.ipfire.org/pipermail/location/2020-October/000199.html
>>
>> Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
>> ---
>> src/python/location-importer.in | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/src/python/location-importer.in b/src/python/location-importer.in
>> index 1e08458..ac678dc 100644
>> --- a/src/python/location-importer.in
>> +++ b/src/python/location-importer.in
>> @@ -624,7 +624,7 @@ class CLI(object):
>> 			return
>>
>> 		# Skip objects with unknown country codes
>> -		if validcountries and inetnum.get("country") not in validcountries:
>> +		if validcountries and (inetnum.get("country") not in validcountries or inetnum.get("country") in ["AQ", "BV"]):
>> 			log.warning("Skipping network with bogus country '%s': %s" % \
>> 				(inetnum.get("country"), inetnum.get("inet6num") or inetnum.get("inetnum")))
>> 			return
>> -- 
>> 2.26.2
>
Michael Tremer April 12, 2021, 9:21 a.m. UTC | #3
Hello,

How about we add the country to the list and mark the continent with a “-“:

  AF      AS      Afghanistan
  YU      -       Yugoslavia

-Michael

> On 10 Apr 2021, at 13:32, Peter Müller <peter.mueller@ipfire.org> wrote:
> 
> Hello Michael,
> 
> thanks for your reply.
> 
> Yes, having this configurable in countries.txt would be nice indeed. Do you propose a certain syntax for this?
> 
> And yes, a tuple is a better idea here. I will wait for your reply and submit a second version of this patch then.
> 
> Thanks, and best regards,
> Peter Müller
> 
> 
>> Hello,
>> 
>> * would we not want this to be configurable in countries.txt?
>> 
>> * The list should probably be a tuple.
>> 
>> -Michael
>> 
>>> On 1 Apr 2021, at 20:57, Peter Müller <peter.mueller@ipfire.org> wrote:
>>> 
>>> Both the Bouvet Island (BV) and Antarctica (AQ) are unpopulated at the
>>> time of writing. Network owners/operators putting these countries into
>>> their RIR data objects are either completely braindead or doing so for
>>> hostile reasons.
>>> 
>>> While we might correct these locations to something useful by manually
>>> creating overrides for them, the rationale behind this patch is not to
>>> let these countries appear on productive systems in the first place, as
>>> we know they _cannot_ be true.
>>> 
>>> Therefore, this patch skips any network object that has either AQ or BV
>>> country code set.
>>> 
>>> See also: https://lists.ipfire.org/pipermail/location/2020-October/000199.html
>>> 
>>> Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
>>> ---
>>> src/python/location-importer.in | 2 +-
>>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>> 
>>> diff --git a/src/python/location-importer.in b/src/python/location-importer.in
>>> index 1e08458..ac678dc 100644
>>> --- a/src/python/location-importer.in
>>> +++ b/src/python/location-importer.in
>>> @@ -624,7 +624,7 @@ class CLI(object):
>>> 			return
>>> 
>>> 		# Skip objects with unknown country codes
>>> -		if validcountries and inetnum.get("country") not in validcountries:
>>> +		if validcountries and (inetnum.get("country") not in validcountries or inetnum.get("country") in ["AQ", "BV"]):
>>> 			log.warning("Skipping network with bogus country '%s': %s" % \
>>> 				(inetnum.get("country"), inetnum.get("inet6num") or inetnum.get("inetnum")))
>>> 			return
>>> -- 
>>> 2.26.2
>>
Peter Müller April 12, 2021, 5:26 p.m. UTC | #4
Hello Michael,

thanks for your reply.

I believe you are mixing up two distinct scenarios here:

(a) Historic ISO-3166-X abbreviations for countries no longer existing anymore, such as YU.

(b) Country codes for existing nations, where we believe with a high level of confidence no
    IP networks are located in any way, either physically or legally. AQ and BV are the most
    notable examples known to me at the time of writing.

Since it does not make sense to add historic country codes, I would suggest to leave networks
assigned to (a) countries as they are, manually creating overrides for them, as I did before.

For (b), simply scrubbing out the country or region they belong to seems to be too harsh to
me, as we use this information for several things, and should not abuse it for our own purposes.
Therefore, I rather suggest adding an additional column to countries.txt for indicating whether
this country code is acceptable to us or not.

Thanks, and best regards,
Peter Müller

> Hello,
> 
> How about we add the country to the list and mark the continent with a “-“:
> 
>   AF      AS      Afghanistan
>   YU      -       Yugoslavia
> 
> -Michael
> 
>> On 10 Apr 2021, at 13:32, Peter Müller <peter.mueller@ipfire.org> wrote:
>>
>> Hello Michael,
>>
>> thanks for your reply.
>>
>> Yes, having this configurable in countries.txt would be nice indeed. Do you propose a certain syntax for this?
>>
>> And yes, a tuple is a better idea here. I will wait for your reply and submit a second version of this patch then.
>>
>> Thanks, and best regards,
>> Peter Müller
>>
>>
>>> Hello,
>>>
>>> * would we not want this to be configurable in countries.txt?
>>>
>>> * The list should probably be a tuple.
>>>
>>> -Michael
>>>
>>>> On 1 Apr 2021, at 20:57, Peter Müller <peter.mueller@ipfire.org> wrote:
>>>>
>>>> Both the Bouvet Island (BV) and Antarctica (AQ) are unpopulated at the
>>>> time of writing. Network owners/operators putting these countries into
>>>> their RIR data objects are either completely braindead or doing so for
>>>> hostile reasons.
>>>>
>>>> While we might correct these locations to something useful by manually
>>>> creating overrides for them, the rationale behind this patch is not to
>>>> let these countries appear on productive systems in the first place, as
>>>> we know they _cannot_ be true.
>>>>
>>>> Therefore, this patch skips any network object that has either AQ or BV
>>>> country code set.
>>>>
>>>> See also: https://lists.ipfire.org/pipermail/location/2020-October/000199.html
>>>>
>>>> Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
>>>> ---
>>>> src/python/location-importer.in | 2 +-
>>>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>>>
>>>> diff --git a/src/python/location-importer.in b/src/python/location-importer.in
>>>> index 1e08458..ac678dc 100644
>>>> --- a/src/python/location-importer.in
>>>> +++ b/src/python/location-importer.in
>>>> @@ -624,7 +624,7 @@ class CLI(object):
>>>> 			return
>>>>
>>>> 		# Skip objects with unknown country codes
>>>> -		if validcountries and inetnum.get("country") not in validcountries:
>>>> +		if validcountries and (inetnum.get("country") not in validcountries or inetnum.get("country") in ["AQ", "BV"]):
>>>> 			log.warning("Skipping network with bogus country '%s': %s" % \
>>>> 				(inetnum.get("country"), inetnum.get("inet6num") or inetnum.get("inetnum")))
>>>> 			return
>>>> -- 
>>>> 2.26.2
>>>
>
Michael Tremer April 14, 2021, 8:58 a.m. UTC | #5
Hi,

> On 12 Apr 2021, at 18:26, Peter Müller <peter.mueller@ipfire.org> wrote:
> 
> Hello Michael,
> 
> thanks for your reply.
> 
> I believe you are mixing up two distinct scenarios here:

I am trying not to. I just added the name because I struggle to remember some country codes.

> (a) Historic ISO-3166-X abbreviations for countries no longer existing anymore, such as YU.

They technically do.

They are still reserved in the ISO 3166 database and they show up (in this case) in the RIPE database. So we will have to handle them no matter what.

YU does not have a succeeding national body (like Eastern Germany would have the Federal Republic of Germany) so we cannot automatically rewrite them.

> (b) Country codes for existing nations, where we believe with a high level of confidence no
>    IP networks are located in any way, either physically or legally. AQ and BV are the most
>    notable examples known to me at the time of writing.
> 
> Since it does not make sense to add historic country codes, I would suggest to leave networks
> assigned to (a) countries as they are, manually creating overrides for them, as I did before.
> 
> For (b), simply scrubbing out the country or region they belong to seems to be too harsh to
> me, as we use this information for several things, and should not abuse it for our own purposes.

(I assume) we still have AS information about those networks and we should keep at least that.

> Therefore, I rather suggest adding an additional column to countries.txt for indicating whether
> this country code is acceptable to us or not.

Okay. What would you suggest?

-Michael

> 
> Thanks, and best regards,
> Peter Müller
> 
>> Hello,
>> 
>> How about we add the country to the list and mark the continent with a “-“:
>> 
>>  AF      AS      Afghanistan
>>  YU      -       Yugoslavia
>> 
>> -Michael
>> 
>>> On 10 Apr 2021, at 13:32, Peter Müller <peter.mueller@ipfire.org> wrote:
>>> 
>>> Hello Michael,
>>> 
>>> thanks for your reply.
>>> 
>>> Yes, having this configurable in countries.txt would be nice indeed. Do you propose a certain syntax for this?
>>> 
>>> And yes, a tuple is a better idea here. I will wait for your reply and submit a second version of this patch then.
>>> 
>>> Thanks, and best regards,
>>> Peter Müller
>>> 
>>> 
>>>> Hello,
>>>> 
>>>> * would we not want this to be configurable in countries.txt?
>>>> 
>>>> * The list should probably be a tuple.
>>>> 
>>>> -Michael
>>>> 
>>>>> On 1 Apr 2021, at 20:57, Peter Müller <peter.mueller@ipfire.org> wrote:
>>>>> 
>>>>> Both the Bouvet Island (BV) and Antarctica (AQ) are unpopulated at the
>>>>> time of writing. Network owners/operators putting these countries into
>>>>> their RIR data objects are either completely braindead or doing so for
>>>>> hostile reasons.
>>>>> 
>>>>> While we might correct these locations to something useful by manually
>>>>> creating overrides for them, the rationale behind this patch is not to
>>>>> let these countries appear on productive systems in the first place, as
>>>>> we know they _cannot_ be true.
>>>>> 
>>>>> Therefore, this patch skips any network object that has either AQ or BV
>>>>> country code set.
>>>>> 
>>>>> See also: https://lists.ipfire.org/pipermail/location/2020-October/000199.html
>>>>> 
>>>>> Signed-off-by: Peter Müller <peter.mueller@ipfire.org>
>>>>> ---
>>>>> src/python/location-importer.in | 2 +-
>>>>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>>>> 
>>>>> diff --git a/src/python/location-importer.in b/src/python/location-importer.in
>>>>> index 1e08458..ac678dc 100644
>>>>> --- a/src/python/location-importer.in
>>>>> +++ b/src/python/location-importer.in
>>>>> @@ -624,7 +624,7 @@ class CLI(object):
>>>>> 			return
>>>>> 
>>>>> 		# Skip objects with unknown country codes
>>>>> -		if validcountries and inetnum.get("country") not in validcountries:
>>>>> +		if validcountries and (inetnum.get("country") not in validcountries or inetnum.get("country") in ["AQ", "BV"]):
>>>>> 			log.warning("Skipping network with bogus country '%s': %s" % \
>>>>> 				(inetnum.get("country"), inetnum.get("inet6num") or inetnum.get("inetnum")))
>>>>> 			return
>>>>> -- 
>>>>> 2.26.2
>>>> 
>>

Patch

diff --git a/src/python/location-importer.in b/src/python/location-importer.in
index 1e08458..ac678dc 100644
--- a/src/python/location-importer.in
+++ b/src/python/location-importer.in
@@ -624,7 +624,7 @@  class CLI(object):
 			return
 
 		# Skip objects with unknown country codes
-		if validcountries and inetnum.get("country") not in validcountries:
+		if validcountries and (inetnum.get("country") not in validcountries or inetnum.get("country") in ["AQ", "BV"]):
 			log.warning("Skipping network with bogus country '%s': %s" % \
 				(inetnum.get("country"), inetnum.get("inet6num") or inetnum.get("inetnum")))
 			return