| Message ID | 20240801184539.2536658-1-jon.murphy@ipfire.org |
|---|---|
| State | New |
| Headers |
Return-Path: <development-bounces@lists.ipfire.org> Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mail01.haj.ipfire.org", Issuer "R11" (verified OK)) by web04.haj.ipfire.org (Postfix) with ESMTPS id 4WZdFx5Pnsz3wwH for <patchwork@web04.haj.ipfire.org>; Thu, 1 Aug 2024 18:45:57 +0000 (UTC) Received: from mail02.haj.ipfire.org (mail02.haj.ipfire.org [172.28.1.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) client-signature ECDSA (secp384r1)) (Client CN "mail02.haj.ipfire.org", Issuer "E5" (verified OK)) by mail01.ipfire.org (Postfix) with ESMTPS id 4WZdFt5CxJz1vr; Thu, 1 Aug 2024 18:45:54 +0000 (UTC) Received: from mail02.haj.ipfire.org (localhost [127.0.0.1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4WZdFt2mc3z34Gd; Thu, 1 Aug 2024 18:45:54 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mail01.haj.ipfire.org", Issuer "R11" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4WZdFp6Pp2z34DG for <development@lists.ipfire.org>; Thu, 1 Aug 2024 18:45:50 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4WZdFn5TY4z1tL; Thu, 1 Aug 2024 18:45:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1722537950; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=92ePKaKa6h/g39gzjSvKJF6tJNPTFMt0eIalM001GP0=; b=Ou3ZdwOB3S85TUlDCe0KzUHr03Ze3cPY+fKFLN/w3REPVnoyUx5kV61xwigOYddKYBh1hd zQca21MrTICi3WGUM/MixxKSAmSzHws5Q2WneKKFMVtcTYL49psCvVuKLmNNgCOYdZlE6K 5Y56Qwn06UScYy8lN44WZtN/RCw4tCF8PCig3VwZ1UYNl2r+kLCMXGFmeCZ7pOlVLpdw0I w5Fjn3HhFpLHYzsHXuPA3KPjO65OGc4GPZtbH4XsMluKiYW52F/9mciCO5JB5qBlL03cQd /78FxCv5tdMQHOAEgU47laNnSYjNrtQhqdympsDBJTRls9sgHyVIgEEjewyM0A== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1722537950; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=92ePKaKa6h/g39gzjSvKJF6tJNPTFMt0eIalM001GP0=; b=TxPmEs20h4klUuawqnVORi99SEltNDPuxP0ByUSs5mx0qwclRpfjJMy21YC0wBTjdS7F5/ +AXbHITaO4RA+6AA== From: Jon Murphy <jon.murphy@ipfire.org> To: development@lists.ipfire.org Subject: [PATCH] RPZ: bug fix and code update Date: Thu, 1 Aug 2024 13:45:39 -0500 Message-Id: <20240801184539.2536658-1-jon.murphy@ipfire.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Message-ID-Hash: RYLATCETVPQU46MPJPG2CLTMB67BI3HG X-Message-ID-Hash: RYLATCETVPQU46MPJPG2CLTMB67BI3HG X-MailFrom: jon.murphy@ipfire.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.8 Precedence: list List-Id: IPFire development talk <development.lists.ipfire.org> Archived-At: <https://lists.ipfire.org/hyperkitty/list/development@lists.ipfire.org/message/RYLATCETVPQU46MPJPG2CLTMB67BI3HG/> List-Archive: <https://lists.ipfire.org/hyperkitty/list/development@lists.ipfire.org/> List-Help: <mailto:development-request@lists.ipfire.org?subject=help> List-Owner: <mailto:development-owner@lists.ipfire.org> List-Post: <mailto:development@lists.ipfire.org> List-Subscribe: <mailto:development-join@lists.ipfire.org> List-Unsubscribe: <mailto:development-leave@lists.ipfire.org> |
| Series |
RPZ: bug fix and code update
|
|
Commit Message
Jon Murphy
Aug. 1, 2024, 6:45 p.m. UTC
changed all paths from `/var/ipfire/rpz/` to `/var/ipfire/dns/rpz/`
(thank you to Adolf!)
rpz-config:
- bug: corrected "Type" test from block to allow
- removed verbose parameter from various commands
rpz-metrics:
- bug: corrected grep for rpz name count
- bug: fixed divide by zero error (thank you Peppe!)
install/uninstall:
- bug: corrected scripts (thank you Bernhard!)
Signed-off-by: Jon Murphy <jon.murphy@ipfire.org>
---
config/backup/includes/rpz | 4 ++--
config/rootfiles/packages/rpz | 6 +++---
config/rpz/rpz-config | 14 +++++++-------
config/rpz/rpz-metrics | 9 +++++----
lfs/rpz | 6 +++---
src/paks/rpz/install.sh | 27 +++++++++++++++++++++++++++
src/paks/rpz/uninstall.sh | 31 +++++++++++++++++++++++++++++++
src/paks/rpz/update.sh | 25 +++++++++++++++++++++++++
8 files changed, 103 insertions(+), 19 deletions(-)
create mode 100644 src/paks/rpz/install.sh
create mode 100644 src/paks/rpz/uninstall.sh
create mode 100644 src/paks/rpz/update.sh
Comments
Adolf, I made the suggested changes (see email below). >> Wouldn't this make more sense to create a log page for unbound-rpz that shows the results in a similar way to the Firewall Logs, IP Address Blocklist Logs etc. > > Yes, I can do this. I did not change the anything related to logging (yet). This will be a future update. >> I think checkconf should be added into the unboundctrl.c file >> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/misc-progs/unboundctrl.c;h=86c6ac42b9010f9d77fa49754bf217875aed3cc0;hb=refs/heads/next >> so that the command can be ensured to be running in safe mode. > > Yes, this can be changed. I do not have a good understanding of the `*ctrl.c` programs so if this is needed, I would need assistance in adding it to c. And this was not changed (yet). I’ll need some assistance. Jon > On Aug 1, 2024, at 1:45 PM, Jon Murphy <jon.murphy@ipfire.org> wrote: > > changed all paths from `/var/ipfire/rpz/` to `/var/ipfire/dns/rpz/` > (thank you to Adolf!) > > rpz-config: > - bug: corrected "Type" test from block to allow > - removed verbose parameter from various commands > > rpz-metrics: > - bug: corrected grep for rpz name count > - bug: fixed divide by zero error (thank you Peppe!) > > install/uninstall: > - bug: corrected scripts (thank you Bernhard!) > > Signed-off-by: Jon Murphy <jon.murphy@ipfire.org> > --- > config/backup/includes/rpz | 4 ++-- > config/rootfiles/packages/rpz | 6 +++--- > config/rpz/rpz-config | 14 +++++++------- > config/rpz/rpz-metrics | 9 +++++---- > lfs/rpz | 6 +++--- > src/paks/rpz/install.sh | 27 +++++++++++++++++++++++++++ > src/paks/rpz/uninstall.sh | 31 +++++++++++++++++++++++++++++++ > src/paks/rpz/update.sh | 25 +++++++++++++++++++++++++ > 8 files changed, 103 insertions(+), 19 deletions(-) > create mode 100644 src/paks/rpz/install.sh > create mode 100644 src/paks/rpz/uninstall.sh > create mode 100644 src/paks/rpz/update.sh > > diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz > index 4d59bb40c..8c7410ebd 100644 > --- a/config/backup/includes/rpz > +++ b/config/backup/includes/rpz > @@ -1,5 +1,5 @@ > -/var/ipfire/rpz/allowlist > -/var/ipfire/rpz/blocklist > +/var/ipfire/dns/rpz/allowlist > +/var/ipfire/dns/rpz/blocklist > /etc/unbound/zonefiles/allow.rpz > /etc/unbound/zonefiles/block.rpz > /etc/unbound/local.d/*rpz.conf > diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/rpz > index 2ffa715dd..183825362 100644 > --- a/config/rootfiles/packages/rpz > +++ b/config/rootfiles/packages/rpz > @@ -6,6 +6,6 @@ usr/sbin/rpz-config > usr/sbin/rpz-metrics > usr/sbin/rpz-sleep > var/ipfire/backup/addons/includes/rpz > -var/ipfire/rpz > -var/ipfire/rpz/allowlist > -var/ipfire/rpz/blocklist > +var/ipfire/dns/rpz > +var/ipfire/dns/rpz/allowlist > +var/ipfire/dns/rpz/blocklist > diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config > index 98dc0a4ca..a24a5c132 100644 > --- a/config/rpz/rpz-config > +++ b/config/rpz/rpz-config > @@ -19,7 +19,7 @@ > # # > ############################################################################### > > -# v22 - 2024-07-12 > +# v23 - 2024-07-30 > > ############### Functions ############### > > @@ -54,11 +54,11 @@ check_unbound_conf () { > make_rpz_file () { > local theType="${1}" # allow or block > > - theList="/var/ipfire/rpz/${theType}list" # input user list of domains > + theList="/var/ipfire/dns/rpz/${theType}list" # input custom list of domains > theZoneFile="/etc/unbound/zonefiles/${theType}.rpz" # output file for RPZ > > theAction='.' > - if [[ "${theType}" =~ "block" ]] ; then > + if [[ "${theType}" =~ "allow" ]] ; then > theAction='rpz-passthru.' > fi > > @@ -131,8 +131,8 @@ case "${theAction}" in > # set-up zone file > /usr/bin/touch "${rpzFile}" > # unbound requires these settings for rpz files > - /bin/chown --verbose nobody:nobody "${rpzFile}" > - /bin/chmod --verbose 644 "${rpzFile}" > + /bin/chown nobody:nobody "${rpzFile}" > + /bin/chmod 644 "${rpzFile}" > ;; > > # trash config file & rpz file > @@ -143,8 +143,8 @@ case "${theAction}" in > fi > > msg_log "info: rpz: remove config file & rpz file \"${theName}\"" > - /bin/rm --verbose "${rpzConfig}" > - /bin/rm --verbose "${rpzFile}" > + /bin/rm "${rpzConfig}" > + /bin/rm "${rpzFile}" > > check_unbound_conf > ;; > diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics > index 0f97c7911..4d932726e 100644 > --- a/config/rpz/rpz-metrics > +++ b/config/rpz/rpz-metrics > @@ -19,7 +19,7 @@ > # # > ############################################################################### > > -# v18 on 2024-07-05 > +# v19 on 2024-07-30 > > ############### Main ############### > > @@ -33,7 +33,7 @@ messageLogs=$( find /var/log/messages* -type f | > > # get the list of RPZ names & counts from the message log(s) > rpzNameCount=$( for logf in ${messageLogs} ; do > - /usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" | > + /usr/bin/zgrep --text --extended-regexp 'info: rpz: applied.* A IN$' "${logf}" | > /usr/bin/awk '$10 ~ /\[\w*]/ { print $10 }' ; > done | /usr/bin/sort | /usr/bin/uniq --count ) > > @@ -107,8 +107,9 @@ do > theLines=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' ) > totalLines=$(( totalLines + theLines )) > > - #hitsPerLine=$( echo "scale=0 ; $theHits / $theLines" | bc ) > - hitsPerLine=$(( 100 * theHits / theLines )) > + if [[ "${theLines}" -gt 2 ]] ; then > + hitsPerLine=$(( 100 * theHits / theLines )) > + fi > fi > > # get modification date > diff --git a/lfs/rpz b/lfs/rpz > index 319c10b7f..73f6f2b1b 100644 > --- a/lfs/rpz > +++ b/lfs/rpz > @@ -67,9 +67,9 @@ $(TARGET) : > $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin > > # Install settings folder and two empty files > - mkdir -pv /var/ipfire/rpz > - touch /var/ipfire/rpz/allowlist > - touch /var/ipfire/rpz/blocklist > + mkdir -pv /var/ipfire/dns/rpz > + touch /var/ipfire/dns/rpz/allowlist > + touch /var/ipfire/dns/rpz/blocklist > > # Add conf file to /etc directory > cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d > diff --git a/src/paks/rpz/install.sh b/src/paks/rpz/install.sh > new file mode 100644 > index 000000000..0a797e158 > --- /dev/null > +++ b/src/paks/rpz/install.sh > @@ -0,0 +1,27 @@ > +#!/bin/bash > +############################################################################### > +# # > +# IPFire.org - A linux based firewall # > +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # > +# # > +# This program is free software: you can redistribute it and/or modify # > +# it under the terms of the GNU General Public License as published by # > +# the Free Software Foundation, either version 3 of the License, or # > +# (at your option) any later version. # > +# # > +# This program is distributed in the hope that it will be useful, # > +# but WITHOUT ANY WARRANTY; without even the implied warranty of # > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # > +# GNU General Public License for more details. # > +# # > +# You should have received a copy of the GNU General Public License # > +# along with this program. If not, see <http://www.gnu.org/licenses/>. # > +# # > +############################################################################### > +# > +. /opt/pakfire/lib/functions.sh > +extract_files > +restore_backup ${NAME} > + > +# restart unbound to load config file > +/etc/init.d/unbound restart > diff --git a/src/paks/rpz/uninstall.sh b/src/paks/rpz/uninstall.sh > new file mode 100644 > index 000000000..4fb20e127 > --- /dev/null > +++ b/src/paks/rpz/uninstall.sh > @@ -0,0 +1,31 @@ > +#!/bin/bash > +############################################################################### > +# # > +# IPFire.org - A linux based firewall # > +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # > +# # > +# This program is free software: you can redistribute it and/or modify # > +# it under the terms of the GNU General Public License as published by # > +# the Free Software Foundation, either version 3 of the License, or # > +# (at your option) any later version. # > +# # > +# This program is distributed in the hope that it will be useful, # > +# but WITHOUT ANY WARRANTY; without even the implied warranty of # > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # > +# GNU General Public License for more details. # > +# # > +# You should have received a copy of the GNU General Public License # > +# along with this program. If not, see <http://www.gnu.org/licenses/>. # > +# # > +############################################################################### > +# > +. /opt/pakfire/lib/functions.sh > + > +# stop unbound to delete RPZ conf file > +/etc/init.d/unbound stop > + > +make_backup ${NAME} > +remove_files > + > +# start unbound to load unbound config file > +/etc/init.d/unbound start > diff --git a/src/paks/rpz/update.sh b/src/paks/rpz/update.sh > new file mode 100644 > index 000000000..938a93a40 > --- /dev/null > +++ b/src/paks/rpz/update.sh > @@ -0,0 +1,25 @@ > +#!/bin/bash > +############################################################################### > +# # > +# IPFire.org - A linux based firewall # > +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # > +# # > +# This program is free software: you can redistribute it and/or modify # > +# it under the terms of the GNU General Public License as published by # > +# the Free Software Foundation, either version 3 of the License, or # > +# (at your option) any later version. # > +# # > +# This program is distributed in the hope that it will be useful, # > +# but WITHOUT ANY WARRANTY; without even the implied warranty of # > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # > +# GNU General Public License for more details. # > +# # > +# You should have received a copy of the GNU General Public License # > +# along with this program. If not, see <http://www.gnu.org/licenses/>. # > +# # > +############################################################################### > +# > +. /opt/pakfire/lib/functions.sh > +extract_backup_includes > +./uninstall.sh > +./install.sh > -- > 2.30.2 >
Reviewed-by: Bernhard Bitsch <bbitsch@ipfire.org> Tested-by: Bernhard Bitsch <bbitsch@ipfire.org> Am 01.08.2024 um 20:45 schrieb Jon Murphy: > changed all paths from `/var/ipfire/rpz/` to `/var/ipfire/dns/rpz/` > (thank you to Adolf!) > > rpz-config: > - bug: corrected "Type" test from block to allow > - removed verbose parameter from various commands > > rpz-metrics: > - bug: corrected grep for rpz name count > - bug: fixed divide by zero error (thank you Peppe!) > > install/uninstall: > - bug: corrected scripts (thank you Bernhard!) > > Signed-off-by: Jon Murphy <jon.murphy@ipfire.org> > --- > config/backup/includes/rpz | 4 ++-- > config/rootfiles/packages/rpz | 6 +++--- > config/rpz/rpz-config | 14 +++++++------- > config/rpz/rpz-metrics | 9 +++++---- > lfs/rpz | 6 +++--- > src/paks/rpz/install.sh | 27 +++++++++++++++++++++++++++ > src/paks/rpz/uninstall.sh | 31 +++++++++++++++++++++++++++++++ > src/paks/rpz/update.sh | 25 +++++++++++++++++++++++++ > 8 files changed, 103 insertions(+), 19 deletions(-) > create mode 100644 src/paks/rpz/install.sh > create mode 100644 src/paks/rpz/uninstall.sh > create mode 100644 src/paks/rpz/update.sh > > diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz > index 4d59bb40c..8c7410ebd 100644 > --- a/config/backup/includes/rpz > +++ b/config/backup/includes/rpz > @@ -1,5 +1,5 @@ > -/var/ipfire/rpz/allowlist > -/var/ipfire/rpz/blocklist > +/var/ipfire/dns/rpz/allowlist > +/var/ipfire/dns/rpz/blocklist > /etc/unbound/zonefiles/allow.rpz > /etc/unbound/zonefiles/block.rpz > /etc/unbound/local.d/*rpz.conf > diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/rpz > index 2ffa715dd..183825362 100644 > --- a/config/rootfiles/packages/rpz > +++ b/config/rootfiles/packages/rpz > @@ -6,6 +6,6 @@ usr/sbin/rpz-config > usr/sbin/rpz-metrics > usr/sbin/rpz-sleep > var/ipfire/backup/addons/includes/rpz > -var/ipfire/rpz > -var/ipfire/rpz/allowlist > -var/ipfire/rpz/blocklist > +var/ipfire/dns/rpz > +var/ipfire/dns/rpz/allowlist > +var/ipfire/dns/rpz/blocklist > diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config > index 98dc0a4ca..a24a5c132 100644 > --- a/config/rpz/rpz-config > +++ b/config/rpz/rpz-config > @@ -19,7 +19,7 @@ > # # > ############################################################################### > > -# v22 - 2024-07-12 > +# v23 - 2024-07-30 > > ############### Functions ############### > > @@ -54,11 +54,11 @@ check_unbound_conf () { > make_rpz_file () { > local theType="${1}" # allow or block > > - theList="/var/ipfire/rpz/${theType}list" # input user list of domains > + theList="/var/ipfire/dns/rpz/${theType}list" # input custom list of domains > theZoneFile="/etc/unbound/zonefiles/${theType}.rpz" # output file for RPZ > > theAction='.' > - if [[ "${theType}" =~ "block" ]] ; then > + if [[ "${theType}" =~ "allow" ]] ; then > theAction='rpz-passthru.' > fi > > @@ -131,8 +131,8 @@ case "${theAction}" in > # set-up zone file > /usr/bin/touch "${rpzFile}" > # unbound requires these settings for rpz files > - /bin/chown --verbose nobody:nobody "${rpzFile}" > - /bin/chmod --verbose 644 "${rpzFile}" > + /bin/chown nobody:nobody "${rpzFile}" > + /bin/chmod 644 "${rpzFile}" > ;; > > # trash config file & rpz file > @@ -143,8 +143,8 @@ case "${theAction}" in > fi > > msg_log "info: rpz: remove config file & rpz file \"${theName}\"" > - /bin/rm --verbose "${rpzConfig}" > - /bin/rm --verbose "${rpzFile}" > + /bin/rm "${rpzConfig}" > + /bin/rm "${rpzFile}" > > check_unbound_conf > ;; > diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics > index 0f97c7911..4d932726e 100644 > --- a/config/rpz/rpz-metrics > +++ b/config/rpz/rpz-metrics > @@ -19,7 +19,7 @@ > # # > ############################################################################### > > -# v18 on 2024-07-05 > +# v19 on 2024-07-30 > > ############### Main ############### > > @@ -33,7 +33,7 @@ messageLogs=$( find /var/log/messages* -type f | > > # get the list of RPZ names & counts from the message log(s) > rpzNameCount=$( for logf in ${messageLogs} ; do > - /usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" | > + /usr/bin/zgrep --text --extended-regexp 'info: rpz: applied.* A IN$' "${logf}" | > /usr/bin/awk '$10 ~ /\[\w*]/ { print $10 }' ; > done | /usr/bin/sort | /usr/bin/uniq --count ) > > @@ -107,8 +107,9 @@ do > theLines=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' ) > totalLines=$(( totalLines + theLines )) > > - #hitsPerLine=$( echo "scale=0 ; $theHits / $theLines" | bc ) > - hitsPerLine=$(( 100 * theHits / theLines )) > + if [[ "${theLines}" -gt 2 ]] ; then > + hitsPerLine=$(( 100 * theHits / theLines )) > + fi > fi > > # get modification date > diff --git a/lfs/rpz b/lfs/rpz > index 319c10b7f..73f6f2b1b 100644 > --- a/lfs/rpz > +++ b/lfs/rpz > @@ -67,9 +67,9 @@ $(TARGET) : > $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin > > # Install settings folder and two empty files > - mkdir -pv /var/ipfire/rpz > - touch /var/ipfire/rpz/allowlist > - touch /var/ipfire/rpz/blocklist > + mkdir -pv /var/ipfire/dns/rpz > + touch /var/ipfire/dns/rpz/allowlist > + touch /var/ipfire/dns/rpz/blocklist > > # Add conf file to /etc directory > cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d > diff --git a/src/paks/rpz/install.sh b/src/paks/rpz/install.sh > new file mode 100644 > index 000000000..0a797e158 > --- /dev/null > +++ b/src/paks/rpz/install.sh > @@ -0,0 +1,27 @@ > +#!/bin/bash > +############################################################################### > +# # > +# IPFire.org - A linux based firewall # > +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # > +# # > +# This program is free software: you can redistribute it and/or modify # > +# it under the terms of the GNU General Public License as published by # > +# the Free Software Foundation, either version 3 of the License, or # > +# (at your option) any later version. # > +# # > +# This program is distributed in the hope that it will be useful, # > +# but WITHOUT ANY WARRANTY; without even the implied warranty of # > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # > +# GNU General Public License for more details. # > +# # > +# You should have received a copy of the GNU General Public License # > +# along with this program. If not, see <http://www.gnu.org/licenses/>. # > +# # > +############################################################################### > +# > +. /opt/pakfire/lib/functions.sh > +extract_files > +restore_backup ${NAME} > + > +# restart unbound to load config file > +/etc/init.d/unbound restart > diff --git a/src/paks/rpz/uninstall.sh b/src/paks/rpz/uninstall.sh > new file mode 100644 > index 000000000..4fb20e127 > --- /dev/null > +++ b/src/paks/rpz/uninstall.sh > @@ -0,0 +1,31 @@ > +#!/bin/bash > +############################################################################### > +# # > +# IPFire.org - A linux based firewall # > +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # > +# # > +# This program is free software: you can redistribute it and/or modify # > +# it under the terms of the GNU General Public License as published by # > +# the Free Software Foundation, either version 3 of the License, or # > +# (at your option) any later version. # > +# # > +# This program is distributed in the hope that it will be useful, # > +# but WITHOUT ANY WARRANTY; without even the implied warranty of # > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # > +# GNU General Public License for more details. # > +# # > +# You should have received a copy of the GNU General Public License # > +# along with this program. If not, see <http://www.gnu.org/licenses/>. # > +# # > +############################################################################### > +# > +. /opt/pakfire/lib/functions.sh > + > +# stop unbound to delete RPZ conf file > +/etc/init.d/unbound stop > + > +make_backup ${NAME} > +remove_files > + > +# start unbound to load unbound config file > +/etc/init.d/unbound start > diff --git a/src/paks/rpz/update.sh b/src/paks/rpz/update.sh > new file mode 100644 > index 000000000..938a93a40 > --- /dev/null > +++ b/src/paks/rpz/update.sh > @@ -0,0 +1,25 @@ > +#!/bin/bash > +############################################################################### > +# # > +# IPFire.org - A linux based firewall # > +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # > +# # > +# This program is free software: you can redistribute it and/or modify # > +# it under the terms of the GNU General Public License as published by # > +# the Free Software Foundation, either version 3 of the License, or # > +# (at your option) any later version. # > +# # > +# This program is distributed in the hope that it will be useful, # > +# but WITHOUT ANY WARRANTY; without even the implied warranty of # > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # > +# GNU General Public License for more details. # > +# # > +# You should have received a copy of the GNU General Public License # > +# along with this program. If not, see <http://www.gnu.org/licenses/>. # > +# # > +############################################################################### > +# > +. /opt/pakfire/lib/functions.sh > +extract_backup_includes > +./uninstall.sh > +./install.sh
More questions! Currently RPZ config files are at `/etc/unbound/local.d` but this directory seems like it is for user (admin) customizations. ``` [root@ipfire ~] # ls -al /etc/unbound/local.d total 68 drwxr-xr-x 2 nobody nobody 4096 Aug 12 13:41 . drwxr-xr-x 4 root root 4096 Aug 12 00:52 .. -rw-r--r-- 1 nobody nobody 436 Jul 12 15:45 00-rpz.conf -rw-r--r-- 1 nobody nobody 285 Mar 1 22:12 AmazonTrkrHZ.rpz.conf -rw-r--r-- 1 nobody nobody 281 Mar 1 22:02 AppleTrkrHZ.rpz.conf -rw-r--r-- 1 nobody nobody 269 Mar 1 21:40 DOHblockHZ.rpz.conf ... -rw-r--r-- 1 nobody nobody 299 Aug 1 19:42 WinTrkrHZ.rpz.conf [root@ipfire ~] # ``` Each file is a config file per category (or one per RPZ file). This makes it easy to add or remove a category (or RPZ file). Should I create a new unbound directory for RPZ config files? Maybe `/etc/unbound/rpz.d`? Or `/etc/unbound/rpz`? Jon > On Aug 1, 2024, at 1:45 PM, Jon Murphy <jon.murphy@ipfire.org> wrote: > > changed all paths from `/var/ipfire/rpz/` to `/var/ipfire/dns/rpz/` > (thank you to Adolf!) > > rpz-config: > - bug: corrected "Type" test from block to allow > - removed verbose parameter from various commands > > rpz-metrics: > - bug: corrected grep for rpz name count > - bug: fixed divide by zero error (thank you Peppe!) > > install/uninstall: > - bug: corrected scripts (thank you Bernhard!) > > Signed-off-by: Jon Murphy <jon.murphy@ipfire.org> > --- > config/backup/includes/rpz | 4 ++-- > config/rootfiles/packages/rpz | 6 +++--- > config/rpz/rpz-config | 14 +++++++------- > config/rpz/rpz-metrics | 9 +++++---- > lfs/rpz | 6 +++--- > src/paks/rpz/install.sh | 27 +++++++++++++++++++++++++++ > src/paks/rpz/uninstall.sh | 31 +++++++++++++++++++++++++++++++ > src/paks/rpz/update.sh | 25 +++++++++++++++++++++++++ > 8 files changed, 103 insertions(+), 19 deletions(-) > create mode 100644 src/paks/rpz/install.sh > create mode 100644 src/paks/rpz/uninstall.sh > create mode 100644 src/paks/rpz/update.sh > > diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz > index 4d59bb40c..8c7410ebd 100644 > --- a/config/backup/includes/rpz > +++ b/config/backup/includes/rpz > @@ -1,5 +1,5 @@ > -/var/ipfire/rpz/allowlist > -/var/ipfire/rpz/blocklist > +/var/ipfire/dns/rpz/allowlist > +/var/ipfire/dns/rpz/blocklist > /etc/unbound/zonefiles/allow.rpz > /etc/unbound/zonefiles/block.rpz > /etc/unbound/local.d/*rpz.conf > diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/rpz > index 2ffa715dd..183825362 100644 > --- a/config/rootfiles/packages/rpz > +++ b/config/rootfiles/packages/rpz > @@ -6,6 +6,6 @@ usr/sbin/rpz-config > usr/sbin/rpz-metrics > usr/sbin/rpz-sleep > var/ipfire/backup/addons/includes/rpz > -var/ipfire/rpz > -var/ipfire/rpz/allowlist > -var/ipfire/rpz/blocklist > +var/ipfire/dns/rpz > +var/ipfire/dns/rpz/allowlist > +var/ipfire/dns/rpz/blocklist > diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config > index 98dc0a4ca..a24a5c132 100644 > --- a/config/rpz/rpz-config > +++ b/config/rpz/rpz-config > @@ -19,7 +19,7 @@ > # # > ############################################################################### > > -# v22 - 2024-07-12 > +# v23 - 2024-07-30 > > ############### Functions ############### > > @@ -54,11 +54,11 @@ check_unbound_conf () { > make_rpz_file () { > local theType="${1}" # allow or block > > - theList="/var/ipfire/rpz/${theType}list" # input user list of domains > + theList="/var/ipfire/dns/rpz/${theType}list" # input custom list of domains > theZoneFile="/etc/unbound/zonefiles/${theType}.rpz" # output file for RPZ > > theAction='.' > - if [[ "${theType}" =~ "block" ]] ; then > + if [[ "${theType}" =~ "allow" ]] ; then > theAction='rpz-passthru.' > fi > > @@ -131,8 +131,8 @@ case "${theAction}" in > # set-up zone file > /usr/bin/touch "${rpzFile}" > # unbound requires these settings for rpz files > - /bin/chown --verbose nobody:nobody "${rpzFile}" > - /bin/chmod --verbose 644 "${rpzFile}" > + /bin/chown nobody:nobody "${rpzFile}" > + /bin/chmod 644 "${rpzFile}" > ;; > > # trash config file & rpz file > @@ -143,8 +143,8 @@ case "${theAction}" in > fi > > msg_log "info: rpz: remove config file & rpz file \"${theName}\"" > - /bin/rm --verbose "${rpzConfig}" > - /bin/rm --verbose "${rpzFile}" > + /bin/rm "${rpzConfig}" > + /bin/rm "${rpzFile}" > > check_unbound_conf > ;; > diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics > index 0f97c7911..4d932726e 100644 > --- a/config/rpz/rpz-metrics > +++ b/config/rpz/rpz-metrics > @@ -19,7 +19,7 @@ > # # > ############################################################################### > > -# v18 on 2024-07-05 > +# v19 on 2024-07-30 > > ############### Main ############### > > @@ -33,7 +33,7 @@ messageLogs=$( find /var/log/messages* -type f | > > # get the list of RPZ names & counts from the message log(s) > rpzNameCount=$( for logf in ${messageLogs} ; do > - /usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" | > + /usr/bin/zgrep --text --extended-regexp 'info: rpz: applied.* A IN$' "${logf}" | > /usr/bin/awk '$10 ~ /\[\w*]/ { print $10 }' ; > done | /usr/bin/sort | /usr/bin/uniq --count ) > > @@ -107,8 +107,9 @@ do > theLines=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' ) > totalLines=$(( totalLines + theLines )) > > - #hitsPerLine=$( echo "scale=0 ; $theHits / $theLines" | bc ) > - hitsPerLine=$(( 100 * theHits / theLines )) > + if [[ "${theLines}" -gt 2 ]] ; then > + hitsPerLine=$(( 100 * theHits / theLines )) > + fi > fi > > # get modification date > diff --git a/lfs/rpz b/lfs/rpz > index 319c10b7f..73f6f2b1b 100644 > --- a/lfs/rpz > +++ b/lfs/rpz > @@ -67,9 +67,9 @@ $(TARGET) : > $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin > > # Install settings folder and two empty files > - mkdir -pv /var/ipfire/rpz > - touch /var/ipfire/rpz/allowlist > - touch /var/ipfire/rpz/blocklist > + mkdir -pv /var/ipfire/dns/rpz > + touch /var/ipfire/dns/rpz/allowlist > + touch /var/ipfire/dns/rpz/blocklist > > # Add conf file to /etc directory > cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d > diff --git a/src/paks/rpz/install.sh b/src/paks/rpz/install.sh > new file mode 100644 > index 000000000..0a797e158 > --- /dev/null > +++ b/src/paks/rpz/install.sh > @@ -0,0 +1,27 @@ > +#!/bin/bash > +############################################################################### > +# # > +# IPFire.org - A linux based firewall # > +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # > +# # > +# This program is free software: you can redistribute it and/or modify # > +# it under the terms of the GNU General Public License as published by # > +# the Free Software Foundation, either version 3 of the License, or # > +# (at your option) any later version. # > +# # > +# This program is distributed in the hope that it will be useful, # > +# but WITHOUT ANY WARRANTY; without even the implied warranty of # > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # > +# GNU General Public License for more details. # > +# # > +# You should have received a copy of the GNU General Public License # > +# along with this program. If not, see <http://www.gnu.org/licenses/>. # > +# # > +############################################################################### > +# > +. /opt/pakfire/lib/functions.sh > +extract_files > +restore_backup ${NAME} > + > +# restart unbound to load config file > +/etc/init.d/unbound restart > diff --git a/src/paks/rpz/uninstall.sh b/src/paks/rpz/uninstall.sh > new file mode 100644 > index 000000000..4fb20e127 > --- /dev/null > +++ b/src/paks/rpz/uninstall.sh > @@ -0,0 +1,31 @@ > +#!/bin/bash > +############################################################################### > +# # > +# IPFire.org - A linux based firewall # > +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # > +# # > +# This program is free software: you can redistribute it and/or modify # > +# it under the terms of the GNU General Public License as published by # > +# the Free Software Foundation, either version 3 of the License, or # > +# (at your option) any later version. # > +# # > +# This program is distributed in the hope that it will be useful, # > +# but WITHOUT ANY WARRANTY; without even the implied warranty of # > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # > +# GNU General Public License for more details. # > +# # > +# You should have received a copy of the GNU General Public License # > +# along with this program. If not, see <http://www.gnu.org/licenses/>. # > +# # > +############################################################################### > +# > +. /opt/pakfire/lib/functions.sh > + > +# stop unbound to delete RPZ conf file > +/etc/init.d/unbound stop > + > +make_backup ${NAME} > +remove_files > + > +# start unbound to load unbound config file > +/etc/init.d/unbound start > diff --git a/src/paks/rpz/update.sh b/src/paks/rpz/update.sh > new file mode 100644 > index 000000000..938a93a40 > --- /dev/null > +++ b/src/paks/rpz/update.sh > @@ -0,0 +1,25 @@ > +#!/bin/bash > +############################################################################### > +# # > +# IPFire.org - A linux based firewall # > +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # > +# # > +# This program is free software: you can redistribute it and/or modify # > +# it under the terms of the GNU General Public License as published by # > +# the Free Software Foundation, either version 3 of the License, or # > +# (at your option) any later version. # > +# # > +# This program is distributed in the hope that it will be useful, # > +# but WITHOUT ANY WARRANTY; without even the implied warranty of # > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # > +# GNU General Public License for more details. # > +# # > +# You should have received a copy of the GNU General Public License # > +# along with this program. If not, see <http://www.gnu.org/licenses/>. # > +# # > +############################################################################### > +# > +. /opt/pakfire/lib/functions.sh > +extract_backup_includes > +./uninstall.sh > +./install.sh > -- > 2.30.2 >
Michael, Sorry for putting you on the spot, but what do you want to do with this RPZ add-on? I saw your comments in the Dev Mailing List of "generally being in favor of trying this path" (bad paraphrasing on my part) I saw your comments in bugzilla at https://bugzilla.ipfire.org/show_bug.cgi?id=13254#c171 > I am not interested in anything regarding the RPZs right now. They have not been properly put on the agenda and looking at how much time we have on our hands, this won't make it on the agenda for years. > > I don't want to build blockers, but this ticket is about a different problem which I want to solve first. How do you want to go forward? Jon > On Aug 12, 2024, at 2:11 PM, jon <jon.murphy@ipfire.org> wrote: > > More questions! > > Currently RPZ config files are at `/etc/unbound/local.d` but this directory seems like it is for user (admin) customizations. > > ``` > [root@ipfire ~] # ls -al /etc/unbound/local.d > total 68 > drwxr-xr-x 2 nobody nobody 4096 Aug 12 13:41 . > drwxr-xr-x 4 root root 4096 Aug 12 00:52 .. > -rw-r--r-- 1 nobody nobody 436 Jul 12 15:45 00-rpz.conf > -rw-r--r-- 1 nobody nobody 285 Mar 1 22:12 AmazonTrkrHZ.rpz.conf > -rw-r--r-- 1 nobody nobody 281 Mar 1 22:02 AppleTrkrHZ.rpz.conf > -rw-r--r-- 1 nobody nobody 269 Mar 1 21:40 DOHblockHZ.rpz.conf > ... > -rw-r--r-- 1 nobody nobody 299 Aug 1 19:42 WinTrkrHZ.rpz.conf > [root@ipfire ~] # > ``` > > > Each file is a config file per category (or one per RPZ file). This makes it easy to add or remove a category (or RPZ file). > > Should I create a new unbound directory for RPZ config files? Maybe `/etc/unbound/rpz.d`? Or `/etc/unbound/rpz`? > > > Jon > > >> On Aug 1, 2024, at 1:45 PM, Jon Murphy <jon.murphy@ipfire.org> wrote: >> >> changed all paths from `/var/ipfire/rpz/` to `/var/ipfire/dns/rpz/` >> (thank you to Adolf!) >> >> rpz-config: >> - bug: corrected "Type" test from block to allow >> - removed verbose parameter from various commands >> >> rpz-metrics: >> - bug: corrected grep for rpz name count >> - bug: fixed divide by zero error (thank you Peppe!) >> >> install/uninstall: >> - bug: corrected scripts (thank you Bernhard!) >> >> Signed-off-by: Jon Murphy <jon.murphy@ipfire.org> >> --- >> config/backup/includes/rpz | 4 ++-- >> config/rootfiles/packages/rpz | 6 +++--- >> config/rpz/rpz-config | 14 +++++++------- >> config/rpz/rpz-metrics | 9 +++++---- >> lfs/rpz | 6 +++--- >> src/paks/rpz/install.sh | 27 +++++++++++++++++++++++++++ >> src/paks/rpz/uninstall.sh | 31 +++++++++++++++++++++++++++++++ >> src/paks/rpz/update.sh | 25 +++++++++++++++++++++++++ >> 8 files changed, 103 insertions(+), 19 deletions(-) >> create mode 100644 src/paks/rpz/install.sh >> create mode 100644 src/paks/rpz/uninstall.sh >> create mode 100644 src/paks/rpz/update.sh >> >> diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz >> index 4d59bb40c..8c7410ebd 100644 >> --- a/config/backup/includes/rpz >> +++ b/config/backup/includes/rpz >> @@ -1,5 +1,5 @@ >> -/var/ipfire/rpz/allowlist >> -/var/ipfire/rpz/blocklist >> +/var/ipfire/dns/rpz/allowlist >> +/var/ipfire/dns/rpz/blocklist >> /etc/unbound/zonefiles/allow.rpz >> /etc/unbound/zonefiles/block.rpz >> /etc/unbound/local.d/*rpz.conf >> diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/rpz >> index 2ffa715dd..183825362 100644 >> --- a/config/rootfiles/packages/rpz >> +++ b/config/rootfiles/packages/rpz >> @@ -6,6 +6,6 @@ usr/sbin/rpz-config >> usr/sbin/rpz-metrics >> usr/sbin/rpz-sleep >> var/ipfire/backup/addons/includes/rpz >> -var/ipfire/rpz >> -var/ipfire/rpz/allowlist >> -var/ipfire/rpz/blocklist >> +var/ipfire/dns/rpz >> +var/ipfire/dns/rpz/allowlist >> +var/ipfire/dns/rpz/blocklist >> diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config >> index 98dc0a4ca..a24a5c132 100644 >> --- a/config/rpz/rpz-config >> +++ b/config/rpz/rpz-config >> @@ -19,7 +19,7 @@ >> # # >> ############################################################################### >> >> -# v22 - 2024-07-12 >> +# v23 - 2024-07-30 >> >> ############### Functions ############### >> >> @@ -54,11 +54,11 @@ check_unbound_conf () { >> make_rpz_file () { >> local theType="${1}" # allow or block >> >> - theList="/var/ipfire/rpz/${theType}list" # input user list of domains >> + theList="/var/ipfire/dns/rpz/${theType}list" # input custom list of domains >> theZoneFile="/etc/unbound/zonefiles/${theType}.rpz" # output file for RPZ >> >> theAction='.' >> - if [[ "${theType}" =~ "block" ]] ; then >> + if [[ "${theType}" =~ "allow" ]] ; then >> theAction='rpz-passthru.' >> fi >> >> @@ -131,8 +131,8 @@ case "${theAction}" in >> # set-up zone file >> /usr/bin/touch "${rpzFile}" >> # unbound requires these settings for rpz files >> - /bin/chown --verbose nobody:nobody "${rpzFile}" >> - /bin/chmod --verbose 644 "${rpzFile}" >> + /bin/chown nobody:nobody "${rpzFile}" >> + /bin/chmod 644 "${rpzFile}" >> ;; >> >> # trash config file & rpz file >> @@ -143,8 +143,8 @@ case "${theAction}" in >> fi >> >> msg_log "info: rpz: remove config file & rpz file \"${theName}\"" >> - /bin/rm --verbose "${rpzConfig}" >> - /bin/rm --verbose "${rpzFile}" >> + /bin/rm "${rpzConfig}" >> + /bin/rm "${rpzFile}" >> >> check_unbound_conf >> ;; >> diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics >> index 0f97c7911..4d932726e 100644 >> --- a/config/rpz/rpz-metrics >> +++ b/config/rpz/rpz-metrics >> @@ -19,7 +19,7 @@ >> # # >> ############################################################################### >> >> -# v18 on 2024-07-05 >> +# v19 on 2024-07-30 >> >> ############### Main ############### >> >> @@ -33,7 +33,7 @@ messageLogs=$( find /var/log/messages* -type f | >> >> # get the list of RPZ names & counts from the message log(s) >> rpzNameCount=$( for logf in ${messageLogs} ; do >> - /usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" | >> + /usr/bin/zgrep --text --extended-regexp 'info: rpz: applied.* A IN$' "${logf}" | >> /usr/bin/awk '$10 ~ /\[\w*]/ { print $10 }' ; >> done | /usr/bin/sort | /usr/bin/uniq --count ) >> >> @@ -107,8 +107,9 @@ do >> theLines=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' ) >> totalLines=$(( totalLines + theLines )) >> >> - #hitsPerLine=$( echo "scale=0 ; $theHits / $theLines" | bc ) >> - hitsPerLine=$(( 100 * theHits / theLines )) >> + if [[ "${theLines}" -gt 2 ]] ; then >> + hitsPerLine=$(( 100 * theHits / theLines )) >> + fi >> fi >> >> # get modification date >> diff --git a/lfs/rpz b/lfs/rpz >> index 319c10b7f..73f6f2b1b 100644 >> --- a/lfs/rpz >> +++ b/lfs/rpz >> @@ -67,9 +67,9 @@ $(TARGET) : >> $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin >> >> # Install settings folder and two empty files >> - mkdir -pv /var/ipfire/rpz >> - touch /var/ipfire/rpz/allowlist >> - touch /var/ipfire/rpz/blocklist >> + mkdir -pv /var/ipfire/dns/rpz >> + touch /var/ipfire/dns/rpz/allowlist >> + touch /var/ipfire/dns/rpz/blocklist >> >> # Add conf file to /etc directory >> cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d >> diff --git a/src/paks/rpz/install.sh b/src/paks/rpz/install.sh >> new file mode 100644 >> index 000000000..0a797e158 >> --- /dev/null >> +++ b/src/paks/rpz/install.sh >> @@ -0,0 +1,27 @@ >> +#!/bin/bash >> +############################################################################### >> +# # >> +# IPFire.org - A linux based firewall # >> +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # >> +# # >> +# This program is free software: you can redistribute it and/or modify # >> +# it under the terms of the GNU General Public License as published by # >> +# the Free Software Foundation, either version 3 of the License, or # >> +# (at your option) any later version. # >> +# # >> +# This program is distributed in the hope that it will be useful, # >> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >> +# GNU General Public License for more details. # >> +# # >> +# You should have received a copy of the GNU General Public License # >> +# along with this program. If not, see <http://www.gnu.org/licenses/>. # >> +# # >> +############################################################################### >> +# >> +. /opt/pakfire/lib/functions.sh >> +extract_files >> +restore_backup ${NAME} >> + >> +# restart unbound to load config file >> +/etc/init.d/unbound restart >> diff --git a/src/paks/rpz/uninstall.sh b/src/paks/rpz/uninstall.sh >> new file mode 100644 >> index 000000000..4fb20e127 >> --- /dev/null >> +++ b/src/paks/rpz/uninstall.sh >> @@ -0,0 +1,31 @@ >> +#!/bin/bash >> +############################################################################### >> +# # >> +# IPFire.org - A linux based firewall # >> +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # >> +# # >> +# This program is free software: you can redistribute it and/or modify # >> +# it under the terms of the GNU General Public License as published by # >> +# the Free Software Foundation, either version 3 of the License, or # >> +# (at your option) any later version. # >> +# # >> +# This program is distributed in the hope that it will be useful, # >> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >> +# GNU General Public License for more details. # >> +# # >> +# You should have received a copy of the GNU General Public License # >> +# along with this program. If not, see <http://www.gnu.org/licenses/>. # >> +# # >> +############################################################################### >> +# >> +. /opt/pakfire/lib/functions.sh >> + >> +# stop unbound to delete RPZ conf file >> +/etc/init.d/unbound stop >> + >> +make_backup ${NAME} >> +remove_files >> + >> +# start unbound to load unbound config file >> +/etc/init.d/unbound start >> diff --git a/src/paks/rpz/update.sh b/src/paks/rpz/update.sh >> new file mode 100644 >> index 000000000..938a93a40 >> --- /dev/null >> +++ b/src/paks/rpz/update.sh >> @@ -0,0 +1,25 @@ >> +#!/bin/bash >> +############################################################################### >> +# # >> +# IPFire.org - A linux based firewall # >> +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # >> +# # >> +# This program is free software: you can redistribute it and/or modify # >> +# it under the terms of the GNU General Public License as published by # >> +# the Free Software Foundation, either version 3 of the License, or # >> +# (at your option) any later version. # >> +# # >> +# This program is distributed in the hope that it will be useful, # >> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >> +# GNU General Public License for more details. # >> +# # >> +# You should have received a copy of the GNU General Public License # >> +# along with this program. If not, see <http://www.gnu.org/licenses/>. # >> +# # >> +############################################################################### >> +# >> +. /opt/pakfire/lib/functions.sh >> +extract_backup_includes >> +./uninstall.sh >> +./install.sh >> -- >> 2.30.2 >> >
Hello, > On 14 Aug 2024, at 18:14, jon <jon.murphy@ipfire.org> wrote: > > Michael, > > Sorry for putting you on the spot, but what do you want to do with this RPZ add-on? I am not sure this is a question for only me. I personally just don’t have any capacity to take on another rather large project as I have a huge backlog of so many things and I feel like a lonely fighter getting these all over the line. For my own sanity I need to have a couple of those closed before thinking about the next ones. A couple of months ago we have talked on the monthly call about RPZ and the decision that was made by the people was to not look at this now, just because there were other more pressing things. We could also not find answers to the questions that led us into RPZ: * Are there any good lists out there that would allow us to replace the URL Filter? That thing is basically on its last leg because of the lack of lists. We need to be able to block ads and pr0n and that very reliably. * We need to look into privacy when RPZs are being realised over DNS - I don’t even understand why we are suddenly starting to pull text files over HTTP again. The IPS seems to have most of these lists already. Regarding your code, there are some issues with the coding style, but I heavily appreciate the pioneering to bring this feature to life. What it would need to be finally merged would be a web UI though. So I think we have a long way ahead of us. -Michael > I saw your comments in the Dev Mailing List of "generally being in favor of trying this path" (bad paraphrasing on my part) > > I saw your comments in bugzilla at https://bugzilla.ipfire.org/show_bug.cgi?id=13254#c171 >> I am not interested in anything regarding the RPZs right now. They have not been properly put on the agenda and looking at how much time we have on our hands, this won't make it on the agenda for years. >> >> I don't want to build blockers, but this ticket is about a different problem which I want to solve first. > > > How do you want to go forward? > > > Jon > > >> On Aug 12, 2024, at 2:11 PM, jon <jon.murphy@ipfire.org> wrote: >> >> More questions! >> >> Currently RPZ config files are at `/etc/unbound/local.d` but this directory seems like it is for user (admin) customizations. >> >> ``` >> [root@ipfire ~] # ls -al /etc/unbound/local.d >> total 68 >> drwxr-xr-x 2 nobody nobody 4096 Aug 12 13:41 . >> drwxr-xr-x 4 root root 4096 Aug 12 00:52 .. >> -rw-r--r-- 1 nobody nobody 436 Jul 12 15:45 00-rpz.conf >> -rw-r--r-- 1 nobody nobody 285 Mar 1 22:12 AmazonTrkrHZ.rpz.conf >> -rw-r--r-- 1 nobody nobody 281 Mar 1 22:02 AppleTrkrHZ.rpz.conf >> -rw-r--r-- 1 nobody nobody 269 Mar 1 21:40 DOHblockHZ.rpz.conf >> ... >> -rw-r--r-- 1 nobody nobody 299 Aug 1 19:42 WinTrkrHZ.rpz.conf >> [root@ipfire ~] # >> ``` >> >> >> Each file is a config file per category (or one per RPZ file). This makes it easy to add or remove a category (or RPZ file). >> >> Should I create a new unbound directory for RPZ config files? Maybe `/etc/unbound/rpz.d`? Or `/etc/unbound/rpz`? >> >> >> Jon >> >> >>> On Aug 1, 2024, at 1:45 PM, Jon Murphy <jon.murphy@ipfire.org> wrote: >>> >>> changed all paths from `/var/ipfire/rpz/` to `/var/ipfire/dns/rpz/` >>> (thank you to Adolf!) >>> >>> rpz-config: >>> - bug: corrected "Type" test from block to allow >>> - removed verbose parameter from various commands >>> >>> rpz-metrics: >>> - bug: corrected grep for rpz name count >>> - bug: fixed divide by zero error (thank you Peppe!) >>> >>> install/uninstall: >>> - bug: corrected scripts (thank you Bernhard!) >>> >>> Signed-off-by: Jon Murphy <jon.murphy@ipfire.org> >>> --- >>> config/backup/includes/rpz | 4 ++-- >>> config/rootfiles/packages/rpz | 6 +++--- >>> config/rpz/rpz-config | 14 +++++++------- >>> config/rpz/rpz-metrics | 9 +++++---- >>> lfs/rpz | 6 +++--- >>> src/paks/rpz/install.sh | 27 +++++++++++++++++++++++++++ >>> src/paks/rpz/uninstall.sh | 31 +++++++++++++++++++++++++++++++ >>> src/paks/rpz/update.sh | 25 +++++++++++++++++++++++++ >>> 8 files changed, 103 insertions(+), 19 deletions(-) >>> create mode 100644 src/paks/rpz/install.sh >>> create mode 100644 src/paks/rpz/uninstall.sh >>> create mode 100644 src/paks/rpz/update.sh >>> >>> diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz >>> index 4d59bb40c..8c7410ebd 100644 >>> --- a/config/backup/includes/rpz >>> +++ b/config/backup/includes/rpz >>> @@ -1,5 +1,5 @@ >>> -/var/ipfire/rpz/allowlist >>> -/var/ipfire/rpz/blocklist >>> +/var/ipfire/dns/rpz/allowlist >>> +/var/ipfire/dns/rpz/blocklist >>> /etc/unbound/zonefiles/allow.rpz >>> /etc/unbound/zonefiles/block.rpz >>> /etc/unbound/local.d/*rpz.conf >>> diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/rpz >>> index 2ffa715dd..183825362 100644 >>> --- a/config/rootfiles/packages/rpz >>> +++ b/config/rootfiles/packages/rpz >>> @@ -6,6 +6,6 @@ usr/sbin/rpz-config >>> usr/sbin/rpz-metrics >>> usr/sbin/rpz-sleep >>> var/ipfire/backup/addons/includes/rpz >>> -var/ipfire/rpz >>> -var/ipfire/rpz/allowlist >>> -var/ipfire/rpz/blocklist >>> +var/ipfire/dns/rpz >>> +var/ipfire/dns/rpz/allowlist >>> +var/ipfire/dns/rpz/blocklist >>> diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config >>> index 98dc0a4ca..a24a5c132 100644 >>> --- a/config/rpz/rpz-config >>> +++ b/config/rpz/rpz-config >>> @@ -19,7 +19,7 @@ >>> # # >>> ############################################################################### >>> >>> -# v22 - 2024-07-12 >>> +# v23 - 2024-07-30 >>> >>> ############### Functions ############### >>> >>> @@ -54,11 +54,11 @@ check_unbound_conf () { >>> make_rpz_file () { >>> local theType="${1}" # allow or block >>> >>> - theList="/var/ipfire/rpz/${theType}list" # input user list of domains >>> + theList="/var/ipfire/dns/rpz/${theType}list" # input custom list of domains >>> theZoneFile="/etc/unbound/zonefiles/${theType}.rpz" # output file for RPZ >>> >>> theAction='.' >>> - if [[ "${theType}" =~ "block" ]] ; then >>> + if [[ "${theType}" =~ "allow" ]] ; then >>> theAction='rpz-passthru.' >>> fi >>> >>> @@ -131,8 +131,8 @@ case "${theAction}" in >>> # set-up zone file >>> /usr/bin/touch "${rpzFile}" >>> # unbound requires these settings for rpz files >>> - /bin/chown --verbose nobody:nobody "${rpzFile}" >>> - /bin/chmod --verbose 644 "${rpzFile}" >>> + /bin/chown nobody:nobody "${rpzFile}" >>> + /bin/chmod 644 "${rpzFile}" >>> ;; >>> >>> # trash config file & rpz file >>> @@ -143,8 +143,8 @@ case "${theAction}" in >>> fi >>> >>> msg_log "info: rpz: remove config file & rpz file \"${theName}\"" >>> - /bin/rm --verbose "${rpzConfig}" >>> - /bin/rm --verbose "${rpzFile}" >>> + /bin/rm "${rpzConfig}" >>> + /bin/rm "${rpzFile}" >>> >>> check_unbound_conf >>> ;; >>> diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics >>> index 0f97c7911..4d932726e 100644 >>> --- a/config/rpz/rpz-metrics >>> +++ b/config/rpz/rpz-metrics >>> @@ -19,7 +19,7 @@ >>> # # >>> ############################################################################### >>> >>> -# v18 on 2024-07-05 >>> +# v19 on 2024-07-30 >>> >>> ############### Main ############### >>> >>> @@ -33,7 +33,7 @@ messageLogs=$( find /var/log/messages* -type f | >>> >>> # get the list of RPZ names & counts from the message log(s) >>> rpzNameCount=$( for logf in ${messageLogs} ; do >>> - /usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" | >>> + /usr/bin/zgrep --text --extended-regexp 'info: rpz: applied.* A IN$' "${logf}" | >>> /usr/bin/awk '$10 ~ /\[\w*]/ { print $10 }' ; >>> done | /usr/bin/sort | /usr/bin/uniq --count ) >>> >>> @@ -107,8 +107,9 @@ do >>> theLines=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' ) >>> totalLines=$(( totalLines + theLines )) >>> >>> - #hitsPerLine=$( echo "scale=0 ; $theHits / $theLines" | bc ) >>> - hitsPerLine=$(( 100 * theHits / theLines )) >>> + if [[ "${theLines}" -gt 2 ]] ; then >>> + hitsPerLine=$(( 100 * theHits / theLines )) >>> + fi >>> fi >>> >>> # get modification date >>> diff --git a/lfs/rpz b/lfs/rpz >>> index 319c10b7f..73f6f2b1b 100644 >>> --- a/lfs/rpz >>> +++ b/lfs/rpz >>> @@ -67,9 +67,9 @@ $(TARGET) : >>> $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin >>> >>> # Install settings folder and two empty files >>> - mkdir -pv /var/ipfire/rpz >>> - touch /var/ipfire/rpz/allowlist >>> - touch /var/ipfire/rpz/blocklist >>> + mkdir -pv /var/ipfire/dns/rpz >>> + touch /var/ipfire/dns/rpz/allowlist >>> + touch /var/ipfire/dns/rpz/blocklist >>> >>> # Add conf file to /etc directory >>> cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d >>> diff --git a/src/paks/rpz/install.sh b/src/paks/rpz/install.sh >>> new file mode 100644 >>> index 000000000..0a797e158 >>> --- /dev/null >>> +++ b/src/paks/rpz/install.sh >>> @@ -0,0 +1,27 @@ >>> +#!/bin/bash >>> +############################################################################### >>> +# # >>> +# IPFire.org - A linux based firewall # >>> +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # >>> +# # >>> +# This program is free software: you can redistribute it and/or modify # >>> +# it under the terms of the GNU General Public License as published by # >>> +# the Free Software Foundation, either version 3 of the License, or # >>> +# (at your option) any later version. # >>> +# # >>> +# This program is distributed in the hope that it will be useful, # >>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >>> +# GNU General Public License for more details. # >>> +# # >>> +# You should have received a copy of the GNU General Public License # >>> +# along with this program. If not, see <http://www.gnu.org/licenses/>. # >>> +# # >>> +############################################################################### >>> +# >>> +. /opt/pakfire/lib/functions.sh >>> +extract_files >>> +restore_backup ${NAME} >>> + >>> +# restart unbound to load config file >>> +/etc/init.d/unbound restart >>> diff --git a/src/paks/rpz/uninstall.sh b/src/paks/rpz/uninstall.sh >>> new file mode 100644 >>> index 000000000..4fb20e127 >>> --- /dev/null >>> +++ b/src/paks/rpz/uninstall.sh >>> @@ -0,0 +1,31 @@ >>> +#!/bin/bash >>> +############################################################################### >>> +# # >>> +# IPFire.org - A linux based firewall # >>> +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # >>> +# # >>> +# This program is free software: you can redistribute it and/or modify # >>> +# it under the terms of the GNU General Public License as published by # >>> +# the Free Software Foundation, either version 3 of the License, or # >>> +# (at your option) any later version. # >>> +# # >>> +# This program is distributed in the hope that it will be useful, # >>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >>> +# GNU General Public License for more details. # >>> +# # >>> +# You should have received a copy of the GNU General Public License # >>> +# along with this program. If not, see <http://www.gnu.org/licenses/>. # >>> +# # >>> +############################################################################### >>> +# >>> +. /opt/pakfire/lib/functions.sh >>> + >>> +# stop unbound to delete RPZ conf file >>> +/etc/init.d/unbound stop >>> + >>> +make_backup ${NAME} >>> +remove_files >>> + >>> +# start unbound to load unbound config file >>> +/etc/init.d/unbound start >>> diff --git a/src/paks/rpz/update.sh b/src/paks/rpz/update.sh >>> new file mode 100644 >>> index 000000000..938a93a40 >>> --- /dev/null >>> +++ b/src/paks/rpz/update.sh >>> @@ -0,0 +1,25 @@ >>> +#!/bin/bash >>> +############################################################################### >>> +# # >>> +# IPFire.org - A linux based firewall # >>> +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # >>> +# # >>> +# This program is free software: you can redistribute it and/or modify # >>> +# it under the terms of the GNU General Public License as published by # >>> +# the Free Software Foundation, either version 3 of the License, or # >>> +# (at your option) any later version. # >>> +# # >>> +# This program is distributed in the hope that it will be useful, # >>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >>> +# GNU General Public License for more details. # >>> +# # >>> +# You should have received a copy of the GNU General Public License # >>> +# along with this program. If not, see <http://www.gnu.org/licenses/>. # >>> +# # >>> +############################################################################### >>> +# >>> +. /opt/pakfire/lib/functions.sh >>> +extract_backup_includes >>> +./uninstall.sh >>> +./install.sh >>> -- >>> 2.30.2 >>> >> >
Comments below. Jon > On Aug 15, 2024, at 10:33 AM, Michael Tremer <michael.tremer@ipfire.org> wrote: > > Hello, > >> On 14 Aug 2024, at 18:14, jon <jon.murphy@ipfire.org> wrote: >> >> Michael, >> >> Sorry for putting you on the spot, but what do you want to do with this RPZ add-on? > > I am not sure this is a question for only me. > > I personally just don’t have any capacity to take on another rather large project as I have a huge backlog of so many things and I feel like a lonely fighter getting these all over the line. For my own sanity I need to have a couple of those closed before thinking about the next ones. > > A couple of months ago we have talked on the monthly call about RPZ and the decision that was made by the people was to not look at this now, just because there were other more pressing things. We could also not find answers to the questions that led us into RPZ: > > * Are there any good lists out there that would allow us to replace the URL Filter? That thing is basically on its last leg because of the lack of lists. We need to be able to block ads and pr0n and that very reliably. I believe this exists with the Hagezi lists at https://github.com/hagezi/dns-blocklists and/or the addition of a chosen "oisd NSFW" list at https://oisd.nl/setup I understand there is no time to review to see if this meet the needs. So we can leave this as "I believe these exist" for now. > > * We need to look into privacy when RPZs are being realised over DNS - I don’t even understand why we are suddenly starting to pull text files over HTTP again. The IPS seems to have most of these lists already. All of the lists that I am pull are HTTPS. And HTTPS only is currently checked (validated) in the existing `rpz-config` code. > > Regarding your code, there are some issues with the coding style, but I heavily appreciate the pioneering to bring this feature to life. My style is camelCase, but if you prefer snake_case that is an easy change (please speak up). As for as other style changes, that can wait until time is available by the core developers. > What it would need to be finally merged would be a web UI though. So I think we have a long way ahead of us. The attachment is an early mock-up I made a few months ago (made with Pixelmator Pro). The WebGUI is being worked on now. Does this help? Jon > > -Michael > >> I saw your comments in the Dev Mailing List of "generally being in favor of trying this path" (bad paraphrasing on my part) >> >> I saw your comments in bugzilla at https://bugzilla.ipfire.org/show_bug.cgi?id=13254#c171 >>> I am not interested in anything regarding the RPZs right now. They have not been properly put on the agenda and looking at how much time we have on our hands, this won't make it on the agenda for years. >>> >>> I don't want to build blockers, but this ticket is about a different problem which I want to solve first. >> >> >> How do you want to go forward? >> >> >> Jon >> >> >>> On Aug 12, 2024, at 2:11 PM, jon <jon.murphy@ipfire.org> wrote: >>> >>> More questions! >>> >>> Currently RPZ config files are at `/etc/unbound/local.d` but this directory seems like it is for user (admin) customizations. >>> >>> ``` >>> [root@ipfire ~] # ls -al /etc/unbound/local.d >>> total 68 >>> drwxr-xr-x 2 nobody nobody 4096 Aug 12 13:41 . >>> drwxr-xr-x 4 root root 4096 Aug 12 00:52 .. >>> -rw-r--r-- 1 nobody nobody 436 Jul 12 15:45 00-rpz.conf >>> -rw-r--r-- 1 nobody nobody 285 Mar 1 22:12 AmazonTrkrHZ.rpz.conf >>> -rw-r--r-- 1 nobody nobody 281 Mar 1 22:02 AppleTrkrHZ.rpz.conf >>> -rw-r--r-- 1 nobody nobody 269 Mar 1 21:40 DOHblockHZ.rpz.conf >>> ... >>> -rw-r--r-- 1 nobody nobody 299 Aug 1 19:42 WinTrkrHZ.rpz.conf >>> [root@ipfire ~] # >>> ``` >>> >>> >>> Each file is a config file per category (or one per RPZ file). This makes it easy to add or remove a category (or RPZ file). >>> >>> Should I create a new unbound directory for RPZ config files? Maybe `/etc/unbound/rpz.d`? Or `/etc/unbound/rpz`? >>> >>> >>> Jon >>> >>> >>>> On Aug 1, 2024, at 1:45 PM, Jon Murphy <jon.murphy@ipfire.org> wrote: >>>> >>>> changed all paths from `/var/ipfire/rpz/` to `/var/ipfire/dns/rpz/` >>>> (thank you to Adolf!) >>>> >>>> rpz-config: >>>> - bug: corrected "Type" test from block to allow >>>> - removed verbose parameter from various commands >>>> >>>> rpz-metrics: >>>> - bug: corrected grep for rpz name count >>>> - bug: fixed divide by zero error (thank you Peppe!) >>>> >>>> install/uninstall: >>>> - bug: corrected scripts (thank you Bernhard!) >>>> >>>> Signed-off-by: Jon Murphy <jon.murphy@ipfire.org> >>>> --- >>>> config/backup/includes/rpz | 4 ++-- >>>> config/rootfiles/packages/rpz | 6 +++--- >>>> config/rpz/rpz-config | 14 +++++++------- >>>> config/rpz/rpz-metrics | 9 +++++---- >>>> lfs/rpz | 6 +++--- >>>> src/paks/rpz/install.sh | 27 +++++++++++++++++++++++++++ >>>> src/paks/rpz/uninstall.sh | 31 +++++++++++++++++++++++++++++++ >>>> src/paks/rpz/update.sh | 25 +++++++++++++++++++++++++ >>>> 8 files changed, 103 insertions(+), 19 deletions(-) >>>> create mode 100644 src/paks/rpz/install.sh >>>> create mode 100644 src/paks/rpz/uninstall.sh >>>> create mode 100644 src/paks/rpz/update.sh >>>> >>>> diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz >>>> index 4d59bb40c..8c7410ebd 100644 >>>> --- a/config/backup/includes/rpz >>>> +++ b/config/backup/includes/rpz >>>> @@ -1,5 +1,5 @@ >>>> -/var/ipfire/rpz/allowlist >>>> -/var/ipfire/rpz/blocklist >>>> +/var/ipfire/dns/rpz/allowlist >>>> +/var/ipfire/dns/rpz/blocklist >>>> /etc/unbound/zonefiles/allow.rpz >>>> /etc/unbound/zonefiles/block.rpz >>>> /etc/unbound/local.d/*rpz.conf >>>> diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/rpz >>>> index 2ffa715dd..183825362 100644 >>>> --- a/config/rootfiles/packages/rpz >>>> +++ b/config/rootfiles/packages/rpz >>>> @@ -6,6 +6,6 @@ usr/sbin/rpz-config >>>> usr/sbin/rpz-metrics >>>> usr/sbin/rpz-sleep >>>> var/ipfire/backup/addons/includes/rpz >>>> -var/ipfire/rpz >>>> -var/ipfire/rpz/allowlist >>>> -var/ipfire/rpz/blocklist >>>> +var/ipfire/dns/rpz >>>> +var/ipfire/dns/rpz/allowlist >>>> +var/ipfire/dns/rpz/blocklist >>>> diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config >>>> index 98dc0a4ca..a24a5c132 100644 >>>> --- a/config/rpz/rpz-config >>>> +++ b/config/rpz/rpz-config >>>> @@ -19,7 +19,7 @@ >>>> # # >>>> ############################################################################### >>>> >>>> -# v22 - 2024-07-12 >>>> +# v23 - 2024-07-30 >>>> >>>> ############### Functions ############### >>>> >>>> @@ -54,11 +54,11 @@ check_unbound_conf () { >>>> make_rpz_file () { >>>> local theType="${1}" # allow or block >>>> >>>> - theList="/var/ipfire/rpz/${theType}list" # input user list of domains >>>> + theList="/var/ipfire/dns/rpz/${theType}list" # input custom list of domains >>>> theZoneFile="/etc/unbound/zonefiles/${theType}.rpz" # output file for RPZ >>>> >>>> theAction='.' >>>> - if [[ "${theType}" =~ "block" ]] ; then >>>> + if [[ "${theType}" =~ "allow" ]] ; then >>>> theAction='rpz-passthru.' >>>> fi >>>> >>>> @@ -131,8 +131,8 @@ case "${theAction}" in >>>> # set-up zone file >>>> /usr/bin/touch "${rpzFile}" >>>> # unbound requires these settings for rpz files >>>> - /bin/chown --verbose nobody:nobody "${rpzFile}" >>>> - /bin/chmod --verbose 644 "${rpzFile}" >>>> + /bin/chown nobody:nobody "${rpzFile}" >>>> + /bin/chmod 644 "${rpzFile}" >>>> ;; >>>> >>>> # trash config file & rpz file >>>> @@ -143,8 +143,8 @@ case "${theAction}" in >>>> fi >>>> >>>> msg_log "info: rpz: remove config file & rpz file \"${theName}\"" >>>> - /bin/rm --verbose "${rpzConfig}" >>>> - /bin/rm --verbose "${rpzFile}" >>>> + /bin/rm "${rpzConfig}" >>>> + /bin/rm "${rpzFile}" >>>> >>>> check_unbound_conf >>>> ;; >>>> diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics >>>> index 0f97c7911..4d932726e 100644 >>>> --- a/config/rpz/rpz-metrics >>>> +++ b/config/rpz/rpz-metrics >>>> @@ -19,7 +19,7 @@ >>>> # # >>>> ############################################################################### >>>> >>>> -# v18 on 2024-07-05 >>>> +# v19 on 2024-07-30 >>>> >>>> ############### Main ############### >>>> >>>> @@ -33,7 +33,7 @@ messageLogs=$( find /var/log/messages* -type f | >>>> >>>> # get the list of RPZ names & counts from the message log(s) >>>> rpzNameCount=$( for logf in ${messageLogs} ; do >>>> - /usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" | >>>> + /usr/bin/zgrep --text --extended-regexp 'info: rpz: applied.* A IN$' "${logf}" | >>>> /usr/bin/awk '$10 ~ /\[\w*]/ { print $10 }' ; >>>> done | /usr/bin/sort | /usr/bin/uniq --count ) >>>> >>>> @@ -107,8 +107,9 @@ do >>>> theLines=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' ) >>>> totalLines=$(( totalLines + theLines )) >>>> >>>> - #hitsPerLine=$( echo "scale=0 ; $theHits / $theLines" | bc ) >>>> - hitsPerLine=$(( 100 * theHits / theLines )) >>>> + if [[ "${theLines}" -gt 2 ]] ; then >>>> + hitsPerLine=$(( 100 * theHits / theLines )) >>>> + fi >>>> fi >>>> >>>> # get modification date >>>> diff --git a/lfs/rpz b/lfs/rpz >>>> index 319c10b7f..73f6f2b1b 100644 >>>> --- a/lfs/rpz >>>> +++ b/lfs/rpz >>>> @@ -67,9 +67,9 @@ $(TARGET) : >>>> $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin >>>> >>>> # Install settings folder and two empty files >>>> - mkdir -pv /var/ipfire/rpz >>>> - touch /var/ipfire/rpz/allowlist >>>> - touch /var/ipfire/rpz/blocklist >>>> + mkdir -pv /var/ipfire/dns/rpz >>>> + touch /var/ipfire/dns/rpz/allowlist >>>> + touch /var/ipfire/dns/rpz/blocklist >>>> >>>> # Add conf file to /etc directory >>>> cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d >>>> diff --git a/src/paks/rpz/install.sh b/src/paks/rpz/install.sh >>>> new file mode 100644 >>>> index 000000000..0a797e158 >>>> --- /dev/null >>>> +++ b/src/paks/rpz/install.sh >>>> @@ -0,0 +1,27 @@ >>>> +#!/bin/bash >>>> +############################################################################### >>>> +# # >>>> +# IPFire.org - A linux based firewall # >>>> +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # >>>> +# # >>>> +# This program is free software: you can redistribute it and/or modify # >>>> +# it under the terms of the GNU General Public License as published by # >>>> +# the Free Software Foundation, either version 3 of the License, or # >>>> +# (at your option) any later version. # >>>> +# # >>>> +# This program is distributed in the hope that it will be useful, # >>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >>>> +# GNU General Public License for more details. # >>>> +# # >>>> +# You should have received a copy of the GNU General Public License # >>>> +# along with this program. If not, see <http://www.gnu.org/licenses/>. # >>>> +# # >>>> +############################################################################### >>>> +# >>>> +. /opt/pakfire/lib/functions.sh >>>> +extract_files >>>> +restore_backup ${NAME} >>>> + >>>> +# restart unbound to load config file >>>> +/etc/init.d/unbound restart >>>> diff --git a/src/paks/rpz/uninstall.sh b/src/paks/rpz/uninstall.sh >>>> new file mode 100644 >>>> index 000000000..4fb20e127 >>>> --- /dev/null >>>> +++ b/src/paks/rpz/uninstall.sh >>>> @@ -0,0 +1,31 @@ >>>> +#!/bin/bash >>>> +############################################################################### >>>> +# # >>>> +# IPFire.org - A linux based firewall # >>>> +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # >>>> +# # >>>> +# This program is free software: you can redistribute it and/or modify # >>>> +# it under the terms of the GNU General Public License as published by # >>>> +# the Free Software Foundation, either version 3 of the License, or # >>>> +# (at your option) any later version. # >>>> +# # >>>> +# This program is distributed in the hope that it will be useful, # >>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >>>> +# GNU General Public License for more details. # >>>> +# # >>>> +# You should have received a copy of the GNU General Public License # >>>> +# along with this program. If not, see <http://www.gnu.org/licenses/>. # >>>> +# # >>>> +############################################################################### >>>> +# >>>> +. /opt/pakfire/lib/functions.sh >>>> + >>>> +# stop unbound to delete RPZ conf file >>>> +/etc/init.d/unbound stop >>>> + >>>> +make_backup ${NAME} >>>> +remove_files >>>> + >>>> +# start unbound to load unbound config file >>>> +/etc/init.d/unbound start >>>> diff --git a/src/paks/rpz/update.sh b/src/paks/rpz/update.sh >>>> new file mode 100644 >>>> index 000000000..938a93a40 >>>> --- /dev/null >>>> +++ b/src/paks/rpz/update.sh >>>> @@ -0,0 +1,25 @@ >>>> +#!/bin/bash >>>> +############################################################################### >>>> +# # >>>> +# IPFire.org - A linux based firewall # >>>> +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # >>>> +# # >>>> +# This program is free software: you can redistribute it and/or modify # >>>> +# it under the terms of the GNU General Public License as published by # >>>> +# the Free Software Foundation, either version 3 of the License, or # >>>> +# (at your option) any later version. # >>>> +# # >>>> +# This program is distributed in the hope that it will be useful, # >>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >>>> +# GNU General Public License for more details. # >>>> +# # >>>> +# You should have received a copy of the GNU General Public License # >>>> +# along with this program. If not, see <http://www.gnu.org/licenses/>. # >>>> +# # >>>> +############################################################################### >>>> +# >>>> +. /opt/pakfire/lib/functions.sh >>>> +extract_backup_includes >>>> +./uninstall.sh >>>> +./install.sh >>>> -- >>>> 2.30.2 >>>> >>> >> >
Hello, > On 15 Aug 2024, at 17:57, jon <jon.murphy@ipfire.org> wrote: > > > Comments below. > > Jon > > >> On Aug 15, 2024, at 10:33 AM, Michael Tremer <michael.tremer@ipfire.org> wrote: >> >> Hello, >> >>> On 14 Aug 2024, at 18:14, jon <jon.murphy@ipfire.org> wrote: >>> >>> Michael, >>> >>> Sorry for putting you on the spot, but what do you want to do with this RPZ add-on? >> >> I am not sure this is a question for only me. >> >> I personally just don’t have any capacity to take on another rather large project as I have a huge backlog of so many things and I feel like a lonely fighter getting these all over the line. For my own sanity I need to have a couple of those closed before thinking about the next ones. >> >> A couple of months ago we have talked on the monthly call about RPZ and the decision that was made by the people was to not look at this now, just because there were other more pressing things. We could also not find answers to the questions that led us into RPZ: >> >> * Are there any good lists out there that would allow us to replace the URL Filter? That thing is basically on its last leg because of the lack of lists. We need to be able to block ads and pr0n and that very reliably. > > I believe this exists with the Hagezi lists at https://github.com/hagezi/dns-blocklists and/or the addition of a chosen "oisd NSFW" list at https://oisd.nl/setup > > I understand there is no time to review to see if this meet the needs. So we can leave this as "I believe these exist" for now. So, I am not sure what to think of all these things on GitHub. It seems that there are many repositories that pop up and presumably disappear in the same kind of way. We see this with the IP blocklists. We want quality blocklists. I know that we don’t have too much on the URL Filter front, but I would prefer to avoid putting so much work in if we in the end stand where we started. >> >> * We need to look into privacy when RPZs are being realised over DNS - I don’t even understand why we are suddenly starting to pull text files over HTTP again. The IPS seems to have most of these lists already. > > All of the lists that I am pull are HTTPS. And HTTPS only is currently checked (validated) in the existing `rpz-config` code. What a lost case. DNS is a globally distributed, cacheable protocol. We could simply put all those blacklists into the DNS caches all around the world. They would respond fast, be local to so many people, but instead we download massive text files. That means that whenever something changes the entire file needs to be transferred again. They are huge. We would want many fast updates which are not possible that way… but I digress. How much traffic did you observe? How is Unbound refreshing these? Is there a local cache? >> Regarding your code, there are some issues with the coding style, but I heavily appreciate the pioneering to bring this feature to life. > > My style is camelCase, but if you prefer snake_case that is an easy change (please speak up). > > As for as other style changes, that can wait until time is available by the core developers. It is mainly that you make things very complicated for yourself. Call “cat”, not “/bin/cat”. There is no guarantee that things will stay where they are now and the shell has lots of builtins that execute a lot faster. And this is all designed to work on the shell. We need something for the web UI. >> What it would need to be finally merged would be a web UI though. So I think we have a long way ahead of us. > > The attachment is an early mock-up I made a few months ago (made with Pixelmator Pro). The WebGUI is being worked on now. > > <mock-up.png> I think we might look for something that is very similar to the IPS ruleset editor or the IP blocklists. I am not even sure if we would require users to manage this or whether we would curate a list like with the IPS and IP blocklists. > Does this help? > Jon > > >> >> -Michael >> >>> I saw your comments in the Dev Mailing List of "generally being in favor of trying this path" (bad paraphrasing on my part) >>> >>> I saw your comments in bugzilla at https://bugzilla.ipfire.org/show_bug.cgi?id=13254#c171 >>>> I am not interested in anything regarding the RPZs right now. They have not been properly put on the agenda and looking at how much time we have on our hands, this won't make it on the agenda for years. >>>> >>>> I don't want to build blockers, but this ticket is about a different problem which I want to solve first. >>> >>> >>> How do you want to go forward? >>> >>> >>> Jon >>> >>> >>>> On Aug 12, 2024, at 2:11 PM, jon <jon.murphy@ipfire.org> wrote: >>>> >>>> More questions! >>>> >>>> Currently RPZ config files are at `/etc/unbound/local.d` but this directory seems like it is for user (admin) customizations. >>>> >>>> ``` >>>> [root@ipfire ~] # ls -al /etc/unbound/local.d >>>> total 68 >>>> drwxr-xr-x 2 nobody nobody 4096 Aug 12 13:41 . >>>> drwxr-xr-x 4 root root 4096 Aug 12 00:52 .. >>>> -rw-r--r-- 1 nobody nobody 436 Jul 12 15:45 00-rpz.conf >>>> -rw-r--r-- 1 nobody nobody 285 Mar 1 22:12 AmazonTrkrHZ.rpz.conf >>>> -rw-r--r-- 1 nobody nobody 281 Mar 1 22:02 AppleTrkrHZ.rpz.conf >>>> -rw-r--r-- 1 nobody nobody 269 Mar 1 21:40 DOHblockHZ.rpz.conf >>>> ... >>>> -rw-r--r-- 1 nobody nobody 299 Aug 1 19:42 WinTrkrHZ.rpz.conf >>>> [root@ipfire ~] # >>>> ``` >>>> >>>> >>>> Each file is a config file per category (or one per RPZ file). This makes it easy to add or remove a category (or RPZ file). >>>> >>>> Should I create a new unbound directory for RPZ config files? Maybe `/etc/unbound/rpz.d`? Or `/etc/unbound/rpz`? >>>> >>>> >>>> Jon >>>> >>>> >>>>> On Aug 1, 2024, at 1:45 PM, Jon Murphy <jon.murphy@ipfire.org> wrote: >>>>> >>>>> changed all paths from `/var/ipfire/rpz/` to `/var/ipfire/dns/rpz/` >>>>> (thank you to Adolf!) >>>>> >>>>> rpz-config: >>>>> - bug: corrected "Type" test from block to allow >>>>> - removed verbose parameter from various commands >>>>> >>>>> rpz-metrics: >>>>> - bug: corrected grep for rpz name count >>>>> - bug: fixed divide by zero error (thank you Peppe!) >>>>> >>>>> install/uninstall: >>>>> - bug: corrected scripts (thank you Bernhard!) >>>>> >>>>> Signed-off-by: Jon Murphy <jon.murphy@ipfire.org> >>>>> --- >>>>> config/backup/includes/rpz | 4 ++-- >>>>> config/rootfiles/packages/rpz | 6 +++--- >>>>> config/rpz/rpz-config | 14 +++++++------- >>>>> config/rpz/rpz-metrics | 9 +++++---- >>>>> lfs/rpz | 6 +++--- >>>>> src/paks/rpz/install.sh | 27 +++++++++++++++++++++++++++ >>>>> src/paks/rpz/uninstall.sh | 31 +++++++++++++++++++++++++++++++ >>>>> src/paks/rpz/update.sh | 25 +++++++++++++++++++++++++ >>>>> 8 files changed, 103 insertions(+), 19 deletions(-) >>>>> create mode 100644 src/paks/rpz/install.sh >>>>> create mode 100644 src/paks/rpz/uninstall.sh >>>>> create mode 100644 src/paks/rpz/update.sh >>>>> >>>>> diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz >>>>> index 4d59bb40c..8c7410ebd 100644 >>>>> --- a/config/backup/includes/rpz >>>>> +++ b/config/backup/includes/rpz >>>>> @@ -1,5 +1,5 @@ >>>>> -/var/ipfire/rpz/allowlist >>>>> -/var/ipfire/rpz/blocklist >>>>> +/var/ipfire/dns/rpz/allowlist >>>>> +/var/ipfire/dns/rpz/blocklist >>>>> /etc/unbound/zonefiles/allow.rpz >>>>> /etc/unbound/zonefiles/block.rpz >>>>> /etc/unbound/local.d/*rpz.conf >>>>> diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/rpz >>>>> index 2ffa715dd..183825362 100644 >>>>> --- a/config/rootfiles/packages/rpz >>>>> +++ b/config/rootfiles/packages/rpz >>>>> @@ -6,6 +6,6 @@ usr/sbin/rpz-config >>>>> usr/sbin/rpz-metrics >>>>> usr/sbin/rpz-sleep >>>>> var/ipfire/backup/addons/includes/rpz >>>>> -var/ipfire/rpz >>>>> -var/ipfire/rpz/allowlist >>>>> -var/ipfire/rpz/blocklist >>>>> +var/ipfire/dns/rpz >>>>> +var/ipfire/dns/rpz/allowlist >>>>> +var/ipfire/dns/rpz/blocklist >>>>> diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config >>>>> index 98dc0a4ca..a24a5c132 100644 >>>>> --- a/config/rpz/rpz-config >>>>> +++ b/config/rpz/rpz-config >>>>> @@ -19,7 +19,7 @@ >>>>> # # >>>>> ############################################################################### >>>>> >>>>> -# v22 - 2024-07-12 >>>>> +# v23 - 2024-07-30 >>>>> >>>>> ############### Functions ############### >>>>> >>>>> @@ -54,11 +54,11 @@ check_unbound_conf () { >>>>> make_rpz_file () { >>>>> local theType="${1}" # allow or block >>>>> >>>>> - theList="/var/ipfire/rpz/${theType}list" # input user list of domains >>>>> + theList="/var/ipfire/dns/rpz/${theType}list" # input custom list of domains >>>>> theZoneFile="/etc/unbound/zonefiles/${theType}.rpz" # output file for RPZ >>>>> >>>>> theAction='.' >>>>> - if [[ "${theType}" =~ "block" ]] ; then >>>>> + if [[ "${theType}" =~ "allow" ]] ; then >>>>> theAction='rpz-passthru.' >>>>> fi >>>>> >>>>> @@ -131,8 +131,8 @@ case "${theAction}" in >>>>> # set-up zone file >>>>> /usr/bin/touch "${rpzFile}" >>>>> # unbound requires these settings for rpz files >>>>> - /bin/chown --verbose nobody:nobody "${rpzFile}" >>>>> - /bin/chmod --verbose 644 "${rpzFile}" >>>>> + /bin/chown nobody:nobody "${rpzFile}" >>>>> + /bin/chmod 644 "${rpzFile}" >>>>> ;; >>>>> >>>>> # trash config file & rpz file >>>>> @@ -143,8 +143,8 @@ case "${theAction}" in >>>>> fi >>>>> >>>>> msg_log "info: rpz: remove config file & rpz file \"${theName}\"" >>>>> - /bin/rm --verbose "${rpzConfig}" >>>>> - /bin/rm --verbose "${rpzFile}" >>>>> + /bin/rm "${rpzConfig}" >>>>> + /bin/rm "${rpzFile}" >>>>> >>>>> check_unbound_conf >>>>> ;; >>>>> diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics >>>>> index 0f97c7911..4d932726e 100644 >>>>> --- a/config/rpz/rpz-metrics >>>>> +++ b/config/rpz/rpz-metrics >>>>> @@ -19,7 +19,7 @@ >>>>> # # >>>>> ############################################################################### >>>>> >>>>> -# v18 on 2024-07-05 >>>>> +# v19 on 2024-07-30 >>>>> >>>>> ############### Main ############### >>>>> >>>>> @@ -33,7 +33,7 @@ messageLogs=$( find /var/log/messages* -type f | >>>>> >>>>> # get the list of RPZ names & counts from the message log(s) >>>>> rpzNameCount=$( for logf in ${messageLogs} ; do >>>>> - /usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" | >>>>> + /usr/bin/zgrep --text --extended-regexp 'info: rpz: applied.* A IN$' "${logf}" | >>>>> /usr/bin/awk '$10 ~ /\[\w*]/ { print $10 }' ; >>>>> done | /usr/bin/sort | /usr/bin/uniq --count ) >>>>> >>>>> @@ -107,8 +107,9 @@ do >>>>> theLines=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' ) >>>>> totalLines=$(( totalLines + theLines )) >>>>> >>>>> - #hitsPerLine=$( echo "scale=0 ; $theHits / $theLines" | bc ) >>>>> - hitsPerLine=$(( 100 * theHits / theLines )) >>>>> + if [[ "${theLines}" -gt 2 ]] ; then >>>>> + hitsPerLine=$(( 100 * theHits / theLines )) >>>>> + fi >>>>> fi >>>>> >>>>> # get modification date >>>>> diff --git a/lfs/rpz b/lfs/rpz >>>>> index 319c10b7f..73f6f2b1b 100644 >>>>> --- a/lfs/rpz >>>>> +++ b/lfs/rpz >>>>> @@ -67,9 +67,9 @@ $(TARGET) : >>>>> $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin >>>>> >>>>> # Install settings folder and two empty files >>>>> - mkdir -pv /var/ipfire/rpz >>>>> - touch /var/ipfire/rpz/allowlist >>>>> - touch /var/ipfire/rpz/blocklist >>>>> + mkdir -pv /var/ipfire/dns/rpz >>>>> + touch /var/ipfire/dns/rpz/allowlist >>>>> + touch /var/ipfire/dns/rpz/blocklist >>>>> >>>>> # Add conf file to /etc directory >>>>> cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d >>>>> diff --git a/src/paks/rpz/install.sh b/src/paks/rpz/install.sh >>>>> new file mode 100644 >>>>> index 000000000..0a797e158 >>>>> --- /dev/null >>>>> +++ b/src/paks/rpz/install.sh >>>>> @@ -0,0 +1,27 @@ >>>>> +#!/bin/bash >>>>> +############################################################################### >>>>> +# # >>>>> +# IPFire.org - A linux based firewall # >>>>> +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # >>>>> +# # >>>>> +# This program is free software: you can redistribute it and/or modify # >>>>> +# it under the terms of the GNU General Public License as published by # >>>>> +# the Free Software Foundation, either version 3 of the License, or # >>>>> +# (at your option) any later version. # >>>>> +# # >>>>> +# This program is distributed in the hope that it will be useful, # >>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >>>>> +# GNU General Public License for more details. # >>>>> +# # >>>>> +# You should have received a copy of the GNU General Public License # >>>>> +# along with this program. If not, see <http://www.gnu.org/licenses/>. # >>>>> +# # >>>>> +############################################################################### >>>>> +# >>>>> +. /opt/pakfire/lib/functions.sh >>>>> +extract_files >>>>> +restore_backup ${NAME} >>>>> + >>>>> +# restart unbound to load config file >>>>> +/etc/init.d/unbound restart >>>>> diff --git a/src/paks/rpz/uninstall.sh b/src/paks/rpz/uninstall.sh >>>>> new file mode 100644 >>>>> index 000000000..4fb20e127 >>>>> --- /dev/null >>>>> +++ b/src/paks/rpz/uninstall.sh >>>>> @@ -0,0 +1,31 @@ >>>>> +#!/bin/bash >>>>> +############################################################################### >>>>> +# # >>>>> +# IPFire.org - A linux based firewall # >>>>> +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # >>>>> +# # >>>>> +# This program is free software: you can redistribute it and/or modify # >>>>> +# it under the terms of the GNU General Public License as published by # >>>>> +# the Free Software Foundation, either version 3 of the License, or # >>>>> +# (at your option) any later version. # >>>>> +# # >>>>> +# This program is distributed in the hope that it will be useful, # >>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >>>>> +# GNU General Public License for more details. # >>>>> +# # >>>>> +# You should have received a copy of the GNU General Public License # >>>>> +# along with this program. If not, see <http://www.gnu.org/licenses/>. # >>>>> +# # >>>>> +############################################################################### >>>>> +# >>>>> +. /opt/pakfire/lib/functions.sh >>>>> + >>>>> +# stop unbound to delete RPZ conf file >>>>> +/etc/init.d/unbound stop >>>>> + >>>>> +make_backup ${NAME} >>>>> +remove_files >>>>> + >>>>> +# start unbound to load unbound config file >>>>> +/etc/init.d/unbound start >>>>> diff --git a/src/paks/rpz/update.sh b/src/paks/rpz/update.sh >>>>> new file mode 100644 >>>>> index 000000000..938a93a40 >>>>> --- /dev/null >>>>> +++ b/src/paks/rpz/update.sh >>>>> @@ -0,0 +1,25 @@ >>>>> +#!/bin/bash >>>>> +############################################################################### >>>>> +# # >>>>> +# IPFire.org - A linux based firewall # >>>>> +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # >>>>> +# # >>>>> +# This program is free software: you can redistribute it and/or modify # >>>>> +# it under the terms of the GNU General Public License as published by # >>>>> +# the Free Software Foundation, either version 3 of the License, or # >>>>> +# (at your option) any later version. # >>>>> +# # >>>>> +# This program is distributed in the hope that it will be useful, # >>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >>>>> +# GNU General Public License for more details. # >>>>> +# # >>>>> +# You should have received a copy of the GNU General Public License # >>>>> +# along with this program. If not, see <http://www.gnu.org/licenses/>. # >>>>> +# # >>>>> +############################################################################### >>>>> +# >>>>> +. /opt/pakfire/lib/functions.sh >>>>> +extract_backup_includes >>>>> +./uninstall.sh >>>>> +./install.sh >>>>> -- >>>>> 2.30.2
Comments below… > On Aug 15, 2024, at 1:33 PM, Michael Tremer <michael.tremer@ipfire.org> wrote: > > Hello, > >> On 15 Aug 2024, at 17:57, jon <jon.murphy@ipfire.org> wrote: >> >> >> Comments below. >> >> Jon >> >> >>> On Aug 15, 2024, at 10:33 AM, Michael Tremer <michael.tremer@ipfire.org> wrote: >>> >>> Hello, >>> >>>> On 14 Aug 2024, at 18:14, jon <jon.murphy@ipfire.org> wrote: >>>> >>>> Michael, >>>> >>>> Sorry for putting you on the spot, but what do you want to do with this RPZ add-on? >>> >>> I am not sure this is a question for only me. >>> >>> I personally just don’t have any capacity to take on another rather large project as I have a huge backlog of so many things and I feel like a lonely fighter getting these all over the line. For my own sanity I need to have a couple of those closed before thinking about the next ones. >>> >>> A couple of months ago we have talked on the monthly call about RPZ and the decision that was made by the people was to not look at this now, just because there were other more pressing things. We could also not find answers to the questions that led us into RPZ: >>> >>> * Are there any good lists out there that would allow us to replace the URL Filter? That thing is basically on its last leg because of the lack of lists. We need to be able to block ads and pr0n and that very reliably. >> >> I believe this exists with the Hagezi lists at https://github.com/hagezi/dns-blocklists and/or the addition of a chosen "oisd NSFW" list at https://oisd.nl/setup >> >> I understand there is no time to review to see if this meet the needs. So we can leave this as "I believe these exist" for now. > > So, I am not sure what to think of all these things on GitHub. It seems that there are many repositories that pop up and presumably disappear in the same kind of way. We see this with the IP blocklists. > > We want quality blocklists. I know that we don’t have too much on the URL Filter front, but I would prefer to avoid putting so much work in if we in the end stand where we started. This makes perfect sense. Based on the comments I’ve seen it seems like Hagezi is growing in popularity and in offerings. Gerd (the owner) is very responsive, involved and I keep running into his posts when searching for RPZ "things". Gerd quickly made changes per my request: see https://github.com/hagezi/dns-blocklists/issues/2301. Now there are two TLD RPZ lists. Like everything else I cannot predict how long it might last. > >>> >>> * We need to look into privacy when RPZs are being realised over DNS - I don’t even understand why we are suddenly starting to pull text files over HTTP again. The IPS seems to have most of these lists already. >> >> All of the lists that I am pull are HTTPS. And HTTPS only is currently checked (validated) in the existing `rpz-config` code. > > What a lost case. DNS is a globally distributed, cacheable protocol. We could simply put all those blacklists into the DNS caches all around the world. They would respond fast, be local to so many people, but instead we download massive text files. That means that whenever something changes the entire file needs to be transferred again. They are huge. We would want many fast updates which are not possible that way… but I digress. I came across one DNS service that offers the Hagezi lists as part of their available DNS services. If interested I can find and post the details. > > How much traffic did you observe? I have 11 RPZ list (honestly too many but I am experimenting) and that eat up about 230M bytes per day. > How is Unbound refreshing these? Unbound RPZ takes care of the updates automagically. There is no cron entry. Each RPZ list includes an SOA line similar to this: `@ SOA localhost. root.localhost. 1723093380 43200 3600 259200 300` The 43200 is the refresh period in seconds. So this RPZ list auto updates every 12 hours. > Is there a local cache? Yes, it is part of unbound. (Nothing we need to setup) > >>> Regarding your code, there are some issues with the coding style, but I heavily appreciate the pioneering to bring this feature to life. >> >> My style is camelCase, but if you prefer snake_case that is an easy change (please speak up). >> >> As for as other style changes, that can wait until time is available by the core developers. > > It is mainly that you make things very complicated for yourself. Call “cat”, not “/bin/cat”. There is no guarantee that things will stay where they are now and the shell has lots of builtins that execute a lot faster. I will remove the executable paths. What builtin did you see that I missed? Anything else? > > And this is all designed to work on the shell. We need something for the web UI. For commands within the WegGUI there are `safe_system` commands on the `.cgi` page. But I am guess you are referring to something else. > >>> What it would need to be finally merged would be a web UI though. So I think we have a long way ahead of us. >> >> The attachment is an early mock-up I made a few months ago (made with Pixelmator Pro). The WebGUI is being worked on now. >> >> <mock-up.png> > > I think we might look for something that is very similar to the IPS ruleset editor or the IP blocklists. > > I am not even sure if we would require users to manage this or whether we would curate a list like with the IPS and IP blocklists. The first "DRAFT" of this was for the user/admin to manage the list. For the RPZ add-on, I’d prefer to keep it this way until we figure out if RPZ is popular and used by the user/admins. And yes, it could be changed to a set list like IPS. FYI - there are 34 different Hagezi list. So it will take some experiments to pick a "the" list. Also, here is current DRAFT (first draft) of user/admin managed WebGUI. See attachment. Jon > >> Does this help? >> Jon >> >> >>> >>> -Michael >>> >>>> I saw your comments in the Dev Mailing List of "generally being in favor of trying this path" (bad paraphrasing on my part) >>>> >>>> I saw your comments in bugzilla at https://bugzilla.ipfire.org/show_bug.cgi?id=13254#c171 >>>>> I am not interested in anything regarding the RPZs right now. They have not been properly put on the agenda and looking at how much time we have on our hands, this won't make it on the agenda for years. >>>>> >>>>> I don't want to build blockers, but this ticket is about a different problem which I want to solve first. >>>> >>>> >>>> How do you want to go forward? >>>> >>>> >>>> Jon >>>> >>>> >>>>> On Aug 12, 2024, at 2:11 PM, jon <jon.murphy@ipfire.org> wrote: >>>>> >>>>> More questions! >>>>> >>>>> Currently RPZ config files are at `/etc/unbound/local.d` but this directory seems like it is for user (admin) customizations. >>>>> >>>>> ``` >>>>> [root@ipfire ~] # ls -al /etc/unbound/local.d >>>>> total 68 >>>>> drwxr-xr-x 2 nobody nobody 4096 Aug 12 13:41 . >>>>> drwxr-xr-x 4 root root 4096 Aug 12 00:52 .. >>>>> -rw-r--r-- 1 nobody nobody 436 Jul 12 15:45 00-rpz.conf >>>>> -rw-r--r-- 1 nobody nobody 285 Mar 1 22:12 AmazonTrkrHZ.rpz.conf >>>>> -rw-r--r-- 1 nobody nobody 281 Mar 1 22:02 AppleTrkrHZ.rpz.conf >>>>> -rw-r--r-- 1 nobody nobody 269 Mar 1 21:40 DOHblockHZ.rpz.conf >>>>> ... >>>>> -rw-r--r-- 1 nobody nobody 299 Aug 1 19:42 WinTrkrHZ.rpz.conf >>>>> [root@ipfire ~] # >>>>> ``` >>>>> >>>>> >>>>> Each file is a config file per category (or one per RPZ file). This makes it easy to add or remove a category (or RPZ file). >>>>> >>>>> Should I create a new unbound directory for RPZ config files? Maybe `/etc/unbound/rpz.d`? Or `/etc/unbound/rpz`? >>>>> >>>>> >>>>> Jon >>>>> >>>>> >>>>>> On Aug 1, 2024, at 1:45 PM, Jon Murphy <jon.murphy@ipfire.org> wrote: >>>>>> >>>>>> changed all paths from `/var/ipfire/rpz/` to `/var/ipfire/dns/rpz/` >>>>>> (thank you to Adolf!) >>>>>> >>>>>> rpz-config: >>>>>> - bug: corrected "Type" test from block to allow >>>>>> - removed verbose parameter from various commands >>>>>> >>>>>> rpz-metrics: >>>>>> - bug: corrected grep for rpz name count >>>>>> - bug: fixed divide by zero error (thank you Peppe!) >>>>>> >>>>>> install/uninstall: >>>>>> - bug: corrected scripts (thank you Bernhard!) >>>>>> >>>>>> Signed-off-by: Jon Murphy <jon.murphy@ipfire.org> >>>>>> --- >>>>>> config/backup/includes/rpz | 4 ++-- >>>>>> config/rootfiles/packages/rpz | 6 +++--- >>>>>> config/rpz/rpz-config | 14 +++++++------- >>>>>> config/rpz/rpz-metrics | 9 +++++---- >>>>>> lfs/rpz | 6 +++--- >>>>>> src/paks/rpz/install.sh | 27 +++++++++++++++++++++++++++ >>>>>> src/paks/rpz/uninstall.sh | 31 +++++++++++++++++++++++++++++++ >>>>>> src/paks/rpz/update.sh | 25 +++++++++++++++++++++++++ >>>>>> 8 files changed, 103 insertions(+), 19 deletions(-) >>>>>> create mode 100644 src/paks/rpz/install.sh >>>>>> create mode 100644 src/paks/rpz/uninstall.sh >>>>>> create mode 100644 src/paks/rpz/update.sh >>>>>> >>>>>> diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz >>>>>> index 4d59bb40c..8c7410ebd 100644 >>>>>> --- a/config/backup/includes/rpz >>>>>> +++ b/config/backup/includes/rpz >>>>>> @@ -1,5 +1,5 @@ >>>>>> -/var/ipfire/rpz/allowlist >>>>>> -/var/ipfire/rpz/blocklist >>>>>> +/var/ipfire/dns/rpz/allowlist >>>>>> +/var/ipfire/dns/rpz/blocklist >>>>>> /etc/unbound/zonefiles/allow.rpz >>>>>> /etc/unbound/zonefiles/block.rpz >>>>>> /etc/unbound/local.d/*rpz.conf >>>>>> diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/rpz >>>>>> index 2ffa715dd..183825362 100644 >>>>>> --- a/config/rootfiles/packages/rpz >>>>>> +++ b/config/rootfiles/packages/rpz >>>>>> @@ -6,6 +6,6 @@ usr/sbin/rpz-config >>>>>> usr/sbin/rpz-metrics >>>>>> usr/sbin/rpz-sleep >>>>>> var/ipfire/backup/addons/includes/rpz >>>>>> -var/ipfire/rpz >>>>>> -var/ipfire/rpz/allowlist >>>>>> -var/ipfire/rpz/blocklist >>>>>> +var/ipfire/dns/rpz >>>>>> +var/ipfire/dns/rpz/allowlist >>>>>> +var/ipfire/dns/rpz/blocklist >>>>>> diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config >>>>>> index 98dc0a4ca..a24a5c132 100644 >>>>>> --- a/config/rpz/rpz-config >>>>>> +++ b/config/rpz/rpz-config >>>>>> @@ -19,7 +19,7 @@ >>>>>> # # >>>>>> ############################################################################### >>>>>> >>>>>> -# v22 - 2024-07-12 >>>>>> +# v23 - 2024-07-30 >>>>>> >>>>>> ############### Functions ############### >>>>>> >>>>>> @@ -54,11 +54,11 @@ check_unbound_conf () { >>>>>> make_rpz_file () { >>>>>> local theType="${1}" # allow or block >>>>>> >>>>>> - theList="/var/ipfire/rpz/${theType}list" # input user list of domains >>>>>> + theList="/var/ipfire/dns/rpz/${theType}list" # input custom list of domains >>>>>> theZoneFile="/etc/unbound/zonefiles/${theType}.rpz" # output file for RPZ >>>>>> >>>>>> theAction='.' >>>>>> - if [[ "${theType}" =~ "block" ]] ; then >>>>>> + if [[ "${theType}" =~ "allow" ]] ; then >>>>>> theAction='rpz-passthru.' >>>>>> fi >>>>>> >>>>>> @@ -131,8 +131,8 @@ case "${theAction}" in >>>>>> # set-up zone file >>>>>> /usr/bin/touch "${rpzFile}" >>>>>> # unbound requires these settings for rpz files >>>>>> - /bin/chown --verbose nobody:nobody "${rpzFile}" >>>>>> - /bin/chmod --verbose 644 "${rpzFile}" >>>>>> + /bin/chown nobody:nobody "${rpzFile}" >>>>>> + /bin/chmod 644 "${rpzFile}" >>>>>> ;; >>>>>> >>>>>> # trash config file & rpz file >>>>>> @@ -143,8 +143,8 @@ case "${theAction}" in >>>>>> fi >>>>>> >>>>>> msg_log "info: rpz: remove config file & rpz file \"${theName}\"" >>>>>> - /bin/rm --verbose "${rpzConfig}" >>>>>> - /bin/rm --verbose "${rpzFile}" >>>>>> + /bin/rm "${rpzConfig}" >>>>>> + /bin/rm "${rpzFile}" >>>>>> >>>>>> check_unbound_conf >>>>>> ;; >>>>>> diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics >>>>>> index 0f97c7911..4d932726e 100644 >>>>>> --- a/config/rpz/rpz-metrics >>>>>> +++ b/config/rpz/rpz-metrics >>>>>> @@ -19,7 +19,7 @@ >>>>>> # # >>>>>> ############################################################################### >>>>>> >>>>>> -# v18 on 2024-07-05 >>>>>> +# v19 on 2024-07-30 >>>>>> >>>>>> ############### Main ############### >>>>>> >>>>>> @@ -33,7 +33,7 @@ messageLogs=$( find /var/log/messages* -type f | >>>>>> >>>>>> # get the list of RPZ names & counts from the message log(s) >>>>>> rpzNameCount=$( for logf in ${messageLogs} ; do >>>>>> - /usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" | >>>>>> + /usr/bin/zgrep --text --extended-regexp 'info: rpz: applied.* A IN$' "${logf}" | >>>>>> /usr/bin/awk '$10 ~ /\[\w*]/ { print $10 }' ; >>>>>> done | /usr/bin/sort | /usr/bin/uniq --count ) >>>>>> >>>>>> @@ -107,8 +107,9 @@ do >>>>>> theLines=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' ) >>>>>> totalLines=$(( totalLines + theLines )) >>>>>> >>>>>> - #hitsPerLine=$( echo "scale=0 ; $theHits / $theLines" | bc ) >>>>>> - hitsPerLine=$(( 100 * theHits / theLines )) >>>>>> + if [[ "${theLines}" -gt 2 ]] ; then >>>>>> + hitsPerLine=$(( 100 * theHits / theLines )) >>>>>> + fi >>>>>> fi >>>>>> >>>>>> # get modification date >>>>>> diff --git a/lfs/rpz b/lfs/rpz >>>>>> index 319c10b7f..73f6f2b1b 100644 >>>>>> --- a/lfs/rpz >>>>>> +++ b/lfs/rpz >>>>>> @@ -67,9 +67,9 @@ $(TARGET) : >>>>>> $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin >>>>>> >>>>>> # Install settings folder and two empty files >>>>>> - mkdir -pv /var/ipfire/rpz >>>>>> - touch /var/ipfire/rpz/allowlist >>>>>> - touch /var/ipfire/rpz/blocklist >>>>>> + mkdir -pv /var/ipfire/dns/rpz >>>>>> + touch /var/ipfire/dns/rpz/allowlist >>>>>> + touch /var/ipfire/dns/rpz/blocklist >>>>>> >>>>>> # Add conf file to /etc directory >>>>>> cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d >>>>>> diff --git a/src/paks/rpz/install.sh b/src/paks/rpz/install.sh >>>>>> new file mode 100644 >>>>>> index 000000000..0a797e158 >>>>>> --- /dev/null >>>>>> +++ b/src/paks/rpz/install.sh >>>>>> @@ -0,0 +1,27 @@ >>>>>> +#!/bin/bash >>>>>> +############################################################################### >>>>>> +# # >>>>>> +# IPFire.org - A linux based firewall # >>>>>> +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # >>>>>> +# # >>>>>> +# This program is free software: you can redistribute it and/or modify # >>>>>> +# it under the terms of the GNU General Public License as published by # >>>>>> +# the Free Software Foundation, either version 3 of the License, or # >>>>>> +# (at your option) any later version. # >>>>>> +# # >>>>>> +# This program is distributed in the hope that it will be useful, # >>>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >>>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >>>>>> +# GNU General Public License for more details. # >>>>>> +# # >>>>>> +# You should have received a copy of the GNU General Public License # >>>>>> +# along with this program. If not, see <http://www.gnu.org/licenses/>. # >>>>>> +# # >>>>>> +############################################################################### >>>>>> +# >>>>>> +. /opt/pakfire/lib/functions.sh >>>>>> +extract_files >>>>>> +restore_backup ${NAME} >>>>>> + >>>>>> +# restart unbound to load config file >>>>>> +/etc/init.d/unbound restart >>>>>> diff --git a/src/paks/rpz/uninstall.sh b/src/paks/rpz/uninstall.sh >>>>>> new file mode 100644 >>>>>> index 000000000..4fb20e127 >>>>>> --- /dev/null >>>>>> +++ b/src/paks/rpz/uninstall.sh >>>>>> @@ -0,0 +1,31 @@ >>>>>> +#!/bin/bash >>>>>> +############################################################################### >>>>>> +# # >>>>>> +# IPFire.org - A linux based firewall # >>>>>> +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # >>>>>> +# # >>>>>> +# This program is free software: you can redistribute it and/or modify # >>>>>> +# it under the terms of the GNU General Public License as published by # >>>>>> +# the Free Software Foundation, either version 3 of the License, or # >>>>>> +# (at your option) any later version. # >>>>>> +# # >>>>>> +# This program is distributed in the hope that it will be useful, # >>>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >>>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >>>>>> +# GNU General Public License for more details. # >>>>>> +# # >>>>>> +# You should have received a copy of the GNU General Public License # >>>>>> +# along with this program. If not, see <http://www.gnu.org/licenses/>. # >>>>>> +# # >>>>>> +############################################################################### >>>>>> +# >>>>>> +. /opt/pakfire/lib/functions.sh >>>>>> + >>>>>> +# stop unbound to delete RPZ conf file >>>>>> +/etc/init.d/unbound stop >>>>>> + >>>>>> +make_backup ${NAME} >>>>>> +remove_files >>>>>> + >>>>>> +# start unbound to load unbound config file >>>>>> +/etc/init.d/unbound start >>>>>> diff --git a/src/paks/rpz/update.sh b/src/paks/rpz/update.sh >>>>>> new file mode 100644 >>>>>> index 000000000..938a93a40 >>>>>> --- /dev/null >>>>>> +++ b/src/paks/rpz/update.sh >>>>>> @@ -0,0 +1,25 @@ >>>>>> +#!/bin/bash >>>>>> +############################################################################### >>>>>> +# # >>>>>> +# IPFire.org - A linux based firewall # >>>>>> +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # >>>>>> +# # >>>>>> +# This program is free software: you can redistribute it and/or modify # >>>>>> +# it under the terms of the GNU General Public License as published by # >>>>>> +# the Free Software Foundation, either version 3 of the License, or # >>>>>> +# (at your option) any later version. # >>>>>> +# # >>>>>> +# This program is distributed in the hope that it will be useful, # >>>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >>>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >>>>>> +# GNU General Public License for more details. # >>>>>> +# # >>>>>> +# You should have received a copy of the GNU General Public License # >>>>>> +# along with this program. If not, see <http://www.gnu.org/licenses/>. # >>>>>> +# # >>>>>> +############################################################################### >>>>>> +# >>>>>> +. /opt/pakfire/lib/functions.sh >>>>>> +extract_backup_includes >>>>>> +./uninstall.sh >>>>>> +./install.sh >>>>>> -- >>>>>> 2.30.2
Hello Jon, > On 15 Aug 2024, at 23:22, jon <jon.murphy@ipfire.org> wrote: > > Comments below… > > >> On Aug 15, 2024, at 1:33 PM, Michael Tremer <michael.tremer@ipfire.org> wrote: >> >> Hello, >> >>> On 15 Aug 2024, at 17:57, jon <jon.murphy@ipfire.org> wrote: >>> >>> >>> Comments below. >>> >>> Jon >>> >>> >>>> On Aug 15, 2024, at 10:33 AM, Michael Tremer <michael.tremer@ipfire.org> wrote: >>>> >>>> Hello, >>>> >>>>> On 14 Aug 2024, at 18:14, jon <jon.murphy@ipfire.org> wrote: >>>>> >>>>> Michael, >>>>> >>>>> Sorry for putting you on the spot, but what do you want to do with this RPZ add-on? >>>> >>>> I am not sure this is a question for only me. >>>> >>>> I personally just don’t have any capacity to take on another rather large project as I have a huge backlog of so many things and I feel like a lonely fighter getting these all over the line. For my own sanity I need to have a couple of those closed before thinking about the next ones. >>>> >>>> A couple of months ago we have talked on the monthly call about RPZ and the decision that was made by the people was to not look at this now, just because there were other more pressing things. We could also not find answers to the questions that led us into RPZ: >>>> >>>> * Are there any good lists out there that would allow us to replace the URL Filter? That thing is basically on its last leg because of the lack of lists. We need to be able to block ads and pr0n and that very reliably. >>> >>> I believe this exists with the Hagezi lists at https://github.com/hagezi/dns-blocklists and/or the addition of a chosen "oisd NSFW" list at https://oisd.nl/setup >>> >>> I understand there is no time to review to see if this meet the needs. So we can leave this as "I believe these exist" for now. >> >> So, I am not sure what to think of all these things on GitHub. It seems that there are many repositories that pop up and presumably disappear in the same kind of way. We see this with the IP blocklists. >> >> We want quality blocklists. I know that we don’t have too much on the URL Filter front, but I would prefer to avoid putting so much work in if we in the end stand where we started. > > This makes perfect sense. Based on the comments I’ve seen it seems like Hagezi is growing in popularity and in offerings. > > Gerd (the owner) is very responsive, involved and I keep running into his posts when searching for RPZ "things". Gerd quickly made changes per my request: see https://github.com/hagezi/dns-blocklists/issues/2301. Now there are two TLD RPZ lists. > > Like everything else I cannot predict how long it might last. If you have such a good connection, could you do me the favour and ask why the prefer distributing those lists using HTTPS and not DNS? Is it simply that GitHub is free and does not offer DNS? :) >> >>>> >>>> * We need to look into privacy when RPZs are being realised over DNS - I don’t even understand why we are suddenly starting to pull text files over HTTP again. The IPS seems to have most of these lists already. >>> >>> All of the lists that I am pull are HTTPS. And HTTPS only is currently checked (validated) in the existing `rpz-config` code. >> >> What a lost case. DNS is a globally distributed, cacheable protocol. We could simply put all those blacklists into the DNS caches all around the world. They would respond fast, be local to so many people, but instead we download massive text files. That means that whenever something changes the entire file needs to be transferred again. They are huge. We would want many fast updates which are not possible that way… but I digress. > > I came across one DNS service that offers the Hagezi lists as part of their available DNS services. If interested I can find and post the details. I run a global DNS system where we could easily add this all to. This would scale like hell. It would be a fun project. Just for fun, I have imported "jpgpi250.github.io <http://jpgpi250.github.io/>” which you can pull from either dfw.lwldns.net <http://dfw.lwldns.net/> or haj.lwldns.net <http://haj.lwldns.net/>. This would be so much better since DNS zone transfer checks the serial of the zone and does not retransfer it if it has not changed (and I suppose none of these lists change that frequently). Sadly our DNS system does not support this, but there is also IXFR for incremental zone updates where you would only load the changes. > > >> >> How much traffic did you observe? > > > I have 11 RPZ list (honestly too many but I am experimenting) and that eat up about 230M bytes per day. Oh wow that is huuuuuge. I assume that is repeat downloads? How large are the zone files just downloaded once? >> How is Unbound refreshing these? > > > Unbound RPZ takes care of the updates automagically. There is no cron entry. > > Each RPZ list includes an SOA line similar to this: > `@ SOA localhost. root.localhost. 1723093380 43200 3600 259200 300` > > The 43200 is the refresh period in seconds. So this RPZ list auto updates every 12 hours. > > >> Is there a local cache? > > Yes, it is part of unbound. (Nothing we need to setup) Would it use the cache for zones transferred over DNS, too? > > >> >>>> Regarding your code, there are some issues with the coding style, but I heavily appreciate the pioneering to bring this feature to life. >>> >>> My style is camelCase, but if you prefer snake_case that is an easy change (please speak up). >>> >>> As for as other style changes, that can wait until time is available by the core developers. >> >> It is mainly that you make things very complicated for yourself. Call “cat”, not “/bin/cat”. There is no guarantee that things will stay where they are now and the shell has lots of builtins that execute a lot faster. > > I will remove the executable paths. What builtin did you see that I missed? Anything else? A builtin is just a command that really isn’t one. So there is a /bin/echo somewhere, but when you just write “echo” in the shell, it will simply output the string itself. That is a lot faster than forking a new process. That is all. >> >> And this is all designed to work on the shell. We need something for the web UI. > > For commands within the WegGUI there are `safe_system` commands on the `.cgi` page. > > But I am guess you are referring to something else. That was mainly for the metrics… We would want those to be shown on the web UI and the tool that you currently have outputs it on the shell. That is what that meant. >> >>>> What it would need to be finally merged would be a web UI though. So I think we have a long way ahead of us. >>> >>> The attachment is an early mock-up I made a few months ago (made with Pixelmator Pro). The WebGUI is being worked on now. >>> >>> <mock-up.png> >> >> I think we might look for something that is very similar to the IPS ruleset editor or the IP blocklists. >> >> I am not even sure if we would require users to manage this or whether we would curate a list like with the IPS and IP blocklists. > > The first "DRAFT" of this was for the user/admin to manage the list. For the RPZ add-on, I’d prefer to keep it this way until we figure out if RPZ is popular and used by the user/admins. > > And yes, it could be changed to a set list like IPS. FYI - there are 34 different Hagezi list. So it will take some experiments to pick a "the" list. Well it doesn’t have to be just one. People might want to block different things… It just seems that they come as a package “Light”, “Normal”, “Pro”, “Pro++” and it is either that or nothing. In URL Filter you would choose the categories more freely I would say. -Michael > Also, here is current DRAFT (first draft) of user/admin managed WebGUI. See attachment. > > <Screenshot 2024-08-11 at 2.07.19 PM copy.png> > > > Jon > > >> >>> Does this help? >>> Jon >>> >>> >>>> >>>> -Michael >>>> >>>>> I saw your comments in the Dev Mailing List of "generally being in favor of trying this path" (bad paraphrasing on my part) >>>>> >>>>> I saw your comments in bugzilla at https://bugzilla.ipfire.org/show_bug.cgi?id=13254#c171 >>>>>> I am not interested in anything regarding the RPZs right now. They have not been properly put on the agenda and looking at how much time we have on our hands, this won't make it on the agenda for years. >>>>>> >>>>>> I don't want to build blockers, but this ticket is about a different problem which I want to solve first. >>>>> >>>>> >>>>> How do you want to go forward? >>>>> >>>>> >>>>> Jon >>>>> >>>>> >>>>>> On Aug 12, 2024, at 2:11 PM, jon <jon.murphy@ipfire.org> wrote: >>>>>> >>>>>> More questions! >>>>>> >>>>>> Currently RPZ config files are at `/etc/unbound/local.d` but this directory seems like it is for user (admin) customizations. >>>>>> >>>>>> ``` >>>>>> [root@ipfire ~] # ls -al /etc/unbound/local.d >>>>>> total 68 >>>>>> drwxr-xr-x 2 nobody nobody 4096 Aug 12 13:41 . >>>>>> drwxr-xr-x 4 root root 4096 Aug 12 00:52 .. >>>>>> -rw-r--r-- 1 nobody nobody 436 Jul 12 15:45 00-rpz.conf >>>>>> -rw-r--r-- 1 nobody nobody 285 Mar 1 22:12 AmazonTrkrHZ.rpz.conf >>>>>> -rw-r--r-- 1 nobody nobody 281 Mar 1 22:02 AppleTrkrHZ.rpz.conf >>>>>> -rw-r--r-- 1 nobody nobody 269 Mar 1 21:40 DOHblockHZ.rpz.conf >>>>>> ... >>>>>> -rw-r--r-- 1 nobody nobody 299 Aug 1 19:42 WinTrkrHZ.rpz.conf >>>>>> [root@ipfire ~] # >>>>>> ``` >>>>>> >>>>>> >>>>>> Each file is a config file per category (or one per RPZ file). This makes it easy to add or remove a category (or RPZ file). >>>>>> >>>>>> Should I create a new unbound directory for RPZ config files? Maybe `/etc/unbound/rpz.d`? Or `/etc/unbound/rpz`? >>>>>> >>>>>> >>>>>> Jon >>>>>> >>>>>> >>>>>>> On Aug 1, 2024, at 1:45 PM, Jon Murphy <jon.murphy@ipfire.org> wrote: >>>>>>> >>>>>>> changed all paths from `/var/ipfire/rpz/` to `/var/ipfire/dns/rpz/` >>>>>>> (thank you to Adolf!) >>>>>>> >>>>>>> rpz-config: >>>>>>> - bug: corrected "Type" test from block to allow >>>>>>> - removed verbose parameter from various commands >>>>>>> >>>>>>> rpz-metrics: >>>>>>> - bug: corrected grep for rpz name count >>>>>>> - bug: fixed divide by zero error (thank you Peppe!) >>>>>>> >>>>>>> install/uninstall: >>>>>>> - bug: corrected scripts (thank you Bernhard!) >>>>>>> >>>>>>> Signed-off-by: Jon Murphy <jon.murphy@ipfire.org> >>>>>>> --- >>>>>>> config/backup/includes/rpz | 4 ++-- >>>>>>> config/rootfiles/packages/rpz | 6 +++--- >>>>>>> config/rpz/rpz-config | 14 +++++++------- >>>>>>> config/rpz/rpz-metrics | 9 +++++---- >>>>>>> lfs/rpz | 6 +++--- >>>>>>> src/paks/rpz/install.sh | 27 +++++++++++++++++++++++++++ >>>>>>> src/paks/rpz/uninstall.sh | 31 +++++++++++++++++++++++++++++++ >>>>>>> src/paks/rpz/update.sh | 25 +++++++++++++++++++++++++ >>>>>>> 8 files changed, 103 insertions(+), 19 deletions(-) >>>>>>> create mode 100644 src/paks/rpz/install.sh >>>>>>> create mode 100644 src/paks/rpz/uninstall.sh >>>>>>> create mode 100644 src/paks/rpz/update.sh >>>>>>> >>>>>>> diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz >>>>>>> index 4d59bb40c..8c7410ebd 100644 >>>>>>> --- a/config/backup/includes/rpz >>>>>>> +++ b/config/backup/includes/rpz >>>>>>> @@ -1,5 +1,5 @@ >>>>>>> -/var/ipfire/rpz/allowlist >>>>>>> -/var/ipfire/rpz/blocklist >>>>>>> +/var/ipfire/dns/rpz/allowlist >>>>>>> +/var/ipfire/dns/rpz/blocklist >>>>>>> /etc/unbound/zonefiles/allow.rpz >>>>>>> /etc/unbound/zonefiles/block.rpz >>>>>>> /etc/unbound/local.d/*rpz.conf >>>>>>> diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/rpz >>>>>>> index 2ffa715dd..183825362 100644 >>>>>>> --- a/config/rootfiles/packages/rpz >>>>>>> +++ b/config/rootfiles/packages/rpz >>>>>>> @@ -6,6 +6,6 @@ usr/sbin/rpz-config >>>>>>> usr/sbin/rpz-metrics >>>>>>> usr/sbin/rpz-sleep >>>>>>> var/ipfire/backup/addons/includes/rpz >>>>>>> -var/ipfire/rpz >>>>>>> -var/ipfire/rpz/allowlist >>>>>>> -var/ipfire/rpz/blocklist >>>>>>> +var/ipfire/dns/rpz >>>>>>> +var/ipfire/dns/rpz/allowlist >>>>>>> +var/ipfire/dns/rpz/blocklist >>>>>>> diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config >>>>>>> index 98dc0a4ca..a24a5c132 100644 >>>>>>> --- a/config/rpz/rpz-config >>>>>>> +++ b/config/rpz/rpz-config >>>>>>> @@ -19,7 +19,7 @@ >>>>>>> # # >>>>>>> ############################################################################### >>>>>>> >>>>>>> -# v22 - 2024-07-12 >>>>>>> +# v23 - 2024-07-30 >>>>>>> >>>>>>> ############### Functions ############### >>>>>>> >>>>>>> @@ -54,11 +54,11 @@ check_unbound_conf () { >>>>>>> make_rpz_file () { >>>>>>> local theType="${1}" # allow or block >>>>>>> >>>>>>> - theList="/var/ipfire/rpz/${theType}list" # input user list of domains >>>>>>> + theList="/var/ipfire/dns/rpz/${theType}list" # input custom list of domains >>>>>>> theZoneFile="/etc/unbound/zonefiles/${theType}.rpz" # output file for RPZ >>>>>>> >>>>>>> theAction='.' >>>>>>> - if [[ "${theType}" =~ "block" ]] ; then >>>>>>> + if [[ "${theType}" =~ "allow" ]] ; then >>>>>>> theAction='rpz-passthru.' >>>>>>> fi >>>>>>> >>>>>>> @@ -131,8 +131,8 @@ case "${theAction}" in >>>>>>> # set-up zone file >>>>>>> /usr/bin/touch "${rpzFile}" >>>>>>> # unbound requires these settings for rpz files >>>>>>> - /bin/chown --verbose nobody:nobody "${rpzFile}" >>>>>>> - /bin/chmod --verbose 644 "${rpzFile}" >>>>>>> + /bin/chown nobody:nobody "${rpzFile}" >>>>>>> + /bin/chmod 644 "${rpzFile}" >>>>>>> ;; >>>>>>> >>>>>>> # trash config file & rpz file >>>>>>> @@ -143,8 +143,8 @@ case "${theAction}" in >>>>>>> fi >>>>>>> >>>>>>> msg_log "info: rpz: remove config file & rpz file \"${theName}\"" >>>>>>> - /bin/rm --verbose "${rpzConfig}" >>>>>>> - /bin/rm --verbose "${rpzFile}" >>>>>>> + /bin/rm "${rpzConfig}" >>>>>>> + /bin/rm "${rpzFile}" >>>>>>> >>>>>>> check_unbound_conf >>>>>>> ;; >>>>>>> diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics >>>>>>> index 0f97c7911..4d932726e 100644 >>>>>>> --- a/config/rpz/rpz-metrics >>>>>>> +++ b/config/rpz/rpz-metrics >>>>>>> @@ -19,7 +19,7 @@ >>>>>>> # # >>>>>>> ############################################################################### >>>>>>> >>>>>>> -# v18 on 2024-07-05 >>>>>>> +# v19 on 2024-07-30 >>>>>>> >>>>>>> ############### Main ############### >>>>>>> >>>>>>> @@ -33,7 +33,7 @@ messageLogs=$( find /var/log/messages* -type f | >>>>>>> >>>>>>> # get the list of RPZ names & counts from the message log(s) >>>>>>> rpzNameCount=$( for logf in ${messageLogs} ; do >>>>>>> - /usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" | >>>>>>> + /usr/bin/zgrep --text --extended-regexp 'info: rpz: applied.* A IN$' "${logf}" | >>>>>>> /usr/bin/awk '$10 ~ /\[\w*]/ { print $10 }' ; >>>>>>> done | /usr/bin/sort | /usr/bin/uniq --count ) >>>>>>> >>>>>>> @@ -107,8 +107,9 @@ do >>>>>>> theLines=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' ) >>>>>>> totalLines=$(( totalLines + theLines )) >>>>>>> >>>>>>> - #hitsPerLine=$( echo "scale=0 ; $theHits / $theLines" | bc ) >>>>>>> - hitsPerLine=$(( 100 * theHits / theLines )) >>>>>>> + if [[ "${theLines}" -gt 2 ]] ; then >>>>>>> + hitsPerLine=$(( 100 * theHits / theLines )) >>>>>>> + fi >>>>>>> fi >>>>>>> >>>>>>> # get modification date >>>>>>> diff --git a/lfs/rpz b/lfs/rpz >>>>>>> index 319c10b7f..73f6f2b1b 100644 >>>>>>> --- a/lfs/rpz >>>>>>> +++ b/lfs/rpz >>>>>>> @@ -67,9 +67,9 @@ $(TARGET) : >>>>>>> $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin >>>>>>> >>>>>>> # Install settings folder and two empty files >>>>>>> - mkdir -pv /var/ipfire/rpz >>>>>>> - touch /var/ipfire/rpz/allowlist >>>>>>> - touch /var/ipfire/rpz/blocklist >>>>>>> + mkdir -pv /var/ipfire/dns/rpz >>>>>>> + touch /var/ipfire/dns/rpz/allowlist >>>>>>> + touch /var/ipfire/dns/rpz/blocklist >>>>>>> >>>>>>> # Add conf file to /etc directory >>>>>>> cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d >>>>>>> diff --git a/src/paks/rpz/install.sh b/src/paks/rpz/install.sh >>>>>>> new file mode 100644 >>>>>>> index 000000000..0a797e158 >>>>>>> --- /dev/null >>>>>>> +++ b/src/paks/rpz/install.sh >>>>>>> @@ -0,0 +1,27 @@ >>>>>>> +#!/bin/bash >>>>>>> +############################################################################### >>>>>>> +# # >>>>>>> +# IPFire.org - A linux based firewall # >>>>>>> +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # >>>>>>> +# # >>>>>>> +# This program is free software: you can redistribute it and/or modify # >>>>>>> +# it under the terms of the GNU General Public License as published by # >>>>>>> +# the Free Software Foundation, either version 3 of the License, or # >>>>>>> +# (at your option) any later version. # >>>>>>> +# # >>>>>>> +# This program is distributed in the hope that it will be useful, # >>>>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >>>>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >>>>>>> +# GNU General Public License for more details. # >>>>>>> +# # >>>>>>> +# You should have received a copy of the GNU General Public License # >>>>>>> +# along with this program. If not, see <http://www.gnu.org/licenses/>. # >>>>>>> +# # >>>>>>> +############################################################################### >>>>>>> +# >>>>>>> +. /opt/pakfire/lib/functions.sh >>>>>>> +extract_files >>>>>>> +restore_backup ${NAME} >>>>>>> + >>>>>>> +# restart unbound to load config file >>>>>>> +/etc/init.d/unbound restart >>>>>>> diff --git a/src/paks/rpz/uninstall.sh b/src/paks/rpz/uninstall.sh >>>>>>> new file mode 100644 >>>>>>> index 000000000..4fb20e127 >>>>>>> --- /dev/null >>>>>>> +++ b/src/paks/rpz/uninstall.sh >>>>>>> @@ -0,0 +1,31 @@ >>>>>>> +#!/bin/bash >>>>>>> +############################################################################### >>>>>>> +# # >>>>>>> +# IPFire.org - A linux based firewall # >>>>>>> +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # >>>>>>> +# # >>>>>>> +# This program is free software: you can redistribute it and/or modify # >>>>>>> +# it under the terms of the GNU General Public License as published by # >>>>>>> +# the Free Software Foundation, either version 3 of the License, or # >>>>>>> +# (at your option) any later version. # >>>>>>> +# # >>>>>>> +# This program is distributed in the hope that it will be useful, # >>>>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >>>>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >>>>>>> +# GNU General Public License for more details. # >>>>>>> +# # >>>>>>> +# You should have received a copy of the GNU General Public License # >>>>>>> +# along with this program. If not, see <http://www.gnu.org/licenses/>. # >>>>>>> +# # >>>>>>> +############################################################################### >>>>>>> +# >>>>>>> +. /opt/pakfire/lib/functions.sh >>>>>>> + >>>>>>> +# stop unbound to delete RPZ conf file >>>>>>> +/etc/init.d/unbound stop >>>>>>> + >>>>>>> +make_backup ${NAME} >>>>>>> +remove_files >>>>>>> + >>>>>>> +# start unbound to load unbound config file >>>>>>> +/etc/init.d/unbound start >>>>>>> diff --git a/src/paks/rpz/update.sh b/src/paks/rpz/update.sh >>>>>>> new file mode 100644 >>>>>>> index 000000000..938a93a40 >>>>>>> --- /dev/null >>>>>>> +++ b/src/paks/rpz/update.sh >>>>>>> @@ -0,0 +1,25 @@ >>>>>>> +#!/bin/bash >>>>>>> +############################################################################### >>>>>>> +# # >>>>>>> +# IPFire.org - A linux based firewall # >>>>>>> +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # >>>>>>> +# # >>>>>>> +# This program is free software: you can redistribute it and/or modify # >>>>>>> +# it under the terms of the GNU General Public License as published by # >>>>>>> +# the Free Software Foundation, either version 3 of the License, or # >>>>>>> +# (at your option) any later version. # >>>>>>> +# # >>>>>>> +# This program is distributed in the hope that it will be useful, # >>>>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >>>>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >>>>>>> +# GNU General Public License for more details. # >>>>>>> +# # >>>>>>> +# You should have received a copy of the GNU General Public License # >>>>>>> +# along with this program. If not, see <http://www.gnu.org/licenses/>. # >>>>>>> +# # >>>>>>> +############################################################################### >>>>>>> +# >>>>>>> +. /opt/pakfire/lib/functions.sh >>>>>>> +extract_backup_includes >>>>>>> +./uninstall.sh >>>>>>> +./install.sh >>>>>>> -- >>>>>>> 2.30.2
Hi Michael, > On Aug 21, 2024, at 5:03 AM, Michael Tremer <michael.tremer@ipfire.org> wrote: > > Hello Jon, > >> On 15 Aug 2024, at 23:22, jon <jon.murphy@ipfire.org> wrote: >> >> Comments below… >> >> >>> On Aug 15, 2024, at 1:33 PM, Michael Tremer <michael.tremer@ipfire.org> wrote: >>> >>> Hello, >>> >>>> On 15 Aug 2024, at 17:57, jon <jon.murphy@ipfire.org> wrote: >>>> >>>> >>>> Comments below. >>>> >>>> Jon >>>> >>>> >>>>> On Aug 15, 2024, at 10:33 AM, Michael Tremer <michael.tremer@ipfire.org> wrote: >>>>> >>>>> Hello, >>>>> >>>>>> On 14 Aug 2024, at 18:14, jon <jon.murphy@ipfire.org> wrote: >>>>>> >>>>>> Michael, >>>>>> >>>>>> Sorry for putting you on the spot, but what do you want to do with this RPZ add-on? >>>>> >>>>> I am not sure this is a question for only me. >>>>> >>>>> I personally just don’t have any capacity to take on another rather large project as I have a huge backlog of so many things and I feel like a lonely fighter getting these all over the line. For my own sanity I need to have a couple of those closed before thinking about the next ones. >>>>> >>>>> A couple of months ago we have talked on the monthly call about RPZ and the decision that was made by the people was to not look at this now, just because there were other more pressing things. We could also not find answers to the questions that led us into RPZ: >>>>> >>>>> * Are there any good lists out there that would allow us to replace the URL Filter? That thing is basically on its last leg because of the lack of lists. We need to be able to block ads and pr0n and that very reliably. >>>> >>>> I believe this exists with the Hagezi lists at https://github.com/hagezi/dns-blocklists and/or the addition of a chosen "oisd NSFW" list at https://oisd.nl/setup >>>> >>>> I understand there is no time to review to see if this meet the needs. So we can leave this as "I believe these exist" for now. >>> >>> So, I am not sure what to think of all these things on GitHub. It seems that there are many repositories that pop up and presumably disappear in the same kind of way. We see this with the IP blocklists. >>> >>> We want quality blocklists. I know that we don’t have too much on the URL Filter front, but I would prefer to avoid putting so much work in if we in the end stand where we started. >> >> This makes perfect sense. Based on the comments I’ve seen it seems like Hagezi is growing in popularity and in offerings. >> >> Gerd (the owner) is very responsive, involved and I keep running into his posts when searching for RPZ "things". Gerd quickly made changes per my request: see https://github.com/hagezi/dns-blocklists/issues/2301. Now there are two TLD RPZ lists. >> >> Like everything else I cannot predict how long it might last. > > If you have such a good connection, could you do me the favour and ask why the prefer distributing those lists using HTTPS and not DNS? I’d be happy to ask Gerd, but I do not understand what you want me to ask. I guessed it is related to this: https://www.ipfire.org/blog/ipfire-location-a-decentralised-signed-database-in-dns https://www.ipfire.org/location/how-to-use/dns I understand the APIs you had added, but I do not understand how this related to the DNS blocked lists. What might the query do? Look up a requested domain? > Is it simply that GitHub is free and does not offer DNS? :) I did not find GitHub DNS server(s) listed anywhere. Just lots of references on setting up DNS on other servers... Let me know what you want asked, specifically, and I will pass it on to Gerd! > >>> >>>>> >>>>> * We need to look into privacy when RPZs are being realised over DNS - I don’t even understand why we are suddenly starting to pull text files over HTTP again. The IPS seems to have most of these lists already. >>>> >>>> All of the lists that I am pull are HTTPS. And HTTPS only is currently checked (validated) in the existing `rpz-config` code. >>> >>> What a lost case. DNS is a globally distributed, cacheable protocol. We could simply put all those blacklists into the DNS caches all around the world. They would respond fast, be local to so many people, but instead we download massive text files. That means that whenever something changes the entire file needs to be transferred again. They are huge. We would want many fast updates which are not possible that way… but I digress. >> >> I came across one DNS service that offers the Hagezi lists as part of their available DNS services. If interested I can find and post the details. > > I run a global DNS system where we could easily add this all to. This would scale like hell. It would be a fun project. > > Just for fun, I have imported "jpgpi250.github.io <http://jpgpi250.github.io/>” which you can pull from either dfw.lwldns.net<http://dfw.lwldns.net/> or haj.lwldns.net <http://haj.lwldns.net/>. This would be so much better since DNS zone transfer checks the serial of the zone and does not retransfer it if it has not changed (and I suppose none of these lists change that frequently). > Peter’s jpgpi250 DoH rpz list updates one per day near 04:00 CEST (02:00 UTC). A handful of adds or deletes every night. Gerd’s block DoH rpz list updates twice per day as needed. Sometime once per day, sometimes twice per day. > Sadly our DNS system does not support this, I do not know what "global DNS system" is support. Is it not unbound based? There are many different formats to feed to DNS system: https://github.com/hagezi/dns-blocklists?tab=readme-ov-file#outbox_tray-encrypted-dns-servers-only- Maybe one of these might help? > but there is also IXFR for incremental zone updates where you would only load the changes. > >> >> >>> >>> How much traffic did you observe? >> >> >> I have 11 RPZ list (honestly too many but I am experimenting) and that eat up about 230M bytes per day. > > Oh wow that is huuuuuge. I assume that is repeat downloads? How large are the zone files just downloaded once? Here is the size of each file (grapped via `ls -lS /etc/unbound/zonefiles` and snipped): 10,655,321 MxProPlusHZ.rpz 90,909 dohJPG.rpz 76,591 HosterHZ.rpz 72,628 DOHblockHZ.rpz 37,411 urlhaus.rpz 27,863 WinTrkrHZ.rpz 12,333 NotSafeSearchHZ.rpz 7,573 AppleTrkrHZ1.rpz 7,308 tldAggHZ.rpz 2,631 tldHZ.rpz 966 allow.rpz 238 block.rpz > >>> How is Unbound refreshing these? >> >> >> Unbound RPZ takes care of the updates automagically. There is no cron entry. >> >> Each RPZ list includes an SOA line similar to this: >> `@ SOA localhost. root.localhost. 1723093380 43200 3600 259200 300` >> >> The 43200 is the refresh period in seconds. So this RPZ list auto updates every 12 hours. >> >> >>> Is there a local cache? >> >> Yes, it is part of unbound. (Nothing we need to setup) > > Would it use the cache for zones transferred over DNS, too? I do not know. > >> >> >>> >>>>> Regarding your code, there are some issues with the coding style, but I heavily appreciate the pioneering to bring this feature to life. >>>> >>>> My style is camelCase, but if you prefer snake_case that is an easy change (please speak up). >>>> >>>> As for as other style changes, that can wait until time is available by the core developers. >>> >>> It is mainly that you make things very complicated for yourself. Call “cat”, not “/bin/cat”. There is no guarantee that things will stay where they are now and the shell has lots of builtins that execute a lot faster. >> >> I will remove the executable paths. What builtin did you see that I missed? Anything else? > > A builtin is just a command that really isn’t one. So there is a /bin/echo somewhere, but when you just write “echo” in the shell, it will simply output the string itself. That is a lot faster than forking a new process. That is all. Got it! All changed to: echo "multi line string" > "${rpzConfig}" > >>> >>> And this is all designed to work on the shell. We need something for the web UI. >> >> For commands within the WegGUI there are `safe_system` commands on the `.cgi` page. >> >> But I am guess you are referring to something else. > > That was mainly for the metrics… We would want those to be shown on the web UI and the tool that you currently have outputs it on the shell. That is what that meant. For short term the metrics will be via shell only. Long term, agree, metrics via the WebGUI! > >>> >>>>> What it would need to be finally merged would be a web UI though. So I think we have a long way ahead of us. >>>> >>>> The attachment is an early mock-up I made a few months ago (made with Pixelmator Pro). The WebGUI is being worked on now. >>>> >>>> <mock-up.png> >>> >>> I think we might look for something that is very similar to the IPS ruleset editor or the IP blocklists. >>> >>> I am not even sure if we would require users to manage this or whether we would curate a list like with the IPS and IP blocklists. >> >> The first "DRAFT" of this was for the user/admin to manage the list. For the RPZ add-on, I’d prefer to keep it this way until we figure out if RPZ is popular and used by the user/admins. >> >> And yes, it could be changed to a set list like IPS. FYI - there are 34 different Hagezi list. So it will take some experiments to pick a "the" list. > > Well it doesn’t have to be just one. People might want to block different things… Agree! > > It just seems that they come as a package “Light”, “Normal”, “Pro”, “Pro++” and it is either that or nothing. More than nothing - Here are the individual (non-package) lists for Hagezi: • Fake - Protects against internet scams, traps & fakes! • Pop-Up Ads - Protects against annoying and malicious pop-up ads! • Threat Intelligence Feeds - Increases security significantly! (Recommended) : Full - Medium - Mini - IPs • Newly Registered Domains - Favoured by threat actors to launch malicious campaigns! : 14 days - 30 days • DoH/VPN/TOR/Proxy Bypass - Prevent methods to bypass your DNS! : Full - DoH only - DoH IPs • Safesearch not supported - Prevent the use of search engines that do not support Safesearch! • Dynamic DNS - Protects against the malicious use of dynamic DNS services! • Badware Hoster - Protects against the malicious use of free host services! • Most Abused TLDs - Protects against known malicious Top Level Domains! • Anti Piracy - Protects against piracy! • Gambling - Protects against gambling content! : Full - Medium - Mini • NSFW (external) - oisd NSFW - Protects against adult content! • Native Tracker - Broadband tracker of devices, services and operating systems And there are others out there also. Do I have your blessing to keep moving forward? Jon PS - Here is the current WebGUI for RPZ. And the EDIT pencil. > -Michael > >> Also, here is current DRAFT (first draft) of user/admin managed WebGUI. See attachment. >> >> <Screenshot 2024-08-11 at 2.07.19 PM copy.png> >> >> >> Jon >> >> >>> >>>> Does this help? >>>> Jon >>>> >>>> >>>>> >>>>> -Michael >>>>> >>>>>> I saw your comments in the Dev Mailing List of "generally being in favor of trying this path" (bad paraphrasing on my part) >>>>>> >>>>>> I saw your comments in bugzilla at https://bugzilla.ipfire.org/show_bug.cgi?id=13254#c171 >>>>>>> I am not interested in anything regarding the RPZs right now. They have not been properly put on the agenda and looking at how much time we have on our hands, this won't make it on the agenda for years. >>>>>>> >>>>>>> I don't want to build blockers, but this ticket is about a different problem which I want to solve first. >>>>>> >>>>>> >>>>>> How do you want to go forward? >>>>>> >>>>>> >>>>>> Jon >>>>>> >>>>>> >>>>>>> On Aug 12, 2024, at 2:11 PM, jon <jon.murphy@ipfire.org> wrote: >>>>>>> >>>>>>> More questions! >>>>>>> >>>>>>> Currently RPZ config files are at `/etc/unbound/local.d` but this directory seems like it is for user (admin) customizations. >>>>>>> >>>>>>> ``` >>>>>>> [root@ipfire ~] # ls -al /etc/unbound/local.d >>>>>>> total 68 >>>>>>> drwxr-xr-x 2 nobody nobody 4096 Aug 12 13:41 . >>>>>>> drwxr-xr-x 4 root root 4096 Aug 12 00:52 .. >>>>>>> -rw-r--r-- 1 nobody nobody 436 Jul 12 15:45 00-rpz.conf >>>>>>> -rw-r--r-- 1 nobody nobody 285 Mar 1 22:12 AmazonTrkrHZ.rpz.conf >>>>>>> -rw-r--r-- 1 nobody nobody 281 Mar 1 22:02 AppleTrkrHZ.rpz.conf >>>>>>> -rw-r--r-- 1 nobody nobody 269 Mar 1 21:40 DOHblockHZ.rpz.conf >>>>>>> ... >>>>>>> -rw-r--r-- 1 nobody nobody 299 Aug 1 19:42 WinTrkrHZ.rpz.conf >>>>>>> [root@ipfire ~] # >>>>>>> ``` >>>>>>> >>>>>>> >>>>>>> Each file is a config file per category (or one per RPZ file). This makes it easy to add or remove a category (or RPZ file). >>>>>>> >>>>>>> Should I create a new unbound directory for RPZ config files? Maybe `/etc/unbound/rpz.d`? Or `/etc/unbound/rpz`? >>>>>>> >>>>>>> >>>>>>> Jon >>>>>>> >>>>>>> >>>>>>>> On Aug 1, 2024, at 1:45 PM, Jon Murphy <jon.murphy@ipfire.org> wrote: >>>>>>>> >>>>>>>> changed all paths from `/var/ipfire/rpz/` to `/var/ipfire/dns/rpz/` >>>>>>>> (thank you to Adolf!) >>>>>>>> >>>>>>>> rpz-config: >>>>>>>> - bug: corrected "Type" test from block to allow >>>>>>>> - removed verbose parameter from various commands >>>>>>>> >>>>>>>> rpz-metrics: >>>>>>>> - bug: corrected grep for rpz name count >>>>>>>> - bug: fixed divide by zero error (thank you Peppe!) >>>>>>>> >>>>>>>> install/uninstall: >>>>>>>> - bug: corrected scripts (thank you Bernhard!) >>>>>>>> >>>>>>>> Signed-off-by: Jon Murphy <jon.murphy@ipfire.org> >>>>>>>> --- >>>>>>>> config/backup/includes/rpz | 4 ++-- >>>>>>>> config/rootfiles/packages/rpz | 6 +++--- >>>>>>>> config/rpz/rpz-config | 14 +++++++------- >>>>>>>> config/rpz/rpz-metrics | 9 +++++---- >>>>>>>> lfs/rpz | 6 +++--- >>>>>>>> src/paks/rpz/install.sh | 27 +++++++++++++++++++++++++++ >>>>>>>> src/paks/rpz/uninstall.sh | 31 +++++++++++++++++++++++++++++++ >>>>>>>> src/paks/rpz/update.sh | 25 +++++++++++++++++++++++++ >>>>>>>> 8 files changed, 103 insertions(+), 19 deletions(-) >>>>>>>> create mode 100644 src/paks/rpz/install.sh >>>>>>>> create mode 100644 src/paks/rpz/uninstall.sh >>>>>>>> create mode 100644 src/paks/rpz/update.sh >>>>>>>> >>>>>>>> diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz >>>>>>>> index 4d59bb40c..8c7410ebd 100644 >>>>>>>> --- a/config/backup/includes/rpz >>>>>>>> +++ b/config/backup/includes/rpz >>>>>>>> @@ -1,5 +1,5 @@ >>>>>>>> -/var/ipfire/rpz/allowlist >>>>>>>> -/var/ipfire/rpz/blocklist >>>>>>>> +/var/ipfire/dns/rpz/allowlist >>>>>>>> +/var/ipfire/dns/rpz/blocklist >>>>>>>> /etc/unbound/zonefiles/allow.rpz >>>>>>>> /etc/unbound/zonefiles/block.rpz >>>>>>>> /etc/unbound/local.d/*rpz.conf >>>>>>>> diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/rpz >>>>>>>> index 2ffa715dd..183825362 100644 >>>>>>>> --- a/config/rootfiles/packages/rpz >>>>>>>> +++ b/config/rootfiles/packages/rpz >>>>>>>> @@ -6,6 +6,6 @@ usr/sbin/rpz-config >>>>>>>> usr/sbin/rpz-metrics >>>>>>>> usr/sbin/rpz-sleep >>>>>>>> var/ipfire/backup/addons/includes/rpz >>>>>>>> -var/ipfire/rpz >>>>>>>> -var/ipfire/rpz/allowlist >>>>>>>> -var/ipfire/rpz/blocklist >>>>>>>> +var/ipfire/dns/rpz >>>>>>>> +var/ipfire/dns/rpz/allowlist >>>>>>>> +var/ipfire/dns/rpz/blocklist >>>>>>>> diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config >>>>>>>> index 98dc0a4ca..a24a5c132 100644 >>>>>>>> --- a/config/rpz/rpz-config >>>>>>>> +++ b/config/rpz/rpz-config >>>>>>>> @@ -19,7 +19,7 @@ >>>>>>>> # # >>>>>>>> ############################################################################### >>>>>>>> >>>>>>>> -# v22 - 2024-07-12 >>>>>>>> +# v23 - 2024-07-30 >>>>>>>> >>>>>>>> ############### Functions ############### >>>>>>>> >>>>>>>> @@ -54,11 +54,11 @@ check_unbound_conf () { >>>>>>>> make_rpz_file () { >>>>>>>> local theType="${1}" # allow or block >>>>>>>> >>>>>>>> - theList="/var/ipfire/rpz/${theType}list" # input user list of domains >>>>>>>> + theList="/var/ipfire/dns/rpz/${theType}list" # input custom list of domains >>>>>>>> theZoneFile="/etc/unbound/zonefiles/${theType}.rpz" # output file for RPZ >>>>>>>> >>>>>>>> theAction='.' >>>>>>>> - if [[ "${theType}" =~ "block" ]] ; then >>>>>>>> + if [[ "${theType}" =~ "allow" ]] ; then >>>>>>>> theAction='rpz-passthru.' >>>>>>>> fi >>>>>>>> >>>>>>>> @@ -131,8 +131,8 @@ case "${theAction}" in >>>>>>>> # set-up zone file >>>>>>>> /usr/bin/touch "${rpzFile}" >>>>>>>> # unbound requires these settings for rpz files >>>>>>>> - /bin/chown --verbose nobody:nobody "${rpzFile}" >>>>>>>> - /bin/chmod --verbose 644 "${rpzFile}" >>>>>>>> + /bin/chown nobody:nobody "${rpzFile}" >>>>>>>> + /bin/chmod 644 "${rpzFile}" >>>>>>>> ;; >>>>>>>> >>>>>>>> # trash config file & rpz file >>>>>>>> @@ -143,8 +143,8 @@ case "${theAction}" in >>>>>>>> fi >>>>>>>> >>>>>>>> msg_log "info: rpz: remove config file & rpz file \"${theName}\"" >>>>>>>> - /bin/rm --verbose "${rpzConfig}" >>>>>>>> - /bin/rm --verbose "${rpzFile}" >>>>>>>> + /bin/rm "${rpzConfig}" >>>>>>>> + /bin/rm "${rpzFile}" >>>>>>>> >>>>>>>> check_unbound_conf >>>>>>>> ;; >>>>>>>> diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics >>>>>>>> index 0f97c7911..4d932726e 100644 >>>>>>>> --- a/config/rpz/rpz-metrics >>>>>>>> +++ b/config/rpz/rpz-metrics >>>>>>>> @@ -19,7 +19,7 @@ >>>>>>>> # # >>>>>>>> ############################################################################### >>>>>>>> >>>>>>>> -# v18 on 2024-07-05 >>>>>>>> +# v19 on 2024-07-30 >>>>>>>> >>>>>>>> ############### Main ############### >>>>>>>> >>>>>>>> @@ -33,7 +33,7 @@ messageLogs=$( find /var/log/messages* -type f | >>>>>>>> >>>>>>>> # get the list of RPZ names & counts from the message log(s) >>>>>>>> rpzNameCount=$( for logf in ${messageLogs} ; do >>>>>>>> - /usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" | >>>>>>>> + /usr/bin/zgrep --text --extended-regexp 'info: rpz: applied.* A IN$' "${logf}" | >>>>>>>> /usr/bin/awk '$10 ~ /\[\w*]/ { print $10 }' ; >>>>>>>> done | /usr/bin/sort | /usr/bin/uniq --count ) >>>>>>>> >>>>>>>> @@ -107,8 +107,9 @@ do >>>>>>>> theLines=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' ) >>>>>>>> totalLines=$(( totalLines + theLines )) >>>>>>>> >>>>>>>> - #hitsPerLine=$( echo "scale=0 ; $theHits / $theLines" | bc ) >>>>>>>> - hitsPerLine=$(( 100 * theHits / theLines )) >>>>>>>> + if [[ "${theLines}" -gt 2 ]] ; then >>>>>>>> + hitsPerLine=$(( 100 * theHits / theLines )) >>>>>>>> + fi >>>>>>>> fi >>>>>>>> >>>>>>>> # get modification date >>>>>>>> diff --git a/lfs/rpz b/lfs/rpz >>>>>>>> index 319c10b7f..73f6f2b1b 100644 >>>>>>>> --- a/lfs/rpz >>>>>>>> +++ b/lfs/rpz >>>>>>>> @@ -67,9 +67,9 @@ $(TARGET) : >>>>>>>> $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin >>>>>>>> >>>>>>>> # Install settings folder and two empty files >>>>>>>> - mkdir -pv /var/ipfire/rpz >>>>>>>> - touch /var/ipfire/rpz/allowlist >>>>>>>> - touch /var/ipfire/rpz/blocklist >>>>>>>> + mkdir -pv /var/ipfire/dns/rpz >>>>>>>> + touch /var/ipfire/dns/rpz/allowlist >>>>>>>> + touch /var/ipfire/dns/rpz/blocklist >>>>>>>> >>>>>>>> # Add conf file to /etc directory >>>>>>>> cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d >>>>>>>> diff --git a/src/paks/rpz/install.sh b/src/paks/rpz/install.sh >>>>>>>> new file mode 100644 >>>>>>>> index 000000000..0a797e158 >>>>>>>> --- /dev/null >>>>>>>> +++ b/src/paks/rpz/install.sh >>>>>>>> @@ -0,0 +1,27 @@ >>>>>>>> +#!/bin/bash >>>>>>>> +############################################################################### >>>>>>>> +# # >>>>>>>> +# IPFire.org - A linux based firewall # >>>>>>>> +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # >>>>>>>> +# # >>>>>>>> +# This program is free software: you can redistribute it and/or modify # >>>>>>>> +# it under the terms of the GNU General Public License as published by # >>>>>>>> +# the Free Software Foundation, either version 3 of the License, or # >>>>>>>> +# (at your option) any later version. # >>>>>>>> +# # >>>>>>>> +# This program is distributed in the hope that it will be useful, # >>>>>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >>>>>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >>>>>>>> +# GNU General Public License for more details. # >>>>>>>> +# # >>>>>>>> +# You should have received a copy of the GNU General Public License # >>>>>>>> +# along with this program. If not, see <http://www.gnu.org/licenses/>. # >>>>>>>> +# # >>>>>>>> +############################################################################### >>>>>>>> +# >>>>>>>> +. /opt/pakfire/lib/functions.sh >>>>>>>> +extract_files >>>>>>>> +restore_backup ${NAME} >>>>>>>> + >>>>>>>> +# restart unbound to load config file >>>>>>>> +/etc/init.d/unbound restart >>>>>>>> diff --git a/src/paks/rpz/uninstall.sh b/src/paks/rpz/uninstall.sh >>>>>>>> new file mode 100644 >>>>>>>> index 000000000..4fb20e127 >>>>>>>> --- /dev/null >>>>>>>> +++ b/src/paks/rpz/uninstall.sh >>>>>>>> @@ -0,0 +1,31 @@ >>>>>>>> +#!/bin/bash >>>>>>>> +############################################################################### >>>>>>>> +# # >>>>>>>> +# IPFire.org - A linux based firewall # >>>>>>>> +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # >>>>>>>> +# # >>>>>>>> +# This program is free software: you can redistribute it and/or modify # >>>>>>>> +# it under the terms of the GNU General Public License as published by # >>>>>>>> +# the Free Software Foundation, either version 3 of the License, or # >>>>>>>> +# (at your option) any later version. # >>>>>>>> +# # >>>>>>>> +# This program is distributed in the hope that it will be useful, # >>>>>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >>>>>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >>>>>>>> +# GNU General Public License for more details. # >>>>>>>> +# # >>>>>>>> +# You should have received a copy of the GNU General Public License # >>>>>>>> +# along with this program. If not, see <http://www.gnu.org/licenses/>. # >>>>>>>> +# # >>>>>>>> +############################################################################### >>>>>>>> +# >>>>>>>> +. /opt/pakfire/lib/functions.sh >>>>>>>> + >>>>>>>> +# stop unbound to delete RPZ conf file >>>>>>>> +/etc/init.d/unbound stop >>>>>>>> + >>>>>>>> +make_backup ${NAME} >>>>>>>> +remove_files >>>>>>>> + >>>>>>>> +# start unbound to load unbound config file >>>>>>>> +/etc/init.d/unbound start >>>>>>>> diff --git a/src/paks/rpz/update.sh b/src/paks/rpz/update.sh >>>>>>>> new file mode 100644 >>>>>>>> index 000000000..938a93a40 >>>>>>>> --- /dev/null >>>>>>>> +++ b/src/paks/rpz/update.sh >>>>>>>> @@ -0,0 +1,25 @@ >>>>>>>> +#!/bin/bash >>>>>>>> +############################################################################### >>>>>>>> +# # >>>>>>>> +# IPFire.org - A linux based firewall # >>>>>>>> +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # >>>>>>>> +# # >>>>>>>> +# This program is free software: you can redistribute it and/or modify # >>>>>>>> +# it under the terms of the GNU General Public License as published by # >>>>>>>> +# the Free Software Foundation, either version 3 of the License, or # >>>>>>>> +# (at your option) any later version. # >>>>>>>> +# # >>>>>>>> +# This program is distributed in the hope that it will be useful, # >>>>>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >>>>>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >>>>>>>> +# GNU General Public License for more details. # >>>>>>>> +# # >>>>>>>> +# You should have received a copy of the GNU General Public License # >>>>>>>> +# along with this program. If not, see <http://www.gnu.org/licenses/>. # >>>>>>>> +# # >>>>>>>> +############################################################################### >>>>>>>> +# >>>>>>>> +. /opt/pakfire/lib/functions.sh >>>>>>>> +extract_backup_includes >>>>>>>> +./uninstall.sh >>>>>>>> +./install.sh >>>>>>>> -- >>>>>>>> 2.30.2 Jon
Morning, > On 22 Aug 2024, at 16:37, jon <jon.murphy@ipfire.org> wrote: > > Hi Michael, > > >> On Aug 21, 2024, at 5:03 AM, Michael Tremer <michael.tremer@ipfire.org> wrote: >> >> Hello Jon, >> >>> On 15 Aug 2024, at 23:22, jon <jon.murphy@ipfire.org> wrote: >>> >>> Comments below… >>> >>> >>>> On Aug 15, 2024, at 1:33 PM, Michael Tremer <michael.tremer@ipfire.org> wrote: >>>> >>>> Hello, >>>> >>>>> On 15 Aug 2024, at 17:57, jon <jon.murphy@ipfire.org> wrote: >>>>> >>>>> >>>>> Comments below. >>>>> >>>>> Jon >>>>> >>>>> >>>>>> On Aug 15, 2024, at 10:33 AM, Michael Tremer <michael.tremer@ipfire.org> wrote: >>>>>> >>>>>> Hello, >>>>>> >>>>>>> On 14 Aug 2024, at 18:14, jon <jon.murphy@ipfire.org> wrote: >>>>>>> >>>>>>> Michael, >>>>>>> >>>>>>> Sorry for putting you on the spot, but what do you want to do with this RPZ add-on? >>>>>> >>>>>> I am not sure this is a question for only me. >>>>>> >>>>>> I personally just don’t have any capacity to take on another rather large project as I have a huge backlog of so many things and I feel like a lonely fighter getting these all over the line. For my own sanity I need to have a couple of those closed before thinking about the next ones. >>>>>> >>>>>> A couple of months ago we have talked on the monthly call about RPZ and the decision that was made by the people was to not look at this now, just because there were other more pressing things. We could also not find answers to the questions that led us into RPZ: >>>>>> >>>>>> * Are there any good lists out there that would allow us to replace the URL Filter? That thing is basically on its last leg because of the lack of lists. We need to be able to block ads and pr0n and that very reliably. >>>>> >>>>> I believe this exists with the Hagezi lists at https://github.com/hagezi/dns-blocklists and/or the addition of a chosen "oisd NSFW" list at https://oisd.nl/setup >>>>> >>>>> I understand there is no time to review to see if this meet the needs. So we can leave this as "I believe these exist" for now. >>>> >>>> So, I am not sure what to think of all these things on GitHub. It seems that there are many repositories that pop up and presumably disappear in the same kind of way. We see this with the IP blocklists. >>>> >>>> We want quality blocklists. I know that we don’t have too much on the URL Filter front, but I would prefer to avoid putting so much work in if we in the end stand where we started. >>> >>> This makes perfect sense. Based on the comments I’ve seen it seems like Hagezi is growing in popularity and in offerings. >>> >>> Gerd (the owner) is very responsive, involved and I keep running into his posts when searching for RPZ "things". Gerd quickly made changes per my request: see https://github.com/hagezi/dns-blocklists/issues/2301. Now there are two TLD RPZ lists. >>> >>> Like everything else I cannot predict how long it might last. >> >> If you have such a good connection, could you do me the favour and ask why the prefer distributing those lists using HTTPS and not DNS? > > I’d be happy to ask Gerd, but I do not understand what you want me to ask. It sounded like you have a personal connection. That is all. > I guessed it is related to this: > https://www.ipfire.org/blog/ipfire-location-a-decentralised-signed-database-in-dns > https://www.ipfire.org/location/how-to-use/dns > > I understand the APIs you had added, but I do not understand how this related to the DNS blocked lists. Yes, this is slightly related to this. I am just running a network of authoritative DNS servers scattered a little bit across the world. It is an essential part of the IPFire infrastructure but also hosts a couple of other things. And since it is there, it is a nice thing to bolt on more things. > What might the query do? Look up a requested domain? The relationship to RPZs is that we could use that service to host them. It is a DNS-based technology after all. >> Is it simply that GitHub is free and does not offer DNS? :) > > I did not find GitHub DNS server(s) listed anywhere. Just lots of references on setting up DNS on other servers... No, GitHub does not offer any DNS services. > Let me know what you want asked, specifically, and I will pass it on to Gerd! The question is only whether it was a conscious decision to distribute the lists over HTTP instead of using DNS. Maybe it was just chosen because that is what was available to them. >> >>>> >>>>>> >>>>>> * We need to look into privacy when RPZs are being realised over DNS - I don’t even understand why we are suddenly starting to pull text files over HTTP again. The IPS seems to have most of these lists already. >>>>> >>>>> All of the lists that I am pull are HTTPS. And HTTPS only is currently checked (validated) in the existing `rpz-config` code. >>>> >>>> What a lost case. DNS is a globally distributed, cacheable protocol. We could simply put all those blacklists into the DNS caches all around the world. They would respond fast, be local to so many people, but instead we download massive text files. That means that whenever something changes the entire file needs to be transferred again. They are huge. We would want many fast updates which are not possible that way… but I digress. >>> >>> I came across one DNS service that offers the Hagezi lists as part of their available DNS services. If interested I can find and post the details. >> >> I run a global DNS system where we could easily add this all to. This would scale like hell. It would be a fun project. >> >> Just for fun, I have imported "jpgpi250.github.io <http://jpgpi250.github.io/>” which you can pull from either dfw.lwldns.net<http://dfw.lwldns.net/> or haj.lwldns.net <http://haj.lwldns.net/>. This would be so much better since DNS zone transfer checks the serial of the zone and does not retransfer it if it has not changed (and I suppose none of these lists change that frequently). >> > > Peter’s jpgpi250 DoH rpz list updates one per day near 04:00 CEST (02:00 UTC). A handful of adds or deletes every night. > > Gerd’s block DoH rpz list updates twice per day as needed. Sometime once per day, sometimes twice per day. Yeah, that is not a lot, so DNS would qualify to distribute the lists. >> Sadly our DNS system does not support this, > > I do not know what "global DNS system" is support. Is it not unbound based? No those servers are running PowerDNS behind a load-balancer. Unbound is just a recursor and not an authoritative DNS server. > There are many different formats to feed to DNS system: > https://github.com/hagezi/dns-blocklists?tab=readme-ov-file#outbox_tray-encrypted-dns-servers-only- > > Maybe one of these might help? > > >> but there is also IXFR for incremental zone updates where you would only load the changes. > > >> >>> >>> >>>> >>>> How much traffic did you observe? >>> >>> >>> I have 11 RPZ list (honestly too many but I am experimenting) and that eat up about 230M bytes per day. >> >> Oh wow that is huuuuuge. I assume that is repeat downloads? How large are the zone files just downloaded once? > > Here is the size of each file (grapped via `ls -lS /etc/unbound/zonefiles` and snipped): > > 10,655,321 MxProPlusHZ.rpz > 90,909 dohJPG.rpz > 76,591 HosterHZ.rpz > 72,628 DOHblockHZ.rpz > 37,411 urlhaus.rpz > 27,863 WinTrkrHZ.rpz > 12,333 NotSafeSearchHZ.rpz > 7,573 AppleTrkrHZ1.rpz > 7,308 tldAggHZ.rpz > 2,631 tldHZ.rpz > 966 allow.rpz > 238 block.rpz > > >> >>>> How is Unbound refreshing these? >>> >>> >>> Unbound RPZ takes care of the updates automagically. There is no cron entry. >>> >>> Each RPZ list includes an SOA line similar to this: >>> `@ SOA localhost. root.localhost. 1723093380 43200 3600 259200 300` >>> >>> The 43200 is the refresh period in seconds. So this RPZ list auto updates every 12 hours. >>> >>> >>>> Is there a local cache? >>> >>> Yes, it is part of unbound. (Nothing we need to setup) >> >> Would it use the cache for zones transferred over DNS, too? > > I do not know. > >> >>> >>> >>>> >>>>>> Regarding your code, there are some issues with the coding style, but I heavily appreciate the pioneering to bring this feature to life. >>>>> >>>>> My style is camelCase, but if you prefer snake_case that is an easy change (please speak up). >>>>> >>>>> As for as other style changes, that can wait until time is available by the core developers. >>>> >>>> It is mainly that you make things very complicated for yourself. Call “cat”, not “/bin/cat”. There is no guarantee that things will stay where they are now and the shell has lots of builtins that execute a lot faster. >>> >>> I will remove the executable paths. What builtin did you see that I missed? Anything else? >> >> A builtin is just a command that really isn’t one. So there is a /bin/echo somewhere, but when you just write “echo” in the shell, it will simply output the string itself. That is a lot faster than forking a new process. That is all. > > > Got it! All changed to: > > echo "multi line string" > "${rpzConfig}" > > >> >>>> >>>> And this is all designed to work on the shell. We need something for the web UI. >>> >>> For commands within the WegGUI there are `safe_system` commands on the `.cgi` page. >>> >>> But I am guess you are referring to something else. >> >> That was mainly for the metrics… We would want those to be shown on the web UI and the tool that you currently have outputs it on the shell. That is what that meant. > > For short term the metrics will be via shell only. > > Long term, agree, metrics via the WebGUI! > > >> >>>> >>>>>> What it would need to be finally merged would be a web UI though. So I think we have a long way ahead of us. >>>>> >>>>> The attachment is an early mock-up I made a few months ago (made with Pixelmator Pro). The WebGUI is being worked on now. >>>>> >>>>> <mock-up.png> >>>> >>>> I think we might look for something that is very similar to the IPS ruleset editor or the IP blocklists. >>>> >>>> I am not even sure if we would require users to manage this or whether we would curate a list like with the IPS and IP blocklists. >>> >>> The first "DRAFT" of this was for the user/admin to manage the list. For the RPZ add-on, I’d prefer to keep it this way until we figure out if RPZ is popular and used by the user/admins. >>> >>> And yes, it could be changed to a set list like IPS. FYI - there are 34 different Hagezi list. So it will take some experiments to pick a "the" list. >> >> Well it doesn’t have to be just one. People might want to block different things… > > Agree! > >> >> It just seems that they come as a package “Light”, “Normal”, “Pro”, “Pro++” and it is either that or nothing. > > More than nothing - Here are the individual (non-package) lists for Hagezi: > > • Fake - Protects against internet scams, traps & fakes! > • Pop-Up Ads - Protects against annoying and malicious pop-up ads! > • Threat Intelligence Feeds - Increases security significantly! (Recommended) : Full - Medium - Mini - IPs > • Newly Registered Domains - Favoured by threat actors to launch malicious campaigns! : 14 days - 30 days > • DoH/VPN/TOR/Proxy Bypass - Prevent methods to bypass your DNS! : Full - DoH only - DoH IPs > • Safesearch not supported - Prevent the use of search engines that do not support Safesearch! > • Dynamic DNS - Protects against the malicious use of dynamic DNS services! > • Badware Hoster - Protects against the malicious use of free host services! > • Most Abused TLDs - Protects against known malicious Top Level Domains! > • Anti Piracy - Protects against piracy! > • Gambling - Protects against gambling content! : Full - Medium - Mini > • NSFW (external) - oisd NSFW - Protects against adult content! > • Native Tracker - Broadband tracker of devices, services and operating systems > > And there are others out there also. Others might exist, but as it seems they all mix and match their lists together. So there will a lot of overlap which becomes a waste of memory and data transfer. Looking at that list, I am not sure what I would want to turn on in my home/if I was a school/a generic office company. > Do I have your blessing to keep moving forward? What do you want to move forward with? What are the next steps? -Michael > Jon > > > > PS - Here is the current WebGUI for RPZ. > > <Screenshot 2024-08-22 at 10.28.54 AM.png> > > > > And the EDIT pencil. > > > <Screenshot 2024-08-22 at 10.29.25 AM.png> > > > >> -Michael >> >>> Also, here is current DRAFT (first draft) of user/admin managed WebGUI. See attachment. >>> >>> <Screenshot 2024-08-11 at 2.07.19 PM copy.png> >>> >>> >>> Jon >>> >>> >>>> >>>>> Does this help? >>>>> Jon >>>>> >>>>> >>>>>> >>>>>> -Michael >>>>>> >>>>>>> I saw your comments in the Dev Mailing List of "generally being in favor of trying this path" (bad paraphrasing on my part) >>>>>>> >>>>>>> I saw your comments in bugzilla at https://bugzilla.ipfire.org/show_bug.cgi?id=13254#c171 >>>>>>>> I am not interested in anything regarding the RPZs right now. They have not been properly put on the agenda and looking at how much time we have on our hands, this won't make it on the agenda for years. >>>>>>>> >>>>>>>> I don't want to build blockers, but this ticket is about a different problem which I want to solve first. >>>>>>> >>>>>>> >>>>>>> How do you want to go forward? >>>>>>> >>>>>>> >>>>>>> Jon >>>>>>> >>>>>>> >>>>>>>> On Aug 12, 2024, at 2:11 PM, jon <jon.murphy@ipfire.org> wrote: >>>>>>>> >>>>>>>> More questions! >>>>>>>> >>>>>>>> Currently RPZ config files are at `/etc/unbound/local.d` but this directory seems like it is for user (admin) customizations. >>>>>>>> >>>>>>>> ``` >>>>>>>> [root@ipfire ~] # ls -al /etc/unbound/local.d >>>>>>>> total 68 >>>>>>>> drwxr-xr-x 2 nobody nobody 4096 Aug 12 13:41 . >>>>>>>> drwxr-xr-x 4 root root 4096 Aug 12 00:52 .. >>>>>>>> -rw-r--r-- 1 nobody nobody 436 Jul 12 15:45 00-rpz.conf >>>>>>>> -rw-r--r-- 1 nobody nobody 285 Mar 1 22:12 AmazonTrkrHZ.rpz.conf >>>>>>>> -rw-r--r-- 1 nobody nobody 281 Mar 1 22:02 AppleTrkrHZ.rpz.conf >>>>>>>> -rw-r--r-- 1 nobody nobody 269 Mar 1 21:40 DOHblockHZ.rpz.conf >>>>>>>> ... >>>>>>>> -rw-r--r-- 1 nobody nobody 299 Aug 1 19:42 WinTrkrHZ.rpz.conf >>>>>>>> [root@ipfire ~] # >>>>>>>> ``` >>>>>>>> >>>>>>>> >>>>>>>> Each file is a config file per category (or one per RPZ file). This makes it easy to add or remove a category (or RPZ file). >>>>>>>> >>>>>>>> Should I create a new unbound directory for RPZ config files? Maybe `/etc/unbound/rpz.d`? Or `/etc/unbound/rpz`? >>>>>>>> >>>>>>>> >>>>>>>> Jon >>>>>>>> >>>>>>>> >>>>>>>>> On Aug 1, 2024, at 1:45 PM, Jon Murphy <jon.murphy@ipfire.org> wrote: >>>>>>>>> >>>>>>>>> changed all paths from `/var/ipfire/rpz/` to `/var/ipfire/dns/rpz/` >>>>>>>>> (thank you to Adolf!) >>>>>>>>> >>>>>>>>> rpz-config: >>>>>>>>> - bug: corrected "Type" test from block to allow >>>>>>>>> - removed verbose parameter from various commands >>>>>>>>> >>>>>>>>> rpz-metrics: >>>>>>>>> - bug: corrected grep for rpz name count >>>>>>>>> - bug: fixed divide by zero error (thank you Peppe!) >>>>>>>>> >>>>>>>>> install/uninstall: >>>>>>>>> - bug: corrected scripts (thank you Bernhard!) >>>>>>>>> >>>>>>>>> Signed-off-by: Jon Murphy <jon.murphy@ipfire.org> >>>>>>>>> --- >>>>>>>>> config/backup/includes/rpz | 4 ++-- >>>>>>>>> config/rootfiles/packages/rpz | 6 +++--- >>>>>>>>> config/rpz/rpz-config | 14 +++++++------- >>>>>>>>> config/rpz/rpz-metrics | 9 +++++---- >>>>>>>>> lfs/rpz | 6 +++--- >>>>>>>>> src/paks/rpz/install.sh | 27 +++++++++++++++++++++++++++ >>>>>>>>> src/paks/rpz/uninstall.sh | 31 +++++++++++++++++++++++++++++++ >>>>>>>>> src/paks/rpz/update.sh | 25 +++++++++++++++++++++++++ >>>>>>>>> 8 files changed, 103 insertions(+), 19 deletions(-) >>>>>>>>> create mode 100644 src/paks/rpz/install.sh >>>>>>>>> create mode 100644 src/paks/rpz/uninstall.sh >>>>>>>>> create mode 100644 src/paks/rpz/update.sh >>>>>>>>> >>>>>>>>> diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz >>>>>>>>> index 4d59bb40c..8c7410ebd 100644 >>>>>>>>> --- a/config/backup/includes/rpz >>>>>>>>> +++ b/config/backup/includes/rpz >>>>>>>>> @@ -1,5 +1,5 @@ >>>>>>>>> -/var/ipfire/rpz/allowlist >>>>>>>>> -/var/ipfire/rpz/blocklist >>>>>>>>> +/var/ipfire/dns/rpz/allowlist >>>>>>>>> +/var/ipfire/dns/rpz/blocklist >>>>>>>>> /etc/unbound/zonefiles/allow.rpz >>>>>>>>> /etc/unbound/zonefiles/block.rpz >>>>>>>>> /etc/unbound/local.d/*rpz.conf >>>>>>>>> diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/rpz >>>>>>>>> index 2ffa715dd..183825362 100644 >>>>>>>>> --- a/config/rootfiles/packages/rpz >>>>>>>>> +++ b/config/rootfiles/packages/rpz >>>>>>>>> @@ -6,6 +6,6 @@ usr/sbin/rpz-config >>>>>>>>> usr/sbin/rpz-metrics >>>>>>>>> usr/sbin/rpz-sleep >>>>>>>>> var/ipfire/backup/addons/includes/rpz >>>>>>>>> -var/ipfire/rpz >>>>>>>>> -var/ipfire/rpz/allowlist >>>>>>>>> -var/ipfire/rpz/blocklist >>>>>>>>> +var/ipfire/dns/rpz >>>>>>>>> +var/ipfire/dns/rpz/allowlist >>>>>>>>> +var/ipfire/dns/rpz/blocklist >>>>>>>>> diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config >>>>>>>>> index 98dc0a4ca..a24a5c132 100644 >>>>>>>>> --- a/config/rpz/rpz-config >>>>>>>>> +++ b/config/rpz/rpz-config >>>>>>>>> @@ -19,7 +19,7 @@ >>>>>>>>> # # >>>>>>>>> ############################################################################### >>>>>>>>> >>>>>>>>> -# v22 - 2024-07-12 >>>>>>>>> +# v23 - 2024-07-30 >>>>>>>>> >>>>>>>>> ############### Functions ############### >>>>>>>>> >>>>>>>>> @@ -54,11 +54,11 @@ check_unbound_conf () { >>>>>>>>> make_rpz_file () { >>>>>>>>> local theType="${1}" # allow or block >>>>>>>>> >>>>>>>>> - theList="/var/ipfire/rpz/${theType}list" # input user list of domains >>>>>>>>> + theList="/var/ipfire/dns/rpz/${theType}list" # input custom list of domains >>>>>>>>> theZoneFile="/etc/unbound/zonefiles/${theType}.rpz" # output file for RPZ >>>>>>>>> >>>>>>>>> theAction='.' >>>>>>>>> - if [[ "${theType}" =~ "block" ]] ; then >>>>>>>>> + if [[ "${theType}" =~ "allow" ]] ; then >>>>>>>>> theAction='rpz-passthru.' >>>>>>>>> fi >>>>>>>>> >>>>>>>>> @@ -131,8 +131,8 @@ case "${theAction}" in >>>>>>>>> # set-up zone file >>>>>>>>> /usr/bin/touch "${rpzFile}" >>>>>>>>> # unbound requires these settings for rpz files >>>>>>>>> - /bin/chown --verbose nobody:nobody "${rpzFile}" >>>>>>>>> - /bin/chmod --verbose 644 "${rpzFile}" >>>>>>>>> + /bin/chown nobody:nobody "${rpzFile}" >>>>>>>>> + /bin/chmod 644 "${rpzFile}" >>>>>>>>> ;; >>>>>>>>> >>>>>>>>> # trash config file & rpz file >>>>>>>>> @@ -143,8 +143,8 @@ case "${theAction}" in >>>>>>>>> fi >>>>>>>>> >>>>>>>>> msg_log "info: rpz: remove config file & rpz file \"${theName}\"" >>>>>>>>> - /bin/rm --verbose "${rpzConfig}" >>>>>>>>> - /bin/rm --verbose "${rpzFile}" >>>>>>>>> + /bin/rm "${rpzConfig}" >>>>>>>>> + /bin/rm "${rpzFile}" >>>>>>>>> >>>>>>>>> check_unbound_conf >>>>>>>>> ;; >>>>>>>>> diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics >>>>>>>>> index 0f97c7911..4d932726e 100644 >>>>>>>>> --- a/config/rpz/rpz-metrics >>>>>>>>> +++ b/config/rpz/rpz-metrics >>>>>>>>> @@ -19,7 +19,7 @@ >>>>>>>>> # # >>>>>>>>> ############################################################################### >>>>>>>>> >>>>>>>>> -# v18 on 2024-07-05 >>>>>>>>> +# v19 on 2024-07-30 >>>>>>>>> >>>>>>>>> ############### Main ############### >>>>>>>>> >>>>>>>>> @@ -33,7 +33,7 @@ messageLogs=$( find /var/log/messages* -type f | >>>>>>>>> >>>>>>>>> # get the list of RPZ names & counts from the message log(s) >>>>>>>>> rpzNameCount=$( for logf in ${messageLogs} ; do >>>>>>>>> - /usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" | >>>>>>>>> + /usr/bin/zgrep --text --extended-regexp 'info: rpz: applied.* A IN$' "${logf}" | >>>>>>>>> /usr/bin/awk '$10 ~ /\[\w*]/ { print $10 }' ; >>>>>>>>> done | /usr/bin/sort | /usr/bin/uniq --count ) >>>>>>>>> >>>>>>>>> @@ -107,8 +107,9 @@ do >>>>>>>>> theLines=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' ) >>>>>>>>> totalLines=$(( totalLines + theLines )) >>>>>>>>> >>>>>>>>> - #hitsPerLine=$( echo "scale=0 ; $theHits / $theLines" | bc ) >>>>>>>>> - hitsPerLine=$(( 100 * theHits / theLines )) >>>>>>>>> + if [[ "${theLines}" -gt 2 ]] ; then >>>>>>>>> + hitsPerLine=$(( 100 * theHits / theLines )) >>>>>>>>> + fi >>>>>>>>> fi >>>>>>>>> >>>>>>>>> # get modification date >>>>>>>>> diff --git a/lfs/rpz b/lfs/rpz >>>>>>>>> index 319c10b7f..73f6f2b1b 100644 >>>>>>>>> --- a/lfs/rpz >>>>>>>>> +++ b/lfs/rpz >>>>>>>>> @@ -67,9 +67,9 @@ $(TARGET) : >>>>>>>>> $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin >>>>>>>>> >>>>>>>>> # Install settings folder and two empty files >>>>>>>>> - mkdir -pv /var/ipfire/rpz >>>>>>>>> - touch /var/ipfire/rpz/allowlist >>>>>>>>> - touch /var/ipfire/rpz/blocklist >>>>>>>>> + mkdir -pv /var/ipfire/dns/rpz >>>>>>>>> + touch /var/ipfire/dns/rpz/allowlist >>>>>>>>> + touch /var/ipfire/dns/rpz/blocklist >>>>>>>>> >>>>>>>>> # Add conf file to /etc directory >>>>>>>>> cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d >>>>>>>>> diff --git a/src/paks/rpz/install.sh b/src/paks/rpz/install.sh >>>>>>>>> new file mode 100644 >>>>>>>>> index 000000000..0a797e158 >>>>>>>>> --- /dev/null >>>>>>>>> +++ b/src/paks/rpz/install.sh >>>>>>>>> @@ -0,0 +1,27 @@ >>>>>>>>> +#!/bin/bash >>>>>>>>> +############################################################################### >>>>>>>>> +# # >>>>>>>>> +# IPFire.org - A linux based firewall # >>>>>>>>> +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # >>>>>>>>> +# # >>>>>>>>> +# This program is free software: you can redistribute it and/or modify # >>>>>>>>> +# it under the terms of the GNU General Public License as published by # >>>>>>>>> +# the Free Software Foundation, either version 3 of the License, or # >>>>>>>>> +# (at your option) any later version. # >>>>>>>>> +# # >>>>>>>>> +# This program is distributed in the hope that it will be useful, # >>>>>>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >>>>>>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >>>>>>>>> +# GNU General Public License for more details. # >>>>>>>>> +# # >>>>>>>>> +# You should have received a copy of the GNU General Public License # >>>>>>>>> +# along with this program. If not, see <http://www.gnu.org/licenses/>. # >>>>>>>>> +# # >>>>>>>>> +############################################################################### >>>>>>>>> +# >>>>>>>>> +. /opt/pakfire/lib/functions.sh >>>>>>>>> +extract_files >>>>>>>>> +restore_backup ${NAME} >>>>>>>>> + >>>>>>>>> +# restart unbound to load config file >>>>>>>>> +/etc/init.d/unbound restart >>>>>>>>> diff --git a/src/paks/rpz/uninstall.sh b/src/paks/rpz/uninstall.sh >>>>>>>>> new file mode 100644 >>>>>>>>> index 000000000..4fb20e127 >>>>>>>>> --- /dev/null >>>>>>>>> +++ b/src/paks/rpz/uninstall.sh >>>>>>>>> @@ -0,0 +1,31 @@ >>>>>>>>> +#!/bin/bash >>>>>>>>> +############################################################################### >>>>>>>>> +# # >>>>>>>>> +# IPFire.org - A linux based firewall # >>>>>>>>> +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # >>>>>>>>> +# # >>>>>>>>> +# This program is free software: you can redistribute it and/or modify # >>>>>>>>> +# it under the terms of the GNU General Public License as published by # >>>>>>>>> +# the Free Software Foundation, either version 3 of the License, or # >>>>>>>>> +# (at your option) any later version. # >>>>>>>>> +# # >>>>>>>>> +# This program is distributed in the hope that it will be useful, # >>>>>>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >>>>>>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >>>>>>>>> +# GNU General Public License for more details. # >>>>>>>>> +# # >>>>>>>>> +# You should have received a copy of the GNU General Public License # >>>>>>>>> +# along with this program. If not, see <http://www.gnu.org/licenses/>. # >>>>>>>>> +# # >>>>>>>>> +############################################################################### >>>>>>>>> +# >>>>>>>>> +. /opt/pakfire/lib/functions.sh >>>>>>>>> + >>>>>>>>> +# stop unbound to delete RPZ conf file >>>>>>>>> +/etc/init.d/unbound stop >>>>>>>>> + >>>>>>>>> +make_backup ${NAME} >>>>>>>>> +remove_files >>>>>>>>> + >>>>>>>>> +# start unbound to load unbound config file >>>>>>>>> +/etc/init.d/unbound start >>>>>>>>> diff --git a/src/paks/rpz/update.sh b/src/paks/rpz/update.sh >>>>>>>>> new file mode 100644 >>>>>>>>> index 000000000..938a93a40 >>>>>>>>> --- /dev/null >>>>>>>>> +++ b/src/paks/rpz/update.sh >>>>>>>>> @@ -0,0 +1,25 @@ >>>>>>>>> +#!/bin/bash >>>>>>>>> +############################################################################### >>>>>>>>> +# # >>>>>>>>> +# IPFire.org - A linux based firewall # >>>>>>>>> +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # >>>>>>>>> +# # >>>>>>>>> +# This program is free software: you can redistribute it and/or modify # >>>>>>>>> +# it under the terms of the GNU General Public License as published by # >>>>>>>>> +# the Free Software Foundation, either version 3 of the License, or # >>>>>>>>> +# (at your option) any later version. # >>>>>>>>> +# # >>>>>>>>> +# This program is distributed in the hope that it will be useful, # >>>>>>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >>>>>>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >>>>>>>>> +# GNU General Public License for more details. # >>>>>>>>> +# # >>>>>>>>> +# You should have received a copy of the GNU General Public License # >>>>>>>>> +# along with this program. If not, see <http://www.gnu.org/licenses/>. # >>>>>>>>> +# # >>>>>>>>> +############################################################################### >>>>>>>>> +# >>>>>>>>> +. /opt/pakfire/lib/functions.sh >>>>>>>>> +extract_backup_includes >>>>>>>>> +./uninstall.sh >>>>>>>>> +./install.sh >>>>>>>>> -- >>>>>>>>> 2.30.2 > > > Jon > > > -- > Jon Murphy > jon.murphy@ipfire.org
Good day! >> Do I have your blessing to keep moving forward? > > What do you want to move forward with? To implement the first phase: An RPZ add-on that is currently shell based. The release will be similar to the Patch way below but with the fixes you suggested. Plus a few bug fixes! I’ll make the current changes and send a new Patch. The main goal of this Phase is to measure user interest in RPZ. The metrics will be feedback within the Community. > What are the next steps? > If there is interest from the user Community, then: • Phase 2: Input WebGUI (image shown in previous post) • Phase 3: RPZ logs / metrics via WebGUI > Looking at that list, I am not sure what I would want to turn on in my home/if I was a school/a generic office company. I could not decide what to use by reading the RPZ descriptions or looking at the LONG rpz lists. That was the reason for creating the `rpz-metrics` script. The output of the script helps me decide what to keep and what to disable. Right now my home (small office also) List will be: • Multi PRO or Multi PRO++ • Amazon Native Tracker • Apple Native Tracker • Windows Native Tracker • Encrypted DNS Servers (DoH server block) • Most Abused TLDs • plus one custom allow list and one custom block list This will bring down my daily RPZ downloads to ~25 MB per day. Much lower than the experimental 230 MB per day! The above are all Hagezi lists. Jon > On Aug 23, 2024, at 4:18 AM, Michael Tremer <michael.tremer@ipfire.org> wrote: > > Morning, > >> On 22 Aug 2024, at 16:37, jon <jon.murphy@ipfire.org> wrote: >> >> Hi Michael, >> >> >>> On Aug 21, 2024, at 5:03 AM, Michael Tremer <michael.tremer@ipfire.org> wrote: >>> >>> Hello Jon, >>> >>>> On 15 Aug 2024, at 23:22, jon <jon.murphy@ipfire.org> wrote: >>>> >>>> Comments below… >>>> >>>> >>>>> On Aug 15, 2024, at 1:33 PM, Michael Tremer <michael.tremer@ipfire.org> wrote: >>>>> >>>>> Hello, >>>>> >>>>>> On 15 Aug 2024, at 17:57, jon <jon.murphy@ipfire.org> wrote: >>>>>> >>>>>> >>>>>> Comments below. >>>>>> >>>>>> Jon >>>>>> >>>>>> >>>>>>> On Aug 15, 2024, at 10:33 AM, Michael Tremer <michael.tremer@ipfire.org> wrote: >>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>>> On 14 Aug 2024, at 18:14, jon <jon.murphy@ipfire.org> wrote: >>>>>>>> >>>>>>>> Michael, >>>>>>>> >>>>>>>> Sorry for putting you on the spot, but what do you want to do with this RPZ add-on? >>>>>>> >>>>>>> I am not sure this is a question for only me. >>>>>>> >>>>>>> I personally just don’t have any capacity to take on another rather large project as I have a huge backlog of so many things and I feel like a lonely fighter getting these all over the line. For my own sanity I need to have a couple of those closed before thinking about the next ones. >>>>>>> >>>>>>> A couple of months ago we have talked on the monthly call about RPZ and the decision that was made by the people was to not look at this now, just because there were other more pressing things. We could also not find answers to the questions that led us into RPZ: >>>>>>> >>>>>>> * Are there any good lists out there that would allow us to replace the URL Filter? That thing is basically on its last leg because of the lack of lists. We need to be able to block ads and pr0n and that very reliably. >>>>>> >>>>>> I believe this exists with the Hagezi lists at https://github.com/hagezi/dns-blocklists and/or the addition of a chosen "oisd NSFW" list at https://oisd.nl/setup >>>>>> >>>>>> I understand there is no time to review to see if this meet the needs. So we can leave this as "I believe these exist" for now. >>>>> >>>>> So, I am not sure what to think of all these things on GitHub. It seems that there are many repositories that pop up and presumably disappear in the same kind of way. We see this with the IP blocklists. >>>>> >>>>> We want quality blocklists. I know that we don’t have too much on the URL Filter front, but I would prefer to avoid putting so much work in if we in the end stand where we started. >>>> >>>> This makes perfect sense. Based on the comments I’ve seen it seems like Hagezi is growing in popularity and in offerings. >>>> >>>> Gerd (the owner) is very responsive, involved and I keep running into his posts when searching for RPZ "things". Gerd quickly made changes per my request: see https://github.com/hagezi/dns-blocklists/issues/2301. Now there are two TLD RPZ lists. >>>> >>>> Like everything else I cannot predict how long it might last. >>> >>> If you have such a good connection, could you do me the favour and ask why the prefer distributing those lists using HTTPS and not DNS? >> >> I’d be happy to ask Gerd, but I do not understand what you want me to ask. > > It sounded like you have a personal connection. That is all. > >> I guessed it is related to this: >> https://www.ipfire.org/blog/ipfire-location-a-decentralised-signed-database-in-dns >> https://www.ipfire.org/location/how-to-use/dns >> >> I understand the APIs you had added, but I do not understand how this related to the DNS blocked lists. > > Yes, this is slightly related to this. I am just running a network of authoritative DNS servers scattered a little bit across the world. It is an essential part of the IPFire infrastructure but also hosts a couple of other things. And since it is there, it is a nice thing to bolt on more things. > >> What might the query do? Look up a requested domain? > > The relationship to RPZs is that we could use that service to host them. It is a DNS-based technology after all. > >>> Is it simply that GitHub is free and does not offer DNS? :) >> >> I did not find GitHub DNS server(s) listed anywhere. Just lots of references on setting up DNS on other servers... > > No, GitHub does not offer any DNS services. > >> Let me know what you want asked, specifically, and I will pass it on to Gerd! > > The question is only whether it was a conscious decision to distribute the lists over HTTP instead of using DNS. Maybe it was just chosen because that is what was available to them. > >>> >>>>> >>>>>>> >>>>>>> * We need to look into privacy when RPZs are being realised over DNS - I don’t even understand why we are suddenly starting to pull text files over HTTP again. The IPS seems to have most of these lists already. >>>>>> >>>>>> All of the lists that I am pull are HTTPS. And HTTPS only is currently checked (validated) in the existing `rpz-config` code. >>>>> >>>>> What a lost case. DNS is a globally distributed, cacheable protocol. We could simply put all those blacklists into the DNS caches all around the world. They would respond fast, be local to so many people, but instead we download massive text files. That means that whenever something changes the entire file needs to be transferred again. They are huge. We would want many fast updates which are not possible that way… but I digress. >>>> >>>> I came across one DNS service that offers the Hagezi lists as part of their available DNS services. If interested I can find and post the details. >>> >>> I run a global DNS system where we could easily add this all to. This would scale like hell. It would be a fun project. >>> >>> Just for fun, I have imported "jpgpi250.github.io <http://jpgpi250.github.io/>” which you can pull from either dfw.lwldns.net<http://dfw.lwldns.net/> or haj.lwldns.net <http://haj.lwldns.net/>. This would be so much better since DNS zone transfer checks the serial of the zone and does not retransfer it if it has not changed (and I suppose none of these lists change that frequently). >>> >> >> Peter’s jpgpi250 DoH rpz list updates one per day near 04:00 CEST (02:00 UTC). A handful of adds or deletes every night. >> >> Gerd’s block DoH rpz list updates twice per day as needed. Sometime once per day, sometimes twice per day. > > Yeah, that is not a lot, so DNS would qualify to distribute the lists. > >>> Sadly our DNS system does not support this, >> >> I do not know what "global DNS system" is support. Is it not unbound based? > > No those servers are running PowerDNS behind a load-balancer. Unbound is just a recursor and not an authoritative DNS server. > >> There are many different formats to feed to DNS system: >> https://github.com/hagezi/dns-blocklists?tab=readme-ov-file#outbox_tray-encrypted-dns-servers-only- >> >> Maybe one of these might help? >> >> >>> but there is also IXFR for incremental zone updates where you would only load the changes. >> >> >>> >>>> >>>> >>>>> >>>>> How much traffic did you observe? >>>> >>>> >>>> I have 11 RPZ list (honestly too many but I am experimenting) and that eat up about 230M bytes per day. >>> >>> Oh wow that is huuuuuge. I assume that is repeat downloads? How large are the zone files just downloaded once? >> >> Here is the size of each file (grapped via `ls -lS /etc/unbound/zonefiles` and snipped): >> >> 10,655,321 MxProPlusHZ.rpz >> 90,909 dohJPG.rpz >> 76,591 HosterHZ.rpz >> 72,628 DOHblockHZ.rpz >> 37,411 urlhaus.rpz >> 27,863 WinTrkrHZ.rpz >> 12,333 NotSafeSearchHZ.rpz >> 7,573 AppleTrkrHZ1.rpz >> 7,308 tldAggHZ.rpz >> 2,631 tldHZ.rpz >> 966 allow.rpz >> 238 block.rpz >> >> >>> >>>>> How is Unbound refreshing these? >>>> >>>> >>>> Unbound RPZ takes care of the updates automagically. There is no cron entry. >>>> >>>> Each RPZ list includes an SOA line similar to this: >>>> `@ SOA localhost. root.localhost. 1723093380 43200 3600 259200 300` >>>> >>>> The 43200 is the refresh period in seconds. So this RPZ list auto updates every 12 hours. >>>> >>>> >>>>> Is there a local cache? >>>> >>>> Yes, it is part of unbound. (Nothing we need to setup) >>> >>> Would it use the cache for zones transferred over DNS, too? >> >> I do not know. >> >>> >>>> >>>> >>>>> >>>>>>> Regarding your code, there are some issues with the coding style, but I heavily appreciate the pioneering to bring this feature to life. >>>>>> >>>>>> My style is camelCase, but if you prefer snake_case that is an easy change (please speak up). >>>>>> >>>>>> As for as other style changes, that can wait until time is available by the core developers. >>>>> >>>>> It is mainly that you make things very complicated for yourself. Call “cat”, not “/bin/cat”. There is no guarantee that things will stay where they are now and the shell has lots of builtins that execute a lot faster. >>>> >>>> I will remove the executable paths. What builtin did you see that I missed? Anything else? >>> >>> A builtin is just a command that really isn’t one. So there is a /bin/echo somewhere, but when you just write “echo” in the shell, it will simply output the string itself. That is a lot faster than forking a new process. That is all. >> >> >> Got it! All changed to: >> >> echo "multi line string" > "${rpzConfig}" >> >> >>> >>>>> >>>>> And this is all designed to work on the shell. We need something for the web UI. >>>> >>>> For commands within the WegGUI there are `safe_system` commands on the `.cgi` page. >>>> >>>> But I am guess you are referring to something else. >>> >>> That was mainly for the metrics… We would want those to be shown on the web UI and the tool that you currently have outputs it on the shell. That is what that meant. >> >> For short term the metrics will be via shell only. >> >> Long term, agree, metrics via the WebGUI! >> >> >>> >>>>> >>>>>>> What it would need to be finally merged would be a web UI though. So I think we have a long way ahead of us. >>>>>> >>>>>> The attachment is an early mock-up I made a few months ago (made with Pixelmator Pro). The WebGUI is being worked on now. >>>>>> >>>>>> <mock-up.png> >>>>> >>>>> I think we might look for something that is very similar to the IPS ruleset editor or the IP blocklists. >>>>> >>>>> I am not even sure if we would require users to manage this or whether we would curate a list like with the IPS and IP blocklists. >>>> >>>> The first "DRAFT" of this was for the user/admin to manage the list. For the RPZ add-on, I’d prefer to keep it this way until we figure out if RPZ is popular and used by the user/admins. >>>> >>>> And yes, it could be changed to a set list like IPS. FYI - there are 34 different Hagezi list. So it will take some experiments to pick a "the" list. >>> >>> Well it doesn’t have to be just one. People might want to block different things… >> >> Agree! >> >>> >>> It just seems that they come as a package “Light”, “Normal”, “Pro”, “Pro++” and it is either that or nothing. >> >> More than nothing - Here are the individual (non-package) lists for Hagezi: >> >> • Fake - Protects against internet scams, traps & fakes! >> • Pop-Up Ads - Protects against annoying and malicious pop-up ads! >> • Threat Intelligence Feeds - Increases security significantly! (Recommended) : Full - Medium - Mini - IPs >> • Newly Registered Domains - Favoured by threat actors to launch malicious campaigns! : 14 days - 30 days >> • DoH/VPN/TOR/Proxy Bypass - Prevent methods to bypass your DNS! : Full - DoH only - DoH IPs >> • Safesearch not supported - Prevent the use of search engines that do not support Safesearch! >> • Dynamic DNS - Protects against the malicious use of dynamic DNS services! >> • Badware Hoster - Protects against the malicious use of free host services! >> • Most Abused TLDs - Protects against known malicious Top Level Domains! >> • Anti Piracy - Protects against piracy! >> • Gambling - Protects against gambling content! : Full - Medium - Mini >> • NSFW (external) - oisd NSFW - Protects against adult content! >> • Native Tracker - Broadband tracker of devices, services and operating systems >> >> And there are others out there also. > > Others might exist, but as it seems they all mix and match their lists together. So there will a lot of overlap which becomes a waste of memory and data transfer. > > Looking at that list, I am not sure what I would want to turn on in my home/if I was a school/a generic office company. > >> Do I have your blessing to keep moving forward? > > What do you want to move forward with? What are the next steps? > > -Michael > >> Jon >> >> >> >> PS - Here is the current WebGUI for RPZ. >> >> <Screenshot 2024-08-22 at 10.28.54 AM.png> >> >> >> >> And the EDIT pencil. >> >> >> <Screenshot 2024-08-22 at 10.29.25 AM.png> >> >> >> >>> -Michael >>> >>>> Also, here is current DRAFT (first draft) of user/admin managed WebGUI. See attachment. >>>> >>>> <Screenshot 2024-08-11 at 2.07.19 PM copy.png> >>>> >>>> >>>> Jon >>>> >>>> >>>>> >>>>>> Does this help? >>>>>> Jon >>>>>> >>>>>> >>>>>>> >>>>>>> -Michael >>>>>>> >>>>>>>> I saw your comments in the Dev Mailing List of "generally being in favor of trying this path" (bad paraphrasing on my part) >>>>>>>> >>>>>>>> I saw your comments in bugzilla at https://bugzilla.ipfire.org/show_bug.cgi?id=13254#c171 >>>>>>>>> I am not interested in anything regarding the RPZs right now. They have not been properly put on the agenda and looking at how much time we have on our hands, this won't make it on the agenda for years. >>>>>>>>> >>>>>>>>> I don't want to build blockers, but this ticket is about a different problem which I want to solve first. >>>>>>>> >>>>>>>> >>>>>>>> How do you want to go forward? >>>>>>>> >>>>>>>> >>>>>>>> Jon >>>>>>>> >>>>>>>> >>>>>>>>> On Aug 12, 2024, at 2:11 PM, jon <jon.murphy@ipfire.org> wrote: >>>>>>>>> >>>>>>>>> More questions! >>>>>>>>> >>>>>>>>> Currently RPZ config files are at `/etc/unbound/local.d` but this directory seems like it is for user (admin) customizations. >>>>>>>>> >>>>>>>>> ``` >>>>>>>>> [root@ipfire ~] # ls -al /etc/unbound/local.d >>>>>>>>> total 68 >>>>>>>>> drwxr-xr-x 2 nobody nobody 4096 Aug 12 13:41 . >>>>>>>>> drwxr-xr-x 4 root root 4096 Aug 12 00:52 .. >>>>>>>>> -rw-r--r-- 1 nobody nobody 436 Jul 12 15:45 00-rpz.conf >>>>>>>>> -rw-r--r-- 1 nobody nobody 285 Mar 1 22:12 AmazonTrkrHZ.rpz.conf >>>>>>>>> -rw-r--r-- 1 nobody nobody 281 Mar 1 22:02 AppleTrkrHZ.rpz.conf >>>>>>>>> -rw-r--r-- 1 nobody nobody 269 Mar 1 21:40 DOHblockHZ.rpz.conf >>>>>>>>> ... >>>>>>>>> -rw-r--r-- 1 nobody nobody 299 Aug 1 19:42 WinTrkrHZ.rpz.conf >>>>>>>>> [root@ipfire ~] # >>>>>>>>> ``` >>>>>>>>> >>>>>>>>> >>>>>>>>> Each file is a config file per category (or one per RPZ file). This makes it easy to add or remove a category (or RPZ file). >>>>>>>>> >>>>>>>>> Should I create a new unbound directory for RPZ config files? Maybe `/etc/unbound/rpz.d`? Or `/etc/unbound/rpz`? >>>>>>>>> >>>>>>>>> >>>>>>>>> Jon >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Aug 1, 2024, at 1:45 PM, Jon Murphy <jon.murphy@ipfire.org> wrote: >>>>>>>>>> >>>>>>>>>> changed all paths from `/var/ipfire/rpz/` to `/var/ipfire/dns/rpz/` >>>>>>>>>> (thank you to Adolf!) >>>>>>>>>> >>>>>>>>>> rpz-config: >>>>>>>>>> - bug: corrected "Type" test from block to allow >>>>>>>>>> - removed verbose parameter from various commands >>>>>>>>>> >>>>>>>>>> rpz-metrics: >>>>>>>>>> - bug: corrected grep for rpz name count >>>>>>>>>> - bug: fixed divide by zero error (thank you Peppe!) >>>>>>>>>> >>>>>>>>>> install/uninstall: >>>>>>>>>> - bug: corrected scripts (thank you Bernhard!) >>>>>>>>>> >>>>>>>>>> Signed-off-by: Jon Murphy <jon.murphy@ipfire.org> >>>>>>>>>> --- >>>>>>>>>> config/backup/includes/rpz | 4 ++-- >>>>>>>>>> config/rootfiles/packages/rpz | 6 +++--- >>>>>>>>>> config/rpz/rpz-config | 14 +++++++------- >>>>>>>>>> config/rpz/rpz-metrics | 9 +++++---- >>>>>>>>>> lfs/rpz | 6 +++--- >>>>>>>>>> src/paks/rpz/install.sh | 27 +++++++++++++++++++++++++++ >>>>>>>>>> src/paks/rpz/uninstall.sh | 31 +++++++++++++++++++++++++++++++ >>>>>>>>>> src/paks/rpz/update.sh | 25 +++++++++++++++++++++++++ >>>>>>>>>> 8 files changed, 103 insertions(+), 19 deletions(-) >>>>>>>>>> create mode 100644 src/paks/rpz/install.sh >>>>>>>>>> create mode 100644 src/paks/rpz/uninstall.sh >>>>>>>>>> create mode 100644 src/paks/rpz/update.sh >>>>>>>>>> >>>>>>>>>> diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz >>>>>>>>>> index 4d59bb40c..8c7410ebd 100644 >>>>>>>>>> --- a/config/backup/includes/rpz >>>>>>>>>> +++ b/config/backup/includes/rpz >>>>>>>>>> @@ -1,5 +1,5 @@ >>>>>>>>>> -/var/ipfire/rpz/allowlist >>>>>>>>>> -/var/ipfire/rpz/blocklist >>>>>>>>>> +/var/ipfire/dns/rpz/allowlist >>>>>>>>>> +/var/ipfire/dns/rpz/blocklist >>>>>>>>>> /etc/unbound/zonefiles/allow.rpz >>>>>>>>>> /etc/unbound/zonefiles/block.rpz >>>>>>>>>> /etc/unbound/local.d/*rpz.conf >>>>>>>>>> diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/rpz >>>>>>>>>> index 2ffa715dd..183825362 100644 >>>>>>>>>> --- a/config/rootfiles/packages/rpz >>>>>>>>>> +++ b/config/rootfiles/packages/rpz >>>>>>>>>> @@ -6,6 +6,6 @@ usr/sbin/rpz-config >>>>>>>>>> usr/sbin/rpz-metrics >>>>>>>>>> usr/sbin/rpz-sleep >>>>>>>>>> var/ipfire/backup/addons/includes/rpz >>>>>>>>>> -var/ipfire/rpz >>>>>>>>>> -var/ipfire/rpz/allowlist >>>>>>>>>> -var/ipfire/rpz/blocklist >>>>>>>>>> +var/ipfire/dns/rpz >>>>>>>>>> +var/ipfire/dns/rpz/allowlist >>>>>>>>>> +var/ipfire/dns/rpz/blocklist >>>>>>>>>> diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config >>>>>>>>>> index 98dc0a4ca..a24a5c132 100644 >>>>>>>>>> --- a/config/rpz/rpz-config >>>>>>>>>> +++ b/config/rpz/rpz-config >>>>>>>>>> @@ -19,7 +19,7 @@ >>>>>>>>>> # # >>>>>>>>>> ############################################################################### >>>>>>>>>> >>>>>>>>>> -# v22 - 2024-07-12 >>>>>>>>>> +# v23 - 2024-07-30 >>>>>>>>>> >>>>>>>>>> ############### Functions ############### >>>>>>>>>> >>>>>>>>>> @@ -54,11 +54,11 @@ check_unbound_conf () { >>>>>>>>>> make_rpz_file () { >>>>>>>>>> local theType="${1}" # allow or block >>>>>>>>>> >>>>>>>>>> - theList="/var/ipfire/rpz/${theType}list" # input user list of domains >>>>>>>>>> + theList="/var/ipfire/dns/rpz/${theType}list" # input custom list of domains >>>>>>>>>> theZoneFile="/etc/unbound/zonefiles/${theType}.rpz" # output file for RPZ >>>>>>>>>> >>>>>>>>>> theAction='.' >>>>>>>>>> - if [[ "${theType}" =~ "block" ]] ; then >>>>>>>>>> + if [[ "${theType}" =~ "allow" ]] ; then >>>>>>>>>> theAction='rpz-passthru.' >>>>>>>>>> fi >>>>>>>>>> >>>>>>>>>> @@ -131,8 +131,8 @@ case "${theAction}" in >>>>>>>>>> # set-up zone file >>>>>>>>>> /usr/bin/touch "${rpzFile}" >>>>>>>>>> # unbound requires these settings for rpz files >>>>>>>>>> - /bin/chown --verbose nobody:nobody "${rpzFile}" >>>>>>>>>> - /bin/chmod --verbose 644 "${rpzFile}" >>>>>>>>>> + /bin/chown nobody:nobody "${rpzFile}" >>>>>>>>>> + /bin/chmod 644 "${rpzFile}" >>>>>>>>>> ;; >>>>>>>>>> >>>>>>>>>> # trash config file & rpz file >>>>>>>>>> @@ -143,8 +143,8 @@ case "${theAction}" in >>>>>>>>>> fi >>>>>>>>>> >>>>>>>>>> msg_log "info: rpz: remove config file & rpz file \"${theName}\"" >>>>>>>>>> - /bin/rm --verbose "${rpzConfig}" >>>>>>>>>> - /bin/rm --verbose "${rpzFile}" >>>>>>>>>> + /bin/rm "${rpzConfig}" >>>>>>>>>> + /bin/rm "${rpzFile}" >>>>>>>>>> >>>>>>>>>> check_unbound_conf >>>>>>>>>> ;; >>>>>>>>>> diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics >>>>>>>>>> index 0f97c7911..4d932726e 100644 >>>>>>>>>> --- a/config/rpz/rpz-metrics >>>>>>>>>> +++ b/config/rpz/rpz-metrics >>>>>>>>>> @@ -19,7 +19,7 @@ >>>>>>>>>> # # >>>>>>>>>> ############################################################################### >>>>>>>>>> >>>>>>>>>> -# v18 on 2024-07-05 >>>>>>>>>> +# v19 on 2024-07-30 >>>>>>>>>> >>>>>>>>>> ############### Main ############### >>>>>>>>>> >>>>>>>>>> @@ -33,7 +33,7 @@ messageLogs=$( find /var/log/messages* -type f | >>>>>>>>>> >>>>>>>>>> # get the list of RPZ names & counts from the message log(s) >>>>>>>>>> rpzNameCount=$( for logf in ${messageLogs} ; do >>>>>>>>>> - /usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" | >>>>>>>>>> + /usr/bin/zgrep --text --extended-regexp 'info: rpz: applied.* A IN$' "${logf}" | >>>>>>>>>> /usr/bin/awk '$10 ~ /\[\w*]/ { print $10 }' ; >>>>>>>>>> done | /usr/bin/sort | /usr/bin/uniq --count ) >>>>>>>>>> >>>>>>>>>> @@ -107,8 +107,9 @@ do >>>>>>>>>> theLines=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' ) >>>>>>>>>> totalLines=$(( totalLines + theLines )) >>>>>>>>>> >>>>>>>>>> - #hitsPerLine=$( echo "scale=0 ; $theHits / $theLines" | bc ) >>>>>>>>>> - hitsPerLine=$(( 100 * theHits / theLines )) >>>>>>>>>> + if [[ "${theLines}" -gt 2 ]] ; then >>>>>>>>>> + hitsPerLine=$(( 100 * theHits / theLines )) >>>>>>>>>> + fi >>>>>>>>>> fi >>>>>>>>>> >>>>>>>>>> # get modification date >>>>>>>>>> diff --git a/lfs/rpz b/lfs/rpz >>>>>>>>>> index 319c10b7f..73f6f2b1b 100644 >>>>>>>>>> --- a/lfs/rpz >>>>>>>>>> +++ b/lfs/rpz >>>>>>>>>> @@ -67,9 +67,9 @@ $(TARGET) : >>>>>>>>>> $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin >>>>>>>>>> >>>>>>>>>> # Install settings folder and two empty files >>>>>>>>>> - mkdir -pv /var/ipfire/rpz >>>>>>>>>> - touch /var/ipfire/rpz/allowlist >>>>>>>>>> - touch /var/ipfire/rpz/blocklist >>>>>>>>>> + mkdir -pv /var/ipfire/dns/rpz >>>>>>>>>> + touch /var/ipfire/dns/rpz/allowlist >>>>>>>>>> + touch /var/ipfire/dns/rpz/blocklist >>>>>>>>>> >>>>>>>>>> # Add conf file to /etc directory >>>>>>>>>> cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d >>>>>>>>>> diff --git a/src/paks/rpz/install.sh b/src/paks/rpz/install.sh >>>>>>>>>> new file mode 100644 >>>>>>>>>> index 000000000..0a797e158 >>>>>>>>>> --- /dev/null >>>>>>>>>> +++ b/src/paks/rpz/install.sh >>>>>>>>>> @@ -0,0 +1,27 @@ >>>>>>>>>> +#!/bin/bash >>>>>>>>>> +############################################################################### >>>>>>>>>> +# # >>>>>>>>>> +# IPFire.org - A linux based firewall # >>>>>>>>>> +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # >>>>>>>>>> +# # >>>>>>>>>> +# This program is free software: you can redistribute it and/or modify # >>>>>>>>>> +# it under the terms of the GNU General Public License as published by # >>>>>>>>>> +# the Free Software Foundation, either version 3 of the License, or # >>>>>>>>>> +# (at your option) any later version. # >>>>>>>>>> +# # >>>>>>>>>> +# This program is distributed in the hope that it will be useful, # >>>>>>>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >>>>>>>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >>>>>>>>>> +# GNU General Public License for more details. # >>>>>>>>>> +# # >>>>>>>>>> +# You should have received a copy of the GNU General Public License # >>>>>>>>>> +# along with this program. If not, see <http://www.gnu.org/licenses/>. # >>>>>>>>>> +# # >>>>>>>>>> +############################################################################### >>>>>>>>>> +# >>>>>>>>>> +. /opt/pakfire/lib/functions.sh >>>>>>>>>> +extract_files >>>>>>>>>> +restore_backup ${NAME} >>>>>>>>>> + >>>>>>>>>> +# restart unbound to load config file >>>>>>>>>> +/etc/init.d/unbound restart >>>>>>>>>> diff --git a/src/paks/rpz/uninstall.sh b/src/paks/rpz/uninstall.sh >>>>>>>>>> new file mode 100644 >>>>>>>>>> index 000000000..4fb20e127 >>>>>>>>>> --- /dev/null >>>>>>>>>> +++ b/src/paks/rpz/uninstall.sh >>>>>>>>>> @@ -0,0 +1,31 @@ >>>>>>>>>> +#!/bin/bash >>>>>>>>>> +############################################################################### >>>>>>>>>> +# # >>>>>>>>>> +# IPFire.org - A linux based firewall # >>>>>>>>>> +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # >>>>>>>>>> +# # >>>>>>>>>> +# This program is free software: you can redistribute it and/or modify # >>>>>>>>>> +# it under the terms of the GNU General Public License as published by # >>>>>>>>>> +# the Free Software Foundation, either version 3 of the License, or # >>>>>>>>>> +# (at your option) any later version. # >>>>>>>>>> +# # >>>>>>>>>> +# This program is distributed in the hope that it will be useful, # >>>>>>>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >>>>>>>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >>>>>>>>>> +# GNU General Public License for more details. # >>>>>>>>>> +# # >>>>>>>>>> +# You should have received a copy of the GNU General Public License # >>>>>>>>>> +# along with this program. If not, see <http://www.gnu.org/licenses/>. # >>>>>>>>>> +# # >>>>>>>>>> +############################################################################### >>>>>>>>>> +# >>>>>>>>>> +. /opt/pakfire/lib/functions.sh >>>>>>>>>> + >>>>>>>>>> +# stop unbound to delete RPZ conf file >>>>>>>>>> +/etc/init.d/unbound stop >>>>>>>>>> + >>>>>>>>>> +make_backup ${NAME} >>>>>>>>>> +remove_files >>>>>>>>>> + >>>>>>>>>> +# start unbound to load unbound config file >>>>>>>>>> +/etc/init.d/unbound start >>>>>>>>>> diff --git a/src/paks/rpz/update.sh b/src/paks/rpz/update.sh >>>>>>>>>> new file mode 100644 >>>>>>>>>> index 000000000..938a93a40 >>>>>>>>>> --- /dev/null >>>>>>>>>> +++ b/src/paks/rpz/update.sh >>>>>>>>>> @@ -0,0 +1,25 @@ >>>>>>>>>> +#!/bin/bash >>>>>>>>>> +############################################################################### >>>>>>>>>> +# # >>>>>>>>>> +# IPFire.org - A linux based firewall # >>>>>>>>>> +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # >>>>>>>>>> +# # >>>>>>>>>> +# This program is free software: you can redistribute it and/or modify # >>>>>>>>>> +# it under the terms of the GNU General Public License as published by # >>>>>>>>>> +# the Free Software Foundation, either version 3 of the License, or # >>>>>>>>>> +# (at your option) any later version. # >>>>>>>>>> +# # >>>>>>>>>> +# This program is distributed in the hope that it will be useful, # >>>>>>>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of # >>>>>>>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # >>>>>>>>>> +# GNU General Public License for more details. # >>>>>>>>>> +# # >>>>>>>>>> +# You should have received a copy of the GNU General Public License # >>>>>>>>>> +# along with this program. If not, see <http://www.gnu.org/licenses/>. # >>>>>>>>>> +# # >>>>>>>>>> +############################################################################### >>>>>>>>>> +# >>>>>>>>>> +. /opt/pakfire/lib/functions.sh >>>>>>>>>> +extract_backup_includes >>>>>>>>>> +./uninstall.sh >>>>>>>>>> +./install.sh >>>>>>>>>> -- >>>>>>>>>> 2.30.2 >> >> >> Jon >> >> >> -- >> Jon Murphy >> jon.murphy@ipfire.org Jon
diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz index 4d59bb40c..8c7410ebd 100644 --- a/config/backup/includes/rpz +++ b/config/backup/includes/rpz @@ -1,5 +1,5 @@ -/var/ipfire/rpz/allowlist -/var/ipfire/rpz/blocklist +/var/ipfire/dns/rpz/allowlist +/var/ipfire/dns/rpz/blocklist /etc/unbound/zonefiles/allow.rpz /etc/unbound/zonefiles/block.rpz /etc/unbound/local.d/*rpz.conf diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/rpz index 2ffa715dd..183825362 100644 --- a/config/rootfiles/packages/rpz +++ b/config/rootfiles/packages/rpz @@ -6,6 +6,6 @@ usr/sbin/rpz-config usr/sbin/rpz-metrics usr/sbin/rpz-sleep var/ipfire/backup/addons/includes/rpz -var/ipfire/rpz -var/ipfire/rpz/allowlist -var/ipfire/rpz/blocklist +var/ipfire/dns/rpz +var/ipfire/dns/rpz/allowlist +var/ipfire/dns/rpz/blocklist diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config index 98dc0a4ca..a24a5c132 100644 --- a/config/rpz/rpz-config +++ b/config/rpz/rpz-config @@ -19,7 +19,7 @@ # # ############################################################################### -# v22 - 2024-07-12 +# v23 - 2024-07-30 ############### Functions ############### @@ -54,11 +54,11 @@ check_unbound_conf () { make_rpz_file () { local theType="${1}" # allow or block - theList="/var/ipfire/rpz/${theType}list" # input user list of domains + theList="/var/ipfire/dns/rpz/${theType}list" # input custom list of domains theZoneFile="/etc/unbound/zonefiles/${theType}.rpz" # output file for RPZ theAction='.' - if [[ "${theType}" =~ "block" ]] ; then + if [[ "${theType}" =~ "allow" ]] ; then theAction='rpz-passthru.' fi @@ -131,8 +131,8 @@ case "${theAction}" in # set-up zone file /usr/bin/touch "${rpzFile}" # unbound requires these settings for rpz files - /bin/chown --verbose nobody:nobody "${rpzFile}" - /bin/chmod --verbose 644 "${rpzFile}" + /bin/chown nobody:nobody "${rpzFile}" + /bin/chmod 644 "${rpzFile}" ;; # trash config file & rpz file @@ -143,8 +143,8 @@ case "${theAction}" in fi msg_log "info: rpz: remove config file & rpz file \"${theName}\"" - /bin/rm --verbose "${rpzConfig}" - /bin/rm --verbose "${rpzFile}" + /bin/rm "${rpzConfig}" + /bin/rm "${rpzFile}" check_unbound_conf ;; diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics index 0f97c7911..4d932726e 100644 --- a/config/rpz/rpz-metrics +++ b/config/rpz/rpz-metrics @@ -19,7 +19,7 @@ # # ############################################################################### -# v18 on 2024-07-05 +# v19 on 2024-07-30 ############### Main ############### @@ -33,7 +33,7 @@ messageLogs=$( find /var/log/messages* -type f | # get the list of RPZ names & counts from the message log(s) rpzNameCount=$( for logf in ${messageLogs} ; do - /usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" | + /usr/bin/zgrep --text --extended-regexp 'info: rpz: applied.* A IN$' "${logf}" | /usr/bin/awk '$10 ~ /\[\w*]/ { print $10 }' ; done | /usr/bin/sort | /usr/bin/uniq --count ) @@ -107,8 +107,9 @@ do theLines=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' ) totalLines=$(( totalLines + theLines )) - #hitsPerLine=$( echo "scale=0 ; $theHits / $theLines" | bc ) - hitsPerLine=$(( 100 * theHits / theLines )) + if [[ "${theLines}" -gt 2 ]] ; then + hitsPerLine=$(( 100 * theHits / theLines )) + fi fi # get modification date diff --git a/lfs/rpz b/lfs/rpz index 319c10b7f..73f6f2b1b 100644 --- a/lfs/rpz +++ b/lfs/rpz @@ -67,9 +67,9 @@ $(TARGET) : $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin # Install settings folder and two empty files - mkdir -pv /var/ipfire/rpz - touch /var/ipfire/rpz/allowlist - touch /var/ipfire/rpz/blocklist + mkdir -pv /var/ipfire/dns/rpz + touch /var/ipfire/dns/rpz/allowlist + touch /var/ipfire/dns/rpz/blocklist # Add conf file to /etc directory cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d diff --git a/src/paks/rpz/install.sh b/src/paks/rpz/install.sh new file mode 100644 index 000000000..0a797e158 --- /dev/null +++ b/src/paks/rpz/install.sh @@ -0,0 +1,27 @@ +#!/bin/bash +############################################################################### +# # +# IPFire.org - A linux based firewall # +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # +# # +# This program is free software: you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation, either version 3 of the License, or # +# (at your option) any later version. # +# # +# This program is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with this program. If not, see <http://www.gnu.org/licenses/>. # +# # +############################################################################### +# +. /opt/pakfire/lib/functions.sh +extract_files +restore_backup ${NAME} + +# restart unbound to load config file +/etc/init.d/unbound restart diff --git a/src/paks/rpz/uninstall.sh b/src/paks/rpz/uninstall.sh new file mode 100644 index 000000000..4fb20e127 --- /dev/null +++ b/src/paks/rpz/uninstall.sh @@ -0,0 +1,31 @@ +#!/bin/bash +############################################################################### +# # +# IPFire.org - A linux based firewall # +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # +# # +# This program is free software: you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation, either version 3 of the License, or # +# (at your option) any later version. # +# # +# This program is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with this program. If not, see <http://www.gnu.org/licenses/>. # +# # +############################################################################### +# +. /opt/pakfire/lib/functions.sh + +# stop unbound to delete RPZ conf file +/etc/init.d/unbound stop + +make_backup ${NAME} +remove_files + +# start unbound to load unbound config file +/etc/init.d/unbound start diff --git a/src/paks/rpz/update.sh b/src/paks/rpz/update.sh new file mode 100644 index 000000000..938a93a40 --- /dev/null +++ b/src/paks/rpz/update.sh @@ -0,0 +1,25 @@ +#!/bin/bash +############################################################################### +# # +# IPFire.org - A linux based firewall # +# Copyright (C) 2024 IPFire Team <info@ipfire.org> # +# # +# This program is free software: you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation, either version 3 of the License, or # +# (at your option) any later version. # +# # +# This program is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with this program. If not, see <http://www.gnu.org/licenses/>. # +# # +############################################################################### +# +. /opt/pakfire/lib/functions.sh +extract_backup_includes +./uninstall.sh +./install.sh