diff mbox series

RPZ: install new add-on

Message ID	20240720174639.1419870-1-jon.murphy@ipfire.org
State	New
Headers	From: Jon Murphy <jon.murphy@ipfire.org> To: development@lists.ipfire.org Subject: [PATCH] RPZ: install new add-on Date: Sat, 20 Jul 2024 12:46:39 -0500 Message-Id: <20240720174639.1419870-1-jon.murphy@ipfire.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Message-ID-Hash: VPI6Y5COEMVFPQLF5VQVJPS6GRYAAK33 Precedence: list Archived-At: <https://lists.ipfire.org/hyperkitty/list/development@lists.ipfire.org/message/VPI6Y5COEMVFPQLF5VQVJPS6GRYAAK33/>
Series	RPZ: install new add-on \| RPZ: install new add-on

Commit Message

Jon Murphy July 20, 2024, 5:46 p.m. UTC

  What is it?
Response Policy Zone (RPZ) is a mechanism to define local policies in a standardised
way and load those policies from external sources.
Bottom line: RPZ allows admins to easily block access to websites via DNS lookup.

RPZ can block websites via categories.  Examples include: fake websites, annoying
pop-up ads, newly registered domains, DoH bypass sites, bad "host" services,
maliscious top level domains (e.g., *.zip, *.mov), piracy, gambling, pornography,
and more.  RPZ lists come from various RPZ providers and their available
catagories.

This RPZ add-on enables the RPZ functionality by adding a couple lines in a
configuration file.  This add-on simply adds a configuration file and adds
three scripts (config, metrics and sleep) to make RPZ easier for the admin to use.

RPZ was release in 2010 and has been part of the IPFire build since ~2015.

Why is it needed?
Some IPFire admin's utilize pihole to block unwanted websites via DNS lookup.
Moving the pihole base functionality (without pretty graphs) to IPFire removes
one device from the admin's local network.
And hopefully this reduces the pihole questions from the Community.

A list of RPZ providers can be recommended by IPFire and coded into a set list.
Or, if prefered, the local admin can choose their own RPZ providers.

This is a:
 * simple replacement for pihole base functionality
 * RPZ can be a nice replacement for the URL Filter

IPFire Wiki
In process at: https://www.ipfire.org/docs/addons/rpz

more info:
 * https://en.wikipedia.org/wiki/Response_policy_zone
 * https://unbound.docs.nlnetlabs.nl/en/latest/topics/filtering/rpz.html

I need help with...
1) The custom allow and block lists are currently located at `/var/ipfire/rpz`.  Is this correct?
2) The three bash scripts are currently located at `/usr/sbin`.  Is this correct?

Signed-off-by: Jon Murphy <jon.murphy@ipfire.org>
---
 config/backup/includes/rpz    |   5 +
 config/rootfiles/packages/rpz |  11 ++
 config/rpz/00-rpz.conf        |  18 ++++
 config/rpz/rpz-config         | 194 ++++++++++++++++++++++++++++++++++
 config/rpz/rpz-metrics        | 143 +++++++++++++++++++++++++
 config/rpz/rpz-sleep          |  58 ++++++++++
 lfs/rpz                       |  88 +++++++++++++++
 make.sh                       |   2 +
 8 files changed, 519 insertions(+)
 create mode 100644 config/backup/includes/rpz
 create mode 100644 config/rootfiles/packages/rpz
 create mode 100644 config/rpz/00-rpz.conf
 create mode 100644 config/rpz/rpz-config
 create mode 100644 config/rpz/rpz-metrics
 create mode 100644 config/rpz/rpz-sleep
 create mode 100644 lfs/rpz

Comments

Jon Murphy July 20, 2024, 6:42 p.m. UTC | #1

I need help with…

1) The custom allow and block lists are currently located at `/var/ipfire/rpz`.  Is this correct?

2) The three bash scripts are currently located at `/usr/sbin`.  Is this correct?

3) Should this commit be separated into multiple posts?  If yes, how do I decide how to split it?


Jon 


> On Jul 20, 2024, at 12:46 PM, Jon Murphy <jon.murphy@ipfire.org> wrote:
> 
> What is it?
> Response Policy Zone (RPZ) is a mechanism to define local policies in a standardised
> way and load those policies from external sources.
> Bottom line: RPZ allows admins to easily block access to websites via DNS lookup.
> 
> RPZ can block websites via categories.  Examples include: fake websites, annoying
> pop-up ads, newly registered domains, DoH bypass sites, bad "host" services,
> maliscious top level domains (e.g., *.zip, *.mov), piracy, gambling, pornography,
> and more.  RPZ lists come from various RPZ providers and their available
> catagories.
> 
> This RPZ add-on enables the RPZ functionality by adding a couple lines in a
> configuration file.  This add-on simply adds a configuration file and adds
> three scripts (config, metrics and sleep) to make RPZ easier for the admin to use.
> 
> RPZ was release in 2010 and has been part of the IPFire build since ~2015.
> 
> Why is it needed?
> Some IPFire admin's utilize pihole to block unwanted websites via DNS lookup.
> Moving the pihole base functionality (without pretty graphs) to IPFire removes
> one device from the admin's local network.
> And hopefully this reduces the pihole questions from the Community.
> 
> A list of RPZ providers can be recommended by IPFire and coded into a set list.
> Or, if prefered, the local admin can choose their own RPZ providers.
> 
> This is a:
> * simple replacement for pihole base functionality
> * RPZ can be a nice replacement for the URL Filter
> 
> IPFire Wiki
> In process at: https://www.ipfire.org/docs/addons/rpz
> 
> more info:
> * https://en.wikipedia.org/wiki/Response_policy_zone
> * https://unbound.docs.nlnetlabs.nl/en/latest/topics/filtering/rpz.html
> 
> I need help with...
> 1) The custom allow and block lists are currently located at `/var/ipfire/rpz`.  Is this correct?
> 2) The three bash scripts are currently located at `/usr/sbin`.  Is this correct?
> 
> Signed-off-by: Jon Murphy <jon.murphy@ipfire.org>
> ---
> config/backup/includes/rpz    |   5 +
> config/rootfiles/packages/rpz |  11 ++
> config/rpz/00-rpz.conf        |  18 ++++
> config/rpz/rpz-config         | 194 ++++++++++++++++++++++++++++++++++
> config/rpz/rpz-metrics        | 143 +++++++++++++++++++++++++
> config/rpz/rpz-sleep          |  58 ++++++++++
> lfs/rpz                       |  88 +++++++++++++++
> make.sh                       |   2 +
> 8 files changed, 519 insertions(+)
> create mode 100644 config/backup/includes/rpz
> create mode 100644 config/rootfiles/packages/rpz
> create mode 100644 config/rpz/00-rpz.conf
> create mode 100644 config/rpz/rpz-config
> create mode 100644 config/rpz/rpz-metrics
> create mode 100644 config/rpz/rpz-sleep
> create mode 100644 lfs/rpz
> 
> diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz
> new file mode 100644
> index 000000000..4d59bb40c
> --- /dev/null
> +++ b/config/backup/includes/rpz
> @@ -0,0 +1,5 @@
> +/var/ipfire/rpz/allowlist
> +/var/ipfire/rpz/blocklist
> +/etc/unbound/zonefiles/allow.rpz
> +/etc/unbound/zonefiles/block.rpz
> +/etc/unbound/local.d/*rpz.conf
> diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/rpz
> new file mode 100644
> index 000000000..2ffa715dd
> --- /dev/null
> +++ b/config/rootfiles/packages/rpz
> @@ -0,0 +1,11 @@
> +etc/unbound/local.d/00-rpz.conf
> +etc/unbound/zonefiles
> +etc/unbound/zonefiles/allow.rpz
> +etc/unbound/zonefiles/block.rpz
> +usr/sbin/rpz-config
> +usr/sbin/rpz-metrics
> +usr/sbin/rpz-sleep
> +var/ipfire/backup/addons/includes/rpz
> +var/ipfire/rpz
> +var/ipfire/rpz/allowlist
> +var/ipfire/rpz/blocklist
> diff --git a/config/rpz/00-rpz.conf b/config/rpz/00-rpz.conf
> new file mode 100644
> index 000000000..72c1d12e5
> --- /dev/null
> +++ b/config/rpz/00-rpz.conf
> @@ -0,0 +1,18 @@
> +server:
> + module-config: "respip validator iterator"
> +
> +rpz:
> +    name: allow.rpz
> +    zonefile: /etc/unbound/zonefiles/allow.rpz
> +    rpz-action-override: passthru
> +    rpz-log: yes
> +    rpz-log-name: allow
> +    rpz-signal-nxdomain-ra: yes
> +
> +rpz:
> +    name: block.rpz
> +    zonefile: /etc/unbound/zonefiles/block.rpz
> +    rpz-action-override: nxdomain
> +    rpz-log: yes
> +    rpz-log-name: block
> +    rpz-signal-nxdomain-ra: yes
> diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config
> new file mode 100644
> index 000000000..98dc0a4ca
> --- /dev/null
> +++ b/config/rpz/rpz-config
> @@ -0,0 +1,194 @@
> +#!/bin/bash
> +###############################################################################
> +#                                                                             #
> +#  IPFire.org - A linux based firewall                                        #
> +#  Copyright (C) 2024  IPFire Team  <info@ipfire.org>                         #
> +#                                                                             #
> +#  This program is free software: you can redistribute it and/or modify       #
> +#  it under the terms of the GNU General Public License as published by       #
> +#  the Free Software Foundation, either version 3 of the License, or          #
> +#  (at your option) any later version.                                        #
> +#                                                                             #
> +#  This program is distributed in the hope that it will be useful,            #
> +#  but WITHOUT ANY WARRANTY; without even the implied warranty of             #
> +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the              #
> +#  GNU General Public License for more details.                               #
> +#                                                                             #
> +#  You should have received a copy of the GNU General Public License          #
> +#  along with this program.  If not, see <http://www.gnu.org/licenses/>.      #
> +#                                                                             #
> +###############################################################################
> +
> +# v22 - 2024-07-12
> +
> +###############     Functions     ###############
> +
> +msg_log () {
> + /usr/bin/logger --tag "${tagName}" "$*"
> + if tty --silent ; then
> + echo "${tagName}:" "$*"
> + fi
> +}
> +
> +check_name () {
> + local testName="${1}"
> + # check for a valid name
> + regex='^[a-zA-Z0-9_]+$'
> + if [[ ! "${testName}" =~ $regex ]] ; then
> + msg_log "error: rpz: the NAME is not valid: \"${testName}\". exit."
> + exit 1
> + fi
> +}
> +
> +check_unbound_conf () {
> + # check the above config files
> + msg_log "info: rpz: check for errors with \"unbound-checkconf\""
> + /usr/sbin/unbound-checkconf
> + exit_code=$?
> + if [[ "${exit_code}" -ne 0 ]] ; then
> + msg_log "error: rpz: unbound-checkconf. exit."
> + exit "${exit_code}"
> + fi
> +}
> +
> +make_rpz_file () {
> + local theType="${1}" # allow or block
> +
> + theList="/var/ipfire/rpz/${theType}list" # input user list of domains
> + theZoneFile="/etc/unbound/zonefiles/${theType}.rpz" # output file for RPZ
> +
> + theAction='.'
> + if [[ "${theType}" =~ "block" ]] ; then
> + theAction='rpz-passthru.'
> + fi
> +
> + # does a list exist?
> + if [[ -s "${theList}" ]] ; then
> + # drop any extra "blanks" and add "CNAME <RPZ action>." to each line
> + actionList=$( /usr/bin/awk '{$1=$1};1' "${theList}" |
> + /bin/sed "/^[^;].*[[:alnum:]]/ s|$| CNAME ${theAction}|" )
> +
> + msg_log "info: rpz: create zonefile for ${theList}"
> +
> + /bin/cat <<-EOF > "${theZoneFile}"
> + ; Name: ${theType} list
> + ; Last modified: $(date "+%Y-%m-%d at %H.%M.%S %Z")
> + ;
> + ; domains with actions list
> + ;
> + ${actionList}
> + EOF
> +
> + # reload the zone that was just updated
> + zoneBase=$( basename "${theZoneFile}" )
> + /usr/sbin/unbound-control auth_zone_reload -q "${zoneBase}"
> + fi
> +}
> +
> +###############       Main        ###############
> +
> +tagName="unbound"
> +
> +theAction="${1}" # input action
> +theName="${2}" # input RPZ name
> +theURL="${3}" # input RPZ URL
> +
> +check_name "${theName}" # is this a valid name?
> +
> +rpzConfig="/etc/unbound/local.d/${theName}.rpz.conf" # output zone conf file
> +rpzFile="/etc/unbound/zonefiles/${theName}.rpz" # output for RPZ file
> +
> +case "${theAction}" in
> +
> + # add new rpz list
> + add )
> + # does this config already exist?  If yes, then exit
> + if [[ -f "${rpzConfig}" ]] ; then
> + msg_log "info: rpz: ${rpzConfig} already exists. exit"
> + exit 1
> + fi
> +
> + # is this a valid URL?
> + regex='^https://[-[:alnum:]\+&@#/%?=~_|!:,.;]*[-[:alnum:]\+&@#/%=~_|]'
> + if ! [[ "${theURL}" =~ $regex ]] ; then
> + msg_log "error: rpz: the URL is not valid: \"${theURL}\". exit."
> + exit 1
> + fi
> +
> + # create the zone config file
> + msg_log "info: rpz: add config file \"${theName}.rpz.conf\""
> + cat <<-EOF > "${rpzConfig}"
> + rpz:
> + name: ${theName}.rpz
> + zonefile: /etc/unbound/zonefiles/${theName}.rpz
> + url:  ${theURL}
> + rpz-action-override: nxdomain
> + rpz-log: yes
> + rpz-log-name: ${theName}
> + rpz-signal-nxdomain-ra: yes
> + EOF
> +
> + # set-up zone file
> + /usr/bin/touch "${rpzFile}"
> + # unbound requires these settings for rpz files
> + /bin/chown --verbose nobody:nobody "${rpzFile}"
> + /bin/chmod --verbose 644 "${rpzFile}"
> + ;;
> +
> + # trash config file & rpz file
> + remove )
> + if ! [[ -f "${rpzConfig}" ]] ; then
> + msg_log "info: rpz: ${rpzConfig} does not exist. exit"
> + exit 1
> + fi
> +
> + msg_log "info: rpz: remove config file & rpz file \"${theName}\""
> + /bin/rm --verbose "${rpzConfig}"
> + /bin/rm --verbose "${rpzFile}"
> +
> + check_unbound_conf
> + ;;
> +
> + # make a new allow or block rpz file
> + make )
> + case "${theName}" in
> + allow )
> + make_rpz_file allow
> + ;;
> +
> + block )
> + make_rpz_file block
> + ;;
> +
> + allowblock )
> + make_rpz_file allow
> + make_rpz_file block
> + ;;
> +
> + * )
> + msg_log "error: rpz: the NAME is not valid: \"${theName}\". exit."
> + exit 1
> + ;;
> + esac
> +
> + check_unbound_conf
> + ;;
> +
> + *)
> + msg_log "error: rpz: missing or incorrect parameter"
> + /usr/bin/printf "Usage:  rpzConfig.sh <ACTION> <NAME> <URL>\n"
> + exit 1
> + ;;
> +
> +esac
> +
> +# reload due to the changes
> +msg_log  "rpz: running \"unbound-control reload\""
> +/usr/sbin/unbound-control reload
> +exit_code=$?
> +if [[ "${exit_code}" -ne 0 ]] ; then
> + msg_log "error: rpz: unbound-control \"${theName}\". exit."
> + exit "${exit_code}"
> +fi
> +
> +exit
> diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics
> new file mode 100644
> index 000000000..0f97c7911
> --- /dev/null
> +++ b/config/rpz/rpz-metrics
> @@ -0,0 +1,143 @@
> +#!/bin/bash
> +###############################################################################
> +#                                                                             #
> +#  IPFire.org - A linux based firewall                                        #
> +#  Copyright (C) 2024  IPFire Team  <info@ipfire.org>                         #
> +#                                                                             #
> +#  This program is free software: you can redistribute it and/or modify       #
> +#  it under the terms of the GNU General Public License as published by       #
> +#  the Free Software Foundation, either version 3 of the License, or          #
> +#  (at your option) any later version.                                        #
> +#                                                                             #
> +#  This program is distributed in the hope that it will be useful,            #
> +#  but WITHOUT ANY WARRANTY; without even the implied warranty of             #
> +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the              #
> +#  GNU General Public License for more details.                               #
> +#                                                                             #
> +#  You should have received a copy of the GNU General Public License          #
> +#  along with this program.  If not, see <http://www.gnu.org/licenses/>.      #
> +#                                                                             #
> +###############################################################################
> +
> +# v18 on 2024-07-05
> +
> +###############       Main        ###############
> +
> +weeks="${1:-2}" # default to two message logs
> +sortBy="${2:-name}"   # by name or by hits
> +
> +# get the list of message logs for N weeks
> +messageLogs=$( find /var/log/messages* -type f |
> + /usr/bin/sort --version-sort |
> + head -"${weeks}" )
> +
> +# get the list of RPZ names & counts from the message log(s)
> +rpzNameCount=$( for logf in ${messageLogs} ; do
> + /usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" |
> + /usr/bin/awk '$10 ~ /\[\w*]/ { print $10 }' ;
> + done | /usr/bin/sort | /usr/bin/uniq --count )
> +
> +# flip results and remove brackets `[` and `]`
> +rpzNameCount=$( /bin/echo "${rpzNameCount}" |
> + /usr/bin/awk '{ print $2, $1 }' |
> + /bin/sed --regexp-extended 's|^\[(.*)\]|\1|' )
> +
> +# grab only names
> +rpzNames=$( /bin/echo "${rpzNameCount}" | /usr/bin/awk '{ print $1 }' )
> +
> +# get list of RPZ files
> +rpzFileList=$( /bin/find /etc/unbound/zonefiles -type f -iname "*.rpz" )
> +
> +# get basename of those files
> +rpzBaseNames=$( /bin/echo "${rpzFileList}" |
> + /bin/sed 's|/etc/unbound/zonefiles/||g ; s|\.rpz||g ;' )
> +
> +# add to rpzNames
> +rpzNames="${rpzNames}"$'\n'"${rpzBaseNames}"
> +
> +# drop duplicate names
> +rpzNames=$( echo "${rpzNames}" | /usr/bin/sort --unique  )
> +
> +# get line count for each RPZ
> +lineCount=$( /bin/echo "${rpzFileList}" | /usr/bin/xargs wc -l )
> +
> +# get comment line count and blank line count for each RPZ
> +commentCount=$( /bin/echo "${rpzFileList}" |
> + /usr/bin/xargs /bin/grep --count -e "^$" -e "^;" )
> +
> +# get modified date each RPZ
> +modDateList=$( /bin/echo "${rpzFileList}" | xargs stat -c '%.10y  %n' )
> +
> +ucListAuthZones=$( /usr/sbin/unbound-control list_auth_zones )
> +
> +# get width of RPZ names
> +pWidth=$( /bin/echo "${rpzNames}" | /usr/bin/awk '{ print $1"   " }' | wc -L )
> +pFormat="%-${pWidth}s %-8s %-8s %-8s %10s %12s\n"
> +
> +# print title line
> +printf "${pFormat}" "name" "hits" "active" "lines" "hits/line%" "last update"
> +printf -- "--------------"
> +
> +theResults=""
> +totalLines=0
> +totalHits=0
> +while read -r theName
> +do
> + printf -- "-" # pretend progress bar
> + # get hit count
> + theHits="0"
> + if output=$( /bin/grep "^${theName}\s" <<< "${rpzNameCount}" ) ; then
> + theHits=$( /bin/echo "${output}" |
> + /usr/bin/awk '{ print $2 }' )
> + totalHits=$(( totalHits + theHits ))
> + fi
> +
> + # is this RPZ list active?
> + theActive="disabled"
> + if /bin/grep --quiet "^${theName}\.rpz" <<< "${ucListAuthZones}"
> + then
> + theActive="enabled"
> + fi
> +
> + # get line count then subtract comment count and blank line count
> + #  from total line count
> + theLines="n/a"
> + hitsPerLine="0"
> + if output=$( /bin/grep --fixed-strings "/${theName}.rpz" <<< "${lineCount}" ) ; then
> + theLines=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' )
> + totalLines=$(( totalLines + theLines ))
> +
> + #hitsPerLine=$( echo "scale=0 ; $theHits / $theLines" | bc )
> + hitsPerLine=$(( 100 * theHits / theLines ))
> + fi
> +
> + # get modification date
> + theModDate="n/a"
> + if output=$( /bin/grep --fixed-strings "/${theName}.rpz" <<< "${modDateList}" ) ; then
> + theModDate=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' )
> + fi
> +
> + # add to results list
> + theResults+="${theName} ${theHits} ${theActive} ${theLines} ${hitsPerLine} ${theModDate}"$'\n'
> +done <<< "${rpzNames}"
> +
> +case "${sortBy}" in
> + names|name) sortArg=(-k3,3r -k1,1) ;; # sort by "active" then by "name"
> +
> + hits|hit) sortArg=(-k3,3r -k2,2nr -k1,1) ;; # sort by "active" then by "hits" then by "name"
> +
> + lines|line) sortArg=(-k3,3r -k4,4nr -k1,1) ;; # sort by "active" then by "lines" then by "name"
> +esac
> +
> +printf -- "--------------\n"
> +# remove blank lines, sort, print as columns
> +/bin/echo "${theResults}" |
> + /usr/bin/awk '!/^[[:space:]]*$/' |
> + /usr/bin/sort "${sortArg[@]}"    |
> + /usr/bin/awk --assign=width="${pWidth}" \
> + '{ printf "%-*s %-8s %-8s %-8s %10s %12s\n", width, $1, $2, $3, $4, $5, $6 }'
> +
> +printf "${pFormat}" "" "=======" "" "========" "" ""
> +printf "${pFormat}" "Totals -->" "${totalHits}" "" "${totalLines}" "" ""
> +
> +exit
> diff --git a/config/rpz/rpz-sleep b/config/rpz/rpz-sleep
> new file mode 100644
> index 000000000..eeef1174a
> --- /dev/null
> +++ b/config/rpz/rpz-sleep
> @@ -0,0 +1,58 @@
> +#!/bin/bash
> +###############################################################################
> +#                                                                             #
> +#  IPFire.org - A linux based firewall                                        #
> +#  Copyright (C) 2024  IPFire Team  <info@ipfire.org>                         #
> +#                                                                             #
> +#  This program is free software: you can redistribute it and/or modify       #
> +#  it under the terms of the GNU General Public License as published by       #
> +#  the Free Software Foundation, either version 3 of the License, or          #
> +#  (at your option) any later version.                                        #
> +#                                                                             #
> +#  This program is distributed in the hope that it will be useful,            #
> +#  but WITHOUT ANY WARRANTY; without even the implied warranty of             #
> +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the              #
> +#  GNU General Public License for more details.                               #
> +#                                                                             #
> +#  You should have received a copy of the GNU General Public License          #
> +#  along with this program.  If not, see <http://www.gnu.org/licenses/>.      #
> +#                                                                             #
> +###############################################################################
> +
> +# v04 on 2024-07-05
> +
> +###############     Functions     ###############
> +
> +# send message to message log
> +msg_log () {
> + /usr/bin/logger --tag "${tagName}" "$*"
> + if /usr/bin/tty --silent ; then
> + echo "${tagName}:" "$*"
> + fi
> +}
> +
> +###############       Main        ###############
> +
> +tagName="unbound"
> +
> +sleepTime="${1:-5m}" # default to sleep for 5m (5 minutes)
> +
> +zoneList=$( /usr/sbin/unbound-control list_auth_zones | /usr/bin/awk '{print $1}' )
> +
> +for zone in ${zoneList} ; do
> + /usr/bin/printf "disable ${zone}\t"
> + /usr/sbin/unbound-control rpz_disable "${zone}"
> +done
> +
> +msg_log "info: rpz: disabled all zones for ${sleepTime}"
> +
> +/bin/sleep "${sleepTime}"
> +
> +for zone in ${zoneList} ; do
> + /usr/bin/printf "enable ${zone}\t"
> + /usr/sbin/unbound-control rpz_enable "${zone}"
> +done
> +
> +msg_log "info: rpz: enabled all zones"
> +
> +exit
> diff --git a/lfs/rpz b/lfs/rpz
> new file mode 100644
> index 000000000..319c10b7f
> --- /dev/null
> +++ b/lfs/rpz
> @@ -0,0 +1,88 @@
> +###############################################################################
> +#                                                                             #
> +# IPFire.org - A linux based firewall                                         #
> +# Copyright (C) 2024  IPFire Team  <info@ipfire.org>                          #
> +#                                                                             #
> +# This program is free software: you can redistribute it and/or modify        #
> +# it under the terms of the GNU General Public License as published by        #
> +# the Free Software Foundation, either version 3 of the License, or           #
> +# (at your option) any later version.                                         #
> +#                                                                             #
> +# This program is distributed in the hope that it will be useful,             #
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of              #
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the               #
> +# GNU General Public License for more details.                                #
> +#                                                                             #
> +# You should have received a copy of the GNU General Public License           #
> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.       #
> +#                                                                             #
> +###############################################################################
> +
> +###############################################################################
> +# Definitions
> +###############################################################################
> +
> +include Config
> +
> +SUMMARY    = response policy zone - RPZ reputation system for unbound DNS
> +
> +VER        = 1.0.0
> +
> +THISAPP    = rpz-$(VER)
> +DIR_APP    = $(DIR_SRC)/$(THISAPP)
> +TARGET     = $(DIR_INFO)/$(THISAPP)
> +
> +PROG       = rpz
> +PAK_VER    = 1
> +
> +DEPS       =
> +
> +SERVICES   =
> +
> +###############################################################################
> +# Top-level Rules
> +###############################################################################
> +
> +install : $(TARGET)
> +
> +check :
> +
> +download :
> +
> +b2 :
> +
> +dist:
> + @$(PAK)
> +
> +###############################################################################
> +# Installation Details
> +###############################################################################
> +
> +$(TARGET) :
> + @$(PREBUILD)
> + @rm -rf $(DIR_APP)
> +
> + # install RPZ scripts
> + install -v -m 755 \
> +  $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin
> +
> + # Install settings folder and two empty files
> + mkdir -pv /var/ipfire/rpz
> + touch /var/ipfire/rpz/allowlist
> + touch /var/ipfire/rpz/blocklist
> +
> + # Add conf file to /etc directory
> + cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d
> +
> + # create zonefiles directory for the RPZ files and add two empty RPZ
> + #  files to avoid a unbound config error
> + mkdir -pv /etc/unbound/zonefiles
> + chown -v nobody:nobody /etc/unbound/zonefiles
> + touch /etc/unbound/zonefiles/allow.rpz
> + touch /etc/unbound/zonefiles/block.rpz
> +
> + # Install backup definition
> + cp -vf $(DIR_CONF)/backup/includes/rpz /var/ipfire/backup/addons/includes/rpz
> +
> + @rm -rf $(DIR_APP)
> + @$(POSTBUILD)
> diff --git a/make.sh b/make.sh
> index 9bbbeb0f1..886d3760a 100755
> --- a/make.sh
> +++ b/make.sh
> @@ -1721,6 +1721,8 @@ buildipfire() {
>   lfsmake2 btrfs-progs
>   lfsmake2 inotify-tools
>   lfsmake2 grub-btrfs
> +  lfsmake2 rpz
> +
> 
>   # Kernelbuild ... current we have no platform that need
>   # multi kernel builds so KCFG is empty
> -- 
> 2.30.2
>

Jon Murphy July 27, 2024, 8:31 p.m. UTC | #2

And another question.

I found an error in the `rpz-config` script file.  Simple easy one line (actually one word) fix.  

Do I resubmit the entire patch?
-or-
Do I wait until approved and then submit the simple patch for the one line change.


Jon


> On Jul 20, 2024, at 1:42 PM, jon <jon.murphy@ipfire.org> wrote:
> 
> I need help with…
> 
> 1) The custom allow and block lists are currently located at `/var/ipfire/rpz`.  Is this correct?
> 
> 2) The three bash scripts are currently located at `/usr/sbin`.  Is this correct?
> 
> 3) Should this commit be separated into multiple posts?  If yes, how do I decide how to split it?
> 
> 
> Jon 
> 
> 
>> On Jul 20, 2024, at 12:46 PM, Jon Murphy <jon.murphy@ipfire.org> wrote:
>> 
>> What is it?
>> Response Policy Zone (RPZ) is a mechanism to define local policies in a standardised
>> way and load those policies from external sources.
>> Bottom line: RPZ allows admins to easily block access to websites via DNS lookup.
>> 
>> RPZ can block websites via categories.  Examples include: fake websites, annoying
>> pop-up ads, newly registered domains, DoH bypass sites, bad "host" services,
>> maliscious top level domains (e.g., *.zip, *.mov), piracy, gambling, pornography,
>> and more.  RPZ lists come from various RPZ providers and their available
>> catagories.
>> 
>> This RPZ add-on enables the RPZ functionality by adding a couple lines in a
>> configuration file.  This add-on simply adds a configuration file and adds
>> three scripts (config, metrics and sleep) to make RPZ easier for the admin to use.
>> 
>> RPZ was release in 2010 and has been part of the IPFire build since ~2015.
>> 
>> Why is it needed?
>> Some IPFire admin's utilize pihole to block unwanted websites via DNS lookup.
>> Moving the pihole base functionality (without pretty graphs) to IPFire removes
>> one device from the admin's local network.
>> And hopefully this reduces the pihole questions from the Community.
>> 
>> A list of RPZ providers can be recommended by IPFire and coded into a set list.
>> Or, if prefered, the local admin can choose their own RPZ providers.
>> 
>> This is a:
>> * simple replacement for pihole base functionality
>> * RPZ can be a nice replacement for the URL Filter
>> 
>> IPFire Wiki
>> In process at: https://www.ipfire.org/docs/addons/rpz
>> 
>> more info:
>> * https://en.wikipedia.org/wiki/Response_policy_zone
>> * https://unbound.docs.nlnetlabs.nl/en/latest/topics/filtering/rpz.html
>> 
>> I need help with...
>> 1) The custom allow and block lists are currently located at `/var/ipfire/rpz`.  Is this correct?
>> 2) The three bash scripts are currently located at `/usr/sbin`.  Is this correct?
>> 
>> Signed-off-by: Jon Murphy <jon.murphy@ipfire.org>
>> ---
>> config/backup/includes/rpz    |   5 +
>> config/rootfiles/packages/rpz |  11 ++
>> config/rpz/00-rpz.conf        |  18 ++++
>> config/rpz/rpz-config         | 194 ++++++++++++++++++++++++++++++++++
>> config/rpz/rpz-metrics        | 143 +++++++++++++++++++++++++
>> config/rpz/rpz-sleep          |  58 ++++++++++
>> lfs/rpz                       |  88 +++++++++++++++
>> make.sh                       |   2 +
>> 8 files changed, 519 insertions(+)
>> create mode 100644 config/backup/includes/rpz
>> create mode 100644 config/rootfiles/packages/rpz
>> create mode 100644 config/rpz/00-rpz.conf
>> create mode 100644 config/rpz/rpz-config
>> create mode 100644 config/rpz/rpz-metrics
>> create mode 100644 config/rpz/rpz-sleep
>> create mode 100644 lfs/rpz
>> 
>> diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz
>> new file mode 100644
>> index 000000000..4d59bb40c
>> --- /dev/null
>> +++ b/config/backup/includes/rpz
>> @@ -0,0 +1,5 @@
>> +/var/ipfire/rpz/allowlist
>> +/var/ipfire/rpz/blocklist
>> +/etc/unbound/zonefiles/allow.rpz
>> +/etc/unbound/zonefiles/block.rpz
>> +/etc/unbound/local.d/*rpz.conf
>> diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/rpz
>> new file mode 100644
>> index 000000000..2ffa715dd
>> --- /dev/null
>> +++ b/config/rootfiles/packages/rpz
>> @@ -0,0 +1,11 @@
>> +etc/unbound/local.d/00-rpz.conf
>> +etc/unbound/zonefiles
>> +etc/unbound/zonefiles/allow.rpz
>> +etc/unbound/zonefiles/block.rpz
>> +usr/sbin/rpz-config
>> +usr/sbin/rpz-metrics
>> +usr/sbin/rpz-sleep
>> +var/ipfire/backup/addons/includes/rpz
>> +var/ipfire/rpz
>> +var/ipfire/rpz/allowlist
>> +var/ipfire/rpz/blocklist
>> diff --git a/config/rpz/00-rpz.conf b/config/rpz/00-rpz.conf
>> new file mode 100644
>> index 000000000..72c1d12e5
>> --- /dev/null
>> +++ b/config/rpz/00-rpz.conf
>> @@ -0,0 +1,18 @@
>> +server:
>> + module-config: "respip validator iterator"
>> +
>> +rpz:
>> +    name: allow.rpz
>> +    zonefile: /etc/unbound/zonefiles/allow.rpz
>> +    rpz-action-override: passthru
>> +    rpz-log: yes
>> +    rpz-log-name: allow
>> +    rpz-signal-nxdomain-ra: yes
>> +
>> +rpz:
>> +    name: block.rpz
>> +    zonefile: /etc/unbound/zonefiles/block.rpz
>> +    rpz-action-override: nxdomain
>> +    rpz-log: yes
>> +    rpz-log-name: block
>> +    rpz-signal-nxdomain-ra: yes
>> diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config
>> new file mode 100644
>> index 000000000..98dc0a4ca
>> --- /dev/null
>> +++ b/config/rpz/rpz-config
>> @@ -0,0 +1,194 @@
>> +#!/bin/bash
>> +###############################################################################
>> +#                                                                             #
>> +#  IPFire.org - A linux based firewall                                        #
>> +#  Copyright (C) 2024  IPFire Team  <info@ipfire.org>                         #
>> +#                                                                             #
>> +#  This program is free software: you can redistribute it and/or modify       #
>> +#  it under the terms of the GNU General Public License as published by       #
>> +#  the Free Software Foundation, either version 3 of the License, or          #
>> +#  (at your option) any later version.                                        #
>> +#                                                                             #
>> +#  This program is distributed in the hope that it will be useful,            #
>> +#  but WITHOUT ANY WARRANTY; without even the implied warranty of             #
>> +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the              #
>> +#  GNU General Public License for more details.                               #
>> +#                                                                             #
>> +#  You should have received a copy of the GNU General Public License          #
>> +#  along with this program.  If not, see <http://www.gnu.org/licenses/>.      #
>> +#                                                                             #
>> +###############################################################################
>> +
>> +# v22 - 2024-07-12
>> +
>> +###############     Functions     ###############
>> +
>> +msg_log () {
>> + /usr/bin/logger --tag "${tagName}" "$*"
>> + if tty --silent ; then
>> + echo "${tagName}:" "$*"
>> + fi
>> +}
>> +
>> +check_name () {
>> + local testName="${1}"
>> + # check for a valid name
>> + regex='^[a-zA-Z0-9_]+$'
>> + if [[ ! "${testName}" =~ $regex ]] ; then
>> + msg_log "error: rpz: the NAME is not valid: \"${testName}\". exit."
>> + exit 1
>> + fi
>> +}
>> +
>> +check_unbound_conf () {
>> + # check the above config files
>> + msg_log "info: rpz: check for errors with \"unbound-checkconf\""
>> + /usr/sbin/unbound-checkconf
>> + exit_code=$?
>> + if [[ "${exit_code}" -ne 0 ]] ; then
>> + msg_log "error: rpz: unbound-checkconf. exit."
>> + exit "${exit_code}"
>> + fi
>> +}
>> +
>> +make_rpz_file () {
>> + local theType="${1}" # allow or block
>> +
>> + theList="/var/ipfire/rpz/${theType}list" # input user list of domains
>> + theZoneFile="/etc/unbound/zonefiles/${theType}.rpz" # output file for RPZ
>> +
>> + theAction='.'
>> + if [[ "${theType}" =~ "block" ]] ; then
>> + theAction='rpz-passthru.'
>> + fi
>> +
>> + # does a list exist?
>> + if [[ -s "${theList}" ]] ; then
>> + # drop any extra "blanks" and add "CNAME <RPZ action>." to each line
>> + actionList=$( /usr/bin/awk '{$1=$1};1' "${theList}" |
>> + /bin/sed "/^[^;].*[[:alnum:]]/ s|$| CNAME ${theAction}|" )
>> +
>> + msg_log "info: rpz: create zonefile for ${theList}"
>> +
>> + /bin/cat <<-EOF > "${theZoneFile}"
>> + ; Name: ${theType} list
>> + ; Last modified: $(date "+%Y-%m-%d at %H.%M.%S %Z")
>> + ;
>> + ; domains with actions list
>> + ;
>> + ${actionList}
>> + EOF
>> +
>> + # reload the zone that was just updated
>> + zoneBase=$( basename "${theZoneFile}" )
>> + /usr/sbin/unbound-control auth_zone_reload -q "${zoneBase}"
>> + fi
>> +}
>> +
>> +###############       Main        ###############
>> +
>> +tagName="unbound"
>> +
>> +theAction="${1}" # input action
>> +theName="${2}" # input RPZ name
>> +theURL="${3}" # input RPZ URL
>> +
>> +check_name "${theName}" # is this a valid name?
>> +
>> +rpzConfig="/etc/unbound/local.d/${theName}.rpz.conf" # output zone conf file
>> +rpzFile="/etc/unbound/zonefiles/${theName}.rpz" # output for RPZ file
>> +
>> +case "${theAction}" in
>> +
>> + # add new rpz list
>> + add )
>> + # does this config already exist?  If yes, then exit
>> + if [[ -f "${rpzConfig}" ]] ; then
>> + msg_log "info: rpz: ${rpzConfig} already exists. exit"
>> + exit 1
>> + fi
>> +
>> + # is this a valid URL?
>> + regex='^https://[-[:alnum:]\+&@#/%?=~_|!:,.;]*[-[:alnum:]\+&@#/%=~_|]'
>> + if ! [[ "${theURL}" =~ $regex ]] ; then
>> + msg_log "error: rpz: the URL is not valid: \"${theURL}\". exit."
>> + exit 1
>> + fi
>> +
>> + # create the zone config file
>> + msg_log "info: rpz: add config file \"${theName}.rpz.conf\""
>> + cat <<-EOF > "${rpzConfig}"
>> + rpz:
>> + name: ${theName}.rpz
>> + zonefile: /etc/unbound/zonefiles/${theName}.rpz
>> + url:  ${theURL}
>> + rpz-action-override: nxdomain
>> + rpz-log: yes
>> + rpz-log-name: ${theName}
>> + rpz-signal-nxdomain-ra: yes
>> + EOF
>> +
>> + # set-up zone file
>> + /usr/bin/touch "${rpzFile}"
>> + # unbound requires these settings for rpz files
>> + /bin/chown --verbose nobody:nobody "${rpzFile}"
>> + /bin/chmod --verbose 644 "${rpzFile}"
>> + ;;
>> +
>> + # trash config file & rpz file
>> + remove )
>> + if ! [[ -f "${rpzConfig}" ]] ; then
>> + msg_log "info: rpz: ${rpzConfig} does not exist. exit"
>> + exit 1
>> + fi
>> +
>> + msg_log "info: rpz: remove config file & rpz file \"${theName}\""
>> + /bin/rm --verbose "${rpzConfig}"
>> + /bin/rm --verbose "${rpzFile}"
>> +
>> + check_unbound_conf
>> + ;;
>> +
>> + # make a new allow or block rpz file
>> + make )
>> + case "${theName}" in
>> + allow )
>> + make_rpz_file allow
>> + ;;
>> +
>> + block )
>> + make_rpz_file block
>> + ;;
>> +
>> + allowblock )
>> + make_rpz_file allow
>> + make_rpz_file block
>> + ;;
>> +
>> + * )
>> + msg_log "error: rpz: the NAME is not valid: \"${theName}\". exit."
>> + exit 1
>> + ;;
>> + esac
>> +
>> + check_unbound_conf
>> + ;;
>> +
>> + *)
>> + msg_log "error: rpz: missing or incorrect parameter"
>> + /usr/bin/printf "Usage:  rpzConfig.sh <ACTION> <NAME> <URL>\n"
>> + exit 1
>> + ;;
>> +
>> +esac
>> +
>> +# reload due to the changes
>> +msg_log  "rpz: running \"unbound-control reload\""
>> +/usr/sbin/unbound-control reload
>> +exit_code=$?
>> +if [[ "${exit_code}" -ne 0 ]] ; then
>> + msg_log "error: rpz: unbound-control \"${theName}\". exit."
>> + exit "${exit_code}"
>> +fi
>> +
>> +exit
>> diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics
>> new file mode 100644
>> index 000000000..0f97c7911
>> --- /dev/null
>> +++ b/config/rpz/rpz-metrics
>> @@ -0,0 +1,143 @@
>> +#!/bin/bash
>> +###############################################################################
>> +#                                                                             #
>> +#  IPFire.org - A linux based firewall                                        #
>> +#  Copyright (C) 2024  IPFire Team  <info@ipfire.org>                         #
>> +#                                                                             #
>> +#  This program is free software: you can redistribute it and/or modify       #
>> +#  it under the terms of the GNU General Public License as published by       #
>> +#  the Free Software Foundation, either version 3 of the License, or          #
>> +#  (at your option) any later version.                                        #
>> +#                                                                             #
>> +#  This program is distributed in the hope that it will be useful,            #
>> +#  but WITHOUT ANY WARRANTY; without even the implied warranty of             #
>> +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the              #
>> +#  GNU General Public License for more details.                               #
>> +#                                                                             #
>> +#  You should have received a copy of the GNU General Public License          #
>> +#  along with this program.  If not, see <http://www.gnu.org/licenses/>.      #
>> +#                                                                             #
>> +###############################################################################
>> +
>> +# v18 on 2024-07-05
>> +
>> +###############       Main        ###############
>> +
>> +weeks="${1:-2}" # default to two message logs
>> +sortBy="${2:-name}"   # by name or by hits
>> +
>> +# get the list of message logs for N weeks
>> +messageLogs=$( find /var/log/messages* -type f |
>> + /usr/bin/sort --version-sort |
>> + head -"${weeks}" )
>> +
>> +# get the list of RPZ names & counts from the message log(s)
>> +rpzNameCount=$( for logf in ${messageLogs} ; do
>> + /usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" |
>> + /usr/bin/awk '$10 ~ /\[\w*]/ { print $10 }' ;
>> + done | /usr/bin/sort | /usr/bin/uniq --count )
>> +
>> +# flip results and remove brackets `[` and `]`
>> +rpzNameCount=$( /bin/echo "${rpzNameCount}" |
>> + /usr/bin/awk '{ print $2, $1 }' |
>> + /bin/sed --regexp-extended 's|^\[(.*)\]|\1|' )
>> +
>> +# grab only names
>> +rpzNames=$( /bin/echo "${rpzNameCount}" | /usr/bin/awk '{ print $1 }' )
>> +
>> +# get list of RPZ files
>> +rpzFileList=$( /bin/find /etc/unbound/zonefiles -type f -iname "*.rpz" )
>> +
>> +# get basename of those files
>> +rpzBaseNames=$( /bin/echo "${rpzFileList}" |
>> + /bin/sed 's|/etc/unbound/zonefiles/||g ; s|\.rpz||g ;' )
>> +
>> +# add to rpzNames
>> +rpzNames="${rpzNames}"$'\n'"${rpzBaseNames}"
>> +
>> +# drop duplicate names
>> +rpzNames=$( echo "${rpzNames}" | /usr/bin/sort --unique  )
>> +
>> +# get line count for each RPZ
>> +lineCount=$( /bin/echo "${rpzFileList}" | /usr/bin/xargs wc -l )
>> +
>> +# get comment line count and blank line count for each RPZ
>> +commentCount=$( /bin/echo "${rpzFileList}" |
>> + /usr/bin/xargs /bin/grep --count -e "^$" -e "^;" )
>> +
>> +# get modified date each RPZ
>> +modDateList=$( /bin/echo "${rpzFileList}" | xargs stat -c '%.10y  %n' )
>> +
>> +ucListAuthZones=$( /usr/sbin/unbound-control list_auth_zones )
>> +
>> +# get width of RPZ names
>> +pWidth=$( /bin/echo "${rpzNames}" | /usr/bin/awk '{ print $1"   " }' | wc -L )
>> +pFormat="%-${pWidth}s %-8s %-8s %-8s %10s %12s\n"
>> +
>> +# print title line
>> +printf "${pFormat}" "name" "hits" "active" "lines" "hits/line%" "last update"
>> +printf -- "--------------"
>> +
>> +theResults=""
>> +totalLines=0
>> +totalHits=0
>> +while read -r theName
>> +do
>> + printf -- "-" # pretend progress bar
>> + # get hit count
>> + theHits="0"
>> + if output=$( /bin/grep "^${theName}\s" <<< "${rpzNameCount}" ) ; then
>> + theHits=$( /bin/echo "${output}" |
>> + /usr/bin/awk '{ print $2 }' )
>> + totalHits=$(( totalHits + theHits ))
>> + fi
>> +
>> + # is this RPZ list active?
>> + theActive="disabled"
>> + if /bin/grep --quiet "^${theName}\.rpz" <<< "${ucListAuthZones}"
>> + then
>> + theActive="enabled"
>> + fi
>> +
>> + # get line count then subtract comment count and blank line count
>> + #  from total line count
>> + theLines="n/a"
>> + hitsPerLine="0"
>> + if output=$( /bin/grep --fixed-strings "/${theName}.rpz" <<< "${lineCount}" ) ; then
>> + theLines=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' )
>> + totalLines=$(( totalLines + theLines ))
>> +
>> + #hitsPerLine=$( echo "scale=0 ; $theHits / $theLines" | bc )
>> + hitsPerLine=$(( 100 * theHits / theLines ))
>> + fi
>> +
>> + # get modification date
>> + theModDate="n/a"
>> + if output=$( /bin/grep --fixed-strings "/${theName}.rpz" <<< "${modDateList}" ) ; then
>> + theModDate=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' )
>> + fi
>> +
>> + # add to results list
>> + theResults+="${theName} ${theHits} ${theActive} ${theLines} ${hitsPerLine} ${theModDate}"$'\n'
>> +done <<< "${rpzNames}"
>> +
>> +case "${sortBy}" in
>> + names|name) sortArg=(-k3,3r -k1,1) ;; # sort by "active" then by "name"
>> +
>> + hits|hit) sortArg=(-k3,3r -k2,2nr -k1,1) ;; # sort by "active" then by "hits" then by "name"
>> +
>> + lines|line) sortArg=(-k3,3r -k4,4nr -k1,1) ;; # sort by "active" then by "lines" then by "name"
>> +esac
>> +
>> +printf -- "--------------\n"
>> +# remove blank lines, sort, print as columns
>> +/bin/echo "${theResults}" |
>> + /usr/bin/awk '!/^[[:space:]]*$/' |
>> + /usr/bin/sort "${sortArg[@]}"    |
>> + /usr/bin/awk --assign=width="${pWidth}" \
>> + '{ printf "%-*s %-8s %-8s %-8s %10s %12s\n", width, $1, $2, $3, $4, $5, $6 }'
>> +
>> +printf "${pFormat}" "" "=======" "" "========" "" ""
>> +printf "${pFormat}" "Totals -->" "${totalHits}" "" "${totalLines}" "" ""
>> +
>> +exit
>> diff --git a/config/rpz/rpz-sleep b/config/rpz/rpz-sleep
>> new file mode 100644
>> index 000000000..eeef1174a
>> --- /dev/null
>> +++ b/config/rpz/rpz-sleep
>> @@ -0,0 +1,58 @@
>> +#!/bin/bash
>> +###############################################################################
>> +#                                                                             #
>> +#  IPFire.org - A linux based firewall                                        #
>> +#  Copyright (C) 2024  IPFire Team  <info@ipfire.org>                         #
>> +#                                                                             #
>> +#  This program is free software: you can redistribute it and/or modify       #
>> +#  it under the terms of the GNU General Public License as published by       #
>> +#  the Free Software Foundation, either version 3 of the License, or          #
>> +#  (at your option) any later version.                                        #
>> +#                                                                             #
>> +#  This program is distributed in the hope that it will be useful,            #
>> +#  but WITHOUT ANY WARRANTY; without even the implied warranty of             #
>> +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the              #
>> +#  GNU General Public License for more details.                               #
>> +#                                                                             #
>> +#  You should have received a copy of the GNU General Public License          #
>> +#  along with this program.  If not, see <http://www.gnu.org/licenses/>.      #
>> +#                                                                             #
>> +###############################################################################
>> +
>> +# v04 on 2024-07-05
>> +
>> +###############     Functions     ###############
>> +
>> +# send message to message log
>> +msg_log () {
>> + /usr/bin/logger --tag "${tagName}" "$*"
>> + if /usr/bin/tty --silent ; then
>> + echo "${tagName}:" "$*"
>> + fi
>> +}
>> +
>> +###############       Main        ###############
>> +
>> +tagName="unbound"
>> +
>> +sleepTime="${1:-5m}" # default to sleep for 5m (5 minutes)
>> +
>> +zoneList=$( /usr/sbin/unbound-control list_auth_zones | /usr/bin/awk '{print $1}' )
>> +
>> +for zone in ${zoneList} ; do
>> + /usr/bin/printf "disable ${zone}\t"
>> + /usr/sbin/unbound-control rpz_disable "${zone}"
>> +done
>> +
>> +msg_log "info: rpz: disabled all zones for ${sleepTime}"
>> +
>> +/bin/sleep "${sleepTime}"
>> +
>> +for zone in ${zoneList} ; do
>> + /usr/bin/printf "enable ${zone}\t"
>> + /usr/sbin/unbound-control rpz_enable "${zone}"
>> +done
>> +
>> +msg_log "info: rpz: enabled all zones"
>> +
>> +exit
>> diff --git a/lfs/rpz b/lfs/rpz
>> new file mode 100644
>> index 000000000..319c10b7f
>> --- /dev/null
>> +++ b/lfs/rpz
>> @@ -0,0 +1,88 @@
>> +###############################################################################
>> +#                                                                             #
>> +# IPFire.org - A linux based firewall                                         #
>> +# Copyright (C) 2024  IPFire Team  <info@ipfire.org>                          #
>> +#                                                                             #
>> +# This program is free software: you can redistribute it and/or modify        #
>> +# it under the terms of the GNU General Public License as published by        #
>> +# the Free Software Foundation, either version 3 of the License, or           #
>> +# (at your option) any later version.                                         #
>> +#                                                                             #
>> +# This program is distributed in the hope that it will be useful,             #
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of              #
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the               #
>> +# GNU General Public License for more details.                                #
>> +#                                                                             #
>> +# You should have received a copy of the GNU General Public License           #
>> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.       #
>> +#                                                                             #
>> +###############################################################################
>> +
>> +###############################################################################
>> +# Definitions
>> +###############################################################################
>> +
>> +include Config
>> +
>> +SUMMARY    = response policy zone - RPZ reputation system for unbound DNS
>> +
>> +VER        = 1.0.0
>> +
>> +THISAPP    = rpz-$(VER)
>> +DIR_APP    = $(DIR_SRC)/$(THISAPP)
>> +TARGET     = $(DIR_INFO)/$(THISAPP)
>> +
>> +PROG       = rpz
>> +PAK_VER    = 1
>> +
>> +DEPS       =
>> +
>> +SERVICES   =
>> +
>> +###############################################################################
>> +# Top-level Rules
>> +###############################################################################
>> +
>> +install : $(TARGET)
>> +
>> +check :
>> +
>> +download :
>> +
>> +b2 :
>> +
>> +dist:
>> + @$(PAK)
>> +
>> +###############################################################################
>> +# Installation Details
>> +###############################################################################
>> +
>> +$(TARGET) :
>> + @$(PREBUILD)
>> + @rm -rf $(DIR_APP)
>> +
>> + # install RPZ scripts
>> + install -v -m 755 \
>> +  $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin
>> +
>> + # Install settings folder and two empty files
>> + mkdir -pv /var/ipfire/rpz
>> + touch /var/ipfire/rpz/allowlist
>> + touch /var/ipfire/rpz/blocklist
>> +
>> + # Add conf file to /etc directory
>> + cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d
>> +
>> + # create zonefiles directory for the RPZ files and add two empty RPZ
>> + #  files to avoid a unbound config error
>> + mkdir -pv /etc/unbound/zonefiles
>> + chown -v nobody:nobody /etc/unbound/zonefiles
>> + touch /etc/unbound/zonefiles/allow.rpz
>> + touch /etc/unbound/zonefiles/block.rpz
>> +
>> + # Install backup definition
>> + cp -vf $(DIR_CONF)/backup/includes/rpz /var/ipfire/backup/addons/includes/rpz
>> +
>> + @rm -rf $(DIR_APP)
>> + @$(POSTBUILD)
>> diff --git a/make.sh b/make.sh
>> index 9bbbeb0f1..886d3760a 100755
>> --- a/make.sh
>> +++ b/make.sh
>> @@ -1721,6 +1721,8 @@ buildipfire() {
>>  lfsmake2 btrfs-progs
>>  lfsmake2 inotify-tools
>>  lfsmake2 grub-btrfs
>> +  lfsmake2 rpz
>> +
>> 
>>  # Kernelbuild ... current we have no platform that need
>>  # multi kernel builds so KCFG is empty
>> -- 
>> 2.30.2
>> 
>

Michael Tremer July 28, 2024, 11:22 a.m. UTC | #3

Hello Jon,

I don’t think it is a good idea to send over the same patch set multiple times.

I think first of all, we need to start at the beginning which is to figure out what this actually is, and why we would want this in IPFire.

I have seen that there have been conversations on community.ipfire.org <http://community.ipfire.org/> many months ago, but I did not take many things away from it. It was from my point of view mostly a prove of concept discussion. Correct me if I have missed anything here.

We have talked about RPZ many times on the monthly call since the URL filter feature is falling more and more out of fashion. I think there is also many posts about this on the forum.

Apart from generally being in favour of trying this path, we decided to postpone any further works into this because we currently have a huge backlog of things to deal with  that are moving at snail speed. This is rooted in so many people being simply tied in with other things.

As you can see, the patch set that you posted did not trigger a response - a common thing on this list recently and not because of this particular patch set.

Therefore I would like to avoid opening another can of worms before we have delivered the other things that have already lots of time spent on them and code written. Otherwise we are only getting a longer pipeline and this all is going to be more frustrating for everyone involved. And this is not about the feature per se; this is about getting things done.

So what I suggest doing is the following:

You have a Git repository, push the code there and commit any fixes to it over time so that nothing is getting lost.

Explain to me from the beginning, please, how it came to this patch set. What is the motivation? What problems does it solve? What problems remain? Basically: Why would anyone want this in IPFire? Where do you see application for this? Why is this realised as an add-on and not part of the core system?

I am particularly interested in what lists are publicly available. You know from URL Filter, that lists have shut down or just been abandoned. There is now very little point running it because the lists are just not as good as they used to be. Not horrible, yet, but over time this will not be getting any better. I am not even aware of any viable paid lists. Is this the same situation for RPZ? A quick Google search did not give me anything that would be free to use over DNS. The situation is somewhat similar with the IPS and IP blocklist feature.

I would be interested in hearing your thoughts on this.

Best,
-Michael

> On 27 Jul 2024, at 21:31, jon <jon.murphy@ipfire.org> wrote:
> 
> And another question.
> 
> I found an error in the `rpz-config` script file.  Simple easy one line (actually one word) fix.  
> 
> Do I resubmit the entire patch?
> -or-
> Do I wait until approved and then submit the simple patch for the one line change.
> 
> 
> Jon
> 
> 
>> On Jul 20, 2024, at 1:42 PM, jon <jon.murphy@ipfire.org> wrote:
>> 
>> I need help with…
>> 
>> 1) The custom allow and block lists are currently located at `/var/ipfire/rpz`.  Is this correct?
>> 
>> 2) The three bash scripts are currently located at `/usr/sbin`.  Is this correct?
>> 
>> 3) Should this commit be separated into multiple posts?  If yes, how do I decide how to split it?
>> 
>> 
>> Jon 
>> 
>> 
>>> On Jul 20, 2024, at 12:46 PM, Jon Murphy <jon.murphy@ipfire.org> wrote:
>>> 
>>> What is it?
>>> Response Policy Zone (RPZ) is a mechanism to define local policies in a standardised
>>> way and load those policies from external sources.
>>> Bottom line: RPZ allows admins to easily block access to websites via DNS lookup.
>>> 
>>> RPZ can block websites via categories.  Examples include: fake websites, annoying
>>> pop-up ads, newly registered domains, DoH bypass sites, bad "host" services,
>>> maliscious top level domains (e.g., *.zip, *.mov), piracy, gambling, pornography,
>>> and more.  RPZ lists come from various RPZ providers and their available
>>> catagories.
>>> 
>>> This RPZ add-on enables the RPZ functionality by adding a couple lines in a
>>> configuration file.  This add-on simply adds a configuration file and adds
>>> three scripts (config, metrics and sleep) to make RPZ easier for the admin to use.
>>> 
>>> RPZ was release in 2010 and has been part of the IPFire build since ~2015.
>>> 
>>> Why is it needed?
>>> Some IPFire admin's utilize pihole to block unwanted websites via DNS lookup.
>>> Moving the pihole base functionality (without pretty graphs) to IPFire removes
>>> one device from the admin's local network.
>>> And hopefully this reduces the pihole questions from the Community.
>>> 
>>> A list of RPZ providers can be recommended by IPFire and coded into a set list.
>>> Or, if prefered, the local admin can choose their own RPZ providers.
>>> 
>>> This is a:
>>> * simple replacement for pihole base functionality
>>> * RPZ can be a nice replacement for the URL Filter
>>> 
>>> IPFire Wiki
>>> In process at: https://www.ipfire.org/docs/addons/rpz
>>> 
>>> more info:
>>> * https://en.wikipedia.org/wiki/Response_policy_zone
>>> * https://unbound.docs.nlnetlabs.nl/en/latest/topics/filtering/rpz.html
>>> 
>>> I need help with...
>>> 1) The custom allow and block lists are currently located at `/var/ipfire/rpz`.  Is this correct?
>>> 2) The three bash scripts are currently located at `/usr/sbin`.  Is this correct?
>>> 
>>> Signed-off-by: Jon Murphy <jon.murphy@ipfire.org>
>>> ---
>>> config/backup/includes/rpz    |   5 +
>>> config/rootfiles/packages/rpz |  11 ++
>>> config/rpz/00-rpz.conf        |  18 ++++
>>> config/rpz/rpz-config         | 194 ++++++++++++++++++++++++++++++++++
>>> config/rpz/rpz-metrics        | 143 +++++++++++++++++++++++++
>>> config/rpz/rpz-sleep          |  58 ++++++++++
>>> lfs/rpz                       |  88 +++++++++++++++
>>> make.sh                       |   2 +
>>> 8 files changed, 519 insertions(+)
>>> create mode 100644 config/backup/includes/rpz
>>> create mode 100644 config/rootfiles/packages/rpz
>>> create mode 100644 config/rpz/00-rpz.conf
>>> create mode 100644 config/rpz/rpz-config
>>> create mode 100644 config/rpz/rpz-metrics
>>> create mode 100644 config/rpz/rpz-sleep
>>> create mode 100644 lfs/rpz
>>> 
>>> diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz
>>> new file mode 100644
>>> index 000000000..4d59bb40c
>>> --- /dev/null
>>> +++ b/config/backup/includes/rpz
>>> @@ -0,0 +1,5 @@
>>> +/var/ipfire/rpz/allowlist
>>> +/var/ipfire/rpz/blocklist
>>> +/etc/unbound/zonefiles/allow.rpz
>>> +/etc/unbound/zonefiles/block.rpz
>>> +/etc/unbound/local.d/*rpz.conf
>>> diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/rpz
>>> new file mode 100644
>>> index 000000000..2ffa715dd
>>> --- /dev/null
>>> +++ b/config/rootfiles/packages/rpz
>>> @@ -0,0 +1,11 @@
>>> +etc/unbound/local.d/00-rpz.conf
>>> +etc/unbound/zonefiles
>>> +etc/unbound/zonefiles/allow.rpz
>>> +etc/unbound/zonefiles/block.rpz
>>> +usr/sbin/rpz-config
>>> +usr/sbin/rpz-metrics
>>> +usr/sbin/rpz-sleep
>>> +var/ipfire/backup/addons/includes/rpz
>>> +var/ipfire/rpz
>>> +var/ipfire/rpz/allowlist
>>> +var/ipfire/rpz/blocklist
>>> diff --git a/config/rpz/00-rpz.conf b/config/rpz/00-rpz.conf
>>> new file mode 100644
>>> index 000000000..72c1d12e5
>>> --- /dev/null
>>> +++ b/config/rpz/00-rpz.conf
>>> @@ -0,0 +1,18 @@
>>> +server:
>>> + module-config: "respip validator iterator"
>>> +
>>> +rpz:
>>> +    name: allow.rpz
>>> +    zonefile: /etc/unbound/zonefiles/allow.rpz
>>> +    rpz-action-override: passthru
>>> +    rpz-log: yes
>>> +    rpz-log-name: allow
>>> +    rpz-signal-nxdomain-ra: yes
>>> +
>>> +rpz:
>>> +    name: block.rpz
>>> +    zonefile: /etc/unbound/zonefiles/block.rpz
>>> +    rpz-action-override: nxdomain
>>> +    rpz-log: yes
>>> +    rpz-log-name: block
>>> +    rpz-signal-nxdomain-ra: yes
>>> diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config
>>> new file mode 100644
>>> index 000000000..98dc0a4ca
>>> --- /dev/null
>>> +++ b/config/rpz/rpz-config
>>> @@ -0,0 +1,194 @@
>>> +#!/bin/bash
>>> +###############################################################################
>>> +#                                                                             #
>>> +#  IPFire.org - A linux based firewall                                        #
>>> +#  Copyright (C) 2024  IPFire Team  <info@ipfire.org>                         #
>>> +#                                                                             #
>>> +#  This program is free software: you can redistribute it and/or modify       #
>>> +#  it under the terms of the GNU General Public License as published by       #
>>> +#  the Free Software Foundation, either version 3 of the License, or          #
>>> +#  (at your option) any later version.                                        #
>>> +#                                                                             #
>>> +#  This program is distributed in the hope that it will be useful,            #
>>> +#  but WITHOUT ANY WARRANTY; without even the implied warranty of             #
>>> +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the              #
>>> +#  GNU General Public License for more details.                               #
>>> +#                                                                             #
>>> +#  You should have received a copy of the GNU General Public License          #
>>> +#  along with this program.  If not, see <http://www.gnu.org/licenses/>.      #
>>> +#                                                                             #
>>> +###############################################################################
>>> +
>>> +# v22 - 2024-07-12
>>> +
>>> +###############     Functions     ###############
>>> +
>>> +msg_log () {
>>> + /usr/bin/logger --tag "${tagName}" "$*"
>>> + if tty --silent ; then
>>> + echo "${tagName}:" "$*"
>>> + fi
>>> +}
>>> +
>>> +check_name () {
>>> + local testName="${1}"
>>> + # check for a valid name
>>> + regex='^[a-zA-Z0-9_]+$'
>>> + if [[ ! "${testName}" =~ $regex ]] ; then
>>> + msg_log "error: rpz: the NAME is not valid: \"${testName}\". exit."
>>> + exit 1
>>> + fi
>>> +}
>>> +
>>> +check_unbound_conf () {
>>> + # check the above config files
>>> + msg_log "info: rpz: check for errors with \"unbound-checkconf\""
>>> + /usr/sbin/unbound-checkconf
>>> + exit_code=$?
>>> + if [[ "${exit_code}" -ne 0 ]] ; then
>>> + msg_log "error: rpz: unbound-checkconf. exit."
>>> + exit "${exit_code}"
>>> + fi
>>> +}
>>> +
>>> +make_rpz_file () {
>>> + local theType="${1}" # allow or block
>>> +
>>> + theList="/var/ipfire/rpz/${theType}list" # input user list of domains
>>> + theZoneFile="/etc/unbound/zonefiles/${theType}.rpz" # output file for RPZ
>>> +
>>> + theAction='.'
>>> + if [[ "${theType}" =~ "block" ]] ; then
>>> + theAction='rpz-passthru.'
>>> + fi
>>> +
>>> + # does a list exist?
>>> + if [[ -s "${theList}" ]] ; then
>>> + # drop any extra "blanks" and add "CNAME <RPZ action>." to each line
>>> + actionList=$( /usr/bin/awk '{$1=$1};1' "${theList}" |
>>> + /bin/sed "/^[^;].*[[:alnum:]]/ s|$| CNAME ${theAction}|" )
>>> +
>>> + msg_log "info: rpz: create zonefile for ${theList}"
>>> +
>>> + /bin/cat <<-EOF > "${theZoneFile}"
>>> + ; Name: ${theType} list
>>> + ; Last modified: $(date "+%Y-%m-%d at %H.%M.%S %Z")
>>> + ;
>>> + ; domains with actions list
>>> + ;
>>> + ${actionList}
>>> + EOF
>>> +
>>> + # reload the zone that was just updated
>>> + zoneBase=$( basename "${theZoneFile}" )
>>> + /usr/sbin/unbound-control auth_zone_reload -q "${zoneBase}"
>>> + fi
>>> +}
>>> +
>>> +###############       Main        ###############
>>> +
>>> +tagName="unbound"
>>> +
>>> +theAction="${1}" # input action
>>> +theName="${2}" # input RPZ name
>>> +theURL="${3}" # input RPZ URL
>>> +
>>> +check_name "${theName}" # is this a valid name?
>>> +
>>> +rpzConfig="/etc/unbound/local.d/${theName}.rpz.conf" # output zone conf file
>>> +rpzFile="/etc/unbound/zonefiles/${theName}.rpz" # output for RPZ file
>>> +
>>> +case "${theAction}" in
>>> +
>>> + # add new rpz list
>>> + add )
>>> + # does this config already exist?  If yes, then exit
>>> + if [[ -f "${rpzConfig}" ]] ; then
>>> + msg_log "info: rpz: ${rpzConfig} already exists. exit"
>>> + exit 1
>>> + fi
>>> +
>>> + # is this a valid URL?
>>> + regex='^https://[-[:alnum:]\+&@#/%?=~_|!:,.;]*[-[:alnum:]\+&@#/%=~_|]'
>>> + if ! [[ "${theURL}" =~ $regex ]] ; then
>>> + msg_log "error: rpz: the URL is not valid: \"${theURL}\". exit."
>>> + exit 1
>>> + fi
>>> +
>>> + # create the zone config file
>>> + msg_log "info: rpz: add config file \"${theName}.rpz.conf\""
>>> + cat <<-EOF > "${rpzConfig}"
>>> + rpz:
>>> + name: ${theName}.rpz
>>> + zonefile: /etc/unbound/zonefiles/${theName}.rpz
>>> + url:  ${theURL}
>>> + rpz-action-override: nxdomain
>>> + rpz-log: yes
>>> + rpz-log-name: ${theName}
>>> + rpz-signal-nxdomain-ra: yes
>>> + EOF
>>> +
>>> + # set-up zone file
>>> + /usr/bin/touch "${rpzFile}"
>>> + # unbound requires these settings for rpz files
>>> + /bin/chown --verbose nobody:nobody "${rpzFile}"
>>> + /bin/chmod --verbose 644 "${rpzFile}"
>>> + ;;
>>> +
>>> + # trash config file & rpz file
>>> + remove )
>>> + if ! [[ -f "${rpzConfig}" ]] ; then
>>> + msg_log "info: rpz: ${rpzConfig} does not exist. exit"
>>> + exit 1
>>> + fi
>>> +
>>> + msg_log "info: rpz: remove config file & rpz file \"${theName}\""
>>> + /bin/rm --verbose "${rpzConfig}"
>>> + /bin/rm --verbose "${rpzFile}"
>>> +
>>> + check_unbound_conf
>>> + ;;
>>> +
>>> + # make a new allow or block rpz file
>>> + make )
>>> + case "${theName}" in
>>> + allow )
>>> + make_rpz_file allow
>>> + ;;
>>> +
>>> + block )
>>> + make_rpz_file block
>>> + ;;
>>> +
>>> + allowblock )
>>> + make_rpz_file allow
>>> + make_rpz_file block
>>> + ;;
>>> +
>>> + * )
>>> + msg_log "error: rpz: the NAME is not valid: \"${theName}\". exit."
>>> + exit 1
>>> + ;;
>>> + esac
>>> +
>>> + check_unbound_conf
>>> + ;;
>>> +
>>> + *)
>>> + msg_log "error: rpz: missing or incorrect parameter"
>>> + /usr/bin/printf "Usage:  rpzConfig.sh <ACTION> <NAME> <URL>\n"
>>> + exit 1
>>> + ;;
>>> +
>>> +esac
>>> +
>>> +# reload due to the changes
>>> +msg_log  "rpz: running \"unbound-control reload\""
>>> +/usr/sbin/unbound-control reload
>>> +exit_code=$?
>>> +if [[ "${exit_code}" -ne 0 ]] ; then
>>> + msg_log "error: rpz: unbound-control \"${theName}\". exit."
>>> + exit "${exit_code}"
>>> +fi
>>> +
>>> +exit
>>> diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics
>>> new file mode 100644
>>> index 000000000..0f97c7911
>>> --- /dev/null
>>> +++ b/config/rpz/rpz-metrics
>>> @@ -0,0 +1,143 @@
>>> +#!/bin/bash
>>> +###############################################################################
>>> +#                                                                             #
>>> +#  IPFire.org - A linux based firewall                                        #
>>> +#  Copyright (C) 2024  IPFire Team  <info@ipfire.org>                         #
>>> +#                                                                             #
>>> +#  This program is free software: you can redistribute it and/or modify       #
>>> +#  it under the terms of the GNU General Public License as published by       #
>>> +#  the Free Software Foundation, either version 3 of the License, or          #
>>> +#  (at your option) any later version.                                        #
>>> +#                                                                             #
>>> +#  This program is distributed in the hope that it will be useful,            #
>>> +#  but WITHOUT ANY WARRANTY; without even the implied warranty of             #
>>> +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the              #
>>> +#  GNU General Public License for more details.                               #
>>> +#                                                                             #
>>> +#  You should have received a copy of the GNU General Public License          #
>>> +#  along with this program.  If not, see <http://www.gnu.org/licenses/>.      #
>>> +#                                                                             #
>>> +###############################################################################
>>> +
>>> +# v18 on 2024-07-05
>>> +
>>> +###############       Main        ###############
>>> +
>>> +weeks="${1:-2}" # default to two message logs
>>> +sortBy="${2:-name}"   # by name or by hits
>>> +
>>> +# get the list of message logs for N weeks
>>> +messageLogs=$( find /var/log/messages* -type f |
>>> + /usr/bin/sort --version-sort |
>>> + head -"${weeks}" )
>>> +
>>> +# get the list of RPZ names & counts from the message log(s)
>>> +rpzNameCount=$( for logf in ${messageLogs} ; do
>>> + /usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" |
>>> + /usr/bin/awk '$10 ~ /\[\w*]/ { print $10 }' ;
>>> + done | /usr/bin/sort | /usr/bin/uniq --count )
>>> +
>>> +# flip results and remove brackets `[` and `]`
>>> +rpzNameCount=$( /bin/echo "${rpzNameCount}" |
>>> + /usr/bin/awk '{ print $2, $1 }' |
>>> + /bin/sed --regexp-extended 's|^\[(.*)\]|\1|' )
>>> +
>>> +# grab only names
>>> +rpzNames=$( /bin/echo "${rpzNameCount}" | /usr/bin/awk '{ print $1 }' )
>>> +
>>> +# get list of RPZ files
>>> +rpzFileList=$( /bin/find /etc/unbound/zonefiles -type f -iname "*.rpz" )
>>> +
>>> +# get basename of those files
>>> +rpzBaseNames=$( /bin/echo "${rpzFileList}" |
>>> + /bin/sed 's|/etc/unbound/zonefiles/||g ; s|\.rpz||g ;' )
>>> +
>>> +# add to rpzNames
>>> +rpzNames="${rpzNames}"$'\n'"${rpzBaseNames}"
>>> +
>>> +# drop duplicate names
>>> +rpzNames=$( echo "${rpzNames}" | /usr/bin/sort --unique  )
>>> +
>>> +# get line count for each RPZ
>>> +lineCount=$( /bin/echo "${rpzFileList}" | /usr/bin/xargs wc -l )
>>> +
>>> +# get comment line count and blank line count for each RPZ
>>> +commentCount=$( /bin/echo "${rpzFileList}" |
>>> + /usr/bin/xargs /bin/grep --count -e "^$" -e "^;" )
>>> +
>>> +# get modified date each RPZ
>>> +modDateList=$( /bin/echo "${rpzFileList}" | xargs stat -c '%.10y  %n' )
>>> +
>>> +ucListAuthZones=$( /usr/sbin/unbound-control list_auth_zones )
>>> +
>>> +# get width of RPZ names
>>> +pWidth=$( /bin/echo "${rpzNames}" | /usr/bin/awk '{ print $1"   " }' | wc -L )
>>> +pFormat="%-${pWidth}s %-8s %-8s %-8s %10s %12s\n"
>>> +
>>> +# print title line
>>> +printf "${pFormat}" "name" "hits" "active" "lines" "hits/line%" "last update"
>>> +printf -- "--------------"
>>> +
>>> +theResults=""
>>> +totalLines=0
>>> +totalHits=0
>>> +while read -r theName
>>> +do
>>> + printf -- "-" # pretend progress bar
>>> + # get hit count
>>> + theHits="0"
>>> + if output=$( /bin/grep "^${theName}\s" <<< "${rpzNameCount}" ) ; then
>>> + theHits=$( /bin/echo "${output}" |
>>> + /usr/bin/awk '{ print $2 }' )
>>> + totalHits=$(( totalHits + theHits ))
>>> + fi
>>> +
>>> + # is this RPZ list active?
>>> + theActive="disabled"
>>> + if /bin/grep --quiet "^${theName}\.rpz" <<< "${ucListAuthZones}"
>>> + then
>>> + theActive="enabled"
>>> + fi
>>> +
>>> + # get line count then subtract comment count and blank line count
>>> + #  from total line count
>>> + theLines="n/a"
>>> + hitsPerLine="0"
>>> + if output=$( /bin/grep --fixed-strings "/${theName}.rpz" <<< "${lineCount}" ) ; then
>>> + theLines=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' )
>>> + totalLines=$(( totalLines + theLines ))
>>> +
>>> + #hitsPerLine=$( echo "scale=0 ; $theHits / $theLines" | bc )
>>> + hitsPerLine=$(( 100 * theHits / theLines ))
>>> + fi
>>> +
>>> + # get modification date
>>> + theModDate="n/a"
>>> + if output=$( /bin/grep --fixed-strings "/${theName}.rpz" <<< "${modDateList}" ) ; then
>>> + theModDate=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' )
>>> + fi
>>> +
>>> + # add to results list
>>> + theResults+="${theName} ${theHits} ${theActive} ${theLines} ${hitsPerLine} ${theModDate}"$'\n'
>>> +done <<< "${rpzNames}"
>>> +
>>> +case "${sortBy}" in
>>> + names|name) sortArg=(-k3,3r -k1,1) ;; # sort by "active" then by "name"
>>> +
>>> + hits|hit) sortArg=(-k3,3r -k2,2nr -k1,1) ;; # sort by "active" then by "hits" then by "name"
>>> +
>>> + lines|line) sortArg=(-k3,3r -k4,4nr -k1,1) ;; # sort by "active" then by "lines" then by "name"
>>> +esac
>>> +
>>> +printf -- "--------------\n"
>>> +# remove blank lines, sort, print as columns
>>> +/bin/echo "${theResults}" |
>>> + /usr/bin/awk '!/^[[:space:]]*$/' |
>>> + /usr/bin/sort "${sortArg[@]}"    |
>>> + /usr/bin/awk --assign=width="${pWidth}" \
>>> + '{ printf "%-*s %-8s %-8s %-8s %10s %12s\n", width, $1, $2, $3, $4, $5, $6 }'
>>> +
>>> +printf "${pFormat}" "" "=======" "" "========" "" ""
>>> +printf "${pFormat}" "Totals -->" "${totalHits}" "" "${totalLines}" "" ""
>>> +
>>> +exit
>>> diff --git a/config/rpz/rpz-sleep b/config/rpz/rpz-sleep
>>> new file mode 100644
>>> index 000000000..eeef1174a
>>> --- /dev/null
>>> +++ b/config/rpz/rpz-sleep
>>> @@ -0,0 +1,58 @@
>>> +#!/bin/bash
>>> +###############################################################################
>>> +#                                                                             #
>>> +#  IPFire.org - A linux based firewall                                        #
>>> +#  Copyright (C) 2024  IPFire Team  <info@ipfire.org>                         #
>>> +#                                                                             #
>>> +#  This program is free software: you can redistribute it and/or modify       #
>>> +#  it under the terms of the GNU General Public License as published by       #
>>> +#  the Free Software Foundation, either version 3 of the License, or          #
>>> +#  (at your option) any later version.                                        #
>>> +#                                                                             #
>>> +#  This program is distributed in the hope that it will be useful,            #
>>> +#  but WITHOUT ANY WARRANTY; without even the implied warranty of             #
>>> +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the              #
>>> +#  GNU General Public License for more details.                               #
>>> +#                                                                             #
>>> +#  You should have received a copy of the GNU General Public License          #
>>> +#  along with this program.  If not, see <http://www.gnu.org/licenses/>.      #
>>> +#                                                                             #
>>> +###############################################################################
>>> +
>>> +# v04 on 2024-07-05
>>> +
>>> +###############     Functions     ###############
>>> +
>>> +# send message to message log
>>> +msg_log () {
>>> + /usr/bin/logger --tag "${tagName}" "$*"
>>> + if /usr/bin/tty --silent ; then
>>> + echo "${tagName}:" "$*"
>>> + fi
>>> +}
>>> +
>>> +###############       Main        ###############
>>> +
>>> +tagName="unbound"
>>> +
>>> +sleepTime="${1:-5m}" # default to sleep for 5m (5 minutes)
>>> +
>>> +zoneList=$( /usr/sbin/unbound-control list_auth_zones | /usr/bin/awk '{print $1}' )
>>> +
>>> +for zone in ${zoneList} ; do
>>> + /usr/bin/printf "disable ${zone}\t"
>>> + /usr/sbin/unbound-control rpz_disable "${zone}"
>>> +done
>>> +
>>> +msg_log "info: rpz: disabled all zones for ${sleepTime}"
>>> +
>>> +/bin/sleep "${sleepTime}"
>>> +
>>> +for zone in ${zoneList} ; do
>>> + /usr/bin/printf "enable ${zone}\t"
>>> + /usr/sbin/unbound-control rpz_enable "${zone}"
>>> +done
>>> +
>>> +msg_log "info: rpz: enabled all zones"
>>> +
>>> +exit
>>> diff --git a/lfs/rpz b/lfs/rpz
>>> new file mode 100644
>>> index 000000000..319c10b7f
>>> --- /dev/null
>>> +++ b/lfs/rpz
>>> @@ -0,0 +1,88 @@
>>> +###############################################################################
>>> +#                                                                             #
>>> +# IPFire.org - A linux based firewall                                         #
>>> +# Copyright (C) 2024  IPFire Team  <info@ipfire.org>                          #
>>> +#                                                                             #
>>> +# This program is free software: you can redistribute it and/or modify        #
>>> +# it under the terms of the GNU General Public License as published by        #
>>> +# the Free Software Foundation, either version 3 of the License, or           #
>>> +# (at your option) any later version.                                         #
>>> +#                                                                             #
>>> +# This program is distributed in the hope that it will be useful,             #
>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of              #
>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the               #
>>> +# GNU General Public License for more details.                                #
>>> +#                                                                             #
>>> +# You should have received a copy of the GNU General Public License           #
>>> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.       #
>>> +#                                                                             #
>>> +###############################################################################
>>> +
>>> +###############################################################################
>>> +# Definitions
>>> +###############################################################################
>>> +
>>> +include Config
>>> +
>>> +SUMMARY    = response policy zone - RPZ reputation system for unbound DNS
>>> +
>>> +VER        = 1.0.0
>>> +
>>> +THISAPP    = rpz-$(VER)
>>> +DIR_APP    = $(DIR_SRC)/$(THISAPP)
>>> +TARGET     = $(DIR_INFO)/$(THISAPP)
>>> +
>>> +PROG       = rpz
>>> +PAK_VER    = 1
>>> +
>>> +DEPS       =
>>> +
>>> +SERVICES   =
>>> +
>>> +###############################################################################
>>> +# Top-level Rules
>>> +###############################################################################
>>> +
>>> +install : $(TARGET)
>>> +
>>> +check :
>>> +
>>> +download :
>>> +
>>> +b2 :
>>> +
>>> +dist:
>>> + @$(PAK)
>>> +
>>> +###############################################################################
>>> +# Installation Details
>>> +###############################################################################
>>> +
>>> +$(TARGET) :
>>> + @$(PREBUILD)
>>> + @rm -rf $(DIR_APP)
>>> +
>>> + # install RPZ scripts
>>> + install -v -m 755 \
>>> +  $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin
>>> +
>>> + # Install settings folder and two empty files
>>> + mkdir -pv /var/ipfire/rpz
>>> + touch /var/ipfire/rpz/allowlist
>>> + touch /var/ipfire/rpz/blocklist
>>> +
>>> + # Add conf file to /etc directory
>>> + cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d
>>> +
>>> + # create zonefiles directory for the RPZ files and add two empty RPZ
>>> + #  files to avoid a unbound config error
>>> + mkdir -pv /etc/unbound/zonefiles
>>> + chown -v nobody:nobody /etc/unbound/zonefiles
>>> + touch /etc/unbound/zonefiles/allow.rpz
>>> + touch /etc/unbound/zonefiles/block.rpz
>>> +
>>> + # Install backup definition
>>> + cp -vf $(DIR_CONF)/backup/includes/rpz /var/ipfire/backup/addons/includes/rpz
>>> +
>>> + @rm -rf $(DIR_APP)
>>> + @$(POSTBUILD)
>>> diff --git a/make.sh b/make.sh
>>> index 9bbbeb0f1..886d3760a 100755
>>> --- a/make.sh
>>> +++ b/make.sh
>>> @@ -1721,6 +1721,8 @@ buildipfire() {
>>> lfsmake2 btrfs-progs
>>> lfsmake2 inotify-tools
>>> lfsmake2 grub-btrfs
>>> +  lfsmake2 rpz
>>> +
>>> 
>>> # Kernelbuild ... current we have no platform that need
>>> # multi kernel builds so KCFG is empty
>>> -- 
>>> 2.30.2
>>> 
>> 
>

Jon Murphy July 29, 2024, 2:28 a.m. UTC | #4

See comments below...

> On Jul 28, 2024, at 6:22 AM, Michael Tremer <michael.tremer@ipfire.org> wrote:
> 
> Hello Jon,
> 
> I don’t think it is a good idea to send over the same patch set multiple times.
> 
> I think first of all, we need to start at the beginning which is to figure out what this actually is, and why we would want this in IPFire.
> 
> I have seen that there have been conversations on community.ipfire.org <http://community.ipfire.org/> many months ago, but I did not take many things away from it. It was from my point of view mostly a prove of concept discussion. Correct me if I have missed anything here.
> 
> We have talked about RPZ many times on the monthly call since the URL filter feature is falling more and more out of fashion. I think there is also many posts about this on the forum.

Yes, I have seen the posts about the url filter feature falling out of fashion.  I did not see notes about RPZ from the monthly calls.  I will go back and search telco call notes.


> 
> Apart from generally being in favour of trying this path, we decided to postpone any further works into this because we currently have a huge backlog of things to deal with  that are moving at snail speed. This is rooted in so many people being simply tied in with other things.
> 
> As you can see, the patch set that you posted did not trigger a response - a common thing on this list recently and not because of this particular patch set.
> 
> Therefore I would like to avoid opening another can of worms before we have delivered the other things that have already lots of time spent on them and code written. Otherwise we are only getting a longer pipeline and this all is going to be more frustrating for everyone involved. And this is not about the feature per se; this is about getting things done.

This is why I wrote the RPZ patch instead of asking for the Developers to write it.  I realize there is a very long pipeline.


> 
> So what I suggest doing is the following:
> 
> You have a Git repository, push the code there and commit any fixes to it over time so that nothing is getting lost.

This was done a few days ago.  But I am not sure if I did it correctly.  If not, please let me know.


> 
> Explain to me from the beginning, please, how it came to this patch set. What is the motivation?

In the beginning (Fall of 2023) the main motivation was blocking DoH.  I had noticed apps using their own method of DNS lookup.  Some were friendly DoH lookups (Apple iCloud?) and some not (mozilla.cloudflare-dns.com but not from Firefox).  Examples I saw are:
mozilla.cloudflare-dns.com <http://mozilla.cloudflare-dns.com/>
www.switch.ch <http://www.switch.ch/>
use-application-dns.net <http://use-application-dns.net/> (a canary for Firefox?)
And a few others...

This worked better than expected and was easier to configure than expected.  All of the main code was part of unbound and I just need to activate it with a config file.  Updates to the RPZ files are already part of the unbound RPZ code.

As I experimented with RPZ I quickly realized it could easily be used to block other websites.  I have been using pihole for a number of years to limit the number of ads and trackers.  And RPZ helped with those areas also.


> What problems does it solve? What problems remain? Basically: Why would anyone want this in IPFire?

It solved the issue of blocking unwanted DoH from apps.  And it solved the issue of unwanted ads and tracking.  For me those are huge benefits and the biggest reason for using RPZ.

To test other possibilities, I increased the "type" of lists to include other categories other than ads & trackers.  Like blocking bad TLDs, gambling, piracy, bad host sites, etc.  This is a nice bonus of using RPZ for me.

> Where do you see application for this?

As a replacement for the URL Filter.  And as a base functional replacement for pihole (but no pretty graphics).


> Why is this realised as an add-on and not part of the core system?

In my opinion it should be part of the core.  But I wanted to "test the waters" first to determine interest.  If only four people use it, then it isn’t worth adding to core and maybe not as an add-on.

Either way I will support it because it helps me!

Here are my metrics for the past eight days (searching `messages` and `messages.1.gz`):

[root@ipfire ~] # rpz-metrics
name               hits     
----------------------------
allow              5560    This is a custom list  
AmazonTrkrHZ       5726     
AppleTrkrHZ        2956     
block              128     This is a custom list  
DOHblockHZ         25077    
dohJPG             3448     
HosterHZ           19       
MxProPlusHZ        17287    
NotSafeSearchHZ    0        
ProxyBypassHZ      11       
tldHZ              15       
WinTrkrHZ          507      
                   =======  
Totals -->         60734    

For the DoH blocking, the above "hits" including blocking Apple DoH.  I’ve been testing this for the past few weeks.  

> 
> I am particularly interested in what lists are publicly available. You know from URL Filter, that lists have shut down or just been abandoned. There is now very little point running it because the lists are just not as good as they used to be. Not horrible, yet, but over time this will not be getting any better. I am not even aware of any viable paid lists. Is this the same situation for RPZ?  A quick Google search did not give me anything that would be free to use over DNS. The situation is somewhat similar with the IPS and IP blocklist feature.

These are free/open lists:

Hagezi - DNS Blocklists 
• at https://github.com/hagezi/dns-blocklists?tab=readme-ov-file#zap-dns-blocklists---for-a-better-internet
• This list was recommended within the IPFire Community and this is the list I strongly recommend.  Gerd (Hagezi) is very responsive to support requests. The Hagezi RPZ list is well supported and the RPZ lists are updated twice per day (if changes are needed).
• Example: I came across a domain name that was blocked `f005.backblazeb2.com` and it was fixed (removed from the block list) within a few days.

ThreatFox - DNS Response Policy Zone (RPZ) 
• at https://threatfox.abuse.ch/export/#rpz

URLHaus - DNS Response Policy Zone (RPZ)
• at https://urlhaus.abuse.ch/api/#rpz

jpgpi250 - DNS block list for DoH
• at https://github.com/jpgpi250/piholemanual/blob/master/DOH.rpz
• this is the original DoH block list that I started with.  Peter (jpgpi250) has a nice PDF document explaining RPZ and was very helpful getting me started with RPZ.  See https://jpgpi250.github.io/piholemanual/doc/Unbound%20response%20policy%20zones.pdf

I have come across other RPZ lists but these are the best supported lists.

I did not look into paid lists.

> 
> 
> I would be interested in hearing your thoughts on this.

Does the above help?

> 
> Best,
> -Michael
> 
>> On 27 Jul 2024, at 21:31, jon <jon.murphy@ipfire.org> wrote:
>> 
>> And another question.
>> 
>> I found an error in the `rpz-config` script file.  Simple easy one line (actually one word) fix.  
>> 
>> Do I resubmit the entire patch?
>> -or-
>> Do I wait until approved and then submit the simple patch for the one line change.
>> 
>> 
>> Jon
>> 
>> 
>>> On Jul 20, 2024, at 1:42 PM, jon <jon.murphy@ipfire.org> wrote:
>>> 
>>> I need help with…
>>> 
>>> 1) The custom allow and block lists are currently located at `/var/ipfire/rpz`.  Is this correct?
>>> 
>>> 2) The three bash scripts are currently located at `/usr/sbin`.  Is this correct?
>>> 
>>> 3) Should this commit be separated into multiple posts?  If yes, how do I decide how to split it?
>>> 
>>> 
>>> Jon 
>>> 
>>> 
>>>> On Jul 20, 2024, at 12:46 PM, Jon Murphy <jon.murphy@ipfire.org> wrote:
>>>> 
>>>> What is it?
>>>> Response Policy Zone (RPZ) is a mechanism to define local policies in a standardised
>>>> way and load those policies from external sources.
>>>> Bottom line: RPZ allows admins to easily block access to websites via DNS lookup.
>>>> 
>>>> RPZ can block websites via categories.  Examples include: fake websites, annoying
>>>> pop-up ads, newly registered domains, DoH bypass sites, bad "host" services,
>>>> maliscious top level domains (e.g., *.zip, *.mov), piracy, gambling, pornography,
>>>> and more.  RPZ lists come from various RPZ providers and their available
>>>> catagories.
>>>> 
>>>> This RPZ add-on enables the RPZ functionality by adding a couple lines in a
>>>> configuration file.  This add-on simply adds a configuration file and adds
>>>> three scripts (config, metrics and sleep) to make RPZ easier for the admin to use.
>>>> 
>>>> RPZ was release in 2010 and has been part of the IPFire build since ~2015.
>>>> 
>>>> Why is it needed?
>>>> Some IPFire admin's utilize pihole to block unwanted websites via DNS lookup.
>>>> Moving the pihole base functionality (without pretty graphs) to IPFire removes
>>>> one device from the admin's local network.
>>>> And hopefully this reduces the pihole questions from the Community.
>>>> 
>>>> A list of RPZ providers can be recommended by IPFire and coded into a set list.
>>>> Or, if prefered, the local admin can choose their own RPZ providers.
>>>> 
>>>> This is a:
>>>> * simple replacement for pihole base functionality
>>>> * RPZ can be a nice replacement for the URL Filter
>>>> 
>>>> IPFire Wiki
>>>> In process at: https://www.ipfire.org/docs/addons/rpz
>>>> 
>>>> more info:
>>>> * https://en.wikipedia.org/wiki/Response_policy_zone
>>>> * https://unbound.docs.nlnetlabs.nl/en/latest/topics/filtering/rpz.html
>>>> 
>>>> I need help with...
>>>> 1) The custom allow and block lists are currently located at `/var/ipfire/rpz`.  Is this correct?
>>>> 2) The three bash scripts are currently located at `/usr/sbin`.  Is this correct?
>>>> 
>>>> Signed-off-by: Jon Murphy <jon.murphy@ipfire.org>
>>>> ---
>>>> config/backup/includes/rpz    |   5 +
>>>> config/rootfiles/packages/rpz |  11 ++
>>>> config/rpz/00-rpz.conf        |  18 ++++
>>>> config/rpz/rpz-config         | 194 ++++++++++++++++++++++++++++++++++
>>>> config/rpz/rpz-metrics        | 143 +++++++++++++++++++++++++
>>>> config/rpz/rpz-sleep          |  58 ++++++++++
>>>> lfs/rpz                       |  88 +++++++++++++++
>>>> make.sh                       |   2 +
>>>> 8 files changed, 519 insertions(+)
>>>> create mode 100644 config/backup/includes/rpz
>>>> create mode 100644 config/rootfiles/packages/rpz
>>>> create mode 100644 config/rpz/00-rpz.conf
>>>> create mode 100644 config/rpz/rpz-config
>>>> create mode 100644 config/rpz/rpz-metrics
>>>> create mode 100644 config/rpz/rpz-sleep
>>>> create mode 100644 lfs/rpz
>>>> 
>>>> diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz
>>>> new file mode 100644
>>>> index 000000000..4d59bb40c
>>>> --- /dev/null
>>>> +++ b/config/backup/includes/rpz
>>>> @@ -0,0 +1,5 @@
>>>> +/var/ipfire/rpz/allowlist
>>>> +/var/ipfire/rpz/blocklist
>>>> +/etc/unbound/zonefiles/allow.rpz
>>>> +/etc/unbound/zonefiles/block.rpz
>>>> +/etc/unbound/local.d/*rpz.conf
>>>> diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/rpz
>>>> new file mode 100644
>>>> index 000000000..2ffa715dd
>>>> --- /dev/null
>>>> +++ b/config/rootfiles/packages/rpz
>>>> @@ -0,0 +1,11 @@
>>>> +etc/unbound/local.d/00-rpz.conf
>>>> +etc/unbound/zonefiles
>>>> +etc/unbound/zonefiles/allow.rpz
>>>> +etc/unbound/zonefiles/block.rpz
>>>> +usr/sbin/rpz-config
>>>> +usr/sbin/rpz-metrics
>>>> +usr/sbin/rpz-sleep
>>>> +var/ipfire/backup/addons/includes/rpz
>>>> +var/ipfire/rpz
>>>> +var/ipfire/rpz/allowlist
>>>> +var/ipfire/rpz/blocklist
>>>> diff --git a/config/rpz/00-rpz.conf b/config/rpz/00-rpz.conf
>>>> new file mode 100644
>>>> index 000000000..72c1d12e5
>>>> --- /dev/null
>>>> +++ b/config/rpz/00-rpz.conf
>>>> @@ -0,0 +1,18 @@
>>>> +server:
>>>> + module-config: "respip validator iterator"
>>>> +
>>>> +rpz:
>>>> +    name: allow.rpz
>>>> +    zonefile: /etc/unbound/zonefiles/allow.rpz
>>>> +    rpz-action-override: passthru
>>>> +    rpz-log: yes
>>>> +    rpz-log-name: allow
>>>> +    rpz-signal-nxdomain-ra: yes
>>>> +
>>>> +rpz:
>>>> +    name: block.rpz
>>>> +    zonefile: /etc/unbound/zonefiles/block.rpz
>>>> +    rpz-action-override: nxdomain
>>>> +    rpz-log: yes
>>>> +    rpz-log-name: block
>>>> +    rpz-signal-nxdomain-ra: yes
>>>> diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config
>>>> new file mode 100644
>>>> index 000000000..98dc0a4ca
>>>> --- /dev/null
>>>> +++ b/config/rpz/rpz-config
>>>> @@ -0,0 +1,194 @@
>>>> +#!/bin/bash
>>>> +###############################################################################
>>>> +#                                                                             #
>>>> +#  IPFire.org - A linux based firewall                                        #
>>>> +#  Copyright (C) 2024  IPFire Team  <info@ipfire.org>                         #
>>>> +#                                                                             #
>>>> +#  This program is free software: you can redistribute it and/or modify       #
>>>> +#  it under the terms of the GNU General Public License as published by       #
>>>> +#  the Free Software Foundation, either version 3 of the License, or          #
>>>> +#  (at your option) any later version.                                        #
>>>> +#                                                                             #
>>>> +#  This program is distributed in the hope that it will be useful,            #
>>>> +#  but WITHOUT ANY WARRANTY; without even the implied warranty of             #
>>>> +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the              #
>>>> +#  GNU General Public License for more details.                               #
>>>> +#                                                                             #
>>>> +#  You should have received a copy of the GNU General Public License          #
>>>> +#  along with this program.  If not, see <http://www.gnu.org/licenses/>.      #
>>>> +#                                                                             #
>>>> +###############################################################################
>>>> +
>>>> +# v22 - 2024-07-12
>>>> +
>>>> +###############     Functions     ###############
>>>> +
>>>> +msg_log () {
>>>> + /usr/bin/logger --tag "${tagName}" "$*"
>>>> + if tty --silent ; then
>>>> + echo "${tagName}:" "$*"
>>>> + fi
>>>> +}
>>>> +
>>>> +check_name () {
>>>> + local testName="${1}"
>>>> + # check for a valid name
>>>> + regex='^[a-zA-Z0-9_]+$'
>>>> + if [[ ! "${testName}" =~ $regex ]] ; then
>>>> + msg_log "error: rpz: the NAME is not valid: \"${testName}\". exit."
>>>> + exit 1
>>>> + fi
>>>> +}
>>>> +
>>>> +check_unbound_conf () {
>>>> + # check the above config files
>>>> + msg_log "info: rpz: check for errors with \"unbound-checkconf\""
>>>> + /usr/sbin/unbound-checkconf
>>>> + exit_code=$?
>>>> + if [[ "${exit_code}" -ne 0 ]] ; then
>>>> + msg_log "error: rpz: unbound-checkconf. exit."
>>>> + exit "${exit_code}"
>>>> + fi
>>>> +}
>>>> +
>>>> +make_rpz_file () {
>>>> + local theType="${1}" # allow or block
>>>> +
>>>> + theList="/var/ipfire/rpz/${theType}list" # input user list of domains
>>>> + theZoneFile="/etc/unbound/zonefiles/${theType}.rpz" # output file for RPZ
>>>> +
>>>> + theAction='.'
>>>> + if [[ "${theType}" =~ "block" ]] ; then
>>>> + theAction='rpz-passthru.'
>>>> + fi
>>>> +
>>>> + # does a list exist?
>>>> + if [[ -s "${theList}" ]] ; then
>>>> + # drop any extra "blanks" and add "CNAME <RPZ action>." to each line
>>>> + actionList=$( /usr/bin/awk '{$1=$1};1' "${theList}" |
>>>> + /bin/sed "/^[^;].*[[:alnum:]]/ s|$| CNAME ${theAction}|" )
>>>> +
>>>> + msg_log "info: rpz: create zonefile for ${theList}"
>>>> +
>>>> + /bin/cat <<-EOF > "${theZoneFile}"
>>>> + ; Name: ${theType} list
>>>> + ; Last modified: $(date "+%Y-%m-%d at %H.%M.%S %Z")
>>>> + ;
>>>> + ; domains with actions list
>>>> + ;
>>>> + ${actionList}
>>>> + EOF
>>>> +
>>>> + # reload the zone that was just updated
>>>> + zoneBase=$( basename "${theZoneFile}" )
>>>> + /usr/sbin/unbound-control auth_zone_reload -q "${zoneBase}"
>>>> + fi
>>>> +}
>>>> +
>>>> +###############       Main        ###############
>>>> +
>>>> +tagName="unbound"
>>>> +
>>>> +theAction="${1}" # input action
>>>> +theName="${2}" # input RPZ name
>>>> +theURL="${3}" # input RPZ URL
>>>> +
>>>> +check_name "${theName}" # is this a valid name?
>>>> +
>>>> +rpzConfig="/etc/unbound/local.d/${theName}.rpz.conf" # output zone conf file
>>>> +rpzFile="/etc/unbound/zonefiles/${theName}.rpz" # output for RPZ file
>>>> +
>>>> +case "${theAction}" in
>>>> +
>>>> + # add new rpz list
>>>> + add )
>>>> + # does this config already exist?  If yes, then exit
>>>> + if [[ -f "${rpzConfig}" ]] ; then
>>>> + msg_log "info: rpz: ${rpzConfig} already exists. exit"
>>>> + exit 1
>>>> + fi
>>>> +
>>>> + # is this a valid URL?
>>>> + regex='^https://[-[:alnum:]\+&@#/%?=~_|!:,.;]*[-[:alnum:]\+&@#/%=~_|]'
>>>> + if ! [[ "${theURL}" =~ $regex ]] ; then
>>>> + msg_log "error: rpz: the URL is not valid: \"${theURL}\". exit."
>>>> + exit 1
>>>> + fi
>>>> +
>>>> + # create the zone config file
>>>> + msg_log "info: rpz: add config file \"${theName}.rpz.conf\""
>>>> + cat <<-EOF > "${rpzConfig}"
>>>> + rpz:
>>>> + name: ${theName}.rpz
>>>> + zonefile: /etc/unbound/zonefiles/${theName}.rpz
>>>> + url:  ${theURL}
>>>> + rpz-action-override: nxdomain
>>>> + rpz-log: yes
>>>> + rpz-log-name: ${theName}
>>>> + rpz-signal-nxdomain-ra: yes
>>>> + EOF
>>>> +
>>>> + # set-up zone file
>>>> + /usr/bin/touch "${rpzFile}"
>>>> + # unbound requires these settings for rpz files
>>>> + /bin/chown --verbose nobody:nobody "${rpzFile}"
>>>> + /bin/chmod --verbose 644 "${rpzFile}"
>>>> + ;;
>>>> +
>>>> + # trash config file & rpz file
>>>> + remove )
>>>> + if ! [[ -f "${rpzConfig}" ]] ; then
>>>> + msg_log "info: rpz: ${rpzConfig} does not exist. exit"
>>>> + exit 1
>>>> + fi
>>>> +
>>>> + msg_log "info: rpz: remove config file & rpz file \"${theName}\""
>>>> + /bin/rm --verbose "${rpzConfig}"
>>>> + /bin/rm --verbose "${rpzFile}"
>>>> +
>>>> + check_unbound_conf
>>>> + ;;
>>>> +
>>>> + # make a new allow or block rpz file
>>>> + make )
>>>> + case "${theName}" in
>>>> + allow )
>>>> + make_rpz_file allow
>>>> + ;;
>>>> +
>>>> + block )
>>>> + make_rpz_file block
>>>> + ;;
>>>> +
>>>> + allowblock )
>>>> + make_rpz_file allow
>>>> + make_rpz_file block
>>>> + ;;
>>>> +
>>>> + * )
>>>> + msg_log "error: rpz: the NAME is not valid: \"${theName}\". exit."
>>>> + exit 1
>>>> + ;;
>>>> + esac
>>>> +
>>>> + check_unbound_conf
>>>> + ;;
>>>> +
>>>> + *)
>>>> + msg_log "error: rpz: missing or incorrect parameter"
>>>> + /usr/bin/printf "Usage:  rpzConfig.sh <ACTION> <NAME> <URL>\n"
>>>> + exit 1
>>>> + ;;
>>>> +
>>>> +esac
>>>> +
>>>> +# reload due to the changes
>>>> +msg_log  "rpz: running \"unbound-control reload\""
>>>> +/usr/sbin/unbound-control reload
>>>> +exit_code=$?
>>>> +if [[ "${exit_code}" -ne 0 ]] ; then
>>>> + msg_log "error: rpz: unbound-control \"${theName}\". exit."
>>>> + exit "${exit_code}"
>>>> +fi
>>>> +
>>>> +exit
>>>> diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics
>>>> new file mode 100644
>>>> index 000000000..0f97c7911
>>>> --- /dev/null
>>>> +++ b/config/rpz/rpz-metrics
>>>> @@ -0,0 +1,143 @@
>>>> +#!/bin/bash
>>>> +###############################################################################
>>>> +#                                                                             #
>>>> +#  IPFire.org - A linux based firewall                                        #
>>>> +#  Copyright (C) 2024  IPFire Team  <info@ipfire.org>                         #
>>>> +#                                                                             #
>>>> +#  This program is free software: you can redistribute it and/or modify       #
>>>> +#  it under the terms of the GNU General Public License as published by       #
>>>> +#  the Free Software Foundation, either version 3 of the License, or          #
>>>> +#  (at your option) any later version.                                        #
>>>> +#                                                                             #
>>>> +#  This program is distributed in the hope that it will be useful,            #
>>>> +#  but WITHOUT ANY WARRANTY; without even the implied warranty of             #
>>>> +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the              #
>>>> +#  GNU General Public License for more details.                               #
>>>> +#                                                                             #
>>>> +#  You should have received a copy of the GNU General Public License          #
>>>> +#  along with this program.  If not, see <http://www.gnu.org/licenses/>.      #
>>>> +#                                                                             #
>>>> +###############################################################################
>>>> +
>>>> +# v18 on 2024-07-05
>>>> +
>>>> +###############       Main        ###############
>>>> +
>>>> +weeks="${1:-2}" # default to two message logs
>>>> +sortBy="${2:-name}"   # by name or by hits
>>>> +
>>>> +# get the list of message logs for N weeks
>>>> +messageLogs=$( find /var/log/messages* -type f |
>>>> + /usr/bin/sort --version-sort |
>>>> + head -"${weeks}" )
>>>> +
>>>> +# get the list of RPZ names & counts from the message log(s)
>>>> +rpzNameCount=$( for logf in ${messageLogs} ; do
>>>> + /usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" |
>>>> + /usr/bin/awk '$10 ~ /\[\w*]/ { print $10 }' ;
>>>> + done | /usr/bin/sort | /usr/bin/uniq --count )
>>>> +
>>>> +# flip results and remove brackets `[` and `]`
>>>> +rpzNameCount=$( /bin/echo "${rpzNameCount}" |
>>>> + /usr/bin/awk '{ print $2, $1 }' |
>>>> + /bin/sed --regexp-extended 's|^\[(.*)\]|\1|' )
>>>> +
>>>> +# grab only names
>>>> +rpzNames=$( /bin/echo "${rpzNameCount}" | /usr/bin/awk '{ print $1 }' )
>>>> +
>>>> +# get list of RPZ files
>>>> +rpzFileList=$( /bin/find /etc/unbound/zonefiles -type f -iname "*.rpz" )
>>>> +
>>>> +# get basename of those files
>>>> +rpzBaseNames=$( /bin/echo "${rpzFileList}" |
>>>> + /bin/sed 's|/etc/unbound/zonefiles/||g ; s|\.rpz||g ;' )
>>>> +
>>>> +# add to rpzNames
>>>> +rpzNames="${rpzNames}"$'\n'"${rpzBaseNames}"
>>>> +
>>>> +# drop duplicate names
>>>> +rpzNames=$( echo "${rpzNames}" | /usr/bin/sort --unique  )
>>>> +
>>>> +# get line count for each RPZ
>>>> +lineCount=$( /bin/echo "${rpzFileList}" | /usr/bin/xargs wc -l )
>>>> +
>>>> +# get comment line count and blank line count for each RPZ
>>>> +commentCount=$( /bin/echo "${rpzFileList}" |
>>>> + /usr/bin/xargs /bin/grep --count -e "^$" -e "^;" )
>>>> +
>>>> +# get modified date each RPZ
>>>> +modDateList=$( /bin/echo "${rpzFileList}" | xargs stat -c '%.10y  %n' )
>>>> +
>>>> +ucListAuthZones=$( /usr/sbin/unbound-control list_auth_zones )
>>>> +
>>>> +# get width of RPZ names
>>>> +pWidth=$( /bin/echo "${rpzNames}" | /usr/bin/awk '{ print $1"   " }' | wc -L )
>>>> +pFormat="%-${pWidth}s %-8s %-8s %-8s %10s %12s\n"
>>>> +
>>>> +# print title line
>>>> +printf "${pFormat}" "name" "hits" "active" "lines" "hits/line%" "last update"
>>>> +printf -- "--------------"
>>>> +
>>>> +theResults=""
>>>> +totalLines=0
>>>> +totalHits=0
>>>> +while read -r theName
>>>> +do
>>>> + printf -- "-" # pretend progress bar
>>>> + # get hit count
>>>> + theHits="0"
>>>> + if output=$( /bin/grep "^${theName}\s" <<< "${rpzNameCount}" ) ; then
>>>> + theHits=$( /bin/echo "${output}" |
>>>> + /usr/bin/awk '{ print $2 }' )
>>>> + totalHits=$(( totalHits + theHits ))
>>>> + fi
>>>> +
>>>> + # is this RPZ list active?
>>>> + theActive="disabled"
>>>> + if /bin/grep --quiet "^${theName}\.rpz" <<< "${ucListAuthZones}"
>>>> + then
>>>> + theActive="enabled"
>>>> + fi
>>>> +
>>>> + # get line count then subtract comment count and blank line count
>>>> + #  from total line count
>>>> + theLines="n/a"
>>>> + hitsPerLine="0"
>>>> + if output=$( /bin/grep --fixed-strings "/${theName}.rpz" <<< "${lineCount}" ) ; then
>>>> + theLines=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' )
>>>> + totalLines=$(( totalLines + theLines ))
>>>> +
>>>> + #hitsPerLine=$( echo "scale=0 ; $theHits / $theLines" | bc )
>>>> + hitsPerLine=$(( 100 * theHits / theLines ))
>>>> + fi
>>>> +
>>>> + # get modification date
>>>> + theModDate="n/a"
>>>> + if output=$( /bin/grep --fixed-strings "/${theName}.rpz" <<< "${modDateList}" ) ; then
>>>> + theModDate=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' )
>>>> + fi
>>>> +
>>>> + # add to results list
>>>> + theResults+="${theName} ${theHits} ${theActive} ${theLines} ${hitsPerLine} ${theModDate}"$'\n'
>>>> +done <<< "${rpzNames}"
>>>> +
>>>> +case "${sortBy}" in
>>>> + names|name) sortArg=(-k3,3r -k1,1) ;; # sort by "active" then by "name"
>>>> +
>>>> + hits|hit) sortArg=(-k3,3r -k2,2nr -k1,1) ;; # sort by "active" then by "hits" then by "name"
>>>> +
>>>> + lines|line) sortArg=(-k3,3r -k4,4nr -k1,1) ;; # sort by "active" then by "lines" then by "name"
>>>> +esac
>>>> +
>>>> +printf -- "--------------\n"
>>>> +# remove blank lines, sort, print as columns
>>>> +/bin/echo "${theResults}" |
>>>> + /usr/bin/awk '!/^[[:space:]]*$/' |
>>>> + /usr/bin/sort "${sortArg[@]}"    |
>>>> + /usr/bin/awk --assign=width="${pWidth}" \
>>>> + '{ printf "%-*s %-8s %-8s %-8s %10s %12s\n", width, $1, $2, $3, $4, $5, $6 }'
>>>> +
>>>> +printf "${pFormat}" "" "=======" "" "========" "" ""
>>>> +printf "${pFormat}" "Totals -->" "${totalHits}" "" "${totalLines}" "" ""
>>>> +
>>>> +exit
>>>> diff --git a/config/rpz/rpz-sleep b/config/rpz/rpz-sleep
>>>> new file mode 100644
>>>> index 000000000..eeef1174a
>>>> --- /dev/null
>>>> +++ b/config/rpz/rpz-sleep
>>>> @@ -0,0 +1,58 @@
>>>> +#!/bin/bash
>>>> +###############################################################################
>>>> +#                                                                             #
>>>> +#  IPFire.org - A linux based firewall                                        #
>>>> +#  Copyright (C) 2024  IPFire Team  <info@ipfire.org>                         #
>>>> +#                                                                             #
>>>> +#  This program is free software: you can redistribute it and/or modify       #
>>>> +#  it under the terms of the GNU General Public License as published by       #
>>>> +#  the Free Software Foundation, either version 3 of the License, or          #
>>>> +#  (at your option) any later version.                                        #
>>>> +#                                                                             #
>>>> +#  This program is distributed in the hope that it will be useful,            #
>>>> +#  but WITHOUT ANY WARRANTY; without even the implied warranty of             #
>>>> +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the              #
>>>> +#  GNU General Public License for more details.                               #
>>>> +#                                                                             #
>>>> +#  You should have received a copy of the GNU General Public License          #
>>>> +#  along with this program.  If not, see <http://www.gnu.org/licenses/>.      #
>>>> +#                                                                             #
>>>> +###############################################################################
>>>> +
>>>> +# v04 on 2024-07-05
>>>> +
>>>> +###############     Functions     ###############
>>>> +
>>>> +# send message to message log
>>>> +msg_log () {
>>>> + /usr/bin/logger --tag "${tagName}" "$*"
>>>> + if /usr/bin/tty --silent ; then
>>>> + echo "${tagName}:" "$*"
>>>> + fi
>>>> +}
>>>> +
>>>> +###############       Main        ###############
>>>> +
>>>> +tagName="unbound"
>>>> +
>>>> +sleepTime="${1:-5m}" # default to sleep for 5m (5 minutes)
>>>> +
>>>> +zoneList=$( /usr/sbin/unbound-control list_auth_zones | /usr/bin/awk '{print $1}' )
>>>> +
>>>> +for zone in ${zoneList} ; do
>>>> + /usr/bin/printf "disable ${zone}\t"
>>>> + /usr/sbin/unbound-control rpz_disable "${zone}"
>>>> +done
>>>> +
>>>> +msg_log "info: rpz: disabled all zones for ${sleepTime}"
>>>> +
>>>> +/bin/sleep "${sleepTime}"
>>>> +
>>>> +for zone in ${zoneList} ; do
>>>> + /usr/bin/printf "enable ${zone}\t"
>>>> + /usr/sbin/unbound-control rpz_enable "${zone}"
>>>> +done
>>>> +
>>>> +msg_log "info: rpz: enabled all zones"
>>>> +
>>>> +exit
>>>> diff --git a/lfs/rpz b/lfs/rpz
>>>> new file mode 100644
>>>> index 000000000..319c10b7f
>>>> --- /dev/null
>>>> +++ b/lfs/rpz
>>>> @@ -0,0 +1,88 @@
>>>> +###############################################################################
>>>> +#                                                                             #
>>>> +# IPFire.org - A linux based firewall                                         #
>>>> +# Copyright (C) 2024  IPFire Team  <info@ipfire.org>                          #
>>>> +#                                                                             #
>>>> +# This program is free software: you can redistribute it and/or modify        #
>>>> +# it under the terms of the GNU General Public License as published by        #
>>>> +# the Free Software Foundation, either version 3 of the License, or           #
>>>> +# (at your option) any later version.                                         #
>>>> +#                                                                             #
>>>> +# This program is distributed in the hope that it will be useful,             #
>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of              #
>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the               #
>>>> +# GNU General Public License for more details.                                #
>>>> +#                                                                             #
>>>> +# You should have received a copy of the GNU General Public License           #
>>>> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.       #
>>>> +#                                                                             #
>>>> +###############################################################################
>>>> +
>>>> +###############################################################################
>>>> +# Definitions
>>>> +###############################################################################
>>>> +
>>>> +include Config
>>>> +
>>>> +SUMMARY    = response policy zone - RPZ reputation system for unbound DNS
>>>> +
>>>> +VER        = 1.0.0
>>>> +
>>>> +THISAPP    = rpz-$(VER)
>>>> +DIR_APP    = $(DIR_SRC)/$(THISAPP)
>>>> +TARGET     = $(DIR_INFO)/$(THISAPP)
>>>> +
>>>> +PROG       = rpz
>>>> +PAK_VER    = 1
>>>> +
>>>> +DEPS       =
>>>> +
>>>> +SERVICES   =
>>>> +
>>>> +###############################################################################
>>>> +# Top-level Rules
>>>> +###############################################################################
>>>> +
>>>> +install : $(TARGET)
>>>> +
>>>> +check :
>>>> +
>>>> +download :
>>>> +
>>>> +b2 :
>>>> +
>>>> +dist:
>>>> + @$(PAK)
>>>> +
>>>> +###############################################################################
>>>> +# Installation Details
>>>> +###############################################################################
>>>> +
>>>> +$(TARGET) :
>>>> + @$(PREBUILD)
>>>> + @rm -rf $(DIR_APP)
>>>> +
>>>> + # install RPZ scripts
>>>> + install -v -m 755 \
>>>> +  $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin
>>>> +
>>>> + # Install settings folder and two empty files
>>>> + mkdir -pv /var/ipfire/rpz
>>>> + touch /var/ipfire/rpz/allowlist
>>>> + touch /var/ipfire/rpz/blocklist
>>>> +
>>>> + # Add conf file to /etc directory
>>>> + cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d
>>>> +
>>>> + # create zonefiles directory for the RPZ files and add two empty RPZ
>>>> + #  files to avoid a unbound config error
>>>> + mkdir -pv /etc/unbound/zonefiles
>>>> + chown -v nobody:nobody /etc/unbound/zonefiles
>>>> + touch /etc/unbound/zonefiles/allow.rpz
>>>> + touch /etc/unbound/zonefiles/block.rpz
>>>> +
>>>> + # Install backup definition
>>>> + cp -vf $(DIR_CONF)/backup/includes/rpz /var/ipfire/backup/addons/includes/rpz
>>>> +
>>>> + @rm -rf $(DIR_APP)
>>>> + @$(POSTBUILD)
>>>> diff --git a/make.sh b/make.sh
>>>> index 9bbbeb0f1..886d3760a 100755
>>>> --- a/make.sh
>>>> +++ b/make.sh
>>>> @@ -1721,6 +1721,8 @@ buildipfire() {
>>>> lfsmake2 btrfs-progs
>>>> lfsmake2 inotify-tools
>>>> lfsmake2 grub-btrfs
>>>> +  lfsmake2 rpz
>>>> +
>>>> 
>>>> # Kernelbuild ... current we have no platform that need
>>>> # multi kernel builds so KCFG is empty
>>>> -- 
>>>> 2.30.2
>>>> 
>>> 
>> 
>

Adolf Belka July 30, 2024, 10:05 a.m. UTC | #5

Hi Jon,
I have given some feedback in your patches below.

On 29/07/2024 04:28, jon wrote:
> See comments below...
>
>> On Jul 28, 2024, at 6:22 AM, Michael Tremer <michael.tremer@ipfire.org> wrote:
>>
>> Hello Jon,
>>
>> I don’t think it is a good idea to send over the same patch set multiple times.
>>
>> I think first of all, we need to start at the beginning which is to figure out what this actually is, and why we would want this in IPFire.
>>
>> I have seen that there have been conversations on community.ipfire.org <http://community.ipfire.org/> many months ago, but I did not take many things away from it. It was from my point of view mostly a prove of concept discussion. Correct me if I have missed anything here.
>>
>> We have talked about RPZ many times on the monthly call since the URL filter feature is falling more and more out of fashion. I think there is also many posts about this on the forum.
> Yes, I have seen the posts about the url filter feature falling out of fashion.  I did not see notes about RPZ from the monthly calls.  I will go back and search telco call notes.
>
>
>> Apart from generally being in favour of trying this path, we decided to postpone any further works into this because we currently have a huge backlog of things to deal with  that are moving at snail speed. This is rooted in so many people being simply tied in with other things.
>>
>> As you can see, the patch set that you posted did not trigger a response - a common thing on this list recently and not because of this particular patch set.
>>
>> Therefore I would like to avoid opening another can of worms before we have delivered the other things that have already lots of time spent on them and code written. Otherwise we are only getting a longer pipeline and this all is going to be more frustrating for everyone involved. And this is not about the feature per se; this is about getting things done.
> This is why I wrote the RPZ patch instead of asking for the Developers to write it.  I realize there is a very long pipeline.
>
>
>> So what I suggest doing is the following:
>>
>> You have a Git repository, push the code there and commit any fixes to it over time so that nothing is getting lost.
> This was done a few days ago.  But I am not sure if I did it correctly.  If not, please let me know.
>
>
>> Explain to me from the beginning, please, how it came to this patch set. What is the motivation?
> In the beginning (Fall of 2023) the main motivation was blocking DoH.  I had noticed apps using their own method of DNS lookup.  Some were friendly DoH lookups (Apple iCloud?) and some not (mozilla.cloudflare-dns.com but not from Firefox).  Examples I saw are:
> mozilla.cloudflare-dns.com <http://mozilla.cloudflare-dns.com/>
> www.switch.ch <http://www.switch.ch/>
> use-application-dns.net <http://use-application-dns.net/> (a canary for Firefox?)
> And a few others...
>
> This worked better than expected and was easier to configure than expected.  All of the main code was part of unbound and I just need to activate it with a config file.  Updates to the RPZ files are already part of the unbound RPZ code.
>
> As I experimented with RPZ I quickly realized it could easily be used to block other websites.  I have been using pihole for a number of years to limit the number of ads and trackers.  And RPZ helped with those areas also.
>
>
>> What problems does it solve? What problems remain? Basically: Why would anyone want this in IPFire?
> It solved the issue of blocking unwanted DoH from apps.  And it solved the issue of unwanted ads and tracking.  For me those are huge benefits and the biggest reason for using RPZ.
>
> To test other possibilities, I increased the "type" of lists to include other categories other than ads & trackers.  Like blocking bad TLDs, gambling, piracy, bad host sites, etc.  This is a nice bonus of using RPZ for me.
>
>> Where do you see application for this?
> As a replacement for the URL Filter.  And as a base functional replacement for pihole (but no pretty graphics).
>
>
>> Why is this realised as an add-on and not part of the core system?
> In my opinion it should be part of the core.  But I wanted to "test the waters" first to determine interest.  If only four people use it, then it isn’t worth adding to core and maybe not as an add-on.
>
> Either way I will support it because it helps me!
>
> Here are my metrics for the past eight days (searching `messages` and `messages.1.gz`):
>
> [root@ipfire ~] # rpz-metrics
> name               hits
> ----------------------------
> allow              5560    This is a custom list
> AmazonTrkrHZ       5726
> AppleTrkrHZ        2956
> block              128     This is a custom list
> DOHblockHZ         25077
> dohJPG             3448
> HosterHZ           19
> MxProPlusHZ        17287
> NotSafeSearchHZ    0
> ProxyBypassHZ      11
> tldHZ              15
> WinTrkrHZ          507
>                     =======
> Totals -->         60734
>
> For the DoH blocking, the above "hits" including blocking Apple DoH.  I’ve been testing this for the past few weeks.
>
>> I am particularly interested in what lists are publicly available. You know from URL Filter, that lists have shut down or just been abandoned. There is now very little point running it because the lists are just not as good as they used to be. Not horrible, yet, but over time this will not be getting any better. I am not even aware of any viable paid lists. Is this the same situation for RPZ?  A quick Google search did not give me anything that would be free to use over DNS. The situation is somewhat similar with the IPS and IP blocklist feature.
> These are free/open lists:
>
> Hagezi - DNS Blocklists
> • at https://github.com/hagezi/dns-blocklists?tab=readme-ov-file#zap-dns-blocklists---for-a-better-internet
> • This list was recommended within the IPFire Community and this is the list I strongly recommend.  Gerd (Hagezi) is very responsive to support requests. The Hagezi RPZ list is well supported and the RPZ lists are updated twice per day (if changes are needed).
> • Example: I came across a domain name that was blocked `f005.backblazeb2.com` and it was fixed (removed from the block list) within a few days.
>
> ThreatFox - DNS Response Policy Zone (RPZ)
> • at https://threatfox.abuse.ch/export/#rpz
>
> URLHaus - DNS Response Policy Zone (RPZ)
> • at https://urlhaus.abuse.ch/api/#rpz
>
> jpgpi250 - DNS block list for DoH
> • at https://github.com/jpgpi250/piholemanual/blob/master/DOH.rpz
> • this is the original DoH block list that I started with.  Peter (jpgpi250) has a nice PDF document explaining RPZ and was very helpful getting me started with RPZ.  See https://jpgpi250.github.io/piholemanual/doc/Unbound%20response%20policy%20zones.pdf
>
> I have come across other RPZ lists but these are the best supported lists.
>
> I did not look into paid lists.
>
>>
>> I would be interested in hearing your thoughts on this.
> Does the above help?
>
>> Best,
>> -Michael
>>
>>> On 27 Jul 2024, at 21:31, jon <jon.murphy@ipfire.org> wrote:
>>>
>>> And another question.
>>>
>>> I found an error in the `rpz-config` script file.  Simple easy one line (actually one word) fix.
>>>
>>> Do I resubmit the entire patch?
>>> -or-
>>> Do I wait until approved and then submit the simple patch for the one line change.
>>>
>>>
>>> Jon
>>>
>>>
>>>> On Jul 20, 2024, at 1:42 PM, jon <jon.murphy@ipfire.org> wrote:
>>>>
>>>> I need help with…
>>>>
>>>> 1) The custom allow and block lists are currently located at `/var/ipfire/rpz`.  Is this correct?
>>>>
>>>> 2) The three bash scripts are currently located at `/usr/sbin`.  Is this correct?
>>>>
>>>> 3) Should this commit be separated into multiple posts?  If yes, how do I decide how to split it?
>>>>
>>>>
>>>> Jon
>>>>
>>>>
>>>>> On Jul 20, 2024, at 12:46 PM, Jon Murphy <jon.murphy@ipfire.org> wrote:
>>>>>
>>>>> What is it?
>>>>> Response Policy Zone (RPZ) is a mechanism to define local policies in a standardised
>>>>> way and load those policies from external sources.
>>>>> Bottom line: RPZ allows admins to easily block access to websites via DNS lookup.
>>>>>
>>>>> RPZ can block websites via categories.  Examples include: fake websites, annoying
>>>>> pop-up ads, newly registered domains, DoH bypass sites, bad "host" services,
>>>>> maliscious top level domains (e.g., *.zip, *.mov), piracy, gambling, pornography,
>>>>> and more.  RPZ lists come from various RPZ providers and their available
>>>>> catagories.
>>>>>
>>>>> This RPZ add-on enables the RPZ functionality by adding a couple lines in a
>>>>> configuration file.  This add-on simply adds a configuration file and adds
>>>>> three scripts (config, metrics and sleep) to make RPZ easier for the admin to use.
>>>>>
>>>>> RPZ was release in 2010 and has been part of the IPFire build since ~2015.
>>>>>
>>>>> Why is it needed?
>>>>> Some IPFire admin's utilize pihole to block unwanted websites via DNS lookup.
>>>>> Moving the pihole base functionality (without pretty graphs) to IPFire removes
>>>>> one device from the admin's local network.
>>>>> And hopefully this reduces the pihole questions from the Community.
>>>>>
>>>>> A list of RPZ providers can be recommended by IPFire and coded into a set list.
>>>>> Or, if prefered, the local admin can choose their own RPZ providers.
>>>>>
>>>>> This is a:
>>>>> * simple replacement for pihole base functionality
>>>>> * RPZ can be a nice replacement for the URL Filter
>>>>>
>>>>> IPFire Wiki
>>>>> In process at: https://www.ipfire.org/docs/addons/rpz
>>>>>
>>>>> more info:
>>>>> * https://en.wikipedia.org/wiki/Response_policy_zone
>>>>> * https://unbound.docs.nlnetlabs.nl/en/latest/topics/filtering/rpz.html
>>>>>
>>>>> I need help with...
>>>>> 1) The custom allow and block lists are currently located at `/var/ipfire/rpz`.  Is this correct?
rpz is supposed to be part of the dns so I would either put all the 
files in /var/ipfire/dns/ or create a directory for rpz under the dns 
directory.
>>>>> 2) The three bash scripts are currently located at `/usr/sbin`.  Is this correct?
>>>>>
>>>>> Signed-off-by: Jon Murphy <jon.murphy@ipfire.org>
>>>>> ---
>>>>> config/backup/includes/rpz    |   5 +
>>>>> config/rootfiles/packages/rpz |  11 ++
>>>>> config/rpz/00-rpz.conf        |  18 ++++
>>>>> config/rpz/rpz-config         | 194 ++++++++++++++++++++++++++++++++++
>>>>> config/rpz/rpz-metrics        | 143 +++++++++++++++++++++++++
>>>>> config/rpz/rpz-sleep          |  58 ++++++++++
>>>>> lfs/rpz                       |  88 +++++++++++++++
>>>>> make.sh                       |   2 +
>>>>> 8 files changed, 519 insertions(+)
>>>>> create mode 100644 config/backup/includes/rpz
>>>>> create mode 100644 config/rootfiles/packages/rpz
>>>>> create mode 100644 config/rpz/00-rpz.conf
>>>>> create mode 100644 config/rpz/rpz-config
>>>>> create mode 100644 config/rpz/rpz-metrics
>>>>> create mode 100644 config/rpz/rpz-sleep
>>>>> create mode 100644 lfs/rpz
>>>>>
>>>>> diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz
>>>>> new file mode 100644
>>>>> index 000000000..4d59bb40c
>>>>> --- /dev/null
>>>>> +++ b/config/backup/includes/rpz
>>>>> @@ -0,0 +1,5 @@
>>>>> +/var/ipfire/rpz/allowlist
>>>>> +/var/ipfire/rpz/blocklist
>>>>> +/etc/unbound/zonefiles/allow.rpz
>>>>> +/etc/unbound/zonefiles/block.rpz
>>>>> +/etc/unbound/local.d/*rpz.conf
>>>>> diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/rpz
>>>>> new file mode 100644
>>>>> index 000000000..2ffa715dd
>>>>> --- /dev/null
>>>>> +++ b/config/rootfiles/packages/rpz
>>>>> @@ -0,0 +1,11 @@
>>>>> +etc/unbound/local.d/00-rpz.conf
>>>>> +etc/unbound/zonefiles
>>>>> +etc/unbound/zonefiles/allow.rpz
>>>>> +etc/unbound/zonefiles/block.rpz
>>>>> +usr/sbin/rpz-config
>>>>> +usr/sbin/rpz-metrics
>>>>> +usr/sbin/rpz-sleep
>>>>> +var/ipfire/backup/addons/includes/rpz
>>>>> +var/ipfire/rpz
>>>>> +var/ipfire/rpz/allowlist
>>>>> +var/ipfire/rpz/blocklist
>>>>> diff --git a/config/rpz/00-rpz.conf b/config/rpz/00-rpz.conf
>>>>> new file mode 100644
>>>>> index 000000000..72c1d12e5
>>>>> --- /dev/null
>>>>> +++ b/config/rpz/00-rpz.conf
>>>>> @@ -0,0 +1,18 @@
>>>>> +server:
>>>>> + module-config: "respip validator iterator"
>>>>> +
>>>>> +rpz:
>>>>> +    name: allow.rpz
>>>>> +    zonefile: /etc/unbound/zonefiles/allow.rpz
>>>>> +    rpz-action-override: passthru
>>>>> +    rpz-log: yes
>>>>> +    rpz-log-name: allow
>>>>> +    rpz-signal-nxdomain-ra: yes
>>>>> +
>>>>> +rpz:
>>>>> +    name: block.rpz
>>>>> +    zonefile: /etc/unbound/zonefiles/block.rpz
>>>>> +    rpz-action-override: nxdomain
>>>>> +    rpz-log: yes
>>>>> +    rpz-log-name: block
>>>>> +    rpz-signal-nxdomain-ra: yes
>>>>> diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config
>>>>> new file mode 100644
>>>>> index 000000000..98dc0a4ca
>>>>> --- /dev/null
>>>>> +++ b/config/rpz/rpz-config
>>>>> @@ -0,0 +1,194 @@
>>>>> +#!/bin/bash
>>>>> +###############################################################################
>>>>> +#                                                                             #
>>>>> +#  IPFire.org - A linux based firewall                                        #
>>>>> +#  Copyright (C) 2024  IPFire Team  <info@ipfire.org>                         #
>>>>> +#                                                                             #
>>>>> +#  This program is free software: you can redistribute it and/or modify       #
>>>>> +#  it under the terms of the GNU General Public License as published by       #
>>>>> +#  the Free Software Foundation, either version 3 of the License, or          #
>>>>> +#  (at your option) any later version.                                        #
>>>>> +#                                                                             #
>>>>> +#  This program is distributed in the hope that it will be useful,            #
>>>>> +#  but WITHOUT ANY WARRANTY; without even the implied warranty of             #
>>>>> +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the              #
>>>>> +#  GNU General Public License for more details.                               #
>>>>> +#                                                                             #
>>>>> +#  You should have received a copy of the GNU General Public License          #
>>>>> +#  along with this program.  If not, see <http://www.gnu.org/licenses/>.      #
>>>>> +#                                                                             #
>>>>> +###############################################################################
>>>>> +
>>>>> +# v22 - 2024-07-12
>>>>> +
>>>>> +###############     Functions     ###############
>>>>> +
>>>>> +msg_log () {
>>>>> + /usr/bin/logger --tag "${tagName}" "$*"
>>>>> + if tty --silent ; then
>>>>> + echo "${tagName}:" "$*"
>>>>> + fi
>>>>> +}
>>>>> +
>>>>> +check_name () {
>>>>> + local testName="${1}"
>>>>> + # check for a valid name
>>>>> + regex='^[a-zA-Z0-9_]+$'
Is there a specific reason why you have chosen only letters, numbers and 
underscore and not also the dash. If you allowed that then you could use 
the validdomainname subroutine in the general-functions.pl file rather 
than creating another valid name function.
Does unbound not accept dash's in the rpz config filename?
>>>>> + if [[ ! "${testName}" =~ $regex ]] ; then
>>>>> + msg_log "error: rpz: the NAME is not valid: \"${testName}\". exit."
>>>>> + exit 1
>>>>> + fi
>>>>> +}
>>>>> +
>>>>> +check_unbound_conf () {
>>>>> + # check the above config files
>>>>> + msg_log "info: rpz: check for errors with \"unbound-checkconf\""
>>>>> + /usr/sbin/unbound-checkconf
I think checkconf should be added into the unboundctrl.c file
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/misc-progs/unboundctrl.c;h=86c6ac42b9010f9d77fa49754bf217875aed3cc0;hb=refs/heads/next
so that the command can be ensured to be running in safe mode.
>>>>> + exit_code=$?
>>>>> + if [[ "${exit_code}" -ne 0 ]] ; then
>>>>> + msg_log "error: rpz: unbound-checkconf. exit."
>>>>> + exit "${exit_code}"
>>>>> + fi
>>>>> +}
>>>>> +
>>>>> +make_rpz_file () {
>>>>> + local theType="${1}" # allow or block
>>>>> +
>>>>> + theList="/var/ipfire/rpz/${theType}list" # input user list of domains
>>>>> + theZoneFile="/etc/unbound/zonefiles/${theType}.rpz" # output file for RPZ
>>>>> +
>>>>> + theAction='.'
>>>>> + if [[ "${theType}" =~ "block" ]] ; then
>>>>> + theAction='rpz-passthru.'
>>>>> + fi
>>>>> +
>>>>> + # does a list exist?
>>>>> + if [[ -s "${theList}" ]] ; then
>>>>> + # drop any extra "blanks" and add "CNAME <RPZ action>." to each line
>>>>> + actionList=$( /usr/bin/awk '{$1=$1};1' "${theList}" |
>>>>> + /bin/sed "/^[^;].*[[:alnum:]]/ s|$| CNAME ${theAction}|" )
>>>>> +
>>>>> + msg_log "info: rpz: create zonefile for ${theList}"
>>>>> +
>>>>> + /bin/cat <<-EOF > "${theZoneFile}"
>>>>> + ; Name: ${theType} list
>>>>> + ; Last modified: $(date "+%Y-%m-%d at %H.%M.%S %Z")
>>>>> + ;
>>>>> + ; domains with actions list
>>>>> + ;
>>>>> + ${actionList}
>>>>> + EOF
>>>>> +
>>>>> + # reload the zone that was just updated
>>>>> + zoneBase=$( basename "${theZoneFile}" )
>>>>> + /usr/sbin/unbound-control auth_zone_reload -q "${zoneBase}"
This should also be run via the unboundctrl.c file especially as the 
unbound-control command is accessing remote systems.
>>>>> + fi
>>>>> +}
>>>>> +
>>>>> +###############       Main        ###############
>>>>> +
>>>>> +tagName="unbound"
>>>>> +
>>>>> +theAction="${1}" # input action
>>>>> +theName="${2}" # input RPZ name
>>>>> +theURL="${3}" # input RPZ URL
>>>>> +
>>>>> +check_name "${theName}" # is this a valid name?
>>>>> +
>>>>> +rpzConfig="/etc/unbound/local.d/${theName}.rpz.conf" # output zone conf file
>>>>> +rpzFile="/etc/unbound/zonefiles/${theName}.rpz" # output for RPZ file
>>>>> +
>>>>> +case "${theAction}" in
>>>>> +
>>>>> + # add new rpz list
>>>>> + add )
>>>>> + # does this config already exist?  If yes, then exit
>>>>> + if [[ -f "${rpzConfig}" ]] ; then
>>>>> + msg_log "info: rpz: ${rpzConfig} already exists. exit"
>>>>> + exit 1
>>>>> + fi
>>>>> +
>>>>> + # is this a valid URL?
>>>>> + regex='^https://[-[:alnum:]\+&@#/%?=~_|!:,.;]*[-[:alnum:]\+&@#/%=~_|]'
>>>>> + if ! [[ "${theURL}" =~ $regex ]] ; then
>>>>> + msg_log "error: rpz: the URL is not valid: \"${theURL}\". exit."
>>>>> + exit 1
>>>>> + fi
>>>>> +
>>>>> + # create the zone config file
>>>>> + msg_log "info: rpz: add config file \"${theName}.rpz.conf\""
>>>>> + cat <<-EOF > "${rpzConfig}"
>>>>> + rpz:
>>>>> + name: ${theName}.rpz
>>>>> + zonefile: /etc/unbound/zonefiles/${theName}.rpz
>>>>> + url:  ${theURL}
>>>>> + rpz-action-override: nxdomain
>>>>> + rpz-log: yes
>>>>> + rpz-log-name: ${theName}
>>>>> + rpz-signal-nxdomain-ra: yes
>>>>> + EOF
>>>>> +
>>>>> + # set-up zone file
>>>>> + /usr/bin/touch "${rpzFile}"
>>>>> + # unbound requires these settings for rpz files
>>>>> + /bin/chown --verbose nobody:nobody "${rpzFile}"
>>>>> + /bin/chmod --verbose 644 "${rpzFile}"
These should only be using --verbose while doing development. In 
production they should not be used as the development should have 
ensured that everything is working as required.
>>>>> + ;;
>>>>> +
>>>>> + # trash config file & rpz file
>>>>> + remove )
>>>>> + if ! [[ -f "${rpzConfig}" ]] ; then
>>>>> + msg_log "info: rpz: ${rpzConfig} does not exist. exit"
>>>>> + exit 1
>>>>> + fi
>>>>> +
>>>>> + msg_log "info: rpz: remove config file & rpz file \"${theName}\""
>>>>> + /bin/rm --verbose "${rpzConfig}"
>>>>> + /bin/rm --verbose "${rpzFile}"
Same thing with the --verbose.
>>>>> +
>>>>> + check_unbound_conf
>>>>> + ;;
>>>>> +
>>>>> + # make a new allow or block rpz file
>>>>> + make )
>>>>> + case "${theName}" in
>>>>> + allow )
>>>>> + make_rpz_file allow
>>>>> + ;;
>>>>> +
>>>>> + block )
>>>>> + make_rpz_file block
>>>>> + ;;
>>>>> +
>>>>> + allowblock )
>>>>> + make_rpz_file allow
>>>>> + make_rpz_file block
>>>>> + ;;
>>>>> +
>>>>> + * )
>>>>> + msg_log "error: rpz: the NAME is not valid: \"${theName}\". exit."
>>>>> + exit 1
>>>>> + ;;
>>>>> + esac
>>>>> +
>>>>> + check_unbound_conf
>>>>> + ;;
>>>>> +
>>>>> + *)
>>>>> + msg_log "error: rpz: missing or incorrect parameter"
>>>>> + /usr/bin/printf "Usage:  rpzConfig.sh <ACTION> <NAME> <URL>\n"
>>>>> + exit 1
>>>>> + ;;
>>>>> +
>>>>> +esac
>>>>> +
>>>>> +# reload due to the changes
>>>>> +msg_log  "rpz: running \"unbound-control reload\""
>>>>> +/usr/sbin/unbound-control reload
Use unboundctrl
>>>>> +exit_code=$?
>>>>> +if [[ "${exit_code}" -ne 0 ]] ; then
>>>>> + msg_log "error: rpz: unbound-control \"${theName}\". exit."
>>>>> + exit "${exit_code}"
>>>>> +fi
>>>>> +
>>>>> +exit
This is a hell of a long bash script. Did you consider doing it in Perl 
or python instead of bash. Maybe it would be better in c even as then it 
would be compiled and could not be modified on the IPFire system if some 
malware got in.
>>>>> diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics
>>>>> new file mode 100644
>>>>> index 000000000..0f97c7911
>>>>> --- /dev/null
>>>>> +++ b/config/rpz/rpz-metrics
>>>>> @@ -0,0 +1,143 @@
>>>>> +#!/bin/bash
>>>>> +###############################################################################
>>>>> +#                                                                             #
>>>>> +#  IPFire.org - A linux based firewall                                        #
>>>>> +#  Copyright (C) 2024  IPFire Team  <info@ipfire.org>                         #
>>>>> +#                                                                             #
>>>>> +#  This program is free software: you can redistribute it and/or modify       #
>>>>> +#  it under the terms of the GNU General Public License as published by       #
>>>>> +#  the Free Software Foundation, either version 3 of the License, or          #
>>>>> +#  (at your option) any later version.                                        #
>>>>> +#                                                                             #
>>>>> +#  This program is distributed in the hope that it will be useful,            #
>>>>> +#  but WITHOUT ANY WARRANTY; without even the implied warranty of             #
>>>>> +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the              #
>>>>> +#  GNU General Public License for more details.                               #
>>>>> +#                                                                             #
>>>>> +#  You should have received a copy of the GNU General Public License          #
>>>>> +#  along with this program.  If not, see <http://www.gnu.org/licenses/>.      #
>>>>> +#                                                                             #
>>>>> +###############################################################################
>>>>> +
>>>>> +# v18 on 2024-07-05
>>>>> +
>>>>> +###############       Main        ###############
>>>>> +
>>>>> +weeks="${1:-2}" # default to two message logs
>>>>> +sortBy="${2:-name}"   # by name or by hits
>>>>> +
>>>>> +# get the list of message logs for N weeks
>>>>> +messageLogs=$( find /var/log/messages* -type f |
>>>>> + /usr/bin/sort --version-sort |
>>>>> + head -"${weeks}" )
>>>>> +
>>>>> +# get the list of RPZ names & counts from the message log(s)
>>>>> +rpzNameCount=$( for logf in ${messageLogs} ; do
>>>>> + /usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" |
>>>>> + /usr/bin/awk '$10 ~ /\[\w*]/ { print $10 }' ;
>>>>> + done | /usr/bin/sort | /usr/bin/uniq --count )
>>>>> +
>>>>> +# flip results and remove brackets `[` and `]`
>>>>> +rpzNameCount=$( /bin/echo "${rpzNameCount}" |
>>>>> + /usr/bin/awk '{ print $2, $1 }' |
>>>>> + /bin/sed --regexp-extended 's|^\[(.*)\]|\1|' )
>>>>> +
>>>>> +# grab only names
>>>>> +rpzNames=$( /bin/echo "${rpzNameCount}" | /usr/bin/awk '{ print $1 }' )
>>>>> +
>>>>> +# get list of RPZ files
>>>>> +rpzFileList=$( /bin/find /etc/unbound/zonefiles -type f -iname "*.rpz" )
>>>>> +
>>>>> +# get basename of those files
>>>>> +rpzBaseNames=$( /bin/echo "${rpzFileList}" |
>>>>> + /bin/sed 's|/etc/unbound/zonefiles/||g ; s|\.rpz||g ;' )
>>>>> +
>>>>> +# add to rpzNames
>>>>> +rpzNames="${rpzNames}"$'\n'"${rpzBaseNames}"
>>>>> +
>>>>> +# drop duplicate names
>>>>> +rpzNames=$( echo "${rpzNames}" | /usr/bin/sort --unique  )
>>>>> +
>>>>> +# get line count for each RPZ
>>>>> +lineCount=$( /bin/echo "${rpzFileList}" | /usr/bin/xargs wc -l )
>>>>> +
>>>>> +# get comment line count and blank line count for each RPZ
>>>>> +commentCount=$( /bin/echo "${rpzFileList}" |
>>>>> + /usr/bin/xargs /bin/grep --count -e "^$" -e "^;" )
>>>>> +
>>>>> +# get modified date each RPZ
>>>>> +modDateList=$( /bin/echo "${rpzFileList}" | xargs stat -c '%.10y  %n' )
>>>>> +
>>>>> +ucListAuthZones=$( /usr/sbin/unbound-control list_auth_zones )
>>>>> +
>>>>> +# get width of RPZ names
>>>>> +pWidth=$( /bin/echo "${rpzNames}" | /usr/bin/awk '{ print $1"   " }' | wc -L )
>>>>> +pFormat="%-${pWidth}s %-8s %-8s %-8s %10s %12s\n"
>>>>> +
>>>>> +# print title line
>>>>> +printf "${pFormat}" "name" "hits" "active" "lines" "hits/line%" "last update"
>>>>> +printf -- "--------------"
>>>>> +
>>>>> +theResults=""
>>>>> +totalLines=0
>>>>> +totalHits=0
>>>>> +while read -r theName
>>>>> +do
>>>>> + printf -- "-" # pretend progress bar
>>>>> + # get hit count
>>>>> + theHits="0"
>>>>> + if output=$( /bin/grep "^${theName}\s" <<< "${rpzNameCount}" ) ; then
>>>>> + theHits=$( /bin/echo "${output}" |
>>>>> + /usr/bin/awk '{ print $2 }' )
>>>>> + totalHits=$(( totalHits + theHits ))
>>>>> + fi
>>>>> +
>>>>> + # is this RPZ list active?
>>>>> + theActive="disabled"
>>>>> + if /bin/grep --quiet "^${theName}\.rpz" <<< "${ucListAuthZones}"
>>>>> + then
>>>>> + theActive="enabled"
>>>>> + fi
>>>>> +
>>>>> + # get line count then subtract comment count and blank line count
>>>>> + #  from total line count
>>>>> + theLines="n/a"
>>>>> + hitsPerLine="0"
>>>>> + if output=$( /bin/grep --fixed-strings "/${theName}.rpz" <<< "${lineCount}" ) ; then
>>>>> + theLines=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' )
>>>>> + totalLines=$(( totalLines + theLines ))
>>>>> +
>>>>> + #hitsPerLine=$( echo "scale=0 ; $theHits / $theLines" | bc )
>>>>> + hitsPerLine=$(( 100 * theHits / theLines ))
>>>>> + fi
>>>>> +
>>>>> + # get modification date
>>>>> + theModDate="n/a"
>>>>> + if output=$( /bin/grep --fixed-strings "/${theName}.rpz" <<< "${modDateList}" ) ; then
>>>>> + theModDate=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' )
>>>>> + fi
>>>>> +
>>>>> + # add to results list
>>>>> + theResults+="${theName} ${theHits} ${theActive} ${theLines} ${hitsPerLine} ${theModDate}"$'\n'
>>>>> +done <<< "${rpzNames}"
>>>>> +
>>>>> +case "${sortBy}" in
>>>>> + names|name) sortArg=(-k3,3r -k1,1) ;; # sort by "active" then by "name"
>>>>> +
>>>>> + hits|hit) sortArg=(-k3,3r -k2,2nr -k1,1) ;; # sort by "active" then by "hits" then by "name"
>>>>> +
>>>>> + lines|line) sortArg=(-k3,3r -k4,4nr -k1,1) ;; # sort by "active" then by "lines" then by "name"
>>>>> +esac
>>>>> +
>>>>> +printf -- "--------------\n"
>>>>> +# remove blank lines, sort, print as columns
>>>>> +/bin/echo "${theResults}" |
>>>>> + /usr/bin/awk '!/^[[:space:]]*$/' |
>>>>> + /usr/bin/sort "${sortArg[@]}"    |
>>>>> + /usr/bin/awk --assign=width="${pWidth}" \
>>>>> + '{ printf "%-*s %-8s %-8s %-8s %10s %12s\n", width, $1, $2, $3, $4, $5, $6 }'
>>>>> +
>>>>> +printf "${pFormat}" "" "=======" "" "========" "" ""
>>>>> +printf "${pFormat}" "Totals -->" "${totalHits}" "" "${totalLines}" "" ""
>>>>> +
>>>>> +exit
If I understand right the rpz-metrics info is being shown on a weekly 
basis. Wouldn't this make more sense to create a log page for 
unbound-rpz that shows the results in a similar way to the Firewall 
Logs, IP Address Blocklist Logs etc. So you can see for a whole month or 
a specific day and you can also export the data into a text format on a 
page of the browser. Basically that would just be copying one of the 
existing cgi scripts and modifying the content elements.
>>>>> diff --git a/config/rpz/rpz-sleep b/config/rpz/rpz-sleep
>>>>> new file mode 100644
>>>>> index 000000000..eeef1174a
>>>>> --- /dev/null
>>>>> +++ b/config/rpz/rpz-sleep
>>>>> @@ -0,0 +1,58 @@
>>>>> +#!/bin/bash
>>>>> +###############################################################################
>>>>> +#                                                                             #
>>>>> +#  IPFire.org - A linux based firewall                                        #
>>>>> +#  Copyright (C) 2024  IPFire Team  <info@ipfire.org>                         #
>>>>> +#                                                                             #
>>>>> +#  This program is free software: you can redistribute it and/or modify       #
>>>>> +#  it under the terms of the GNU General Public License as published by       #
>>>>> +#  the Free Software Foundation, either version 3 of the License, or          #
>>>>> +#  (at your option) any later version.                                        #
>>>>> +#                                                                             #
>>>>> +#  This program is distributed in the hope that it will be useful,            #
>>>>> +#  but WITHOUT ANY WARRANTY; without even the implied warranty of             #
>>>>> +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the              #
>>>>> +#  GNU General Public License for more details.                               #
>>>>> +#                                                                             #
>>>>> +#  You should have received a copy of the GNU General Public License          #
>>>>> +#  along with this program.  If not, see <http://www.gnu.org/licenses/>.      #
>>>>> +#                                                                             #
>>>>> +###############################################################################
>>>>> +
>>>>> +# v04 on 2024-07-05
>>>>> +
>>>>> +###############     Functions     ###############
>>>>> +
>>>>> +# send message to message log
>>>>> +msg_log () {
>>>>> + /usr/bin/logger --tag "${tagName}" "$*"
>>>>> + if /usr/bin/tty --silent ; then
>>>>> + echo "${tagName}:" "$*"
>>>>> + fi
>>>>> +}
>>>>> +
>>>>> +###############       Main        ###############
>>>>> +
>>>>> +tagName="unbound"
>>>>> +
>>>>> +sleepTime="${1:-5m}" # default to sleep for 5m (5 minutes)
>>>>> +
>>>>> +zoneList=$( /usr/sbin/unbound-control list_auth_zones | /usr/bin/awk '{print $1}' )
>>>>> +
>>>>> +for zone in ${zoneList} ; do
>>>>> + /usr/bin/printf "disable ${zone}\t"
>>>>> + /usr/sbin/unbound-control rpz_disable "${zone}"
unboundctrl
>>>>> +done
>>>>> +
>>>>> +msg_log "info: rpz: disabled all zones for ${sleepTime}"
>>>>> +
>>>>> +/bin/sleep "${sleepTime}"
>>>>> +
>>>>> +for zone in ${zoneList} ; do
>>>>> + /usr/bin/printf "enable ${zone}\t"
>>>>> + /usr/sbin/unbound-control rpz_enable "${zone}"
unboundctrl
>>>>> +done
>>>>> +
>>>>> +msg_log "info: rpz: enabled all zones"
>>>>> +
>>>>> +exit
>>>>> diff --git a/lfs/rpz b/lfs/rpz
>>>>> new file mode 100644
>>>>> index 000000000..319c10b7f
>>>>> --- /dev/null
>>>>> +++ b/lfs/rpz
>>>>> @@ -0,0 +1,88 @@
>>>>> +###############################################################################
>>>>> +#                                                                             #
>>>>> +# IPFire.org - A linux based firewall                                         #
>>>>> +# Copyright (C) 2024  IPFire Team  <info@ipfire.org>                          #
>>>>> +#                                                                             #
>>>>> +# This program is free software: you can redistribute it and/or modify        #
>>>>> +# it under the terms of the GNU General Public License as published by        #
>>>>> +# the Free Software Foundation, either version 3 of the License, or           #
>>>>> +# (at your option) any later version.                                         #
>>>>> +#                                                                             #
>>>>> +# This program is distributed in the hope that it will be useful,             #
>>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of              #
>>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the               #
>>>>> +# GNU General Public License for more details.                                #
>>>>> +#                                                                             #
>>>>> +# You should have received a copy of the GNU General Public License           #
>>>>> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.       #
>>>>> +#                                                                             #
>>>>> +###############################################################################
>>>>> +
>>>>> +###############################################################################
>>>>> +# Definitions
>>>>> +###############################################################################
>>>>> +
>>>>> +include Config
>>>>> +
>>>>> +SUMMARY    = response policy zone - RPZ reputation system for unbound DNS
>>>>> +
>>>>> +VER        = 1.0.0
>>>>> +
>>>>> +THISAPP    = rpz-$(VER)
>>>>> +DIR_APP    = $(DIR_SRC)/$(THISAPP)
>>>>> +TARGET     = $(DIR_INFO)/$(THISAPP)
>>>>> +
>>>>> +PROG       = rpz
>>>>> +PAK_VER    = 1
>>>>> +
>>>>> +DEPS       =
>>>>> +
>>>>> +SERVICES   =
>>>>> +
>>>>> +###############################################################################
>>>>> +# Top-level Rules
>>>>> +###############################################################################
>>>>> +
>>>>> +install : $(TARGET)
>>>>> +
>>>>> +check :
>>>>> +
>>>>> +download :
>>>>> +
>>>>> +b2 :
>>>>> +
>>>>> +dist:
>>>>> + @$(PAK)
>>>>> +
>>>>> +###############################################################################
>>>>> +# Installation Details
>>>>> +###############################################################################
>>>>> +
>>>>> +$(TARGET) :
>>>>> + @$(PREBUILD)
>>>>> + @rm -rf $(DIR_APP)
>>>>> +
>>>>> + # install RPZ scripts
>>>>> + install -v -m 755 \
>>>>> +  $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin
>>>>> +
>>>>> + # Install settings folder and two empty files
>>>>> + mkdir -pv /var/ipfire/rpz
>>>>> + touch /var/ipfire/rpz/allowlist
>>>>> + touch /var/ipfire/rpz/blocklist
>>>>> +
>>>>> + # Add conf file to /etc directory
>>>>> + cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d
>>>>> +
>>>>> + # create zonefiles directory for the RPZ files and add two empty RPZ
>>>>> + #  files to avoid a unbound config error
>>>>> + mkdir -pv /etc/unbound/zonefiles
>>>>> + chown -v nobody:nobody /etc/unbound/zonefiles
>>>>> + touch /etc/unbound/zonefiles/allow.rpz
>>>>> + touch /etc/unbound/zonefiles/block.rpz
>>>>> +
>>>>> + # Install backup definition
>>>>> + cp -vf $(DIR_CONF)/backup/includes/rpz /var/ipfire/backup/addons/includes/rpz
>>>>> +
>>>>> + @rm -rf $(DIR_APP)
>>>>> + @$(POSTBUILD)
>>>>> diff --git a/make.sh b/make.sh
>>>>> index 9bbbeb0f1..886d3760a 100755
>>>>> --- a/make.sh
>>>>> +++ b/make.sh
>>>>> @@ -1721,6 +1721,8 @@ buildipfire() {
>>>>> lfsmake2 btrfs-progs
>>>>> lfsmake2 inotify-tools
>>>>> lfsmake2 grub-btrfs
>>>>> +  lfsmake2 rpz
>>>>> +
>>>>>
>>>>> # Kernelbuild ... current we have no platform that need
>>>>> # multi kernel builds so KCFG is empty
>>>>> -- 
>>>>> 2.30.2
>>>>>

Jon Murphy July 30, 2024, 4:32 p.m. UTC | #6

Comments below...

> On Jul 30, 2024, at 5:05 AM, Adolf Belka <adolf.belka@ipfire.org> wrote:
> 
> Hi Jon,
> I have given some feedback in your patches below.
> 
> On 29/07/2024 04:28, jon wrote:
>> See comments below...
>> 
>>> On Jul 28, 2024, at 6:22 AM, Michael Tremer <michael.tremer@ipfire.org> wrote:
>>> 
>>> Hello Jon,
>>> 
>>> I don’t think it is a good idea to send over the same patch set multiple times.
>>> 
>>> I think first of all, we need to start at the beginning which is to figure out what this actually is, and why we would want this in IPFire.
>>> 
>>> I have seen that there have been conversations on community.ipfire.org <http://community.ipfire.org/> many months ago, but I did not take many things away from it. It was from my point of view mostly a prove of concept discussion. Correct me if I have missed anything here.
>>> 
>>> We have talked about RPZ many times on the monthly call since the URL filter feature is falling more and more out of fashion. I think there is also many posts about this on the forum.
>> Yes, I have seen the posts about the url filter feature falling out of fashion.  I did not see notes about RPZ from the monthly calls.  I will go back and search telco call notes.
>> 
>> 
>>> Apart from generally being in favour of trying this path, we decided to postpone any further works into this because we currently have a huge backlog of things to deal with  that are moving at snail speed. This is rooted in so many people being simply tied in with other things.
>>> 
>>> As you can see, the patch set that you posted did not trigger a response - a common thing on this list recently and not because of this particular patch set.
>>> 
>>> Therefore I would like to avoid opening another can of worms before we have delivered the other things that have already lots of time spent on them and code written. Otherwise we are only getting a longer pipeline and this all is going to be more frustrating for everyone involved. And this is not about the feature per se; this is about getting things done.
>> This is why I wrote the RPZ patch instead of asking for the Developers to write it.  I realize there is a very long pipeline.
>> 
>> 
>>> So what I suggest doing is the following:
>>> 
>>> You have a Git repository, push the code there and commit any fixes to it over time so that nothing is getting lost.
>> This was done a few days ago.  But I am not sure if I did it correctly.  If not, please let me know.
>> 
>> 
>>> Explain to me from the beginning, please, how it came to this patch set. What is the motivation?
>> In the beginning (Fall of 2023) the main motivation was blocking DoH.  I had noticed apps using their own method of DNS lookup.  Some were friendly DoH lookups (Apple iCloud?) and some not (mozilla.cloudflare-dns.com but not from Firefox).  Examples I saw are:
>> mozilla.cloudflare-dns.com <http://mozilla.cloudflare-dns.com/>
>> www.switch.ch <http://www.switch.ch/>
>> use-application-dns.net <http://use-application-dns.net/> (a canary for Firefox?)
>> And a few others...
>> 
>> This worked better than expected and was easier to configure than expected.  All of the main code was part of unbound and I just need to activate it with a config file.  Updates to the RPZ files are already part of the unbound RPZ code.
>> 
>> As I experimented with RPZ I quickly realized it could easily be used to block other websites.  I have been using pihole for a number of years to limit the number of ads and trackers.  And RPZ helped with those areas also.
>> 
>> 
>>> What problems does it solve? What problems remain? Basically: Why would anyone want this in IPFire?
>> It solved the issue of blocking unwanted DoH from apps.  And it solved the issue of unwanted ads and tracking.  For me those are huge benefits and the biggest reason for using RPZ.
>> 
>> To test other possibilities, I increased the "type" of lists to include other categories other than ads & trackers.  Like blocking bad TLDs, gambling, piracy, bad host sites, etc.  This is a nice bonus of using RPZ for me.
>> 
>>> Where do you see application for this?
>> As a replacement for the URL Filter.  And as a base functional replacement for pihole (but no pretty graphics).
>> 
>> 
>>> Why is this realised as an add-on and not part of the core system?
>> In my opinion it should be part of the core.  But I wanted to "test the waters" first to determine interest.  If only four people use it, then it isn’t worth adding to core and maybe not as an add-on.
>> 
>> Either way I will support it because it helps me!
>> 
>> Here are my metrics for the past eight days (searching `messages` and `messages.1.gz`):
>> 
>> [root@ipfire ~] # rpz-metrics
>> name               hits
>> ----------------------------
>> allow              5560    This is a custom list
>> AmazonTrkrHZ       5726
>> AppleTrkrHZ        2956
>> block              128     This is a custom list
>> DOHblockHZ         25077
>> dohJPG             3448
>> HosterHZ           19
>> MxProPlusHZ        17287
>> NotSafeSearchHZ    0
>> ProxyBypassHZ      11
>> tldHZ              15
>> WinTrkrHZ          507
>>                    =======
>> Totals -->         60734
>> 
>> For the DoH blocking, the above "hits" including blocking Apple DoH.  I’ve been testing this for the past few weeks.
>> 
>>> I am particularly interested in what lists are publicly available. You know from URL Filter, that lists have shut down or just been abandoned. There is now very little point running it because the lists are just not as good as they used to be. Not horrible, yet, but over time this will not be getting any better. I am not even aware of any viable paid lists. Is this the same situation for RPZ?  A quick Google search did not give me anything that would be free to use over DNS. The situation is somewhat similar with the IPS and IP blocklist feature.
>> These are free/open lists:
>> 
>> Hagezi - DNS Blocklists
>> • at https://github.com/hagezi/dns-blocklists?tab=readme-ov-file#zap-dns-blocklists---for-a-better-internet
>> • This list was recommended within the IPFire Community and this is the list I strongly recommend.  Gerd (Hagezi) is very responsive to support requests. The Hagezi RPZ list is well supported and the RPZ lists are updated twice per day (if changes are needed).
>> • Example: I came across a domain name that was blocked `f005.backblazeb2.com` and it was fixed (removed from the block list) within a few days.
>> 
>> ThreatFox - DNS Response Policy Zone (RPZ)
>> • at https://threatfox.abuse.ch/export/#rpz
>> 
>> URLHaus - DNS Response Policy Zone (RPZ)
>> • at https://urlhaus.abuse.ch/api/#rpz
>> 
>> jpgpi250 - DNS block list for DoH
>> • at https://github.com/jpgpi250/piholemanual/blob/master/DOH.rpz
>> • this is the original DoH block list that I started with.  Peter (jpgpi250) has a nice PDF document explaining RPZ and was very helpful getting me started with RPZ.  See https://jpgpi250.github.io/piholemanual/doc/Unbound%20response%20policy%20zones.pdf
>> 
>> I have come across other RPZ lists but these are the best supported lists.
>> 
>> I did not look into paid lists.
>> 
>>> 
>>> I would be interested in hearing your thoughts on this.
>> Does the above help?
>> 
>>> Best,
>>> -Michael
>>> 
>>>> On 27 Jul 2024, at 21:31, jon <jon.murphy@ipfire.org> wrote:
>>>> 
>>>> And another question.
>>>> 
>>>> I found an error in the `rpz-config` script file.  Simple easy one line (actually one word) fix.
>>>> 
>>>> Do I resubmit the entire patch?
>>>> -or-
>>>> Do I wait until approved and then submit the simple patch for the one line change.
>>>> 
>>>> 
>>>> Jon
>>>> 
>>>> 
>>>>> On Jul 20, 2024, at 1:42 PM, jon <jon.murphy@ipfire.org> wrote:
>>>>> 
>>>>> I need help with…
>>>>> 
>>>>> 1) The custom allow and block lists are currently located at `/var/ipfire/rpz`.  Is this correct?
>>>>> 
>>>>> 2) The three bash scripts are currently located at `/usr/sbin`.  Is this correct?
>>>>> 
>>>>> 3) Should this commit be separated into multiple posts?  If yes, how do I decide how to split it?
>>>>> 
>>>>> 
>>>>> Jon
>>>>> 
>>>>> 
>>>>>> On Jul 20, 2024, at 12:46 PM, Jon Murphy <jon.murphy@ipfire.org> wrote:
>>>>>> 
>>>>>> What is it?
>>>>>> Response Policy Zone (RPZ) is a mechanism to define local policies in a standardised
>>>>>> way and load those policies from external sources.
>>>>>> Bottom line: RPZ allows admins to easily block access to websites via DNS lookup.
>>>>>> 
>>>>>> RPZ can block websites via categories.  Examples include: fake websites, annoying
>>>>>> pop-up ads, newly registered domains, DoH bypass sites, bad "host" services,
>>>>>> maliscious top level domains (e.g., *.zip, *.mov), piracy, gambling, pornography,
>>>>>> and more.  RPZ lists come from various RPZ providers and their available
>>>>>> catagories.
>>>>>> 
>>>>>> This RPZ add-on enables the RPZ functionality by adding a couple lines in a
>>>>>> configuration file.  This add-on simply adds a configuration file and adds
>>>>>> three scripts (config, metrics and sleep) to make RPZ easier for the admin to use.
>>>>>> 
>>>>>> RPZ was release in 2010 and has been part of the IPFire build since ~2015.
>>>>>> 
>>>>>> Why is it needed?
>>>>>> Some IPFire admin's utilize pihole to block unwanted websites via DNS lookup.
>>>>>> Moving the pihole base functionality (without pretty graphs) to IPFire removes
>>>>>> one device from the admin's local network.
>>>>>> And hopefully this reduces the pihole questions from the Community.
>>>>>> 
>>>>>> A list of RPZ providers can be recommended by IPFire and coded into a set list.
>>>>>> Or, if prefered, the local admin can choose their own RPZ providers.
>>>>>> 
>>>>>> This is a:
>>>>>> * simple replacement for pihole base functionality
>>>>>> * RPZ can be a nice replacement for the URL Filter
>>>>>> 
>>>>>> IPFire Wiki
>>>>>> In process at: https://www.ipfire.org/docs/addons/rpz
>>>>>> 
>>>>>> more info:
>>>>>> * https://en.wikipedia.org/wiki/Response_policy_zone
>>>>>> * https://unbound.docs.nlnetlabs.nl/en/latest/topics/filtering/rpz.html
>>>>>> 
>>>>>> I need help with...
>>>>>> 1) The custom allow and block lists are currently located at `/var/ipfire/rpz`.  Is this correct?
> rpz is supposed to be part of the dns so I would either put all the files in /var/ipfire/dns/ or create a directory for rpz under the dns directory.

Yes, I can do this.  I’ll go with `/var/ipfire/dns/rpz` if OK.

>>>>>> 2) The three bash scripts are currently located at `/usr/sbin`.  Is this correct?
>>>>>> 
>>>>>> Signed-off-by: Jon Murphy <jon.murphy@ipfire.org>
>>>>>> ---
>>>>>> config/backup/includes/rpz    |   5 +
>>>>>> config/rootfiles/packages/rpz |  11 ++
>>>>>> config/rpz/00-rpz.conf        |  18 ++++
>>>>>> config/rpz/rpz-config         | 194 ++++++++++++++++++++++++++++++++++
>>>>>> config/rpz/rpz-metrics        | 143 +++++++++++++++++++++++++
>>>>>> config/rpz/rpz-sleep          |  58 ++++++++++
>>>>>> lfs/rpz                       |  88 +++++++++++++++
>>>>>> make.sh                       |   2 +
>>>>>> 8 files changed, 519 insertions(+)
>>>>>> create mode 100644 config/backup/includes/rpz
>>>>>> create mode 100644 config/rootfiles/packages/rpz
>>>>>> create mode 100644 config/rpz/00-rpz.conf
>>>>>> create mode 100644 config/rpz/rpz-config
>>>>>> create mode 100644 config/rpz/rpz-metrics
>>>>>> create mode 100644 config/rpz/rpz-sleep
>>>>>> create mode 100644 lfs/rpz
>>>>>> 
>>>>>> diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz
>>>>>> new file mode 100644
>>>>>> index 000000000..4d59bb40c
>>>>>> --- /dev/null
>>>>>> +++ b/config/backup/includes/rpz
>>>>>> @@ -0,0 +1,5 @@
>>>>>> +/var/ipfire/rpz/allowlist
>>>>>> +/var/ipfire/rpz/blocklist
>>>>>> +/etc/unbound/zonefiles/allow.rpz
>>>>>> +/etc/unbound/zonefiles/block.rpz
>>>>>> +/etc/unbound/local.d/*rpz.conf
>>>>>> diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/rpz
>>>>>> new file mode 100644
>>>>>> index 000000000..2ffa715dd
>>>>>> --- /dev/null
>>>>>> +++ b/config/rootfiles/packages/rpz
>>>>>> @@ -0,0 +1,11 @@
>>>>>> +etc/unbound/local.d/00-rpz.conf
>>>>>> +etc/unbound/zonefiles
>>>>>> +etc/unbound/zonefiles/allow.rpz
>>>>>> +etc/unbound/zonefiles/block.rpz
>>>>>> +usr/sbin/rpz-config
>>>>>> +usr/sbin/rpz-metrics
>>>>>> +usr/sbin/rpz-sleep
>>>>>> +var/ipfire/backup/addons/includes/rpz
>>>>>> +var/ipfire/rpz
>>>>>> +var/ipfire/rpz/allowlist
>>>>>> +var/ipfire/rpz/blocklist
>>>>>> diff --git a/config/rpz/00-rpz.conf b/config/rpz/00-rpz.conf
>>>>>> new file mode 100644
>>>>>> index 000000000..72c1d12e5
>>>>>> --- /dev/null
>>>>>> +++ b/config/rpz/00-rpz.conf
>>>>>> @@ -0,0 +1,18 @@
>>>>>> +server:
>>>>>> + module-config: "respip validator iterator"
>>>>>> +
>>>>>> +rpz:
>>>>>> +    name: allow.rpz
>>>>>> +    zonefile: /etc/unbound/zonefiles/allow.rpz
>>>>>> +    rpz-action-override: passthru
>>>>>> +    rpz-log: yes
>>>>>> +    rpz-log-name: allow
>>>>>> +    rpz-signal-nxdomain-ra: yes
>>>>>> +
>>>>>> +rpz:
>>>>>> +    name: block.rpz
>>>>>> +    zonefile: /etc/unbound/zonefiles/block.rpz
>>>>>> +    rpz-action-override: nxdomain
>>>>>> +    rpz-log: yes
>>>>>> +    rpz-log-name: block
>>>>>> +    rpz-signal-nxdomain-ra: yes
>>>>>> diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config
>>>>>> new file mode 100644
>>>>>> index 000000000..98dc0a4ca
>>>>>> --- /dev/null
>>>>>> +++ b/config/rpz/rpz-config
>>>>>> @@ -0,0 +1,194 @@
>>>>>> +#!/bin/bash
>>>>>> +###############################################################################
>>>>>> +#                                                                             #
>>>>>> +#  IPFire.org - A linux based firewall                                        #
>>>>>> +#  Copyright (C) 2024  IPFire Team  <info@ipfire.org>                         #
>>>>>> +#                                                                             #
>>>>>> +#  This program is free software: you can redistribute it and/or modify       #
>>>>>> +#  it under the terms of the GNU General Public License as published by       #
>>>>>> +#  the Free Software Foundation, either version 3 of the License, or          #
>>>>>> +#  (at your option) any later version.                                        #
>>>>>> +#                                                                             #
>>>>>> +#  This program is distributed in the hope that it will be useful,            #
>>>>>> +#  but WITHOUT ANY WARRANTY; without even the implied warranty of             #
>>>>>> +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the              #
>>>>>> +#  GNU General Public License for more details.                               #
>>>>>> +#                                                                             #
>>>>>> +#  You should have received a copy of the GNU General Public License          #
>>>>>> +#  along with this program.  If not, see <http://www.gnu.org/licenses/>.      #
>>>>>> +#                                                                             #
>>>>>> +###############################################################################
>>>>>> +
>>>>>> +# v22 - 2024-07-12
>>>>>> +
>>>>>> +###############     Functions     ###############
>>>>>> +
>>>>>> +msg_log () {
>>>>>> + /usr/bin/logger --tag "${tagName}" "$*"
>>>>>> + if tty --silent ; then
>>>>>> + echo "${tagName}:" "$*"
>>>>>> + fi
>>>>>> +}
>>>>>> +
>>>>>> +check_name () {
>>>>>> + local testName="${1}"
>>>>>> + # check for a valid name
>>>>>> + regex='^[a-zA-Z0-9_]+$'
> Is there a specific reason why you have chosen only letters, numbers and underscore and not also the dash. If you allowed that then you could use the validdomainname subroutine in the general-functions.pl file rather than creating another valid name function.
> Does unbound not accept dash's in the rpz config filename?

A dash did not work with RPZ. 

The RPZ "name"s appear in multiple places.  I needed something that would work with RPZ and work with a filename and work within the message log.  

>>>>>> + if [[ ! "${testName}" =~ $regex ]] ; then
>>>>>> + msg_log "error: rpz: the NAME is not valid: \"${testName}\". exit."
>>>>>> + exit 1
>>>>>> + fi
>>>>>> +}
>>>>>> +
>>>>>> +check_unbound_conf () {
>>>>>> + # check the above config files
>>>>>> + msg_log "info: rpz: check for errors with \"unbound-checkconf\""
>>>>>> + /usr/sbin/unbound-checkconf
> I think checkconf should be added into the unboundctrl.c file
> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/misc-progs/unboundctrl.c;h=86c6ac42b9010f9d77fa49754bf217875aed3cc0;hb=refs/heads/next
> so that the command can be ensured to be running in safe mode.

Yes, this can be changed.  I do not have a good understanding of the `*ctrl.c` programs so if this is needed, I would need assistance in adding it to c.


>>>>>> + exit_code=$?
>>>>>> + if [[ "${exit_code}" -ne 0 ]] ; then
>>>>>> + msg_log "error: rpz: unbound-checkconf. exit."
>>>>>> + exit "${exit_code}"
>>>>>> + fi
>>>>>> +}
>>>>>> +
>>>>>> +make_rpz_file () {
>>>>>> + local theType="${1}" # allow or block
>>>>>> +
>>>>>> + theList="/var/ipfire/rpz/${theType}list" # input user list of domains
>>>>>> + theZoneFile="/etc/unbound/zonefiles/${theType}.rpz" # output file for RPZ
>>>>>> +
>>>>>> + theAction='.'
>>>>>> + if [[ "${theType}" =~ "block" ]] ; then
>>>>>> + theAction='rpz-passthru.'
>>>>>> + fi
>>>>>> +
>>>>>> + # does a list exist?
>>>>>> + if [[ -s "${theList}" ]] ; then
>>>>>> + # drop any extra "blanks" and add "CNAME <RPZ action>." to each line
>>>>>> + actionList=$( /usr/bin/awk '{$1=$1};1' "${theList}" |
>>>>>> + /bin/sed "/^[^;].*[[:alnum:]]/ s|$| CNAME ${theAction}|" )
>>>>>> +
>>>>>> + msg_log "info: rpz: create zonefile for ${theList}"
>>>>>> +
>>>>>> + /bin/cat <<-EOF > "${theZoneFile}"
>>>>>> + ; Name: ${theType} list
>>>>>> + ; Last modified: $(date "+%Y-%m-%d at %H.%M.%S %Z")
>>>>>> + ;
>>>>>> + ; domains with actions list
>>>>>> + ;
>>>>>> + ${actionList}
>>>>>> + EOF
>>>>>> +
>>>>>> + # reload the zone that was just updated
>>>>>> + zoneBase=$( basename "${theZoneFile}" )
>>>>>> + /usr/sbin/unbound-control auth_zone_reload -q "${zoneBase}"
> This should also be run via the unboundctrl.c file especially as the unbound-control command is accessing remote systems.
>>>>>> + fi
>>>>>> +}
>>>>>> +
>>>>>> +###############       Main        ###############
>>>>>> +
>>>>>> +tagName="unbound"
>>>>>> +
>>>>>> +theAction="${1}" # input action
>>>>>> +theName="${2}" # input RPZ name
>>>>>> +theURL="${3}" # input RPZ URL
>>>>>> +
>>>>>> +check_name "${theName}" # is this a valid name?
>>>>>> +
>>>>>> +rpzConfig="/etc/unbound/local.d/${theName}.rpz.conf" # output zone conf file
>>>>>> +rpzFile="/etc/unbound/zonefiles/${theName}.rpz" # output for RPZ file
>>>>>> +
>>>>>> +case "${theAction}" in
>>>>>> +
>>>>>> + # add new rpz list
>>>>>> + add )
>>>>>> + # does this config already exist?  If yes, then exit
>>>>>> + if [[ -f "${rpzConfig}" ]] ; then
>>>>>> + msg_log "info: rpz: ${rpzConfig} already exists. exit"
>>>>>> + exit 1
>>>>>> + fi
>>>>>> +
>>>>>> + # is this a valid URL?
>>>>>> + regex='^https://[-[:alnum:]\+&@#/%?=~_|!:,.;]*[-[:alnum:]\+&@#/%=~_|]'
>>>>>> + if ! [[ "${theURL}" =~ $regex ]] ; then
>>>>>> + msg_log "error: rpz: the URL is not valid: \"${theURL}\". exit."
>>>>>> + exit 1
>>>>>> + fi
>>>>>> +
>>>>>> + # create the zone config file
>>>>>> + msg_log "info: rpz: add config file \"${theName}.rpz.conf\""
>>>>>> + cat <<-EOF > "${rpzConfig}"
>>>>>> + rpz:
>>>>>> + name: ${theName}.rpz
>>>>>> + zonefile: /etc/unbound/zonefiles/${theName}.rpz
>>>>>> + url:  ${theURL}
>>>>>> + rpz-action-override: nxdomain
>>>>>> + rpz-log: yes
>>>>>> + rpz-log-name: ${theName}
>>>>>> + rpz-signal-nxdomain-ra: yes
>>>>>> + EOF
>>>>>> +
>>>>>> + # set-up zone file
>>>>>> + /usr/bin/touch "${rpzFile}"
>>>>>> + # unbound requires these settings for rpz files
>>>>>> + /bin/chown --verbose nobody:nobody "${rpzFile}"
>>>>>> + /bin/chmod --verbose 644 "${rpzFile}"
> These should only be using --verbose while doing development. In production they should not be used as the development should have ensured that everything is working as required.

Yes, verbose can be removed.

>>>>>> + ;;
>>>>>> +
>>>>>> + # trash config file & rpz file
>>>>>> + remove )
>>>>>> + if ! [[ -f "${rpzConfig}" ]] ; then
>>>>>> + msg_log "info: rpz: ${rpzConfig} does not exist. exit"
>>>>>> + exit 1
>>>>>> + fi
>>>>>> +
>>>>>> + msg_log "info: rpz: remove config file & rpz file \"${theName}\""
>>>>>> + /bin/rm --verbose "${rpzConfig}"
>>>>>> + /bin/rm --verbose "${rpzFile}"
> Same thing with the --verbose.
>>>>>> +
>>>>>> + check_unbound_conf
>>>>>> + ;;
>>>>>> +
>>>>>> + # make a new allow or block rpz file
>>>>>> + make )
>>>>>> + case "${theName}" in
>>>>>> + allow )
>>>>>> + make_rpz_file allow
>>>>>> + ;;
>>>>>> +
>>>>>> + block )
>>>>>> + make_rpz_file block
>>>>>> + ;;
>>>>>> +
>>>>>> + allowblock )
>>>>>> + make_rpz_file allow
>>>>>> + make_rpz_file block
>>>>>> + ;;
>>>>>> +
>>>>>> + * )
>>>>>> + msg_log "error: rpz: the NAME is not valid: \"${theName}\". exit."
>>>>>> + exit 1
>>>>>> + ;;
>>>>>> + esac
>>>>>> +
>>>>>> + check_unbound_conf
>>>>>> + ;;
>>>>>> +
>>>>>> + *)
>>>>>> + msg_log "error: rpz: missing or incorrect parameter"
>>>>>> + /usr/bin/printf "Usage:  rpzConfig.sh <ACTION> <NAME> <URL>\n"
>>>>>> + exit 1
>>>>>> + ;;
>>>>>> +
>>>>>> +esac
>>>>>> +
>>>>>> +# reload due to the changes
>>>>>> +msg_log  "rpz: running \"unbound-control reload\""
>>>>>> +/usr/sbin/unbound-control reload
> Use unboundctrl
>>>>>> +exit_code=$?
>>>>>> +if [[ "${exit_code}" -ne 0 ]] ; then
>>>>>> + msg_log "error: rpz: unbound-control \"${theName}\". exit."
>>>>>> + exit "${exit_code}"
>>>>>> +fi
>>>>>> +
>>>>>> +exit
> This is a hell of a long bash script.

The above script is only 200 lines.

The below script is only 143 lines

> Did you consider doing it in Perl or python instead of bash.

No.  The script did not require the speed or complexity of Perl or Python  And I don’t have the skills needed to write in those languages. 

> Maybe it would be better in c even as then it would be compiled and could not be modified on the IPFire system if some malware got in.

If some odd malware got into IPFire I would venture to guess it would not go after just this code...

>>>>>> diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics
>>>>>> new file mode 100644
>>>>>> index 000000000..0f97c7911
>>>>>> --- /dev/null
>>>>>> +++ b/config/rpz/rpz-metrics
>>>>>> @@ -0,0 +1,143 @@
>>>>>> +#!/bin/bash
>>>>>> +###############################################################################
>>>>>> +#                                                                             #
>>>>>> +#  IPFire.org - A linux based firewall                                        #
>>>>>> +#  Copyright (C) 2024  IPFire Team  <info@ipfire.org>                         #
>>>>>> +#                                                                             #
>>>>>> +#  This program is free software: you can redistribute it and/or modify       #
>>>>>> +#  it under the terms of the GNU General Public License as published by       #
>>>>>> +#  the Free Software Foundation, either version 3 of the License, or          #
>>>>>> +#  (at your option) any later version.                                        #
>>>>>> +#                                                                             #
>>>>>> +#  This program is distributed in the hope that it will be useful,            #
>>>>>> +#  but WITHOUT ANY WARRANTY; without even the implied warranty of             #
>>>>>> +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the              #
>>>>>> +#  GNU General Public License for more details.                               #
>>>>>> +#                                                                             #
>>>>>> +#  You should have received a copy of the GNU General Public License          #
>>>>>> +#  along with this program.  If not, see <http://www.gnu.org/licenses/>.      #
>>>>>> +#                                                                             #
>>>>>> +###############################################################################
>>>>>> +
>>>>>> +# v18 on 2024-07-05
>>>>>> +
>>>>>> +###############       Main        ###############
>>>>>> +
>>>>>> +weeks="${1:-2}" # default to two message logs
>>>>>> +sortBy="${2:-name}"   # by name or by hits
>>>>>> +
>>>>>> +# get the list of message logs for N weeks
>>>>>> +messageLogs=$( find /var/log/messages* -type f |
>>>>>> + /usr/bin/sort --version-sort |
>>>>>> + head -"${weeks}" )
>>>>>> +
>>>>>> +# get the list of RPZ names & counts from the message log(s)
>>>>>> +rpzNameCount=$( for logf in ${messageLogs} ; do
>>>>>> + /usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" |
>>>>>> + /usr/bin/awk '$10 ~ /\[\w*]/ { print $10 }' ;
>>>>>> + done | /usr/bin/sort | /usr/bin/uniq --count )
>>>>>> +
>>>>>> +# flip results and remove brackets `[` and `]`
>>>>>> +rpzNameCount=$( /bin/echo "${rpzNameCount}" |
>>>>>> + /usr/bin/awk '{ print $2, $1 }' |
>>>>>> + /bin/sed --regexp-extended 's|^\[(.*)\]|\1|' )
>>>>>> +
>>>>>> +# grab only names
>>>>>> +rpzNames=$( /bin/echo "${rpzNameCount}" | /usr/bin/awk '{ print $1 }' )
>>>>>> +
>>>>>> +# get list of RPZ files
>>>>>> +rpzFileList=$( /bin/find /etc/unbound/zonefiles -type f -iname "*.rpz" )
>>>>>> +
>>>>>> +# get basename of those files
>>>>>> +rpzBaseNames=$( /bin/echo "${rpzFileList}" |
>>>>>> + /bin/sed 's|/etc/unbound/zonefiles/||g ; s|\.rpz||g ;' )
>>>>>> +
>>>>>> +# add to rpzNames
>>>>>> +rpzNames="${rpzNames}"$'\n'"${rpzBaseNames}"
>>>>>> +
>>>>>> +# drop duplicate names
>>>>>> +rpzNames=$( echo "${rpzNames}" | /usr/bin/sort --unique  )
>>>>>> +
>>>>>> +# get line count for each RPZ
>>>>>> +lineCount=$( /bin/echo "${rpzFileList}" | /usr/bin/xargs wc -l )
>>>>>> +
>>>>>> +# get comment line count and blank line count for each RPZ
>>>>>> +commentCount=$( /bin/echo "${rpzFileList}" |
>>>>>> + /usr/bin/xargs /bin/grep --count -e "^$" -e "^;" )
>>>>>> +
>>>>>> +# get modified date each RPZ
>>>>>> +modDateList=$( /bin/echo "${rpzFileList}" | xargs stat -c '%.10y  %n' )
>>>>>> +
>>>>>> +ucListAuthZones=$( /usr/sbin/unbound-control list_auth_zones )
>>>>>> +
>>>>>> +# get width of RPZ names
>>>>>> +pWidth=$( /bin/echo "${rpzNames}" | /usr/bin/awk '{ print $1"   " }' | wc -L )
>>>>>> +pFormat="%-${pWidth}s %-8s %-8s %-8s %10s %12s\n"
>>>>>> +
>>>>>> +# print title line
>>>>>> +printf "${pFormat}" "name" "hits" "active" "lines" "hits/line%" "last update"
>>>>>> +printf -- "--------------"
>>>>>> +
>>>>>> +theResults=""
>>>>>> +totalLines=0
>>>>>> +totalHits=0
>>>>>> +while read -r theName
>>>>>> +do
>>>>>> + printf -- "-" # pretend progress bar
>>>>>> + # get hit count
>>>>>> + theHits="0"
>>>>>> + if output=$( /bin/grep "^${theName}\s" <<< "${rpzNameCount}" ) ; then
>>>>>> + theHits=$( /bin/echo "${output}" |
>>>>>> + /usr/bin/awk '{ print $2 }' )
>>>>>> + totalHits=$(( totalHits + theHits ))
>>>>>> + fi
>>>>>> +
>>>>>> + # is this RPZ list active?
>>>>>> + theActive="disabled"
>>>>>> + if /bin/grep --quiet "^${theName}\.rpz" <<< "${ucListAuthZones}"
>>>>>> + then
>>>>>> + theActive="enabled"
>>>>>> + fi
>>>>>> +
>>>>>> + # get line count then subtract comment count and blank line count
>>>>>> + #  from total line count
>>>>>> + theLines="n/a"
>>>>>> + hitsPerLine="0"
>>>>>> + if output=$( /bin/grep --fixed-strings "/${theName}.rpz" <<< "${lineCount}" ) ; then
>>>>>> + theLines=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' )
>>>>>> + totalLines=$(( totalLines + theLines ))
>>>>>> +
>>>>>> + #hitsPerLine=$( echo "scale=0 ; $theHits / $theLines" | bc )
>>>>>> + hitsPerLine=$(( 100 * theHits / theLines ))
>>>>>> + fi
>>>>>> +
>>>>>> + # get modification date
>>>>>> + theModDate="n/a"
>>>>>> + if output=$( /bin/grep --fixed-strings "/${theName}.rpz" <<< "${modDateList}" ) ; then
>>>>>> + theModDate=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' )
>>>>>> + fi
>>>>>> +
>>>>>> + # add to results list
>>>>>> + theResults+="${theName} ${theHits} ${theActive} ${theLines} ${hitsPerLine} ${theModDate}"$'\n'
>>>>>> +done <<< "${rpzNames}"
>>>>>> +
>>>>>> +case "${sortBy}" in
>>>>>> + names|name) sortArg=(-k3,3r -k1,1) ;; # sort by "active" then by "name"
>>>>>> +
>>>>>> + hits|hit) sortArg=(-k3,3r -k2,2nr -k1,1) ;; # sort by "active" then by "hits" then by "name"
>>>>>> +
>>>>>> + lines|line) sortArg=(-k3,3r -k4,4nr -k1,1) ;; # sort by "active" then by "lines" then by "name"
>>>>>> +esac
>>>>>> +
>>>>>> +printf -- "--------------\n"
>>>>>> +# remove blank lines, sort, print as columns
>>>>>> +/bin/echo "${theResults}" |
>>>>>> + /usr/bin/awk '!/^[[:space:]]*$/' |
>>>>>> + /usr/bin/sort "${sortArg[@]}"    |
>>>>>> + /usr/bin/awk --assign=width="${pWidth}" \
>>>>>> + '{ printf "%-*s %-8s %-8s %-8s %10s %12s\n", width, $1, $2, $3, $4, $5, $6 }'
>>>>>> +
>>>>>> +printf "${pFormat}" "" "=======" "" "========" "" ""
>>>>>> +printf "${pFormat}" "Totals -->" "${totalHits}" "" "${totalLines}" "" ""
>>>>>> +
>>>>>> +exit
> If I understand right the rpz-metrics info is being shown on a weekly basis. Wouldn't this make more sense to create a log page for unbound-rpz that shows the results in a similar way to the Firewall Logs, IP Address Blocklist Logs etc. So you can see for a whole month or a specific day and you can also export the data into a text format on a page of the browser. Basically that would just be copying one of the existing cgi scripts and modifying the content elements.

Yes, I can do this.


>>>>>> diff --git a/config/rpz/rpz-sleep b/config/rpz/rpz-sleep
>>>>>> new file mode 100644
>>>>>> index 000000000..eeef1174a
>>>>>> --- /dev/null
>>>>>> +++ b/config/rpz/rpz-sleep
>>>>>> @@ -0,0 +1,58 @@
>>>>>> +#!/bin/bash
>>>>>> +###############################################################################
>>>>>> +#                                                                             #
>>>>>> +#  IPFire.org - A linux based firewall                                        #
>>>>>> +#  Copyright (C) 2024  IPFire Team  <info@ipfire.org>                         #
>>>>>> +#                                                                             #
>>>>>> +#  This program is free software: you can redistribute it and/or modify       #
>>>>>> +#  it under the terms of the GNU General Public License as published by       #
>>>>>> +#  the Free Software Foundation, either version 3 of the License, or          #
>>>>>> +#  (at your option) any later version.                                        #
>>>>>> +#                                                                             #
>>>>>> +#  This program is distributed in the hope that it will be useful,            #
>>>>>> +#  but WITHOUT ANY WARRANTY; without even the implied warranty of             #
>>>>>> +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the              #
>>>>>> +#  GNU General Public License for more details.                               #
>>>>>> +#                                                                             #
>>>>>> +#  You should have received a copy of the GNU General Public License          #
>>>>>> +#  along with this program.  If not, see <http://www.gnu.org/licenses/>.      #
>>>>>> +#                                                                             #
>>>>>> +###############################################################################
>>>>>> +
>>>>>> +# v04 on 2024-07-05
>>>>>> +
>>>>>> +###############     Functions     ###############
>>>>>> +
>>>>>> +# send message to message log
>>>>>> +msg_log () {
>>>>>> + /usr/bin/logger --tag "${tagName}" "$*"
>>>>>> + if /usr/bin/tty --silent ; then
>>>>>> + echo "${tagName}:" "$*"
>>>>>> + fi
>>>>>> +}
>>>>>> +
>>>>>> +###############       Main        ###############
>>>>>> +
>>>>>> +tagName="unbound"
>>>>>> +
>>>>>> +sleepTime="${1:-5m}" # default to sleep for 5m (5 minutes)
>>>>>> +
>>>>>> +zoneList=$( /usr/sbin/unbound-control list_auth_zones | /usr/bin/awk '{print $1}' )
>>>>>> +
>>>>>> +for zone in ${zoneList} ; do
>>>>>> + /usr/bin/printf "disable ${zone}\t"
>>>>>> + /usr/sbin/unbound-control rpz_disable "${zone}"
> unboundctrl
>>>>>> +done
>>>>>> +
>>>>>> +msg_log "info: rpz: disabled all zones for ${sleepTime}"
>>>>>> +
>>>>>> +/bin/sleep "${sleepTime}"
>>>>>> +
>>>>>> +for zone in ${zoneList} ; do
>>>>>> + /usr/bin/printf "enable ${zone}\t"
>>>>>> + /usr/sbin/unbound-control rpz_enable "${zone}"
> unboundctrl
>>>>>> +done
>>>>>> +
>>>>>> +msg_log "info: rpz: enabled all zones"
>>>>>> +
>>>>>> +exit
>>>>>> diff --git a/lfs/rpz b/lfs/rpz
>>>>>> new file mode 100644
>>>>>> index 000000000..319c10b7f
>>>>>> --- /dev/null
>>>>>> +++ b/lfs/rpz
>>>>>> @@ -0,0 +1,88 @@
>>>>>> +###############################################################################
>>>>>> +#                                                                             #
>>>>>> +# IPFire.org - A linux based firewall                                         #
>>>>>> +# Copyright (C) 2024  IPFire Team  <info@ipfire.org>                          #
>>>>>> +#                                                                             #
>>>>>> +# This program is free software: you can redistribute it and/or modify        #
>>>>>> +# it under the terms of the GNU General Public License as published by        #
>>>>>> +# the Free Software Foundation, either version 3 of the License, or           #
>>>>>> +# (at your option) any later version.                                         #
>>>>>> +#                                                                             #
>>>>>> +# This program is distributed in the hope that it will be useful,             #
>>>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of              #
>>>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the               #
>>>>>> +# GNU General Public License for more details.                                #
>>>>>> +#                                                                             #
>>>>>> +# You should have received a copy of the GNU General Public License           #
>>>>>> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.       #
>>>>>> +#                                                                             #
>>>>>> +###############################################################################
>>>>>> +
>>>>>> +###############################################################################
>>>>>> +# Definitions
>>>>>> +###############################################################################
>>>>>> +
>>>>>> +include Config
>>>>>> +
>>>>>> +SUMMARY    = response policy zone - RPZ reputation system for unbound DNS
>>>>>> +
>>>>>> +VER        = 1.0.0
>>>>>> +
>>>>>> +THISAPP    = rpz-$(VER)
>>>>>> +DIR_APP    = $(DIR_SRC)/$(THISAPP)
>>>>>> +TARGET     = $(DIR_INFO)/$(THISAPP)
>>>>>> +
>>>>>> +PROG       = rpz
>>>>>> +PAK_VER    = 1
>>>>>> +
>>>>>> +DEPS       =
>>>>>> +
>>>>>> +SERVICES   =
>>>>>> +
>>>>>> +###############################################################################
>>>>>> +# Top-level Rules
>>>>>> +###############################################################################
>>>>>> +
>>>>>> +install : $(TARGET)
>>>>>> +
>>>>>> +check :
>>>>>> +
>>>>>> +download :
>>>>>> +
>>>>>> +b2 :
>>>>>> +
>>>>>> +dist:
>>>>>> + @$(PAK)
>>>>>> +
>>>>>> +###############################################################################
>>>>>> +# Installation Details
>>>>>> +###############################################################################
>>>>>> +
>>>>>> +$(TARGET) :
>>>>>> + @$(PREBUILD)
>>>>>> + @rm -rf $(DIR_APP)
>>>>>> +
>>>>>> + # install RPZ scripts
>>>>>> + install -v -m 755 \
>>>>>> +  $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin
>>>>>> +
>>>>>> + # Install settings folder and two empty files
>>>>>> + mkdir -pv /var/ipfire/rpz
>>>>>> + touch /var/ipfire/rpz/allowlist
>>>>>> + touch /var/ipfire/rpz/blocklist
>>>>>> +
>>>>>> + # Add conf file to /etc directory
>>>>>> + cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d
>>>>>> +
>>>>>> + # create zonefiles directory for the RPZ files and add two empty RPZ
>>>>>> + #  files to avoid a unbound config error
>>>>>> + mkdir -pv /etc/unbound/zonefiles
>>>>>> + chown -v nobody:nobody /etc/unbound/zonefiles
>>>>>> + touch /etc/unbound/zonefiles/allow.rpz
>>>>>> + touch /etc/unbound/zonefiles/block.rpz
>>>>>> +
>>>>>> + # Install backup definition
>>>>>> + cp -vf $(DIR_CONF)/backup/includes/rpz /var/ipfire/backup/addons/includes/rpz
>>>>>> +
>>>>>> + @rm -rf $(DIR_APP)
>>>>>> + @$(POSTBUILD)
>>>>>> diff --git a/make.sh b/make.sh
>>>>>> index 9bbbeb0f1..886d3760a 100755
>>>>>> --- a/make.sh
>>>>>> +++ b/make.sh
>>>>>> @@ -1721,6 +1721,8 @@ buildipfire() {
>>>>>> lfsmake2 btrfs-progs
>>>>>> lfsmake2 inotify-tools
>>>>>> lfsmake2 grub-btrfs
>>>>>> +  lfsmake2 rpz
>>>>>> +
>>>>>> 
>>>>>> # Kernelbuild ... current we have no platform that need
>>>>>> # multi kernel builds so KCFG is empty
>>>>>> -- 
>>>>>> 2.30.2
>>>>>> 
> 
> -- 
> Sent from my laptop

diff mbox series

Patch

diff --git a/config/backup/includes/rpz b/config/backup/includes/rpz
new file mode 100644
index 000000000..4d59bb40c
--- /dev/null
+++ b/config/backup/includes/rpz
@@ -0,0 +1,5 @@ 
+/var/ipfire/rpz/allowlist
+/var/ipfire/rpz/blocklist
+/etc/unbound/zonefiles/allow.rpz
+/etc/unbound/zonefiles/block.rpz
+/etc/unbound/local.d/*rpz.conf
diff --git a/config/rootfiles/packages/rpz b/config/rootfiles/packages/rpz
new file mode 100644
index 000000000..2ffa715dd
--- /dev/null
+++ b/config/rootfiles/packages/rpz
@@ -0,0 +1,11 @@ 
+etc/unbound/local.d/00-rpz.conf
+etc/unbound/zonefiles
+etc/unbound/zonefiles/allow.rpz
+etc/unbound/zonefiles/block.rpz
+usr/sbin/rpz-config
+usr/sbin/rpz-metrics
+usr/sbin/rpz-sleep
+var/ipfire/backup/addons/includes/rpz
+var/ipfire/rpz
+var/ipfire/rpz/allowlist
+var/ipfire/rpz/blocklist
diff --git a/config/rpz/00-rpz.conf b/config/rpz/00-rpz.conf
new file mode 100644
index 000000000..72c1d12e5
--- /dev/null
+++ b/config/rpz/00-rpz.conf
@@ -0,0 +1,18 @@ 
+server:
+	module-config: "respip validator iterator"
+
+rpz:
+    name:					allow.rpz
+    zonefile:				/etc/unbound/zonefiles/allow.rpz
+    rpz-action-override:	passthru
+    rpz-log:				yes
+    rpz-log-name:			allow
+    rpz-signal-nxdomain-ra:	yes
+
+rpz:
+    name:					block.rpz
+    zonefile:				/etc/unbound/zonefiles/block.rpz
+    rpz-action-override:	nxdomain
+    rpz-log:				yes
+    rpz-log-name:			block
+    rpz-signal-nxdomain-ra:	yes
diff --git a/config/rpz/rpz-config b/config/rpz/rpz-config
new file mode 100644
index 000000000..98dc0a4ca
--- /dev/null
+++ b/config/rpz/rpz-config
@@ -0,0 +1,194 @@ 
+#!/bin/bash
+###############################################################################
+#                                                                             #
+#  IPFire.org - A linux based firewall                                        #
+#  Copyright (C) 2024  IPFire Team  <info@ipfire.org>                         #
+#                                                                             #
+#  This program is free software: you can redistribute it and/or modify       #
+#  it under the terms of the GNU General Public License as published by       #
+#  the Free Software Foundation, either version 3 of the License, or          #
+#  (at your option) any later version.                                        #
+#                                                                             #
+#  This program is distributed in the hope that it will be useful,            #
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of             #
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the              #
+#  GNU General Public License for more details.                               #
+#                                                                             #
+#  You should have received a copy of the GNU General Public License          #
+#  along with this program.  If not, see <http://www.gnu.org/licenses/>.      #
+#                                                                             #
+###############################################################################
+
+#	v22 - 2024-07-12
+
+###############     Functions     ###############
+
+msg_log () {
+	/usr/bin/logger --tag "${tagName}" "$*"
+	if tty --silent ; then
+		echo "${tagName}:" "$*"
+	fi
+}
+
+check_name () {
+	local testName="${1}"
+	#	check for a valid name
+	regex='^[a-zA-Z0-9_]+$'
+	if [[ ! "${testName}" =~ $regex ]] ; then
+		msg_log "error: rpz: the NAME is not valid: \"${testName}\". exit."
+		exit 1
+	fi
+}
+
+check_unbound_conf () {
+	#	check the above config files
+	msg_log "info: rpz: check for errors with \"unbound-checkconf\""
+	/usr/sbin/unbound-checkconf
+	exit_code=$?
+	if [[ "${exit_code}" -ne 0 ]] ; then
+		msg_log "error: rpz: unbound-checkconf. exit."
+		exit "${exit_code}"
+	fi
+}
+
+make_rpz_file () {
+	local theType="${1}"	#	allow or block
+
+	theList="/var/ipfire/rpz/${theType}list"				#	input user list of domains
+	theZoneFile="/etc/unbound/zonefiles/${theType}.rpz"		#	output file for RPZ
+
+	theAction='.'
+	if [[ "${theType}" =~ "block" ]] ; then
+		theAction='rpz-passthru.'
+	fi
+
+	#	does a list exist?
+	if [[ -s "${theList}" ]] ; then
+		#	drop any extra "blanks" and add "CNAME <RPZ action>." to each line
+		actionList=$( /usr/bin/awk '{$1=$1};1' "${theList}" |
+			/bin/sed "/^[^;].*[[:alnum:]]/ s|$|	CNAME	${theAction}|" )
+
+		msg_log "info: rpz: create zonefile for ${theList}"
+
+		/bin/cat <<-EOF > "${theZoneFile}"
+		; Name:				${theType} list
+		; Last modified:	$(date "+%Y-%m-%d at %H.%M.%S %Z")
+		;
+		;	domains with actions list
+		;
+		${actionList}
+		EOF
+
+		#	reload the zone that was just updated
+		zoneBase=$( basename "${theZoneFile}" )
+		/usr/sbin/unbound-control auth_zone_reload -q "${zoneBase}"
+	fi
+}
+
+###############       Main        ###############
+
+tagName="unbound"
+
+theAction="${1}"										#	input action
+theName="${2}"											#	input RPZ name
+theURL="${3}"											#	input RPZ URL
+
+check_name "${theName}"									#	is this a valid name?
+
+rpzConfig="/etc/unbound/local.d/${theName}.rpz.conf"	#	output zone conf file
+rpzFile="/etc/unbound/zonefiles/${theName}.rpz"			#	output for RPZ file
+
+case "${theAction}" in
+
+	#	add new rpz list
+	add )
+		#	does this config already exist?  If yes, then exit
+		if [[ -f "${rpzConfig}" ]] ; then
+			msg_log "info: rpz: ${rpzConfig} already exists. exit"
+			exit 1
+		fi
+
+		#	is this a valid URL?
+		regex='^https://[-[:alnum:]\+&@#/%?=~_|!:,.;]*[-[:alnum:]\+&@#/%=~_|]'
+		if ! [[ "${theURL}" =~ $regex ]] ; then
+			msg_log "error: rpz: the URL is not valid: \"${theURL}\". exit."
+			exit 1
+		fi
+
+		#	create the zone config file
+		msg_log "info: rpz: add config file \"${theName}.rpz.conf\""
+		cat <<-EOF > "${rpzConfig}"
+		rpz:
+			name:					${theName}.rpz
+			zonefile:				/etc/unbound/zonefiles/${theName}.rpz
+			url:  ${theURL}
+			rpz-action-override:	nxdomain
+			rpz-log:				yes
+			rpz-log-name:			${theName}
+			rpz-signal-nxdomain-ra:	yes
+		EOF
+
+		#	set-up zone file
+		/usr/bin/touch "${rpzFile}"
+		#	unbound requires these settings for rpz files
+		/bin/chown --verbose nobody:nobody "${rpzFile}"
+		/bin/chmod --verbose 644 "${rpzFile}"
+		;;
+
+	#	trash config file & rpz file
+	remove )
+		if ! [[ -f "${rpzConfig}" ]] ; then
+			msg_log "info: rpz: ${rpzConfig} does not exist. exit"
+			exit 1
+		fi
+
+		msg_log "info: rpz: remove config file & rpz file \"${theName}\""
+		/bin/rm --verbose "${rpzConfig}"
+		/bin/rm --verbose "${rpzFile}"
+
+		check_unbound_conf
+		;;
+
+	#	make a new allow or block rpz file
+	make )
+		case "${theName}" in
+			allow )
+				make_rpz_file allow
+			;;
+
+			block )
+				make_rpz_file block
+			;;
+
+			allowblock )
+				make_rpz_file allow
+				make_rpz_file block
+			;;
+
+			* )
+				msg_log "error: rpz: the NAME is not valid: \"${theName}\". exit."
+				exit 1
+			;;
+		esac
+
+		check_unbound_conf
+		;;
+
+	*)
+		msg_log "error: rpz: missing or incorrect parameter"
+		/usr/bin/printf "Usage:  rpzConfig.sh <ACTION> <NAME> <URL>\n"
+		exit 1
+		;;
+
+esac
+
+#	reload due to the changes
+msg_log  "rpz: running \"unbound-control reload\""
+/usr/sbin/unbound-control reload
+exit_code=$?
+if [[ "${exit_code}" -ne 0 ]] ; then
+	msg_log "error: rpz: unbound-control \"${theName}\". exit."
+	exit "${exit_code}"
+fi
+
+exit
diff --git a/config/rpz/rpz-metrics b/config/rpz/rpz-metrics
new file mode 100644
index 000000000..0f97c7911
--- /dev/null
+++ b/config/rpz/rpz-metrics
@@ -0,0 +1,143 @@ 
+#!/bin/bash
+###############################################################################
+#                                                                             #
+#  IPFire.org - A linux based firewall                                        #
+#  Copyright (C) 2024  IPFire Team  <info@ipfire.org>                         #
+#                                                                             #
+#  This program is free software: you can redistribute it and/or modify       #
+#  it under the terms of the GNU General Public License as published by       #
+#  the Free Software Foundation, either version 3 of the License, or          #
+#  (at your option) any later version.                                        #
+#                                                                             #
+#  This program is distributed in the hope that it will be useful,            #
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of             #
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the              #
+#  GNU General Public License for more details.                               #
+#                                                                             #
+#  You should have received a copy of the GNU General Public License          #
+#  along with this program.  If not, see <http://www.gnu.org/licenses/>.      #
+#                                                                             #
+###############################################################################
+
+#	v18 on 2024-07-05
+
+###############       Main        ###############
+
+weeks="${1:-2}"			#	default to two message logs
+sortBy="${2:-name}"  	#	by name or by hits
+
+#	get the list of message logs for N weeks
+messageLogs=$( find /var/log/messages* -type f |
+	/usr/bin/sort --version-sort |
+	head -"${weeks}" )
+
+#	get the list of RPZ names & counts from the message log(s)
+rpzNameCount=$( for logf in ${messageLogs} ; do
+	/usr/bin/zgrep --text --fixed-strings 'info: rpz: applied' "${logf}" |
+	/usr/bin/awk '$10 ~ /\[\w*]/ { print $10 }' ;
+	done | /usr/bin/sort | /usr/bin/uniq --count )
+
+#	flip results and remove brackets `[` and `]`
+rpzNameCount=$( /bin/echo "${rpzNameCount}" |
+	/usr/bin/awk '{ print $2, $1 }' |
+	/bin/sed --regexp-extended 's|^\[(.*)\]|\1|' )
+
+#	grab only names
+rpzNames=$( /bin/echo "${rpzNameCount}" | /usr/bin/awk '{ print $1 }' )
+
+#	get list of RPZ files
+rpzFileList=$( /bin/find /etc/unbound/zonefiles -type f -iname "*.rpz" )
+
+#	get basename of those files
+rpzBaseNames=$( /bin/echo "${rpzFileList}" |
+	/bin/sed 's|/etc/unbound/zonefiles/||g ; s|\.rpz||g ;' )
+
+#	add to rpzNames
+rpzNames="${rpzNames}"$'\n'"${rpzBaseNames}"
+
+#	drop duplicate names
+rpzNames=$( echo "${rpzNames}" | /usr/bin/sort --unique  )
+
+#	get line count for each RPZ
+lineCount=$( /bin/echo "${rpzFileList}" | /usr/bin/xargs wc -l )
+
+#	get comment line count and blank line count for each RPZ
+commentCount=$( /bin/echo "${rpzFileList}" |
+	/usr/bin/xargs /bin/grep --count -e "^$" -e "^;" )
+
+#	get modified date each RPZ
+modDateList=$( /bin/echo "${rpzFileList}" | xargs stat -c '%.10y  %n' )
+
+ucListAuthZones=$( /usr/sbin/unbound-control list_auth_zones )
+
+#	get width of RPZ names
+pWidth=$( /bin/echo "${rpzNames}" | /usr/bin/awk '{ print $1"   " }' | wc -L )
+pFormat="%-${pWidth}s %-8s %-8s %-8s %10s %12s\n"
+
+#	print title line
+printf "${pFormat}" "name" "hits" "active" "lines" "hits/line%" "last update"
+printf -- "--------------"
+
+theResults=""
+totalLines=0
+totalHits=0
+while read -r theName
+do
+	printf -- "-"									#	pretend progress bar
+	#	get hit count
+	theHits="0"
+	if output=$( /bin/grep "^${theName}\s" <<< "${rpzNameCount}" ) ; then
+		theHits=$( /bin/echo "${output}" |
+		/usr/bin/awk '{ print $2 }' )
+		totalHits=$(( totalHits + theHits ))
+	fi
+
+	#	is this RPZ list active?
+	theActive="disabled"
+	if /bin/grep --quiet "^${theName}\.rpz" <<< "${ucListAuthZones}"
+	then
+		theActive="enabled"
+	fi
+
+	#	get line count then subtract comment count and blank line count
+	#	  from total line count
+	theLines="n/a"
+	hitsPerLine="0"
+	if output=$( /bin/grep --fixed-strings "/${theName}.rpz" <<< "${lineCount}" ) ; then
+		theLines=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' )
+		totalLines=$(( totalLines + theLines ))
+
+		#hitsPerLine=$( echo "scale=0 ; $theHits / $theLines" | bc )
+		hitsPerLine=$(( 100 * theHits / theLines ))
+	fi
+
+	#	get modification date
+	theModDate="n/a"
+	if output=$( /bin/grep --fixed-strings "/${theName}.rpz" <<< "${modDateList}" ) ; then
+		theModDate=$( /bin/echo "${output}" | /usr/bin/awk '{ print $1 }' )
+	fi
+
+	#	add to results list
+	theResults+="${theName} ${theHits} ${theActive} ${theLines} ${hitsPerLine} ${theModDate}"$'\n'
+done <<< "${rpzNames}"
+
+case "${sortBy}" in
+	names|name)	sortArg=(-k3,3r -k1,1)			;;	#	sort by "active" then by "name"
+
+	hits|hit)	sortArg=(-k3,3r -k2,2nr -k1,1)	;;	#	sort by "active" then by "hits" then by "name"
+
+	lines|line)	sortArg=(-k3,3r -k4,4nr -k1,1)	;;	#	sort by "active" then by "lines" then by "name"
+esac
+
+printf -- "--------------\n"
+#	remove blank lines, sort, print as columns
+/bin/echo "${theResults}" |
+	/usr/bin/awk '!/^[[:space:]]*$/' |
+	/usr/bin/sort "${sortArg[@]}"    |
+	/usr/bin/awk --assign=width="${pWidth}" \
+		'{ printf "%-*s %-8s %-8s %-8s %10s %12s\n", width, $1, $2, $3, $4, $5, $6 }'
+
+printf "${pFormat}" "" "=======" "" "========" "" ""
+printf "${pFormat}" "Totals -->" "${totalHits}" "" "${totalLines}" "" ""
+
+exit
diff --git a/config/rpz/rpz-sleep b/config/rpz/rpz-sleep
new file mode 100644
index 000000000..eeef1174a
--- /dev/null
+++ b/config/rpz/rpz-sleep
@@ -0,0 +1,58 @@ 
+#!/bin/bash
+###############################################################################
+#                                                                             #
+#  IPFire.org - A linux based firewall                                        #
+#  Copyright (C) 2024  IPFire Team  <info@ipfire.org>                         #
+#                                                                             #
+#  This program is free software: you can redistribute it and/or modify       #
+#  it under the terms of the GNU General Public License as published by       #
+#  the Free Software Foundation, either version 3 of the License, or          #
+#  (at your option) any later version.                                        #
+#                                                                             #
+#  This program is distributed in the hope that it will be useful,            #
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of             #
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the              #
+#  GNU General Public License for more details.                               #
+#                                                                             #
+#  You should have received a copy of the GNU General Public License          #
+#  along with this program.  If not, see <http://www.gnu.org/licenses/>.      #
+#                                                                             #
+###############################################################################
+
+#	v04 on 2024-07-05
+
+###############     Functions     ###############
+
+#	send message to message log
+msg_log () {
+	/usr/bin/logger --tag "${tagName}" "$*"
+	if /usr/bin/tty --silent ; then
+		echo "${tagName}:" "$*"
+	fi
+}
+
+###############       Main        ###############
+
+tagName="unbound"
+
+sleepTime="${1:-5m}"			#	default to sleep for 5m (5 minutes)
+
+zoneList=$( /usr/sbin/unbound-control list_auth_zones | /usr/bin/awk '{print $1}' )
+
+for zone in ${zoneList} ; do
+	/usr/bin/printf "disable ${zone}\t"
+	/usr/sbin/unbound-control rpz_disable "${zone}"
+done
+
+msg_log "info: rpz: disabled all zones for ${sleepTime}"
+
+/bin/sleep "${sleepTime}"
+
+for zone in ${zoneList} ; do
+	/usr/bin/printf "enable ${zone}\t"
+	/usr/sbin/unbound-control rpz_enable "${zone}"
+done
+
+msg_log "info: rpz: enabled all zones"
+
+exit
diff --git a/lfs/rpz b/lfs/rpz
new file mode 100644
index 000000000..319c10b7f
--- /dev/null
+++ b/lfs/rpz
@@ -0,0 +1,88 @@ 
+###############################################################################
+#                                                                             #
+# IPFire.org - A linux based firewall                                         #
+# Copyright (C) 2024  IPFire Team  <info@ipfire.org>                          #
+#                                                                             #
+# This program is free software: you can redistribute it and/or modify        #
+# it under the terms of the GNU General Public License as published by        #
+# the Free Software Foundation, either version 3 of the License, or           #
+# (at your option) any later version.                                         #
+#                                                                             #
+# This program is distributed in the hope that it will be useful,             #
+# but WITHOUT ANY WARRANTY; without even the implied warranty of              #
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the               #
+# GNU General Public License for more details.                                #
+#                                                                             #
+# You should have received a copy of the GNU General Public License           #
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.       #
+#                                                                             #
+###############################################################################
+
+###############################################################################
+# Definitions
+###############################################################################
+
+include Config
+
+SUMMARY    = response policy zone - RPZ reputation system for unbound DNS
+
+VER        = 1.0.0
+
+THISAPP    = rpz-$(VER)
+DIR_APP    = $(DIR_SRC)/$(THISAPP)
+TARGET     = $(DIR_INFO)/$(THISAPP)
+
+PROG       = rpz
+PAK_VER    = 1
+
+DEPS       =
+
+SERVICES   =
+
+###############################################################################
+# Top-level Rules
+###############################################################################
+
+install : $(TARGET)
+
+check :
+
+download :
+
+b2 :
+
+dist:
+	@$(PAK)
+
+###############################################################################
+# Installation Details
+###############################################################################
+
+$(TARGET) :
+	@$(PREBUILD)
+	@rm -rf $(DIR_APP)
+
+	#	install RPZ scripts
+	install -v -m 755 \
+	  $(DIR_CONF)/rpz/{rpz-config,rpz-metrics,rpz-sleep} -t /usr/sbin
+
+	# Install settings folder and two empty files
+	mkdir -pv /var/ipfire/rpz
+	touch /var/ipfire/rpz/allowlist
+	touch /var/ipfire/rpz/blocklist
+
+	# Add conf file to /etc directory
+	cp -vf $(DIR_CONF)/rpz/00-rpz.conf /etc/unbound/local.d
+
+	#	create zonefiles directory for the RPZ files and add two empty RPZ
+	#	  files to avoid a unbound config error
+	mkdir -pv /etc/unbound/zonefiles
+	chown -v nobody:nobody /etc/unbound/zonefiles
+	touch /etc/unbound/zonefiles/allow.rpz
+	touch /etc/unbound/zonefiles/block.rpz
+
+	# Install backup definition
+	cp -vf $(DIR_CONF)/backup/includes/rpz /var/ipfire/backup/addons/includes/rpz
+
+	@rm -rf $(DIR_APP)
+	@$(POSTBUILD)
diff --git a/make.sh b/make.sh
index 9bbbeb0f1..886d3760a 100755
--- a/make.sh
+++ b/make.sh
@@ -1721,6 +1721,8 @@  buildipfire() {
   lfsmake2 btrfs-progs
   lfsmake2 inotify-tools
   lfsmake2 grub-btrfs
+  lfsmake2 rpz
+
 
   # Kernelbuild ... current we have no platform that need
   # multi kernel builds so KCFG is empty